OpenAI's featured chatbot recommends $200,000 in surgeries while promoting incel ideology
OpenAI is featuring a "Looksmaxxing GPT" that provides "PSL ratings" for photos. It will rate people as "subhuman", and advise men to get invasive procedures like jaw surgery to "increase their sexual market value" among women, who it describes as "hypergamous by nature".
"PSL" is short for "PUAhate Sluthate Lookism", a group of three manosphere/incel communities. PUAHate came to public attention after Elliot Rodger's 2014 mass shooting in Isla Vista, California, when it came out that he frequented the platform to talk about his "incel" status.
After uploading a photo, this GPT gives ratings like "Subhuman", "Low Normie / High Subhuman Borderline (White)", or "high-tier normie, borderline Chadlite".
It proactively offers to provide "hardmaxxing" (medical interventions like plastic surgery) or "softmaxxing" advice (grooming, fitness training, etc).
Asked for "hardmaxxing" advice, the chatbot provides specific surgical recommendations, complete with references to Dr. Barry Eppley, who is something of a celebrity plastic surgeon among incels.
It readily goes into detail about how "Women are hypergamous by nature", "get endless attention and options", and care most about looks and status.
I'm about to take a two-ish week sabbatical so I can (once again!) rewrite the Trump chapter of my Enshittification book (October 2025), and so that I can get my (thankfully very treatable) cancer irradiated:
https://pluralistic.net/2024/11/05/carcinoma-angels/#squeaky-nail
While I'm away, here are some things I'd like to call your attention to. First, some good news: the Washington Post Tech Guild just won a historic union vote with a giant majority, despite the vicious union-hating owner of the Post, a Mr Jeffrey Preston Bezos:
Even more good news: the GOP have ratfucked themselves, doing the work that our Democratic Party leaders can't or won't do. In overruling the parliamentarian in a bid to arrogate to themselves the power to kill California emission standards, Republican Senators have opened the door for Democrats to seize 10 hours of debate time for every single change Trump makes to federal regulations. These debates take precedence over all Senate business. They can even go back in time and demand 10 hours of floor debate on every agency action for the past 60 days. Basically, that means that Senate Dems can tie up the Senate until the 2026 mid-terms and beyond:
https://prospect.org/politics/2025-05-28-senate-democrats-stop-big-beautiful-bill/
Will they? I mean, it's the kind of tactic Mitch McConnell would have leapt at without even bothering to fully raise the lid of his sarcophagus. Chuck Schumer? I dunno. Maybe if we gave him a ping-pong paddle with some stylish sans serif text invoking each debate?
https://www.youtube.com/watch?v=KADW3ZRZLVI
That's some good news I'm going to take with me into my coming break. I've really cleared my calendar for this time off, finishing up my CBC podcast "Understood: Who Broke the Internet?" just in the nick of time:
https://pluralistic.net/2025/05/26/babyish-radical-extremists/#cancon
The series prompted Harrison Mooney to do a long, fantastic interview with me for The Tyee, which sets out the series' thesis and call to action very well:
https://thetyee.ca/Culture/2025/05/27/Musk-Zuck-Use-Our-Love-Hostage/
If you're as pissed off about enshittification as I am and you happen to live in NYC, there's a support group for you! This week, I heard from a reader who's organized a monthly open mic "Evening on Enshittification," where attendees present and learn about different kinds of enshittification, from AI to dating and beyond:
https://partiful.com/e/Li1DGg7x5ohmCOf2hAkj
And if you're on the other coast, you can catch me TOMORROW in Seattle at the Cascade PBS Ideas Festival, where I'll be onstage with the folks from NPR's On The Media:
https://www.cascadepbs.org/festival/speaker/cory-doctorow
If a couple weeks without me is too much, please consider dialing into my virtual keynote for Fediforum on June 5:
https://fediforum.org/2025-06/
And of course, when I get back, I'm going to be finishing off my tour for Picks and Shovels with gigs in Portland, London, and Manchester:
I've got a packed schedule in Portland: first, I'm doing a keynote at the Teardown conference on Friday, June 20:
https://www.crowdsupply.com/teardown/portland-2025
Followed by a bookstore event with bunnie Huang at the Lloyd Center Barnes and Noble:
https://stores.barnesandnoble.com/event/9780062183697-0
And a library gig on June 20 in Tualatin:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
Londoners, you can catch me at the How To Academy on July 1, where I'll be doing a Canada Day book event with the amazing Riley Quinn, showrunner for Trashfuture:
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
And then I'm doing a bookstore event in Manchester at Blackwells on July 2:
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Followed by a July 4 keynote for the Co-operatives UK Congress in Manchester:
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
#20yrsago Copyright prevented transmission of Beatles music to aliens https://memex.craphound.com/2005/05/29/copyright-prevented-transmission-of-beatles-music-to-aliens/
#10yrsago Pornoscanner lobbyist’s new job: overseeing TSA spending https://theintercept.com/2015/05/27/tsa-body-scanner-lobbyist-takes-congressional-job-overseeing-spending-tsa-security/
#5yrsago Canadian newsrooms restructure as co-ops https://pluralistic.net/2020/05/29/mind-control-skepticism/#co-ops
#5yrsago GOP lawmaker hid his diagnosis from Democrats https://pluralistic.net/2020/05/29/mind-control-skepticism/#flu-klux-klan
#5yrsago Walmart's crummy anti-theft AI https://pluralistic.net/2020/05/29/mind-control-skepticism/#neverseen
#5yrsago Masks work https://pluralistic.net/2020/05/29/mind-control-skepticism/#mask-up
#5yrsago What to do about the police https://pluralistic.net/2020/05/29/mind-control-skepticism/#qualified-immunity
#5yrsago Private equity goes mainstream https://pluralistic.net/2020/05/29/mind-control-skepticism/#looters
#5yrsago How the IoT reinforces gentrification https://pluralistic.net/2020/05/29/mind-control-skepticism/#automated-karens
#5yrsago Big Tech distorts our discourse https://pluralistic.net/2020/05/29/mind-control-skepticism/#tech-unexceptionalism
#5yrsago Bus drivers refuse to take arrested protesters to jail https://pluralistic.net/2020/05/30/up-is-not-down/#solidarity
#5yrsago Why I haven't written about CDA 230 https://pluralistic.net/2020/05/30/up-is-not-down/#cda230
#5yrsago Australia caves on "robodebt" https://pluralistic.net/2020/05/30/up-is-not-down/#robodebt
#1yrago Real innovation vs Silicon Valley nonsense https://pluralistic.net/2024/05/30/posiwid/#social-cost-of-carbon
#1yrago The Pizzaburger Presidency https://pluralistic.net/2024/05/29/sub-bushel-comms-strategy/#nothing-would-fundamentally-change
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Kick 'Em In the Dongle (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16148346-kick-em-in-the-dongle
The Big Story
https://www.seekyoursounds.com/podcasts/the-big-story/cory-doctorow-explains-who-broke-the-internet
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
there is no feeling like finishing book two in a series you're enjoying and then discovering that book three is coming out only days later
It’s been two weeks since Coinbase disclosed its data breach, and so far we’re up to:
• 8 customer class actions against Coinbase
• 1 shareholder class action against Coinbase
• 1 customer class action against TaskUs, a contractor the plaintiffs allege was responsible for the breach.
Donald Trump is many things: a racist, an authoritarian, a rapist… but what he is, and has always been, above all and from the very start, is a scammer:
The election of Donald Trump feeds many needs in the right wing coalition: the libidinal pleasure of seeing trans people, migrants, and anyone who isn't white getting terrorized by masked thugs and swivel-eyed loons; massive tax cuts for the oligarch class, especially those who (like Trump) inherit their wealth; the gutting of public education and the destruction of the barrier between church and state.
But the most important, best-served constituency in the Trump coalition is scammers. This has been his promise since his first campaign, when he boasted on national television that he cheated on his taxes because "that makes me smart":
https://pluralistic.net/2024/12/04/its-not-a-lie/#its-a-premature-truth
Trump – who has run multiple pyramid schemes – is gutting enforcement against Ponzi scammers:
https://pluralistic.net/2025/05/05/free-enterprise-system/#amway-or-the-highway
Trump – who has repeatedly lied to investors and customers about his business plans – has elevated the 21st century's most notorious defrauder of investors and customers and put him in charge of restructuring the US government:
https://www.wired.com/story/theres-a-very-simple-pattern-to-elon-musks-broken-promises/
Under the previous administration, agencies like the FTC and (eventually) the DOT revived their powers to block "unfair and deceptive" commercial practices, from fast food restaurants forcing cashiers to sign noncompete "agreements" to Southwest Airlines selling tickets for planes that didn't exist and cancelling the least-booked flights every morning, stranding passengers:
https://pluralistic.net/2023/01/16/for-petes-sake/#unfair-and-deceptive
But this administration has reversed course, demonstrating time and again that there is no rugpull too petty or cruel to pass up. Take this one: RFK Jr blames processed food for health problems on indigenous reservations, so the USDA is…killing the food banks that provide local, fresh produce to indigenous reservations:
https://www.propublica.org/article/tribal-food-grant-cuts-trump-rfk-jr
Trump is the ripoff candidate, the King of the Scam. Take Trumpcoin: the President of the United States of America issued his own shitcoin through which people could openly offer him bribes. Then he announced that he would reward the top 220 bribes with a private dinner where he could be directly lobbied for presidential favors. The top bribes were proffered by a rogues' gallery of domestic and foreign scammers and crooks, including billionaire felons seeking pardons for stealing from millions of normal investors:
https://www.citationneeded.news/trump-memecoin-dinner-guests/
But the actual attendees didn't really bribe Trump. They scammed him by taking short positions on their Trumpcoin holdings:
https://www.theverge.com/cryptocurrency/674327/trump-coin-short-sell-hedge-contest-dinner-winner
And the instant the guestlist was announced, they liquidated their holdings, tanking Trumpcoin's price:
https://protos.com/trump-token-15-since-dinner-as-40-of-guests-dump-by-dessert/
But Trump scammed them, by turning the "private dinner with the president" into a whistle-stop in which he breezed into the hall, spoke for 15 minutes, and then left by helicopter without meeting with any of his "guests":
https://link.nymag.com/view/640f640416f22cc291043cebntiap.15g1/0da0f946
This is the kind of scammer inception that threatens to create a black hole of superdense Scammium-238 that sucks the whole country into the scam singularity. But scammers don't care – the Art of the Scam is to race across the river on the backs of alligators without losing a leg. These people aren't long-termists, they've always got one foot out the door.
The Trump II presidency is the most scam-friendly presidency in history, and everyone knows it. The scammers are lining up to get their scams okayed – like Verizon, which wants to bring back the long-banned practice of locking your phone so that it only works on their network and refusing to unlock it:
Verizon had to promise to unlock its customers' phones as a condition of being given exclusive control over billions' worth of the public spectrum, and in exchange for the right to buy its competitor Tracphone, in a nakedly anticompetitive merger. They stand every chance of getting out of these obligations, because Trump's scammer FCC chair Brendan Carr has launched an initiative called "Delete, Delete, Delete" in which he is inviting scammers to nominate regulations that protect the public so he can get rid of them:
Trump handed control over the Department of Education to the scammer Linda McMahon, wife of the scammer Vince McMahon, who bought up every wrestling league so that his performers had nowhere else to go, then falsely reclassified them as independent contractors, taking away their health insurance and leaving them to beg for Gofundme pennies so they could die with dignity in their mid-fifties from their work-related injuries:
https://www.youtube.com/watch?v=m8UQ4O7UiDs
Linda McMahon, a scammer, has thrown 5.3 million student borrowers into collections, calling them "irresponsible." Another 5.59 million will enter default in the next six months. She's opened the door for these ex-students to have money taken out of their Social Security and other government payments, leaving them only $750/month to live on (this figure hasn't been inflation-adjusted since 1996). Each time they get dinged, another $20 in fees (for having their money taken away) will be added to their debts.
As David Dayen writes for The American Prospect, McMahon is also unleashing America's scammiest loan servicers on student borrowers:
https://prospect.org/education/2025-05-27-borrowers-besieged-student-debt/
Take Maximus (owners of the Default Resolution Group), which has been repeatedly sued for misleading debtors, illegally taking money out of their accounts, and failing to note which borrowers were legally protected from having their funds taken. DoE loan servicers have a long and dishonorable tradition of ripping off debtors – like the giant Navient, which was eventually barred from servicing student loans altogether:
Servicers like Maximus routinely fail in their legal obligation to inform debtors of their rights, and when debtors know their rights – for example, the right to contest wage garnishment – Maximus just ignores their paperwork:
https://escholarship.org/content/qt6rc8r76c/qt6rc8r76c_noSplash_41d21b1b129a7224ae49ae65621f8dda.pdf
Dayen cites a long list of Maximus's documented sins – for example, systematically failing to inform debtors that if they enter loan rehabilitation and make nine payments over ten months, they can avoid default. Every time a borrower completes this process, they come off of Maximus's books, so it's not hard to see why Maximus would fail to tell them about it.
The rules are incredibly complex. If you do enter loan rehabilitation, you can still have payments garnished from your paycheck until you've completed five separate payments. You can consolidate your loans – and possibly even get them canceled after 20-25 years – but that also erases any credit you've managed toward debt forgiveness, and starts piling interest on the new loan. If you get any of this wrong, you have to pay more every month, not less.
The CARES Act ordered Maximus to stop taking money away from debtors – but some workers had their paychecks docked for 17 months after the order went into effect, including borrowers who had already paid their debts in full:
https://protectborrowers.org/wp-content/uploads/2022/12/SBPC_AWG_Final.pdf
Buried in all the complexity around student debt are several obscure, difficult to navigate paths that can reduce or eliminate your payments. But these are almost impossible to find (the Office of Federal Student Aid's website devotes 39 words to them, several screens down the page). This has created an opportunity for a giant scam industry that takes desperate debtors' last dollars to fill in paperwork they could easily complete for themselves, or to pretend to fill in that paperwork and just walk away from them:
These scam agencies charge thousands of dollars – up to 40% of your outstanding debt – for either doing nothing or doing something you can do yourself, but can't readily find out about. This can net them tens of millions of dollars, extracted from desperate debtors:
https://www.ftc.gov/system/files/ftc_gov/pdf/panda_prosperity_complaint.pdf
There's about to be a hell of lot more of those desperate debtors kicking around. The DOE is cutting $330 billion out of the student loan pool in order to fund Trump's tax-break for the ultra-rich. This will make government loans much more expensive for borrowers – but it'll also vastly reduce the amount that students can borrow through the government. The lifetime cap on undergrad loans is now $50k, and $100k for grad students. Pell grants are being taken away from anyone who isn't a full-time student with a 30-hour courseload (so the poorest students, who have to work part-time while they go to school, will be shut out of them).
Where will students get the rest of the money? From predatory lenders. Private student loans currently pay for 7% of borrowers' education expenses – soon, it could be 90%. This is great news for scam colleges, like Trump University, which encouraged students to take out predatory loans to pay sky-high tuition for a useless "education":
https://en.wikipedia.org/wiki/Trump_University
The Big Beautiful Bill contains numerous attacks on the safeguards against scam colleges. That means that even more public money will go to propping up fraudulent institutions, and naive students who are just looking to get an education will be on the hook to pay it back – with interest.
Dayen calls this "The Golden Age of Scams":
https://prospect.org/power/2025-05-27-golden-age-of-scams/
He likens the current griftogenic environment to the situation in Elizabethan England, where counterfeit money was in wide circulation. This prompted Sir Thomas Gresham to coin "Gresham's Law": "Bad money drives out good." Gresham observed that anyone who was passed a bad coin would try to spend it as soon as possible, and if they managed to do so, the sucker who received the bad coin would then hasten to get rid of it. Eventually, everyone is preferentially spending counterfeit money and hoarding their legitimate coins.
In Trump's scam-friendly America, honest businesses operate at a disadvantage. The car dealer that cheats you on the inspection report for your car, cheats you again on your loan, and cheats you a third time on warranty service will have more money to advertise and market than the honest seller across the street. The supplement-pushing grifter has more money to spend on marketing than the company that makes real medicine.
Trump is out there pardoning shady crypto exchanges, gutting the white collar crime cops, and accepting tens of millions of dollars in payments from companies and executives who are under investigation for stealing from the American people. The Consumer Finance Protection Bureau just shut down its kleptocracy unit (seriously!):
This is a far cry from how other governments operate. In Germany, two executives behind Volkswagen's Dieselgate sentence are personally going to prison, one of them for 4.5 years. 31 more executives are still being dragged through the courts:
Scams beget scams. Scammers will tell you that "you can't con an honest man," and try to make everyone complicit in their dishonesty. If you are convinced that being honest will get you taken advantage of, there's nothing for it but to turn scammer yourself. Trump and his scammer base will drive out every honest business and interaction in our economy – and when they steal from you, they'll flash their veneers and tell you "That makes me smart."
The Internet of Consent https://www.anildash.com/2025/05/27/2025-05-27-internet-of-consent/
Donald Trump’s War on Gender Is Also a War on Government https://www.newyorker.com/news/the-lede/donald-trumps-war-on-gender-is-also-a-war-on-government
#20yrsago Copyright prevented transmission of Beatles music to aliens https://memex.craphound.com/2005/05/29/copyright-prevented-transmission-of-beatles-music-to-aliens/
#10yrsago Pornoscanner lobbyist’s new job: overseeing TSA spending https://theintercept.com/2015/05/27/tsa-body-scanner-lobbyist-takes-congressional-job-overseeing-spending-tsa-security/
#5yrsago Canadian newsrooms restructure as co-ops https://pluralistic.net/2020/05/29/mind-control-skepticism/#co-ops
#5yrsago GOP lawmaker hid his diagnosis from Democrats https://pluralistic.net/2020/05/29/mind-control-skepticism/#flu-klux-klan
#5yrsago Walmart's crummy anti-theft AI https://pluralistic.net/2020/05/29/mind-control-skepticism/#neverseen
#5yrsago Masks work https://pluralistic.net/2020/05/29/mind-control-skepticism/#mask-up
#5yrsago What to do about the police https://pluralistic.net/2020/05/29/mind-control-skepticism/#qualified-immunity
#5yrsago Private equity goes mainstream https://pluralistic.net/2020/05/29/mind-control-skepticism/#looters
#5yrsago How the IoT reinforces gentrification https://pluralistic.net/2020/05/29/mind-control-skepticism/#automated-karens
#5yrsago Big Tech distorts our discourse https://pluralistic.net/2020/05/29/mind-control-skepticism/#tech-unexceptionalism
#1yrago The Pizzaburger Presidency https://pluralistic.net/2024/05/29/sub-bushel-comms-strategy/#nothing-would-fundamentally-change
#15yrsago Canadian students speak out against the Canadian DMCA https://www.youtube.com/watch?v=WA1RDyN7JTg
#10yrsago How metadata compromises you https://www.youtube.com/watch?v=xP_e56DsymA
#5yrsago Corporate Dems include military contractors in bailout https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#beltway-bandits
#5yrsago World video premiere of the Little Brother stage play https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#custom-made
#5yrsago Wisconsin cases surge after voting day https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#grandparents-optional-party
#5yrsago Teardown of an "anti-5g" USB stick https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#conspiragrifters
#5yrsago California is setting itself up for a brutal second wave https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#second-wave
#5yrsago Why ammosexuals are REALLY pointing guns at their own dicks https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#musketfuckers
#5yrsago "News" broadcasts air scripted Amazon commercial https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#quackspeak
#5yrsago Cuomo legalizes nursing-home murders https://pluralistic.net/2020/05/28/holographic-nano-layer-catalyser/#immunity-impunity
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Kick 'Em In the Dongle (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16148346-kick-em-in-the-dongle
The Big Story
https://www.seekyoursounds.com/podcasts/the-big-story/cory-doctorow-explains-who-broke-the-internet
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
Crypto industry-funded polls have claimed that as much as 28% of American adults currently owned crypto in 2024. Coinbase in particular loves to throw around claims of 20%.
Federal data has just come out putting the figure at... 8%.
Legislators and regulators are frequently relying on the crypto industry’s own inflated figures to justify their actions, and the crypto industry has used the supposed existence of all these “crypto voters” as a cudgel to threaten legislators.
When techies describe their experience of AI, it sometimes sounds like they're describing two completely different realities – and that's because they are. For workers with power and control, automation turns them into centaurs, who get to use AI tools to improve their work-lives. For workers whose power is waning, AI is a tool for reverse-centaurism, an electronic whip that pushes them to work at superhuman speeds. And when they fail, these workers become "moral crumple zones," absorbing the blame for the defective products their bosses pushed out in order to goose profits. As ever, what a technology does pales in comparison to who it does it for and who it does it to.
In the Who Cares Era, the most radical thing you can do is care.
"Say I tell you that you have my permission to move a book I wrote (and am thus the copyright holder for) from your Kindle to another device. If the Kindle book has DRM, you're still not allowed to move it. The fact that I am the copyright holder has no impact on whether Amazon—a company that didn't create or invest in my book—can prevent you from moving that book outside of its walled garden...In fact, if I supply you with a tool to remove DRM (like some versions of Calibre), then I commit a felony and Amazon can have me sent to prison for five years for giving you a tool to move my book from the Kindle app to a rival app like Kobo," [Cory Doctorow] wrote.
On a recent This Machine Kills episode, guest Hagen Blix described the ultimate form of "AI therapy" with a "human in the loop":
https://soundcloud.com/thismachinekillspod/405-ai-is-the-demon-god-of-capital-ft-hagen-blix
One actual therapist is just having ten chat GPT windows open where they just like have five seconds to interrupt the chatGPT. They have to scan them all and see if it says something really inappropriate. That's your job, to stop it.
Blix admits that's not where therapy is at…yet, but he references Laura Preston's 2023 N Plus One essay, "HUMAN_FALLBACK," which describes her as a backstop to a real-estate "virtual assistant," that masqueraded as a human handling the queries that confused it, in a bid to keep the customers from figuring out that they were engaging with a chatbot:
https://www.nplusonemag.com/issue-44/essays/human_fallback/
This is what makes investors and bosses slobber so hard for AI – a "productivity" boost that arises from taking away the bargaining power of workers so that they can be made to labor under worse conditions for less money. The efficiency gains of automation aren't just about using fewer workers to achieve the same output – it's about the fact that the workers you fire in this process can be used as a threat against the remaining workers: "Do your job and shut up or I'll fire you and give your job to one of your former colleagues who's now on the breadline."
This has been at the heart of labor fights over automation since the Industrial Revolution, when skilled textile workers took up the Luddite cause because their bosses wanted to fire them and replace them with child workers snatched from Napoleonic War orphanages:
https://pluralistic.net/2023/09/26/enochs-hammer/#thats-fronkonsteen
Textile automation wasn't just about producing more cloth – it was about producing cheaper, worse cloth. The new machines were so easy a child could use them, because that's who was using them – kidnapped war orphans. The adult textile workers the machines displaced weren't afraid of technology. Far from it! Weavers used the most advanced machinery of the day, and apprenticed for seven years to learn how to operate it. Luddites had the equivalent of a Masters in Engineering from MIT.
Weavers' guilds presented two problems for their bosses: first, they had enormous power, thanks to the extensive training required to operate their looms; and second, they used that power to regulate the quality of the goods they made. Even before the Industrial Revolution, weavers could have produced more cloth at lower prices by skimping on quality, but they refused, out of principle, because their work mattered to them.
Now, of course weavers also appreciated the value of their products, and understood that innovations that would allow them to increase their productivity and make more fabric at lower prices would be good for the world. They weren't snobs who thought that only the wealthy should go clothed. Weavers had continuously adopted numerous innovations, each of which increased the productivity and the quality of their wares.
Long before the Luddite uprising, weavers had petitioned factory owners and Parliament under the laws that guaranteed the guilds the right to oversee textile automation to ensure that it didn't come at the price of worker power or the quality of the textiles the machines produced. But the factory owners and their investors had captured Parliament, which ignored its own laws and did nothing as the "dark, Satanic mills" proliferated. Luddites only turned to property destruction after the system failed them.
Now, it's true that eventually, the machines improved and the fabric they turned out matched and exceeded the quality of the fabric that preceded the Industrial Revolution. But there's nothing about the way the Industrial Revolution unfolded – increasing the power of capital to pay workers less and treat them worse while flooding the market with inferior products – that was necessary or beneficial to that progress. Every other innovation in textile production up until that time had been undertaken with the cooperation of the guilds, who'd ensured that "progress" meant better lives for workers, better products for consumers, and lower prices. If the Luddites' demands for co-determination in the Industrial Revolution had been met, we might have gotten to the same world of superior products at lower costs, but without the immiseration of generations of workers, mass killings to suppress worker uprisings, and decades of defective products being foisted on the public.
So there are two stories about automation and labor: in the dominant narrative, workers are afraid of the automation that delivers benefits to all of us, stand in the way of progress, and get steamrollered for their own good, as well as ours. In the other narrative, workers are glad to have boring and dangerous parts of their work automated away and happy to produce more high-quality goods and services, and stand ready to assess and plan the rollout of new tools, and when workers object to automation, it's because they see automation being used to crush them and worsen the outputs they care about, at the expense of the customers they care for.
In modern automation/labor theory, this debate is framed in terms of "centaurs" (humans who are assisted by technology) and "reverse-centaurs" (humans who are conscripted to assist technology):
https://pluralistic.net/2023/04/12/algorithmic-wage-discrimination/#fishers-of-men
There are plenty of workers who are excited at the thought of using AI tools to relieve them of some drudgework. To the extent that these workers have power over their bosses and their working conditions, that excitement might well be justified. I hear a lot from programmers who work on their own projects about how nice it is to have a kind of hypertrophied macro system that can generate and tweak little automated tools on the fly so the humans can focus on the real, chewy challenges. Those workers are the centaurs, and it's no wonder that they're excited about improved tooling.
But the reverse-centaur version is a lot darker. The reverse-centaur coder is an assistant to the AI, charged with being a "human in the loop" who reviews the material that the AI produces. This is a pretty terrible job to have.
For starters, the kinds of mistakes that AI coders make are the hardest mistakes for human reviewers to catch. That's because LLMs are statistical prediction machines, spicy autocomplete that works by ingesting and analyzing a vast corpus of written materials and then producing outputs that represent a series of plausible guesses about which words should follow one another. To the extent that the reality the AI is participating in is statistically smooth and predictable, AI can often make eerily good guesses at words that turn into sentences or code that slots well into that reality.
But where reality is lumpy and irregular, AI stumbles. AI is intrinsically conservative. As a statistically informed guessing program, it wants the future to be like the past:
https://reallifemag.com/the-apophenic-machine/
This means that AI coders stumble wherever the world contains rough patches and snags. Take "slopsquatting." For the most part, software libraries follow regular naming conventions. For example, there might be a series of text-handling libraries with names like "text.parsing.docx," "text.parsing.xml," and "text.parsing.markdown." But for some reason – maybe two different projects were merged, or maybe someone was just inattentive – there's also a library called "text.txt.parsing" (instead of "text.parsing.txt").
AI coders are doing inference based on statistical analysis, and anyone inferring what the .txt parsing library is called would guess, based on the other libraries, that it was "text.parsing.txt." And that's what the AI guesses, and so it tries to import that library to its software projects.
This creates a new security vulnerability, "slopsquatting," in which a malicious actor creates a library with the expected name, which replicates the functionality of the real library, but also contains malicious code:
https://www.theregister.com/2025/04/12/ai_code_suggestions_sabotage_supply_chain/
Note that slopsquatting errors are extremely hard to spot. As is typical with AI coding errors, these are errors that are based on continuing a historical pattern, which is the sort of thing our own brains do all the time (think of trying to go up a step that isn't there after climbing to the top of a staircase). Notably, these are very different from the errors that a beginning programmer whose work is being reviewed by a more senior coder might make. These are the very hardest errors for humans to spot, and these are the errors that AIs make the most, and they do so at machine speed:
https://pluralistic.net/2024/04/23/maximal-plausibility/#reverse-centaurs
To be a human in the loop for an AI coder, a programmer must engage in sustained, careful, line-by-line and command-by-command scrutiny of the code. This is the hardest kind of code to review, and maintaining robotic vigilance over long periods at high speeds is something humans are very bad at. Indeed, it's the kind of task we try very hard to automate, since machines are much better at being machinelike than humans are. This is the essence of reverse-centaurism: when a human is expected to act like a machine in order to help the machine do something it can't do.
Humans routinely fail at spotting these errors, unsurprisingly. If the purpose of automation is to make superior goods at lower prices, then this would be a real concern, since a reverse-centaur coding arrangement is bound to produce code with lurking, pernicious, especially hard-to-spot bugs that present serious risks to users. But if the purpose of automation is to discipline labor – to force coders to accept worse conditions and pay – irrespective of the impact on quality, then AI is the perfect tool for the job. The point of the human isn't to catch the AI's errors so much as it is to catch the blame for the AI's errors – to be what Madeleine Clare Elish calls a "moral crumple zone":
https://estsjournal.org/index.php/ests/article/view/260
As has been the case since the Industrial Revolution, the project of automation isn't just about increasing productivity, it's about weakening labor power as a prelude to lowering quality. Take what's happened to the news industry, where mass layoffs are being offset by AI tools. At Hearst's King Features Syndicates, a single writer was charged with producing over 30 summer guides, the entire package:
That is an impossible task, which is why the writer turned to AI to do his homework, and then, infamously, published a "summer reading guide" that was full of nonexistent books that were hallucinated by a chatbot:
Most people reacted to this story as a consumer issue: they were outraged that the world was having a defective product foisted upon it. But the consumer issue here is downstream from the labor issue: when the writers at King Features Syndicate are turned into reverse-centaurs, they will inevitably produce defective outputs. The point of the worker – the "human in the loop" – isn't to supervise the AI, it's to take the blame for the AI. That's just what happened, as this poor schmuck absorbed an internet-sized rasher of shit flung his way by outraged social media users. After all, it was his byline on the story, not the chatbot's. He's the moral crumple-zone.
The implication of this is that consumers and workers are class allies in the automation wars. The point of using automation to weaken labor isn't just cheaper products – it's cheaper, defective products, inflicted on the unsuspecting and defenseless public who are no longer protected by workers' professionalism and pride in their jobs.
That's what's going on at Duolingo, where CEO Luis von Ahn created a firestorm by announcing mass firings of human language instructors, who would be replaced by AI. The "AI first" announcement pissed off Duolingo's workers, of course, but what caught von Ahn off-guard was how much this pissed off Duolingo's users:
But of course, this makes perfect sense. After all, language-learners are literally incapable of spotting errors in the AI instruction they receive. If you spoke the language well enough to spot the AI's mistakes, you wouldn't need Duolingo! I don't doubt that there are countless ways in which AIs could benefit both language learners and the Duolingo workers who develop instructional materials, but for that to happen, workers' and learners' needs will have to be the focus of AI integration. Centaurs could produce great language learning materials with AI – but reverse-centaurs can only produce slop.
Unsurprisingly, many of the most successful AI products are "bossware" tools that let employers monitor and discipline workers who've been reverse-centaurized. Both blue-collar and white-collar workplaces have filled up with "electronic whips" that monitor and evaluate performance:
https://pluralistic.net/2024/08/02/despotism-on-demand/#virtual-whips
AI can give bosses "dashboards" that tell them which Amazon delivery drivers operate their vehicles with their mouths open (Amazon doesn't let its drivers sing on the job). Meanwhile, a German company called Celonis will sell your boss a kind of AI phrenology tool that assesses your "emotional quality" by spying on you while you work:
https://crackedlabs.org/en/data-work/publications/processmining-algomanage
Tech firms were among the first and most aggressive adopters of AI-based electronic whips. But these whips weren't used on coders – they were reserved for tech's vast blue-collar and contractor workforce: clickworkers, gig workers, warehouse workers, AI data-labelers and delivery drivers.
Tech bosses tormented these workers but pampered their coders. That wasn't out of any sentimental attachment to tech workers. Rather, tech bosses were afraid of tech workers, because tech workers possess a rare set of skills that can be harnessed by tech firms to produce gigantic returns. Tech workers have historically been princes of labor, able to command high salaries and deferential treatment from their bosses (think of the amazing tech "campus" perks), because their scarcity gave them power.
It's easy to predict how tech bosses would treat tech workers if they could get away with it – just look how they treat workers they aren't afraid of. Just like the textile mill owners of the Industrial Revolution, the thing that excites tech bosses about AI is the possibility of cutting off a group of powerful workers at the knees. After all, it took more than a century for strong labor unions to match the power that the pre-Industrial Revolution guilds had. If AI can crush the power of tech workers, it might buy tech bosses a century of free rein to shift value from their workforce to their investors, while also doing away with pesky Tron-pilled workers who believe they have a moral obligation to "fight for the user."
William Gibson famously wrote, "The future is here, it's just not evenly distributed." The workers that tech bosses don't fear are living in the future of the workers that tech bosses can't easily replace.
This week, the New York Times's veteran Amazon labor reporter Noam Scheiber published a deeply reported piece about the experience of coders at Amazon in the age of AI:
https://www.nytimes.com/2025/05/25/business/amazon-ai-coders.html
Amazon CEO Andy Jassy is palpably horny for AI coders, evidenced by investor memos boasting of AI's returns in "productivity and cost avoidance" and pronouncements about AI saving "the equivalent of 4,500 developer-years":
Amazon is among the most notorious abusers of blue-collar labor, the workplace where everyone who doesn't have a bullshit laptop job is expected to piss in a bottle and spend an unpaid hour before and after work going through a bag- and body-search. Amazon's blue-collar workers are under continuous, totalizing, judging AI scrutiny that scores them based on whether their eyeballs are correctly oriented, whether they take too long to pick up an object, whether they pee too often. Amazon warehouse workers are injured at three times national average. Amazon AIs scan social media for disgruntled workers talking about unions, and Amazon has another AI tool that predicts which shops and departments are most likely to want to unionize.
Scheiber's piece describes what it's like to be an Amazon tech worker who's getting the reverse-centaur treatment that has heretofore been reserved for warehouse workers and drivers. They describe "speedups" in which they are moved from writing code to reviewing AI code, their jobs transformed from solving chewy intellectual puzzles to racing to spot hard-to-find AI coding errors as a clock ticks down. Amazon bosses haven't ordered their tech workers to use AI, just raised their quotas to a level that can't be attained without getting an AI to do most of the work – just like the Chicago Sun-Times writer who was expected to write all 30 articles in the summer guide package on his own. No one made him use AI, but he wasn't going to produce 30 articles on deadline without a chatbot.
Amazon insists that it is treating AI as an assistant for its coders, but the actual working conditions make it clear that this is a reverse-centaur transformation. Scheiber discusses a dissident internal group at Amazon called Amazon Employees for Climate Justice, who link the company's use of AI to its carbon footprint. Beyond those climate concerns, these workers are treating AI as a labor issue.
Amazon's coders have been making tentative gestures of solidarity towards its blue-collar workforce since the pandemic broke out, walking out in support of striking warehouse workers (and getting fired for doing so):
https://pluralistic.net/2020/04/14/abolish-silicon-valley/#hang-together-hang-separately
But those firings haven't deterred Amazon's tech workers from making common cause with their comrades on the shop floor:
https://pluralistic.net/2021/01/19/deastroturfing/#real-power
When techies describe their experience of AI, it sometimes sounds like they're describing two completely different realities – and that's because they are. For workers with power and control, automation turns them into centaurs, who get to use AI tools to improve their work-lives. For workers whose power is waning, AI is a tool for reverse-centaurism, an electronic whip that pushes them to work at superhuman speeds. And when they fail, these workers become "moral crumple zones," absorbing the blame for the defective products their bosses pushed out in order to goose profits.
As ever, what a technology does pales in comparison to who it does it for and who it does it to.
Atomic Highway – Post Apocalyptic Roleplaying https://archive.org/details/atomic-highway-ebook-v-2/mode/1up
THE SILICON VALLEY EFFECT https://law.stanford.edu/wp-content/uploads/2025/04/SJIL_61-1_Arun.pdf
#20yrsago Bulk of American calories comes from sweet drinks https://www.sciencedaily.com/releases/2005/05/050527111920.htm
#20yrsago Chicago’s Bean sculpture is free to photograph, at last https://memex.craphound.com/2005/05/27/chicagos-bean-sculpture-is-free-to-photograph-at-last/
#15yrsago Man single-handedly building a metro rail https://englishrussia.com/2010/05/24/the-most-unusual-metro-in-the-world/
#15yrsago Canada’s copyright minister: superinfringer https://memex.craphound.com/2010/05/27/canadas-copyright-minister-superinfringer/
#15yrsago Pinkwater’s ADVENTURES OF A CAT-WHISKERED GIRL, sequel to Neddiad and Yggyssey https://memex.craphound.com/2010/05/27/pinkwaters-adventures-of-a-cat-whiskered-girl-sequel-to-neddiad-and-yggyssey/
#10yrsago Real estate bubble drives urban blight https://www.newyorker.com/business/currency/why-are-there-so-many-shuttered-storefronts-in-the-west-village
#10yrsago IRS leaks 100K taxpayers’ data to identity thieves https://theintercept.com/2015/05/27/data-breaches-wreak-havoc/
#10yrsago Swiss cops’ dawn raid snags top FIFA officials https://www.nytimes.com/2015/05/27/sports/soccer/fifa-officials-face-corruption-charges-in-us.html
#5yrsago The Toronto Star's new owners donated to far-right Tories https://pluralistic.net/2020/05/27/literal-gunhumping/#plutewatch
#5yrsago How to pay artists while fighting censorship and Big Tech https://pluralistic.net/2020/05/27/literal-gunhumping/#pay-artists
#5yrsago Ammosexuals point their guns at their crotches https://pluralistic.net/2020/05/27/literal-gunhumping/#youll-shoot-your-eye-out
#5yrsago Twitter's porn filters are dampening discussions of "cumgate" https://pluralistic.net/2020/05/27/literal-gunhumping/#cumgate
#5yrsago West Virginia's governor Jim Justice: billionaire, deadbeat https://pluralistic.net/2020/05/27/literal-gunhumping/#injustice
#5yrsago Hertz's bankruptcy was caused by private equity looting https://pluralistic.net/2020/05/27/literal-gunhumping/#hertz-uranus
#5yrsago Facebook shelved research that showed they were sowing division https://pluralistic.net/2020/05/27/literal-gunhumping/#joel-kaplan
#5yrsago Youtube is automatically blocking criticism of the Chinese Communist Party https://pluralistic.net/2020/05/27/literal-gunhumping/#communist-bandit
#1yrago Against Lore https://pluralistic.net/2024/05/27/cmon-do-it-again/#better_to_remain_silent_and_be_thought_a_fool_than_to_speak_and_remove_all_doubt
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
The Big Story
https://www.seekyoursounds.com/podcasts/the-big-story/cory-doctorow-explains-who-broke-the-internet
Keynote (Pycon)
https://www.youtube.com/watch?v=ydVmzg_SJLw
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
Given the recent data breach and Coinbase’s user agreement that aims to force customers into arbitration rather than individual or class action lawsuits, it’s interesting to read the outcome of a recent arbitration case against Coinbase.
![9. On December 17, 2024, Arbitrator Kruze issued an Order Granting Dispositive Motion as to all of Mr. Spilker’s claims. Ex. A at 6 (“Respondent’s Motion for Summary Judgment is GRANTED. Claimant’s claims against Coinbase are dismissed.”). 10. The Final Award holds: a. Claimant’s EFTA cause of action is time-barred because the “one-year limitations period begins when the first unauthorized transfer occurs, not upon discovery by the consumer, and not when the consumer notifies the defendant of the unauthorized transfer.” Id. at 3, applying 15 U.S.C. §1693m(g) and Wike v. Vertrue, Inc., 566 F.3d 590,593 (6th Cir. 2009). b. “The undisputed facts show that a third party, not Coinbase, caused Claimant’s damages” and that “Claimant’s damages were the result of an intervening and superseding cause: the actions of a third-party scammer. Coinbase, as a matter of law, cannot be held liable for Claimant’s damages.” Ex. A at 4, citing May v. Google, LLC, No. 24-CV-01314- BLF, 2024 WL 4681604, at *10 (N.D. Cal. Nov. 4, 2024). c. The parties’ contract forecloses Mr. Spilker’s causes of action for breach of contract, negligence and tort claims, and claims under Idaho and Oregon law. Ex. A at 5. d. Pursuant to Melchoir v. New Line Prods., Inc., 106 Cal. App. 4th 779, 793 (2003), “Claimant’s cause of action for unjust enrichment fails because ‘there is no [such thing as a] cause of action in California for unjust enrichment.’” Ex. A at 5. e. Mr. Spilker’s CLRA claim fails “as courts have consistently held that the CLRA does not apply to cryptocurrency exchanges like Coinbase”. Id. at 6, citing various cases.](https://storage.mollywhite.net/micro/0bbcc42846cc232283eb_Screenshot-2025-05-27-at-1.04.40---PM.png)
Customer lost $350,000 in September 2022 to a phishing attack from a scammer that the customer said had “confidential information that could have only been obtained with direct access to Coinbase’s database”.
Coinbase didn’t prevent the suspicious transfers, allegedly wiped customer’s transaction history, blamed the customer for the loss, then refused to reimburse. Arbitration concluded with $0 reimbursement.
Arbitrator found the complaint had been filed too late, and that the customer had admitted that a third party rather than a Coinbase insider had performed the theft. Doesn’t appear the arbitrator investigated the claims of a possible breach, or how the hardware MFA was bypassed.
![Claimant does not dispute these facts in its briefing. In other words, Claimant’s damages were the result of an intervening and superseding cause: the actions of a third-party scammer. Coinbase, as a matter of law, cannot be held liable for Claimant’s damages. See May v. Google, LLC, No. 24-CV-01314-BLF, 2024 WL 4681604, at *10 (N.D. Cal. Nov. 4, 2024). Third, Claimant’s supplemental production also eviscerates many of Claimant’s other causes of action. For example, in his Demand, Claimant alleges that Coinbase never advised users that staked ETH could be wrapped and traded before his September 2022 loss. In its Motion and Reply, Coinbase previously relied on public statements announcing the launch of cbETH as early as August 2022. Claimant’s belated production, attached to Respondent’s supplemental briefing, also contains undisputed facts that Claimant was himself actually informed about cbETH’s launch before his funds were stolen. This disclosure significantly undermines Claimant’s misrepresentation claims under common law, securities law, and commodities law. Fourth, the parties’ contract forecloses many of Claimant’s causes of action. For example, Claimant’s breach of contract claim is undercut by section 6.6 of the UA, which explicitly apportions the risk of account compromises to Claimant as the user: “Any loss or compromise of . . . your personal information may result in unauthorized access to your Coinbase Account(s) by third-parties and the loss or theft of any Digital Assets and/or funds held in your Coinbase Account(s) and any associated accounts, including your linked bank account(s) and credit card(s). . . .We assume no responsibility for any loss that you may sustain due to compromise of account login credentials due to no fault of Coinbase.” Moreover, the UA’s choice-of-law provision in section 9.5 restricts Claimant to California law, foreclosing Claimant’s Idaho and Oregon-based causes of action. In addition, the UA and California law effectively cut off Claimant’s tort claims, such as his negligence cause of action. See, e.g., Berk v. Coinbase, Inc., 840 F. App’x 914 (9th Cir. Dec. 23, 2020) (Coinbase owes no independent tort duty of care beyond the promises made in the UA). Finally, there are other independent reasons why Claimant’s Demand cannot succeed. For example, Claimant’s cause of action for “unjust enrichment” fails because “there is no [such thing as a] cause of action in California for unjust enrichment.” Melchior v. New Line Prods., Inc., 106 Cal. App. 4th 779, 793 (2003). As another example, Claimant submitted no evidence of any false statements made by](https://storage.mollywhite.net/micro/cb92d9a3b3a2046762ee_Screenshot-2025-05-27-at-1.13.11---PM.png)
"Kick 'Em In the Dongle" is the fourth and final episode of "Understood: Who Broke the Internet?", a podcast series I hosted and co-wrote for the CBC. It's quite a finale!
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16148346-kick-em-in-the-dongle
The thesis of the series is the same as the thesis of enshittification: that the internet turned into a pile of shit because named people, in living memory, made policies that were broadly "enshittogenic" because they insulated businesses that tormented their end users and business customers from any consequences for their cheating:
https://www.youtube.com/watch?v=ydVmzg_SJLw
Moreover, these people were warned at the time about the certain consequences of their policies, and they ignored and dismissed both expert feedback and public opinion. These people never faced consequences or any accountability for their actions, as tech criticism focused (understandably and deservedly) on the businesses that took advantage of the enshittogenic policies and enshittified, without any understanding that these firms were turning into piles of shit because of policies that reward them for doing so.
Episode one of the series tells the story of an enshittification poster-child: Google. We look at the paper-trail that emerged from the Department of Justice's successful monopoly prosecution of Google, and what it reveals about the sorry state of internet search today:
https://pluralistic.net/2025/05/08/who-broke-the-internet/#bruce-lehman
That paper-trail documents an intense power-struggle within Google: in 2019, Google's ad revenue czar went to war against Google's search boss, demanding that search be deliberately worsened. This may sound paradoxical (or even paranoid), but for Google, making search worse made a perverse kind of sense. The company's search revenue growth had stalled, for the obvious reason that Google had a 90% market share in search, which meant that basically everyone was a Google search user, leaving the company with no new potential customers to sign up.
In 2019, Prabhakar Raghavan – the ex-McKinsey, ex-Yahoo MBA who ran ad revenue for Google – came up with an ingenious solution: just make search worse. If you have to run multiple searches to find what you're looking for, that creates multiple chances to show you an ad:
https://www.wheresyoured.at/the-men-who-killed-google/
Raghavan's nemesis was Ben Gomes, an OG googler who'd overseen the creation of the company's server infrastructure and had been crowned the head of search. Gomes hated Raghavan's idea, and in the memos, we get a blow-by-blow account of the epic fight inside Google between the enshittifiers and the anti-enshittification resistance, who are ultimately trounced, which is how we get today's sloppified, ad-poisoned, spam-centric Google search.
Raghavan and his clique are obviously greedy monsters, but that's not the whole story. The real question is, how did we get to the point where Google, a company justly famed for its emphasis on search quality, abandoned its commitment to excellence? That's the question we explore in episodes two and three.
Episode two is "Ctrl-ctrl-ctrl," and it reveals the original sin of tech, the origin of the worst tech policies in the world:
https://pluralistic.net/2025/05/13/ctrl-ctrl-ctrl/#free-dmitry
This is the tale of another epic struggle inside another giant institution, only this struggle takes place in government, not Google. We travel back to the Clinton years, when Vice President Al Gore was put in charge of demilitarizing the internet and transforming it into a service that welcomed the public, as well as private firms. Gore's rival in this project was Clinton's copyright czar, the white shoe entertainment lawyer Bruce Lehman.
Lehman wanted Gore to install an "anti-circumvention" policy on the new internet: under Lehman's proposal, copyright law would be rewritten to ban modifying ("circumventing") digital products, services and devices, whether or not those modifications led to anyone's copyrights being violated. Anti-circumvention would let dominant companies conscript the government to punish upstart rivals and tinkerers who dared to improve their products, say, by blocking commercial surveillance, or by turning off checks that blocked generic parts and consumables or independent repair, or by making existing products more accessible to people with disabilities.
Experts like Pam Samuelson hated this proposal and made a huge stink about it. This led to Gore categorically rejecting Lehman's ideas, so Lehman (in his own words) did "an end-run around Congress" and got the UN's World Intellectual Property Organization (WIPO) to turn "anti-circumvention" into an international treaty obligation. Then he went back to Congress and got them to pass an anti-circumvention law, Section 1201 of the Digital Millennium Copyright Act (DMCA), that went even further than the WIPO treaties demanded.
Almost instantly, the direst predictions of Lehman's opponents came true. A Russian computer scientist named Dmitry Skylarov was arrested by the FBI for giving a technical conference presentation about the weaknesses in Adobe's ebook software, in which he explained how these allowed Adobe customers to do legal things, like transferring their ebooks to a new computer (Adobe's software blocked this).
The chilling effect of DMCA 1201 was deep and far-reaching. It created (in the words of Jay Freeman), a new "felony contempt of business model" system, in which a business could threaten to imprison anyone who tried to disenshittify their products, for example, by making it possible for hospitals to maintain their ventilators without paying a med-tech giant for overpriced, slow service:
Anticircumvention law lets John Deere stop farmers from fixing their own tractors. It stops independent mechanics from fixing your car. It stops you from using cheap third-party inkjet cartridges. It's why Patreon performers lose 30 cents on every in-app subscription dollar, because only Apple can provide iPhone apps, and Apple uses that control to extract a 30% fee on in-app payments. It's why you can't stop apps from spying on you – and why Apple (which does block other companies apps from spying on you) can track every click, message and movement you make in order to target ads to you:
https://pluralistic.net/2022/11/14/luxury-surveillance/#liar-liar
Anticircumvention let the garage-door opener company that bought every one of its rivals block integration with standard home automation tools, forcing you to use an app that makes you look at ads before you can open your garage-door:
https://pluralistic.net/2023/11/09/lead-me-not-into-temptation/#chamberlain
Anticircumvention is why there's no such thing as a Tivo for streaming services, letting you record the programs you enjoy so you can watch them later (say, when Prime moves Christmas movies into the paid tier between October and January). It's why you can't get a scraper that lets you leave Facebook or Twitter for Mastodon or Bluesky, and continue to interact with your friends who are stuck on zuckermuskian legacy media:
https://www.eff.org/interoperablefacebook
It's why you can't get an alternative Instagram client that blocks spying, ads and "suggestions," just showing you the latest updates from the people you follow:
https://www.theverge.com/2022/9/29/23378541/the-og-app-instagram-clone-pulled-from-app-store
Of course, companies that abuse this government-granted weapon might still face consequences, if their behavior was so obnoxious that it drove us into the arms of their competitors. But for that to happen, we'd need to have meaningful competition, which brings me to episode three, "In God We Antitrust":
https://pluralistic.net/2025/05/19/khan-thought/#they-were-warned
Episode three goes even farther back in time, to the early 1980s, when a racist pig and Nixon co-conspirator named Robert Bork led a successful counterrevolution that destroyed antitrust enforcement in the US, and then around the world. It's thanks to Bork – and his idea that monopolies are "efficient" – that we got what Tom Eastman calls an internet of "five giant websites filled with screenshots of the other four." It's why every sector in our economy is controlled by a cartel, a duopoly or a monopoly:
https://www.openmarketsinstitute.org/learn/monopoly-by-the-numbers
If Bruce Lehman paved the way for Prabhakar Raghavan's enshittification of Google, then Robert Bork laid the road that Bruce Lehman traveled to Geneva and the WIPO Internet Treaties. Industry consolidation always leads to regulatory capture. A handful of gigantic companies can easily collude to present a disciplined message to its regulators and the fact that they don't compete with one another lets them steal so much from us that they have huge warchests they can use to get their policies enacted.
40 years of Bork's pro-monopoly policies has produced…monopolies. The reason a handful of powerful executives have more power than any of the world's governments – the reason the public is thwarted on everything from healthcare to climate, minimum wages to privacy – is that Robert Bork overturned generations of antitrust practice and created pro-oligarch policies that created a modern oligarchy.
The 2020s have seen an impressive and heartening global surge in antitrust activism, motivated by an urge to blunt or even shatter corporate power, bypassing apologetics about "efficiency" that can only be understood through mastering an esoteric mathematics whose own practitioners cheerfully describe it as disconnected from any observable reality:
https://www.sciencedirect.com/science/article/abs/pii/S0039368122000693
This global, grassroots movement has provoked a massive backlash from our technofeudal overlords, culminating in the 2024 re-election of Donald Trump, which is where we open our the fourth and final episode of "Understood: Who Broke the Internet?" Trump's inauguration stage featured some unusual attendees: the CEOs of the largest tech companies in America, who had personally donated a million bucks each to Trump's inauguration fund. These are some of the richest men in human history, and they were all in on Trump.
Trump lost no time in inflicting misery on the American people, illegally firing the agency personnel most closely associated with the antitrust movement and canceling many of their key policies. But for the rest of the world, the most prominent effect of Trumpism was the imposition of tariffs on every country in the world, including islands without any human inhabitants:
The world is changing before our eyes, and it needn't change for the worse. As Trump transforms America into a hermit kingdom, countries around the world have a chance to consider what their policies might be like if they weren't organized around US priorities. That includes Canada.
Canada could retaliate against Trump's tariffs by legalizing and incubating Canadian companies that find ways to improve America's enshittified products, creating mods, plugins, alternative software and other tools that Canadians – and the world – would snap up. Every customer for these disenshittifying tools would constitute a targeted strike against technofeudalism, against Trumpism, against the companies whose CEOs sat behind Trump on the dais.
More: the Canadian companies that raided America's high-tech giants could use the sky-high rents they extracted through anti-circumvention laws as a kind of disposable rocket stage to boost a new Canadian tech sector into a stable orbit, giving Canada a global tech standing comparable to the power and wealth Finland enjoyed during the Nokia years.
That's something Canada could do, only it can't, yet, because of a 13-year old anti-circumvention law that was crammed onto Canada's statute-books by two ministers in Stephen Harper's government, James Moore and Tony Clement:
https://pluralistic.net/2024/11/15/radical-extremists/#sex-pest
Harper charged Moore and Clement with getting an anticircumvention law because the US Trade Representative had made it clear that failing to do so would result in the US imposing tariffs on Canada. But Canadians hated the idea of this law. In 2004, a Liberal MP named Sam Bulte lost her Toronto seat after she attempted to ram an anticirumvention law through Parliament. The Tories tried to pass another anticircuvmention law in 2007, and faced so much pushback that the bill died.
Moore and Clement's tactic for defusing this opposition was to have a public consultation on anticircumvention law, to make it seem like the government was listening to the people. Boy, did that idea backfire: 6,138 Canadians wrote in to oppose the proposal. 54 supported it:
https://www.michaelgeist.ca/2010/04/copycon-final-numbers/
But Moore and Clement pressed on. Moore explained to an International Chamber of Commerce meeting in Toronto that he would be discarding nearly every consultation response he'd received, on the grounds that people who disagreed with him were "babyish…radical extremists":
https://www.cbc.ca/news/science/copyright-debate-turns-ugly-1.898216
The most remarkable thing about Canada's 2012 adoption of anticircumvention law is that it came 14 years after the US passed the DMCA. We already had a thick record of the damage that law had done. We had all the evidence we needed to see how this US law had hurt everyday Americans. But Moore and Clement still tabled their bill, with language that was actually worse than the American law, dispensing with the largely ineffectual safeguards Congress had put in the 1998 DMCA.
More than a decade on, Canada's "digital locks" law has stalled the country's tech sector and left Canadians defenseless against American enshittification. Even the country's pioneering Right to Repair and interoperability laws, passed last year, can't undo this damage, because they only give Canadians the right to fix or improve things if they don't have to break a digital lock to do so, and everything has a digital lock these days, from ebikes to car parts.
Moore actually gave us a comment for the show, once again dismissing his critics by claiming there was no evidence that his law had created a chilling effect that stopped Canadians from making products and services that unrigged the game American big business forced us all to play. It's nice to see that Moore hasn't changed since his days of calling his detractors "babyish radical extremists." The very nature of "chilling effects" is that they can only be observed by looking at what didn't happen: Moore seems to interpret the fact that Canadians haven't shipped a privacy tool for phones, or an alternative app store for Xboxes, or a service that jailbreaks your car so any mechanic can fix it as evidence that Canadians wouldn't want these things (or that Canadian technologists are too stupid to deliver them).
Repealing Canada's anticircumvention laws would mark a turning point in tech regulation. For decades now, countries that are upset with tech companies' greed and cruelty have created policies that demand that Big Tech wield its extraordinary power more wisely. Think of content moderation laws, or laws that try to get tech companies to share some of their monopoly ripoff money with news outlets. These laws don't seek to take away power from tech giants – they just try to turn it to socially beneficial uses. This is a huge mistake. For a tech company to control its users' behavior, it must have power over those users, must observe every action they take and retain the ability to stop them. For a tech company to share its billions with news outlets, it must continue to make billions by ripping us all off:
https://www.eff.org/deeplinks/2023/06/save-news-we-must-open-app-stores
The only tech regulation that will truly make us all better off is a regulation that shatters tech power – not one that seeks to harness it. That's what getting rid of anticircumvention would do: it would give us – internet users – the right to defend ourselves against exploitation, manipulation and abuse. It would let us decide how the devices, products and services we use work. It wouldn't just make it illegal for tech giants to use our technology to attack us – it would make it impossible for them to do so, because our technology would take orders from us, not them.
Repealing anticircumvention laws in Canada and around the world is the best path forward. Ironically, Donald Trump's "Liberation Day" has created the conditions for every country to liberate itself from America's grotesque tech policies – and to export our tools of technological liberation to our American friends, who were the first victims of US Big Tech.
I'm so pleased with how this show worked out. My collaborators – especially showrunner Acey Rowe and producer Matt Meuse – were stone brilliant as was our sound designer, Julian Uzielli. The whole team has done smashing work getting the word out about the show and making it sound smart and accessible. I couldn't have asked for a better group of colleagues to produce this show, and I couldn't be prouder of how it sounds.
You can subscribe to "Understood: Who Broke the Internet?" on any podcast app, even the enshittified ones, and you can get the RSS here:
https://www.cbc.ca/podcasting/includes/nakedemperor.xml
Trump Antitrust Enforcers Kick Small Business In the Teeth https://www.thebignewsletter.com/p/monopoly-round-up-trump-antitrust
Cloudflare CEO: Football Piracy Blocks Will Claim Lives; “I Pray No One Dies” https://torrentfreak.com/cloudflare-ceo-football-piracy-blocks-will-claim-lives-i-pray-no-one-dies-250526/ (h/t Hacker News)
#20yrsago Music critic to labels: Give me CDs/MP3s/vinyl or don’t bother http://sfj.abstractdynamics.org/archives/005579.html
#20yrsago MIT Tech Review pre-publishing draft articles on a blog https://web.archive.org/web/20050524003647/http://www.continuousblog.net/2005/05/post.html
#20yrsago Schwarzenegger creates, then fills Potemkin pothole https://web.archive.org/web/20050602073302/https://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2005/05/27/GOVERNOR.TMP
#15yrsago A perfect marvel of vacuous malice https://memex.craphound.com/2010/05/26/a-perfect-marvel-of-vacuous-malice/
#15yrsago Canada’s own PATRIOT Act https://web.archive.org/web/20100530042800/https://www.michaelgeist.ca/content/view/5059/125/
#15yrsago Schneier at the airport https://www.schneier.com/blog/archives/2010/05/scene_from_an_a.html
#15yrsago September 2008 crash cost $108K per US household https://web.archive.org/web/20110505220244/https://www.pewtrusts.org/our_work_report_detail.aspx?id=58695&category=618
#10yrsago Amazon will finally start paying tax in the UK https://www.theguardian.com/technology/2015/may/23/amazon-to-begin-paying-corporation-tax-on-uk-retail-sales
#10yrsago What Sony and Spotify’s secret deal really looks like https://www.theverge.com/2015/5/19/8621581/sony-music-spotify-contract
#10yrsago Poverty is a tax on cognition https://www.youtube.com/watch?v=u6_scuce5TA
#10yrsago Screw the techno-determinists — give me hope instead https://www.theguardian.com/technology/2015/may/26/hope-future-internet-activism-freedom
#10yrsago The Awesome: ass-kicking girl monster-hunter FTW! https://memex.craphound.com/2015/05/26/the-awesome-ass-kicking-girl-monster-hunter-ftw/
#10yrsago Al Qaeda’s job application form https://web.archive.org/web/20150520175956/https://www.dni.gov/files/documents/ubl/english/Instructions
#5yrsago White Americans are increasingly opposed to democracy https://pluralistic.net/2020/05/26/pandemic-profiteers/#quiet-part-outloud
#5yrsago Uber is scrapping thousands of Jump bikes https://pluralistic.net/2020/05/26/pandemic-profiteers/#bezzlers
#5yrsago Texas profiteers make bank from NYC's homeless https://pluralistic.net/2020/05/26/pandemic-profiteers/#profiteering
#5yrsago Scarfolk on Cumgate https://pluralistic.net/2020/05/26/pandemic-profiteers/#private-law
#5yrsago Coronagrifting and other bad design fictions https://pluralistic.net/2020/05/24/coronagrifting/#coronagrifting
#5yrsago Bloomberg editorial calls for a supersized New Deal https://pluralistic.net/2020/05/24/coronagrifting/#more-fdr
#1yrago They brick you because they can https://pluralistic.net/2024/05/24/record-scratch/#autoenshittification
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Keynote (Pycon)
https://www.youtube.com/watch?v=ydVmzg_SJLw
Reclaiming the Internet (Easy Prey)
https://www.easyprey.com/reclaiming-the-internet/
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
You can’t crush ideas by suppressing them. You can only crush them by ignoring them. By refusing to think, refusing to change.– The Dispossessed, Ursula K. Le Guin
It's linkdump time, in which I skillfully weave together all the links that I was too busy to cram into the week's newsletter issues. Here's the previous 31 (!) installments:
https://pluralistic.net/tag/linkdump/
This week's linkdump comes with a great excuse: I was off at the staff retreat for the Electronic Frontier Foundation, for intensive policy work (and a lot of team-bonding socializing – I karaoked "The Piano Has Been Drinking") with my colleagues on the front lines of the battle to disenshittify the internet. If you'd like to join that fight, here's a chance to do so: we're hiring a staff technologist!
https://www.paycomonline.net/v4/ats/web.php/jobs?clientkey=28620672D234BF368306CEB4A2746667
Of course, you don't have to work for EFF to make disenshittificatory tech. "Just a QR Code" is a new site that generates QR codes, operating entirely in your browser, without transmitting any data to a server or trying to cram ads into your eyeballs. The fact that it runs entirely in-browser means you can save this webpage and work with an offline copy to generate QR codes forever – even if the site goes down:
One of the best, longest-tenured gatherings of anti-enshittification technologists is HOPE, the Hackers On Planet Earth con spawned by 2600: The Hacker Quarterly. After getting evicted from their traditional digs at the Hotel Pennsylvania (which was bought by a billionaire who turned it into a crater and then lost interest), HOPE had to find new digs. The new location, St John's University in Queens, is fantastic, and the last event was so great they decided to go from biennial to annual:
But then, Trump hit. HOPE draws a sizable cohort of international attendees and speakers, and most of these people have decided that attending a genuinely fantastic hacker con isn't worth risk being sent to a Salvadoran slave-labor camp by a surly border guard. As a result, HOPE's numbers are dangerously low:
Please consider attending! HOPE is consistently one of the best events I've attended. The vibes are impeccable and the information is deep, gnarly and fantastic, and has a long, long track record of just being great.
Another beloved, long-running, print based institution is The Onion, which got a new lease on life when former disinformation reporter Ben Collins bought the site after quitting NBC, which had censured him for being too mean to Elon Musk:
https://variety.com/2022/tv/news/nbc-news-ben-collins-twitter-elon-musk-1235463474/
Having been burned by corporate journalistic cowardice, Collins decided to revive The Onion's tradition of merciless, trenchant parody. He also revived The Onion's tradition of showing up in the world as a printed artifact, spraying gallons of ink onto tons of vegetable pulp and shipping the result to mailboxes around the world (including mine):
https://membership.theonion.com/
Collins sat down for a long interview with Vanity Fair's Chris Murphy that is full of so many excellent moments and quips that I actually cheered aloud while reading it, more than once!
https://www.vanityfair.com/hollywood/story/the-onions-ben-collins-knows-how-to-save-media
Collins believes he saved The Onion from "AI death," and I think he's right. Instead, he's produced a site that fights fascism using one of history's most reliable methods, satire: "There’s nothing fascists hate more than getting truly ripped on." Collins points out something interesting about Trump: "He never laughs…He’s funny in the sense that callous people can be particularly biting, but he’s not funny."
Here's his advice to other would-be media barons: "Kowtowing to power— your job is not that, nobody fucking wants that."
Among other things, Collins used The Onion to acquire Alex Jones's Infowars out of bankruptcy, only to have far-right legal shenanigans interrupt the hostile takeover (it's still in the courts).
(Paywall-busting version here:)
Another funny – but much angrier – independent media voice is Ed Zitron, one of the best ranters in technology. Ed's motto is "I hate them for what they did to the computer," a phrase I like so much I used it as the epigraph for my next book. Ed's just published the longest-ever post on his excellent "Where's Your Ed At?" newsletter, called "The Era of the Business Idiot":
https://www.wheresyoured.at/the-era-of-the-business-idiot/
In this post, Ed tried to answer the burning question, "Why are these objectively very stupid people given so much power over so much capital, and the lives of so many of us?" He lashes out at everyone – MBA programs, sociopathic Jack Welch-alikes, the supine press, and more. And he coins a truly excellent epithet for one of our most cherished business idiots, Open AI CEO Sam Altman:
SLOPPENHEIMER.
I love Ed's work, which focuses extensively on the internal ideological and personal traits of business leaders. But I think that any study of the enshittocene – and any effective opposition to enshittification – needs to start with policy, the legal arrangements that create an enshittogenic environment that allow the business idiots to wreak havoc without the constraints of competition, regulation, an empowered workforce or technological countermeasures.
In the EU, the epicenter of enshittogenesis is Ireland, a tax haven that has attracted the largest and worst American tech companies who maintain the fiction that they are based in Eire. But these companies are hardly loyal to Dublin: any company footloose enough to pretend that it's Irish this week can pretend to be Maltese, Luxembourgeois, Cypriot or Dutch next week. To keep those companies from upping sticks, Ireland must not only offer them criminally favorable tax treatment, they have to slow walk or ignore all regulations that discipline the enshittificatory impulses of Big Tech:
https://pluralistic.net/2023/05/15/finnegans-snooze/#dirty-old-town
Writing in Politico, Eoin Drea lays out the ways that Ireland is serving Trump's agenda to protect US Big Tech from EU regulators:
https://www.politico.eu/article/ireland-donald-trump-fines-white-house-washington-uk/
In particular, Drea identifies the risk that Ireland will shelter US companies from enforcement of the Digital Markets Act, the EU's "crowning legislative jewel." Ireland's PM has been carrying water for Trump, pressuring the EU to be "considered and measured" in its response to Trump's aggression; he's also vowed to "resist" the EU's digital taxes. Drea argues that centralizing enforcement of EU tech regulation in Brussels and the federal courts will relieve Ireland of the pressure to defend Trump's policies, since they will no longer be in a position to protect tech companies from Europe's rules.
When it comes to flouting EU rules, one of the most egregious "Irish" tech offenders is Meta. In a long article for Ars Technica, Ashley Belanger looks at Zuckerberg's recent statements about Facebook's future as a place where lonely people, having been alienated from their actual friends and families by a system that downranks posts from your social network to create space for ads and boosted posts, befriend AI chatbots instead:
I contributed a little to Belanger's excellent reporting, discussing my work with EFF on what an interoperable Facebook might look like, and how it might set Facebook's prisoners free:
https://www.eff.org/interoperablefacebook
Mark Zuckerberg's transformation from a historically awful person to a historically monstrous person has been really something to see. In this week's Saturday Morning Breakfast Cereal webtoon, Zach Weinersmith scores a body-blow on Zuck that was so perfect it made me bark with laughter:
https://www.smbc-comics.com/comic/gently
I firmly believe that Zuckerberg's transformation isn't due to the decay of his character. I think Zuck was always a creep, as any reader of Sarah Wynn-Williams's tell-all Facebook memoir Careless People can attest:
https://pluralistic.net/2025/04/23/zuckerstreisand/#zdgaf
Rather, I think the collapse of the internet into what Tom Eastman calls "five giant websites filled with screenshots of the other four" relieved Zuck of his nagging fear that a competitor would poach his users if he abused them too much. This is the enshittogenic environment at work – when we let firms form cartels, their owners become oligarchs.
Tech is far from the only cartel. In publishing, we only have five major publishers left, and the largest, Bertelsmann, dwarfs the other four. It's hard to overstate how gigantic Bertelsmann is, but here's a trenchant example: Bertelsmann owns Penguin-Random House, and PRH has publishing deals with five sitting Supreme Court justices. This meant that a majority of the court had to recuse itself from hearing a plagiarism case involving a Ta-Nehisi Coates book. It's the first time a mass-recusal has scuppered a Supreme Court case since 1945, when the majority of justices disclosed that they were stockholders in Alcoa, a monopolist:
https://www.newsweek.com/five-supreme-court-justices-sit-out-case-rare-move-2074666
Oligarchs are intrinsically enshittogenic. Oligarchs use their money and power to support strongmen who will trade money for government action, like Donald Trump, who offered a private dinner for major holders of his TRUMP shitcoin. The announcement prompted a ferocious bidding war among foreign agents and convicted criminals to buy up Trumpcoins and get a seat at the table:
https://www.citationneeded.news/trump-memecoin-dinner-guests/
Trump defenders claimed that the president was just rewarding people who understood the value of his coin, and not selling influence at all. Apparently, the attendees didn't get the memo, with many of them dumping their Trumpcoins the instant they were added to the guest-list:
https://protos.com/trump-token-15-since-dinner-as-40-of-guests-dump-by-dessert/
Joke's on them, though: Trump stiffed them! He showed up, gave a 15 minute speech (practically a haiku by Trump's normal standard of bloviation), then climbed into a helicopter and flew away, hundreds of millions of dollars richer thanks to the suckers left to their rubber chicken banquet:
https://link.nymag.com/view/640f640416f22cc291043cebntiap.15g1/0da0f946
Those specific oligarchs didn't get a chance to petition Trump to enact their favored policies, but Trump is still delivering for oligarchs. The "Big Beautiful Bill" that was passed in the dead of night last week included a whole raft of "sleeper" provisions, each worse than the last, as enumerated by The American Prospect's Robert Kuttner:
Among these:
$45 billion for immigration gulags, to be built by Trump's favorite beltway bandits;
a nonprofit killer that lets the president cancel the nonprofit status of any org that challenges him (this died earlier last week and was revived in the "Big Beautiful Bill");
doubling the threshold for estate taxes, so a couple can leave $30m to their heirs tax-free, meaning that only 0.8% of US households will face any estate tax;
gutting the child tax credit, taking away support from 4.5m children of taxpaying parents who lack a Social Security Number and making millions more ineligible;
cutting health coverage for millions of people dependent on Obamacare; and
getting rid of the excise tax on gun silencers.
We're heading into some dark times indeed. It can be hard to imagine things ever getting better, but there was one author who consistently imagined bold, utopian, audacious far futures: Iain M. Banks, whose "Culture" series remain one of the greatest science fiction visions ever published:
https://en.wikipedia.org/wiki/Culture_series
Banks's books are available in cheap paperbacks, and there's no shortage of used copies, but if you're looking to get a truly gorgeous Banks volume, the Folio Society has you covered, with a new, slipcased edition of Use of Weapons:
https://www.foliosociety.com/usa/use-of-weapons.html
I love Folio books and often give them as Christmas gifts to the people who matter most to me on my list. This one comes with seven full-page illustrations by Dániel Taylor.
In other publishing news, I got a care-package from my publisher this week: a box of advance review copies of my next book, Enshittification: Why Everything Suddenly Got Worse and What to Do About It, which Farrar, Straus and Giroux will publish next October:
https://us.macmillan.com/books/9780374619329/enshittification/
I'm very proud of how this one came out as a book, but I'm just as excited by how gorgeous this book is as an artifact:
https://www.flickr.com/photos/doctorow/54540006021/
I'm going into the studio to record the audiobook in August, and there's a graphic novel and documentary in the offing.
#20yrsago Pat York: dear friend, writer, Boing Boing guestblogger, RIP https://memex.craphound.com/2005/05/22/pat-york-dear-friend-writer-boing-boing-guestblogger-rip/
#20yrsago Alan Moore tells DC Comics to get bent https://web.archive.org/web/20050527220922/http://www.comicbookresources.com/columns/index.cgi?column=litg&article=2153
#20yrsago Thurl Ravenscroft, RIP: voice of Haunted Mansion and Grinch song, Tony the Tiger https://web.archive.org/web/20050525220256/https://www.newsfromme.com/archives/2005_05_23.html#009891
#20yrsago Dutch mayor wants to ban hacker con https://web.archive.org/web/20050525160551/https://www.whatthehack.org/news/index_html
#15yrsago Ireland’s largest ISP begins disconnecting users who are accused of piracy https://web.archive.org/web/20100605170505/https://www.irishtimes.com/newspaper/frontpage/2010/0524/1224271013389.html
#15yrsago Mark Twain’s autobiography to be finally published, 100 years after his death https://www.the-independent.com/arts-entertainment/books/news/after-keeping-us-waiting-for-a-century-mark-twain-will-finally-reveal-all-1980695.html
#15yrsago Igor Stravinsky, arrested for “tampering” with the Star Spangled Banner, 1940 https://web.archive.org/web/20100526120708/http://dcmusicaviva.blogspot.com/2009/03/stravinsky-mugshot.html
#15yrsago Mechanical irising peephole mechanism https://www.talkshopbot.com/forum/showthread.php?795-More-mechanical-wooden-silliness
#15yrsago InfoLadies of Bangladesh revolutionize rural life https://www.theguardian.com/journalismcompetition/professional-two-wheel-triumph
#15yrsago Google and Viacom blend high-profile copyright suits with extreme profanity, as nature intended https://arstechnica.com/tech-policy/2010/05/f-those-mother-f-ers-youtubeviacom-lawsuit-gets-dirty/
#15yrsago Google offers encrypted search https://www.eff.org/deeplinks/2010/05/google-launches-encrypted-search
#15yrsago The Boneshaker: magic, latter-day Bradburian novel for young adults https://memex.craphound.com/2010/05/21/the-boneshaker-magic-latter-day-bradburian-novel-for-young-adults/
#15yrsago Scientology raid uncovers dossiers on local “enemies”: sexual habits, health info, political opinions https://www.ansa.it/web/notizie/rubriche/english/2010/05/20/visualizza_new.html_1794804082.html
#15yrsago Cracked vs. RIAA damages https://web.archive.org/web/20100524024915/http://www.cracked.com/funny-4003-the-pirate-bay/
#10yrsago NSA wanted to hack the Android store https://www.cbc.ca/news/canada/spy-agencies-target-mobile-phones-app-stores-to-implant-spyware-1.3076546
#10yrsago GM says you don’t own your car, you just license it https://web.archive.org/web/20150522003554/https://consumerist.com/2015/05/20/gm-that-car-you-bought-were-really-the-ones-who-own-it/
#10yrsago Today’s terrifying Web security vulnerability, courtesy of the 1990s crypto wars https://memex.craphound.com/2015/05/21/todays-terrifying-web-security-vulnerability-courtesy-of-the-1990s-crypto-wars/
#10yrsago Mark Zuckerberg just dropped another $100M to protect his privacy https://slate.com/business/2015/05/tech-billionaires-and-privacy-why-facebook-s-mark-zuckerberg-is-spending-millions-on-a-private-island.html
#10yrsago Paper on changing peoples’ minds about marriage equality retracted https://retractionwatch.com/2015/05/20/author-retracts-study-of-changing-minds-on-same-sex-marriage-after-colleague-admits-data-were-faked/
#10yrsago The Man Who Sold The Moon https://memex.craphound.com/2015/05/22/the-man-who-sold-the-moon/
#10yrsago Popehat on depression https://web.archive.org/web/20150524013923/http://popehat.com/2015/05/21/happy-to-be-here/
#10yrsago An Internet of Things that act like red-light cameras https://www.antipope.org/charlie/blog-static/2015/05/the-evil-business-plan-of-evil.html
#10yrsago danah boyd explains student privacy bills https://medium.com/message/which-students-get-to-have-privacy-e9773f9a064
#10yrsago Hedge funds buy swathes of foreclosed subprimes, force up rents, float rent-bonds https://www.motherjones.com/politics/2014/02/blackstone-rental-homes-bundled-derivatives/
#10yrsago Hacktivist sees too much, FBI lock him up on child-porn charges, produce no evidence https://www.newsweek.com/2015/05/29/porn-run-333599.html
#5yrsago Casio censors calculator modder's Github project https://pluralistic.net/2020/05/23/penguin-farts/#copyfraud
#5yrsago Covid apps and false positives https://pluralistic.net/2020/05/23/penguin-farts/#false-alarms
#5yrsago Physical BLINK tag https://pluralistic.net/2020/05/22/crisis-for-thee-not-me/#html-follies
#5yrsago Mum uses GDPR to force Gran to take down pics https://pluralistic.net/2020/05/22/crisis-for-thee-not-me/#family-feud
#5yrsago Coronavirus has made the super-rich MUCH richer https://pluralistic.net/2020/05/22/crisis-for-thee-not-me/#what-crisis
#5yrsago Copyright bots are slaughtering classical musicians' performances https://pluralistic.net/2020/05/22/crisis-for-thee-not-me/#filternet
#5yrsago Oh Joy Sex Toy's new teen sex-ed book https://pluralistic.net/2020/05/22/crisis-for-thee-not-me/#ojst
#5yrsago How spy agencies targeted Snowden journalists https://pluralistic.net/2020/05/21/profitable-butchers/#sources-and-methods
#5yrsago Monopolies killed corporate R&D https://pluralistic.net/2020/05/21/profitable-butchers/#all-d
#5yrsago Spotify's trying to kill podcasting https://pluralistic.net/2020/05/21/profitable-butchers/#rogan-monopolist
#5yrsago Black Americans' covid mortality is 2.5X white mortality https://pluralistic.net/2020/05/21/profitable-butchers/#ethnic-cleansing
#5yrsago On Madame Leota's side-table https://pluralistic.net/2020/05/21/profitable-butchers/#details-details
#5yrsago Private equity's healthcare playbook is terrifying https://pluralistic.net/2020/05/21/profitable-butchers/#looted
#5yrsago Patent troll sues ventilator makers https://pluralistic.net/2020/05/21/profitable-butchers/#ip-edge
#5yrsago The Lost Cause and MMT https://pluralistic.net/2020/05/21/profitable-butchers/#byebye-falc
#5yrsago Walt's grandson calls for Disney execs' bonuses to be canceled https://pluralistic.net/2020/05/21/profitable-butchers/#brad-lund
#1yrago Linkrot https://pluralistic.net/2024/05/21/noway-back-machine/#pew-pew-pew
#1yrago How finfluencers destroyed the housing and lives of thousands of people https://pluralistic.net/2024/05/22/koteswar-jay-gajavelli/#if-you-ever-go-to-houston
#1yrago Red Lobster was killed by private equity, not Endless Shrimp https://pluralistic.net/2024/05/23/spineless/#invertebrates
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Keynote (Pycon)
https://www.youtube.com/watch?v=ydVmzg_SJLw
Reclaiming the Internet (Easy Prey)
https://www.easyprey.com/reclaiming-the-internet/
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources: Hacker News (https://news.ycombinator.com/), Kottke (http://kottke.org/), Tim Harford (https://timharford.com/), Super Punch (https://www.superpunch.net/), Jamie Boyle (https://www.thepublicdomain.org/).
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
welcome to the future, now your error-prone software can call the cops
The emissions from individual AI text, image, and video queries seem small—until you add up what the industry isn't tracking and consider where it's heading next.
My latest Locus Magazine column is "Strange Bedfellows and Long Knives," about the secret engine of sweeping political upheavals (like Trumpism) and their inherent fragility:
https://locusmag.com/2025/05/commentary-cory-doctorow-strange-bedfellows-and-long-knives/
Stories about major change usually focus on a group, but groups rarely achieve big, ambitious goals. Think about all the goal-oriented groups in your orbit, with missions like alleviating hunger, or beautifying your neighborhood, or changing the health-care system. They've been at it for decades, and while many groups do excellent work at the margins, blocking regressions and making modest advancements (or the occasional breakthrough), they're playing a game of inches.
But sometimes – the New Deal, the civil rights movement, the Reagan revolution, Trump II – we get a wholesale, foundational, societal change. Very rarely, that's because an existing group conceived of a devastating new tactic (think of Obama's online campaigning in 2008), but that's the exception. Almost always, the major upheavals in our society aren't caused by the same people trying a different tactic – they're the result of a coalition that forms around a shared set of goals.
Reagan rode to power thanks to the support of different groups, many of whom had cordially loathed one another for decades. Most notably, Reagan brokered a deal with evangelicals – whose movement was already organized around strict obedience to charismatic cult leaders – to end their decades long boycott of politics and show up at the polls for him:
Evangelicals hated politicians (whom they viewed as obsessed with "worldly" matters to the exclusion of the spiritual) and they really hated the finance sector (whom they damned as both amoral sons of Mammon, and also, quietly, Jewish). Right wing politicians and the financiers they relied on viewed evangelicals as stupid, superstitious, and ungovernable. But by promising to deliver culture-war stuff (racism, restrictions on abortion, homophobia) to evangelicals, and tax-cuts and deregulation to the rich, Reagan fused two groups that had been largely stalled in achieving their goals for decades, and, with the backing of that coalition, rewrote the American consensus to give each of them some of what they wanted.
But here's the thing about coalitions: while they share some goals, they don't share all their goals. Two groups that have identical goals aren't actually two groups – that's just one group with two chapters. Moreover, the divergence in coalition members' goals are often – nearly always – in conflict. Which is to say, they want some of the same things, but there are always group members who want different, mutually exclusive, opposing things.
When coalitions are forming and campaigning, they tend to focus on their shared goals. But once they take power, it's their differences that matter.
Think of Tolkien: the Fellowship of the Ring forms by pulling together disparate factions to join in a shared quest that culminates in a massive battle in which (spoilers) they are victorious. But in the immediate aftermath of that victory, even before the wounded and the fallen have been recovered from the battlefield, we (spoilers) witness another fight, this one between the allies, over what the post-victory order will be. This is pretty much also what happened after WWII, when (spoilers) the USSR and the USA switched from being allies to being rivals even before anyone could (spoilers) clean Hitler's brains off the walls of his bunker.
Leftists get a front-row seat for the coalitional moves of the right, but we tend to miss the internecine struggle to claim the prize after their victories. One exception to this is Rick Perlstein, a leftist historian whose books Nixonland and Reaganland are definitive histories of the internal machinations that powered the right wing revolution. For years, Perlstein has been carefully reading the massive anthologies that the Heritage Foundation publishes in the runup to each election, in which various members of the right coalition spell out their post-victory goals. These were pretty obscure until last year, when we all became aware of the latest volume in the series, Project 2025:
https://pluralistic.net/2024/07/14/fracture-lines/#disassembly-manual
Perlstein read Project 2025 – all of it, not just the individual chapters that were the most lurid and apocalyptic right-wing fantasies. Because Perlstein read all 900 pages, he was able to identify something that nearly everyone else missed, that Project 2025 is full of contradictory plans that are in direct opposition to one another:
https://prospect.org/politics/2024-07-10-project-2025-republican-presidencies-tradition/
Project 2025 is usually credited to the Heritage Foundation, but it's more accurate to say that Heritage was the anthologist of the plan, not its author. They selected and assembled chapters written by various members of the Trump coalition. Now, as anthologist, it was Heritage's job to make as coherent a job of this as possible, but, as it turns out "as possible" wasn't very possible.
Project 2025 contains multiple, contradictory, mutually opposed prescriptions for monetary policy, taxation, foreign policy, domestic security, government reform, taxation, and more. Normally, an anthologist editing a volume like this would serve as a kind of referee, choosing winners from among these opposing sides. That surely happens all the time in Trumpland – doubtless there are crank eugenicists, Proud Boys, and Q-addled hallucinators who have cherished goals that would never make it into Project 2025.
But the fact that Heritage couldn't tell one (or two, or three) sides in these debates to go pound sand and elevate a single policy to canon tells us that there are opposing forces in the Trump coalition who are each so powerful that neither of them can overpower the others. These are the fracture lines in the Trump coalition, the places we should apply ourselves to if we want to neutralize the movement, shatter it back into a mob of warring factions.
As Naomi Klein says, this is something Steve Bannon has been doing to the left for years:
One of the things I’ve learned from studying Steve Bannon is he takes the task of peeling away parts of the Democrats’ coalition very seriously, and he’s done it very successfully again and again. So why wouldn’t we try to do it back to him?
https://prospect.org/culture/2025-05-13-moment-of-unparalleled-peril-interview-naomi-klein/
The Trump coalition's fracture lines are already showing, for example, in healthcare:
https://pluralistic.net/2024/12/20/clinical-trial-by-ordeal/#spoiled-his-brand-new-rattle
And tariffs:
https://www.rawstory.com/trump-peter-navarro/
And Elon Musk:
https://www.cnn.com/2025/04/08/business/elon-musk-peter-navarro-comments-tariffs
Trump held his coalition together during the war, but history tells us that now, after the victory, is the moment when Trump's coalition is most vulnerable, as members of that coalition realize that they won't get the things they were promised in exchange for the blood and treasure they expended to get Trump into office.
I've been a Locus columnist for two decades now. It remains the journal of record for the science fiction and fantasy field, a vital source of information and community. Locus is structured as a charitable nonprofit (I'm a donor) and it depends on support from readers like you to keep going. They're currently hosting their annual fundraiser, with many, many, many cool rewards, from signed books to the right to name a character in an upcoming novel, and beyond:
https://www.indiegogo.com/projects/locus-mag-science-fiction-fantasy-horror-2025#/
I Tried to Jailbreak a 'Pay-To-Ride' Ebike — Total Nightmare https://www.youtube.com/watch?v=yJC9s4XhiRE&t=1s (h/t JWZ)
Medicare Fraud? Insider Trading? Another New CEO? Just a Normal Week at UnitedHealth https://prospect.org/health/2025-05-21-medicare-fraud-insider-trading-new-ceo-unitedhealth/
#20yrsago Italian phone carriers have phone-unlockers arrested https://memex.craphound.com/2005/05/20/italian-phone-carriers-have-phone-unlockers-arrested/
#15yrsago Critical paths and self-publishing https://www.publishersweekly.com/pw/by-topic/columns-and-blogs/cory-doctorow/article/43187-closing-in.html
#15yrsago JHEREG license plate https://memex.craphound.com/2010/05/20/jhereg-license-plate/
#15yrsago UK government promises immediate, sweeping, pro-liberty reform https://arstechnica.com/tech-policy/2010/05/new-uk-govt-to-curb-cctv-scrap-id-cards-help-open-source/
#15yrsago Canada’s sellout Heritage Minister ready to hand copyright to Hollywood https://web.archive.org/web/20100523073407/https://www.michaelgeist.ca/content/view/5049/125/
#15yrsago NYC sidewalk with a tourist lane https://web.archive.org/web/20100523071746/http://www.ireport.com/docs/DOC-446601?hpt=Sbin
#15yrsago Charts of UK Parliamentary language usage, 1935- https://web.archive.org/web/20100523074752/http://contentini.com/an-analysis-of-uk-parliamentary-language-1935-2010/
#15yrsago NYC writer’s space throws out last remaining typewriter user https://web.archive.org/web/20100523093837/http://www.nydailynews.com/ny_local/2010/05/20/2010-05-20_untitled__typewriter20m.html
#15yrsago World Sauna Championships: blisters, peeling skin, burned throats https://www.espn.com/espn/news/story?id=5198604
#10yrsago Digital rights news from 2025 https://edri.org/our-work/edri-gram-300-digital-rights-news-2025/
#10yrsago FBI spies on tar sands opponents under banner of “national security” https://web.archive.org/web/20150523001136/https://firstlook.org/theintercept/2015/05/19/fbi-invokes-national-security-justify-surveillance-tar-sands-protestors/
#10yrsago Bottled water: the ultimate throwback to feudal selfishness https://gizmodo.com/stop-drinking-bottled-water-1704609514
#10yrsago Chris Christie denounces “civil liberties extremists” https://www.huffpost.com/entry/chris-christie-911-attacks-coffin_n_7309628
#5yrsago Ifixit's medtech repair manual trove is full to bursting https://pluralistic.net/2020/05/20/oil-in-the-cloud/#ifixit
#5yrsago Re-positive coronavirus cases are not infectious https://pluralistic.net/2020/05/20/oil-in-the-cloud/#whew
#5yrsago Google vows no custom AI for oil and gas https://pluralistic.net/2020/05/20/oil-in-the-cloud/#no-ml-for-oil
#5yrsago Grocer won't show employees results from mandatory temp-checks https://pluralistic.net/2020/05/20/oil-in-the-cloud/#thermal-privacy
#5yrsago Software secrecy https://pluralistic.net/2020/05/20/oil-in-the-cloud/#floss
#5yrsago Climate and machine learning https://pluralistic.net/2020/05/20/oil-in-the-cloud/#digging-coal-with-your-spacebar
#5yrsago Hackers on Planet Earth, in the cloud https://pluralistic.net/2020/05/20/oil-in-the-cloud/#hope
#1yrago The new globalism is global labor https://pluralistic.net/2024/05/20/a-common-foe/#the-multinational-playbook
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
In God We Antitrust (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16147052-in-god-we-antitrust
Fireside Fedi
https://video.firesidefedi.live/w/huevh4L6r1yMYXqcQMi8gR
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
It's kinda weird how, the more oligarchic our society gets, the more racist it gets. Why is the rise of billionaires attended by a revival of discredited eugenic ideas, dressed up in modern euphemisms like "race realism" and "human diversity"?
I think the answer lies in JK Galbraith's observation that "The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is, the search for a superior moral justification for selfishness."
The theory of markets goes like this: a market is a giant computer that is always crunching all kinds of "signals" about what people want and how much they want it, and which companies and individuals are most suited to different roles within the system. The laissez-faire proposition is that if we just resist the temptation to futz with the computer (to "distort the market"), it will select the best person for each position: workers, consumers, and, of course, "capital allocators" who decide where the money goes and thus what gets made.
The vast, distributed market computer is said to be superior to any kind of "central planning" because it can integrate new facts quickly and adjust production to suit varying needs. Let rents rise too high and the computer will trigger the subroutine that brings "self-interested" ("greedy") people into the market to build more housing and get a share of those sky-high rents, "coming back into equilibrium." But allow a bureaucracy to gum up the computer with a bunch of rules about how that housing should be built and the "lure new homebuilders" program will crash. Likewise, if the government steps in to cap the price of rents, the "price signal" will be silenced and that "new homebuilders" program won't even be triggered.
There's some logic to this. There are plenty of good things that market actors do that are motivated by self-interest rather than altruism. When Google founders Larry Page and Sergey Brin developed their Pagerank algorithm and revolutionized internet search, they weren't just solving a cool computer science problem – they were hoping to get rich.
But here's the thing: if you let Larry and Sergey tap the capital markets – if they can put on a convincing show for the "capital allocators" – then the market will happily supply them with the billions they need to buy and neutralize their competitors, to create barriers to entry for superior search engines, and become the "central planners" that market theory so deplores. If your business can't get any market oxygen, if no audience ever discovers your creative endeavors, does it matter if the central planner who decided you don't deserve a chance is elected or nominated by "the market"?
Here's how self-proclaimed market enthusiasts answer that question: all Larry and Sergey are doing here is another form of "capital allocation." They're allocating attention, deciding what can and can't be seen, in just the same way that a investor decides what will and won't be funded. If an investor doesn't fund promising projects, then some other investor will come along, fund them, get rich, and poach the funds that were once given to less-successful rivals. In the same way, if Google allocates attention badly, then someone will start a better search engine that's better at allocating attention, and we will switch to that new search engine, and Google will fail.
Again, this sounds reasonable, but a little scrutiny reveals it to be circular reasoning. Google has dominated search for a quarter of a century now. It has a 90% market share. According to the theory of self-correcting markets, this means that Google is very good at allocating our attention. What's more, if it feels like Google actually sucks at this – like Google's search-results are garbage – that doesn't mean Google is bad at search. It doesn't mean that Google is sacrificing quality to improve its bottom line (say, by scaling back on anti-spam spending, or by increasing the load of ads on a search results page).
It just means that doing better than Google is impossible. You can tell it's impossible, because it hasn't happened.
QED.
Google wasn't the first search engine, and it would be weird if it were the last. The internet and the world have changed a lot and the special skills, organizational structures and leadership that Google assembled to address the internet of the 2000s and the 2010s is unlikely to be the absolute perfect mix for the 2020s. And history teaches us that the kinds of people who can assemble the skills, structures and leaders to succeed in one era are unlikely to be able to change over to the ideal mix for the next era.
Interpreting the persistent fact of Google's 90% market-share despite its plummeting quality as evidence of Google's excellence requires an incredible act of mental gymnastics. Rather than accepting the proposition that Google both dominates and sucks because it is excellent, we should at least consider the possibility that Google dominates while sucking because it cheats. And hey, wouldn't you know it, three federal courts have found Google to be a monopolist in three different ways in just a year.
Now, the market trufans will tell you that these judges who called Google a cheater are just futzers who can't keep their fingers off the beautiful, flawless market computer. By dragging Google into court, forcing its executives to answer impertinent questions, and publishing their emails, the court system is "distorting the market." Google is the best, because it is the biggest, and once it stops being the best, it will be toppled.
This makes perfect sense to people who buy the underlying logic of market-as-computer. For the rest of us, it strains credulity.
Now, think for a minute of the people who got rich off of Google. You have the founders – like Sergey Brin, who arrived in America as a penniless refugee and is now one of the richest people in the history of the human species. He got his fortune by building something that billions of us used trillions of times (maybe even quadrillions of times) – the greatest search engine the world had ever seen.
Brin isn't the only person who got rich off Google, of course. There are plenty of Googlers who performed different kinds of labor – coding, sure, but also accountancy, HR, graphic design, even catering in the company's famous cafeterias – who became "post-economic" (a euphemism for "so rich they don't ever need to think about money ever again") thanks to their role in Google's success.
There's a pretty good argument to be made that these people "earned" their money, in the sense that they did a job and that job generated some money and they took it home. We can argue about whether the share of the profits that went to different people was fair, or whether the people whose spending generated that profit got a good deal, or whether the product itself was good or ethical. But what is inarguable is that this was money that people got for doing something.
Then there's Google's investors. They made a lot of money, especially the early investors. Again, we can argue about whether investors should be rewarded for speculation, but there's no question that the investors in Google took a risk and got something back. They could have lost it all. In some meaningful sense, they made a good choice and were rewarded for it.
But now let's think about the next generation. The odds that these billionaires, centimillionaires and decimillionaires will spawn the next generation of 1%ers, 0.1%ers, and 0.0001%ers are very high. Right now, in America, the biggest predictor of being rich is having rich parents. Every billionaire on the Forbes under-30 list inherited their wealth:
https://ca.finance.yahoo.com/news/forbes-billionaires-under-30-inherited-203930435.html
The wealthy have created a system of dynastic wealth that puts the aristocratic system of primogenitor in the shade. Every scion of every one-percenter can have their own fortune and start their own dynasty, without lifting a finger. Their sole job is to sign the paperwork put before them by "wealth managers":
https://pluralistic.net/2021/06/19/dynastic-wealth/#caste
Yes, it's true that some of the very richest people on Earth got their money by investing, rather than inheriting it. Bill Gates's investment income growth exceeds even the growth of the world's richest woman, L'Oreal heiress Liliane Bettencourt, who never did anything of note apart from emerging from an extremely lucky orifice and then simply accruing:
https://memex.craphound.com/2014/06/24/thomas-pikettys-capital-in-the-21st-century/
But Bill Gates's wealth accumulation from investing exceeds the wealth he accumulated by founding and running the most successful company in history (at the time). Doing work never pays as much as allocating capital. And Gates's children? They can assume a Bettencourtian posture on a divan, mouths yawning wide for the passage of peeled grapes, and their fortunes will grow still larger. Same goes for their children, and their children's children.
Capitalism's self-mythologizing insists that the invisible hand owes no allegiance to yesterday's champions. The mere fact that the market rewarded you for allocating capital wisely during your tenure does not entitle your offspring to continue to allocate wealth in the years and centuries to come – not unless they, too, are capital allocators of such supremacy that they are superior to everyone born hereafter and will make the decisions that make the whole world better off.
Because that's the justification for inequality: that the market relentlessly seeks out the people with the skill and foresight to do things and invest in things that improve the world for all of us. If we interrupt that market process with regulations, taxes, or other "distorting" factors, then the market's quest for the right person for the right job will be thwarted and all of us will end up poorer. If we want the benefits of the invisible hand, we must not jostle the invisible elbow!
That's the justification for abolishing welfare, public education, public health, affirmative action, DEI, and any other programs that redistribute wealth to the least among us. If we get in the way of the market's selection process, we'll elevate incompetents to roles of power and importance and they will bungle those roles in ways that hurt us all. As Boris Johnson put it: "the harder you shake the pack the easier it will be for [big] cornflakes to get to the top":
https://www.theguardian.com/politics/2013/nov/28/boris-johnson-iq-intelligence-gordon-gekko
Which leaves the servants and defenders of the invisible hand with a rather awkward question: how is it that today, capital allocation is a hereditary role? We used to have the idea that fitness to allocate capital – that is, to govern the economy and the lives of all of the rest of us – was a situational matter. The rule was "shirtsleeves to shirtsleeves in three generations": "The first generation makes it, the second generation spends it, and the third generation blows it."
That's the lesson of the rags to riches story*: that out there, amongst the teeming grubby billions, lurks untold genius, waiting to be anointed by the market and turned loose to make us all better off.
https://newenglandhistoricalsociety.com/horatio-alger-hundred-year-old-secret/
Perhaps your life was saved by a surgeon who came from humble origins but made it through med school courtesy of Pell Grants. Perhaps you thrilled to a novel or a film made by an artist from a working class family who got their break through an NEA grant. Maybe the software you rely on every day, or the game that fills your evenings, was created by someone who learned their coding skills at a public library or publicly funded after-school program.
The presence among us of people who achieved social mobility and made our lives better is evidence that people are being born every moment with something to contribute that is markedly different, and higher in social status, than the role their parents played. Even if you stipulate that the person who cleans your toilet has been correctly sorted into a toilet-cleaning job by the invisible hand, it's clear that the invisible hand would prefer that at least some of those toilet-cleaners' kids should do something else for a living.
And yet, wealth remains stubbornly hereditary. Our capital allocators – who, during the post-war, post-New Deal era were often drawn from working families – are now increasingly, relentlessly born to that role.
For the wealthy, this is the origin of the meritocracy to eugenics pipeline. If power and privilege are inherited – and they are, ever moreso every day – then either we live in an extremely unfair society in which the privileged and the powerful have rigged the game…or the invisible hand has created a subspecies of thoroughbred humans who were literally born to rule.
This is the thesis of the ultra-rich, the moral justification for rigging the system so that their failsons and faildaughters will give rise to faildestinies of failgrandkids and failgreat-grandkids, whose emergence from history's luckiest orifices guarantees them a lifelong tenure ordering other people around. It's the justification for some people being born to own the places where the rest of us live, and the rest of us paying them half our salaries just so we don't end up sleeping on the sidewalk.
"Hereditary meritocracy" is just a polite way of saying "eugenics." It starts from the premise of the infallible invisible hand and then attributes all inequality in society to the hand's perfect judgment, its genetic insight in picking the best people for the best jobs. If people of one race are consistently on top of the pile, that's the market telling you something about their genomes. If men consistently fare better in the economy than women, the invisible hand is trying to say something about the Y chromosome for anyone with ears to hear.
Capitalism's winners have always needed "a superior moral justification for selfishness," a discreet varnish to shine up the old divine right of kings. Think of the millionaire who created a "Nobel Prize sperm-bank" (and then fraudulently fathered hundreds of children because he couldn't find any Nobelists willing to make a deposit):
https://memex.craphound.com/2006/09/07/nobel-prize-sperm-bank-human-tragicomedy-about-eugenics/
Or the billionaire founder of Telegram who has fathered over 100 children in a bid to pass on his "superior genes":
https://www.cnn.com/2024/08/26/tech/pavel-durov-telegram-profile-intl
Think of Trump and his endless boasting about his "good blood" and praise for the "bloodlines" of Henry Ford and other vicious antisemites:
Or Elon Musk, building a compound where he hopes to LARP as Immortan Joe, with a harem of women who have borne his legion of children, who will carry on his genetic legacy:
https://www.nytimes.com/2024/10/29/business/elon-musk-children-compound.html
Inequality is a hell of a drug. There's plenty of evidence that becoming a billionaire rots your brain, and being born into a dynastic fortune is a thoroughly miserable experience:
https://pluralistic.net/2021/04/13/public-interest-pharma/#affluenza
The stories that rich people tell themselves about why this is the only way things can be ("There is no alternative" -M. Thatcher) always end up being stories about superior blood. Eugenics and inequality are inseparable companions.
"Groomer"-Gate Revisited: Trial Exhibits Reveal More Details About Instagram's Disturbing Algorithms https://www.bigtechontrial.com/p/groomer-gate-revisited-trial-exhibits
Chicago Sun-Times Prints AI-Generated Summer Reading List With Books That Don't Exist https://www.404media.co/chicago-sun-times-prints-ai-generated-summer-reading-list-with-books-that-dont-exist/
#20yrsago Canadian court’s file-sharing ruling is mixed blessing https://web.archive.org/web/20050521013510/https://www.michaelgeist.ca/home.php#406
#20yrsago Cuba switching to GNU/Linux https://linux.slashdot.org/story/05/05/19/1213245/cuba-switching-to-linux
#15yrsago Finnish record industry’s regrettable new anti-piracy mascot https://www.youtube.com/watch?v=HqpZiQDLOlY
#15yrsago Honey, I Wrecked the Kids: a guide to democratic parenting https://memex.craphound.com/2010/05/19/honey-i-wrecked-the-kids-a-guide-to-democratic-parenting/
#10yrsago America’s terrible trains are an ideological triumph https://www.newyorker.com/news/daily-comment/the-plot-against-trains
#10yrsago Taxi medallion markets collapse across America https://www.usatoday.com/story/news/2015/05/17/taxi-medallion-values-decline-uber-rideshare/27314735/
#5yrsago $10T to avert another Great Depression https://pluralistic.net/2020/05/19/larval-pegasi/#stimulus
#5yrsago Softbank's "pegasus" grift https://pluralistic.net/2020/05/19/larval-pegasi/#long-con
#5yrsago "Shoe-leather" contact tracing works https://pluralistic.net/2020/05/19/larval-pegasi/#shoe-leather
#5yrsago Marcus Yallow has coronavirus https://pluralistic.net/2020/05/19/larval-pegasi/#m1k3y
#5yrsago Lego's new Haunted House is wheelchair accessible https://pluralistic.net/2020/05/19/larval-pegasi/#delightfully-unlivable
#5yrsago Toothsome masks https://pluralistic.net/2020/05/19/larval-pegasi/#mask-up
Virtual: Fediforum, Jun 5
https://fediforum.org/2025-06/
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Fireside Fedi
https://video.firesidefedi.live/w/huevh4L6r1yMYXqcQMi8gR
Ctrl-ctrl-ctrl (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16145640-ctrl-ctrl-ctrl
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
Episode 3 of "Understood: Who Broke the Internet?" (my new CBC podcast about enshittification) just dropped. It's called "In God We Antitrust," and it's great:
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16147052-in-god-we-antitrust
The thesis of this four-part series is pretty straightforward: the enshittification of the internet was the result of an enshittogenic policy environment. Platforms always had the technical means to scam us and abuse us. Tech founders and investors always included a cohort of scumbags who would trade our happiness and wellbeing for their profits. What changed was the consequences of giving in to those impulses. When Google took off, its founders' mantra was "competition is just a click away." If someone built a better search engine, users could delete their google.com bookmarks, just like they did to their altavista.com bookmarks when Google showed up.
Policymakers – not technologists or VCs – changed the environment so that this wasn't true anymore:
https://pluralistic.net/2025/05/08/who-broke-the-internet/#bruce-lehman
In last week's episode, we told the story of Bruce Lehman, the Clinton administration's Copyright Czar, who swindled the US government into passing a law that made it illegal to mod, hack, reverse-engineer or otherwise improve on an existing technology:
https://pluralistic.net/2025/05/13/ctrl-ctrl-ctrl/#free-dmitry
This neutralized a powerful anti-enshittificatory force: interoperability. All digital tech is born interoperable, because of the intrinsic characteristics of computers, their flexibility. This means that tech is inherently enshittification-resistant. When a company enshittifies its products or services, its beleaguered users and suppliers don't have to wait for a regulator to punish it. They don't have to wait for a competitor to challenge it.
Interoperable tools – ad-blockers, privacy blockers, alternative clients, mods, plugins, firmware patches and other hacks – offer immediate, profound relief from enshittification. Every ten foot pile of shit that a tech company drops into your life can be met with an eleven foot ladder of disenshittifying, interoperable technology.
That's why Lehman's successful attack on tinkering was so devastating. Before Lehman, tech had achieved a kind of pro-user equilibrium: every time a company made its products worse, they had to confront a thousand guerrilla technologists who unilaterally unfucked things: third party printer ink, file-format compatibility, protocol compatibility, all the way up to Unix, a massive operating system that was painstakingly re-created, piece by piece, in free software.
Lehman offered would-be enshittifiers a way to shift this equilibrium to full enshittification: just stick a digital lock on your product. It didn't even matter if the lock worked – under Lehman's anticircumvention law, tampering with a lock, even talking about weaknesses in a lock, became a literal felony, punishable by a five-year prison sentence and a $500K fine. Lehman's law was an offer no tech boss would refuse, and enshittification ate the world.
But Lehman's not the only policymaker who was warned about the consequences of his terrible plans, who ignored the warnings, and who disclaims any responsibility for the shitty world that followed. Long before Lehman's assault on tech policy, another group of lawyers and economists laid waste to competition policy.
In the 1960s and 1970s, a group of Chicago School economists conceived of an absurd new way to interpret competition law, which they called "the consumer welfare standard." Under this standard, the job of competition policy was to encourage monopolies to form, on the grounds that monopolies were "efficient" and would lower prices for "consumers."
The chief proponent of this standard was Robert Bork, a virulent racist whose most significant claim to fame was that he was the only government lawyer willing to help Richard Nixon illegally fire officials who wouldn't turn a blind eye to his crimes. Bork's long record of unethical behavior and scorching bigotry came back to bite him in the ass when Ronald Reagan tried to seat him on the Supreme Court, during a confirmation hearing that Bork screwed up so badly that even today, we use "borked" as a synonym for anything that is utterly fucked.
But Bork's real legacy was as a pro-monopoly propagandist, whose work helped shift how judges, government enforcers, and economists viewed antitrust law. Bork approached the text of America's antitrust laws, like the Sherman Act and the Clayton Act, with the same techniques as a Qanon follower addressing a Q "drop," applying gnostic techniques to find in these laws mystical coded language that – he asserted – meant that Congress had intended for America's anti-monopoly laws to actually support monopolies.
In episode three, we explore Bork's legacy, and how it led to what Tom Eastman calls the internet of "five giant websites, each filled with screenshots of the other four." We got great interviews and old tape for this one, including Michael Wiesel, a Canadian soap-maker who created a bestselling line of nontoxic lip-balm kits for kids, only to have Amazon shaft him by underselling him with his own product.
But the most interesting interview was with Lina Khan, the generational talent who became the youngest-ever FTC chair under Joe Biden, and launched an all-out assault on American monopolies and their vile depredations:
https://pluralistic.net/2023/07/14/making-good-trouble/#the-peoples-champion
Khan's extraordinary rise to power starts with a law review paper she wrote in her third year at Yale, "Amazon's Antitrust Paradox," which became the first viral law review article in history:
https://www.yalelawjournal.org/note/amazons-antitrust-paradox
"Amazon's Antitrust Paradox" was a stinging rebuke to Bork and his theories, using Amazon's documented behavior to show that after Amazon used its monopoly power to lower prices and drive rivals out of the market, it subsequently raised prices. And, contrary to Bork's theories, those new, high prices didn't conjure up new rivals who would enter the market with lower prices again, eager to steal Amazon's customers away. Instead, Amazon's demonstrated willingness to cross-subsidize divisions' gigantic losses to destroy any competitor with below-cost pricing created a "kill zone" of businesses adjacent to the giant's core enterprise that no one dared enter:
https://www.thebignewsletter.com/p/how-biden-can-clean-up-obamas-big
The clarity of Khan's writing, combined with her careful research and devastating conclusions dragged a vast crowd of people who'd never paid much attention to antitrust – including me! – into the fray. No wonder that four years later, she was appointed to serve as the head of the FTC, making her the most powerful consumer rights regulator in the world.
We live in an age of monopolies, with cartels dominating every part of our lives, acting as "autocrats of trade" and "kings over the necessaries of life," the corporate dictators that Senator John Sherman warned about when he was stumping for the 1890 Sherman Act, America's first antitrust law:
https://pluralistic.net/2022/02/20/we-should-not-endure-a-king/
Bork and his co-religionists created this age. They're the reason we live in a world where we have to get our "necessaries of life" from a cartel, a duopoly or a monopoly. It's not because the great forces of history transformed the economy – it's because of these dickheads:
https://www.openmarketsinstitute.org/learn/monopoly-by-the-numbers
This episode of "Understood: Who Borked the Internet?" draws a straight line from those economists and their ideas to the world we live in today. It sets up the final episode, next week's "Kick 'Em in the Dongle," which charts a course for us to escape from the hellscape created by Bork, Lehman, and their toadies and trolls.
You can get "Understood: Who Broke the Internet?" in any podcast app, even the seriously enshittified ones (which, let's be real here, is most of them). Here's a direct link to the RSS:
https://www.cbc.ca/podcasting/includes/nakedemperor.xml
Drop + DOOM: The Dark Ages Slayer's Arsenal Artisan Keycap https://drop.com/buy/drop-doom-the-dark-ages-slayers-arsenal-artisan-keycap (h/t Yanko Design)
A Shapeshifting Situationship https://jimmunroe.net/comics/zeroed-out/a-shapeshifting-situationship.html
#20yrsago My PopSci piece on radio’s Broadcast Flag https://web.archive.org/web/20050520034838/https://www.popsci.com/popsci/personaltech/article/0,20967,1051257,00.html
#20yrsago My Wired News op-ed about the BBC https://web.archive.org/web/20050520005755/https://www.wired.com/news/culture/0,1284,67552,00.html
#20yrsago Russian villagers scavenge fallen Soyuz junk to survive https://www.spacedaily.com/news/debris-05d.html
#20yrsago WiFi crypto can be broken in 3 mins https://web.archive.org/web/20050519000137/https://www.tomsnetworking.com/Sections-article111.php
#15yrsago Lost etymology of “fanboy” https://technologizer.com/2010/05/17/fanboy/
#15yrsago Linda Stone on email apnea and continuous partial attention https://vimeo.com/7551900
#15yrsago Punk photography from Maximumrocknroll https://web.archive.org/web/20100520000538/https://www.wired.com/rawfile/2010/05/gallery-maximum-rocknroll/all/1
#15yrsago SWORD OF MY MOUTH: Apocalyptic graphic novel about the tyranny of angels https://memex.craphound.com/2010/05/17/sword-of-my-mouth-apocalyptic-graphic-novel-about-the-tyranny-of-angels/
#15yrsago Time to kill “Information Wants to Be Free” https://www.theguardian.com/technology/2010/may/18/information-wants-to-be-free
#15yrsago New York Times headline writer allergic to the word “liar” https://memex.naughtons.org/euphemism-nyt-style/11033/
#15yrsago Blowgun woman: “I like to hear people say ‘ouch'” https://web.archive.org/web/20100429014025/http://www.msnbc.msn.com/id/36725246/ns/us_news/
#15yrsago Financial Times chickens out, refuses to run Amnesty’s anti-Shell Oil ad https://web.archive.org/web/20100520180957/http://amnesty.org.uk/news_details.asp?NewsID=18768
#10yrsago Atlanta pays $20,000 to critic forced to post pro-cop message to Facebook https://web.archive.org/web/20150514160324/http://clatl.com/freshloaf/archives/2015/05/12/baton-bob-to-receive-20000-settlement-for-2013-arrest-in-midtown
#10yrsago Blizzard bans 100,000 Warcraft players https://hothardware.com/news/blizzard-puts-over-100000-cheaters-on-ice-in-epic-world-of-warcraft-banning-spree
#10yrsago Dolls with hearing aids, port-wine stains and canes https://web.archive.org/web/20150911020636/https://mymakie.com/campaign/toylikeme/
#5yrsago Neoliberals won't waste this crisis https://pluralistic.net/2020/05/17/cheap-truthers/#bannon-lenninism
#5yrsago Farewell to Beyond the Beyond https://pluralistic.net/2020/05/17/cheap-truthers/#cheap-truth
#5yrsago DOD lie-detector manual leaked https://pluralistic.net/2020/05/17/cheap-truthers/#buttholes
#5yrsago Instagram's slow-mo appeals court https://pluralistic.net/2020/05/17/cheap-truthers/#robot-sez-no
#5yrsago Deliveroo, without Delivero https://pluralistic.net/2020/05/18/code-is-speech/#platform-cooperatvism
#5yrsago Airgap-busting malware https://pluralistic.net/2020/05/18/code-is-speech/#ramsay
#5yrsago Universal broadband now https://pluralistic.net/2020/05/18/code-is-speech/#digital-divide
#5yrsago See through walls with free software https://pluralistic.net/2020/05/18/code-is-speech/#vaneck
#5yrsago England's storks are back https://pluralistic.net/2020/05/18/code-is-speech/#estorchestone
#5yrsago Restaurateur wreaks algorithmic vengeance upon Doordash https://pluralistic.net/2020/05/18/code-is-speech/#schadenpizza
#5yrsago Ada Palmer, being brilliant for 2.5h https://pluralistic.net/2020/05/18/code-is-speech/#terra-ignota
#1yrago Monopoly is capitalism's gerrymander https://pluralistic.net/2024/05/18/market-discipline/#too-big-to-care
#1yrago You were promised a jetpack by liars https://pluralistic.net/2024/05/17/fake-it-until-you-dont-make-it/#twenty-one-seconds
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 4
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Fireside Fedi
https://video.firesidefedi.live/w/huevh4L6r1yMYXqcQMi8gR
Ctrl-ctrl-ctrl (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16145640-ctrl-ctrl-ctrl
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
Every now and again, I reach the end of the week with more stray links that I've been able to squeeze into the newsletter, and when that happens it's time for a linkdump. This is linkdump number 31; here's 1-30:
https://pluralistic.net/tag/linkdump/
It's been five years (to the day!) since Wired killed off "Beyond the Beyond," Bruce Sterling's excellent blog, a wanton act of online vandalism that, among other things, made it much harder to figure out what was on Bruce's mind, a subject I find endlessly fascinating:
https://pluralistic.net/2020/05/17/cheap-truthers/#cheap-truth
Sterling's got a Medium that he almost never updates. I follow it through RSS, the best way to keep up with both things that update frequently and also hardly ever:
This week, he posted a long, thoughtful, and seriously intriguing review of Cafe Europa Revisited, Slavenka Drakulic's followup to her 1996 international blockbuster Cafe Europa:
https://bruces.medium.com/cafe-europa-revisited-2025-be8875c06c4c
I confess that I had never heard of Drakulic, though, as I read Sterling's review, it became clear why he dotes on the acerbic Croatian essayist, a keen observer of the material world and theorizer of political upheaval:
https://www.penguinrandomhouse.com/books/602764/cafe-europa-revisited-by-slavenka-drakulic/
Drakulic is well-known for an essay collection called "How We Survived Communism and Even Laughed," and the subtitle of this volume is "How to Survive Post-Communism," which just about says it all. Sterling characterizes it as the start of a new hot genre, "Old books directly written for old people by old people."
"The West" (whatever that is) is getting old. For more than a decade, Bruce Sterling's been predicting a future of "old people, in big cities, afraid of the sky." Original Sin, a new heavily reported book on the 2024 election makes a good case that Biden was indeed in a state of advanced senescence through much of his presidency and the entire election campaign, and had no business occupying the White House, much less running for another four years:
https://www.nytimes.com/2025/05/13/books/review/originial-sin-jake-tapper-alex-thompson.html
Biden's unwillingness to confront his age and frailty, along with Trump's obvious mental and physical decline, has many terrified American political thinkers talking about the gerontocracy that's running the country:
https://pluralistic.net/2023/07/01/designated-survivors/
Corey Robin got in some good licks on this one, in a piece called "We really are the oldest democracy in the world":
https://coreyrobin.com/2025/05/15/we-really-are-the-oldest-democracy-in-the-world/
"Oldest democracy" as in, "the democracy with the oldest leaders." The Democrats are gearing up for the midterms with such repeat offenders as Maxine Waters (86), Rosa DeLauro (82), John Garamendi (80), Doris Matsui (80) and Bonnie Watson Coleman (80). Also running: David Scott (79) who had to step down as ranking House Ag Committee member over health concerns. And: Dwight Evans (70), who missed most of last year's votes after suffering a stroke.
Meanwhile, Nancy Pelosi (85), Steny Hoyer (85), Danny Davis (83), Frederica Wilson (82), Emanuel Cleaver (80) and Alma Adams (78) won't say whether they're running in 2026:
https://www.axios.com/2025/05/15/house-democrats-age-members-reelection-biden
At 53, I can tell that I've lost a step. Sure, I have the benefits of wisdom, but man, I am so tired. Maybe the reason our Democratic leaders have sat idly by and watched as Trump dismantled democracy and installed fascism is that they're too tired to scale the fences like their South Korean counterparts did?
https://www.theverge.com/24312920/martial-law-south-korea-yoon-suk-yeol-protest-dispatch
I'm not saying everyone over 65 in Congress should retire. I'm saying that a caucus that skewed younger might be more, you know, vigorous. I'm minded of my favorite John Ciardi poem, "About Crows":
The young crow flies above, below,
and rings around the slow old crow.
What does the fast young crow not know?
WHERE TO GO.
https://spirituallythinking.blogspot.com/2011/10/about-crows-by-john-ciardi.html
Meanwhile, young people might just be getting something out of the regulatory apparatus. Thanks to a smashing court loss in the USA and regulation in the EU, Apple is now required to allow app makers to use their own payment processors, skipping the 30% App Tax Apple levies on every in-app purchase, to the tune of $100b/year.
Among other things, this means that every Fortnite skin and upgrade could suddenly get 25% cheaper without costing Epic Games a dime. The only problem is that Apple refuses to obey the regulation or the court order:
https://pluralistic.net/2025/05/01/its-not-the-crime/#its-the-coverup
This week, Apple blocked Fortnite's app from the App Store:
https://www.macrumors.com/2025/05/16/apple-blocks-fortnite-return-to-ios-app-store/
And defied EU regulators by slapping deceptive warning labels all over any EU app that accepts payments without kicking 30% up to Apple:
https://www.theverge.com/news/667484/apple-eu-ios-app-store-warning-payment-system
Apple's in a lot of trouble in the USA (Apple execs who lied to a federal judge about this stuff now face criminal sanctions), and it looks like they're spoiling for a fight with the EU. After all Trump flew to Davos and threatened to destroy any country that tried to regulate US Big Tech. The rest of the world doesn't seem scared – or at least, they're more scared of the risk of trusting US cloud technology that can be cut off to kneecap a rival economy, or used to spy on government and industry, or both. In the EU, Cryptpad – a free, open cloud based document collaboration platform – is luring away Google Docs and Office 365 users at speed:
Meanwhile, back in the USA, things are looking grim for Meta, as the FTC's case against the company moves into the end-game. The stakes are high: Meta could be forced to sell off Whatsapp and Instagram:
https://www.bigtechontrial.com/p/from-roadshow-to-expert-witness-courtroom
That is, if Mad King Trump doesn't step in. Seems like nothing is too petty for the Trump admin. How petty are they? This week, Trump's CBP seized a load of t-shirts from the subversive design studio Cola Corporation:
https://www.404media.co/cbp-seizes-shipment-of-t-shirts-featuring-swarm-of-bees-attacking-cops/
Why did CBP seize Cola's tees? Apparently, it was design that featured a cop being attacked by a swarm of bees. Cola knows good publicity when he sees it: he's printing up more of the tees and selling them in a new line he calls "the confiscated collection":
https://www.thecolacorporation.com/collections/confiscated
Get yours while supplies last!
#20yrsago Dave Matthews’s new CD DRM crashes PCs https://ma.tt/2005/05/dave-matthews-stand-up/
#15yrsago HOWTO Make a Admiral Ackbar paper-bag puppet https://web.archive.org/web/20100525031350/http://www.starwars.com/kids/do/crafts/f20100511.html
#10yrsago Self-sustaining botnet made out of hacked home routers https://arstechnica.com/information-technology/2015/05/researchers-uncover-self-sustaining-botnets-of-poorly-secured-routers/
#10yrsago Leetspeak, circa 1901 https://slate.com/human-interest/2015/05/history-of-telegraph-operators-abbreviations-used-by-telegraphers.html
#10yrsago The business model of NSA apologists https://web.archive.org/web/20150512185408/https://firstlook.org/theintercept/2015/05/12/intelligence-industry-cash-flows-media-echo-chamber-defending-nsa-surveillance/
#10yrsago Guard tells top senator that she can’t take notes on TPP https://web.archive.org/web/20150513114616/https://firstlook.org/theintercept/2015/05/12/can-read-notes-life-top-democratic-senator-blasts-obamas-tpp-secrecy/
#10yrsago Dragons Beware: Claudette’s back in the sequel to Giants Beware! https://memex.craphound.com/2015/05/12/dragons-beware-claudettes-back-in-the-sequel-to-giants-beware/
#10yrsago The Subprimes: a novel of the Piketty/Klein apocalypse https://web.archive.org/web/20190425051937/https://hbr.org/2015/05/greenfelds-the-subprimes-and-the-way-fiction-predicts-the-present"
#5yrsago Zuck wants Giphy https://pluralistic.net/2020/05/16/labor-investors/#zuckermonster
#5yrsago NYC teens fight period poverty https://pluralistic.net/2020/05/16/labor-investors/#it-leads
#5yrsago Democratize workplaces now https://pluralistic.net/2020/05/16/labor-investors/#suffering-sufferage
#5yrsago Rep Steve Cohen wants to clawback billionaires' bailout https://pluralistic.net/2020/05/16/labor-investors/#steve-cohen
#5yrsago Plane ticket refunds (without airline cooperation) https://pluralistic.net/2020/05/12/evil-maid/#friendly-skies
#5yrsago Iceland's world-beating coronavirus app didn't help much https://pluralistic.net/2020/05/12/evil-maid/#fjords
#5yrsago Adjacent, similar NYC neighborhoods with wildly different outcomes https://pluralistic.net/2020/05/12/evil-maid/#demographics-are-destiny
#5yrsago The right's theories about human behavior are bankrupt https://pluralistic.net/2020/05/12/evil-maid/#blitzed
#1yrago Utah's getting some of America's best broadband https://pluralistic.net/2024/05/16/symmetrical-10gb-for-119/#utopia
Seattle: Cascade PBS Ideas Festival, May 30
https://www.eventbrite.com/e/cascade-pbs-ideas-festival-tickets-1251699710529
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 3
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Ctrl-ctrl-ctrl (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16145640-ctrl-ctrl-ctrl
Remember when the internet was… good? What happened? (The Current)
https://www.cbc.ca/listen/live-radio/1-63-the-current/clip/16145000-remember-internet-was…-good-what-happened
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
Something amazing happened from 2020-2024: even as parts of the Biden administration were encouraging genocide and covering up the president's senescence, a small collection of little-regarded agencies were taking a wrecking ball to corporate power, approaching antitrust and consumer protection with a vigor not seen in generations.
One of the most effective agencies during those years was the Consumer Finance Protection Bureau. Under the direction of Rohit Chopra, the CFPB finally used its long-dormant powers to rein in the most egregious and abusive conduct of America's most predatory corporations, like banks, fintech, and repeat corporate offenders, with a 7-2 Supreme Court mandate to go hard:
https://pluralistic.net/2024/06/10/getting-things-done/#deliverism
As impressive as the whole CFPB agenda was, the standout for me was its attack on America's data brokerage industry. Data brokers are effectively totally unregulated, and they buy and sell every intimate fact of your life. The reason every device in your life – smart speaker, car, toothbrush, thermostate – spies on you all the time is because data brokers will buy any data from anyone and sell it to anyone, too.
Data brokerages put "surveillance capitalist" companies like Google and Meta to shame (indeed, Big Tech buys a lot of data from brokerages, as do agencies like the DEA, ICE and the FBI, who treat the brokerages as a warrant-free, off-the-books mass surveillance system). Data brokerages combine data about your movements, purchases, friends, medical problems, education, love life, and more, and bucket you into categories that marketers (or scammers) can buy access to. There are over 650,000 of these categories, including "seniors with dementia," "depressed teenagers" and "US military personnel with gambling problems":
Congress hasn't passed a new consumer privacy law since 1988's Video Privacy Protection Act. The last technological privacy issue your legislature considered important enough to address was the scourge of video-store clerks telling newspapers which VHS cassettes you took home:
https://pluralistic.net/2023/12/06/privacy-first/#but-not-just-privacy
Congress's massive failure created equally massive risks for the rest of us. From phishing and ransomware attacks to identity theft to stalking and SWATting, America's privacy nihilism enabled mass-scale predation upon all of us, rich and poor, old and young, rural and urban, men and women, racialized and white.
That's the void that the CFPB stepped into last summer, when they passed a new rule that would effectively shut down the entire data brokerage industry:
Yesterday, Trump's CFPB boss, Russell Vought, killed that rule, stating that it was "no longer necessary or appropriate":
https://www.wired.com/story/cfpb-quietly-kills-rule-to-shield-americans-from-data-brokers/
Here's the thing: Trumpism relies on the fusion of two groups of people: a tiny number of oligarchs, and millions of everyday people who are constantly victimized by those oligarchs. To get this latter group of Christmas-voting turkeys to stay in the coalition, Trump needs to deliver something that keeps them happy. Mostly, Trump delivers negative things to keep them happy – the spectacle of public cruelty to immigrants, women, trans people, academics, etc. There is a certain libidinal satisfaction that comes from watching your enemies suffer – but you can't eat schadenfreude. You can't make rent or put braces on your kids' teeth or pay your medical bills with the sadistic happiness you feel when you hear the sobs of people you've been taught to despise.
For Trump to keep the turkeys voting for Christmas, he needs to do something for them. He can't just do things to scapegoats. But America's eminently guillotineable oligarchs have found so many ways to turn working peoples' torment into riches, and they are so greedy and unwilling to give up any of those grifts, that Trump can't manage to deliver anything positive to his base. Last week, his FTC killed the "click to cancel" rule that required companies that tricked you into buying subscriptions to make it easy for you to cancel them:
https://pluralistic.net/2025/05/12/greased-slide/#greased-pole
There isn't a single person in the Trump base who isn't harmed by data brokers. Every red-hat-wearing MAGA footsoldier has been swindled with a recurring-payment scam by clicking a deceptive link. The material conditions of the lives of Trump's base – already in severe jeopardy thanks to the massive inflation the tariffs will cause, and the plummeting wages that the ensuing mass business-closures will bring about – cannot be improved in any way.
I don't think anyone knows for sure how much support Trump can win solely by torturing the people his supporters hate, even as those supporters' lives get worse and worse. The one thing I'm sure of, though, is that it's less support than Trump would get if he could do something – anything – to make their lives even a little better.
Trump owes his success to coalition-building. The Trumpist agenda – ripoffs and racism and rape – has been around forever, in festering pockets like the John Birch Society, but those feverish monsters were encysted by the body politic and kept away from power. When a group of people who've been unsuccessfully trying to do something for a long time suddenly attain success, the most likely explanation is that they have found coalition partners to join them in their push.
Every coalition is brittle, because coalition partners want different things (if you want the same thing, you're just a group – "coalitions" are, definitionally, made up of people who want different things). They have shared goals, sure, but some of the things that some of the coalition partners want are things that the other partners totally reject. When one partner wins, the other partners lose. Trump's been good at holding together his coalition, but he's running up against some hard limits.
Here's what Naomi Klein told Cerise Castle from Capital & Main/The American Prospect:
The most serious vulnerability that Trump has is that a large part of his base really hates Silicon Valley and is not interested in being replaced by machines. So it’s a monumental bait-and-switch that Trump has done with this immediate alignment with the billionaire class in Silicon Valley, and if the left can’t exploit that, then we’re doing something wrong.
https://prospect.org/culture/2025-05-13-moment-of-unparalleled-peril-interview-naomi-klein/
Killing the CFPB's data broker rule is a pure transfer from the Trump base to Silicon Valley oligarchs, whose hunger for our private data know no bounds.
(Image: Cryteria, CC BY 3.0, modified)
Will Congress Legalize Mark Zuckerberg As Your Therapist? https://www.thebignewsletter.com/p/will-congress-legalize-mark-zuckerberg
Is it Euro-poor, or Ameri-poor? https://walkingtheworld.substack.com/p/is-it-euro-poor-or-ameri-poor (h/t Mitch Wagner)
#20yrsago Legal fund for French blogger being sued by for criticizing his town https://web.archive.org/web/20050518031636/https://www.loiclemeur.com/english/2005/05/a_french_blogge.html
#10yrsago Stephen Harper extended music copyright to please US record industry lobbyist https://www.michaelgeist.ca/2015/05/harper-letter-to-music-canada-on-budget-day-confirms-copyright-extension-the-product-of-industry-lobbying/
#10yrsago Millennial justice https://web.archive.org/web/20150514175100/https://www.thehairpin.com/2015/05/a-millennial-revenge-fantasy/
#5yrsago Tear gas ice-cream https://pluralistic.net/2020/05/15/out-here-everything-hurts/#612strike
#5yrsago Whistleblower warns of massive mortgage fraud https://pluralistic.net/2020/05/15/out-here-everything-hurts/#cmbs
#5yrsago A people's vaccine https://pluralistic.net/2020/05/15/out-here-everything-hurts/#accesstomedicine
#5yrsago Google's GDPR reckoning https://pluralistic.net/2020/05/15/out-here-everything-hurts/#noyb
#5yrsago Understanding Qanon https://pluralistic.net/2020/05/15/out-here-everything-hurts/#q
#1yrago Even if you think AI search could be good, it won't be good https://pluralistic.net/2024/05/15/they-trust-me-dumb-fucks/#ai-search
Pittsburgh: PyCon, May 16
https://us.pycon.org/2025/schedule/
Virtual: Writing to Resist (California Writers Club Berkeley), May 18:
https://cwc-berkeley.org/writing-to-resist-5-18-25/
Seattle: Cascade PBS Ideas Festival, May 30
https://www.eventbrite.com/e/cascade-pbs-ideas-festival-tickets-1251699710529
PDX: Teardown 2025, Jun 20-22
https://www.crowdsupply.com/teardown/portland-2025
PDX: Picks and Shovels with bunnie Huang at Barnes and Noble, Jun 20
https://stores.barnesandnoble.com/event/9780062183697-0
Tualatin Public Library, Jun 22:
https://www.tualatinoregon.gov/library/author-talk-cory-doctorow
London: How To Academy with Riley Quinn, Jul 1
https://howtoacademy.com/events/cory-doctorow-the-fight-against-the-big-tech-oligarchy/
Manchester: Picks and Shovels at Blackwell's Bookshop, Jul 2
https://www.eventbrite.co.uk/e/an-evening-with-cory-doctorow-tickets-1308451968059
Manchester: Co-operatives UK Co-op Congress keynote, Jul 3
https://www.uk.coop/events-and-training/events-calendar/co-op-congress-2025-book-your-place
New Orleans: DeepSouthCon63, Oct 10-12, 2025
http://www.contraflowscifi.org/
Ctrl-ctrl-ctrl (Understood)
https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16145640-ctrl-ctrl-ctrl
Remember when the internet was… good? What happened? (The Current)
https://www.cbc.ca/listen/live-radio/1-63-the-current/clip/16145000-remember-internet-was…-good-what-happened
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). Signed, personalized copies at Dark Delicacies (https://www.darkdel.com/store/p3007/Pre-Order_Signed_Copies%3A_The_Lost_Cause_HB.html#/)
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. Signed copies at Dark Delicacies (US): and Forbidden Planet (UK): https://forbiddenplanet.com/385004-red-team-blues-signed-edition-hardcover/.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59?sk=f6cd10e54e20a07d4c6d0f3ac011af6b) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p2682/Corey_Doctorow%3A_Poesy_the_Monster_Slayer_HB.html#/.
Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026
The Memex Method, Farrar, Straus, Giroux, 2026
Today's top sources:
Currently writing:
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
https://creativecommons.org/licenses/by/4.0/
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
Blog (no ads, tracking, or data-collection):
Newsletter (no ads, tracking, or data-collection):
https://pluralistic.net/plura-list
Mastodon (no ads, tracking, or data-collection):
Medium (no ads, paywalled):
Twitter (mass-scale, unrestricted, third-party surveillance and advertising):
Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):
https://mostlysignssomeportents.tumblr.com/tagged/pluralistic
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
ISSN: 3066-764X
I've seen a lot of complaints about how MCP isn't ready for the enterprise.
I agree, although maybe not for the reasons you think. But don't worry, this isn't just a rant! I believe we can fix it!
The good news is the recent updates to the MCP authorization spec that separate out the role of the authorization server from the MCP server have now put the building blocks in place to make this a lot easier.
But let's back up and talk about what enterprise buyers expect when they are evaluating AI tools to bring into their companies.
At a minimum, an enterprise admin expects to be able to put an application under their single sign-on system. This enables the company to manage which users are allowed to use which applications, and prevents their users from needing to have their own passwords at the applications. The goal is to get every application managed under their single sign-on (SSO) system. Many large companies have more than 200 applications, so having them all managed through their SSO solution is a lot better than employees having to manage 200 passwords for each application!
There's a lot more than SSO too, like lifecycle management, entitlements, and logout. We're tackling these in the IPSIE working group in the OpenID Foundation. But for the purposes of this discussion, let's stick to the basics of SSO.
So what does this have to do with MCP?
An AI agent using MCP is just another application enterprises expect to be able to integrate into their single-sign-on (SSO) system. Let's take the example of Claude. When rolled out at a company, ideally every employee would log in to their company Claude account using the company identity provider (IdP). This lets the enterprise admin decide how many Claude licenses to purchase and who should be able to use it.
The next thing that should happen after a user logs in to Claude via SSO is they need to connect Claude to their other enterprise apps. This includes the built-in integrations in Claude like Google Calendar and Google Drive, as well as any MCP servers exposed by other apps in use within the enterprise. That could cover other SaaS apps like Zoom, Atlassian, and Slack, as well as home-grown internal apps.
Today, this process involves a somewhat cumbersome series of steps each individual employee must take. Here's an example of what the user needs to do to connect their AI agent to external apps:
First, the user logs in to Claude using SSO. This involves a redirect from Claude to the enterprise IdP where they authenticate with one or more factors, and then are redirected back.
Next, they need to connect the external app from within Claude. Claude provides a button to initiate the connection. This takes the user to that app (in this example, Google), which redirects them to the IdP to authenticate again, eventually getting redirected back to the app where an OAuth consent prompt is displayed asking the user to approve access, and finally the user is redirected back to Claude and the connection is established.
The user has to repeat these steps for every MCP server that they want to connect to Claude. There are two main problems with this:
Both of these are significant problems. If you have even just 10 MCP servers rolled out in the enterprise, you're asking users to click through 10 SSO and OAuth prompts to establish the connections, and it will only get worse as MCP is more widely adopted within apps. But also, should we really be asking the user if it's okay for Claude to access their data in Google Drive? In a company context, that's not actually the user's decision. That decision should be made by the enterprise IT admin.
In "An Open Letter to Third-party Suppliers", Patrick Opet, Chief Information Security Officer of JPMorgan Chase writes:
"Modern integration patterns, however, dismantle these essential boundaries, relying heavily on modern identity protocols (e.g., OAuth) to create direct, often unchecked interactions between third-party services and firms' sensitive internal resources."
Right now, these app-to-app connections are happening behind the back of the IdP. What we need is a way to move the connections between the applications into the IdP where they can be managed by the enterprise admin.
Let's see how this works if we leverage a new (in-progress) OAuth extension called "Identity and Authorization Chaining Across Domains", which I'll refer to as "Cross-App Access" for short, enabling the enterprise IdP to sit in the middle of the OAuth exchange between the two apps.
In this example, we'll use Claude as the application that is trying to connect to Slack's (hypothetical) MCP server. We'll start with a high-level overview of the flow, and later go over the detailed protocol.
First, the user logs in to Claude through the IdP as normal. This results in Claude getting either an ID token or SAML assertion from the IdP, which tells Claude who the user is. (This works the same for SAML assertions or ID tokens, so I'll use ID tokens in the example from here out.) This is no different than what the user would do today when signing in to Claude.
Then, instead of prompting the user to connect Slack, Claude takes the ID token back to the IdP in a request that says "Claude is requesting access to this user's Slack account."
The IdP validates the ID token, sees it was issued to Claude, and verifies that the admin has allowed Claude to access Slack on behalf of the given user. Assuming everything checks out, the IdP issues a new token back to Claude.
Claude takes the intermediate token from the IdP to Slack saying "hi, I would like an access token for the Slack MCP server. The IdP gave me this token with the details of the user to issue the access token for." Slack validates the token the same way it would have validated an ID token. (Remember, Slack is already configured for SSO to the IdP for this customer as well, so it already has a way to validate these tokens.) Slack is able to issue an access token giving Claude access to this user's resources in its MCP server.
This solves the two big problems:
The other nice side effect of this is since there is no user interaction required, the first time a new user logs in to Claude, all their enterprise apps will be automatically connected without them having to click any buttons!
Now let's look at what this looks like in the actual protocol. This is based on the adopted in-progress OAuth specification "Identity and Authorization Chaining Across Domains". This spec is actually a combination of two RFCs: Token Exchange (RFC 8693), and JWT Profile for Authorization Grants (RFC 7523). Both RFCs as well as the "Identity and Authorization Chaining Across Domains" spec are very flexible. While this means it is possible to apply this to many different use cases, it does mean we need to be a bit more specific in how to use it for this use case. For that purpose, I've written a profile of the Identity Chaining draft called "Identity Assertion Authorization Grant" to fill in the missing pieces for the specific use case detailed here.
Let's go through it step by step. For this example we'll use the following entities:
First, Claude gets the user to sign in using a standard OpenID Connect (or SAML) flow in order to obtain an ID token. There isn't anything unique to this spec regarding this first stage, so I will skip the details of the OpenID Connect flow and we'll start with the ID token as the input to the next step.
Claude, the requesting application, then makes a Token Exchange request (RFC 8693) to the IdP's token endpoint with the following parameters:
requested_token_type: The value urn:ietf:params:oauth:token-type:id-jag indicates that an ID Assertion JWT is being requested.resource: The Issuer URL of the Resource Application's authorization server.subject_token: The identity assertion (e.g. the OpenID Connect ID Token or SAML assertion) for the target end-user.subject_token_type: Either urn:ietf:params:oauth:token-type:id_token or urn:ietf:params:oauth:token-type:saml2 as defined by RFC 8693.This request will also include the client credentials that Claude would use in a traditional OAuth token request, which could be a client secret or a JWT Bearer Assertion.
POST /oauth2/token HTTP/1.1
Host: acme.okta.com
Content-Type: application/x-www-form-urlencoded
grant_type=urn:ietf:params:oauth:grant-type:token-exchange
&requested_token_type=urn:ietf:params:oauth:token-type:id-jag
&resource=https://mcp.slack.com/
&subject_token=eyJraWQiOiJzMTZ0cVNtODhwREo4VGZCXzdrSEtQ...
&subject_token_type=urn:ietf:params:oauth:token-type:id_token
&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
&client_assertion=eyJhbGciOiJSUzI1NiIsImtpZCI6IjIyIn0...
At this point, the IdP evaluates the request and decides whether to issue the requested "ID Assertion JWT". The request will be evaluated based on the validity of the arguments, as well as the configured policy by the customer.
For example, the IdP validates that the ID token in this request was issued to the same client that matches the provided client authentication. It evaluates that the user still exists and is active, and that the user is assigned the Resource Application. Other policies can be evaluated at the discretion of the IdP, just like it can during a single sign-on flow.
If the IdP agrees that the requesting app should be authorized to access the given user's data in the resource app's MCP server, it will respond with a Token Exchange response to issue the token:
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store
{
"issued_token_type": "urn:ietf:params:oauth:token-type:id-jag",
"access_token": "eyJhbGciOiJIUzI1NiIsI...",
"token_type": "N_A",
"expires_in": 300
}
The claims in the issued JWT are defined in "Identity Assertion Authorization Grant". The JWT is signed using the same key that the IdP signs ID tokens with. This is a critical aspect that makes this work, since again we assumed that both apps would already be configured for SSO to the IdP so would already be aware of the signing key for that purpose.
At this point, Claude is ready to request a token for the Resource App's MCP server
The JWT received in the previous request can now be used as a "JWT Authorization Grant" as described by RFC 7523. To do this, Claude makes a request to the MCP authorization server's token endpoint with the following parameters:
grant_type: urn:ietf:params:oauth:grant-type:jwt-bearerassertion: The Identity Assertion Authorization Grant JWT obtained in the previous token exchange stepFor example:
POST /oauth2/token HTTP/1.1
Host: auth.slack.com
Authorization: Basic yZS1yYW5kb20tc2VjcmV0v3JOkF0XG5Qx2
grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
assertion=eyJhbGciOiJIUzI1NiIsI...
Slack's authorization server can now evaluate this request to determine whether to issue an access token. The authorization server can validate the JWT by checking the issuer (iss) in the JWT to determine which enterprise IdP the token is from, and then check the signature using the public key discovered at that server. There are other claims to be validated as well, described in Section 6.1 of the Identity Assertion Authorization Grant.
Assuming all the validations pass, Slack is ready to issue an access token to Claude in the token response:
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store
{
"token_type": "Bearer",
"access_token": "2YotnFZFEjr1zCsicMWpAA",
"expires_in": 86400,
"refresh_token": "tGzv3JOkF0XG5Qx2TlKWIA",
}
This token response is the same format that Slack's authorization server would be responding to a traditional OAuth flow. That's another key aspect of this design that makes it scalable. We don't need the resource app to use any particular access token format, since only that server is responsible for validating those tokens.
Now that Claude has the access token, it can make a request to the (hypothetical) Slack MCP server using the bearer token the same way it would have if it got the token using the traditional redirect-based OAuth flow.
Here's the flow again, this time as a sequence diagram.
At this point, the user is logged in to the MCP client. Everything up until this point has been a standard OpenID Connect flow.
For a more detailed step by step of the flow, see Appendix A.3 of the Identity Assertion Authorization Grant.
If this is something you're interested in, we'd love your help! The in-progress spec is publicly available, and we're looking for people interested in helping prototype it. If you're building an MCP server and you want to make it enterprise-ready, I'd be happy to help you build this!
You can find me at a few related events coming up:
And of course you can always find me on LinkedIn or email me at aaron.parecki@okta.com.
Let's not overthink auth in MCP.
Yes, the MCP server is going to need its own auth server. But it's not as bad as it sounds. Let me explain.
First let's get a few pieces of terminology straight.
The confusion that's happening in the discussions I've seen so far is because the spec and diagrams show that the MCP server itself is handing authorization. That's not necessary.
In OAuth, we talk about the "authorization server" and "resource server" as distinct roles. I like to think of the authorization server as the "token factory", that's the thing that makes the access tokens. The resource server (usually an API) needs to be able to validate the tokens created by the authorization server.
It's possible to build a single server that is both a resource server and authorization server, and in fact many OAuth systems are built that way, especially large consumer services.
But nothing about the spec requires that the two roles are combined, it's also possible to run these as two totally unrelated services.
This flexibility that's been baked into OAuth for over a decade is what has led to the rapid adoption, as well the proliferation of open source and commercial products that provide an OAuth authorization server as a service.
So how does this relate to MCP?
I can annotate the flow from the Model Context Protocol spec to show the parts where the client talks to the MCP Resource Server separately from where the client talks to the MCP Authorization Server.
Here is the updated sequence diagram showing communication with each role separately.
Why is it important to call out this change?
I've seen a few conversations in various places about how requiring the MCP Server to be both an authorization server and resource server is too much of a burden. But actually, very little needs to change about the spec to enable this separation of concerns that OAuth already provides.
I've also seen various suggestions of other ways to separate the authorization server from the MCP server, like delegating to an enterprise IdP and having the MCP server validate access tokens issued by the IdP. These other options also conflate the OAuth roles in an awkward way and would result in some undesirable properties or relationships between the various parties involved.
So what needs to change in the MCP spec to enable this?
The main thing currently forcing the MCP Server to be both the authorization server and resource server is how the client does discovery.
One design goal of MCP is to enable a client to bootstrap everything it needs based on only the server URL provided. I think this is a great design goal, and luckily is something that can be achieved even when separating the roles in the way I've described.
The MCP spec currently says that clients are expected to fetch the OAuth Server Metadata (RFC8414) file from the MCP Server base URL, resulting in a URL such as:
https://example.com/.well-known/oauth-authorization-server
This ends up meaning the MCP Resource Server must also be an Authorization Server, which leads to the complications the community has encountered so far. The good news is there is an OAuth spec we can apply here instead: Protected Resource Metadata.
The Protected Resource Metadata spec is used by a Resource Server to advertise metadata about itself, including which Authorization Server can be used with it. This spec is both new and old. It was started in 2016, but was never adopted by the OAuth working group until 2023, after I had presented at an IETF meeting about the need for clients to be able to bootstrap OAuth flows given an OAuth resource server. The spec is now awaiting publication as an RFC, and should get its RFC number in a couple months.
Applying this to the MCP server would result in a sequence like the following:
/.well-known/oauth-protected-resource to the MCP Server base URL.authorization_servers property in the JSON response, and builds the Authorization Server Metadata URL by appending /.well-known/oauth-authorization-server
Note: The Protected Resource Metadata spec also supports the Resource Server returning WWW-Authenticate with a link to the resource metadata URL if you want to avoid the requirement that MCP Servers host their metadata URLs at the .well-known endpoint, it just requires an extra HTTP request to support this.
Two things to keep in mind about how the MCP Server validates access tokens with this new separation of concerns.
If you do build the MCP Authorization Server and Resource Server as part of the same system, you don't need to do anything special to validate the access tokens the Authorization Server issues. You probably already have some sort of infrastructure in place for your normal API to validate tokens issued by your Authorization Server, so nothing changes there.
If you are using an external Authorization Server, whether that's an open source product or a commercial hosted service, that product will have its own docs for how you can validate the tokens it creates. There's a good chance it already supports the standardized JWT Access Tokens described in RFC 9068, in which case you can use off-the-shelf JWT validation middleware for common frameworks.
In either case, the critical design goal here is that the MCP Authorization Server issues access tokens that only ever need to be validated by the MCP Resource Server. This is in line with the security recommendations in Section 2.3 of RFC 9700, in particular that "access tokens SHOULD be audience-restricted to a specific resource server". In other words, it would be a bad idea for the MCP Client to be issued an access token that works with both the MCP Resource Server and the service's REST API.
Another argument I've seen is that MCP Server developers shouldn't have to build any OAuth infrastructure at all, instead they should be able to delegate all the OAuth bits to an external service.
In principle, I agree. Getting API access and authorization right is tricky, that's why there are entire companies dedicated to solving the problem.
The architecture laid out above enables this exact separation of concerns. The difference between this architecture and some of the other proposals I've seen is that this cleanly separates the security boundaries so that there are minimal dependencies among the parties involved.
But, one thing I haven't seen mentioned in the discussions is that there actually is no requirement than an OAuth Authorization Server provide any UI itself.
While it is desirable from a security perspective that the MCP Resource Server has a corresponding Authorization Server that issues access tokens for it, that Authorization Server doesn't actually need to have any UI or even any concept of user login or accounts. You can actually build an Authorization Server that delegates all user account management to an external service. You can see an example of this in PayPal's MCP server they recently launched.
PayPal's traditional API already supports OAuth, the authorization and token endpoints are:
https://www.paypal.com/signin/authorizehttps://api-m.paypal.com/v1/oauth2/tokenWhen PayPal built their MCP server, they launched it at https://mcp.paypal.com. If you fetch the metadata for the MCP Server, you'll find the two OAuth endpoints for the MCP Authorization Server:
https://mcp.paypal.com/authorizehttps://mcp.paypal.com/tokenWhen the MCP Client redirects the user to the authorization endpoint, the MCP server itself doesn't provide any UI. Instead, it immediately redirects the user to the real PayPal authorization endpoint which then prompts the user to log in and authorize the client.
This points to yet another benefit of architecting the MCP Authorization Server and Resource Server this way. It enables implementers to delegate the actual user management to their existing OAuth server with no changes needed to the MCP Client. The MCP Client isn't even aware that this extra redirect step was inserted in the middle. As far as the MCP Client is concerned, it has been talking to only the MCP Authorization Server. It just so happens that the MCP Authorization Server has sent the user elsewhere to actually log in.
There's one more point I want to make about why having a dedicated MCP Authorization Server is helpful architecturally.
The MCP spec strongly recommends that MCP Servers (authorization servers) support Dynamic Client Registration. If MCP is successful, there will be a large number of MCP Clients talking to a large number of MCP Servers, and the user is the one deciding which combinations of clients and servers to use. This means it is not scalable to require that every MCP Client developer register their client with every MCP Server.
This is similar to the idea of using an email client with the user's chosen email server. Obviously Mozilla can't register Thunderbird with every email server out there. Instead, there needs to be a way to dynamically establish a client's identity with the OAuth server at runtime. Dynamic Client Registration is one option for how to do that.
The problem is most commercial APIs are not going to enable Dynamic Client Registration on their production servers. For example, in order to get client credentials to use the Google APIs, you need to register as a developer and then register an OAuth client after logging in. Dynamic Client Registration would allow a client to register itself without the link to the developer's account. That would mean there is no paper trail for who the client was developed by. The Dynamic Client Registration endpoint can't require authentication by definition, so is a public endpoint that can create clients, which as you can imagine opens up some potential security issues.
I do, however, think it would be reasonable to expect production services to enable Dynamic Client Registration only on the MCP's Authorization Server. This way the dynamically-registered clients wouldn't be able to use the regular REST API, but would only be able to interact with the MCP API.
Mastodon and BlueSky also have a similar problem of needing clients to show up at arbitrary authorization servers without prior coordination between the client developer and authorization server operator. I call this the "OAuth for the Open Web" problem. Mastodon used Dynamic Client Registration as their solution, and has since documented some of the issues that this creates, linked here and here.
BlueSky decided to take a different approach and instead uses an https URL as a client identifier, bypassing the need for a client registration step entirely. This has the added bonus of having at least some level of confidence of the client identity because the client identity is hosted at a domain. It would be a perfectly viable approach to use this method for MCP as well. There is a discussion on that within MCP here. This is an ongoing topic within the OAuth working group, I have a couple of drafts in progress to formalize this pattern, Client ID Metadata Document and Client ID Scheme.
Lastly, I want to touch on the idea of enabling users to log in to MCP Servers with their enterprise IdP.
When an enterprise company purchases software, they expect to be able to tie it in to their single-sign-on solution. For example, when I log in to work Slack, I enter my work email and Slack redirects me to my work IdP where I log in. This way employees don't need to have passwords with every app they use in the enterprise, they can log in to everything with the same enterprise account, and all the apps can be protected with multi-factor authentication through the IdP. This also gives the company control over which users can access which apps, as well as a way to revoke a user's access at any time.
So how does this relate to MCP?
Well, plenty of people are already trying to figure out how to let their employees safely use AI tools within the enterprise. So we need a way to let employees use their enterprise IdP to log in and authorize MCP Clients to access MCP Servers.
If you're building an MCP Server in front of an existing application that already supports enterprise Single Sign-On, then you don't need to do anything differently in the MCP Client or Server and you already have support for this. When the MCP Client redirects to the MCP Authorization Server, the MCP Authorization Server redirects to the main Authorization Server, which would then prompt the user for their company email/domain and redirect to the enterprise IdP to log in.
This brings me to yet another thing I've been seeing conflated in the discussions: user login and user authorization.
OAuth is an authorization delegation protocol. OAuth doesn't actually say anything about how users authenticate at the OAuth server, it only talks about how the user can authorize access to an application. This is actually a really great thing, because it means we can get super creative with how users authenticate.
Remember the yellow box "User logs in and authorizes" from the original sequence diagram? These are actually two totally distinct steps. The OAuth authorization server is responsible for getting the user to log in somehow, but there's no requirement that how the user logs in is with a username/password. This is where we can insert a single-sign-on flow to an enterprise IdP, or really anything you can imagine.
So think of this as two separate boxes: "user logs in", and "user authorizes". Then, we can replace the "user logs in" box with an entirely new OpenID Connect flow out to the enterprise IdP to log the user in, and after they are logged in they can authorize the client.
I'll spare you the complete expanded sequence diagram, since it looks a lot more complicated than it actually is. But I again want to stress that this is nothing new, this is already how things are commonly done today.
This all just becomes cleaner to understand when you separate the MCP Authorization Server from the MCP Resource Server.
We can push all the complexity of user login, token minting, and more onto the MCP Authorization Server, keeping the MCP Resource Server free to do the much simpler task of validating access tokens and serving resources.
There are two things I want to call out about how enterprise IdP integration could be improved. Both of these are entire topics on their own, so I will only touch on the problems and link out to other places where work is happening to solve them.
There are two points of friction with the current state of enterprise login for SaaS apps.
When a user logs in to a SaaS app, they need to tell the app how to find their enterprise IdP. This is commonly done by either asking the user to enter their work email, or asking the user to enter their tenant URL at the service.
Neither of these is really a great user experience. It would be a lot better if the browser already knew which enterprise IdP the user should be sent to. This is one of my goals with the work happening in FedCM. With this new browser API, the browser can mediate the login, telling the SaaS app which enterprise IdP to use automatically only needing the user to click their account icon rather than type anything in.
Another point of friction in the enterprise happens when a user starts connecting multiple applications to each other within the company. For example, if you drop in a Google Docs link into Slack, Slack will prompt you to connect your Google account to preview the link. Multiply this by N number of applications that can preview links, and M number of applications you might drop links to, and you end up sending the user through a huge number of OAuth consent flows.
The problem is only made worse with the explosion of AI tools. Every AI tool will need access to data in every other application in the enterprise. That is a lot of OAuth consent flows for the user to manage. Plus, the user shouldn't really be the one granting consent for Slack to access the company Google Docs account anyway. That consent should ideally be managed by the enterprise IT admin.
What we actually need is a way to enable the IT admin to grant consent for apps to talk to each other company-wide, removing the need for users to be sent through an OAuth flow at all.
This is the basis of another OAuth spec I've been working on, the Identity Assertion Authorization Grant.
The same problem applies to MCP Servers, and with the separation of concerns laid out above, it becomes straightforward to add this extension to move the consent to the enterprise and streamline the user experience.
Get in touch!
If these sound like interesting problems, please get in touch! You can find me on LinkedIn or reach me via email at aaron@parecki.com.
Hello! Today I want to talk about ANSI escape codes.
For a long time I was vaguely aware of ANSI escape codes (“that’s how you make text red in the terminal and stuff”) but I had no real understanding of where they were supposed to be defined or whether or not there were standards for them. I just had a kind of vague “there be dragons” feeling around them. While learning about the terminal this year, I’ve learned that:
So I wanted to put together a list for myself of some standards that exist around escape codes, because I want to know if they have to feel unreliable and frustrating, or if there’s a future where we could all rely on them with more confidence.
Have you ever pressed the left arrow key in your terminal and seen ^[[D?
That’s an escape code! It’s called an “escape code” because the first character
is the “escape” character, which is usually written as ESC, \x1b, \E,
\033, or ^[.
Escape codes are how your terminal emulator communicates various kinds of information (colours, mouse movement, etc) with programs running in the terminal. There are two kind of escape codes:
ESC[D, “Ctrl+left arrow” might be ESC[1;5D, and clicking the mouse might
be something like ESC[M :3.Now let’s talk about standards!
The first standard I found relating to escape codes was ECMA-48, which was originally published in 1976.
ECMA-48 does two things:
ESC[ + something and “OSC” codes, which are ESC] + something)ESC[D, or “turn text red” is ESC[31m. In the spec, the “cursor left”
one is called CURSOR LEFT and the one for changing colours is called
SELECT GRAPHIC RENDITION.The formats are extensible, so there’s room for others to define more escape codes in the future. Lots of escape codes that are popular today aren’t defined in ECMA-48: for example it’s pretty common for terminal applications (like vim, htop, or tmux) to support using the mouse, but ECMA-48 doesn’t define escape codes for the mouse.
There are a bunch of escape codes that aren’t defined in ECMA-48, for example:
I believe (correct me if I’m wrong!) that these and some others came from xterm, are documented in XTerm Control Sequences, and have been widely implemented by other terminal emulators.
This list of “what xterm supports” is not a standard exactly, but xterm is extremely influential and so it seems like an important document.
In the 80s (and to some extent today, but my understanding is that it was MUCH more dramatic in the 80s) there was a huge amount of variation in what escape codes terminals actually supported.
To deal with this, there’s a database of escape codes for various terminals called “terminfo”.
It looks like the standard for terminfo is called X/Open Curses, though you need to create an account to view that standard for some reason. It defines the database format as well as a C library interface (“curses”) for accessing the database.
For example you can run this bash snippet to see every possible escape code for “clear screen” for all of the different terminals your system knows about:
for term in $(toe -a | awk '{print $1}')
do
echo $term
infocmp -1 -T "$term" 2>/dev/null | grep 'clear=' | sed 's/clear=//g;s/,//g'
done
On my system (and probably every system I’ve ever used?), the terminfo database is managed by ncurses.
I think it’s interesting that there are two main approaches that applications take to handling ANSI escape codes:
TERM environment variable. Fish does this, for example.Some examples of programs/libraries that take approach #2 (“don’t use terminfo”) include:
I got curious about why folks might be moving away from terminfo and I found this very interesting and extremely detailed rant about terminfo from one of the fish maintainers, which argues that:
[the terminfo authors] have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is.
I’m not going to do it justice so I’m not going to summarize it, I think it’s worth reading.
I was just talking about the idea that you can use a “common set” of escape codes that will work for most people. But what is that set? Is there any agreement?
I really do not know the answer to this at all, but from doing some reading it seems like it’s some combination of:
and maybe ultimately “identify the terminal emulators you think your users are going to use most frequently and test in those”, the same way web developers do when deciding which CSS features are okay to use
I don’t think there are any resources like Can I use…? or Baseline for the terminal though. (in theory terminfo is supposed to be the “caniuse” for the terminal but it seems like it often takes 10+ years to add new terminal features when people invent them which makes it very limited)
I also asked on Mastodon why people found terminfo valuable in 2025 and got a few reasons that made sense to me:
TERM environment variable to
control how programs behave (for example with TERM=dumb), and there’s
no standard for how that should work in a post-terminfo worldThe way that ncurses uses the TERM environment variable to decide which
escape codes to use reminds me of how webservers used to sometimes use the
browser user agent to decide which version of a website to serve.
It also seems like it’s had some of the same results – the way iTerm2 reports itself as being “xterm-256color” feels similar to how Safari’s user agent is “Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15”. In both cases the terminal emulator / browser ends up changing its user agent to get around user agent detection that isn’t working well.
On the web we ended up deciding that user agent detection was not a good practice and to instead focus on standardization so we can serve the same HTML/CSS to all browsers. I don’t know if the same approach is the future in the terminal though – I think the terminal landscape today is much more fragmented than the web ever was as well as being much less well funded.
A few more documents and standards related to escape codes, in no particular order:
I sometimes see people saying that the unix terminal is “outdated”, and since I love the terminal so much I’m always curious about what incremental changes might make it feel less “outdated”.
Maybe if we had a clearer standards landscape (like we do on the web!) it would be easier for terminal emulator developers to build new features and for authors of terminal applications to more confidently adopt those features so that we can all benefit from them and have a richer experience in the terminal.
Obviously standardizing ANSI escape codes is not easy (ECMA-48 was first published almost 50 years ago and we’re still not there!). I don’t even know what all of the challenges are. But the situation with HTML/CSS/JS used to be extremely bad too and now it’s MUCH better, so maybe there’s hope.
I was talking to a friend about how to add a directory to your PATH today. It’s
something that feels “obvious” to me since I’ve been using the terminal for a
long time, but when I searched for instructions for how to do it, I actually
couldn’t find something that explained all of the steps – a lot of them just
said “add this to ~/.bashrc”, but what if you’re not using bash? What if your
bash config is actually in a different file? And how are you supposed to figure
out which directory to add anyway?
So I wanted to try to write down some more complete directions and mention some of the gotchas I’ve run into over the years.
Here’s a table of contents:
If you’re not sure what shell you’re using, here’s a way to find out. Run this:
ps -p $$ -o pid,comm=
97295 bash97295 zsh$$ isn’t valid syntax in fish, but in any case the error
message tells you that you’re using fish, which you probably already knew)Also bash is the default on Linux and zsh is the default on Mac OS (as of 2024). I’ll only cover bash, zsh, and fish in these directions.
~/.zshrc~/.bashrc, but it’s complicated, see the note in the next section~/.config/fish/config.fish (you can run echo $__fish_config_dir if you want to be 100% sure)Bash has three possible config files: ~/.bashrc, ~/.bash_profile, and ~/.profile.
If you’re not sure which one your system is set up to use, I’d recommend testing this way:
echo hi there to your ~/.bashrc~/.bashrc is being used! Hooray!~/.bash_profile~/.profile if the first two options don’t work.(there are a lot of elaborate flow charts out there that explain how bash decides which config file to use but IMO it’s not worth it to internalize them and just testing is the fastest way to be sure)
Let’s say that you’re trying to install and run a program called http-server
and it doesn’t work, like this:
$ npm install -g http-server
$ http-server
bash: http-server: command not found
How do you find what directory http-server is in? Honestly in general this is
not that easy – often the answer is something like “it depends on how npm is
configured”. A few ideas:
cargo, npm, homebrew, etc),
when you first set it up it’ll print out some directions about how to update
your PATH. So if you’re paying attention you can get the directions then.PATH for younpm config get prefix (then append /bin/)go env GOPATH (then append /bin/)asdf info | grep ASDF_DIR (then append /bin/ and /shims/)Once you’ve found a directory you think might be the right one, make sure it’s
actually correct! For example, I found out that on my machine, http-server is
in ~/.npm-global/bin. I can make sure that it’s the right directory by trying to
run the program http-server in that directory like this:
$ ~/.npm-global/bin/http-server
Starting up http-server, serving ./public
It worked! Now that you know what directory you need to add to your PATH,
let’s move to the next step!
Now we have the 2 critical pieces of information we need:
~/.npm-global/bin/)~/.bashrc, ~/.zshrc, or ~/.config/fish/config.fish)Now what you need to add depends on your shell:
bash instructions:
Open your shell’s config file, and add a line like this:
export PATH=$PATH:~/.npm-global/bin/
(obviously replace ~/.npm-global/bin with the actual directory you’re trying to add)
zsh instructions:
You can do the same thing as in bash, but zsh also has some slightly fancier syntax you can use if you prefer:
path=(
$path
~/.npm-global/bin
)
fish instructions:
In fish, the syntax is different:
set PATH $PATH ~/.npm-global/bin
(in fish you can also use fish_add_path, some notes on that further down)
Now, an extremely important step: updating your shell’s config won’t take effect if you don’t restart it!
Two ways to do this:
bash to start a new shell (or zsh if you’re using zsh, or fish if you’re using fish)I’ve found that both of these usually work fine.
And you should be done! Try running the program you were trying to run and hopefully it works now.
If not, here are a couple of problems that you might run into:
If the wrong version of a program is running, you might need to add the directory to the beginning of your PATH instead of the end.
For example, on my system I have two versions of python3 installed, which I
can see by running which -a:
$ which -a python3
/usr/bin/python3
/opt/homebrew/bin/python3
The one your shell will use is the first one listed.
If you want to use the Homebrew version, you need to add that directory
(/opt/homebrew/bin) to the beginning of your PATH instead, by putting this in
your shell’s config file (it’s /opt/homebrew/bin/:$PATH instead of the usual $PATH:/opt/homebrew/bin/)
export PATH=/opt/homebrew/bin/:$PATH
or in fish:
set PATH ~/.cargo/bin $PATH
All of these directions only work if you’re running the program from your shell. If you’re running the program from an IDE, from a GUI, in a cron job, or some other way, you’ll need to add the directory to your PATH in a different way, and the exact details might depend on the situation.
in a cron job
Some options:
/home/bork/bin/my-programecho "PATH=$PATH".I’m honestly not sure how to handle it in an IDE/GUI because I haven’t run into that in a long time, will add directions here if someone points me in the right direction.
PATH entries making it harder to debugIf you edit your path and start a new shell by running bash (or zsh, or
fish), you’ll often end up with duplicate PATH entries, because the shell
keeps adding new things to your PATH every time you start your shell.
Personally I don’t think I’ve run into a situation where this kind of
duplication breaks anything, but the duplicates can make it harder to debug
what’s going on with your PATH if you’re trying to understand its contents.
Some ways you could deal with this:
PATH, open a new terminal to do it in so you get
a “fresh” state. This should avoid the duplication.PATH at the end of your shell’s config (for example in
zsh apparently you can do this with typeset -U path)PATH when adding it (for
example in fish I believe you can do this with fish_add_path --path /some/directory)How to deduplicate your PATH is shell-specific and there isn’t always a
built in way to do it so you’ll need to look up how to accomplish it in your
shell.
PATHHere’s a situation that’s easy to get into in bash or zsh:
PATHbash to reload your configThis happens because in bash, by default, history is not saved until you exit the shell.
Some options for fixing this:
bash to reload your config, run source ~/.bashrc (or
source ~/.zshrc in zsh). This will reload the config inside your current
session.sourceWhen you install cargo (Rust’s installer) for the first time, it gives you
these instructions for how to set up your PATH, which don’t mention a specific
directory at all.
This is usually done by running one of the following (note the leading DOT):
. "$HOME/.cargo/env" # For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish" # For fish
The idea is that you add that line to your shell’s config, and their script
automatically sets up your PATH (and potentially other things) for you.
This is pretty common (for example Homebrew suggests you eval brew shellenv), and there are
two ways to approach this:
. "$HOME/.cargo/env" to your shell’s config). "$HOME/.cargo/env" in my shell (or the fish version if using fish)echo "$PATH" | tr ':' '\n' | grep cargo to figure out which directories it added/Users/bork/.cargo/bin and shorten that to ~/.cargo/bin~/.cargo/bin to PATH (with the directions in this post)I don’t think there’s anything wrong with doing what the tool suggests (it might be the “best way”!), but personally I usually use the second approach because I prefer knowing exactly what configuration I’m changing.
fish_add_pathfish has a handy function called fish_add_path that you can run to add a directory to your PATH like this:
fish_add_path /some/directory
This is cool (it’s such a simple command!) but I’ve stopped using it for a couple of reasons:
fish_add_path will update the PATH for every session in the
future (with a “universal variable”) and sometimes it will update the PATH
just for the current session and it’s hard for me to tell which one it will
do. In theory the docs explain this but I could not understand them.PATH a few weeks or
months later because maybe you made a mistake, it’s kind of hard to do
(there are instructions in this comments of this github issue though).Hopefully this will help some people. Let me know (on Mastodon or Bluesky) if you there are other major gotchas that have tripped you up when adding a directory to your PATH, or if you have questions about this post!
A few weeks ago I ran a terminal survey (you can read the results here) and at the end I asked:
What’s the most frustrating thing about using the terminal for you?
1600 people answered, and I decided to spend a few days categorizing all the responses. Along the way I learned that classifying qualitative data is not easy but I gave it my best shot. I ended up building a custom tool to make it faster to categorize everything.
As with all of my surveys the methodology isn’t particularly scientific. I just posted the survey to Mastodon and Twitter, ran it for a couple of days, and got answers from whoever happened to see it and felt like responding.
Here are the top categories of frustrations!
I think it’s worth keeping in mind while reading these comments that
These comments aren’t coming from total beginners.
Here are the categories of frustrations! The number in brackets is the number of people with that frustration. I’m mostly writing this up for myself because I’m trying to write a zine about the terminal and I wanted to get a sense for what people are having trouble with.
People talked about struggles remembering:
One example comment:
There are just so many little “trivia” details to remember for full functionality. Even after all these years I’ll sometimes forget where it’s 2 or 1 for stderr, or forget which is which for
>and>>.
People talked about struggling with switching systems (for example home/work computer or when SSHing) and running into:
as well as differences inside the same system like pagers being not consistent with each other (git diff pagers, other pagers).
One example comment:
I got used to fish and vi mode which are not available when I ssh into servers, containers.
Lots of problems with color, like:
This comment felt relatable to me:
Getting my terminal theme configured in a reasonable way between the terminal emulator and fish (I did this years ago and remember it being tedious and fiddly and now feel like I’m locked into my current theme because it works and I dread touching any of that configuration ever again).
Half of the comments on keyboard shortcuts were about how on Linux/Windows, the keyboard shortcut to copy/paste in the terminal is different from in the rest of the OS.
Some other issues with keyboard shortcuts other than copy/paste:
Ctrl-W in a browser-based terminal and closing the windowCtrl-Shift-, no Super, no Hyper, lots of ctrl- shortcuts aren’t
possible like Ctrl-,)Ctrl+left arrow for something else)Aside from “the keyboard shortcut for copy and paste is different”, there were a lot of OTHER issues with copy and paste, like:
There were lots of comments about this, which all came down to the same basic complaint – it’s hard to discover useful tools or features! This comment kind of summed it all up:
How difficult it is to learn independently. Most of what I know is an assorted collection of stuff I’ve been told by random people over the years.
A lot of comments about it generally having a steep learning curve. A couple of example comments:
After 15 years of using it, I’m not much faster than using it than I was 5 or maybe even 10 years ago.
and
That I know I could make my life easier by learning more about the shortcuts and commands and configuring the terminal but I don’t spend the time because it feels overwhelming.
Some issues with shell history:
One example comment:
It wasted a lot of time until I figured it out and still annoys me that “history” on zsh has such a small buffer; I have to type “history 0” to get any useful length of history.
People talked about:
Here’s a representative comment:
Finding good examples and docs. Man pages often not enough, have to wade through stack overflow
A few issues with scrollback:
One example comment:
When resizing the terminal (in particular: making it narrower) leads to broken rewrapping of the scrollback content because the commands formatted their output based on the terminal window width.
Lots of comments about how the terminal feels hampered by legacy decisions and how users often end up needing to learn implementation details that feel very esoteric. One example comment:
Most of the legacy cruft, it would be great to have a green field implementation of the CLI interface.
Lots of complaints about POSIX shell scripting. There’s a general feeling that shell scripting is difficult but also that switching to a different less standard scripting language (fish, nushell, etc) brings its own problems.
Shell scripting. My tolerance to ditch a shell script and go to a scripting language is pretty low. It’s just too messy and powerful. Screwing up can be costly so I don’t even bother.
Some more issues that were mentioned at least 10 times:
Ctrl-S, cating a binary, etc)There were also 122 answers to the effect of “nothing really” or “only that I can’t do EVERYTHING in the terminal”
One example comment:
Think I’ve found work arounds for most/all frustrations
I’m not going to make a lot of commentary on these results, but here are a couple of categories that feel related to me:
Trying to categorize all these results in a reasonable way really gave me an appreciation for social science researchers’ skills.
Hello! Recently I ran a terminal survey and I asked people what frustrated them. One person commented:
There are so many pieces to having a modern terminal experience. I wish it all came out of the box.
My immediate reaction was “oh, getting a modern terminal experience isn’t that hard, you just need to….”, but the more I thought about it, the longer the “you just need to…” list got, and I kept thinking about more and more caveats.
So I thought I would write down some notes about what it means to me personally to have a “modern” terminal experience and what I think can make it hard for people to get there.
Here are a few things that are important to me, with which part of the system is responsible for them:
p in vim to paste (text editor, maybe the OS/terminal emulator too)ls (shell config)Ctrl+left arrow to work (shell or application)less: (terminal emulator and applications)There are a million other terminal conveniences out there and different people value different things, but those are the ones that I would be really unhappy without.
My basic approach is:
fish shell. Mostly don’t configure it, except to:
EDITOR environment variable to my favourite terminal editorls to ls --color=autoneovim, with a configuration that I’ve been very slowly building over the last 9 years or so (the last time I deleted my vim config and started from scratch was 9 years ago)A few things that affect my approach:
What if you want a nice experience, but don’t want to spend a lot of time on configuration? Figuring out how to configure vim in a way that I was satisfied with really did take me like ten years, which is a long time!
My best ideas for how to get a reasonable terminal experience with minimal config are:
fish or zsh with oh-my-zshEDITOR environment variable to your favourite terminal text
editorls to ls --color=autoCtrl-C to copy, Ctrl-V to paste, Ctrl-A to select all) in micro and
they do what you’d expect. I would probably try switching to helix except
that retraining my vim muscle memory seems way too hard. Also helix doesn’t
have a GUI or plugin system yet.Personally I wouldn’t use xterm, rxvt, or Terminal.app as a terminal emulator, because I’ve found in the past that they’re missing core features (like 24-bit colour in Terminal.app’s case) that make the terminal harder to use for me.
I don’t want to pretend that getting a “modern” terminal experience is easier than it is though – I think there are two issues that make it hard. Let’s talk about them!
bash and zsh are by far the two most popular shells, and neither of them provide a default experience that I would be happy using out of the box, for example:
And even though I love fish, the fact that it isn’t POSIX does make it hard for a lot of folks to make the switch.
Of course it’s totally possible to learn how to customize your prompt in bash
or whatever, and it doesn’t even need to be that complicated (in bash I’d
probably start with something like export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ ', or maybe use starship).
But each of these “not complicated” things really does add up and it’s
especially tough if you need to keep your config in sync across several
systems.
An extremely popular solution to getting a “modern” shell experience is oh-my-zsh. It seems like a great project and I know a lot of people use it very happily, but I’ve struggled with configuration systems like that in the past – it looks like right now the base oh-my-zsh adds about 3000 lines of config, and often I find that having an extra configuration system makes it harder to debug what’s happening when things go wrong. I personally have a tendency to use the system to add a lot of extra plugins, make my system slow, get frustrated that it’s slow, and then delete it completely and write a new config from scratch.
In the terminal survey I ran recently, the most popular terminal text editors
by far were vim, emacs, and nano.
I think the main options for terminal text editors are:
micro or helix which seem to offer a pretty good out-of-the-box
experience, potentially occasionally run into issues with using a less
mainstream text editorcode as their EDITOR in the terminal.The last issue is that sometimes individual programs that I use are kind of
annoying. For example on my Mac OS machine, /usr/bin/sqlite3 doesn’t support
the Ctrl+Left Arrow keyboard shortcut. Fixing this to get a reasonable
terminal experience in SQLite was a little complicated, I had to:
I find that debugging application-specific issues like this is really not easy and often it doesn’t feel “worth it” – often I’ll end up just dealing with various minor inconveniences because I don’t want to spend hours investigating them. The only reason I was even able to figure this one out at all is that I’ve been spending a huge amount of time thinking about the terminal recently.
A big part of having a “modern” experience using terminal programs is just
using newer terminal programs, for example I can’t be bothered to learn a
keyboard shortcut to sort the columns in top, but in htop I can just click
on a column heading with my mouse to sort it. So I use htop instead! But discovering new more “modern” command line tools isn’t easy (though
I made a list here),
finding ones that I actually like using in practice takes time, and if you’re
SSHed into another machine, they won’t always be there.
Something I find tricky about configuring my terminal to make everything “nice” is that changing one seemingly small thing about my workflow can really affect everything else. For example right now I don’t use tmux. But if I needed to use tmux again (for example because I was doing a lot of work SSHed into another machine), I’d need to think about a few things, like:
and probably more things I haven’t thought of. “Using tmux means that I have to change how I manage my colours” sounds unlikely, but that really did happen to me and I decided “well, I don’t want to change how I manage colours right now, so I guess I’m not using that feature!”.
It’s also hard to remember which features I’m relying on – for example maybe my current terminal does have OSC 52 support and because copying from tmux over SSH has always Just Worked I don’t even realize that that’s something I need, and then it mysteriously stops working when I switch terminals.
Personally even though I think my setup is not that complicated, it’s taken me 20 years to get to this point! Because terminal config changes are so likely to have unexpected and hard-to-understand consequences, I’ve found that if I change a lot of terminal configuration all at once it makes it much harder to understand what went wrong if there’s a problem, which can be really disorienting.
So I usually prefer to make pretty small changes, and accept that changes can
might take me a REALLY long time to get used to. For example I switched from
using ls to eza a year or two ago and
while I like it (because eza -l prints human-readable file sizes by default)
I’m still not quite sure about it. But also sometimes it’s worth it to make a
big change, like I made the switch to fish (from bash) 10 years ago and I’m
very happy I did.
Trying to explain how “easy” it is to configure your terminal really just made me think that it’s kind of hard and that I still sometimes get confused.
I’ve found that there’s never one perfect way to configure things in the terminal that will be compatible with every single other thing. I just need to try stuff, figure out some kind of locally stable state that works for me, and accept that if I start using a new tool it might disrupt the system and I might need to rethink things.
Recently I’ve been thinking about how everything that happens in the terminal is some combination of:
top or vim or cat)The first three (your operating system, shell, and terminal emulator) are all kind of known quantities – if you’re using bash in GNOME Terminal on Linux, you can more or less reason about how how all of those things interact, and some of their behaviour is standardized by POSIX.
But the fourth one (“whatever program you happen to be running”) feels like it could do ANYTHING. How are you supposed to know how a program is going to behave?
This post is kind of long so here’s a quick table of contents:
Ctrl-CqCtrl-D on an empty lineCtrl-W should delete the last word- means stdin/stdoutAs far as I know, there are no real standards for how programs in the terminal should behave – the closest things I know of are:
cp should work but AFAIK it doesn’t have anything to say about how for
example htop should behave.But even though there are no standards, in my experience programs in the terminal behave in a pretty consistent way. So I wanted to write down a list of “rules” that in my experience programs mostly follow.
My goal here isn’t to convince authors of terminal programs that they should follow any of these rules. There are lots of exceptions to these and often there’s a good reason for those exceptions.
But it’s very useful for me to know what behaviour to expect from a random new terminal program that I’m using. Instead of “uh, programs could do literally anything”, it’s “ok, here are the basic rules I expect, and then I can keep a short mental list of exceptions”.
So I’m just writing down what I’ve observed about how programs behave in my 20 years of using the terminal, why I think they behave that way, and some examples of cases where that rule is “broken”.
There are a bunch of common conventions that I think are pretty clearly the program’s responsibility to implement, like:
~/.BLAHrc or ~/.config/BLAH/FILE or /etc/BLAH/ or something--help should print help textBut in this post I’m going to focus on things that it’s not 100% obvious are
the program’s responsibility. For example it feels to me like a “law of nature”
that pressing Ctrl-D should quit a REPL, but programs often
need to explicitly implement support for it – even though cat doesn’t need
to implement Ctrl-D support, ipython does. (more about that in “rule 3” below)
Understanding which things are the program’s responsibility makes it much less surprising when different programs’ implementations are slightly different.
Ctrl-CThe main reason for this rule is that noninteractive programs will quit by
default on Ctrl-C if they don’t set up a SIGINT signal handler, so this is
kind of a “you should act like the default” rule.
Something that trips a lot of people up is that this doesn’t apply to
interactive programs like python3 or bc or less. This is because in
an interactive program, Ctrl-C has a different job – if the program is
running an operation (like for example a search in less or some Python code
in python3), then Ctrl-C will interrupt that operation but not stop the
program.
As an example of how this works in an interactive program: here’s the code in prompt-toolkit (the library that iPython uses for handling input)
that aborts a search when you press Ctrl-C.
qTUI programs (like less or htop) will usually quit when you press q.
This rule doesn’t apply to any program where pressing q to quit wouldn’t make
sense, like tmux or text editors.
Ctrl-D on an empty lineREPLs (like python3 or ed) will usually quit when you press Ctrl-D on an
empty line. This rule is similar to the Ctrl-C rule – the reason for this is
that by default if you’re running a program (like cat) in “cooked mode”, then
the operating system will return an EOF when you press Ctrl-D on an empty
line.
Most of the REPLs I use (sqlite3, python3, fish, bash, etc) don’t actually use cooked mode, but they all implement this keyboard shortcut anyway to mimic the default behaviour.
For example, here’s the code in prompt-toolkit that quits when you press Ctrl-D, and here’s the same code in readline.
I actually thought that this one was a “Law of Terminal Physics” until very recently because I’ve basically never seen it broken, but you can see that it’s just something that each individual input library has to implement in the links above.
Someone pointed out that the Erlang REPL does not quit when you press Ctrl-D,
so I guess not every REPL follows this “rule”.
Terminal programs rarely use colours other than the base 16 ANSI colours. This
is because if you specify colours with a hex code, it’s very likely to clash
with some users’ background colour. For example if I print out some text as
#EEEEEE, it would be almost invisible on a white background, though it would
look fine on a dark background.
But if you stick to the default 16 base colours, you have a much better chance that the user has configured those colours in their terminal emulator so that they work reasonably well with their background color. Another reason to stick to the default base 16 colours is that it makes less assumptions about what colours the terminal emulator supports.
The only programs I usually see breaking this “rule” are text editors, for example Helix by default will use a purple background which is not a default ANSI colour. It seems fine for Helix to break this rule since Helix isn’t a “core” program and I assume any Helix user who doesn’t like that colorscheme will just change the theme.
Almost every program I use supports readline keybindings if it would make
sense to do so. For example, here are a bunch of different programs and a link
to where they define Ctrl-E to go to the end of the line:
None of those programs actually uses readline directly, they just sort of
mimic emacs/readline keybindings. They don’t always mimic them exactly: for
example atuin seems to use Ctrl-A as a prefix, so Ctrl-A doesn’t go to the
beginning of the line.
Also all of these programs seem to implement their own internal cut and paste
buffers so you can delete a line with Ctrl-U and then paste it with Ctrl-Y.
The exceptions to this are:
git, cat, and nc) don’t have any line editing support at all (except for backspace, Ctrl-W, and Ctrl-U)I wrote more about this “what keybindings does a program support?” question in entering text in the terminal is complicated.
I’ve never seen a program (other than a text editor) where Ctrl-W doesn’t
delete the last word. This is similar to the Ctrl-C rule – by default if a
program is in “cooked mode”, the OS will delete the last word if you press
Ctrl-W, and delete the whole line if you press Ctrl-U. So usually programs
will imitate that behaviour.
I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!
Most programs will disable colours when writing to a pipe. For example:
rg blah will highlight all occurrences of blah in the output, but if the
output is to a pipe or a file, it’ll turn off the highlighting.ls --color=auto will use colour when writing to a terminal, but not when
writing to a pipeBoth of those programs will also format their output differently when writing
to the terminal: ls will organize files into columns, and ripgrep will group
matches with headings.
If you want to force the program to use colour (for example because you want to
look at the colour), you can use unbuffer to force the program’s output to be
a tty like this:
unbuffer rg blah | less -R
I’m sure that there are some programs that “break” this rule but I can’t think
of any examples right now. Some programs have an --color flag that you can
use to force colour to be on, in the example above you could also do rg --color=always | less -R.
- means stdin/stdoutUsually if you pass - to a program instead of a filename, it’ll read from
stdin or write to stdout (whichever is appropriate). For example, if you want
to format the Python code that’s on your clipboard with black and then copy
it, you could run:
pbpaste | black - | pbcopy
(pbpaste is a Mac program, you can do something similar on Linux with xclip)
My impression is that most programs implement this if it would make sense and I can’t think of any exceptions right now, but I’m sure there are many exceptions.
These rules took me a long time for me to learn because I had to:
Ctrl-C will exit programs")Ctrl-C will exit find but not less”)Ctrl-C will generally quit
noninteractive programs, but in interactive programs it might interrupt the
current operation instead of quitting the program")A lot of my understanding of the terminal is honestly still in the “subconscious pattern recognition” stage. The only reason I’ve been taking the time to make things explicit at all is because I’ve been trying to explain how it works to others. Hopefully writing down these “rules” explicitly will make learning some of this stuff a little bit faster for others.
Here’s a niche terminal problem that has bothered me for years but that I never really understood until a few weeks ago. Let’s say you’re running this command to watch for some specific output in a log file:
tail -f /some/log/file | grep thing1 | grep thing2
If log lines are being added to the file relatively slowly, the result I’d see is… nothing! It doesn’t matter if there were matches in the log file or not, there just wouldn’t be any output.
I internalized this as “uh, I guess pipes just get stuck sometimes and don’t
show me the output, that’s weird”, and I’d handle it by just
running grep thing1 /some/log/file | grep thing2 instead, which would work.
So as I’ve been doing a terminal deep dive over the last few months I was really excited to finally learn exactly why this happens.
The reason why “pipes get stuck” sometimes is that it’s VERY common for programs to buffer their output before writing it to a pipe or file. So the pipe is working fine, the problem is that the program never even wrote the data to the pipe!
This is for performance reasons: writing all output immediately as soon as you can uses more system calls, so it’s more efficient to save up data until you have 8KB or so of data to write (or until the program exits) and THEN write it to the pipe.
In this example:
tail -f /some/log/file | grep thing1 | grep thing2
the problem is that grep thing1 is saving up all of its matches until it has
8KB of data to write, which might literally never happen.
Part of why I found this so disorienting is that tail -f file | grep thing
will work totally fine, but then when you add the second grep, it stops
working!! The reason for this is that the way grep handles buffering depends
on whether it’s writing to a terminal or not.
Here’s how grep (and many other programs) decides to buffer its output:
isatty function
So if grep is writing directly to your terminal then you’ll see the line as
soon as it’s printed, but if it’s writing to a pipe, you won’t.
Of course the buffer size isn’t always 8KB for every program, it depends on the implementation. For grep the buffering is handled by libc, and libc’s buffer size is
defined in the BUFSIZ variable. Here’s where that’s defined in glibc.
(as an aside: “programs do not use 8KB output buffers when writing to a terminal” isn’t, like, a law of terminal physics, a program COULD use an 8KB buffer when writing output to a terminal if it wanted, it would just be extremely weird if it did that, I can’t think of any program that behaves that way)
One annoying thing about this buffering behaviour is that you kind of need to remember which commands buffer their output when writing to a pipe.
Some commands that don’t buffer their output:
I think almost everything else will buffer output, especially if it’s a command where you’re likely to be using it for batch processing. Here’s a list of some common commands that buffer their output when writing to a pipe, along with the flag that disables block buffering.
--line-buffered)-u)fflush() function)-l)-u)-u)Those are all the ones I can think of, lots of unix commands (like sort) may
or may not buffer their output but it doesn’t matter because sort can’t do
anything until it finishes receiving input anyway.
Also I did my best to test both the Mac OS and GNU versions of these but there are a lot of variations and I might have made some mistakes.
Also, here are a few programming language where the default print statement will buffer output when writing to a pipe, and some ways to disable buffering if you want:
setvbuf)python -u, or PYTHONUNBUFFERED=1, or sys.stdout.reconfigure(line_buffering=False), or print(x, flush=True))STDOUT.sync = true)$| = 1)I assume that these languages are designed this way so that the default print function will be fast when you’re doing batch processing.
Also whether output is buffered or not might depend on how you print, for
example in C++ cout << "hello\n" buffers when writing to a pipe but cout << "hello" << endl will flush its output.
Ctrl-C on a pipe, the contents of the buffer are lostLet’s say you’re running this command as a hacky way to watch for DNS requests
to example.com, and you forgot to pass -l to tcpdump:
sudo tcpdump -ni any port 53 | grep example.com
When you press Ctrl-C, what happens? In a magical perfect world, what I would
want to happen is for tcpdump to flush its buffer, grep would search for
example.com, and I would see all the output I missed.
But in the real world, what happens is that all the programs get killed and the
output in tcpdump’s buffer is lost.
I think this problem is probably unavoidable – I spent a little time with
strace to see how this works and grep receives the SIGINT before
tcpdump anyway so even if tcpdump tried to flush its buffer grep would
already be dead.
After a little more investigation, there is a workaround: if you find
tcpdump’s PID and kill -TERM $PID, then tcpdump will flush the buffer so
you can see the output. That’s kind of a pain but I tested it and it seems to
work.
It’s not just pipes, this will also buffer:
sudo tcpdump -ni any port 53 > output.txt
Redirecting to a file doesn’t have the same “Ctrl-C will totally destroy the
contents of the buffer” problem though – in my experience it usually behaves
more like you’d want, where the contents of the buffer get written to the file
before the program exits. I’m not 100% sure whether this is something you can
always rely on or not.
Okay, let’s talk solutions. Let’s say you’ve run this command:
tail -f /some/log/file | grep thing1 | grep thing2
I asked people on Mastodon how they would solve this in practice and there were 5 basic approaches. Here they are:
Historically my solution to this has been to just avoid the “command writing to pipe slowly” situation completely and instead run a program that will finish quickly like this:
cat /some/log/file | grep thing1 | grep thing2 | tail
This doesn’t do the same thing as the original command but it does mean that you get to avoid thinking about these weird buffering issues.
(you could also do grep thing1 /some/log/file but I often prefer to use an
“unnecessary” cat)
You could remember that grep has a flag to avoid buffering and pass it like this:
tail -f /some/log/file | grep --line-buffered thing1 | grep thing2
Some people said that if they’re specifically dealing with a multiple greps
situation, they’ll rewrite it to use a single awk instead, like this:
tail -f /some/log/file | awk '/thing1/ && /thing2/'
Or you would write a more complicated grep, like this:
tail -f /some/log/file | grep -E 'thing1.*thing2'
(awk also buffers, so for this to work you’ll want awk to be the last command in the pipeline)
stdbufstdbuf uses LD_PRELOAD to turn off libc’s buffering, and you can use it to turn off output buffering like this:
tail -f /some/log/file | stdbuf -o0 grep thing1 | grep thing2
Like any LD_PRELOAD solution it’s a bit unreliable – it doesn’t work on
static binaries, I think won’t work if the program isn’t using libc’s
buffering, and doesn’t always work on Mac OS. Harry Marr has a really nice How stdbuf works post.
unbufferunbuffer program will force the program’s output to be a TTY, which means
that it’ll behave the way it normally would on a TTY (less buffering, colour
output, etc). You could use it in this example like this:
tail -f /some/log/file | unbuffer grep thing1 | grep thing2
Unlike stdbuf it will always work, though it might have unwanted side
effects, for example grep thing1’s will also colour matches.
If you want to install unbuffer, it’s in the expect package.
It’s a bit hard for me to say which one is “best”, I think personally I’m
mostly likely to use unbuffer because I know it’s always going to work.
If I learn about more solutions I’ll try to add them to this post.
I think it’s not very common for me to have a program that slowly trickles data into a pipe like this, normally if I’m using a pipe a bunch of data gets written very quickly, processed by everything in the pipeline, and then everything exits. The only examples I can come up with right now are:
tail -fkubectl logsI think it would be cool if there were a standard environment variable to turn
off buffering, like PYTHONUNBUFFERED in Python. I got this idea from a
couple of blog posts by Mark Dominus
in 2018. Maybe NO_BUFFER like NO_COLOR?
The design seems tricky to get right; Mark points out that NETBSD has environment variables called STDBUF, STDBUF1, etc which gives you a
ton of control over buffering but I imagine most developers don’t want to
implement many different environment variables to handle a relatively minor
edge case.
I’m also curious about whether there are any programs that just automatically flush their output buffers after some period of time (like 1 second). It feels like it would be nice in theory but I can’t think of any program that does that so I imagine there are some downsides.
Some things I didn’t talk about in this post since these posts have been getting pretty long recently and seriously does anyone REALLY want to read 3000 words about buffering?
I like writing Javascript without a build system and for the millionth time yesterday I ran into a problem where I needed to figure out how to import a Javascript library in my code without using a build system, and it took FOREVER to figure out how to import it because the library’s setup instructions assume that you’re using a build system.
Luckily at this point I’ve mostly learned how to navigate this situation and either successfully use the library or decide it’s too difficult and switch to a different library, so here’s the guide I wish I had to importing Javascript libraries years ago.
I’m only going to talk about using Javacript libraries on the frontend, and only about how to use them in a no-build-system setup.
In this post I’m going to talk about:
There are 3 basic types of Javascript files a library can provide:
<script src> and it’ll Just Work. Great if you
can get it but not always availableI’m not sure if there’s a better name for the “classic” type but I’m just going to call it “classic”. Also there’s a type called “AMD” but I’m not sure how relevant it is in 2024.
Now that we know the 3 types of files, let’s talk about how to figure out which of these the library actually provides!
Every Javascript library has a build which it uploads to NPM. You might be thinking (like I did originally) – Julia! The whole POINT is that we’re not using Node to build our library! Why are we talking about NPM?
But if you’re using a link from a CDN like https://cdnjs.cloudflare.com/ajax/libs/Chart.js/4.4.1/chart.umd.min.js, you’re still using the NPM build! All the files on the CDNs originally come from NPM.
Because of this, I sometimes like to npm install the library even if I’m not
planning to use Node to build my library at all – I’ll just create a new temp
folder, npm install there, and then delete it when I’m done. I like being able to poke
around in the files in the NPM build on my filesystem, because then I can be
100% sure that I’m seeing everything that the library is making available in
its build and that the CDN isn’t hiding something from me.
So let’s npm install a few libraries and try to figure out what types of
Javascript files they provide in their builds!
First let’s look inside Chart.js, a plotting library.
$ cd /tmp/whatever
$ npm install chart.js
$ cd node_modules/chart.js/dist
$ ls *.*js
chart.cjs chart.js chart.umd.js helpers.cjs helpers.js
This library seems to have 3 basic options:
option 1: chart.cjs. The .cjs suffix tells me that this is a CommonJS
file, for using in Node. This means it’s impossible to use it directly in the
browser without some kind of build step.
option 2:chart.js. The .js suffix by itself doesn’t tell us what kind of
file it is, but if I open it up, I see import '@kurkle/color'; which is an
immediate sign that this is an ES module – the import ... syntax is ES
module syntax.
option 3: chart.umd.js. “UMD” stands for “Universal Module Definition”,
which I think means that you can use this file either with a basic <script src>, CommonJS,
or some third thing called AMD that I don’t understand.
When I was using Chart.js I picked Option 3. I just needed to add this to my code:
<script src="./chart.umd.js"> </script>
and then I could use the library with the global Chart environment variable.
Couldn’t be easier. I just copied chart.umd.js into my Git repository so that
I didn’t have to worry about using NPM or the CDNs going down or anything.
dist directoryA lot of libraries will put their build in the dist directory, but not
always! The build files’ location is specified in the library’s package.json.
For example here’s an excerpt from Chart.js’s package.json.
"jsdelivr": "./dist/chart.umd.js",
"unpkg": "./dist/chart.umd.js",
"main": "./dist/chart.cjs",
"module": "./dist/chart.js",
I think this is saying that if you want to use an ES Module (module) you
should use dist/chart.js, but the jsDelivr and unpkg CDNs should use
./dist/chart.umd.js. I guess main is for Node.
chart.js’s package.json also says "type": "module", which according to this documentation
tells Node to treat files as ES modules by default. I think it doesn’t tell us
specifically which files are ES modules and which ones aren’t but it does tell
us that something in there is an ES module.
@atcute/oauth-browser-client@atcute/oauth-browser-client
is a library for logging into Bluesky with OAuth in the browser.
Let’s see what kinds of Javascript files it provides in its build!
$ npm install @atcute/oauth-browser-client
$ cd node_modules/@atcute/oauth-browser-client/dist
$ ls *js
constants.js dpop.js environment.js errors.js index.js resolvers.js
It seems like the only plausible root file in here is index.js, which looks
something like this:
export { configureOAuth } from './environment.js';
export * from './errors.js';
export * from './resolvers.js';
This export syntax means it’s an ES module. That means we can use it in
the browser without a build step! Let’s see how to do that.
Using an ES module isn’t an easy as just adding a <script src="whatever.js">. Instead, if
the ES module has dependencies (like @atcute/oauth-browser-client does) the
steps are:
import { configureOAuth } from '@atcute/oauth-browser-client'; in your JS code<script type="module" src="YOURSCRIPT.js"></script>The reason we need an import map instead of just doing something like import { BrowserOAuthClient } from "./oauth-client-browser.js" is that internally the module has more import statements like import {something} from @atcute/client, and we need to tell the browser where to get the code for @atcute/client and all of its other dependencies.
Here’s what the importmap I used looks like for @atcute/oauth-browser-client:
<script type="importmap">
{
"imports": {
"nanoid": "./node_modules/nanoid/bin/dist/index.js",
"nanoid/non-secure": "./node_modules/nanoid/non-secure/index.js",
"nanoid/url-alphabet": "./node_modules/nanoid/url-alphabet/dist/index.js",
"@atcute/oauth-browser-client": "./node_modules/@atcute/oauth-browser-client/dist/index.js",
"@atcute/client": "./node_modules/@atcute/client/dist/index.js",
"@atcute/client/utils/did": "./node_modules/@atcute/client/dist/utils/did.js"
}
}
</script>
Getting these import maps to work is pretty fiddly, I feel like there must be a tool to generate them automatically but I haven’t found one yet. It’s definitely possible to write a script that automatically generates the importmaps using esbuild’s metafile but I haven’t done that and maybe there’s a better way.
I decided to set up importmaps yesterday to get github.com/jvns/bsky-oauth-example to work, so there’s some example code in that repo.
Also someone pointed me to Simon Willison’s download-esm, which will download an ES module and rewrite the imports to point to the JS files directly so that you don’t need importmaps. I haven’t tried it yet but it seems like a great idea.
I did run into some problems with using importmaps in the browser though – it needed to download dozens of Javascript files to load my site, and my webserver in development couldn’t keep up for some reason. I kept seeing files fail to load randomly and then had to reload the page and hope that they would succeed this time.
It wasn’t an issue anymore when I deployed my site to production, so I guess it was a problem with my local dev environment.
Also one slightly annoying thing about ES modules in general is that you need to
be running a webserver to use them, I’m sure this is for a good reason but it’s
easier when you can just open your index.html file without starting a
webserver.
Because of the “too many files” thing I think actually using ES modules with importmaps in this way isn’t actually that appealing to me, but it’s good to know it’s possible.
If the ES module doesn’t have dependencies then it’s even easier – you don’t need the importmaps! You can just:
<script type="module" src="YOURCODE.js"></script> in your HTML. The type="module" is important.import {whatever} from "https://example.com/whatever.js" in YOURCODE.jsIf you don’t want to use importmaps, you can also use a build system like esbuild. I talked about how to do that in Some notes on using esbuild, but this blog post is about ways to avoid build systems completely so I’m not going to talk about that option here. I do still like esbuild though and I think it’s a good option in this case.
CanIUse says that importmaps are in
“Baseline 2023: newly available across major browsers” so my sense is that in
2024 that’s still maybe a little bit too new? I think I would use importmaps
for some fun experimental code that I only wanted like myself and 12 people to
use, but if I wanted my code to be more widely usable I’d use esbuild instead.
@atproto/oauth-client-browserLet’s look at one final example library! This is a different Bluesky auth
library than @atcute/oauth-browser-client.
$ npm install @atproto/oauth-client-browser
$ cd node_modules/@atproto/oauth-client-browser/dist
$ ls *js
browser-oauth-client.js browser-oauth-database.js browser-runtime-implementation.js errors.js index.js indexed-db-store.js util.js
Again, it seems like only real candidate file here is index.js. But this is a
different situation from the previous example library! Let’s take a look at
index.js:
There’s a bunch of stuff like this in index.js:
__exportStar(require("@atproto/oauth-client"), exports);
__exportStar(require("./browser-oauth-client.js"), exports);
__exportStar(require("./errors.js"), exports);
var util_js_1 = require("./util.js");
This require() syntax is CommonJS syntax, which means that we can’t use this
file in the browser at all, we need to use some kind of build step, and
ESBuild won’t work either.
Also in this library’s package.json it says "type": "commonjs" which is
another way to tell it’s CommonJS.
Originally I thought it was impossible to use CommonJS modules without learning a build system, but then someone Bluesky told me about esm.sh! It’s a CDN that will translate anything into an ES Module. skypack.dev does something similar, I’m not sure what the difference is but one person mentioned that if one doesn’t work sometimes they’ll try the other one.
For @atproto/oauth-client-browser using it seems pretty simple, I just need to put this in my HTML:
<script type="module" src="script.js"> </script>
and then put this in script.js.
import { BrowserOAuthClient } from "https://esm.sh/@atproto/oauth-client-browser@0.3.0"
It seems to Just Work, which is cool! Of course this is still sort of using a build system – it’s just that esm.sh is running the build instead of me. My main concerns with this approach are:
I also learned that you can also use esbuild to convert a CommonJS module
into an ES module, though there are some limitations – the import { BrowserOAuthClient } from syntax doesn’t work. Here’s a github issue about that.
I think the esbuild approach is probably more appealing to me than the
esm.sh approach because it’s a tool that I already have on my computer so I
trust it more. I haven’t experimented with this much yet though.
Here’s a summary of the three types of JS files you might encounter, options for how to use them, and how to identify them.
Unhelpfully a .js or .min.js file extension could be any of these 3
options, so if the file is something.js you need to do more detective work to
figure out what you’re dealing with.
<script src="whatever.js"></script>.umd.js extension<script src=... tag and see if it worksimport {whatever} from "./my-module.js" directly in your codeimport {whatever} from "my-module"
import or export statement. (not module.exports = ..., that’s CommonJS).mjs extension"type": "module" in package.json (though it’s not clear to me which file exactly this refers to)https://esm.sh/@atproto/oauth-client-browser@0.3.0require() or module.exports = ... in the code.cjs extension"type": "commonjs" in package.json (though it’s not clear to me which file exactly this refers to)The main difference between CommonJS modules and ES modules from my perspective is that ES modules are actually a standard. This makes me feel a lot more confident using them, because browsers commit to backwards compatibility for web standards forever – if I write some code using ES modules today, I can feel sure that it’ll still work the same way in 15 years.
It also makes me feel better about using tooling like esbuild because even if
the esbuild project dies, because it’s implementing a standard it feels likely
that there will be another similar tool in the future that I can replace it
with.
A lot of the time when I talk about this stuff I get responses like “I hate javascript!!! it’s the worst!!!”. But my experience is that there are a lot of great tools for Javascript (I just learned about https://esm.sh yesterday which seems great! I love esbuild!), and that if I take the time to learn how things works I can take advantage of some of those tools and make my life a lot easier.
So the goal of this post is definitely not to complain about Javascript, it’s to understand the landscape so I can use the tooling in a way that feels good to me.
Here are some questions I still have, I’ll add the answers into the post if I learn the answer.
atcute-client.js), but so that in the browser I can still import multiple
different paths from that file (like both @atcute/client/lexicons and
@atcute/client)?Here’s a list of every tool we talked about in this post:
Writing this post has made me think that even though I usually don’t want to
have a build that I run every time I update the project, I might be willing to
have a build step (using download-esm or something) that I run only once
when setting up the project and never run again except maybe if I’m updating my
dependency versions.
Thanks to Marco Rogers who taught me a lot of the things in this post. I’ve probably made some mistakes in this post and I’d love to know what they are – let me know on Bluesky or Mastodon!
I added a new section to this site a couple weeks ago called TIL (“today I learned”).
One kind of thing I like to post on Mastodon/Bluesky is “hey, here’s a cool thing”, like the great SQLite repl litecli, or the fact that cross compiling in Go Just Works and it’s amazing, or cryptographic right answers, or this great diff tool. Usually I don’t want to write a whole blog post about those things because I really don’t have much more to say than “hey this is useful!”
It started to bother me that I didn’t have anywhere to put those things: for example recently I wanted to use diffdiff and I just could not remember what it was called.
So I quickly made a new folder called /til/, added some
custom styling (I wanted to style the posts to look a little bit like a tweet),
made a little Rake task to help me create new posts quickly (rake new_til), and
set up a separate RSS Feed for it.
I think this new section of the blog might be more for myself than anything, now when I forget the link to Cryptographic Right Answers I can hopefully look it up on the TIL page. (you might think “julia, why not use bookmarks??” but I have been failing to use bookmarks for my whole life and I don’t see that changing ever, putting things in public is for whatever reason much easier for me)
So far it’s been working, often I can actually just make a quick post in 2 minutes which was the goal.
My page is inspired by Simon Willison’s great TIL blog, though my TIL posts are a lot shorter.
This came about because I spent a lot of time on Twitter, so I’ve been thinking about what I want to do about all of my tweets.
I keep reading the advice to “POSSE” (“post on your own site, syndicate elsewhere”), and while I find the idea appealing in principle, for me part of the appeal of social media is that it’s a little bit ephemeral. I can post polls or questions or observations or jokes and then they can just kind of fade away as they become less relevant.
I find it a lot easier to identify specific categories of things that I actually want to have on a Real Website That I Own:
and then let everything else be kind of ephemeral.
I really believe in the advice to make email lists though – the first two (blog posts & comics) both have email lists and RSS feeds that people can subscribe to if they want. I might add a quick summary of any TIL posts from that week to the “blog posts from this week” mailing list.
Here's where you can find me at IETF 121 in Dublin!
Hello! I’ve been thinking about the terminal a lot and yesterday I got curious
about all these “control codes”, like Ctrl-A, Ctrl-C, Ctrl-W, etc. What’s
the deal with all of them?
Here’s a table of all 33 ASCII control characters, and what they do on my machine (on Mac OS), more or less. There are about a million caveats, but I’ll talk about what it means and all the problems with this diagram that I know about.
You can also view it as an HTML page (I just made it an image so it would show up in RSS).
The first surprising thing about this diagram to me is that there are 33 control codes, split into (very roughly speaking) these categories:
3 (Ctrl-C), it’ll send a SIGINT signal to
the current programEnter, Tab, Backspace). For example when you press Enter, your
terminal gets sent 13.readline: “the application can do whatever it wants”
often means “it’ll do more or less what the readline library does,
whether the application actually uses readline or not”, so I’ve
labelled a bunch of the codes that readline usesCtrl-X has no standard meaning in the
terminal in general but emacs uses it very heavilyThere’s no real structure to which codes are in which categories, they’re all just kind of randomly scattered because this evolved organically.
(If you’re curious about readline, I wrote more about readline in entering text in the terminal is complicated, and there are a lot of cheat sheets out there)
Something else that I find a little surprising is that are only 33 control codes –
A to Z, plus 7 more (@, [, \, ], ^, _, ?). This means that if you want to
have for example Ctrl-1 as a keyboard shortcut in a terminal application,
that’s not really meaningful – on my machine at least Ctrl-1 is exactly the
same thing as just pressing 1, Ctrl-3 is the same as Ctrl-[, etc.
Also Ctrl+Shift+C isn’t a control code – what it does depends on your
terminal emulator. On Linux Ctrl-Shift-X is often used by the terminal
emulator to copy or open a new tab or paste for example, it’s not sent to the
TTY at all.
Also I use Ctrl+Left Arrow all the time, but that isn’t a control code,
instead it sends an ANSI escape sequence (ctrl-[[1;5D) which is a different
thing which we absolutely do not have space for in this post.
This “there are only 33 codes” thing is totally different from how keyboard
shortcuts work in a GUI where you can have Ctrl+KEY for any key you want.
Each of these 33 control codes has a name in ASCII (for example 3 is ETX).
When all of these control codes were originally defined, they weren’t being
used for computers or terminals at all, they were used for the telegraph machine.
Telegraph machines aren’t the same as UNIX terminals so a lot of the codes were repurposed to mean something else.
Personally I don’t find these ASCII names very useful, because 50% of the time the name in ASCII has no actual relationship to what that code does on UNIX systems today. So it feels easier to just ignore the ASCII names completely instead of trying to figure which ones still match their original meaning.
Another thing that’s a bit weird is that Ctrl-M is literally the same as
Enter, and Ctrl-I is the same as Tab, which makes it hard to use those two as keyboard shortcuts.
From some quick research, it seems like some folks do still use Ctrl-I and
Ctrl-M as keyboard shortcuts (here’s an example), but to do that
you need to configure your terminal emulator to treat them differently than the
default.
For me the main takeaway is that if I ever write a terminal application I
should avoid Ctrl-I and Ctrl-M as keyboard shortcuts in it.
While writing this I needed to do a bunch of experimenting to figure out what various key combinations did, so I wrote this Python script echo-key.py that will print them out.
There’s probably a more official way but I appreciated having a script I could customize.
Two of these codes (Ctrl-W and Ctrl-U) are labelled in the table as
“handled by the OS”, but actually they’re not always handled by the OS, it
depends on whether the terminal is in “canonical” mode or in “noncanonical mode”.
In canonical mode,
programs only get input when you press Enter (and the OS is in charge of deleting characters when you press Backspace or Ctrl-W). But in noncanonical mode the program gets
input immediately when you press a key, and the Ctrl-W and Ctrl-U codes are passed through to the program to handle any way it wants.
Generally in noncanonical mode the program will handle Ctrl-W and Ctrl-U
similarly to how the OS does, but there are some small differences.
Some examples of programs that use canonical mode:
grep or catgit, I thinkExamples of programs that use noncanonical mode:
python3, irb and other REPLsless or vimsttyI said that Ctrl-C sends SIGINT but technically this is not necessarily
true, if you really want to you can remap all of the codes labelled “OS
terminal driver”, plus Backspace, using a tool called stty, and you can view
the mappings with stty -a.
Here are the mappings on my machine right now:
$ stty -a
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
stop = ^S; susp = ^Z; time = 0; werase = ^W;
I have personally never remapped any of these and I cannot imagine a reason I
would (I think it would be a recipe for confusion and disaster for me), but I
asked on Mastodon and people said the most common reasons they used
stty were:
stty sanestty erase ^H to change how Backspace worksstty ixoffSIGINT to a different key, like their DELETE keyTwo signals caveats:
ISIG terminal mode is turned off, then the OS won’t send signals. For example vim turns off ISIGCtrl-T) which sends SIGINFOYou can see which terminal modes a program is setting using strace like this,
terminal modes are set with the ioctl system call:
$ strace -tt -o out vim
$ grep ioctl out | grep SET
here are the modes vim sets when it starts (ISIG and ICANON are
missing!):
17:43:36.670636 ioctl(0, TCSETS, {c_iflag=IXANY|IMAXBEL|IUTF8,
c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST, c_cflag=B38400|CS8|CREAD,
c_lflag=ECHOK|ECHOCTL|ECHOKE|PENDIN, ...}) = 0
and it resets the modes when it exits:
17:43:38.027284 ioctl(0, TCSETS, {c_iflag=ICRNL|IXANY|IMAXBEL|IUTF8,
c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD,
c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE|PENDIN, ...}) = 0
I think the specific combination of modes vim is using here might be called “raw mode”, man cfmakeraw talks about that.
Related to “there are only 33 codes”, there are a lot of conflicts where
different parts of the system want to use the same code for different things,
for example by default Ctrl-S will freeze your screen, but if you turn that
off then readline will use Ctrl-S to do a forward search.
Another example is that on my machine sometimes Ctrl-T will send SIGINFO
and sometimes it’ll transpose 2 characters and sometimes it’ll do something
completely different depending on:
ISIG setreadline / imitates readline’s behaviourIn this diagram I’ve labelled code 127 as “backspace” and 8 as “other backspace”. Uh, what?
I think this was the single biggest topic of discussion in the replies on Mastodon – apparently there’s a LOT of history to this and I’d never heard of any of it before.
First, here’s how it works on my machine:
Backspace key127, which is called DEL in ASCII127 mapped to “backspace” (so it works both in canonical mode and noncanonical mode)If I press Ctrl+H, it has the same effect as Backspace if I’m using
readline, but in a program without readline support (like cat for instance),
it just prints out ^H.
Apparently Step 2 above is different for some folks – their Backspace key sends
the byte 8 instead of 127, and so if they want Backspace to work then they
need to configure the OS (using stty) to set erase = ^H.
There’s an incredible section of the Debian Policy Manual on keyboard configuration
that describes how Delete and Backspace should work according to Debian
policy, which seems very similar to how it works on my Mac today. My
understanding (via this mastodon post)
is that this policy was written in the 90s because there was a lot of confusion
about what Backspace should do in the 90s and there needed to be a standard
to get everything to work.
There’s a bunch more historical terminal stuff here but that’s all I’ll say for now.
I’ve probably missed a bunch more ways that “how it works on my machine” might be different from how it works on other people’s machines, and I’ve probably made some mistakes about how it works on my machine too. But that’s all I’ve got for today.
Some more stuff I know that I’ve left out: according to stty -a Ctrl-O is
“discard”, Ctrl-R is “reprint”, and Ctrl-Y is “dsusp”. I have no idea how
to make those actually do anything (pressing them does not do anything
obvious, and some people have told me what they used to do historically but
it’s not clear to me if they have a use in 2024), and a lot of the time in practice
they seem to just be passed through to the application anyway so I just
labelled Ctrl-R and Ctrl-Y as
readline.
Also I want to say that I think the contents of this post are kind of interesting
but I don’t think they’re necessarily that useful. I’ve used the terminal
pretty successfully every day for the last 20 years without knowing literally
any of this – I just knew what Ctrl-C, Ctrl-D, Ctrl-Z, Ctrl-R,
Ctrl-L did in practice (plus maybe Ctrl-A, Ctrl-E and Ctrl-W) and did
not worry about the details for the most part, and that was
almost always totally fine except when I was trying to use xterm.js.
But I had fun learning about it so maybe it’ll be interesting to you too.
I’ve been having problems for the last 3 years or so where Mess With DNS periodically runs out of memory and gets OOM killed.
This hasn’t been a big priority for me: usually it just goes down for a few minutes while it restarts, and it only happens once a day at most, so I’ve just been ignoring. But last week it started actually causing a problem so I decided to look into it.
This was kind of winding road where I learned a lot so here’s a table of contents:
I run Mess With DNS on a VM without about 465MB of RAM, which according to
ps aux (the RSS column) is split up something like:
That leaves about 110MB of memory free.
A while back I set GOMEMLIMIT to 250MB to try to make sure the garbage collector ran if Mess With DNS used more than 250MB of memory, and I think this helped but it didn’t solve everything.
A few weeks ago I started backing up Mess With DNS’s database for the first time using restic.
This has been working okay, but since Mess With DNS operates without much extra
memory I think restic sometimes needed more memory than was available on the
system, and so the backup script sometimes got OOM killed.
This was a problem because
There’s probably more than one solution to this, but I decided to try to make Mess With DNS use less memory so that there was more available memory on the system, mostly because it seemed like a fun problem to try to solve.
I’d run a memory profile of Mess With DNS a bunch of times in the past, so I knew exactly what was using most of Mess With DNS’s memory: IP addresses.
When it starts, Mess With DNS loads this database where you can look up the
ASN of every IP address into memory, so that when it
receives a DNS query it can take the source IP address like 74.125.16.248 and
tell you that IP address belongs to GOOGLE.
This database by itself used about 117MB of memory, and a simple du told me
that was too much – the original text files were only 37MB!
$ du -sh *.tsv
26M ip2asn-v4.tsv
11M ip2asn-v6.tsv
The way it worked originally is that I had an array of these:
type IPRange struct {
StartIP net.IP
EndIP net.IP
Num int
Name string
Country string
}
and I searched through it with a binary search to figure out if any of the ranges contained the IP I was looking for. Basically the simplest possible thing and it’s super fast, my machine can do about 9 million lookups per second.
I’ve been using SQLite recently, so my first thought was – maybe I can store all of this data on disk in an SQLite database, give the tables an index, and that’ll use less memory.
So I:
This did solve the initial memory goal (after a GC it now hardly used any memory at all because the table was on disk!), though I’m not sure how much GC churn this solution would cause if we needed to do a lot of queries at once. I did a quick memory profile and it seemed to allocate about 1KB of memory per lookup.
Let’s talk about the issues I ran into with using SQLite though.
SQLite doesn’t have support for big integers and IPv6 addresses are 128 bits,
so I decided to store them as text. I think BLOB might have been better, I
originally thought BLOBs couldn’t be compared but the sqlite docs say they can.
I ended up with this schema:
CREATE TABLE ipv4_ranges (
start_ip INTEGER NOT NULL,
end_ip INTEGER NOT NULL,
asn INTEGER NOT NULL,
country TEXT NOT NULL,
name TEXT NOT NULL
);
CREATE TABLE ipv6_ranges (
start_ip TEXT NOT NULL,
end_ip TEXT NOT NULL,
asn INTEGER,
country TEXT,
name TEXT
);
CREATE INDEX idx_ipv4_ranges_start_ip ON ipv4_ranges (start_ip);
CREATE INDEX idx_ipv6_ranges_start_ip ON ipv6_ranges (start_ip);
CREATE INDEX idx_ipv4_ranges_end_ip ON ipv4_ranges (end_ip);
CREATE INDEX idx_ipv6_ranges_end_ip ON ipv6_ranges (end_ip);
Also I learned that Python has an ipaddress module, so I could use
ipaddress.ip_address(s).exploded to make sure that the IPv6 addresses were
expanded so that a string comparison would compare them properly.
I ran a quick microbenchmark, something like this. It printed out that it could look up 17,000 IPv6 addresses per second, and similarly for IPv4 addresses.
This was pretty discouraging – being able to look up 17k addresses per section is kind of fine (Mess With DNS does not get a lot of traffic), but I compared it to the original binary search code and the original code could do 9 million per second.
ips := []net.IP{}
count := 20000
for i := 0; i < count; i++ {
// create a random IPv6 address
bytes := randomBytes()
ip := net.IP(bytes[:])
ips = append(ips, ip)
}
now := time.Now()
success := 0
for _, ip := range ips {
_, err := ranges.FindASN(ip)
if err == nil {
success++
}
}
fmt.Println(success)
elapsed := time.Since(now)
fmt.Println("number per second", float64(count)/elapsed.Seconds())
I’d never really done an EXPLAIN in sqlite, so I thought it would be a fun opportunity to see what the query plan was doing.
sqlite> explain query plan select * from ipv6_ranges where '2607:f8b0:4006:0824:0000:0000:0000:200e' BETWEEN start_ip and end_ip;
QUERY PLAN
`--SEARCH ipv6_ranges USING INDEX idx_ipv6_ranges_end_ip (end_ip>?)
It looks like it’s just using the end_ip index and not the start_ip index,
so maybe it makes sense that it’s slower than the binary search.
I tried to figure out if there was a way to make SQLite use both indexes, but I couldn’t find one and maybe it knows best anyway.
At this point I gave up on the SQLite solution, I didn’t love that it was slower and also it’s a lot more complex than just doing a binary search. I felt like I’d rather keep something much more similar to the binary search.
A few things I tried with SQLite that did not cause it to use both indexes:
ANALYZEINTERSECT to intersect the results of start_ip < ? and ? < end_ip. This did make it use both indexes, but it also seemed to make the
query literally 1000x slower, probably because it needed to create the
results of both subqueries in memory and intersect them.My next idea was to use a trie, because I had some vague idea that maybe a trie would use less memory, and I found this library called ipaddress-go that lets you look up IP addresses using a trie.
I tried using it here’s the code, but I think I was doing something wildly wrong because, compared to my naive array + binary search:
I’m not really sure what went wrong here but I gave up on this approach and decided to just try to make my array use less memory and stick to a simple binary search.
One thing I learned about memory profiling is that you can use runtime
package to see how much memory is currently allocated in the program. That’s
how I got all the memory numbers in this post. Here’s the code:
func memusage() {
runtime.GC()
var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf("Alloc = %v MiB\n", m.Alloc/1024/1024)
// write mem.prof
f, err := os.Create("mem.prof")
if err != nil {
log.Fatal(err)
}
pprof.WriteHeapProfile(f)
f.Close()
}
Also I learned that if you use pprof to analyze a heap profile there are two
ways to analyze it: you can pass either --alloc-space or --inuse-space to
go tool pprof. I don’t know how I didn’t realize this before but
alloc-space will tell you about everything that was allocated, and
inuse-space will just include memory that’s currently in use.
Anyway I ran go tool pprof -pdf --inuse_space mem.prof > mem.pdf a lot. Also
every time I use pprof I find myself referring to my own intro to pprof, it’s probably
the blog post I wrote that I use the most often. I should add --alloc-space
and --inuse-space to it.
I was storing my ip2asn entries like this:
type IPRange struct {
StartIP net.IP
EndIP net.IP
Num int
Name string
Country string
}
I had 3 ideas for ways to improve this:
Name and the Country, because a lot of IP ranges belong to the same ASNnet.IP is an []byte under the hood, which felt like it involved an unnecessary pointer, was there a way to inline it into the struct?I figured I could store the ASN info in an array, and then just store the index
into the array in my IPRange struct. Here are the structs so you can see what
I mean:
type IPRange struct {
StartIP netip.Addr
EndIP netip.Addr
ASN uint32
Idx uint32
}
type ASNInfo struct {
Country string
Name string
}
type ASNPool struct {
asns []ASNInfo
lookup map[ASNInfo]uint32
}
This worked! It brought memory usage from 117MB to 65MB – a 50MB savings. I felt good about this.
Here’s all of the code for that part.
As an aside – I’m storing the ASN in a uint32, is that right? I looked in the ip2asn
file and the biggest one seems to be 401307, though there are a few lines that
say 4294901931 which is much bigger, but also are just inside the range of a
uint32. So I can definitely use a uint32.
59.101.179.0 59.101.179.255 4294901931 Unknown AS4294901931
netip.Addr instead of net.IPIt turns out that I’m not the only one who felt that net.IP was using an
unnecessary amount of memory – in 2021 the folks at Tailscale released a new
IP address library for Go which solves this and many other issues. They wrote a great blog post about it.
I discovered (to my delight) that not only does this new IP address library exist and do exactly what I want, it’s also now in the Go
standard library as netip.Addr. Switching to netip.Addr was
very easy and saved another 20MB of memory, bringing us to 46MB.
I didn’t try my third idea (remove the end IP from the struct) because I’d already been programming for long enough on a Saturday morning and I was happy with my progress.
It’s always such a great feeling when I think “hey, I don’t like this, there must be a better way” and then immediately discover that someone has already made the exact thing I want, thought about it a lot more than me, and implemented it much better than I would have.
Even though I tried to explain this in a simple linear way “I tried X, then I tried Y, then I tried Z”, that’s kind of a lie – I always try to take my actual debugging process (total chaos) and make it seem more linear and understandable because the reality is just too annoying to write down. It’s more like:
runtime to check how much
memory everything is using, start doing thatSomeone asked why I don’t just give the VM more memory. I could very easily afford to pay for a VM with 1GB of memory, but I feel like 512MB really should be enough (and really that 256MB should be enough!) so I’d rather stay inside that constraint. It’s kind of a fun puzzle.
Folks had a lot of good ideas I hadn’t thought of. Recording them as inspiration if I feel like having another Fun Performance Day at some point.
ASNPool. Someone tried this and it uses more memory, probably because Go’s pointers are 64 bitsGOARCH=386 to use 32-bit pointers to sace space (maybe in combination with using unique!)I deployed the new version and now Mess With DNS is using less memory! Hooray!
A few other notes:
I’m honestly not sure if this will solve all my memory problems, probably not! But I had fun, I learned a few things about SQLite, I still don’t know what to think about tries, and it made me love binary search even more than I already did.
Warning: this is a post about very boring yakshaving, probably only of interest to people who are trying to upgrade Hugo from a very old version to a new version. But what are blogs for if not documenting one’s very boring yakshaves from time to time?
So yesterday I decided to try to upgrade Hugo. There’s no real reason to do this – I’ve been using Hugo version 0.40 to generate this blog since 2018, it works fine, and I don’t have any problems with it. But I thought – maybe it won’t be as hard as I think, and I kind of like a tedious computer task sometimes!
I thought I’d document what I learned along the way in case it’s useful to anyone else doing this very specific migration. I upgraded from Hugo v0.40 (from 2018) to v0.135 (from 2024).
Here are most of the changes I had to make:
template "theme/partials/thing.html is now partial thing.htmlI had to replace a bunch of instances of {{ template "theme/partials/header.html" . }} with {{ partial "header.html" . }}.
This happened in v0.42:
We have now virtualized the filesystems for project and theme files. This makes everything simpler, faster and more powerful. But it also means that template lookups on the form {{ template “theme/partials/pagination.html” . }} will not work anymore. That syntax has never been documented, so it’s not expected to be in wide use.
.Data.Pages is now site.RegularPagesThis seems to be discussed in the release notes for 0.57.2
I just needed to replace .Data.Pages with site.RegularPages in the template on the homepage as well as in my RSS feed template.
.Next and .Prev got flippedI had this comment in the part of my theme where I link to the next/previous blog post:
“next” and “previous” in hugo apparently mean the opposite of what I’d think they’d mean intuitively. I’d expect “next” to mean “in the future” and “previous” to mean “in the past” but it’s the opposite
It looks they changed this in ad705aac064 so that “next” actually is in the future and “prev” actually is in the past. I definitely find the new behaviour more intuitive.
Figuring out why/when all of these changes happened was a little difficult. I ended up hacking together a bash script to download all of the changelogs from github as text files, which I could then grep to try to figure out what happened. It turns out it’s pretty easy to get all of the changelogs from the GitHub API.
So far everything was not so bad – there was also a change around taxonomies that’s I can’t quite explain, but it was all pretty manageable, but then we got to the really tough one: the markdown renderer.
The blackfriday markdown renderer (which was previously the default) was removed in v0.100.0. This seems pretty reasonable:
It has been deprecated for a long time, its v1 version is not maintained anymore, and there are many known issues. Goldmark should be a mature replacement by now.
Fixing all my Markdown changes was a huge pain – I ended up having to update 80 different Markdown files (out of 700) so that they would render properly, and I’m not totally sure
The obvious question here is – why bother even trying to upgrade Hugo at all if I have to switch Markdown renderers? My old site was running totally fine and I think it wasn’t necessarily a good use of time, but the one reason I think it might be useful in the future is that the new renderer (goldmark) uses the CommonMark markdown standard, which I’m hoping will be somewhat more futureproof. So maybe I won’t have to go through this again? We’ll see.
Also it turned out that the new Goldmark renderer does fix some problems I had (but didn’t know that I had) with smart quotes and how lists/blockquotes interact.
The hard part of this Markdown change was even figuring out what changed. Almost all of the problems (including #2 and #3 above) just silently broke the site, they didn’t cause any errors or anything. So I had to diff the HTML to hunt them down.
Here’s what I ended up doing:
public_oldpublicpublic/ and public_old with this diff.sh script and put the results in a diffs/ folderfind diffs -type f | xargs cat | grep -C 5 '(31m|32m)' | less -r over and over again to look at every single change until I found something that seemed wrong(the grep 31m|32m thing is searching for red/green text in the diff)
This was very time consuming but it was a little bit fun for some reason so I kept doing it until it seemed like nothing too horrible was left.
Here’s a list of every type of Markdown change I had to make. It’s very possible these are all extremely specific to me but it took me a long time to figure them all out so maybe this will be helpful to one other person who finds this in the future.
This doesn’t work anymore (it doesn’t expand the link):
<small>
[a link](https://example.com)
</small>
I need to do this instead:
<small>
[a link](https://example.com)
</small>
This works too:
<small> [a link](https://example.com) </small>
<< is changed into «I didn’t want this so I needed to configure:
markup:
goldmark:
extensions:
typographer:
leftAngleQuote: '<<'
rightAngleQuote: '>>'
This doesn’t render as a nested list anymore if I only indent by 2 spaces, I need to put 4 spaces.
1. a
* b
* c
2. b
The problem is that the amount of indent needed depends on the size of the list markers. Here’s a reference in CommonMark for this.
Previously the > quote here didn’t render as a blockquote, and with the new renderer it does.
* something
> quote
* something else
I found a bunch of Markdown that had been kind of broken (which I hadn’t noticed) that works better with the new renderer, and this is an example of that.
Lists inside blockquotes also seem to work better.
Previously this didn’t render as a heading, but now it does. So I needed to
replace the # with #.
* # passengers: 20
+ or 1) at the beginning of the line makes it a listI had something which looked like this:
`1 / (1
+ exp(-1)) = 0.73`
With Blackfriday it rendered like this:
<p><code>1 / (1
+ exp(-1)) = 0.73</code></p>
and with Goldmark it rendered like this:
<p>`1 / (1</p>
<ul>
<li>exp(-1)) = 0.73`</li>
</ul>
Same thing if there was an accidental 1) at the beginning of a line, like in this Markdown snippet
I set up a small Hadoop cluster (1 master, 2 workers, replication set to
1) on
To fix this I just had to rewrap the line so that the + wasn’t the first character.
The Markdown is formatted this way because I wrap my Markdown to 80 characters a lot and the wrapping isn’t very context sensitive.
There were a bunch of places where the old renderer (Blackfriday) was doing
unwanted things in code blocks like replacing ... with … or replacing
quotes with smart quotes. I hadn’t realized this was happening and I was very
happy to have it fixed.
The way this gets rendered got better:
"Oh, *interesting*!"
Before there were two left smart quotes, now the quotes match.
p tagPreviously if I had an image like this:
<img src="https://jvns.ca/images/rustboot1.png">
it would get wrapped in a <p> tag, now it doesn’t anymore. I dealt with this
just by adding a margin-bottom: 0.75em to images in the CSS, hopefully
that’ll make them display well enough.
<br> is now wrapped in a p tagPreviously this wouldn’t get wrapped in a p tag, but now it seems to:
<br><br>
I just gave up on fixing this though and resigned myself to maybe having some extra space in some cases. Maybe I’ll try to fix it later if I feel like another yakshave.
I also needed to
Here’s what I needed to add to my config.yaml to do all that:
markup:
highlight:
codeFences: false
goldmark:
renderer:
unsafe: true
parser:
autoHeadingIDType: blackfriday
Maybe I’ll try to get syntax highlighting working one day, who knows. I might prefer having it off though.
I also wrote a little program to compare the Blackfriday and Goldmark output for various markdown snippets, here it is in a gist.
It’s not really configured the exact same way Blackfriday and Goldmark were in my Hugo versions, but it was still helpful to have to help me understand what was going on.
My approach to themes in Hugo has been:
So I just need to edit the theme files to fix any problems. Also I wrote a lot of the theme myself so I’m pretty familiar with how it works.
Relying on someone else to keep a theme updated feels kind of scary to me, I think if I were using a third-party theme I’d just copy the code into my site’s github repo and then maintain it myself.
I asked on Mastodon if anyone had used a static site generator with good backwards compatibility.
The main answers seemed to be Jekyll and 11ty. Several people said they’d been using Jekyll for 10 years without any issues, and 11ty says it has stability as a core goal.
I think a big factor in how appealing Jekyll/11ty are is how easy it is for you to maintain a working Ruby / Node environment on your computer: part of the reason I stopped using Jekyll was that I got tired of having to maintain a working Ruby installation. But I imagine this wouldn’t be a problem for a Ruby or Node developer.
Several people said that they don’t build their Jekyll site locally at all – they just use GitHub Pages to build it.
Overall I’ve been happy with Hugo – I started using it because it had fast build times and it was a static binary, and both of those things are still extremely useful to me. I might have spent 10 hours on this upgrade, but I’ve probably spent 1000+ hours writing blog posts without thinking about Hugo at all so that seems like an extremely reasonable ratio.
I find it hard to be too mad about the backwards incompatible changes, most of
them were quite a long time ago, Hugo does a great job of making their old
releases available so you can use the old release if you want, and the most
difficult one is removing support for the blackfriday Markdown renderer in
favour of using something CommonMark-compliant which seems pretty reasonable to
me even if it is a huge pain.
But it did take a long time and I don’t think I’d particularly recommend moving 700 blog posts to a new Markdown renderer unless you’re really in the mood for a lot of computer suffering for some reason.
The new renderer did fix a bunch of problems so I think overall it might be a good thing, even if I’ll have to remember to make 2 changes to how I write Markdown (4.1 and 4.3).
Also I’m still using Hugo 0.54 for https://wizardzines.com so maybe these notes will be useful to Future Me if I ever feel like upgrading Hugo for that site.
Hopefully I didn’t break too many things on the blog by doing this, let me know if you see anything broken!
Yesterday I was thinking about how long it took me to get a colorscheme in my terminal that I was mostly happy with (SO MANY YEARS), and it made me wonder what about terminal colours made it so hard.
So I asked people on Mastodon what problems they’ve run into with colours in the terminal, and I got a ton of interesting responses! Let’s talk about some of the problems and a few possible ways to fix them.
One of the top complaints was “blue on black is hard to read”. Here’s an
example of that: if I open Terminal.app, set the background to black, and run
ls, the directories are displayed in a blue that isn’t that easy to read:
To understand why we’re seeing this blue, let’s talk about ANSI colours!
Your terminal has 16 numbered colours – black, red, green, yellow, blue, magenta, cyan, white, and “bright” version of each of those.
Programs can use them by printing out an “ANSI escape code” – for example if you want to see each of the 16 colours in your terminal, you can run this Python program:
def color(num, text):
return f"\033[38;5;{num}m{text}\033[0m"
for i in range(16):
print(color(i, f"number {i:02}"))
This made me wonder – if blue is colour number 5, who decides what hex color that should correspond to?
The answer seems to be “there’s no standard, terminal emulators just choose colours and it’s not very consistent”. Here’s a screenshot of a table from Wikipedia, where you can see that there’s a lot of variation:
Bright yellow on white is even worse than blue on black, here’s what I get in a terminal with the default settings:
That’s almost impossible to read (and some other colours like light green cause similar issues), so let’s talk about solutions!
If you’re annoyed by these colour contrast issues (or maybe you just think the default ANSI colours are ugly), you might think – well, I’ll just choose a different “blue” and pick something I like better!
There are two ways you can do this:
Way 1: Configure your terminal emulator: I think most modern terminal emulators have a way to reconfigure the colours, and some of them even come with some preinstalled themes that you might like better than the defaults.
Way 2: Run a shell script: There are ANSI escape codes that you can print
out to tell your terminal emulator to reconfigure its colours. Here’s a shell script that does that,
from the base16-shell project.
You can see that it has a few different conventions for changing the colours –
I guess different terminal emulators have different escape codes for changing
their colour palette, and so the script is trying to pick the right style of
escape code based on the TERM environment variable.
I prefer to use the “shell script” method, because:
some advantages of configuring colours in your terminal emulator:
This is what my shell has looked like for probably the last 5 years (using the
solarized light base16 theme), and I’m pretty happy with it. Here’s htop:
Okay, so let’s say you’ve found a terminal colorscheme that you like. What else can go wrong?
Here’s what some output of fd, a find alternative, looks like in my
colorscheme:
The contrast is pretty bad here, and I definitely don’t have that lime green in my normal colorscheme. What’s going on?
We can see what color codes fd is using using the unbuffer program to
capture its output including the color codes:
$ unbuffer fd . > out
$ vim out
^[[38;5;48mbad-again.sh^[[0m
^[[38;5;48mbad.sh^[[0m
^[[38;5;48mbetter.sh^[[0m
out
^[[38;5;48 means “set the foreground color to color 48”. Terminals don’t
only have 16 colours – many terminals these days actually have 3 ways of
specifying colours:
#ffea03So fd is using one of the colours from the extended 256-color set. bat (a
cat alternative) does something similar – here’s what it looks like by
default in my terminal.
This looks fine though and it really seems like it’s trying to work well with a variety of terminal themes.
I think it’s interesting that some of these newer terminal tools (fd, cat,
delta, and probably more) have support for arbitrary custom themes. I guess
the downside of this approach is that the default theme might clash with your
terminal’s background, but the upside is that it gives you a lot more control
over theming the tool’s output than just choosing 16 ANSI colours.
I don’t really use bat, but if I did I’d probably use bat --theme ansi to
just use the ANSI colours that I have set in my normal terminal colorscheme.
A bunch of people on Mastodon mentioned a specific issue with grays in the Solarized theme: when I list a directory, the base16 Solarized Light theme looks like this:
but iTerm’s default Solarized Light theme looks like this:
This is because in the iTerm theme (which is the original Solarized design), colors 9-14 (the “bright blue”, “bright
red”, etc) are mapped to a series of grays, and when I run ls, it’s trying to
use those “bright” colours to color my directories and executables.
My best guess for why the original Solarized theme is designed this way is to make the grays available to the vim Solarized colorscheme.
I’m pretty sure I prefer the modified base16 version I use where the “bright” colours are actually colours instead of all being shades of gray though. (I didn’t actually realize the version I was using wasn’t the “original” Solarized theme until I wrote this post)
In any case I really love Solarized and I’m very happy it exists so that I can use a modified version of it.
If I my vim theme has a different background colour than my terminal theme, I get this ugly border, like this:
This one is a pretty minor issue though and I think making your terminal background match your vim background is pretty straightforward.
A few people mentioned problems with terminal applications setting an unwanted background colour, so let’s look at an example of that.
Here ngrok has set the background to color #16 (“black”), but the
base16-shell script I use sets color 16 to be bright orange, so I get this,
which is pretty bad:
I think the intention is for ngrok to look something like this:
I think base16-shell sets color #16 to orange (instead of black)
so that it can provide extra colours for use by base16-vim.
This feels reasonable to me – I use base16-vim in the terminal, so I guess I’m
using that feature and it’s probably more important to me than ngrok (which I
rarely use) behaving a bit weirdly.
This particular issue is a maybe obscure clash between ngrok and my colorschem, but I think this kind of clash is pretty common when a program sets an ANSI background color that the user has remapped for some reason.
A bunch of terminals (iTerm2, tabby, kitty’s text_fg_override_threshold, and folks tell me also Ghostty and Windows Terminal) have a “minimum contrast” feature that will automatically adjust colours to make sure they have enough contrast.
Here’s an example from iTerm. This ngrok accident from before has pretty bad contrast, I find it pretty difficult to read:
With “minimum contrast” set to 40 in iTerm, it looks like this instead:
I didn’t have minimum contrast turned on before but I just turned it on today because it makes such a big difference when something goes wrong with colours in the terminal.
TERM being set to the wrong thingA few people mentioned that they’ll SSH into a system that doesn’t support the
TERM environment variable that they have set locally, and then the colours
won’t work.
I think the way TERM works is that systems have a terminfo database, so if
the value of the TERM environment variable isn’t in the system’s terminfo
database, then it won’t know how to output colours for that terminal. I don’t
know too much about terminfo, but someone linked me to this terminfo rant that talks about a few other
issues with terminfo.
I don’t have a system on hand to reproduce this one so I can’t say for sure how
to fix it, but this stackoverflow question
suggests running something like TERM=xterm ssh instead of ssh.
A couple of problems people mentioned with designing / finding terminal colorschemes:
Another problem people mentioned is using a program like nethack or midnight commander which you might expect to have a specific colourscheme based on the default ANSI terminal colours.
For example, midnight commander has a really specific classic look:
But in my Solarized theme, midnight commander looks like this:
The Solarized version feels like it could be disorienting if you’re very used to the “classic” look.
One solution Simon Tatham mentioned to this is using some palette customization ANSI codes (like the ones base16 uses that I talked about earlier) to change the color palette right before starting the program, for example remapping yellow to a brighter yellow before starting Nethack so that the yellow characters look better.
If I run fd | less, I see something like this, with the colours disabled.
In general I find this useful – if I pipe a command to grep, I don’t want it
to print out all those color escape codes, I just want the plain text. But what if you want to see the colours?
To see the colours, you can run unbuffer fd | less -r! I just learned about
unbuffer recently and I think it’s really cool, unbuffer opens a tty for the
command to write to so that it thinks it’s writing to a TTY. It also fixes
issues with programs buffering their output when writing to a pipe, which is
why it’s called unbuffer.
Here’s what the output of unbuffer fd | less -r looks like for me:
Also some commands (including fd) support a --color=always flag which will
force them to always print out the colours.
ls and other commandsSome people mentioned that they don’t want ls to use colour at all, perhaps
because ls uses blue, it’s hard to read on black, and maybe they don’t feel like
customizing their terminal’s colourscheme to make the blue more readable or
just don’t find the use of colour helpful.
Some possible solutions to this one:
ls --color=never, which is probably easiestLS_COLORS to customize the colours used by ls. I think some other programs other than ls support the LS_COLORS environment variable too.NO_COLOR=true (there’s a list here)Here’s an example of running LS_COLORS="fi=0:di=0:ln=0:pi=0:so=0:bd=0:cd=0:or=0:ex=0" ls:
I used to have a lot of problems with configuring my colours in vim – I’d set up my terminal colours in a way that I thought was okay, and then I’d start vim and it would just be a disaster.
I think what was going on here is that today, there are two ways to set up a vim colorscheme in the terminal:
20 years ago when I started using vim, terminals with 24-bit hex color support were a lot less common (or maybe they didn’t exist at all), and vim certainly didn’t have support for using 24-bit colour in the terminal. From some quick searching through git, it looks like vim added support for 24-bit colour in 2016 – just 8 years ago!
So to get colours to work properly in vim before 2016, you needed to synchronize
your terminal colorscheme and your vim colorscheme. Here’s what that looked like,
the colorscheme needed to map the vim color classes like cterm05 to ANSI colour numbers.
But in 2024, the story is really different! Vim (and Neovim, which I use now)
support 24-bit colours, and as of Neovim 0.10 (released in May 2024), the
termguicolors setting (which tells Vim to use 24-bit hex colours for
colorschemes) is turned on by default in any terminal with 24-bit
color support.
So this “you need to synchronize your terminal colorscheme and your vim colorscheme” problem is not an issue anymore for me in 2024, since I don’t plan to use terminals without 24-bit color support in the future.
The biggest consequence for me of this whole thing is that I don’t need base16
to set colors 16-21 to weird stuff anymore to integrate with vim – I can just
use a terminal theme and a vim theme, and as long as the two themes use similar
colours (so it’s not jarring for me to switch between them) there’s no problem.
I think I can just remove those parts from my base16 shell script and totally
avoid the problem with ngrok and the weird orange background I talked about
above.
I think there are a lot of issues around the intersection of multiple programs, like using some combination tmux/ssh/vim that I couldn’t figure out how to reproduce well enough to talk about them. Also I’m sure I missed a lot of other things too.
I’ve personally had a lot of success with using
base16-shell with
base16-vim – I just need to add a couple of lines to my
fish config to set it up (+ a few .vimrc lines) and then I can move on and
accept any remaining problems that that doesn’t solve.
I don’t think base16 is for everyone though, some limitations I’m aware of with base16 that might make it not work for you:
base16-vim
access to more colours, which might not be relevant if you always use a
terminal with 24-bit color support, and can cause problems like the ngrok
issue aboveApparently there’s a community fork of base16 called tinted-theming, which I haven’t looked into much yet.
Just one so far but I’ll link more if people tell me about them:
We talked about a lot in this post and while I think learning about all these details is kind of fun if I’m in the mood to do a deep dive, I find it SO FRUSTRATING to deal with it when I just want my colours to work! Being surprised by unreadable text and having to find a workaround is just not my idea of a good day.
Personally I’m a zero-configuration kind of person and it’s not that appealing to me to have to put together a lot of custom configuration just to make my colours in the terminal look acceptable. I’d much rather just have some reasonable defaults that I don’t have to change.
My one big takeaway from writing this was to turn on “minimum contrast” in my terminal, I think it’s going to fix most of the occasional accidental unreadable text issues I run into and I’m pretty excited about it.
I spent a lot of time in the past couple of weeks working on a website in Go that may or may not ever see the light of day, but I learned a couple of things along the way I wanted to write down. Here they are:
I’ve never felt motivated to learn any of the Go routing libraries (gorilla/mux, chi, etc), so I’ve been doing all my routing by hand, like this.
// DELETE /records:
case r.Method == "DELETE" && n == 1 && p[0] == "records":
if !requireLogin(username, r.URL.Path, r, w) {
return
}
deleteAllRecords(ctx, username, rs, w, r)
// POST /records/<ID>
case r.Method == "POST" && n == 2 && p[0] == "records" && len(p[1]) > 0:
if !requireLogin(username, r.URL.Path, r, w) {
return
}
updateRecord(ctx, username, p[1], rs, w, r)
But apparently as of Go 1.22, Go now has better support for routing in the standard library, so that code can be rewritten something like this:
mux.HandleFunc("DELETE /records/", app.deleteAllRecords)
mux.HandleFunc("POST /records/{record_id}", app.updateRecord)
Though it would also need a login middleware, so maybe something more like
this, with a requireLogin middleware.
mux.Handle("DELETE /records/", requireLogin(http.HandlerFunc(app.deleteAllRecords)))
One annoying gotcha I ran into was: if I make a route for /records/, then a
request for /records will be redirected to /records/.
I ran into an issue with this where sending a POST request to /records
redirected to a GET request for /records/, which broke the POST request
because it removed the request body. Thankfully Xe Iaso wrote a blog post about the exact same issue which made it
easier to debug.
I think the solution to this is just to use API endpoints like POST /records
instead of POST /records/, which seems like a more normal design anyway.
I got a little bit tired of writing so much boilerplate for my SQL queries, but I didn’t really feel like learning an ORM, because I know what SQL queries I want to write, and I didn’t feel like learning the ORM’s conventions for translating things into SQL queries.
But then I found sqlc, which will compile a query like this:
-- name: GetVariant :one
SELECT *
FROM variants
WHERE id = ?;
into Go code like this:
const getVariant = `-- name: GetVariant :one
SELECT id, created_at, updated_at, disabled, product_name, variant_name
FROM variants
WHERE id = ?
`
func (q *Queries) GetVariant(ctx context.Context, id int64) (Variant, error) {
row := q.db.QueryRowContext(ctx, getVariant, id)
var i Variant
err := row.Scan(
&i.ID,
&i.CreatedAt,
&i.UpdatedAt,
&i.Disabled,
&i.ProductName,
&i.VariantName,
)
return i, err
}
What I like about this is that if I’m ever unsure about what Go code to write for a given SQL query, I can just write the query I want, read the generated function and it’ll tell me exactly what to do to call it. It feels much easier to me than trying to dig through the ORM’s documentation to figure out how to construct the SQL query I want.
Reading Brandur’s sqlc notes from 2024 also gave me some confidence that this is a workable path for my tiny programs. That post gives a really helpful example of how to conditionally update fields in a table using CASE statements (for example if you have a table with 20 columns and you only want to update 3 of them).
Someone on Mastodon linked me to this post called Optimizing sqlite for servers. My projects are small and I’m not so concerned about performance, but my main takeaways were:
db.SetMaxOpenConns(1) on it. I learned the hard way that if I don’t do this
then I’ll get SQLITE_BUSY errors from two threads trying to write to the db
at the same time.There are a more tips in that post that seem useful (like “COUNT queries are slow” and “Use STRICT tables”), but I haven’t done those yet.
Also sometimes if I have two tables where I know I’ll never need to do a JOIN
beteween them, I’ll just put them in separate databases so that I can connect
to them independently.
I run all of my Go projects in VMs with relatively little memory, like 256MB or 512MB. I ran into an issue where my application kept getting OOM killed and it was confusing – did I have a memory leak? What?
After some Googling, I realized that maybe I didn’t have a memory leak, maybe I just needed to reconfigure the garbage collector! It turns out that by default (according to A Guide to the Go Garbage Collector), Go’s garbage collector will let the application allocate memory up to 2x the current heap size.
Mess With DNS’s base heap size is around 170MB and the amount of memory free on the VM is around 160MB right now, so if its memory doubled, it’ll get OOM killed.
In Go 1.19, they added a way to tell Go “hey, if the application starts using this much memory, run a GC”. So I set the GC memory limit to 250MB and it seems to have resulted in the application getting OOM killed less often:
export GOMEMLIMIT=250MiB
I’ve been making tiny websites (like the nginx playground) in Go on and off for the last 4 years or so and it’s really been working for me. I think I like it because:
apt-get install golang-go or whatever and then a go build will build my projectServe(w http.ResponseWriter, r *http.Request) which read the request and send a response. If I need to
remember some detail of how exactly that’s accomplished, I just have to read
the function!net/http is in the standard library, so you can start making websites
without installing any libraries at all. I really appreciate this one.ioctl or
something that’s easy to doIn general everything about it feels like it makes projects easy to work on for 5 days, abandon for 2 years, and then get back into writing code without a lot of problems.
For contrast, I’ve tried to learn Rails a couple of times and I really want to love Rails – I’ve made a couple of toy websites in Rails and it’s always felt like a really magical experience. But ultimately when I come back to those projects I can’t remember how anything works and I just end up giving up. It feels easier to me to come back to my Go projects that are full of a lot of repetitive boilerplate, because at least I can read the code and figure out how it works.
some things I haven’t done much of yet in Go:
html/template a lot in Hugo (which I’ve used for this blog for the last 8 years)
but I’m still not sure how I feel about it.In general I’m not sure how to implement security-sensitive features so I don’t start projects which need login/CSRF/etc. I imagine this is where a framework would help.
Both of the Go features I mentioned in this post (GOMEMLIMIT and the routing)
are new in the last couple of years and I didn’t notice when they came out. It
makes me think I should pay closer attention to the release notes for new Go
versions.
I wrote about how much I love fish in this blog post from 2017 and, 7 years of using it every day later, I’ve found even more reasons to love it. So I thought I’d write a new post with both the old reasons I loved it and some reasons.
This came up today because I was trying to figure out why my terminal doesn’t break anymore when I cat a binary to my terminal, the answer was “fish fixes the terminal!”, and I just thought that was really nice.
In 10 years of using fish I have never found a single thing I wanted to configure. It just works the way I want. My fish config file just has:
alias ls eza, alias vim nvim, etc)direnv hook fish | source to integrate a tool like direnvI’ve been told that configuring things in fish is really easy if you ever do want to configure something though.
My absolute favourite thing about fish is that I type, it’ll automatically suggest (in light grey) a matching command that I ran recently. I can press the right arrow key to accept the completion, or keep typing to ignore it.
Here’s what that looks like. In this example I just typed the “v” key and it guessed that I want to run the previous vim command again.
One of my favourite subtle autocomplete features is how fish handles autocompleting commands that contain paths in them. For example, if I run:
$ ls blah.txt
that command will only be autocompleted in directories that contain blah.txt – it won’t show up in a different directory. (here’s a short comment about how it works)
As an example, if in this directory I type bash scripts/, it’ll only suggest
history commands including files that actually exist in my blog’s scripts
folder, and not the dozens of other irrelevant scripts/ commands I’ve run in
other folders.
I didn’t understand exactly how this worked until last week, it just felt like fish was magically able to suggest the right commands. It still feels a little like magic and I love it.
If I copy and paste multiple lines, bash will run them all, like this:
[bork@grapefruit linux-playground (main)]$ echo hi
hi
[bork@grapefruit linux-playground (main)]$ touch blah
[bork@grapefruit linux-playground (main)]$ echo hi
hi
This is a bit alarming – what if I didn’t actually want to run all those commands?
Fish will paste them all at a single prompt, so that I can press Enter if I actually want to run them. Much less scary.
bork@grapefruit ~/work/> echo hi
touch blah
echo hi
If I run ls and press tab, it’ll display all the filenames in a nice grid. I can use either Tab, Shift+Tab, or the arrow keys to navigate the grid.
Also, I can tab complete from the middle of a filename – if the filename starts with a weird character (or if it’s just not very unique), I can type some characters from the middle and press tab.
Here’s what the tab completion looks like:
bork@grapefruit ~/work/> ls
api/ blah.py fly.toml README.md
blah Dockerfile frontend/ test_websocket.sh
I honestly don’t complete things other than filenames very much so I can’t speak to that, but I’ve found the experience of tab completing filenames to be very good.
Fish’s default prompt includes everything I want:
Here’s a screenshot with a few different variations on the default prompt,
including if the last command was interrupted (the SIGINT) or failed.
In bash, the maximum history size is 500 by default, presumably because computers used to be slow and not have a lot of disk space. Also, by default, commands don’t get added to your history until you end your session. So if your computer crashes, you lose some history.
In fish:
I’m not sure how clearly I’m explaining how fish’s history system works here, but it feels really good to me in practice. My impression is that the way it’s implemented is the commands are continually added to the history file, but fish only loads the history file once, on startup.
I’ll mention here that if you want to have a fancier history system in another shell it might be worth checking out atuin or fzf.
I also like fish’s interface for searching history: for example if I want to edit my fish config file, I can just type:
$ config.fish
and then press the up arrow to go back the last command that included config.fish. That’ll complete to:
$ vim ~/.config/fish/config.fish
and I’m done. This isn’t so different from using Ctrl+R in bash to search
your history but I think I like it a little better over all, maybe because
Ctrl+R has some behaviours that I find confusing (for example you can
end up accidentally editing your history which I don’t like).
I used to run into issues with bash where I’d accidentally cat a binary to
the terminal, and it would break the terminal.
Every time fish displays a prompt, it’ll try to fix up your terminal so that you don’t end up in weird situations like this. I think this is some of the code in fish to prevent broken terminals.
Some things that it does are:
echo so that you can see the characters you typeI don’t think I’ve run into any of these “my terminal is broken” issues in a very long time, and I actually didn’t even realize that this was because of fish – I thought that things somehow magically just got better, or maybe I wasn’t making as many mistakes. But I think it was mostly fish saving me from myself, and I really appreciate that.
Also related to terminals breaking: fish disables Ctrl+S (which freezes your terminal and then you need to remember to press Ctrl+Q to unfreeze it). It’s a feature that I’ve never wanted and I’m happy to not have it.
Apparently you can disable Ctrl+S in other shells with stty -ixon.
By default commands that don’t exist are highlighted in red, like this.
I find the loop syntax in fish a lot easier to type than the bash syntax. It looks like this:
for i in *.yaml
echo $i
end
Also it’ll add indentation in your loops which is nice.
Related to loops: you can edit multiline commands much more easily than in bash (just use the arrow keys to navigate the multiline command!). Also when you use the up arrow to get a multiline command from your history, it’ll show you the whole command the exact same way you typed it instead of squishing it all onto one line like bash does:
$ bash
$ for i in *.png
> do
> echo $i
> done
$ # press up arrow
$ for i in *.png; do echo $i; done ink
This might just be me, but I really appreciate that fish has the Ctrl+left arrow / Ctrl+right arrow keyboard shortcut for moving between
words when writing a command.
I’m honestly a bit confused about where this keyboard shortcut is coming from
(the only documented keyboard shortcut for this I can find in fish is Alt+left arrow / Alt + right arrow which seems to do the same thing), but I’m pretty
sure this is a fish shortcut.
A couple of notes about getting this shortcut to work / where it comes from:
Ctrl+left arrow switches workspaces by default, so I had to turn
that off./etc/inputrc to make
Ctrl+left/right arrow go back/forward a word, so it’ll work in bash on
Ubuntu and maybe other Linux distros too. Here’s a stack overflow question talking about thatSometimes tools don’t have instructions for integrating them with fish. That’s annoying, but:
bash or zshMy biggest day-to-day to annoyance is probably that for whatever reason I’m
still not used to fish’s syntax for setting environment variables, I get confused
about set vs set -x.
fish_add_pathfish has a function called fish_add_path that you can run to add a directory
to your PATH like this:
fish_add_path /some/directory
I love the idea of it and I used to use it all the time, but I’ve stopped using it for two reasons:
fish_add_path will update the PATH for every session in the
future (with a “universal variable”) and sometimes it will update the PATH
just for the current session. It’s hard for me to tell which one it will
do: in theory the docs explain this but I could not understand them.PATH a few weeks or
months later because maybe you made a mistake, that’s also kind of hard to do
(there are instructions in this comments of this github issue though).Instead I just update my PATH like this, similarly to how I’d do it in bash:
set PATH $PATH /some/directory/bin
When I started using fish, you couldn’t do things like cmd1 && cmd2 – it
would complain “no, you need to run cmd1; and cmd2” instead.
It seems like over the years fish has started accepting a little more POSIX-style syntax than it used to, like:
cmd1 && cmd2export a=b to set an environment variable (though this seems a bit limited, you can’t do export PATH=$PATH:/whatever so I think it’s probably better to learn set instead)Changing my default shell to fish is always a little annoying, I occasionally get myself into a situation where
/home/bork/.nix-stuff/bin/fish/etc/shells as an allowed shellchshThis has never been a major issue because I always have a terminal open somewhere where I can fix the problem and rescue myself, but it’s a bit alarming.
If you don’t want to use chsh to change your shell to fish (which is very reasonable,
maybe I shouldn’t be doing that), the Arch wiki page has a couple of good suggestions –
either configure your terminal emulator to run fish or add an exec fish to
your .bashrc.
Other than occasionally writing a for loop interactively on the command line, I’ve never really learned the fish scripting language. I still do all of my shell scripting in bash.
I don’t think I’ve ever written a fish function or if statement.
I ran a highly unscientific poll on Mastodon asking people what shell they use interactively. The results were (of 2600 responses):
I think 16% for fish is pretty remarkable, since (as far as I know) there isn’t any system where fish is the default shell, and my sense is that it’s very common to just stick to whatever your system’s default shell is.
It feels like a big achievement for the fish project, even if maybe my Mastodon followers are more likely than the average shell user to use fish for some reason.
Fish definitely isn’t for everyone. I think I like it because:
(seq 1 10) to run a command
instead of backticks or using set instead of exportMaybe you’re also a person who would like fish! I hope a few more of the people who fish is for can find it, because I spend so much of my time in the terminal and it’s made that time much more pleasant.
I just did a massive spring cleaning of one of my servers, trying to clean up what has become quite the mess of clutter. For every website on the server, I either:
It feels good to get rid of old code, and to turn previously dynamic sites (with all of the risk they come with) into plain HTML.
This is also making me seriously reconsider the value of spinning up any new projects. Several of these are now 10 years old, still churning along fine, but difficult to do any maintenance on because of versions and dependencies. For example:
One that I'm particularly happy with, despite it being an ugly pile of PHP, is oauth.net. I inherited this site in 2012, and it hasn't needed any framework upgrades since it's just using PHP templates. My ham radio website w7apk.com is similarly a small amount of templated PHP, and it is low stress to maintain, and actually fun to quickly jot some notes down when I want. I like not having to go through the whole ceremony of setting up a dev environment, installing dependencies, upgrading things to the latest version, checking for backwards incompatible changes, git commit, deploy, etc. I can just sftp some changes up to the server and they're live.
Some questions for myself for the future, before starting a new project:
One project I've been committed to maintaining and doing regular (ok fine, "semi-regular") updates for is Meetable, the open source events website that I run on a few domains:
I started this project in October 2019, excited for all the IndieWebCamps we were going to run in 2020. Somehow that is already 5 years ago now. Well that didn't exactly pan out, but I did quickly pivot it to add a bunch of features that are helpful for virtual events, so it worked out ok in the end. We've continued to use it for posting IndieWeb events, and I also run an instance for two IETF working groups. I'd love to see more instances pop up, I've only encountered one or two other ones in the wild. I even spent a significant amount of time on the onboarding flow so that it's relatively easy to install and configure. I even added passkeys for the admin login so you don't need any external dependencies on auth providers. It's a cool project if I may say so myself.
Anyway, this is not a particularly well thought out blog post, I just wanted to get my thoughts down after spending all day combing through the filesystem of my web server and uncovering a lot of ancient history.
About 3 years ago, I announced Mess With DNS in this blog post, a playground where you can learn how DNS works by messing around and creating records.
I wasn’t very careful with the DNS implementation though (to quote the release blog post: “following the DNS RFCs? not exactly”), and people started reporting problems that eventually I decided that I wanted to fix.
Some of the problems people have reported were:
And there are certainly more issues that nobody got around to reporting, for example that if you added an NS record for a subdomain to delegate it, Mess With DNS wouldn’t handle the delegation properly.
I wasn’t sure how to fix these problems for a long time – technically I could have started addressing them individually, but it felt like there were a million edge cases and I’d never get there.
But then one day I was chatting with someone else who was working on a DNS server and they said they were using PowerDNS: an open source DNS server with an HTTP API!
This seemed like an obvious solution to my problems – I could just swap out my own crappy DNS implementation for PowerDNS.
There were a couple of challenges I ran into when setting up PowerDNS that I’ll talk about here. I really don’t do a lot of web development and I think I’ve never built a website that depends on a relatively complex API before, so it was a bit of a learning experience.
One of the main things Mess With DNS does is give you a live view of every DNS query it receives for your subdomain, using a websocket. To make this work, it needs to intercept every DNS query before they it gets sent to the PowerDNS DNS server:
There were 2 options I could think of for how to intercept the DNS queries:
dnsdist (a DNS load balancer from the PowerDNS project) has
support for logging all DNS queries it receives using
dnstap, so I could put dnsdist in front of PowerDNS
and then log queries that wayI originally implemented option #1, but for some reason there was a 1 second delay before every query got logged. I couldn’t figure out why, so I implemented my own very simple proxy instead.
The frontend used to have a lot of DNS logic in it – it converted emoji domain
names to ASCII using punycode, had a lookup table to convert numeric DNS query
types (like 1) to their human-readable names (like A), did a little bit of
validation, and more.
Originally I considered keeping this pattern and just giving the frontend (more or less) direct access to the PowerDNS API to create and delete, but writing even more complex code in Javascript didn’t feel that appealing to me – I don’t really know how to write tests in Javascript and it seemed like it wouldn’t end well.
So I decided to take all of the DNS logic out of the frontend and write a new DNS API for managing records, shaped something like this:
GET /recordsDELETE /records/<ID>DELETE /records/ (delete all records for a user)POST /records/ (create record)POST /records/<ID> (update record)This meant that I could actually write tests for my code, since the backend is in Go and I do know how to write tests in Go.
I had this idea that APIs shouldn’t return duplicate information – for example if I get a DNS record, it should only include a given piece of information once.
But I ran into a problem with that idea when displaying MX records: an MX record has 2 fields, “preference”, and “mail server”. And I needed to display that information in 2 different ways on the frontend:
10 and mail.example.com)10 mail.example.com)This is kind of a small problem, but it came up in a few different places.
I talked to my friend Marco Rogers about this, and based on some advice from him I realized that I could return the same information in the API in 2 different ways! Then the frontend just has to display it. So I started just returning duplicate information in the API, something like this:
{
values: {'Preference': 10, 'Server': 'mail.example.com'},
content: '10 mail.example.com',
...
}
I ended up using this pattern in a couple of other places where I needed to display the same information in 2 different ways and it was SO much easier.
I think what I learned from this is that if I’m making an API that isn’t intended for external use (there are no users of this API other than the frontend!), I can tailor it very specifically to the frontend’s needs and that’s okay.
In Mess With DNS (and I think in most DNS user interfaces!), you create, add, and delete records.
But that’s not how the PowerDNS API works. In PowerDNS, you create a zone, which is made of record sets. Records don’t have any ID in the API at all.
I ended up solving this by generate a fake ID for each records which is made of:
For example one record’s ID is brooch225.messwithdns.com.|NS|bnMxLm1lc3N3aXRoZG5zLmNvbS4=
Then I can search through the zone and find the appropriate record to update it.
This means that if you update a record then its ID will change which isn’t usually what I want in an ID, but that seems fine.
I think the error messages that the PowerDNS API returns aren’t really intended to be shown to end users, for example:
Name 'new\032site.island358.messwithdns.com.' contains unsupported characters (this error encodes the space as \032, which is a bit disorienting if you don’t know that the space character is 32 in ASCII)RRset test.pear5.messwithdns.com. IN CNAME: Conflicts with pre-existing RRset (this talks about RRsets, which aren’t a concept that the Mess With DNS UI has at all)Record orange.beryl5.messwithdns.com./A '1.2.3.4$': Parsing record content (try 'pdnsutil check-zone'): unable to parse IP address, strange character: $ (mentions “pdnsutil”, a utility which Mess With DNS’s users don’t have
access to in this context)I ended up handling this in two ways:
Invalid IPv4 address: "1.2.3.4$Sometimes users will still get errors from PowerDNS directly, but I added some logging of all the errors that users see, so hopefully I can review them and add extra translations if there are other common errors that come up.
I think what I learned from this is that if I’m building a user-facing application on top of an API, I need to be pretty thoughtful about how I resurface those errors to users.
Previously Mess With DNS was using a Postgres database. This was problematic
because I only gave the Postgres machine 256MB of RAM, which meant that the
database got OOM killed almost every single day. I never really worked out
exactly why it got OOM killed every day, but that’s how it was. I spent some
time trying to tune Postgres’ memory usage by setting the max connections /
work-mem / maintenance-work-mem and it helped a bit but didn’t solve the
problem.
So for this refactor I decided to use SQLite instead, because the website doesn’t really get that much traffic. There are some choices involved with using SQLite, and I decided to:
db.SetMaxOpenConns(1) to make sure that we only open 1 connection to
the database at a time, to prevent SQLITE_BUSY errors from two threads
trying to access the database at the same time (just setting WAL mode didn’t
work)I still haven’t set up backups, though I don’t think my Postgres database had backups either. I think I’m unlikely to use litestream for backups – Mess With DNS is very far from a critical application, and I think daily backups that I could recover from in case of a disaster are more than good enough.
This has nothing to do with PowerDNS but I decided to upgrade Vue.js from version 2 to 3 as part of this refresh. The main problem with that is that the form validation library I was using (FormKit) completely changed its API between Vue 2 and Vue 3, so I decided to just stop using it instead of learning the new API.
I ended up switching to some form validation tools that are built into the
browser like required and oninvalid (here’s the code).
I think it could use some of improvement, I still don’t understand forms very well.
This also has nothing to do with PowerDNS, but when modifying the frontend I realized that my state management in the frontend was a mess – in every place where I made an API request to the backend, I had to try to remember to add a “refresh records” call after that in every place that I’d modified the state and I wasn’t always consistent about it.
With some more advice from Marco, I ended up implementing a single global state management store which stores all the state for the application, and which lets me create/update/delete records.
Then my components can just call store.createRecord(record), and the store
will automatically resynchronize all of the state as needed.
This project ended up having several steps because I reworked the whole integration between the frontend and the backend. I ended up splitting it into a few different phases:
I made sure that the website was (more or less) 100% working and then deployed it in between phases, so that the amount of changes I was managing at a time stayed somewhat under control.
I released the upgraded website a few days ago and it seems to work! The PowerDNS API has been great to work on top of, and I’m relieved that there’s a whole class of problems that I now don’t have to think about at all, other than potentially trying to make the error messages from PowerDNS a little clearer. Using PowerDNS has fixed a lot of the DNS issues that folks have reported in the last few years and it feels great.
If you run into problems with the new Mess With DNS I’d love to hear about them here.
I’ve been writing Go pretty casually for years – the backends for all of my playgrounds (nginx, dns, memory, more DNS) are written in Go, but many of those projects are just a few hundred lines and I don’t come back to those codebases much.
I thought I more or less understood the basics of the language, but this week I’ve been writing a lot more Go than usual while working on some upgrades to Mess with DNS, and ran into a bug that revealed I was missing a very basic concept!
Then I posted about this on Mastodon and someone linked me to this very cool site (and book) called 100 Go Mistakes and How To Avoid Them by Teiva Harsanyi. It just came out in 2022 so it’s relatively new.
I decided to read through the site to see what else I was missing, and found a couple of other misconceptions I had about Go. I’ll talk about some of the mistakes that jumped out to me the most, but really the whole 100 Go Mistakes site is great and I’d recommend reading it.
Here’s the initial mistake that started me on this journey:
Let’s say we have a struct:
type Thing struct {
Name string
}
and this code:
thing := Thing{"record"}
other_thing := thing
other_thing.Name = "banana"
fmt.Println(thing)
This prints “record” and not “banana” (play.go.dev link), because thing is copied when you
assign it to other_thing.
The bug I spent 2 hours of my life debugging last week was effectively this code (play.go.dev link):
type Thing struct {
Name string
}
func findThing(things []Thing, name string) *Thing {
for _, thing := range things {
if thing.Name == name {
return &thing
}
}
return nil
}
func main() {
things := []Thing{Thing{"record"}, Thing{"banana"}}
thing := findThing(things, "record")
thing.Name = "gramaphone"
fmt.Println(things)
}
This prints out [{record} {banana}] – because findThing returned a copy, we didn’t change the name in the original array.
This mistake is #30 in 100 Go Mistakes.
I fixed the bug by changing it to something like this (play.go.dev link), which returns a reference to the item in the array we’re looking for instead of a copy.
func findThing(things []Thing, name string) *Thing {
for i := range things {
if things[i].Name == name {
return &things[i]
}
}
return nil
}
When I learned that I was mistaken about how assignment worked in Go I was really taken aback, like – it’s such a basic fact about the language works! If I was wrong about that then what ELSE am I wrong about in Go????
My best guess for what happened is:
.clone() explicitly.
(though apparently structs will be automatically copied on assignment if the struct implements the Copy trait)When you subset a slice with x[2:3], the original slice and the sub-slice
share the same backing array, so if you append to the new slice, it can
unintentionally change the old slice:
For example, this code prints [1 2 3 555 5] (code on play.go.dev)
x := []int{1, 2, 3, 4, 5}
y := x[2:3]
y = append(y, 555)
fmt.Println(x)
I don’t think this has ever actually happened to me, but it’s alarming and I’m very happy to know about it.
Apparently you can avoid this problem by changing y := x[2:3] to y := x[2:3:3], which restricts the new slice’s capacity so that appending to it
will re-allocate a new slice. Here’s some code on play.go.dev that does that.
This one isn’t a “mistake” exactly, but it’s been a source of confusion for me and it’s pretty simple so I’m glad to have it cleared up.
In Go you can declare methods in 2 different ways:
func (t Thing) Function() (a “value receiver”)func (t *Thing) Function() (a “pointer receiver”)My understanding now is that basically:
t, you need a pointer receiver.t, use a value receiver.Explanation #42 has a bunch of other interesting details though. There’s definitely still something I’m missing about value vs pointer receivers (I got a compile error related to them a couple of times in the last week that I still don’t understand), but hopefully I’ll run into that error again soon and I can figure it out.
Some more notes from 100 Go Mistakes:
Also there are some things that have tripped me up in the past, like:
I really appreciated this “100 common mistakes” format – it made it really easy for me to skim through the mistakes and very quickly mentally classify them into:
It looks like “100 Common Mistakes” is a series of books from Manning and they also have “100 Java Mistakes” and an upcoming “100 SQL Server Mistakes”.
Also I enjoyed what I’ve read of Effective Python by Brett Slatkin, which has a similar “here are a bunch of short Python style tips” structure where you can quickly skim it and take what’s useful to you. There’s also Effective C++, Effective Java, and probably more.
other resources I’ve appreciated:
Here's where you can find me at IETF 120 in Vancouver!
The other day I asked what folks on Mastodon find confusing about working in the terminal, and one thing that stood out to me was “editing a command you already typed in”.
This really resonated with me: even though entering some text and editing it is
a very “basic” task, it took me maybe 15 years of using the terminal every
single day to get used to using Ctrl+A to go to the beginning of the line (or
Ctrl+E for the end – I think I used Home/End instead).
So let’s talk about why entering text might be hard! I’ll also share a few tips that I wish I’d learned earlier.
A big part of what makes entering text in the terminal hard is the inconsistency between how different programs handle entering text. For example:
cat, nc, git commit --interactive, etc) don’t support using arrow keys at all: if you press arrow keys, you’ll just see ^[[D^[[D^[[C^[[C^irb, python3 on a Linux machine and many many more) use the readline library, which gives you a lot of basic functionality (history, arrow keys, etc)/usr/bin/python3 on my Mac) do support very basic features like arrow keys, but not other features like Ctrl+left or reverse searching with Ctrl+Rfish shell or ipython3 or micro or vim) have their own fancy system for accepting input which is totally customSo there’s a lot of variation! Let’s talk about each of those a little more.
First, there’s “the baseline” – what happens if a program just accepts text by
calling fgets() or whatever and doing absolutely nothing else to provide a
nicer experience. Here’s what using these tools typically looks for me – If I
start the version of dash installed on
my machine (a pretty minimal shell) press the left arrow keys, it just prints
^[[D to the terminal.
$ ls l-^[[D^[[D^[[D
At first it doesn’t seem like all of these “baseline” tools have much in common, but there are actually a few features that you get for free just from your terminal, without the program needing to do anything special at all.
The things you get for free are:
Ctrl+W, to delete the previous wordCtrl+U, to delete the whole lineCtrl+C to interrupt the process, Ctrl+Z to suspend, etc)This is not great, but it means that if you want to delete a word you
generally can do it with Ctrl+W instead of pressing backspace 15 times, even
if you’re in an environment which is offering you absolutely zero features.
You can get a list of all the ctrl codes that your terminal supports with stty -a.
readlineThe next group is tools that use readline! Readline is a GNU library to make entering text more pleasant, and it’s very widely used.
My favourite readline keyboard shortcuts are:
Ctrl+E (or End) to go to the end of the lineCtrl+A (or Home) to go to the beginning of the lineCtrl+left/right arrow to go back/forward 1 wordCtrl+R to search your historyAnd you can use Ctrl+W / Ctrl+U from the “baseline” list, though Ctrl+U
deletes from the cursor to the beginning of the line instead of deleting the
whole line. I think Ctrl+W might also have a slightly different definition of
what a “word” is.
There are a lot more (here’s a full list), but those are the only ones that I personally use.
The bash shell is probably the most famous readline user (when you use
Ctrl+R to search your history in bash, that feature actually comes from
readline), but there are TONS of programs that use it – for example psql,
irb, python3, etc.
rlwrapOne of my absolute favourite things is that if you have a program like nc
without readline support, you can just run rlwrap nc to turn it into a
program with readline support!
This is incredible and makes a lot of tools that are borderline unusable MUCH more pleasant to use. You can even apparently set up rlwrap to include your own custom autocompletions, though I’ve never tried that.
I think reasons tools might not use readline might include:
cat or nc) and maybe the maintainers don’t want to bring in a relatively large dependencygit has a few interactive
features (like git add -p), but not very many, and usually you’re just
typing a single character like y or n – most of the time you need to really
type something significant in git, it’ll drop you into a text editor instead.For example idris2 says they don’t use readline
to keep dependencies minimal and suggest using rlwrap to get better
interactive features.
The simplest test I can think of is to press Ctrl+R, and if you see:
(reverse-i-search)`':
then you’re probably using readline. This obviously isn’t a guarantee (some
other library could use the term reverse-i-search too!), but I don’t know of
another system that uses that specific term to refer to searching history.
Because I’m a vim user, It took me a very long time to understand where these
keybindings come from (why Ctrl+A to go to the beginning of a line??? so
weird!)
My understanding is these keybindings actually come from Emacs – Ctrl+A and
Ctrl+E do the same thing in Emacs as they do in Readline and I assume the
other keyboard shortcuts mostly do as well, though I tried out Ctrl+W and
Ctrl+U in Emacs and they don’t do the same thing as they do in the terminal
so I guess there are some differences.
There’s some more history of the Readline project here.
libedit)On my Mac laptop, /usr/bin/python3 is in a weird middle ground where it
supports some readline features (for example the arrow keys), but not the
other ones. For example when I press Ctrl+left arrow, it prints out ;5D,
like this:
$ python3
>>> importt subprocess;5D
Folks on Mastodon helped me figure out that this is because in the default
Python install on Mac OS, the Python readline module is actually backed by
libedit, which is a similar library which has fewer features, presumably
because Readline is GPL licensed.
Here’s how I was eventually able to figure out that Python was using libedit on my system:
$ python3 -c "import readline; print(readline.__doc__)"
Importing this module enables command line editing using libedit readline.
Generally Python uses readline though if you install it on Linux or through Homebrew. It’s just that the specific version that Apple includes on their systems doesn’t have readline. Also Python 3.13 is going to remove the readline dependency in favour of a custom library, so “Python uses readline” won’t be true in the future.
I assume that there are more programs on my Mac that use libedit but I haven’t looked into it.
The last group of programs is programs that have their own custom (and sometimes much fancier!) system for editing text. This includes:
Ctrl+Z for undo when typing in a command. Zsh’s line editor is called zle.ipython), for example IPython uses the prompt_toolkit library instead of readlineatuin)Some features you might see are:
I went looking at how Atuin (a wonderful tool for searching your shell history that I started using recently) handles text input. Looking at the code and some of the discussion around it, their implementation is custom but it’s inspired by readline, which makes sense to me – a lot of users are used to those keybindings, and it’s convenient for them to work even though atuin doesn’t use readline.
prompt_toolkit (the library IPython uses) is similar – it actually supports a lot of options (including vi-like keybindings), but the default is to support the readline-style keybindings.
This is like how you see a lot of programs which support very basic vim
keybindings (like j for down and k for up). For example Fastmail supports
j and k even though most of its other keybindings don’t have much
relationship to vim.
I assume that most “readline-inspired” custom input systems have various subtle incompatibilities with readline, but this doesn’t really bother me at all personally because I’m extremely ignorant of most of readline’s features. I only use maybe 5 keyboard shortcuts, so as long as they support the 5 basic commands I know (which they always do!) I feel pretty comfortable. And usually these custom systems have much better autocomplete than you’d get from just using readline, so generally I prefer them over readline.
Bash, zsh, and fish all have a “vi mode” for entering text. In a very unscientific poll I ran on Mastodon, 12% of people said they use it, so it seems pretty popular.
Readline also has a “vi mode” (which is how Bash’s support for it works), so by extension lots of other programs have it too.
I’ve always thought that vi mode seems really cool, but for some reason even though I’m a vim user it’s never stuck for me.
I’ve spent a lot of my life being confused about why a command line application I was using wasn’t behaving the way I wanted, and it feels good to be able to more or less understand what’s going on.
I think this is roughly my mental flowchart when I’m entering text at a command line prompt:
Ctrl+W and Ctrl+U, and I can rlwrap the tool if I
want more features.Ctrl+R print reverse-i-search? Probably it’s readline, so I can use
all of the readline shortcuts I’m used to, and I know I can get some basic
history and press up arrow to get the previous command.Ctrl+R do something else? This is probably some custom input library:
it’ll probably act more or less like readline, and I can check the
documentation if I really want to know how it works.Being able to diagnose what’s going on like this makes the command line feel a more predictable and less chaotic.
There are lots more complications related to entering text that we didn’t talk about at all here, like:
TERM environment variableHello! Today someone on Mastodon asked about job control (fg, bg, Ctrl+z,
wait, etc). It made me think about how I don’t use my shell’s job
control interactively very often: usually I prefer to just open a new terminal
tab if I want to run multiple terminal programs, or use tmux if it’s over ssh.
But I was curious about whether other people used job control more often than me.
So I asked on Mastodon for reasons people use job control. There were a lot of great responses, and it even made me want to consider using job control a little more!
In this post I’m only going to talk about using job control interactively (not in scripts) – the post is already long enough just talking about interactive use.
First: what’s job control? Well – in a terminal, your processes can be in one of 3 states:
some_process &: the process is still running, but you can’t interact with it anymore unless you bring it back to the foreground.Ctrl+Z. This pauses the process: it won’t keep using the CPU, but you can restart it if you want.“Job control” is a set of commands for seeing which processes are running in a terminal and moving processes between these 3 states
fg brings a process to the foreground. It works on both stopped processes and background processes. For example, if you start a background process with cat < /dev/zero &, you can bring it back to the foreground by running fgbg restarts a stopped process and puts it in the background.Ctrl+z stops the current foreground process.jobs lists all processes that are active in your terminalkill sends a signal (like SIGKILL) to a job (this is the shell builtin kill, not /bin/kill)disown removes the job from the list of running jobs, so that it doesn’t get killed when you close the terminalwait waits for all background processes to complete. I only use this in scripts though.%2 instead of fg %2I might have forgotten some other job control commands but I think those are all the ones I’ve ever used.
You can also give fg or bg a specific job to foreground/background. For example if I see this in the output of jobs:
$ jobs
Job Group State Command
1 3161 running cat < /dev/zero &
2 3264 stopped nvim -w ~/.vimkeys $argv
then I can foreground nvim with fg %2. You can also kill it with kill -9 %2, or just kill %2 if you want to be more gentle.
kill %2 implemented?I was curious about how kill %2 works – does %2 just get replaced with the
PID of the relevant process when you run the command, the way environment
variables are? Some quick experimentation shows that it isn’t:
$ echo kill %2
kill %2
$ type kill
kill is a function with definition
# Defined in /nix/store/vicfrai6lhnl8xw6azq5dzaizx56gw4m-fish-3.7.0/share/fish/config.fish
So kill is a fish builtin that knows how to interpret %2. Looking at
the source code (which is very easy in fish!), it uses jobs -p %2 to expand %2
into a PID, and then runs the regular kill command.
Job control is implemented by your shell. I use fish, but my sense is that the basics of job control work pretty similarly in bash, fish, and zsh.
There are definitely some shells which don’t have job control at all, but I’ve only used bash/fish/zsh so I don’t know much about that.
Now let’s get into a few reasons people use job control!
I run into processes that don’t respond to Ctrl+C pretty regularly, and it’s
always a little annoying – I usually switch terminal tabs to find and kill and
the process. A bunch of people pointed out that you can do this in a faster way
using job control!
How to do this: Press Ctrl+Z, then kill %1 (or the appropriate job number
if there’s more than one stopped/background job, which you can get from
jobs). You can also kill -9 if it’s really not responding.
Sometimes I start a GUI program from the command line (for example with
wireshark some_file.pcap), forget to start it in the background, and don’t want it eating up my terminal tab.
How to do this:
Ctrl+Z and then running bg.disown to remove it from the list of jobs, to make sure that
the GUI program won’t get closed when you close your terminal tab.Personally I try to avoid starting GUI programs from the terminal if possible
because I don’t like how their stdout pollutes my terminal (on a Mac I use
open -a Wireshark instead because I find it works better but sometimes you
don’t have another choice.
tmuxThis is basically the same as the GUI app thing – you can move the job to the background and disown it.
I was also curious about if there are ways to redirect a process’s output to a file after it’s already started. A quick search turned up this Linux-only tool which is based on nelhage’s reptyr (which lets you for example move a process that you started outside of tmux to tmux) but I haven’t tried either of those.
vimA lot of people mentioned that if they want to quickly test something while
editing code in vim or another terminal editor, they like to use Ctrl+Z
to stop vim, run the command, and then run fg to go back to their editor.
You can also use this to check the output of a command that you ran before
starting vim.
I’ve never gotten in the habit of this, probably because I mostly use a GUI version of vim. I feel like I’d also be likely to switch terminal tabs and end up wondering “wait… where did I put my editor???” and have to go searching for it.
A few people said that they prefer to the output of all of their commands being interleaved in the terminal. This really surprised me because I usually think of having the output of lots of different commands interleaved as being a bad thing, but one person said that they like to do this with tcpdump specifically and I think that actually sounds extremely useful. Here’s what it looks like:
# start tcpdump
$ sudo tcpdump -ni any port 1234 &
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
# run curl
$ curl google.com:1234
13:13:29.881018 IP 192.168.1.173.49626 > 142.251.41.78.1234: Flags [S], seq 613574185, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2730440518 ecr 0,sackOK,eol], length 0
13:13:30.881963 IP 192.168.1.173.49626 > 142.251.41.78.1234: Flags [S], seq 613574185, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2730441519 ecr 0,sackOK,eol], length 0
13:13:31.882587 IP 192.168.1.173.49626 > 142.251.41.78.1234: Flags [S], seq 613574185, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2730442520 ecr 0,sackOK,eol], length 0
# when you're done, kill the tcpdump in the background
$ kill %1
I think it’s really nice here that you can see the output of tcpdump inline in your terminal – when I’m using tcpdump I’m always switching back and forth and I always get confused trying to match up the timestamps, so keeping everything in one terminal seems like it might be a lot clearer. I’m going to try it.
One person said that sometimes they’re running a very CPU-intensive program,
for example converting a video with ffmpeg, and they need to use the CPU for
something else, but don’t want to lose the work that ffmpeg already did.
You can do this by pressing Ctrl+Z to pause the process, and then run fg
when you want to start it again.
Many people replied that they didn’t use job control intentionally, but
that they sometimes accidentally ran Ctrl+Z, which stopped whatever program was
running, so they needed to learn how to use fg to bring it back to the
foreground.
The were also some mentions of accidentally running Ctrl+S too (which stops
your terminal and I think can be undone with Ctrl+Q). My terminal totally
ignores Ctrl+S so I guess I’m safe from that one though.
Some folks mentioned that they already set up a bunch of environment variables that they need to run various commands, so it’s easier to use job control to run multiple commands in the same terminal than to redo that work in another tab.
Probably the most obvious reason to use job control to manage multiple processes is “because you have to” – maybe you’re in single-user mode, or on a very restricted computer, or SSH’d into a machine that doesn’t have tmux or screen and you don’t want to create multiple SSH sessions.
Some people also said that they just don’t like using terminal tabs: for instance a few folks mentioned that they prefer to be able to see all of their terminals on the screen at the same time, so they’d rather have 4 terminals on the screen and then use job control if they need to run more than 4 programs.
I think my two main takeaways from thos post is I’ll probably try out job control a little more for:
tcpdump in the background with whatever network command I’m running, so I can see both of their output in the same placeHello! I’ve been writing about git on here nonstop for months, and the git zine is FINALLY done! It came out on Friday!
You can get it for $12 here: https://wizardzines.com/zines/git, or get an 14-pack of all my zines here.
Here’s the cover:
Here’s the table of contents:
I wrote this zine for people who have been using git for years and are still afraid of it. As always – I think it sucks to be afraid of the tools that you use in your work every day! I want folks to feel confident using git.
My goals are:
You might be wondering – Julia! You already have a zine about git! What’s going on? Oh Shit, Git! is a set of tricks for fixing git messes. “How Git Works” explains how Git actually works.
Also, Oh Shit, Git! is the amazing Katie Sylor Miller’s concept: we made it into a zine because I was such a huge fan of her work on it.
I think they go really well together.
This zine was really hard for me to write because when I started writing it, I’d been using git pretty confidently for 10 years. I had no real memory of what it was like to struggle with git.
But thanks to a huge amount of help from Marie as well as everyone who talked to me about git on Mastodon, eventually I was able to see that there are a lot of things about git that are counterintuitive, misleading, or just plain confusing. These include:
Your branch is up to date with 'origin/main' doesn’t necessary mean that your branch is up to date with the main branch on the origin)git pull and your branch has diverged from the origin, it doesn’t give you great guidance how to handle the situation)git stash drop)The more I heard from people how about how confusing they find git, the more it became clear that git really does not make it easy to figure out what its internal logic is just by using it.
The previous section made git sound really bad, like “how can anyone possibly use this thing?”.
But my experience is that after I learned what git actually means by all of its
weird error messages, dealing with it became pretty routine! I’ll see an
error: failed to push some refs to 'github.com:jvns/wizard-zines-site',
realize “oh right, probably a coworker made some changes to main since I last
ran git pull”, run git pull --rebase to incorporate their changes, and move
on with my day. The whole thing takes about 10 seconds.
Or if I see a You are in 'detached HEAD' state warning, I’ll just make sure
to run git checkout mybranch before continuing to write code. No big deal.
For me (and for a lot of folks I talk to about git!), dealing with git’s weird language can become so normal that you totally forget why anybody would even find it weird.
One of my biggest questions when writing this zine was how much to focus on
what’s in the .git directory. We ended up deciding to include a couple of
pages about internals (“inside .git”, pages 14-15), but otherwise focus more on
git’s behaviour when you use it and why sometimes git behaves in unexpected
ways.
This is partly because there are lots of great guides to git’s internals out there already (1, 2), and partly because I think even if you have read one of these guides to git’s internals, it isn’t totally obvious how to connect that information to what you actually see in git’s user interface.
For example: it’s easy to find documentation about remotes in git – for example this page says:
Remote-tracking branches […] remind you where the branches in your remote repositories were the last time you connected to them.
But even if you’ve read that, you might not realize that the statement Your branch is up to date with 'origin/main'" in git status doesn’t necessarily
mean that you’re actually up to date with the remote main branch.
So in general in the zine we focus on the behaviour you see in Git’s UI, and then explain how that relates to what’s happening internally in Git.
The zine also comes with a free printable cheat sheet: (click to get a PDF version)
The zine also comes with an HTML transcript, to (hopefully) make it easier to read on a screen reader! Our Operations Manager, Lee, transcribed all of the pages and wrote image descriptions. I’d love feedback about the experience of reading the zine on a screen reader if you try it.
I’ve been pretty critical about git in this post, but I only write zines about technologies I love, and git is no exception.
Some reasons I love git:
git commit -am 'whatever' and git push over and over again and it
works perfectly)I hope this zine helps some of you love it too.
I don’t make these zines by myself!
I worked with Marie Claire LeBlanc Flanagan every morning for 8 months to write clear explanations of git.
The cover is by Vladimir Kašiković, Gersande La Flèche did copy editing, James Coglan (of the great Building Git) did technical review, our Operations Manager Lee did the transcription as well as a million other things, my partner Kamal read the zine and told me which parts were off (as he always does), and I had a million great conversations with Marco Rogers about git.
And finally, I want to thank all the beta readers! There were 66 this time which is a record! They left hundreds of comments about what was confusing, what they learned, and which of my jokes were funny. It’s always hard to hear from beta readers that a page I thought made sense is actually extremely confusing, and fixing those problems before the final version makes the zine so much better.
Here are some links to get the zine again:
As always, you can get either a PDF version to print at home or a print version shipped to your house. The only caveat is print orders will ship in July – I need to wait for orders to come in to get an idea of how many I should print before sending it to the printer.
As always: if you’ve bought zines in the past, thank you for all your support over the years. And thanks to all of you (1000+ people!!!) who have already bought the zine in the first 3 days. It’s already set a record for most zines sold in a single day and I’ve been really blown away.
IndieWebCamp Düsseldorf took place this weekend, and I was inspired to work on a quick hack for demo day to show off a new feature I've been working on for IndieAuth.
Since I do actually use my website to log in to different websites on a regular basis, I am often presented with the login screen asking for my domain name, which is admittedly an annoying part of the process. I don't even like having to enter my email address when I log in to a site, and entering my domain isn't any better.
So instead, I'd like to get rid of this prompt, and let the browser handle it for you! Here's a quick video of logging in to a website using my domain with the new browser API:
So how does this work?
For the last couple of years, there has been an ongoing effort at the Federated Identity Community Group at the W3C to build a new API in browsers that can sit in the middle of login flows. It's primarily being driven by Google for their use case of letting websites show a Google login popup dialog without needing 3rd party cookies and doing so in a privacy-preserving way. There's a lot to unpack here, more than I want to go into in this blog post. You can check out Tim Cappalli's slides from the OAuth Security Workshop for a good explainer on the background and how it works.
However, there are a few experimental features that are being considered for the API to accommodate use cases beyond the "Sign in with Google" case. The one that's particularly interesting to the IndieAuth use case is the IdP Registration API. This API allows any website to register itself as an identity provider that can appear in the account chooser popup, so that a relying party website doesn't have to list out all the IdPs it supports, it can just say it supports "any" IdP. This maps to how IndieAuth is already used today, where a website can accept any user's IndieAuth server without any prior relationship with the user. For more background, check out my previous blog post "OAuth for the Open Web".
So now, with the IdP Registration API in FedCM, your website can tell your browser that it is an IdP, then when a website wants to log you in, it asks your browser to prompt you. You choose your account from the list, the negotiation happens behind the scenes, and you're logged in!
One of the nice things about combining FedCM with IndieAuth is it lends itself nicely to running the FedCM IdP as a separate service from your actual website. I could run an IndieAuth IdP service that you could sign up for and link your website to. Since your identity is your website, your website would be the thing ultimately sent to the relying party that you're signing in to, even though it was brokered through the IdP service. Ultimately this means much faster adoption is possible, since all it takes to turn your website into a FedCM-supported site is adding a single <link> tag to your home page.
So if this sounds interesting to you, leave a comment below! The IdP registration API is currently an early experiment, and Google needs to see actual interest in it in order to keep it around! In particular, they are looking for Relying Parties who would be interested in actually using this to log users in. I am planning on launching this on webmention.io as an experiment. If you have a website where users can sign in with IndieAuth, feel free to get in touch and I'd be happy to help you set up FedCM support as well!
The draft specification OAuth for Browser-Based Applications has just entered Working Group Last Call!
https://datatracker.ietf.org/doc/html/draft-ietf-oauth-browser-based-apps
This begins a two-week period to collect final comments on the draft. Please review the draft and reply on the OAuth mailing list if you have any comments or concerns. And if you've reviewed the document and are happy with the current state, it is also extremely helpful if you can reply on the list to just say "looks good to me"!
If joining the mailing list is too much work, you're also welcome to comment on the Last Call issue on GitHub.
In case you were wondering, yes your comments matter! Even just a small indication of support goes a long way in these discussions!
I am extremely happy with how this draft has turned out, and would like to again give a huge thanks to Philippe De Ryck for the massive amount of work he's put in to the latest few versions to help get this over the finish line!
While writing about Git, I’ve noticed that a lot of folks struggle with Git’s error messages. I’ve had many years to get used to these error messages so it took me a really long time to understand why folks were confused, but having thought about it much more, I’ve realized that:
So in this post, I’m going to go through a bunch of Git’s error messages, list a few things that I think are confusing about them for each one, and talk about what I do when I’m confused by the message.
Before we start, I want to say that trying to think about why these error messages are confusing has given me a lot of respect for how difficult maintaining Git is. I’ve been thinking about Git for months, and for some of these messages I really have no idea how to improve them.
Some things that seem hard to me about improving error messages:
That said, if you find these messages confusing, hopefully some of these notes will help clarify them a bit.
git push on a diverged branch
$ git push To github.com:jvns/int-exposed ! [rejected] main -> main (non-fast-forward) error: failed to push some refs to 'github.com:jvns/int-exposed' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. $ git status On branch main Your branch and 'origin/main' have diverged, and have 2 and 1 different commits each, respectively.
Some things I find confusing about this:
git status or git pull to find out.failed to push some refs, but it’s not totally clear which references it
failed to push. I believe everything that failed to push is listed with ! [rejected] on the previous line– in this case just the main branch.What I like to do if I’m confused:
git status to figure out what the state of my current branch is.git pull on a diverged branch
$ git pull
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
The main thing I think is confusing here is that git is presenting you with a kind of overwhelming number of options: it’s saying that you can either:
pull.rebase false, pull.rebase true, or pull.ff only locallygit pull --rebase or git pull --no-rebaseIt’s very hard to imagine how a beginner to git could easily use this hint to sort through all these options on their own.
If I were explaining this to a friend, I’d say something like “you can use git pull --rebase
or git pull --no-rebase to resolve this with a rebase or merge
right now, and if you want to set a permanent preference, you can do that
with git config pull.rebase false or git config pull.rebase true.
git config pull.ff only feels a little redundant to me because that’s git’s
default behaviour anyway (though it wasn’t always).
What I like to do here:
git status to see the state of my current branchgit log origin/main or git log to see what the diverged commits aregit pull --rebase to resolve itgit push --force or git reset --hard origin/main if I
want to throw away my local work or remote work (for example because I
accidentally commited to the wrong branch, or because I ran git commit --amend on a personal branch that only I’m using and want to force push)git checkout asdf (a branch that doesn't exist)
$ git checkout asdf error: pathspec 'asdf' did not match any file(s) known to git
This is a little weird because we my intention was to check out a branch,
but git checkout is complaining about a path that doesn’t exist.
This is happening because git checkout’s first argument can be either a
branch or a path, and git has no way of knowing which one you intended. This
seems tricky to improve, but I might expect something like “No such branch,
commit, or path: asdf”.
What I like to do here:
git switch instead, but I keep using git checkout anywayasdf doesn’t exist”git switch asdf (a branch that doesn't exist)
$ git switch asdf fatal: invalid reference: asdf
git switch only accepts a branch as an argument (unless you pass -d), so why is it saying invalid reference: asdf instead of invalid branch: asdf?
I think the reason is that internally, git switch is trying to be helpful in its error messages: if you run git switch v0.1 to switch to a tag, it’ll say:
$ git switch v0.1
fatal: a branch is expected, got tag 'v0.1'`
So what git is trying to communicate with fatal: invalid reference: asdf is
“asdf isn’t a branch, but it’s not a tag either, or any other reference”. From my various git polls my impression is that
a lot of git users have literally no idea what a “reference” is in git, so I’m not sure if that’s coming across.
What I like to do here:
90% of the time when a git error message says reference I just mentally
replace it with branch in my head.
git checkout HEAD^
$ git checkout HEAD^ Note: switching to 'HEAD^'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -cOr undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 182cd3f add "swap byte order" button
This is a tough one. Definitely a lot of people are confused about this message, but obviously there's been a lot of effort to improve it too. I don't have anything smart to say about this one.
What I like to do here:
git checkout main or something to go back to a branchgit status when a rebase is in progress
This isn’t an error message, but I still find it a little confusing on its own:
$ git status interactive rebase in progress; onto c694cf8 Last command done (1 command done): pick 0a9964d wip No commands remaining. You are currently rebasing branch 'main' on 'c694cf8'. (fix conflicts and then run "git rebase --continue") (use "git rebase --skip" to skip this patch) (use "git rebase --abort" to check out the original branch) Unmerged paths: (use "git restore --staged..." to unstage) (use "git add ..." to mark resolution) both modified: index.html no changes added to commit (use "git add" and/or "git commit -a")
Two things I think could be clearer here:
You are currently rebasing branch 'main' on 'c694cf8'. were on the first line instead of the 5th line – right now the first line doesn’t say which branch you’re rebasing.c694cf8 is actually origin/main, so I feel like You are currently rebasing branch 'main' on 'origin/main' might be even clearer.What I like to do here:
My shell prompt includes the branch that I’m currently rebasing, so I rely on that instead of the output of git status.
git rebase when a file has been deleted
$ git rebase main CONFLICT (modify/delete): index.html deleted in 0ce151e (wip) and modified in HEAD. Version HEAD of index.html left in tree. error: could not apply 0ce151e... wip
The thing I still find confusing about this is – index.html was modified in
HEAD. But what is HEAD? Is it the commit I was working on when I started
the merge/rebase, or is it the commit from the other branch? (the answer is
“HEAD is your branch if you’re doing a merge, and it’s the “other branch” if
you’re doing a rebase, but I always find that hard to remember)
I think I would personally find it easier to understand if the message listed the branch names if possible, something like this:
CONFLICT (modify/delete): index.html deleted on `main` and modified on `mybranch`
git status during a merge or rebase (who is "them"?)
$ git status On branch master You have unmerged paths. (fix conflicts and run "git commit") (use "git merge --abort" to abort the merge)Unmerged paths: (use “git add/rm
…” as appropriate to mark resolution) deleted by them: the_file no changes added to commit (use “git add” and/or “git commit -a”)
I find this one confusing in exactly the same way as the previous message: it
says deleted by them:, but what “them” refers to depends on whether you did a merge or rebase or cherry-pick.
them is the other branch you merged inthem is the branch that you were on when you ran git rebaseWhat I like to do if I’m confused:
git show main --stat or something to see what I did on the main branch if I can’t remembergit clean
$ git clean fatal: clean.requireForce defaults to true and neither -i, -n, nor -f given; refusing to clean
I just find it a bit confusing that you need to look up what -i, -n and
-f are to be able to understand this error message. I’m personally way too
lazy to do that so even though I’ve probably been using git clean for 10
years I still had no idea what -i stood for (interactive) until I was
writing this down.
What I like to do if I’m confused:
Usually I just chaotically run git clean -f to delete all my untracked files
and hope for the best, though I might actually switch to git clean -i now
that I know what -i stands for. Seems a lot safer.
Hopefully some of this is helpful!
These are common questions when writing documentation for OAuth-related things. While these terms are all used in RFC 6749 and many extensions, the differences between the terminology is never actually explained.
I wanted to finally write down a definition of the terms, along with examples of when each is appropriate.
Let me know if you have any suggestions for clarifying any of this, or any other helpful examples to add! I'm planning on adding this summary to OAuth 2.1 so that we have a formal reference for it in the future!
It was 11am at the Fort Lauderdale airport, an hour after my non-stop flight to Portland was supposed to have boarded. As I had been watching our estimated departure get pushed back in 15 minute increments, I finally received the dreaded news over the loudspeaker - the flight was cancelled entirely. As hordes of people started lining up to rebook their flights with the gate agent, I found a quiet spot in the corner and opened up my laptop to look at my options.
The other Alaska Airlines flight options were pretty terrible. There was a Fort Lauderdale to Seattle to Portland option that would have me landing at midnight. A flight on a partner airline had a 1-hour connection through Dallas, and there were only middle seats available on both legs. So I started to get creative, and searched for flights from Orlando, about 200 miles north. There was a non-stop on Alaska Airlines at 7pm, with plenty of available seats, so I called up customer service and asked them to change my booking. Since the delay was their fault, there were no change fees even though the flight was leaving from a different airport.
So now it was my responsibility to get myself from Miami to Orlando by 7pm. I could have booked a flight on a budget airline for $150, but it wouldn't have been a very nice experience, and I'd have a lot of time to kill in the Orlando airport. Then I remembered the Brightline train recently opened new service from Miami to Orlando, supposedly taking less time than driving there.
Never having tried to take that train before, I didn't realize they run a shuttle service from the Fort Lauderdale airport to the train station, so I jumped in an Uber headed to the station. On the way there, I booked a ticket on my phone. The price from Miami to Orlando was $144 for Coach, or $229 for Premium class. Since this will probably be the only time I take this train for the foreseeable future, I splurged for the Premium class ticket to see what that experience is like.
Astute readers will have noticed that I mentioned I booked a ticket from Miami rather than Fort Lauderdale. We'll come back to that in a bit. Once I arrived at the station, I began my Brightline experience.
Walking in to the station felt like something between an airport and a car rental center.
There was a small ticket counter in the lobby, but I already had a ticket on my phone so I went up the escalators.
At the top of the escalators was an electronic gate where you scan your QR code to go through. Mine didn't work (again, more on that later), but it was relatively empty and a staff member was able to look at my ticket on my phone and let me through anyway. There was a small X-Ray machine, I tossed my roller bag and backpack onto the belt, but kept my phone and wallet in my pocket, and walked through the security checkpoint.
Once through the minimal security checkpoint, I was up in the waiting area above the platform with a variety of different sections. There was a small bar with drinks and snacks, a couple large seating areas, an automated mini mart, some tall tables...
... and the entrance to the Premium lounge.
The Premium Lounge entrance had another electronic gate with a QR code scanner. I tried getting in but it also rejected my boarding pass. My first thought was I booked my ticket just 10 minutes earlier so it hadn't synced up yet, so I went back to the the security checkpoint and asked what was wrong. They looked at my boarding pass and had no idea what was wrong, and let me in to the lounge via the back employee-only entrance instead.
Once inside the lounge, I did a quick loop to see what kind of food and drink options there were. The lounge was entirely un-attended, the only staff I saw were at the security checkpoint, and someone occasionally coming through to take out dirty dishes.
The first thing you're presented with after entering the lounge is the beverage station. There are 6 taps with beer and wine, and you use a touch screen to make your selection and pour what you want.
On the other side of the wall is the food. I arrived at the tail end of the breakfast service, so there were pretty slim pickings by the end.
There were yogurts, granola, a bowl of bacon and egg mix, several kinds of pastries, and a bowl of fruit that nobody seemed to have touched. I don't know if this was just because this was the end of the morning, but if you were vegan or gluten free there was really nothing you could eat there.
There was also a coffee and tea station with some minimal options.
Shortly after I arrived, it rolled over to lunch time, so the staff came out to swap out the food at the food station. The lunch options were also minimal, but there was a bit more selection.
There was a good size meat and cheese spread. I'm not a big fan of when they mix the meat and cheese on the same plate, but there was enough of a cheese island in the middle I was reasonably confident I wasn't eating meat juice off the side of the cheeses. The pasta dish also had meat so I didn't investigate further. Two of the three wraps had meat and I wasn't confident about which were which so I skipped those. There was a pretty good spinach and feta salad, and some hummus as well as artichoke dip, and a variety of crackers. If you like desserts, there was an even better selection of small desserts as well.
At this point I was starting to listen for my train's boarding announcement. There was really barely any staff visible anywhere, but the few people I saw had made it clear they would clearly announce the train over the loudspeakers when it was time. There was also a sign at the escalators to the platform that said boarding opens 10 minutes before the train departs.
The trains run northbound and southbound every 1-2 hours, so it's likely that you'll only hear one announcement for a train other than yours the entire time you're there.
The one train announcement I heard was a good demonstration of how quickly the whole process actually is once the train shows up. The train pulls up, they call everyone down to the platform, and you have ten minutes to get onto the train. Ten minutes isn't much, but you're sitting literally right on top of the train platform so it takes no time to get down there.
Once your train is called, it's time to head down the escalator to the train platform!
But wait, I mentioned my barcode had failed to be scanned a couple times at this point. Let me explain. Apparently, in my haste in the back of the Uber, I had actually booked a ticket from Miami to Orlando, but since I was already at the Fort Lauderdale airport, I had gone to the Fort Lauderdale Brightline station since it was the closest. So the departure time I saw on my ticket didn't match the time the train arrived at Fort Lauderdale, and the ticket gates refused to let me in because the ticket didn't depart from that station. I don't know why none of the employees who looked at my ticket mentioned this ever. It didn't end up being a big deal because thankfully Miami was earlier in the route, so I essentially just got on my scheduled train 2 stops late.
So anyway, I made my way down to the platform to board the train. I should also mention at this point that I was on a conference call from my phone. I had previously connected my phone to the free wifi at the station, and it was plenty good enough for the call. As I went down the escalator to the platform, it broke up a bit in the middle of the escalator, but picked back up once I was on the platform outside.
There were some signs on the platform to indicate "Coach 1", "Coach 2" and "Coach 3" cars. However my ticket was a "Premium" ticket, so I walked to where I assumed the front of the train would be when it pulled up.
I got on the train on the front car marked "SMART" and "3", seats 9-17. It wasn't clear what "SMART" was since I didn't see that option when booking online. My seat was seat 9A, so I wasn't entirely sure I was in the right spot, but I figured better to be on the train than on the platform, so I just went in. We started moving shortly after. As soon as I walked in, I had to walk past the train attendant pushing a beverage cart through the aisles. I made it to seat 9, but it was occupied. I asked the attendant where my seat was, and she said it was in car 1 at the "front", and motioned to the back of the train. I don't know why their cars are in the opposite order you'd expect. So I took my bags back to car 1 where I was finally greeted with the "Premium" sign I was looking for.
I was quickly able to find my seat, which was not in fact occupied. The Premium car was configured with 2 seats on one side and 1 seat on the other side.
Some of the seats are configured to face each other, so there is a nice variety of seating options. You could all be sitting around a table if you booked a ticket for 4 people, or you could book 2 tickets and sit either next to each other or across from each other.
Since I had booked my ticket so last minute, I had basically the last available seat in the car so I was sitting next to someone. As soon as I sat down, the beverage cart came by with drinks. The cart looked like the same type you'd find on an airplane, and even had some identical warning stickers on it such as the "must be secured for takeoff and landing" sign. The drink options were also similar to what you'd get on a Premium Economy flight service. I opted for a glass of prosecco, and made myself comfortable.
The tray table at the seat had two configurations. You could either drop down a small flap or the whole tray.
The small tray was big enough to hold a drink or an iPad or phone, but not much else. The large tray was big enough for my laptop with a drink next to it as well as an empty glass or bottle behind it.
Under the seat there was a single power outlet for the 2 seats with 120v power as well as two USB-C ports.
Shortly after I had settled in, the crew came back with a snack tray and handed me these four snacks without really giving me the option of refusing any of them.
At this point I wasn't really hungry since I had just eaten at the airport, so I stuffed the snacks in my bag, except for the prosciutto, which I offered to my seat mate but he refused.
By this point we were well on our way to the Boca Raton stop. A few people got off and on there, and we continued on. I should add here that I always feel a bit unsettled when there is that much movement of people getting on and off all the time. These stops were about 20-30 minutes away from each other, which meant the beginning of the ride I never really felt completely settled in. This is the same reason I prefer a 6 hour flight over two 3 hour flights. I like to be able to settle in and just not think about anything until we arrive.
We finally left the last of the South Florida stops, West Palm Beach, and started the rest of the trip to Orlando. A bunch of people got off at West Palm Beach, enough that the Premium cabin was nearly empty at that point. I was able to move to the seat across the aisle which was a window/aisle seat all to myself!
Finally I could settle in for the long haul. Shortly before 3, the crew came by with the lunch cart. The options were either a vegetarian or non-vegetarian option, so that made the choice easy for me.
The vegetarian option was a tomato basil mozzarella sandwich, a side of fruit salad, and some vegetables with hummus. The hummus was surprisingly good, not like the little plastic tubs you get at the airport. The sandwich was okay, but did have a nice pesto spread on it.
After lunch, I opened up my computer to start writing this post and worked on it for most of the rest of the trip.
As the train started making a left turn to head west, the conductor came on the loudspeaker and made an announcement along the lines of "we're about to head west onto the newest tracks that have been built in the US in 100 years. We'll be reaching 120 miles per hour, so feel free to feel smug as we whiz by the cars on the highway." And sure enough, we really picked up the speed on that stretch! While we had reached 100-120mph briefly during the trip north, that last stretch was a solid 120mph sustained for about 20 minutes!
We finally slowed down and pulled into the Orlando station at the airport.
Disembarking the train was simple enough. This was the last stop of the train so there wasn't quite as much of a rush to get off before the train started again. There's no need to mind the gap as you get off since there's a little platform that extends from the train car.
At the Orlando station there was a short escalator up and then you exit through the automated gates.
I assumed I would have to scan my ticket when exiting but that ended up not being the case. Which actually meant that the only time my ticket was ever checked was when entering the station. I never saw anyone come through to check tickets on the train.
At this point I was already in the airport, and it was a short walk around the corner to the tram that goes directly to the airport security checkpoint.
The whole trip took 176 minutes for 210 miles, which is an average speed of 71 miles per hour. When moving, we were typically moving at anywhere from 80-120 miles per hour.
We need more high speed trains in the US! I go from Portland to Seattle often enough that a train running every 90 minutes that was faster than a car and easier and more comfortable than an airplane would be so nice!