2025-05-08T19:32:21+00:00 Fullscreen Open in Tab
GENIUS Act stablecoin bill stalls in the Senate

GENIUS Act stablecoin bill stalls in the Senate


The GENIUS Act stablecoin bill has stalled in the Senate with key Democrats — including some of its early co-sponsors — voting against it. Democratic opposition increased after Trump’s World Liberty Financial cryptocurrency platform released its own stablecoin, allowing the president to profit from transactions including a $2 billion deal between the Emirati state-owned investment firm and the Binance crypto platform.

Senators who voted against the bill include Ruben Gallego (AZ), Mark Warner (VA), Lisa Blunt Rochester (DE), Andy Kim (NJ), Kirsten Gillibrand (NY), and Angela Alsobrooks (MD). Gallego was one of the top recipients of crypto industry campaign funds in 2024, receiving $10 million in crypto super PAC backing.

Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
Thu, 08 May 2025 17:54:14 +0000 Fullscreen Open in Tab
Pluralistic: Who broke the internet? (08 May 2025)


Today's links



The logo for 'Who Broke the Internet' - a shattered hard-drive with the CBC logomark.

Who broke the internet? (permalink)

"Who Broke the Internet?" is a new podcast from CBC Understood that I host and co-wrote – it's a four-part series that explains how the enshitternet came about, and, more importantly, what we can do about it. Episode one is out this week:

https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16144078-dont-be-evil

The thesis of the series – and indeed, of my life's work – is that the internet didn't turn to shit because of the "great forces of history," or "network effects," or "returns to scale." Rather, the Great Enshittening is the result of specific policy choices, made in living memory, by named individuals, who were warned at the time that this would happen, and they did it anyway. These wreckers are the largely forgotten authors of our misery, and they mingle with impunity in polite society, never fearing that someone might be sizing them up for a pitchfork.

"Who Broke the Internet?" aims to change that. But the series isn't just about holding these named people accountable for their enshittificatory deeds: it's about understanding the policies that created the enshittocene, so that we can dismantle them and build a new, good internet that is fit for purpose, namely, helping us overcome and survive environmental collapse, oligarchic control, fascism and genocide.

The crux of enshittification theory is this: tech bosses made their products and services so much worse in order to extract more rents from end-users and business customers. The reason they did this is because they could. Over 20+ years, our policymakers created an environment of impunity for enshittifying companies, sitting idly by (or even helping out) as tech companies bought or destroyed their competitors; captured their regulators; neutered tech workers' power; and expanded IP laws to ensure that technology could only ever be used to attack us, but never to defend us.

These four forces – competition, regulation, labor power and interoperability – once acted as constraints, because they punished enshittifying gambits. Make your product worse and users, workers and suppliers would defect to a competitor; or a regulator would fine you or even bring criminal charges; or your irreplaceable workers would down tools and refuse to obey your orders; or another technologist would come up with an alternative client, an ad-blocker, a scraper, or compatible spare parts, plugins or mods that would permanently sever your relationship with whomever you were tormenting.

As these constraints fell away, the environment became enshittogenic: rather than punishing enshittification, it rewarded it. Individual enshittifiers within companies triumphed in their factional struggles with corporate rivals, like the Google revenue czar who vanquished the Search czar, deliberately worsening search results so we'd have to repeatedly search to get the answers we seek, creating more opportunities to show us ads:

https://www.wheresyoured.at/the-men-who-killed-google/

An enshittogenic environment meant that individuals within companies who embraced plans to worsen things to juice profits were promoted, displacing workers and managers who felt an ethical or professional obligation to make good and useful things. Top tech bosses – the C-suite – went from being surrounded by "adult supervision" who checked their worst impulses with dire warnings about competition, government punishments, or worker revolt to being encysted in a casing of enthusiastic enshittifiers who competed to see who could come up with the most outrageously enshittificatory gambits.

"Who Broke the Internet?" covers the collapse of all of these constraints, but its main focus is on IP law – specifically, anticircumvention law, which bans technologists from reverse-engineering and modifying the technologies we own and use (AKA "interoperability" or "adversarial interoperability").

Interoperability is at the center of the enshittification story because interop is an unavoidable characteristic of anything built out of computers. Computers are, above all else, flexible. Formally speaking, our computers are "Turing-complete universal von Neumann machines," which is to say that every one of our computers is capable of running every valid program.

That flexibility is why we call computers a "general purpose" technology. The same computer that helps your optometrist analyze your retina can also control your car's anti-lock braking system, and it can also play Doom.

Enshittification runs on that flexibility. It's that flexibility that allows a digital products or service to offer different prices, search rankings, recommendations, and costs to every user, every time they interact with it:

https://pluralistic.net/2023/02/19/twiddler/

It's that flexibility that lets tech companies send over-the-air "updates" to your property that takes away functionality you paid for and valued, and then sell it back to you as an "upgrade" or worse, a monthly subscription:

https://pluralistic.net/2023/10/26/hit-with-a-brick/#graceful-failure

But that flexibility cuts both ways. The fact that every computer can run every valid program means that every enshittificatory app and update, there's a disenshittificatory program you could install that would reverse the damage. For every program that tells your HP printer to reject third-party ink, forcing you to buy HP's own colored water at $10,000/gallon, there's another program that tells your HP printer to enthusiastically accept third-party ink that costs mere pennies:

https://www.eff.org/deeplinks/2020/11/ink-stained-wretches-battle-soul-digital-freedom-taking-place-inside-your-printer

In other worse, show me a 10-foot enshittifying wall, and I'll show you an 11-foot disenshittifying ladder.

Interoperability has long been technology's most important disenshittifier. Interop harnesses the rapaciousness of tech bros and puts it in service to making things better. Someone who hacks Instagram to take out the ads and recommendations and just show you posts from people you follow need not be motivated by the desire to make your life better – they can be motivated by the desire to poach Instagram users and build a rival business, and still make life better for you:

https://www.digitaltrends.com/mobile/the-og-app-instagram-alternative-ad-free/

And if they succeed and then recapitulate the sins of Instagram's bosses, turning the screws on users with ads, suggestions and slop? That just invites more disenshittifying interoperators to do unto them as they did unto Zuck.

That's the way it used to work: the 10-foot piles of shit deployed by tech bosses conjured up 11-foot ladders. This is what disruption is, when it is at its best. There's nothing wrong with moving fast and breaking things – provided the things you're breaking belong to billionaire enshittifiers. Those things need to be broken.

Enter IP law. For the past 25+ years, IP law has been relentlessly expanded in ways that ensure that disruption is always for thee, never me. "IP" has come to mean, "Any law that lets a dominant company reach out and exert control over its critics, competitors and customers":

https://locusmag.com/2020/09/cory-doctorow-ip/

The most pernicious IP law is far and away "anticircumvention." Under anticircumvention, it is illegal to "break a digital lock" that controls access to a copyrighted work, including software (and digital locks are software, so any digital lock automatically gets this protection).

This is mind-bending, particularly because it's one of those things that's so unreasonable, so very, very stupid that it's easy to think you're misunderstanding it, because surely it can't be that stupid.

But oh, it is.

One of the best ways to grasp this point is to start with what you might do in a world without digital locks. Take your printer: if HP raises the price of ink, you might start to refill your cartridges or buy third-party cartridges. Obviously, this is not a copyright violation. Ink is not a copyrighted work. But once HP puts a digital lock on the printer that checks to see if you've done an end-run around the HP ink ripoff, then refilling your cartridge becomes illegal, because you have to break that digital lock to get your printer to use the ink you've chosen.

Or think about cars: taking your car to your mechanic does not violate anyone's copyright. If your car, you decide who fixes it. But all car makers use digital locks to prevent mechanics from reading out the diagnostic information they need to access to fix your car. If a mechanic wants to know why your check engine light has turned on, they have to buy a tool – spending five-figure sums every year for every manufacturer – in order to decode that error. Now, it's your car, and error messages aren't copyrighted works, but bypassing the lock that prevents independent diagnosis is a crime, thanks to anticircumvention law.

Then there's app stores. You bought your console. You bought your phone. These devices are your property. If I want to sell you some software I've written so you can run it on your device, that's not a copyright violations. It is the literal opposite of a copyright violation: an author selling their copyrighted works to a customer who gets to enjoy those works using their own property. But the digital lock on your iPhone, Xbox, Playstation and Switch all prevent your device from running software unless it is delivered by the manufacturer's app store, which takes 30 cents out of every dollar you spend. Installing software without going through the manufacturer's app store requires that you break the device's digital lock, and that's a crime, which means that buying a copyrighted work from its author becomes a copyright violation!

This is what Jay Freeman calls "felony contempt of business model." We created laws – again, in living memory, thanks to known individuals – that had the foreseeable, explicit intent of making it illegal to disenshittify the products and services you rely on. We created this enshittogenic environment, and we got the enshittocene.

That's where "Who Broke the Internet?" comes in. We tell the story of Bruce Lehman, who was Bill Clinton's IP czar. Anticircumvention was really Lehman's brainchild, and he had a plan to make it the law of the land. When Al Gore was overseeing the demilitarization of the internet (the "Information Superhighway" proceedings), Lehman pitched this idea to him as the new rules of the road for the internet. To Gore's eternal credit, he flatly rejected Lehman's proposal as the batshit nonsense it plainly was.

So Lehman scuttled to Switzerland, where a UN agency, the World Intellectual Property Organization (WIPO) was crafting a pair of new treaties to create a global system of internet regulation. Lehman lobbied the national delegations to WIPO to put anticircumvention in their treaties, and he succeeded – partially. WIPO is a very bad agency, since the majority of delegations that are sent to Geneva by the world's nations come from poor countries in the global south, and they're made up of experts in things like water, agriculture and child health. The vast majority of national reps at WIPO are not experts in IP, and they are often easy prey for fast-talking lobbyists from US-based media, pharma and tech companies, as well as the US government reps who carry their water.

But even at WIPO, Lehman's proposal was viewed as far too extreme. In the end, the anticircumvention rules embedded in the WIPO treaties are much more reasonable than Lehman's demands. Under the WIPO treaty, signatories must pass laws that make copyright infringement extra illegal if you have to break a digital lock on the way. But if you break a lock and you don't infringe copyright (say, because you refilled a printer cartridge, took your car to an independent mechanic, or got some software without using an app store), then you're fine.

Lehman's next move was to convince Congress that they needed to pass a version of the anticircumvention rule that went far beyond the obligations in the WIPO treaties. In this, he was joined by powerful, deep-pocketed lobbyists from Big Content, and later, Big Tech. They successfully pressured Congress into passing Section 1201 of the Digital Millennium Copyright Act in 1998 – a law that protects digital locks, at the expense of copyright and the creative workers whom copyright is said to serve.

Lehman has repeatedly, publicly described this maneuver as "doing an end-run around Congress." Once America adopted this extreme anticircumvention rule, the US Trade Representative made it America's top priority to ram identical laws through the legislatures of all of America's trading partners, under the explicit or tacit threat of tariffs on any country that refused (the information minister of a Central American country once told me that the USTR threatened them, saying that if they didn't accept anticircumvention as a clause in the Central American Free Trade Agreement – CAFTA – they would lose their ability to export soybeans to America).

Canada took more than a decade to enact its own version of the anticircumvention rule, which was the source of public outrage by the USTR and US industry lobbyists. These neocolonialists found plenty of Parliamentary sellouts willing to introduce laws on their behalf, but every time this happened, the Canadian people reacted with a kind of mass outrage that had never been seen in response to highly technical proposals for internet regulation. For example, the Liberal MP Sam Bulte was challenged on her support of the rule by her Parkdale constituents at a public meeting, and had a screeching meltdown, screaming that she would not be "bullied by user-rights zealots and EFF members." Voters put "User-Rights Zealot" signs on their lawns and voted her out of office.

Anticircumvention remained a priority for the US, and they found new MPs to do their dirty work. Stephen Harper's Conservatives made multiple tries at this. After Jim Prentice utterly failed to get the rule through Parliament, the brief was picked up by Heritage Minister James Moore (who liked to call himself "the iPad Minister") and now-disgraced Industry Minister Tony Clement. Clement and Moore tried to diffuse the opposition to the proposal by conducting a public consultation on it.

This backfired horribly. Over 6,000 Canadians wrote into the consultation with individual, detailed, personal critiques of anticircumvention, explaining how the rule would hurt them at work and at home. Only 53 submissions supported the rule. Moore threw away these 6,130 negative responses, justifying this by publicly calling them the "babyish" views of "radical extremists":

https://pluralistic.net/2024/11/15/radical-extremists/#sex-pest

Named individuals created policies in living memory. They were warned about the foreseeable outcomes of those proposals. They passed them anyway – and then no one held them accountable.

Until now.

The point of remembering where these policies came from isn't (merely) to ensure that these people are forever remembered as the monsters they showed themselves to be. Rather, it is to recover the true history of enshittification, the choices we made that led to enshittification, so that we can reverse those policies, disenshittify our tech, and give rise to a new, good internet that's fit for the purpose of being the global digital nervous system for a species facing a polycrisis of climate catastrophe, oligarchy, fascism and genocide.

There's never been a more urgent moment to reconsider those enshittificatory policies – and there's never been a more auspicious moment, either. After all, Canada's anticircumvention law exists because it was supposed to guarantee tariff-free access to American markets. That promise has been shattered, permanently. It's time to get rid of that law, and make it legal for Canadian technologists to give the Canadian public the tools they need to escape from America's Big Tech bullies, who pick our pockets with junk-fees and lock-in, and who attack our social, legal and civil lives with social media walled gardens:

https://pluralistic.net/2025/01/15/beauty-eh/#its-the-only-war-the-yankees-lost-except-for-vietnam-and-also-the-alamo-and-the-bay-of-ham

"Understood: Who Broke the Internet" is streaming now. We've got three more episodes to go – part two drops on Monday (and it's a banger). You can subscribe to it wherever you get your podcasts, and here's the RSS feed:

https://www.cbc.ca/podcasting/includes/nakedemperor.xml


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Self-referential multiple-choice test https://web.archive.org/web/20050126084907/https://math.wisc.edu/~propp/srat-Q

#15yrsago Big Content’s depraved indifference https://memex.craphound.com/2010/05/08/big-contents-depraved-indifference/

#15yrsago Use rust particles to reveal the data on your credit-card’s magstripe https://www.tetherdcow.com/another-science-experiment/

#15yrsago FCC hands Hollywood the keys to your PC, home theater and future https://memex.craphound.com/2010/05/07/fcc-hands-hollywood-the-keys-to-your-pc-home-theater-and-future/

#15yrsago Walt Disney World’s Haunted Mansion: stupendous essay https://passport2dreams.blogspot.com/2010/05/history-and-haunted-mansion.html

#15yrsago Eating IHOP’s cheesecake-stuffed pancakes https://www.salon.com/2010/05/07/ihop_pancake_stackers_the_new_kfc_double_down/

#10yrsago Drug pump is “most insecure” devices ever seen by researcher https://securityledger.com/2015/05/researcher-drug-pump-the-least-secure-ip-device-ive-ever-seen/

#10yrsago Appeals Court rejects NSA’s bulk phone-record collection program https://www.eff.org/press/releases/eff-hails-court-ruling-rejecting-nsa-bulk-collection-americans-phone-records

#10yrsago Keurig CEO blames disastrous financials on DRM https://money.cnn.com/2015/05/06/investing/keurig-green-mountain-earnings-stock-fall/index.html

#5yrsago Volcano gods demand workers https://pluralistic.net/2020/05/08/volcano-gods/#reopening

#5yrsago US public health officials on apps: "Meh" https://pluralistic.net/2020/05/08/volcano-gods/#shoe-leather

#5yrsago Wechat spies on non-Chinese users for in-China censorship https://pluralistic.net/2020/05/07/just-look-at-it/#training-data

#5yrsago Sidewalk Labs pulls out of Toronto https://pluralistic.net/2020/05/07/just-look-at-it/#ding-dong

#5yrsago Unix and Adversarial Interoperability https://pluralistic.net/2020/05/07/just-look-at-it/#multics

#5yrsago EU: "Cookie walls violate the GDPR" https://pluralistic.net/2020/05/07/just-look-at-it/#cookie-theatre

#5yrsago Helicopter flyover of deserted Disneyland https://pluralistic.net/2020/05/07/just-look-at-it/#emptyland

#5yrsago Wink will brick your smart home if you don't pay a monthly fee <a #ebook"="" 05="" 07="" 2020="" href="https://pluralistic.net/2020/05/07/just-look-at-it/#wink'>https://pluralistic.net/2020/05/07/just-look-at-it/#wink</a>

#5yrsago EFF's Guide to Digital Rights During the Pandemic <a href=" https:="" just-look-at-it="" pluralistic.net="">https://pluralistic.net/2020/05/07/just-look-at-it/#ebook

#5yrsago America is united https://pluralistic.net/2020/05/07/just-look-at-it/#national-unity

#5yrsago The TSA is hoarding N95s https://pluralistic.net/2020/05/07/just-look-at-it/#taking-shit-away

#1yrago The disenshittified internet starts with loyal "user agents" https://pluralistic.net/2024/05/07/treacherous-computing/#rewilding-the-internet


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-05-08T00:28:37+00:00 Fullscreen Open in Tab
Published on Citation Needed: "Meet Trump’s memecoin dinner guests"
Wed, 07 May 2025 12:51:57 +0000 Fullscreen Open in Tab
Pluralistic: Mark Zuckerberg announces mind-control ray (again) (07 May 2025)


Today's links



A portrait of a bearded, glaring Rasputin. His face has been replaced with Mark Zuckerberg's metaverse avatar; the pupils of the avatar's eyes have been replaced with the glaring red eye of HAL 9000 from Kubrick's '2001: A Space Odyssey.'

Mark Zuckerberg announces mind-control ray (again) (permalink)

Mark Zuckerberg has told investors how he plans to make back the tens of billions he's spending on AI: he's going to use it to make advertisements that can bypass our critical faculties and convince anyone to buy anything. In other words, Meta will make an AI mind-control ray and rent it out to grateful advertisers.

Here, Zuck is fulfilling the fundamental duty of every CEO of every high-growth tech company: explaining how his company will continue to grow. These growth stories are key, because growth stocks trade at a huge premium relative to the stocks of "mature" companies. Every dollar Meta brings in boosts their share price to a much greater degree than the dollars earned by companies with similar rates of profit, but slower rates of growth. This premium represents a bet by investors that Meta will continue to grow, which means that the instant Meta stops growing, the value of its shares will plummet, to reflect the fact that it is a "mature" company, not a "growth" company.

So Zuck needs to do everything he can to keep investors believing that Meta will continue to grow. After all, Zuck's key employees and top managers all take much (or even most!) of their compensation in Meta stock, which means that the instant the company stops growing, those workers' pay will plummet and they will seek employment elsewhere, depriving Meta of the workers it needs to successfully create or conquer a new market and once again become a growth stock.

This is why Zuck keeps telling stories. The most important story Zuck tells is about himself, the boy genius who converted a tool for nonconsensually rating the fuckability of Harvard undergrads into a social media monopoly with four billion users. Zuck's cult of personality isn't the product of mere narcissism – it's a tool for creating the material conditions for ongoing investor confidence:

https://www.businessinsider.com/mark-zuckerberg-shirt-latin-what-does-it-say-explained-words-2024-9

If Zuck is a boy genius, then Zuck's pronouncements take on the character of prophesy. When Zuck announced the "pivot to video," investors poured tens of billions into Facebook stock and into video-first online news production, despite the fact that Zuck was obviously lying:

https://slate.com/technology/2018/10/facebook-online-video-pivot-metrics-false.html

The "boy genius" story is an example of Silicon Valley's storied "reality distortion field," pioneered by Steve Jobs. Like Jobs, Zuck is a Texas marksman, who fires a shotgun into the side of a barn and then draws a target around the holes. Jobs is remembered for his successes, and forgiven his (many, many) flops, and so is Zuck. The fact that pivot to video was well understood to have been a catastrophic scam didn't stop people from believing Zuck when he announced "metaverse."

Zuck lost more than $70b on metaverse, but, being a boy genius Texas marksman, he is still able to inspire confidence from credulous investors. Zuck's AI initiatives generated huge interest in Meta's stock, with investors betting that Zuck would find ways to keep Meta's growth going, despite the fact that AI has the worst unit economics of any tech venture in living memory. AI is a business that gets more expensive as time goes on, and where the market's willingness to pay goes down over time. This makes the old dotcom economics of "losing money on every sale, but making it up in volume" look positively rosy:

https://www.wheresyoured.at/reality-check/

Now, Zuck has finally described how he's going to turn AI's terrible economics around: he's going to ask AI to design his advertisers' campaigns, and these will be so devastatingly effective that advertisers will pay a huge premium to advertise on Meta:

https://finance.yahoo.com/news/the-ai-revolution-is-an-advertising-revolution-morning-brief-100001467.html

This narrative is especially galling because it's literally the same story Zuck has been telling for decades: "Facebook has built a mind-control ray out of Big Data, and we can sell anything to anyone":

https://pluralistic.net/2021/09/30/dont-believe-the-criti-hype/#ordinary-mediocrities

This is a facially absurd proposition. After all, everyone who's ever claimed to have perfected mind-control – Rasputin, Mesmer, MK-ULTRA, neurolinguistic programming grifters and pathetic "pick up artists" – was a liar. Either they were lying to themselves, or to everyone else. Or both.

But many of tech's critics helped sell this narrative (and thus helped Meta sell ads). Many critics have fallen prey to the sin of "criti-hype," Lee Vinsel's term for critiquing the claims of your adversary without bothering to ask whether they are true:

https://pluralistic.net/2021/02/02/euthanize-rentiers/#dont-believe-the-hype

The project of convincing investors that tech's "dopamine hackers" had perfected mind-control with warmed over, non-replicable Skinnerian behavior-mod techniques and mass surveillance sold a hell of a lot of ads. After all, if there's one kind of person the advertising sector has always been able to sell to, it's advertising executives, who are the easiest of marks for a story about how easy it is to trick the public into buying whatever you're selling:

https://pluralistic.net/2020/10/05/florida-man/#wannamakers-ghost

Every ad-tech sales-bro who takes a meeting with an advertising executive finds himself pushing on an open door. Advertisers desperately wants to believe in mind-control rays. Think of the department store magnate John Wannamaker, who said, "half my advertising spending is wasted – I just don't know which half." Imagine: some advertising exec convinced John Wannamaker that he was only wasting half of his advertising spending!

I've long maintained that the threat from AI to workers isn't that AI can do your job – it's that an AI salesman can convince your boss to fire you and replace you with an AI that can't do your job:

https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#government-by-spicy-autocomplete

The corollary here is that it doesn't matter if AI can design ads that work, not so long as an AI ad salesman can sell this proposition to an advertisers, and not so long as a tech CEO can sell it to investors.

AI keeps passing the worst kinds of Turing tests – for example, it's great at helping people who are prone to life-destroying hallucinations that they are talking to God:

https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/

Zuck kept up his growth story with this mind control narrative for more than a decade, got caught committing a string of spectacular frauds, and then lured investors back into his stock offerings by telling the same story. This isn't just an indictment of Zuck, it's a stinging rebuke to the whole idea that markets are a kind of infallible computer for assessing and operationalizing information. The market's "thought process" demonstrably lacks the object permanence that most babies acquire by the time they are a year old. You can tell when your child has acquired object permanence by the fact that they cease to enjoy "peek-a-boo" (object permanence means they understand where you have gone when your face is hidden).

In claiming that AI will give him an infinite growth mind-control ray, Mark Zuckerberg is challenging the market to a game of peek-a-boo – and he's winning.

(Image: Cryteria, CC BY 3.0, modified)


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago V-TV DAY: WE WON THE BROADCAST FLAG FIGHT! https://memex.craphound.com/2005/05/06/v-tv-day-we-won-the-broadcast-flag-fight/

#20yrsago Google Accelerator is bad news for Web apps https://signalvnoise.com/archives2/google_web_accelerator_hey_not_so_fast_an_alert_for_web_app_designers

#15yrsago UNESCO’s bizarre World Anti-Piracy Observatory https://web.archive.org/web/20100308001509/http://portal.unesco.org/culture/en/ev.php-URL_ID=39055&URL_DO=DO_TOPIC&URL_SECTION=201.html

#15yrsago Naked scanner reveals airport screener’s tiny penis, sparks steel baton fight with fellow officers https://www.staugustine.com/story/news/nation-world/2010/05/08/airport-screener-charged-attack-over-daily-jokes/16232427007/

#15yrsago India’s e-voting machines vulnerable to fraud https://indiaevm.org

#15yrsago Roy Lichtenstein’s estate claims copyright over the images he appropriated <a "="" 05="" 06="" 2015="" balkaninsight.com="" href="https://web.archive.org/web/20100509220603/http://www.elsinoremusic.net/'>https://web.archive.org/web/20100509220603/http://www.elsinoremusic.net/</a>

#10yrsago Mass protests, brutal crackdown in Macedonia <a href=" https:="" macedonia-police-quells-anti-government-protest="">https://balkaninsight.com/2015/05/06/macedonia-police-quells-anti-government-protest/

#10yrsago Which UK MPs rebel against the party, and with whom do they ally? https://vartree.blogspot.com/2015/05/vizualizing-rebel-alliances-in-uk.html

#10yrsago Legal threat against security researcher claims he violated lock’s copyright https://arstechnica.com/information-technology/2015/05/lawyers-threaten-researcher-over-key-cloning-bug-in-high-security-lock/

#10yrsago Talent, practice and doing the hard stuff https://locusmag.com/2015/05/cory-doctorow-shorter/

#5yrsago Ohio's got snitchline for bosses whose workers who won't go back https://pluralistic.net/2020/05/06/moloch-demands-death/#life-or-death

#5yrsago Appeals court says Miami jail doesn't need to provide soap https://pluralistic.net/2020/05/06/moloch-demands-death/#death-sentence

#5yrsago Scarfolk death statistics https://pluralistic.net/2020/05/06/moloch-demands-death/#scarfolk

#5yrsago Sacrifice banks to save businesses https://pluralistic.net/2020/05/06/moloch-demands-death/#no-choice

#5yrsago America's corporate lobbyists want a bailout https://pluralistic.net/2020/05/06/moloch-demands-death/#human-centipede

#1yrago Amazon illegally interferes with an historic UK warehouse election https://pluralistic.net/2024/05/06/one-click-to-quit-the-union/#foxglove


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources: Slashdot (https://slashdot.org/).

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-05-06T21:37:29+00:00 Fullscreen Open in Tab
Note published on May 6, 2025 at 9:37 PM UTC

A trader bought 6,000 $TRUMP on April 21, two days before the announcement that top holders would be invited to dinner with the president. They sold it a few hours after the announcement two days later caused the token price to go up, earning $33k on the trade. Impressive timing!

Solscan screenshot showing the trader purchased 6,041.52 TRUMP for $51,051.35 in USDT on April 21
Solscan screenshot showing the trader sold 6,041.52 TRUMP for $84,087.42 in USDT on April 21

They had traded the Trump token a bit for the first few days after it launched in January, then apparently lost interest until suddenly rediscovering it on April 21.

Image

At the moment they still hold 9,130 TRUMP, so as of now they’ll be joining Trump at the dinner in a few weeks.

They regularly transfer funds from Bybit, suggesting they are not based in the US.

Tue, 06 May 2025 15:35:01 +0000 Fullscreen Open in Tab
Pluralistic: The Adventures of Mary Darling (06 May 2025)


Today's links



The cover of the Tachyon edition of 'The Adventures of Mary Darling.'

The Adventures of Mary Darling (permalink)

Science fiction great Pat Murphy has written some classics – including books that were viciously suppressed by the heirs of JRR Tolkien! – but with The Adventures of Mary Darling, she's outdone even her own impressive self:

https://tachyonpublications.com/product/the-adventures-of-mary-darling/

The titular Mary Darling here is the mother of Wendy, John and and Michael Darling, the three children who are taken by Peter Pan to Neverland in JM Barrie's 1902 book The Little White Bird, which later became Peter Pan. If you recall your Barrie, you'll remember that it ends with the revelation that Wendy, John and Michael weren't the first Darlings to go to Neverland: when Mary Darling was a girl, she, too, made the journey.

Murphy's novel opens with Mary Darling and her husband George coming home from a dinner party to discover their three children missing, the window open, and their nanny, a dog called Nana, barking frantically in the yard. John is frightened, but Mary is practically petrified, inconsolable and rigid with fear.

Soon, Mary's beloved uncle, John Watson, is summoned to the house, along with his famous roommate, the detective Sherlock Holmes. With Holmes on the case, surely the children will be found?

Of course not. Holmes is incapable of understanding where the Darling children have gone, because to do so would be to admit the existence of the irrational and fantastic, and, more importantly, to accept the testimony of women, lower-class people, and pirates. Holmes has all the confidence of the greatest detective alive, which means he is of no help at all.

Neither is George Darling, who, as a kind of act of penance for letting his children be stolen away, takes to Nana's doghouse, and insists that he will not emerge from it until the children are returned. He takes his meals in the doghouse, and is carried in it to and from the taxis that bring him to work and home again.

Only Mary can rescue her children. John Watson discovers her consorting with Sam, a one-legged Pacific Islander who is a known fence and the finest rat-leather glovemaker in London, these being much prized by London's worst criminal gangs. Horrified that Mary is keeping such ill company, Watson confronts her and Sam (and Sam's parrot, who screeches nonstop piratical nonsense), only to be told that Mary knows what she is doing, and that she is determined to see her children home safe.

Mary, meanwhile, is boning up on her swordplay and self-defense (taught by a Suffragist swordmaster in a room above an Aerated Bread Company tearoom, these being the only public place in Victorian London where a respectable woman can enjoy herself without a male escort). She's acquiring nautical maps. She's going to Neverland.

What follows is a very rough guide to fairyland. It's a story that recovers the dark asides from Barrie's original Pan stories, which were soaked with blood, cruelty and death. The mermaids want to laugh as you drown. The fairies hate you and want you to die. And Peter Pan doesn't care how many poorly trained Lost Boy starvelings die in his sorties against pirates, because he knows where there are plenty more Lost Boys to be found in the alienated nurseries of Victorian London, an ocean away.

More importantly, it's a story that revolves around the women in Barrie's world, who are otherwise confined to the edges and shadows of the action. In Barrie's Pan, Wendy is a "mother," Tiger Lily is a "princess," and Mary is a barely-there adult whose main role is to smile wistfully at the memory of when she was a girl and got to serve as Peter's "mother."

And Holmes? Apart from one love interest and a stalwart housekeeper, Holmes has very little time or regard for women. This is so central to the Holmes cannon that the Arthur Conan Doyle estate actually sued over Netflix's Enola Holmes movie, arguing that Enola displayed basic respect for women, a feature that doesn't appear until the very end of the Holmes canon, and – the estate argued – those final stories were still in copyright:

https://www.cbr.com/why-enola-holmes-has-nice-version-sherlock/

Murphy's woman's-eye-view of Peter Pan, Neverland and the Lost Boys dilates the narrow aperture through which Peter Pan plays out, revealing a great deal of exciting, fun, frightening stuff that was always off in the wings. She gives flesh and substance to characters like Tiger Lily, by giving her the semi-fictionalized identity of one of the many American First Nations people who toured Europe and Africa, putting on Wild West shows that won eternal fame and cultural currency for the "American Indian," even as the USA was seeking to exterminate them and their memory.

Likewise, Murphy's pirates are grounded in the reality of pirate ships: democratic, anarchic, and far more fun than Robert Louis Stevenson would have you believe. While Murphy's pirates are about a century too late (as are Barrie's), they are in other regards pretty rigorous, which makes them extraordinarily great literary figures.

If you read David Graeber's posthumous Pirate Enlightenment, you'll know about the Zana-Malata of Madagascar, the descendants of anarchist pirates and matriarchal Malagasy women, who pranked and hoaxed British merchant sailors for generations, deliberately creating a mythology of south seas pirate kings:

https://pluralistic.net/2023/01/24/zana-malata/#libertalia

This hybrid culture of bold, fierce matriarchal Malagasy women and their anarchist pirate husbands play a central role in the book's resolution, and Murphy's pirate utopia is so well drawn and homely that I found myself wanting to move there.

This is a profoundly political book, but it's such a romp, too! Murphy has a real flair for this kind of thing. Back in 1999, she published the brilliant There and Back Again, an all-female retelling of The Hobbit (in spaaaaace!) that was widely celebrated…right up to the moment that Christopher Tolkien used baseless copyright threats to get the book withdrawn from sale:

https://en.wikipedia.org/wiki/There_and_Back_Again_(novel)

Billionaire failsons of long-dead writers notwithstanding, you can still read There and Back Again by borrowing a copy of the book from the Internet Archive's Open Library:

https://openlibrary.org/works/OL15436385W/There_and_back_again

Murphy's mashup of Holmes, Pan, South Seas pirate anarchists, and other salutary and exciting personages, milieux, furniture and tropes of the Victorian adventure story is an unmissable triumph, a romp, a delight.


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Space Needle to be converted to WiFi antenna https://web.archive.org/web/20050506113417/http://www.komotv.com/stories/36658.htm

#20yrsago How tech could replace the US healthcare system https://web.archive.org/web/20050427012918/http://www.wired.com/wired/archive/13.05/view.html?pg=5

#20yrsago DRM and music research https://web.archive.org/web/20060106133157/http://blogs.sun.com/roller/page/plamere/20050505#listen_only_music

#15yrsago How I got phished https://locusmag.com/2010/05/cory-doctorow-persistence-pays-parasites/

#15yrsago Stross explains why he’s voting LibDem https://www.antipope.org/charlie/blog-static/2010/05/party-election-broadcast.html

#15yrsago HOWTO Tell a debt-collector to go to hell https://web.archive.org/web/20100507214045/https://consumerist.com/2010/05/this-is-how-you-tell-a-zombie-debt-collector-to-buzz-off.html

#15yrsago Scholarly essay nails Gilligan’s Island’s hidden subtext https://web.archive.org/web/20100508025302/http://www.fightthebias.com/Resources/Humor/island.htm

#15yrsago Canadian Prime Minister promises to enact a Canadian DMCA in six weeks https://web.archive.org/web/20100508202357/https://www.michaelgeist.ca/content/view/5008/125/

#10yrsago House Republicans hold hearing on politics in science, don’t invite any scientists https://www.science.org/content/article/house-panel-holds-hearing-politically-driven-science-sans-scientists

#5yrsago Teen Vogue on socialist feminism https://pluralistic.net/2020/05/05/the-hard-stuff/#wages-for-housework

#5yrsago A federal jobs guarantee https://pluralistic.net/2020/05/05/the-hard-stuff/#jobs-guarantee

#5yrsago Pandemic profiteering could create social chaos https://pluralistic.net/2020/05/05/the-hard-stuff/#pandemic-profiteers

#5yrsago Leaked Trump doc projects 3000 US deaths/day https://pluralistic.net/2020/05/05/the-hard-stuff/#lethal-incompetence

#5yrsago Negativland's "This is Not Normal" https://pluralistic.net/2020/05/05/the-hard-stuff/#letter-u-numeral-2

#5yrsago What "writing rules" actually mean https://pluralistic.net/2020/05/05/the-hard-stuff/#said-bookism


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-05-05T16:45:47+00:00 Fullscreen Open in Tab
Note published on May 5, 2025 at 4:45 PM UTC
Mon, 05 May 2025 13:52:07 +0000 Fullscreen Open in Tab
Pluralistic: Bridget Read's 'Little Bosses Everywhere' (05 May 2025)


Today's links



The Penguin Random House cover for Bridget Read's 'Little Bosses Everywhere.'

Bridget Read's 'Little Bosses Everywhere' (permalink)

Pyramid schemes are as American as apple pie. If you doubt it, just read Little Bosses Everywhere, Bridget Read's deeply researched, horrifying, amazing investigative book on the subject, which is out today from Crown:

https://www.penguinrandomhouse.com/books/715421/little-bosses-everywhere-by-bridget-read/

Read, an investigative journalist at Curbed, takes us through the history of the "industry," which evolved out of Depression-era snake oil salesmen, Tupperware parties, and magical thinking cults built around books like Think and Grow Rich. This fetid swamp gives rise to a group of self-mythologizing scam artists who found companies like Amway and Mary Kay, claiming outlandish – and easily debunked – origin stories that the credulous press repeats, alongside their equally nonsensical claims about the "opportunities" they are creating for their victims.

In Read's telling, there's only two kinds of MLM participants: suckers (who lose lots and lots of money) and predators (who rake in that money). MLMs pretend that they're doing "direct sales," cutting out the middleman to peddle vitamins, household cleaners, cosmetics, tights or jewelry. But the actual sales volume of these products rounds to zero. The money in the system – tens of billions of dollars per year in the US alone – is almost entirely being spent by "salespeople" who are required to buy a certain amount of "product" every month, either as a condition of membership, or in order to attain some kind of bonus or status.

The "salespeople" in these systems are effectively in a cult, and the high-pressure techniques that Read describes will be instantly recognizable to anyone familiar with cultic dynamics, or even just a casual listener to the Conspirituality podcast:

https://www.conspirituality.net/episodes

And, as with other cults, MLM members are tormented endlessly by other cult members into trying to recruit their friends and family-members. Sometimes, they succeed, and the cult grows a little – but usually not for very long. Most people who get recruited into an MLM quickly figure out that it's impossible to make any money – indeed, it's impossible to avoid losing a lot of money – and bail.

The meat-and-potatoes of the MLM industry are the minority who don't see through the scam. They believe that they are deficient, because everyone else is reporting such incredible returns from "the program." They charge more product to their credit cards, insisting to their "uplines" that they are selling machines (and not that they are filling their garages and attics and living rooms and kitchen cupboards with unsold, unsellable junk). What they don't understand is that all the "successes" in the cult are either scammers who are getting rich off people like them, or they are people like them, going deep into debt and desperately trying to pretend that they're selling as well as those uplines.

The US government and various law enforcement agencies have taken various runs at these cults, but the cults have always won. That's down to enforcers buying into the cult leader/scammers' essential lie: that, at the end of the day, MLM is a system for selling things to people. That isn't true, has never been true, and never will be true. But by crafting rules and tests that attempt to sort the "legitimate" MLMs from the "scam" MLMs, enforcers fall into the scammers' trap. The scammers welcome rules that distinguish "good" MLMs from "bad" MLMs, because it's trivial to create the superficial appearance of adherence to these rules while flouting them. For example, if the rule says that "independent sales representatives" must sell to at least ten outside customers, they can simply make up the names of ten people and charge it to their card. This happens routinely, but there's no auditing, and besides, the MLM victims are all "independent business owners," so if there were any penalties for these violations, they would fall to the victims, not the cult.

Meanwhile, the scammers know it's a scam, and the failure of their victims to sell the useless "product" the cult is nominally organized around is a feature, not a bug. The hordes of indebted, cost-sunk, self-castigating failures are suckers for yet another scam: selling victims "training" to improve their sales technique. After all, if everyone around you is selling this crap without breaking a sweat, the failing must be your own. You need coaching, training, seminars, cassettes, books, retreats, all of it piling debt on debt.

The internal operations of these cults are shrouded in mystery, but Read lifts the veil and makes masterful sense of the horrors lurking beneath. In this, she is somewhat aided by MLM cult leaders' propensity for suing one another, as various sub-bosses build up massive followings of their own and seek to usurp the cult leader by founding their own parallel cults or sub-cults. These lawsuits sometimes drag the cults' dirty laundry out in public, and Read sorts through these court filings very carefully. Unfortunately, the cults' propensity for suing also helps suppress a lot of dirty laundry, because MLM leaders love to sue ex-cult members who participate in online forums where they document their expenses, and they use these cult victims' own money to pay for the court cases that silence them.

MLMs aren't just cults, they're religious cults. Since the very earliest days, pyramid scheme runners have declared themselves to be engaged in an extension of their Christian (mostly Calvinist) faith. The engine of a pyramid scheme needs social capital for fuel: to bring in new recruits, a cult member has to draw on the bonds of trust, fellowship and solidarity in order to convince their targets that this is a bona fide enterprise (and not a cult). Faith groups – especially fringe faith groups – have this kind of capital in spades. This goes double for faiths that demand large families (which is why we see such deep penetration of MLMs into Mormonism and orthodox Judiasm). If your faith demands that you produce a "quiverfull" of mouths to feed, then the chances are that you will not be able to survive without being enmeshed in a mutual support network with your co-religionists. MLMs convert this trust, generosity and mutual dependency into cash (at a ruinous exchange rate) and then funnel it "upline" to the cult leaders, who reap billions.

Of course, those kinds of bonds are not solely forged on the basis of faith: racialized people, women, and other groups who face systemic discrimination depend on one another for mutual aid, which makes them vulnerable to another MLM pitch: "predatory inclusion":

https://pluralistic.net/2023/09/27/predatory-inclusion/#equal-opportunity-scammers

Predatory inclusion is when scam artists adopt the language of social justice to pitch their cons – think of all the crypto bros who sold their ripoff schemes as a way to "achieve independence for women" or "build Black wealth" (thanks, Spike Lee):

https://www.vice.com/en/article/spike-lee-made-an-ad-for-cryptocurrency-atms-and-its-bizarre/

Predatory inclusion is parasitic upon the bonds of solidarity forged in adversity, and this goes double for the MLM variety. As MLMs cut away the strands of the web of mutual support, the cult leaders replace them with rabid anti-Communism, the kind of far-right rhetoric that brought Christian conservatives into the Reagan coalition and ultimately led to Trump's fascist takeover.

Here's how that move works: "You are a small, independent businessperson, the backbone of America. You will realize the American dream through your own backbone and work ethic (and therefore your current failure is due to your own lack of both). People who want to shut down pyramid schemes say they want to protect you, but really they want the government to decide who can and can't own a business. They're Communists, and in coming for MLMs, they're coming for America itself."

Some of America's richest family dynasties owe their wealth to pyramid schemes. They are dynasties of fraud, and they funneled their criminal gains into far right political projects. The Heritage Foundation – the authors of Project 2025 and Trump's master strategists – got their start with money from Rich DeVos (father in law of Betsy DeVos, who served as Secretary of Education in the first Trump cabinet). The far-right dark money machine runs on MLM money.

In fact, there's a good case to be made that everything rotten in today's world is built on the tactics of MLMs. Take the "gig economy." Companies like Uber promise drivers a high hourly wage. A small number of drivers are randomly allocated extremely large payouts by the system, in order to convert them into Judas goats, who fill gig-work message boards with tales of their good fortune. As Veena Dubal documents in her seminal work on "algorithmic wage discrimination," this tactic is devastatingly effective, convincing other Uber drivers to put in extremely long hours for sub-starvation wages, and then blame themselves for "being bad at Uber" – just like the downlines at Mary Kay and Amway who think the problem is with them:

https://pluralistic.net/2023/04/12/algorithmic-wage-discrimination/#fishers-of-men

Trump, of course, is the ultimate expression of the MLM grift – and not only because he licensed his name to two different pyramid schemes. Trump embodies the MLM ethic of lying about how rich you are so that marks send you their money to get in on the "opportunity" and then blame themselves when the promised riches never materialize.

Erik Baker once described MLMs as a kind of bizarro-world version of unions. In the world of labor organizing, success lies in finding the people with the most social capital, the ones who are trusted by their coworkers, and teaching them to have a structured organizing conversation. This is exactly what MLMs do – but the difference lies in the goal of that structured organizing conversation. For union organizers, the goal is build solidarity as a means to improving the lives of everyone in the community. For MLM organizers, the goal is to destroy solidarity, atomizing the community, shattering its bonds, leaving its members defenseless as they are fleeced by the cult's leaders and their henchmen:

https://pluralistic.net/2025/02/05/power-of-positive-thinking/#the-socialism-of-fools

Neoliberalism's war-cry is Thatcher's "There is no such thing as society." The past 40 years have been a long process of tearing us away from one another, teaching us to see one another as marks, to mistrust systems of mutual aid as Communism. Read's Little Bosses Everywhere is a brilliantly told, deeply researched history of the past and present of the ultimate business model for late-stage capitalism: destroying the lives of everyone around you while pretending to be a small businessperson.


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Haunted Mansion executed in sand https://web.archive.org/web/20050507151039/http://strangecosmos.com/content/item/101751.html

#20yrsago Ukranian TV sign-language interpreter blew whistle on crooked vote https://web.archive.org/web/20050503154101/http://www.sun-sentinel.com/news/nationworld/sfl-ainterpreter01may01,0,341946.story?coll=sfla-news-nationworld

#20yrsago 3D printer made from Meccano and hot-glue <a 04="" 2005="" brazil.aids"="" href="https://blog.rebang.com/?p=101'>https://blog.rebang.com/?p=101</a>

#20yrsago Brazil rejects Bush’s faith-based AIDS money <a href=" https:="" may="" world="" www.theguardian.com="">https://www.theguardian.com/world/2005/may/04/brazil.aids

#20yrsago Boy Scout badge in Intellectual Property https://web.archive.org/web/20050522080926/http://news.com.com/2061-10796_3-5693563.html">

#15yrsago Satellite photos catch Greek tax-evaders https://www.nytimes.com/2010/05/02/world/europe/02evasion.html

#15yrsago Six reasons to hate Facebook’s new anti-privacy system, “Connections” https://www.eff.org/deeplinks/2010/05/things-you-need-know-about-facebook

#15yrsago Viacom is becoming a lawsuit company instead of a TV company https://www.theguardian.com/technology/blog/2010/may/04/viacom-youtube

#15yrsago Bruce Schneier explains how to prevent terrorist attacks like Times Square car-bomb https://archive.nytimes.com/roomfordebate.blogs.nytimes.com/2010/05/03/times-square-bombs-and-big-crowds/

#15yrsago London cops’ signs in Internet cafe warn against loading porn and “extremism” https://www.flickr.com/photos/doctorow/4581218328/

#15yrsago Windsor Executive Solutions: Bruce Sterling and Chris Nakashima-Brown’s transhuman monarch story https://futurismic.com/2010/05/04/new-fiction-windsor-executive-solutions-by-chris-nakashima-brown-and-bruce-sterling/

#15yrsago BBC to project real-time election results on Big Ben’s tower https://news.bbc.co.uk/2/hi/uk_news/politics/election_2010/8656578.stm

#15yrsago Terry Pratchett: Doctor Who isn’t science fiction https://web.archive.org/web/20100506141559/http://www.sfx.co.uk/2010/05/03/guest-blog-terry-pratchett-on-doctor-who/

#15yrsago HOWTO live out of your car https://www.reddit.com/r/reddit.com/comments/byaac/comment/c0p5qkh/

#15yrsago Star Wars Loteria tribute https://web.archive.org/web/20100505204544/http://www.chepo.net/

#10yrsago FBI replies to Stingray Freedom of Information request with 5,000 blank pages https://www.techdirt.com/2015/05/01/fbi-hands-over-5000-pages-stingray-info-to-muckrock-redacts-nearly-all-it/,/a>

#10yrsago Court says DEA is allowed to secretly fill your truck with weed, get into firefights with Zetas https://www.techdirt.com/2015/05/01/who-pays-when-dea-destroys-your-vehicle-kills-your-employee-during-botched-sting-hint-not-dea/

#10yrsago Disrupting Richard Scarry https://welcometobusinesstown.tumblr.com

#10yrsago Every question in every Q&A session https://web.archive.org/web/20150502151150/http://the-toast.net/2015/05/01/every-question-in-every-qa-session-ever/

#10yrsago Algorithmic guilt: using secret algorithms to kick people off welfare https://slate.com/technology/2015/04/the-dangers-of-letting-algorithms-enforce-policy.html

#5yrsago Kim Stanley Robinson on "our rewritten imagination" https://pluralistic.net/2020/05/02/rewriting-our-imaginations/#ksr

#5yrsago Ticketmaster gets $500m from Mohammad bin Salman <a <a="" apps="" be="" contact="" could="" href="https://pluralistic.net/2020/05/02/rewriting-our-imaginations/#exposure-notification" than="" tracing"="" useless="" worse="">https://pluralistic.net/2020/05/02/rewriting-our-imaginations/#exposure-notification

#5yrsago Prisons, meatpacking plants, nursing homes https://pluralistic.net/2020/05/02/rewriting-our-imaginations/#superclusters

#5yrsago My talk at Republica online https://pluralistic.net/2020/05/02/rewriting-our-imaginations/#republica

#5yrsago The Making of Prince of Persia https://pluralistic.net/2020/05/03/give-me-slack/#pop

#5yrsago Lockdown CO2 and structural roots of the climate emergency https://pluralistic.net/2020/05/03/give-me-slack/#reality-is-a-leftist

#5yrsago XML inventor quits Amazon over whistleblower firings https://pluralistic.net/2020/05/04/which-side-are-you-on/#tim-bray

#5yrsago The failure of software licensing https://pluralistic.net/2020/05/04/which-side-are-you-on/#tivoization-and-beyond

#5yrsago Wired workers have unionized https://pluralistic.net/2020/05/04/which-side-are-you-on/#solidarity

#5yrsago Hospital CEOs making millions amid cuts https://pluralistic.net/2020/05/04/which-side-are-you-on/#administrative-bloat

#5yrsago Pandemic could make Big Tech our permanent overlords https://pluralistic.net/2020/05/04/which-side-are-you-on/#trustbusting

#1yrago Rosemary Kirstein's "The Steerswoman" https://pluralistic.net/2024/05/04/the-wulf/#underground-fave

#1yrago Google is (still) losing the spam wars to zombie news-brands https://pluralistic.net/2024/05/03/keyword-swarming/#site-reputation-abuse

#1yrago CDA 230 bans Facebook from blocking interoperable tools https://pluralistic.net/2024/05/02/kaiju-v-kaiju/#cda-230-c-2-b


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

Fri, 02 May 2025 20:28:37 +0000 Fullscreen Open in Tab
Pluralistic: AI and the fatfinger economy (02 May 2025)


Today's links



A leg-hold trap whose trigger disc has been replaced with the hostile, glaring eye of HAL 9000 from Kubrick's '2001: A Space Odyssey.' A giant man's finger enters the frame from one corner, aimed at the trigger.

AI and the fatfinger economy (permalink)

Have you noticed that all the buttons you click most frequently to invoke routine, useful functions in your device have been moved, and their former place is now taken up by a curiously butthole-esque icon that summons an unwanted AI?

https://velvetshark.com/ai-company-logos-that-look-like-buttholes

These traps for the unwary aren't accidental, but neither are they placed there solely because tech companies think that if they can trick you into using their AI, you'll be so impressed that you'll become a regular user. To understand why you find yourself repeatedly fatfingering your way into an unwanted AI interaction – and why those interactions are so hard to exit – you have to understand something about both the macro- and microeconomics of high-growth tech companies.

Growth is a heady advantage for tech companies, and not because of an ideological commitment to "growth at all costs," but because companies with growth stocks enjoy substantial, material benefits. A growth stock trades at a higher "price to earnings ratio" ("P:E") than a "mature" stock. Because of this, there are a lot of actors in the economy who will accept shares in a growing company as though they were cash (indeed, some might prefer shares to cash). This means that a growing company can outbid their rivals when acquiring other companies and/or hiring key personnel, because they can bid with shares (which they get by typing zeroes into a spreadsheet), while their rivals need cash (which they can only get by selling things or borrowing money).

The problem is that all growth ends. Google has a 90% share of the search market. Google isn't going to appreciably increase the number of searchers, short of desperate gambits like raising a billion new humans to maturity and convincing them to become Google users (this is the strategy behind Google Classroom, of course). To continue posting growth, Google needs gimmicks. For example, in 2019, Google intentionally made Search less accurate so that users would have to run multiple queries (and see multiple rounds of ads) to find the answers to their questions:

https://www.wheresyoured.at/the-men-who-killed-google/

Thanks to Google's monopoly, worsening search perversely resulted in increased earnings, and Wall Street rewarded Google by continuing to trade its stock with that prized high P:E. But for Google – and other tech giants – the most enduring and convincing growth stories comes from moving into adjacent lines of business, which is why we've lived through so many hype bubbles: metaverse, web3, cryptocurrency, and now, of course, AI.

For a company like Google, the promise of these bubbles is that it will be able to double or triple in size, by dominating an entirely new sector. With that promise comes peril: growth must eventually stop ("anything that can't go on forever eventually stops"). When that happens, the company's stock instantaneously goes from being a "growth stock" to being a "mature stock" which means that its P:E is way too high. Anyone holding growth stock knows that there will come a day when those stocks will transition, in an eyeblink, from being undervalued to being grossly overvalued, and that when that day comes, there will be a mass sell-off. If you're still holding the stock when that happens, you stand to lose bigtime:

https://pluralistic.net/2025/03/06/privacy-last/#exceptionally-american

So everyone holding a growth stock sleeps with one eye open and their fists poised over the "sell" button. Managers of growth companies know how jittery their investors are, and they do everything they can to keep the growth story alive, as a matter of life and death.

But mass sell-offs aren't just bad for the company – it's also very bad for the company's key employees, that is, anyone who's been given stock in addition to their salary. Those people's portfolios are extremely heavy on their employer's shares, and they stand to disproportionately lose in the event of a selloff. So they are personally motivated to keep the growth story alive.

That's where these growth-at-all-stakes maneuvers bent on capturing an adjacent sector come from. If you remember the Google Plus days, you'll remember that every Google service you interacted with had some important functionality ripped out of it and replaced with a G+-based service. To make sure that happened, Google's bosses decreed that the company's bonuses would be tied to the amount of G+ activity each division generated. In companies where bonuses can amount to 90% of your annual salary or more, this was a powerful motivator. It meant that every product team at Google was fully aligned on a project to cram G+ buttons into their product design. Whether or not these made sense for users, they always made sense for the product team, whose ability to take a fancy Christmas holiday, buy a new car, or pay their kids' private school tuition depended on getting you to use G+.

Once you understand how corporate growth stories are converted to "key performance indicators" that drive product design, many of the annoyances of digital services suddenly make a great deal of sense. You know how it's almost impossible to watch a show on a streaming video service without accidentally tapping a part of the screen that whisks you to a completely different video?

The reason you have to handle your phone like a photonegative while watching a movie – the reason every millimeter of screen real-estate has been boobytrapped with an icon that takes you somewhere else – is that streaming services believe that their customers are apt to leave when they feel like there's nothing new to watch. These bosses have made their product teams' bonuses dependent on successfully "recommending" a show you've never seen or expressed any interest in to you:

https://pluralistic.net/2022/05/15/the-fatfinger-economy/

Of course, bosses understand that their workers will be tempted to game this metric. They want to distinguish between "real" clicks that lead to interest in a new video, and fake fatfinger clicks that you instantaneously regret. The easiest way to distinguish between these two types of click is to measure how long you watch the new show before clicking away.

Of course, this is also entirely gameable: all the product manager has to do is take away the "back" button, so that an accidental click to a new video is extremely hard to cancel. The five seconds you spend figuring out how to get back to your show are enough to count as a successful recommendation, and the product team is that much closer to a luxury ski vacation next Christmas.

So this is why you keep invoking AI by accident, and why the AI that is so easy to invoke is so hard to dispel. Like a demon, a chatbot is much easier to summon than it is to rid yourself of.

Google is an especially grievous offender here. Familiar buttons in Gmail, Gdocs, and the Android message apps have been replaced with AI-summoning fatfinger traps. Android is filled with these pitfalls – for example, the bottom-of-screen swipe gesture used to switch between open apps now summons an AI, while ridding yourself of that AI takes multiple clicks.

This is an entirely material phenomenon. Google doesn't necessarily believe that you will ever want to use AI, but they must convince investors that their AI offerings are "getting traction." Google – like other tech companies – gets to invent metrics to prove this proposition, like "how many times did a user click on the AI button" and "how long did the user spend with the AI after clicking?" The fact that your entire "AI use" consisted of hunting for a way to get rid of the AI doesn't matter – at least, not for the purposes of maintaining Google's growth story.

Goodhart's Law holds that "When a measure becomes a target, it ceases to be a good measure." For Google and other AI narrative-pushers, every measure is designed to be a target, a line that can be made to go up, as managers and product teams align to sell the company's growth story, lest we all sell off the company's shares.

(Image: Pogrebnoj-Alexandroff, CC BY-SA 3.0; Cryteria, CC BY 3.0; modified)


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#10yrsago Encryption backdoors are like TSA luggage-locks for the Internet https://www.theguardian.com/technology/2015/may/01/encryption-wont-work-if-it-has-a-back-door-only-the-good-guys-have-keys-to-

#10yrsago Tell the Copyright Office not to criminalize using unapproved goop in a 3D printer https://makezine.com/article/digital-fabrication/3d-printing-workshop/really-3d-printer/

#10yrsago Stupid patent for the ages: “Changing order quantities” https://www.eff.org/deeplinks/2015/04/stupid-patent-month-eclipse-ip-casts-shadow-over-innovation

#10yrsago Computer scientist/Congressman: crypto backdoors are “technologically stupid,” DA is “offensive” https://www.youtube.com/watch?v=YG0bUmuj4tg

#5yrsago Americans overwhelmingly support pandemic containment measures

#5yrsago How Big Ag destroyed our food supply's resilience https://pluralistic.net/2020/05/01/icann-can-and-did/#big-ag

#5yrsago San Francisco's legion of billionaires won't shell out for the city's covid fund https://pluralistic.net/2020/05/01/icann-can-and-did/#give2sf

#5yrsago "Financial literacy" will not make poor people better off https://pluralistic.net/2020/05/01/icann-can-and-did/#victim-blaming

#5yrsago Amazon and Trump officials neutered worker protection initiative https://pluralistic.net/2020/05/01/icann-can-and-did/#trumpazon

#5yrsago Frontier deliberately denied fiber to millions https://pluralistic.net/2020/05/01/icann-can-and-did/#fiber-now

#5yrsago .ORG has been snatched from the grasp of rapacious private equity billionaires https://pluralistic.net/2020/05/01/icann-can-and-did/#we-won

#1yrago Boeing's deliberately defective fleet of flying sky-wreckage https://pluralistic.net/2024/05/01/boeing-boeing/#mrsa


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-05-02T19:50:25+00:00 Fullscreen Open in Tab
Reviewing the 18 books I read in April
Thu, 01 May 2025 20:42:05 +0000 Fullscreen Open in Tab
Pluralistic: Apple faces criminal sanctions for defying App Store antitrust order (01 May 2025)


Today's links



A giant rotten apple behind sepia-toned prison bars. A worm descends from the prison's ceiling to perch on the apple. The scene is surmounted with Apple's 'Think Different.' wordmark.

Apple faces criminal sanctions for defying App Store antitrust order (permalink)

Epic, makers of the wildly popular Fortnite video-game, have waged a one-company war against the "app tax" – the 15-30% rake that the mobile duopoly of Apple/Google take out of every penny we spend inside of apps.

Epic's own digital practices are hardly spotless: just this year, the company was caught cheating players – many of them children – with deceptive practices and had to refund over $72m:

https://www.ftc.gov/enforcement/refunds/fortnite-refunds

But in this fight, Epic is on the side of the angels. The 30% that Apple/Google sucks out of the mobile economy is a brutal tax, and not just on app makers. Patreon performers recently raised a stink when the company announced that it would be clawing back 30% of the money pledged by their supporters – that 30% surcharge is passed straight through to Apple/Google:

https://www.theverge.com/2024/8/12/24218629/patreon-membership-ios-30-percent-apple-tax

From independent news outlets to crafters selling their work out of small storefronts, all the way up to massive entertainment services like Disney Plus and Fortnite, the mobile cartel takes 30% out of every dollar, a racket they maintain with onerous rules that ban apps from using their own payment processors, or even from encouraging users to click a link that brings them to a web-based payment screen.

30% is a gigantic markup on payment processing. It's ten times the going rate for payments in the USA, already one of the most expensive places in the world to transfer money from one party to another. In the EU, payment processing typically runs 1%…or less.

But crafters, Patreon podcasters and small-town newspapers are in no position to fight Google and Apple. Instead, we get Epic, a multi-billion-dollar company that's gone to the mattresses to fight these multi-trillion-dollar companies. Personally, I dote on billionaire-on-trillionaire violence.

Epic was wildly successful. It mopped up the floor with Google, securing an especially punitive award from a judge who was furious that Google had destroyed evidence:

https://pluralistic.net/2023/12/12/im-feeling-lucky/#hugger-mugger

Epic also won against Apple, though not as thoroughly as it had with Google, because Apple had the commonsense not to get up to the kind of shenanigans that make federal judges very, very mad. In the Google case, the court found that Google had acted as a monopolist and ordered it to open up the payment system in Google Play, a direct blow to the Android app tax.

In the Apple case, the judge did not find that Apple had acted as a monopolist, but did rule that the App Store's payment processing racket violated the law, and ordered Apple to end its own app tax:

https://techcrunch.com/2025/04/30/epic-games-just-scored-a-major-win-against-apple/

That's where things get gnarly. Apple is addicted to corrupt sources of income – like the tens of billions it illegally receives every year in bribes from Google to make it the default search:

https://apnews.com/article/google-antitrust-search-engine-verdict-apple-319a61f20fb11510097845a30abaefd8

And it really, really loves the app tax. When the EU ordered Apple to allow third-party app stores (as a way of killing the app tax), the company cooked up a malicious compliance plan that was comically corrupt:

https://pluralistic.net/2024/02/06/spoil-the-bunch/#dma

So, the mere fact that a federal judge had ordered Apple to open up its app store to competing payment processors was not going convince Apple to actually do it. Instead, Apple cooked up a set of rules for third-party payment processing that would make it more costly to use someone else's payments, piling up a mountain of junk fees and using scare screens and other deceptive warnings to discourage users from making payments through a rival system:

https://www.thebignewsletter.com/p/judge-rules-apple-executive-lied

That's the kind of thing that is apt to make a federal judge angry – and, as noted, angry federal judges can make life very hard for tech monopolists, a lesson Google learned when it destroyed key evidence in its Epic case. But Apple didn't just flout the court order – they lied about it to cover it up, and Judge Yvonne Gonzalez Rogers is furious. She held that Alex Roman, Apple's Vice-President of Finance, "outright lied under oath," and she has raised the possibility of criminal contempt penalties for Apple:

https://regmedia.co.uk/2025/05/01/pacer_epic_vs_apple_injunction_judgement.pdf

The judge further wrote:

This is an injunction, not a negotiation. There are no do-overs once a party willfully disregards a court order. Time is of the essence. The Court will not tolerate further delays. As previously ordered, Apple will not impede competition. The Court enjoins Apple from implementing its new anticompetitive acts to avoid compliance with the Injunction. Effective immediately Apple will no longer impede developers’ ability to communicate with users nor will they levy or impose a new commission on off-app purchases

In other words, any junk fees, any impediments to opening up third party payments, will be swiftly and harshly dealt with. As of right now developers can start to build third-party payments into their apps and Apple cannot block them. It's the end of the app tax, a source of about $100b/year for Apple:

https://www.theregister.com/2025/05/01/apple_epic_lies_possible_crime/

The world is on fire and everything is terrible, but we are also living through the most consequential season in the history of the war on corporate tech power. Google has been convicted three times of being a monopolist and is almost certainly going to have to sell off Chrome, most of its ad-tech stack, and possibly Android. Meta just put up a pathetic showing in an equally serious antitrust case that could see it forced to sell off Instagram and Whatsapp:

https://pluralistic.net/2025/04/11/it-is-better-to-buy/#than-to-compete

Countries around the world have passed big, sweeping, muscular antitrust laws specifically aimed at smashing corporate tech power, like the EU's Digital Markets Act and Digital Services Act:

https://www.eff.org/pages/adoption-dsadma-notre-analyse

Most importantly, all of this is happening from the bottom up. There is no dark money campaign to fuck up the tech companies. The politicians and enforcers who are taking on Big Tech are being shoved from behind by billions of everyday people who are furious and refuse to take it any longer:

https://pluralistic.net/2025/04/10/solidarity-forever-2/#oligarchism

I am deeply grateful for the public servants who have championed this cause, but I also know that these people are the effect of our movement, not the cause. When Kier Starmer fires Britain's brilliant and effective top competition enforcer and replaces him with the former head of Amazon UK, that does nothing to tamp down the political outrage that Britons feel towards America's tech giants:

https://pluralistic.net/2025/01/22/autocrats-of-trade/#dingo-babysitter

All over the world, countries that passed IP laws to protect US tech interests in exchange for tariff-free access to US markets are grappling with the end of free trade with America. This represents a generational opportunity to pass laws that enable local technologists to jailbreak US tech exports and liberate their people from the extractive practices of Big Tech forever:

https://archive.is/CiBIz

There is nothing harder to stop than an idea whose time has come to pass.

(Image: Alex Popovkin, Bahia, Brazil from Brazil, CC BY 2.0; Hubertl, CC BY-SA 4.0; modified)


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Doonesbury on DRM https://web.archive.org/web/20050501170432/http://www.doonesbury.com/strip/dailydose/index.html?uc_full_date=20050501

#15yrsago AT&T asks government to create national censorwall and system for disconnecting accused infringers https://arstechnica.com/tech-policy/2010/04/att-calls-for-us-3-strikes-tribunal-web-censorship/

#15yrsago Asimov’s opens to electronic submission https://web.archive.org/web/20050514080931/http://www.asimovs.com/info/guidelines.shtml

#10yrsago Lego store detains 11 year old customer, accuses his father of being an unfit parent https://www.freerangekids.com/lego-store-detains-boy-11-for-being-too-young-to-shop-alone/

#10yrsago Telescreen watch: Vizio adds spyware to its TVs https://web.archive.org/web/20150905093150/http://www.vizio.com/smartinteractivity

#5yrsago AMC: "We will never show another Universal movie" https://pluralistic.net/2020/04/30/day-and-date/#vertical-integration

#5yrsago Financial services workers dying for junk mail https://pluralistic.net/2020/04/30/day-and-date/#broadridge-financial-solutions

#5yrsago Swedish covid death rates soar above neighbors' https://pluralistic.net/2020/04/30/day-and-date/#tubers

#5yrsago Medicare for All (Congressjerks) https://pluralistic.net/2020/04/30/day-and-date/#m4a4c

#5yrsago Berlin in color, after the Reich's fall https://pluralistic.net/2020/04/30/day-and-date/#bouncing-rubble

#1yrago Live Nation/Ticketmaster is buying Congress https://pluralistic.net/2024/04/30/nix-fix-the-tix/#something-must-be-done-there-we-did-something


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-05-01T14:06:40+00:00 Fullscreen Open in Tab
Read "Why I Am Leaving the USA"
Read:
The author, a trans woman and mother of neurodivergent kids, has been monitoring this nation’s political climate since Trump’s first term. Now that her worst fears are fast becoming a reality, she’s had to make the most difficult decision of her life.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-05-01T13:26:57+00:00 Fullscreen Open in Tab
Read "Trump's Quest for Crypto Riches Is a Constitutional Scandal Waiting to Happen"
Thu, 01 May 2025 00:58:40 +0000 Fullscreen Open in Tab
Pluralistic: Republicans want to force students to pay off scam college loans (30 Apr 2025)


Today's links



A 1942 photo of two smiling university graduates (a young man and a young woman) in gown and mortarboards, standing before the steps of a columnated university building. Behind them is a dancing skeleton, also wearing a mortarboard, drumiming on a snare drum.

Republicans want to force students to pay off scam college loans (permalink)

House Republicans have a great plan to pay for Trump's tax-cuts for the rich: jacking up the cost of federal student loans, while eliminating protections for students who are scammed by fake universities:

https://prospect.org/education/2025-04-30-republicans-education-upper-class-privilege-student-loans/

Every GOP legislator and especially Congressional committee chairs are scrambling to find cuts that can offset Trump's plans to make his 2017 tax cuts permanent and then add more cuts on top of that. The failure of Doge to make any appreciable savings has left Trump high and dry, with unfunded tax cuts that will flunk even the most compliant, ass-kissing Congressional Budget Office analysis:

https://www.msnbc.com/opinion/msnbc-opinion/elon-musk-doge-savings-trump-rcna203051

Enter the House Education and Workforce Committee, whose Republican members have found a way to save $330b over the next decade, through the simple expedient of making working families choose between foregoing education for their kids, or burdening those kids with brutal, crushing debts for the rest of their lives – debts that can't be discharged in bankruptcy, even if the student becomes totally, permanently disabled – not even if the "university" that charged them all that tuition is later shut down for running a scam.

Trump knows a lot about scams in higher ed, of course. His own ill-fated "Trump 'University,'" a fraudulent, non-accredited institution that stole millions of dollars from unsuspecting students:

https://en.wikipedia.org/wiki/Trump_University

Trump U isn't the only scam college out there, not by a damn sight. The Department of Education's "Borrower Defense to Repayment" system allows students who've been scammed by fraudulent institutions to have their debts canceled. That's the clause that the GOP members of the House Education and Workforce Committee plan to kill. This will not only leave fraud victims on the hook for a lifetime of debt – it will also make it easier for scam institutions to re-open and prey upon even more students. The Republicans' giveaway to scam universities kills the "gainful employment" rule that requires that universities prove that their grads can actually get work in the fields they graduate in.

The GOP plan will kill all subsidized undergrad loans, meaning that interest will be piled on student loans while students are still at school, so a grad with a four-year degree will also owe four years worth of compounded interest on their freshman year loans. Undergrad loans are capped at $50k, less than half the price of a degree at most state colleges. The GOP members say that the $50k cap covers the "median tuition" – meaning that it is lower than tuition at half the country's institutions.

GOP members have also called for changes to "income based repayment," with sharply rising payments that will shoot up every time a graduate's income crosses a line. Under this plan, a student grad earning $10k-$20k would have to pay 1% of their income to service their loans. For each $10k increase in graduate pay, repayment goes up by 1% – so if a grad earning $99.9k gets a raise to $100k, their repayments will shoot up from 9% of their annual income to 10%. That means a $100 raise could leave a graduate $850 poorer.

This proposal will roll back Biden-era changes to the interest charged to borrowers on income-based repayment. Under the new rules, interest will continue to compound on your loan even if you're earning peanuts, meaning that the poorest grads will have the highest lifetime interest charges and likely die with unpaid student loans that exceed the principle several times over (remember, the only debt that can be charged against your Social Security is student loans).

The Republican proposal also screws grads working through a Public Service Loan Forgiveness plan, which cancels your student debt after ten years of work in public service. The Republicans want to increase the payments due from grads during that decade of public service. Also, med-school grads would no longer receive credit towards PSLF debt cancellation for the years they spend in residencies, which will drain the supply of freshly minted doctors who staff community health clinics.

They also want to gut Pell grants, changing eligibility to limits grants to "full time" students (30+ hours/week of courses), which will strike hardest at the poorest students, who often attend school part time while working.

Raising the price of a good education and lowering protections against receiving a bad education is an attack on the very idea of education as a source of social mobility. After all, the students most likely to be trapped by a scam college are students from families without a lot of college grads, who lack the means of assessing educational quality.

During the New Deal, America created two parallel paths to social mobility: labor protections and subsidized home ownership. As with every American social initiative, the New Deal was undermined by racism, sexism and xenophobia, and excluded many of America's most disfavored minorities from its benefits. After WWII, two groups of Americans fought to change the New Deal. The wealthy fought to roll back its protections, while the rest of us fought to extend those benefits to Black people, indigenous people, Latino people, women, queers, and others who were left out from the start:

https://pluralistic.net/2024/02/26/horsehoe-crab/#substantive-disagreement

They made a lot of progress, but then came the Reagan revolution, which wiped out labor protections (including defined benefit pensions) and doubled down on home ownership as the only means of securing a comfortable and dignified life. Over the next quarter-century, this turned a lucky group of workers into real-estate millionaires, even as their wages stagnated and the cost of education and health care skyrocketed:

https://pluralistic.net/2021/06/06/the-rents-too-damned-high/

Housing prices also skyrocketed. Of course they did: the only way that owning a house could be an "investment" (as opposed to a way to fulfill the human need for shelter) is if the price of keeping a roof over your head went up. But owning an expensive house in a world of stagnant wages and rising health and education costs is a recipe for not owning a house anymore, because you'll have to liquidate that home to cover your bills or get your kids through school. This century hasn't just been a time in which housing grew more valuable (and thus more expensive) – it's been an era in which its easier than ever to be forced out of your home:

https://pluralistic.net/2022/11/06/the-end-of-the-road-to-serfdom/

Trading labor protection for real-estate speculation was always going to end badly for workers. The retreat of organized labor has paved the way for a rollback of all the post-war prosperity, allowing America's oligarchs to create a new Gilded Age where education is reserved for failsons of wealthy families, which is fine, because the rest of us won't need a degree to shine their shoes, clean their toilets, and screw the little screws in on iPhones:

https://www.theverge.com/news/644320/us-commerce-secretary-howard-lutnick-says-well-be-making-iphones-in-the-us


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Danny O’Brien goes to work at EFF! https://web.archive.org/web/20050507123924/https://www.oblomovka.com/entries/2005/04/29#1114782180

#15yrsago 1939 World’s Fair: the future’s cradle, in pictures https://web.archive.org/web/20100501170616/http://www.wired.com/thisdayintech/2010/04/gallery-1939-worlds-fair/

#10yrsago British austerity: a failed experiment abandoned by the rest of the world https://www.theguardian.com/business/ng-interactive/2015/apr/29/the-austerity-delusion

#10yrsago Translation: once they learn the truth, techies hate and fear us https://www.wired.com/2015/04/us-defense-secretary-snowden-caused-tensions-techies/

#10yrsago FBI’s crypto backdoor plans require them to win the war on general purpose computing http://webpolicy.org/2015/04/28/you-cant-backdoor-a-platform/

#10yrsago Anyone can open a Master Lock padlock in under two minutes https://www.youtube.com/watch?v=09UgmwtL12c

#10yrsago Couples counsellor who assigns Ikea furniture assembly calls Liatorp “The Divorcemaker” https://web.archive.org/web/20150430183654/https://laist.com/2015/04/28/santa_monica_therapist_uses_ikea_as.php

#10yrsago UK Tories forged letter of support in the Telegraph from “5,000 small businesses” https://sturdyblog.wordpress.com/2015/04/27/small-business-letter-to-the-telegraph-an-attempt-to-defraud-the-electorate/

#5yrsago How monopolism crashed the US food supply https://pluralistic.net/2020/04/29/banjo-nazis/#butchery

#5yrsago Legendary troubleshooting stories https://pluralistic.net/2020/04/29/banjo-nazis/#cuckoos-egg

#5yrsago Medical debt collection during the pandemic https://pluralistic.net/2020/04/29/banjo-nazis/#armbreakers

#5yrsago British Library releases 1.9m images https://pluralistic.net/2020/04/29/banjo-nazis/#it-belongs-in-a-museum

#5yrsago NSO Group employee used Pegasus cyberweapon to stalk a woman https://pluralistic.net/2020/04/29/banjo-nazis/#loveint

#5yrsago Founder of AI surveillance company was a Nazi who helped shoot up a synagogue https://pluralistic.net/2020/04/29/banjo-nazis/#damien-patton-nazi

#5yrsago Talking Radicalized with the CBC https://pluralistic.net/2020/04/29/banjo-nazis/#zuckervegans

#5yrsago Bayesian reasoning and covid-19 https://pluralistic.net/2020/04/29/banjo-nazis/#uncertainty

#5yrsago Cigna claims to be rolling in dough and on the verge of bankruptcy https://pluralistic.net/2020/04/29/banjo-nazis/#someones-lying

#1yrago Cigna's nopeinator https://pluralistic.net/2024/04/29/what-part-of-no/#dont-you-understand


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-04-30T18:53:26+00:00 Fullscreen Open in Tab
Finished reading World War Z
Finished reading:
Cover image of World War Z
Published . 342 pages.
Started ; completed April 30, 2025.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-04-30T15:06:46+00:00 Fullscreen Open in Tab
Read "DOGE Aide Dismantling CFPB Owns Stock in Companies That Could Benefit From Cuts"
2025-04-29T20:41:33+00:00 Fullscreen Open in Tab
Finished reading Beware of Chicken
Finished reading:
Cover image of Beware of Chicken
Beware of Chicken series, book 1.
Published . 485 pages.
Started ; completed April 29, 2025.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
Tue, 29 Apr 2025 19:42:28 +0000 Fullscreen Open in Tab
Pluralistic: Mike Lee and Jim Jordan want to kill the law that bans companies from cheating you (29 Apr 2025)


Today's links



Three men playing cards and having a drink. The men are dressed in long trousers and shirts. One man passes a card to another player with the card between his toes under the table, unbeknownst to the third player. The card-passer has Trump's hair and orange skin. The card-receiver wears a MAGA hat. The background is a heavily halftoned, desaturated, waving US flag.

Mike Lee and Jim Jordan want to kill the law that bans companies from cheating you (permalink)

House and Senate Republicans are on the verge of killing Section 5 of the Federal Trade Commission Act, one of the most potent anti-corruption laws on the US statute books:

https://www.thebignewsletter.com/p/house-gop-proposes-eliminating-key

More than a century ago, Congress passed the FTCA, and they made a point of including a clause that granted the new independent agency broad authority to investigate and prohibit "unfair and deceptive methods of competition." As Matt Stoller writes, over the ensuing 100 years, the FTC has used Section 5 to go after "illegal commissions, firms spying on rivals, sabotage, messing around with patents or regulations."

But starting with the Reagan era, both Republican and Democratic presidents have appointed FTC chairs who were loath to invoke FTCA 5, shying away from the power and duty Congress had given them. This all changed with Biden's FTC chair Lina Khan, who revived the law, using it to punish companies for invading your privacy, blocking repair, locking workers in with noncompete clauses, and more:

https://www.ftc.gov/system/files/ftc_gov/pdf/P221202Section5PolicyStatement.pdf

FTCA has been repeatedly upheld by the Supreme Court, and Congress liked the way it worked so much that when they created the Department of Transport, they copy-pasted the language of FTCA into the DOT's enabling legislation. Pete Buttigieg, Biden's Transport secretary, refused to use this power, but when Khan's chief of staff moved over to Transport, it became a powerhouse regulator, fighting ripoffs and scams in aviation, rail and more:

https://pluralistic.net/2023/01/10/the-courage-to-govern/#whos-in-charge

Neoclassical economists hate laws like Section 5. The entire basis of neoliberal economics is that the economy can be modeled – and thus controlled – using mathematics. This ideology requires that economists ignore all qualitative aspects of society. Notoriously, economic modeling treats power as irrelevant, because it can't be quantified and plugged into a model:

https://pluralistic.net/2023/04/03/all-models-are-wrong/#some-are-useful

This is a hell of a deal for the powerful. Ignoring power lets a rich person who buys a starving person's kidneys claim to be engaged in a "voluntary transaction." Ignoring power lets private equity funds claim that gouging you on emergency room care and ambulance rides is fine, because you "freely chose" to be rushed to their hospital while dying of a heart attack. If we can all agree that power doesn't matter, then we can do away with all workplace protections, from the minimum wage to worker safety. Take power out of the equation, and you can claim that any worker on starvation wages who loses an arm in a badly maintained machine "freely contracted" into that situation.

Oligarchs and their lickspittles have waged a generations-long war on the very concept of power, and this assault on Section 5 of the FTC Act is just the latest skirmish. You see, "unfair and deceptive" is a qualitative idea, one that requires consideration of power relationships.

The abolition of fairness as a concept is central to Trumpism. Notoriously, Trump has claimed that any time he successfully rips someone off, "That makes him smart":

https://pluralistic.net/2024/12/04/its-not-a-lie/#its-a-premature-truth

The Trump movement is full of extremely successful cheats and liars. There's VCs like Mark Andreesen, whose fund paid a $100m bribe to Kickstarter execs in exchange for a fake cryptocurrency launch, in a bid to lure more retail investors into the crypto bubble that Andreesen-Horowitz played a central role in:

https://finance.yahoo.com/news/untold-story-kickstarter-crypto-hail-120000205.html

And of course, there's Elon Musk, who lies about his cars, his robots, his rockets, his AI, and everything else. No wonder Elon Musk wants to get rid of a law that bans "unfair and deceptive methods of competition."

The bid to kill Section 5 of the FTC Act is hidden deep in a budget reconciliation amendment introduced by Rep Jim Jordan (R-OH), which pastes in sloppy language drafted by Sen Mike Lee (R-UT). The mechanism by which this amendment will neuter Section 5 is eye-glazingly complex, though Stoller does his best to make it comprehensible.

Far more important than the method by which Section 5 of the FTC Act will be gutted is the consequence of doing so. Stripping the FTC of the power to chase unfair and deceptive conduct will fire a starting pistol for even more ripoffs and scams. Worse than that, the Jordan amendment will kill enforcement of existing consent decrees from companies that have been successfully prosecuted under Section 5, allowing them to restart the scams that attracted regulatory scrutiny.

The Trump administration has been touting antitrust as its "alternative to regulation," drawing an arbitrary line between "regulation" and "antitrust." Antitrust is absolutely regulation:

https://finance.yahoo.com/news/antitrust-enforcement-not-regulation-doj-163251704.html

Indeed, antitrust is the most important regulation of all, because it's the regulation that keeps companies from getting so large and powerful that they can ignore all the other regulations. Without antitrust, companies become too big to fail, then too big to jail, then too big to care. The Trump admin will absolutely continue to do antitrust, but in the Trumpiest way possible – by attacking companies that offend Trump, rather than attacking companies that harm the public:

https://pluralistic.net/2024/11/12/the-enemy-of-your-enemy/#is-your-enemyis

This is gangsterism, the thing that comes after capitalism collapses into feudalism. In gangsterism, "fairness" and "power" have no place. All that's left is a kind of caveat emptor brainworm that insists that if you got scammed, you should have shopped more carefully. And if you got scammed at gunpoint, you just need to understand that the gun was held by the invisible hand, and it was pointed at you in an economically efficient manner.


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Tell US govt not to steal tax-funded weather from the public https://web.archive.org/web/20050429182910/https://www.opendemocracy.net/debates/article-8-40-2452.jsp

#20yrsago Fairfax libraries waste tax-dollars on DRM https://www.his.com/~pshapiro/audiobook.html

#20yrsago Dirty tricks at WIPO https://web.archive.org/web/20050429182910/https://www.opendemocracy.net/debates/article-8-40-2452.jsp

#15yrsago Goodhart’s Law: Once you measure something, it changes https://en.wikipedia.org/wiki/Goodhart's_law

#15yrsago All of Gopherspace as a single download https://changelog.complete.org/archives/1466-download-a-piece-of-internet-history

#15yrsago Timeline of Facebook privacy policy: from reasonable (2005) to apocalyptic (2010) https://web.archive.org/web/20100501041847/http://www.eff.org/deeplinks/2010/04/facebook-timeline/

#15yrsago Top US psychiatric pharmaceuticals, 2009 edition https://web.archive.org/web/20100429082228/https://psychcentral.com/lib/2010/top-25-psychiatric-prescriptions-for-2009

#15yrsago Canadian record industry won’t say what it wants https://www.michaelgeist.ca/2010/04/cria-on-copyright-specifics/

#15yrsago Music industry spokesman loves child porn https://christianengstrom.wordpress.com/2010/04/27/ifpis-child-porn-strategy/

#15yrsago Mississippi school purges top student from yearbook for being lesbian https://www.jacksonfreepress.com/news/2010/apr/26/school-cuts-gay-student-photo-from-yearbook/

#10yrsago Lifting the lid on Scientology’s fatally woo version of rehab https://www.muckrock.com/news/archives/2015/apr/27/narconon-incident-reports/

#5yrsago "Essential" workers will strike across America for May Day https://pluralistic.net/2020/04/28/force-multiplier/#mayday

#5yrsago Citizen DJ https://pluralistic.net/2020/04/28/force-multiplier/#brianfoo

#5yrsago NYC will pedestrianize 40 miles of city streets https://pluralistic.net/2020/04/28/force-multiplier/#pandemic-urbanism

#5yrsago Foreclosure vultures hold illegal auctions on courthouse steps https://pluralistic.net/2020/04/28/force-multiplier/#first-against-the-wall

#5yrsago The law is free https://pluralistic.net/2020/04/28/force-multiplier/#go-carl-go

#5yrsago DRM and CHI https://pluralistic.net/2020/04/28/force-multiplier/#fraud


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-04-29T14:48:21+00:00 Fullscreen Open in Tab
Note published on April 29, 2025 at 2:48 PM UTC

Prosecutors have requested Alex Mashinsky, CEO of the collapsed Celsius cryptocurrency company, be sentenced to at least twenty years in prison for his "sustained, calculated campaign of deceit carried out over years, targeting ordinary people."

Prosecutors say that such a severe sentence is necessary not only for deterrence, but because despite his guilty plea, Mashinsky has not fully accepted responsibility for his actions.

Sentencing guidelines recommend a 30 year sentence, which is only because of the statutory cap resulting from his guilty plea agreement; otherwise the recommendation would be life. Mashinsky has argued he should be sentenced to no more than a year in prison. Sentencing will happen on May 8.

Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-04-29T14:13:46+00:00 Fullscreen Open in Tab
Read "Secret Deals, Foreign Investments, Presidential Policy Changes: The Rise of Trump's Crypto Firm"
2025-04-29T02:31:35+00:00 Fullscreen Open in Tab
Finished reading This Inevitable Ruin
Finished reading:
Cover image of This Inevitable Ruin
Dungeon Crawler Carl series, book 7.
Published . 724 pages.
Started ; completed April 28, 2025.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-04-28T19:24:18+00:00 Fullscreen Open in Tab
Read "Government Actually Threatens Wikipedia's Editorial Freedom; Self-Proclaimed Free Speech Warriors Suddenly Have Other Plans"
Read:
Even if Wikipedia’s content was biased (it isn’t), even if every editor was actively trying to push an anti-Israel narrative (they aren’t), that would still be protected by the First Amendment. The government doesn’t get to threaten organizations over their editorial choices, no matter how much certain prosecutors or publications might dislike those choices.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-04-28T18:44:48+00:00 Fullscreen Open in Tab
Note published on April 28, 2025 at 6:44 PM UTC
2025-04-28T17:04:20+00:00 Fullscreen Open in Tab
Note published on April 28, 2025 at 5:04 PM UTC
Sun, 27 Apr 2025 21:37:10 +0000 Fullscreen Open in Tab
Pluralistic: The enshittification of tech jobs (27 Apr 2025)


Today's links



An altered version of J.C. Leyendecker's Labor Day 1946 cover illustration for Hearst's 'American Weekly' magazine. The original features a muscular worker in dungarees sitting atop a banner-draped globe, holding a sledgehammer. In this version, his head has been replaced with a faceless hacker-in-a-hoodie, and his sledgehammer has been filled with Matrix code-waterfall characters. Leyendecker's signature has been replaced with an IWW graphic depicting workers with upraised fists all joining together to form a gigantic fist.

The enshittification of tech jobs (permalink)

Tech workers are a weird choice for "princes of labor," but for decades they've enjoyed unparalleled labor power, expressed in high wages, lavish stock grants, and whimsical campuses with free laundry and dry-cleaning, gourmet cafeterias, and kombucha on tap:

https://www.youtube.com/watch?v=nhUtdgVZ7MY

All of this, despite the fact that tech union density is so low it can barely be charted. Tech workers' power didn't come from solidarity, it came from scarcity. When you're getting five new recruiter emails every day, you don't need a shop steward to tell your boss to go fuck themselves at the morning scrum. You can do it yourself, secure in the knowledge that there's a company across the road who'll give you a better job by lunchtime.

Tech bosses sucked up to their workers because tech workers are insanely productive. Even with sky-high salaries, every hour a tech worker puts in on the job translates into massive profits. Which created a conundrum for tech bosses: if tech workers produce incalculable value for the company every time they touch their keyboards, and if there aren't enough tech workers to go around, how do you get whichever tech workers you can hire to put in as many hours as possible?

The answer is a tactic that Fobazi Ettarh called "vocational awe":

https://www.inthelibrarywiththeleadpipe.org/2018/vocational-awe/

"Vocational awe" describes the feeling that your work matters so much that you should accept all manner of tradeoffs and calamities to get the job done. Ettarh uses the term to describe the pathology of librarians, teachers, nurses and other underpaid, easily exploited workers in "caring professions." Tech workers are weird candidates for vocational awe, given how well-paid they are, but never let it be said that tech bosses don't know how to innovate – they successfully transposed an exploitation tactic from the most precarious professionals to the least precarious.

As farcical as all the engineer-pampering tech bosses got up to for the first couple decades of this century was, it certainly paid off. Tech workers stayed at the office for every hour that god sent, skipping their parents' funerals and their kids' graduations to ship on time. Snark all you like about empty platitudes like "organize the world's information and make it useful" or "bring the world closer together," but you can't argue with results: workers who could – and did – bargain for anything from their bosses…except a 40-hour work-week.

But for tech bosses, this vocational awe wheeze had a fatal flaw: if you convince your workforce that they are monk-warriors engaged in the holy labor of bringing forth a new, better technological age, they aren't going to be very happy when you order them to enshittify the products they ruined their lives to ship. "I fight for the user" has been lurking in the hindbrains of so many tech workers since the Tron years, somehow nestling comfortably alongside of the idea that "I don't need a union, I'm a temporarily embarrassed founder."

Tech bosses don't actually like workers. You can tell by the way they treat the workers they don't fear. Sure, Tim Cook's engineers get beer-fattened, chestnut finished and massaged like Kobe cows, but Cook's factory workers in China are so maltreated that Foxconn (the cutout Apple uses to run "iPhone City" where Apple's products are made) had to install suicide nets to reduce the amount of spatter from workers who would rather die than put in another hour at Tim Apple's funtime distraction rectangle factory:

https://www.theguardian.com/technology/2017/jun/18/foxconn-life-death-forbidden-city-longhua-suicide-apple-iphone-brian-merchant-one-device-extract

Jeff Bezos's engineers get soft-play areas, one imported Australian barista for each mini-kitchen, and the kind of Japanese toilet that doesn't just wash you after but also offers you a trim and dye-job, but Amazon delivery drivers are monitored by AIs that narc them out for driving with their mouths open (singing is prohibited in Uncle Jeff's delivery pods!) and have to piss in bottles; meanwhile, Amazon warehouse workers are injured at three times the rate of other warehouse workers.

This is how tech bosses would treat tech workers…if they could.

And now? They can.

Writing for the Wall Street Journal, Katherine Bindley describes the new labor dynamics at Big Tech:

https://www.msn.com/en-us/money/companies/tech-workers-are-just-like-the-rest-of-us-miserable-at-work/ar-AA1DDKjh

It starts with Meta, who just announced a 5% across-the-board layoff – on the same day that it doubled executive bonuses. But it's not just the workers who get shown the door who suffer in this new tech reality – the workers on the job are having to do two or three jobs, for worse pay, and without all those lovely perks.

Take Google, where founder Sergey Brin just told his workers that they should be aiming for a "sweet spot" of 60 hours/week. Brin returned to Google to oversee its sweaty and desperate "pivot to AI," and like so many tech execs, he's been trumpeting the increased productivity that chatbots will deliver for coders. But a coder who picks up their fired colleagues' work load by pulling 60-hour work-weeks isn't "more productive," they're more exploited.

Amazon is another firm whose top exec, Andy Jassy, has boasted about the productivity gains of AI, but an Amazon Web Services manager who spoke to Bindley says that he's lost so many coders that he's now writing code for the first time in a decade.

Then there's a Meta recruiter who got fired and then immediately re-hired, but as a "short term employee" with no merit pay, stock grants, or promotions. She has to continuously reapply for her job, and has picked up the workload of several fired colleagues who weren't re-hired. Meta managers (the ones whose bonuses were just doubled) call this initiative "agility." Amazon is famous for spying on its warehouse workers and drivers – and now its tech staff report getting popups warning them that their keystrokes are being monitored and analyzed, and their screens are being recorded.

Bindley spoke to David Markley, an Amazon veteran turned executive coach, who attributed the worsening conditions (for example, managers being given 30 direct reports) to the "narrative" of AI. Not, you'll note, the actual reality of AI, but rather, the story that AI lets you "collapse the organization," slash headcount and salaries, and pauperize the (former) princes of labor.

The point of AI isn't to make workers more productive, it's to make them weaker when they bargain with their bosses. Another of Bindley's sources went through eight rounds of interviews with a company, received an offer, countered with a request for 12% more than the offer, and had the job withdrawn, because "the company didn’t want to move ahead anymore based on the way the compensation conversation had gone."

For decades, tech workers were able to flatter themselves that they were peers with their bosses – that "temporarily embarrassed founder" syndrome again. The Google founders and Zuck held regular "town hall" meetings where the rank-and-file engineers could ask impertinent questions. At Google, these have been replaced with "tightly scripted events." Zuckerberg has discontinued his participation in company-wide Q&As, because they are "no longer a good use of his time."

Companies are scaling back perks in both meaningful ways (Netflix hacking away at parental leave), and petty ones (Netflix and Google cutting back on free branded swag for workers). Google's hacked back its "fun budget" for offsite team-building activities and replacement laptops for workers needing faster machines (so much for prioritizing "increasing worker productivity").

Trump's new gangster capitalism pits immiserated blue collar workers against the "professional and managerial class," attacking universities and other institutions that promised social mobility to the children of working families. Trump had a point when he lionized factory work as a source of excellent wages and benefits for working people without degrees, but he conspicuously fails to mention that factory work was deadly, low-waged and miserable – until factory workers formed unions:

https://www.laborpolitics.com/p/unions-not-just-factories-will-make

Re-shoring industrial jobs to the USA is a perfectly reasonable goal. Between uncertain geopolitics, climate chaos, monopolization and the lurking spectre of the next pandemic, we should assume that supply-chains will be repeatedly and cataclysmicly shocked over the next century or more. And yes, re-shoring product could provide good jobs to working people – but only if they're unionized.

But Trump has gutted the National Labor Relations Board and stacked his administration with bloodsucking scabs like Elon Musk. Trump doesn't want to bring good jobs back to America – he wants to bring bad jobs back to America. He wants to reshore manufacturing jobs from territories with terrible wages, deadly labor conditions, and no environment controls by taking away Americans' wages, labor rights and environmental protections. He doesn't just want to bring home iPhone production, he wants to import the suicide nets of iPhone City, too.

Tech workers are workers, and they once held the line against enshittification, refusing to break the things they'd built for their bosses in meaningless all-nighters motivated by vocational awe. Long after tech bosses were able to buy all their competitors, capture their regulators, and expand IP law to neutralize the threat of innovative, interoperable products like alternative app stores, ad-blockers and jailbreaking kits, tech workers held the line.

There've been half a million US tech layoffs since 2023. Tech workers' scarcity-derived power has been vaporized. Tech workers can avoid the fate of the factory, warehouse and delivery workers their bosses literally work to death – but only by unionizing.

In other words, the workers in re-shored factories and tech workers need the same thing. They are class allies – and tech bosses are their class enemies. This is class war.


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago Architectures of Control: DRM in hardware https://web.archive.org/web/20050425184527/http://www.danlockton.co.uk/research/architectures.html

#20yrsago Insect photos in naturalistic http://macro-focus https://bugdreams.com

#20yrsago BBC: DRM makes music customers mad https://news.bbc.co.uk/2/hi/technology/4474143.stm

#20yrsago Guckert was at the White House even when there were no press briefings https://web.archive.org/web/20050428034248/https://www.rawstory.com/exclusives/byrne/secret_service_gannon_424.htm

#20yrsago FBI warnings ruin CD art & art is the reason for buying CDs http://www.yarnivore.com/francis/archives/001102.html

#20yrsago US govt admits RFID passports are danger to Americans https://www.nytimes.com/2005/04/27/politics/bowing-to-critics-us-to-alter-design-of-electronic-passports.html

#15yrsago Considering cities as “dense meshes of active, communicating public objects” https://speedbird.wordpress.com/2010/04/24/frameworks-for-citizen-responsiveness-enhanced-toward-a-readwrite-urbanism/

#15yrsago Peter Watts won’t go to jail https://memex.craphound.com/2010/04/26/peter-watts-wont-go-to-jail/

#15yrsago Canada’s Heritage Minister ready to bring back DMCA-style copyright, throwing out results of copyright consultation https://web.archive.org/web/20100428113301/http://www.michaelgeist.ca/content/view/4979/135/

#15yrsago In praise of SFWA’s Grievance Committee https://memex.craphound.com/2010/04/25/in-praise-of-sfwas-grievance-committee/

#15yrsago UK’s super-rich get even richer https://news.bbc.co.uk/2/hi/business/8642021.stm

#15yrsago Protect your copyrights, boycott DRM-locked platforms https://www.publishersweekly.com/pw/by-topic/digital/devices/article/42869-can-you-survive-a-benevolent-dictatorship.html

#15yrsago Why I won’t buy an iPad, the podcast edition https://web.archive.org/web/20110114222040/https://podcasts.tvo.org/searchengine/audio/800832_48k.mp3

#15yrsago On Peter Watts’s sentencing hearing https://web.archive.org/web/20100429105210/http://www.tor.com/index.php?option=com_content&view=blog&id=59215

#15yrsago The “fair use economy” is enormous, growing, and endangered by the relatively tiny entertainment industry https://web.archive.org/web/20110128152731/https://www.wired.com/images_blogs/threatlevel/2010/04/fairuseeconomy.pdf

#15yrsago UK election: ask your candidates if they’ll repeal the Digital Economy Act https://web.archive.org/web/20100430090624/http://action.openrightsgroup.org/ea-campaign/clientcampaign.do?ea.client.id=1422&ea.campaign.id=6449

#10yrsago Town will cut off power to families of kids who commit vandalism https://web.archive.org/web/20150419053210/https://www.illinoishomepage.net/story/d/story/cutting-vandalism-off-at-the-source/26297/gSM2PYl6P0CRIttIu_95BQ

#10yrsago Portraits of e-waste pickers in Ghana https://www.wired.com/2015/04/kevin-mcelvaney-agbogbloshie/

#10yrsago In the 21st century, only corporations get to own property and we’re their tenants https://web.archive.org/web/20150428173001/https://www.theglobeandmail.com/technology/how-digital-rights-management-keeps-value-in-hands-of-the-manufacturer/article24130876/

#10yrsago Obituary for an amazing history teacher https://web.archive.org/web/20150426235723/https://thescientificparent.org/teachers-be-like-robin-barker-james/

#10yrsago What the UK Greens actually believe about copyright http://tomchance.org/2015/04/24/making-copyright-work-for-creatives/

#10yrsago School bus driver bans little girl from reading https://www.cbc.ca/news/canada/montreal/quebec-girl-told-to-stop-reading-book-by-school-bus-driver-1.3043652?cmp=rss

#10yrsago Variations on the Trolley Problem https://www.mcsweeneys.net/articles/lesser-known-trolley-problem-variations

#10yrsago Senators announce “Aaron Swartz Should Have Faced More Jail Time” bill https://www.techdirt.com/2015/04/23/senators-introduce-anti-aarons-law-to-increase-jail-terms-unauthorized-access-to-computers/

#10yrsago Kansas kid corrects anti-drug teacher, cops raid his house https://web.archive.org/web/20150423174017/http://benswann.com/exclusive-cops-raid-cannabis-oil-activist-because-her-son-discussed-medical-pot-facts-at-school/

#5yrsago Makers in a time of pandemic https://pluralistic.net/2020/04/25/send-pics/#makers

#5yrsago A deflationary pandemic https://pluralistic.net/2020/04/25/send-pics/#fiscal-dominance

#5yrsago The vernacular signage of the pandemic https://pluralistic.net/2020/04/25/send-pics/#frankfurt

#5yrsago California Adventure, Minecraft edition https://pluralistic.net/2020/04/24/slicey-boi/#minecraft

#5yrsago Security expert conned out of $10,000 https://pluralistic.net/2020/04/24/slicey-boi/#overconfidence

#5yrsago Facebook let advertisers target "pseudoscience" and "conspiracy" https://pluralistic.net/2020/04/24/slicey-boi/#upton-sinclair-disease

#5yrsago Amazon uses its sellers' data to figure out which products to clone https://pluralistic.net/2020/04/24/slicey-boi/#moral-hazard

#5yrsago US telcoms sector isn't doing better than Europe's https://pluralistic.net/2020/04/24/slicey-boi/#opportunists

#5yrsago Masks work https://pluralistic.net/2020/04/24/slicey-boi/#pewpew

#5yrsago US healthcare fails insured people too https://pluralistic.net/2020/04/24/slicey-boi/#m4a

#5yrsago "Inject disinfectant" vs both sides-ism https://pluralistic.net/2020/04/24/slicey-boi/#both-sides-ism

#5yrsago A labradoodle breeder is in charge of America's vaccines https://pluralistic.net/2020/04/24/slicey-boi/#labradoodles

#5yrsago Which guillotine is right for you https://pluralistic.net/2020/04/24/slicey-boi/#slicey-boi

#5yrsago Hospital cuts healthcare workers' pay, pays six-figure exec bonuses https://pluralistic.net/2020/04/27/in-this-together/#all-in-this-together

#5yrsago Pandemic proves ISP data-caps were always a pretense https://pluralistic.net/2020/04/27/in-this-together/#concast

#5yrsago Billionaires thriving on our pandemic losses https://pluralistic.net/2020/04/27/in-this-together/#socialized-losses

#5yrsago Podcasting Someone Comes to Town, Someone Leaves Town https://pluralistic.net/2020/04/27/in-this-together/#alan-abel-andrew

#5yrsago Indie booksellers during the pandemic https://pluralistic.net/2020/04/27/in-this-together/#glimmers-of-hope

#1yrago The tax sharks are back and they're coming for your home https://pluralistic.net/2024/04/27/for-the-little-people/#alden-capital

#1yrago The specific process by which Google enshittified its search https://pluralistic.net/2024/04/24/naming-names/#prabhakar-raghavan

#1yrago Antitrust is a labor issue https://pluralistic.net/2024/04/25/capri-v-tapestry/#aiming-at-dollars-not-men


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources: Slashdot (https://slashdot.org).

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-04-27T18:46:53+00:00 Fullscreen Open in Tab
Finished reading Legends & Lattes
Finished reading:
Cover image of Legends & Lattes
Legends & Lattes series, book 1.
Published . 296 pages.
Started ; completed April 27, 2025.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-04-27T15:33:38+00:00 Fullscreen Open in Tab
Note published on April 27, 2025 at 3:33 PM UTC
2025-04-27T01:44:39+00:00 Fullscreen Open in Tab
Finished reading A Captured Cauldron
Finished reading:
Cover image of A Captured Cauldron
Side Quest Row series, book 2.
Published . 334 pages.
Started ; completed April 26, 2025.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
2025-04-26T14:48:08+00:00 Fullscreen Open in Tab
Note published on April 26, 2025 at 2:48 PM UTC

Looking forward to the full-throated condemnation of this letter by those who accused the Biden administration of pressuring social media firms to moderate COVID-19 and election fraud misinfo! Any second now...

Ed Martin, interim US attorney for DC, has written a letter to the Wikimedia Foundation, threatening its status as a nonprofit entity.
Illustration of Molly White sitting and typing on a laptop, on a purple background with 'Molly White' in white serif.
Thu, 24 Apr 2025 11:56:57 +0000 Fullscreen Open in Tab
Pluralistic: Every complex ecosystem has parasites (24 Apr 2025)


Today's links



A rainforest in Chiapas, green and intergrown.

Every complex ecosystem has parasites (permalink)

Patrick "patio11" McKenzie is a fantastic explainer, the kind of person who breaks topics down in ways that stay with you, and creep into your understanding of other subjects, too. Take his 2022 essay, "The optimal amount of fraud is non-zero":

https://www.bitsaboutmoney.com/archive/optimal-amount-of-fraud/

It's a very well-argued piece, and here's the nut of it:

The marginal return of permitting fraud against you is plausibly greater than zero, and therefore, you should welcome greater than zero fraud.

In other words, if you allow some fraud, you will also allow through a lot of non-fraudulent business that would otherwise trip your fraud meter. Or, put it another way, the only way to prevent all fraud is to chase away a large proportion of your customers, whose transactions are in some way abnormal or unexpected.

Another great explainer is Bruce Schneier, the security expert. In the wake of 9/11, lots of pundits (and senior government officials) ran around saying, "No price is too high to prevent another terrorist attack on our aviation system." Schneier had a foolproof way of shutting these fools up: "Fine, just ground all civilian aircraft, forever." Turns out, there is a price that's too high to pay for preventing air-terrorism.

Latent in these two statements is the idea that the most secure systems are simple, and while simplicity is a fine goal to strive for, we should always keep in mind the maxim attributed to Einstein, "Everything should be made as simple as possible, but not simpler." That is to say, some things are just complicated.

20 years ago, my friend Kathryn Myronuk and I were talking about the spam wars, which were raging at the time. The spam wars were caused by the complexity of email: as a protocol (rather than a product), email is heterogeneous. There are lots of different kinds of email servers and clients, and many different ways of creating and rendering an email. All this flexibility makes email really popular, and it also means that users have a wide variety of use-cases for it. As a result, identifying spam is really hard. There's no reliable automated way of telling whether an email is spam or not – you can't just block a given server, or anyone using a kind of server software, or email client. You can't choose words or phrases to block and only block spam.

Many solutions were proposed to this at the height of the spam wars, and they all sucked, because they all assumed that the way the proposer used email was somehow typical, thus we could safely build a system to block things that were very different from this "typical" use and not catch too many dolphins in our tuna nets:

https://craphound.com/spamsolutions.txt

So Kathryn and I were talking about this, and she said, "Yeah, all complex ecosystems have parasites." I was thunderstruck. The phrase entered my head and never left. I even gave a major speech with that title later that year, at the O'Reilly Emerging Technology Conference:

https://craphound.com/complexecosystems.txt

Truly, a certain degree of undesirable activity is the inevitable price you pay once you make something general purpose, generative, and open. Open systems – like the web, or email – succeed because they are so adaptable, which means that all kinds of different people with different needs find ways to make use of them. The undesirable activity in open systems is, well, undesirable, and it's valid and useful to try to minimize it. But minimization isn't the same as elimination. "The optimal amount of fraud is non-zero," because "everything should be made as simple as possible, but not simpler." Complexity is generative, but "all complex ecosystems have parasites."

America is a complex system. It has, for example, a Social Security apparatus that has to serve more than 65 million people. By definition, a cohort of 65 million people will experience 65 one-in-a-million outliers every day. Social Security has to accommodate 65 million variations on the (surprisingly complicated) concept of a "street address":

https://gist.github.com/almereyda/85fa289bfc668777fe3619298bbf0886

It will have to cope with 65 million variations on the absolutely, maddeningly complicated idea of a "name":

https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/

In cybernetics, we say that a means of regulating a system must be capable of representing as many states as the system itself – that is, if you're building a control box for a thing with five functions, the box needs at least five different settings:

http://pespmc1.vub.ac.be/REQVAR.html

So when we're talking about managing something as complicated as Social Security, we need to build a Social Security Administration that is just as complicated. Anything that complicated is gonna have parasites – once you make something capable of managing the glorious higgledy piggledy that is the human experience of names, dates of birth, and addresses, you will necessarily create exploitable failure modes that bad actors can use to steal Social Security. You can build good fraud detection systems (as the SSA has), and you can investigate fraud (as the SSA does), and you can keep this to a manageable number – in the case of the SSA, that number is well below one percent:

https://www.congress.gov/crs_external_products/IF/PDF/IF12948/IF12948.2.pdf

But if you want to reduce Social Security fraud from "a fraction of one percent" to "zero percent," you can either expend a gigantic amount of money (far more than you're losing to fraud) to get a little closer to zero – or you can make Social Security far simpler. For example, you could simply declare that anyone whose life and work history can't fit in a simple database schema is not eligible for Social Security, kick tens of millions of people off the SSI rolls, and cause them to lose their homes and starve on the streets. This isn't merely cruel, it's also very, very expensive, since homelessness costs the system far more than Social Security. The optimum amount of fraud is non-zero.

Conservatives hate complexity. That's why the Trump administration banned all research grants for proposals that contained the word "systemic" (as a person with so-far-local cancer, I sure worry about what happens when and if my lymphoma become systemic). I once described the conservative yearning for "simpler times," as a desire to be a child again. After all, the thing that made your childhood "simpler" wasn't that the world was less complicated – it's that your parents managed that complexity and shielded you from it. There's always been partner abuse, divorce, gender minorities, mental illness, disability, racial discrimination, geopolitical crises, refugees, and class struggle. The only people who don't have to deal with this stuff are (lucky) children.

Complexity is an unavoidable attribute of all complicated processes. Evolution is complicated, so it produces complexity. It's convenient to think about a simplified model of genes in which individual genes produce specific traits, but it turns out genes all influence each other, are influenced in turn by epigenetics, and that developmental factors play a critical role in our outcomes. From eye-color to gender, evolution produces spectra, not binaries. It's ineluctably (and rather gloriously) complicated.

The conservative project to insist that things can be neatly categorized – animal or plant, man or woman, planet or comet – tries to take graceful bimodal curves and simplify them into a few simple straight lines – one or zero (except even the values of the miniature transistors on your computer's many chips are never at "one" or "zero" – they're "one-ish" and "mostly zero").

Like Social Security, fraud in the immigration system is a negligible rounding error. The US immigration system is a baroque, ramified, many-tendriled thing (I have the receipts from the immigration lawyers who helped me get a US visa, a green card, and citizenship to prove it). It is already so overweighted with pitfalls and traps for the unwary that a good immigration lawyer might send you to apply for a visa with 600 pages of documentation (the most I ever presented) just to make sure that every possible requirement is met:

https://www.flickr.com/photos/doctorow/2242342898/in/photolist-zp6PxJ-4q9Aqs-2nVHTZK-2pFKHyf

After my decades of experience with the US immigration system, I am prepared to say that the system is now at a stage where it is experiencing sharply diminishing returns from its anti-fraud systems. The cost of administering all this complexity is high, and the marginal amount of fraud caught by any new hoop the system gins up for migrants to jump through will round to zero.

Which poses a problem for Trump and trumpists: having whipped up a national panic about out of control immigration and open borders, the only way to make the system better at catching the infinitesimal amount of fraud it currently endures is to make the rules simpler, through the blunt-force tactic of simply excluding people who should be allowed in the country. For example, you could ban college kids planning to spend the summer in the US on the grounds that they didn't book all their hotels in advance, because they're planning to go from city to city and wing it:

https://www.newsweek.com/germany-tourists-deported-hotel-maria-lepere-charlotte-pohl-hawaii-2062046

Or you could ban the only research scientist in the world who knows how to interpret the results of the most promising new cancer imaging technology because a border guard was confused about the frog embryos she was transporting (she's been locked up for two months now):

https://www.msn.com/en-us/health/other/horrified-harvard-scientists-ice-arrest-leaves-cancer-researchers-scrambling/ar-AA1DlUt8

Of course, the US has long operated a policy of "anything that confuses a border guard is grounds for being refused entry" but the Trump administration has turned the odd, rare outrage into business-as-usual.

But they can lock up or turn away as many people as they want, and they still won't get the amount of fraud to zero. The US is a complicated place. People have complicated reasons for entering the USA – work, family reunion, leisure, research, study, and more. The only immigration system that doesn't leak a little at the seams is an immigration system that is so simple that it has no seams – a toy immigration system for a trivial country in which so little is going on that everything is going on.

The only garden without weeds is a monoculture under a dome. The only email system without spam is a closed system managed by one company that only allows a carefully vetted cluster of subscribers to communicate with one another. The only species with just two genders is one wherein members who fit somewhere else on the spectrum are banished or killed, a charnel process that never ends because there are always newborns that are outside of the first sigma of the two peaks in the bimodal distribution.

A living system – a real country – is complicated. It's a system, where people do things you'll never understand for perfectly good reasons (and vice versa). To accommodate all that complexity, we need complex systems, and all complex ecosystems have parasites. Yes, you can burn the rainforest to the ground and plant monocrops in straight rows, but then what you have is a farm, not a forest, vulnerable to pests and plagues and fire and flood. Complex systems have parasites, sure, but complex systems are resilient. The optimal level of fraud is never zero, because a system that has been simplified to the point where no fraud can take place within it is a system that is so trivial and brittle as to be useless.


Hey look at this (permalink)



A Wayback Machine banner.

Object permanence (permalink)

#20yrsago French court bans DRM for DVDs https://web.archive.org/web/20050424023258/https://www.01net.com/editorial/274752/droit/la-justice-interdit-de-proteger-les-dvd-contre-la-copie/

#20yrsago Why governments make stupid copyrights https://www.ft.com/content/39b697dc-b25e-11d9-bcc6-00000e2511c8

#20yrsago London Review of Books’s personals are really dirty and funny https://web.archive.org/web/20050426005000/http://www.lrb.co.uk/classified/index.php#PERSONALS

#20yrsago German crooner’s megaphone-style covers of modern rock https://www.palast-orchester.de/en

#15yrsago British Airways leaves stranded passengers all over world, jacks up prices on tickets home https://www.theguardian.com/news/blog/2010/apr/23/iceland-volcano-thousands-passengers-stranded

#15yrsago Google highlights fair use defense to YouTube takedowns https://publicpolicy.googleblog.com/2010/04/content-id-and-fair-use.html

#15yrsago Microsoft wins its $100M tax-break and amnesty from broke-ass Washington State https://web.archive.org/web/20100429061500/http://microsofttaxdodge.com/2010/04/microsoft-gets-nevada-royalty-tax-cut-and-tax-amnesty.html?all

#10yrsago Privilege: you’re probably not the one percent https://jacobin.com/2015/04/1-99-percent-class-inequality

#10yrsago Marissa Mayer makes 1,100 Yahooers jobless, calls it a “remix” https://web.archive.org/web/20150425183847/http://news.dice.com/2015/04/22/yahoo-called-its-layoffs-a-remix-dont-do-that/?CMPID=AF_SD_UP_JS_AV_OG_DNA_

#10yrsago Canadian Big Content spokesjerk says the public domain is against the public interest https://www.michaelgeist.ca/2015/04/canadian-recording-industry-works-entering-the-public-domain-are-not-in-the-public-interest/

#5yrsago Riot Baby https://pluralistic.net/2020/04/23/riot-baby/#Tochi-Onyebuchi

#5yrsago Mayor of Las Vegas says the "free market" will decide what's safe https://pluralistic.net/2020/04/23/riot-baby/#carolyn-goodman

#1yrago "Humans in the loop" must detect the hardest-to-spot errors, at superhuman speed https://pluralistic.net/2024/04/23/maximal-plausibility/#reverse-centaurs


Upcoming appearances (permalink)

A photo of me onstage, giving a speech, pounding the podium.



A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)



A grid of my books with Will Stahle covers..

Latest books (permalink)



A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Enshittification: Why Everything Suddenly Got Worse and What to Do About It, Farrar, Straus, Giroux, October 7 2025
    https://us.macmillan.com/books/9780374619329/enshittification/

  • Unauthorized Bread: a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • Enshittification, Why Everything Suddenly Got Worse and What to Do About It (the graphic novel), Firstsecond, 2026

  • The Memex Method, Farrar, Straus, Giroux, 2026



Colophon (permalink)

Today's top sources:

Currently writing:

  • Enshittification: a nonfiction book about platform decay for Farrar, Straus, Giroux. Status: second pass edit underway (readaloud)

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS FEB 2025

Latest podcast: Nimby and the D-Hoppers CONCLUSION https://craphound.com/stories/2025/04/13/nimby-and-the-d-hoppers-conclusion/


This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.


How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-04-03T16:39:37-07:00 Fullscreen Open in Tab
Let's fix OAuth in MCP

Let's not overthink auth in MCP.

Yes, the MCP server is going to need its own auth server. But it's not as bad as it sounds. Let me explain.

First let's get a few pieces of terminology straight.

The confusion that's happening in the discussions I've seen so far is because the spec and diagrams show that the MCP server itself is handing authorization. That's not necessary.

oauth roles

In OAuth, we talk about the "authorization server" and "resource server" as distinct roles. I like to think of the authorization server as the "token factory", that's the thing that makes the access tokens. The resource server (usually an API) needs to be able to validate the tokens created by the authorization server.

combined AS and RS

It's possible to build a single server that is both a resource server and authorization server, and in fact many OAuth systems are built that way, especially large consumer services.

separate AS and RS

But nothing about the spec requires that the two roles are combined, it's also possible to run these as two totally unrelated services.

This flexibility that's been baked into OAuth for over a decade is what has led to the rapid adoption, as well the proliferation of open source and commercial products that provide an OAuth authorization server as a service.

So how does this relate to MCP?

I can annotate the flow from the Model Context Protocol spec to show the parts where the client talks to the MCP Resource Server separately from where the client talks to the MCP Authorization Server.

MCP Flow showing AS and RS highlighted

Here is the updated sequence diagram showing communication with each role separately.

New MCP diagram showing separate AS and RS

Why is it important to call out this change?

I've seen a few conversations in various places about how requiring the MCP Server to be both an authorization server and resource server is too much of a burden. But actually, very little needs to change about the spec to enable this separation of concerns that OAuth already provides.

I've also seen various suggestions of other ways to separate the authorization server from the MCP server, like delegating to an enterprise IdP and having the MCP server validate access tokens issued by the IdP. These other options also conflate the OAuth roles in an awkward way and would result in some undesirable properties or relationships between the various parties involved.

So what needs to change in the MCP spec to enable this?

Discovery

The main thing currently forcing the MCP Server to be both the authorization server and resource server is how the client does discovery.

One design goal of MCP is to enable a client to bootstrap everything it needs based on only the server URL provided. I think this is a great design goal, and luckily is something that can be achieved even when separating the roles in the way I've described.

The MCP spec currently says that clients are expected to fetch the OAuth Server Metadata (RFC8414) file from the MCP Server base URL, resulting in a URL such as:

https://example.com/.well-known/oauth-authorization-server

This ends up meaning the MCP Resource Server must also be an Authorization Server, which leads to the complications the community has encountered so far. The good news is there is an OAuth spec we can apply here instead: Protected Resource Metadata.

Protected Resource Metadata

The Protected Resource Metadata spec is used by a Resource Server to advertise metadata about itself, including which Authorization Server can be used with it. This spec is both new and old. It was started in 2016, but was never adopted by the OAuth working group until 2023, after I had presented at an IETF meeting about the need for clients to be able to bootstrap OAuth flows given an OAuth resource server. The spec is now awaiting publication as an RFC, and should get its RFC number in a couple months.

Applying this to the MCP server would result in a sequence like the following:

New discovery flow for MCP

  1. The MCP Client fetches the Resource Server Metadata file by appending /.well-known/oauth-protected-resource to the MCP Server base URL.
  2. The MCP Client finds the authorization_servers property in the JSON response, and builds the Authorization Server Metadata URL by appending /.well-known/oauth-authorization-server
  3. The MCP Client fetches the Authorization Server Metadata to find the endpoints it needs for the OAuth flow, the authorization endpoint and token endpoint
  4. The MCP Client initiates an OAuth flow and continues as normal


Note: The Protected Resource Metadata spec also supports the Resource Server returning WWW-Authenticate with a link to the resource metadata URL if you want to avoid the requirement that MCP Servers host their metadata URLs at the .well-known endpoint, it just requires an extra HTTP request to support this.

Access Token Validation

Two things to keep in mind about how the MCP Server validates access tokens with this new separation of concerns.

If you do build the MCP Authorization Server and Resource Server as part of the same system, you don't need to do anything special to validate the access tokens the Authorization Server issues. You probably already have some sort of infrastructure in place for your normal API to validate tokens issued by your Authorization Server, so nothing changes there.

If you are using an external Authorization Server, whether that's an open source product or a commercial hosted service, that product will have its own docs for how you can validate the tokens it creates. There's a good chance it already supports the standardized JWT Access Tokens described in RFC 9068, in which case you can use off-the-shelf JWT validation middleware for common frameworks.

In either case, the critical design goal here is that the MCP Authorization Server issues access tokens that only ever need to be validated by the MCP Resource Server. This is in line with the security recommendations in Section 2.3 of RFC 9700, in particular that "access tokens SHOULD be audience-restricted to a specific resource server". In other words, it would be a bad idea for the MCP Client to be issued an access token that works with both the MCP Resource Server and the service's REST API.

Why Require the MCP Server to have an Authorization Server in the first place?

Another argument I've seen is that MCP Server developers shouldn't have to build any OAuth infrastructure at all, instead they should be able to delegate all the OAuth bits to an external service.

In principle, I agree. Getting API access and authorization right is tricky, that's why there are entire companies dedicated to solving the problem.

The architecture laid out above enables this exact separation of concerns. The difference between this architecture and some of the other proposals I've seen is that this cleanly separates the security boundaries so that there are minimal dependencies among the parties involved.

But, one thing I haven't seen mentioned in the discussions is that there actually is no requirement than an OAuth Authorization Server provide any UI itself.

An Authorization Server with no UI?

While it is desirable from a security perspective that the MCP Resource Server has a corresponding Authorization Server that issues access tokens for it, that Authorization Server doesn't actually need to have any UI or even any concept of user login or accounts. You can actually build an Authorization Server that delegates all user account management to an external service. You can see an example of this in PayPal's MCP server they recently launched.

PayPal's traditional API already supports OAuth, the authorization and token endpoints are:

  • https://www.paypal.com/signin/authorize
  • https://api-m.paypal.com/v1/oauth2/token

When PayPal built their MCP server, they launched it at https://mcp.paypal.com. If you fetch the metadata for the MCP Server, you'll find the two OAuth endpoints for the MCP Authorization Server:

  • https://mcp.paypal.com/authorize
  • https://mcp.paypal.com/token

When the MCP Client redirects the user to the authorization endpoint, the MCP server itself doesn't provide any UI. Instead, it immediately redirects the user to the real PayPal authorization endpoint which then prompts the user to log in and authorize the client.

Roles with backend API and Authorization Servers

This points to yet another benefit of architecting the MCP Authorization Server and Resource Server this way. It enables implementers to delegate the actual user management to their existing OAuth server with no changes needed to the MCP Client. The MCP Client isn't even aware that this extra redirect step was inserted in the middle. As far as the MCP Client is concerned, it has been talking to only the MCP Authorization Server. It just so happens that the MCP Authorization Server has sent the user elsewhere to actually log in.

Dynamic Client Registration

There's one more point I want to make about why having a dedicated MCP Authorization Server is helpful architecturally.

The MCP spec strongly recommends that MCP Servers (authorization servers) support Dynamic Client Registration. If MCP is successful, there will be a large number of MCP Clients talking to a large number of MCP Servers, and the user is the one deciding which combinations of clients and servers to use. This means it is not scalable to require that every MCP Client developer register their client with every MCP Server.

This is similar to the idea of using an email client with the user's chosen email server. Obviously Mozilla can't register Thunderbird with every email server out there. Instead, there needs to be a way to dynamically establish a client's identity with the OAuth server at runtime. Dynamic Client Registration is one option for how to do that.

The problem is most commercial APIs are not going to enable Dynamic Client Registration on their production servers. For example, in order to get client credentials to use the Google APIs, you need to register as a developer and then register an OAuth client after logging in. Dynamic Client Registration would allow a client to register itself without the link to the developer's account. That would mean there is no paper trail for who the client was developed by. The Dynamic Client Registration endpoint can't require authentication by definition, so is a public endpoint that can create clients, which as you can imagine opens up some potential security issues.

I do, however, think it would be reasonable to expect production services to enable Dynamic Client Registration only on the MCP's Authorization Server. This way the dynamically-registered clients wouldn't be able to use the regular REST API, but would only be able to interact with the MCP API.

Mastodon and BlueSky also have a similar problem of needing clients to show up at arbitrary authorization servers without prior coordination between the client developer and authorization server operator. I call this the "OAuth for the Open Web" problem. Mastodon used Dynamic Client Registration as their solution, and has since documented some of the issues that this creates, linked here and here.

BlueSky decided to take a different approach and instead uses an https URL as a client identifier, bypassing the need for a client registration step entirely. This has the added bonus of having at least some level of confidence of the client identity because the client identity is hosted at a domain. It would be a perfectly viable approach to use this method for MCP as well. There is a discussion on that within MCP here. This is an ongoing topic within the OAuth working group, I have a couple of drafts in progress to formalize this pattern, Client ID Metadata Document and Client ID Scheme.

Enterprise IdP Integration

Lastly, I want to touch on the idea of enabling users to log in to MCP Servers with their enterprise IdP.

When an enterprise company purchases software, they expect to be able to tie it in to their single-sign-on solution. For example, when I log in to work Slack, I enter my work email and Slack redirects me to my work IdP where I log in. This way employees don't need to have passwords with every app they use in the enterprise, they can log in to everything with the same enterprise account, and all the apps can be protected with multi-factor authentication through the IdP. This also gives the company control over which users can access which apps, as well as a way to revoke a user's access at any time.

So how does this relate to MCP?

Well, plenty of people are already trying to figure out how to let their employees safely use AI tools within the enterprise. So we need a way to let employees use their enterprise IdP to log in and authorize MCP Clients to access MCP Servers.

If you're building an MCP Server in front of an existing application that already supports enterprise Single Sign-On, then you don't need to do anything differently in the MCP Client or Server and you already have support for this. When the MCP Client redirects to the MCP Authorization Server, the MCP Authorization Server redirects to the main Authorization Server, which would then prompt the user for their company email/domain and redirect to the enterprise IdP to log in.

This brings me to yet another thing I've been seeing conflated in the discussions: user login and user authorization.

OAuth is an authorization delegation protocol. OAuth doesn't actually say anything about how users authenticate at the OAuth server, it only talks about how the user can authorize access to an application. This is actually a really great thing, because it means we can get super creative with how users authenticate.

User logs in and authorizes

Remember the yellow box "User logs in and authorizes" from the original sequence diagram? These are actually two totally distinct steps. The OAuth authorization server is responsible for getting the user to log in somehow, but there's no requirement that how the user logs in is with a username/password. This is where we can insert a single-sign-on flow to an enterprise IdP, or really anything you can imagine.

So think of this as two separate boxes: "user logs in", and "user authorizes". Then, we can replace the "user logs in" box with an entirely new OpenID Connect flow out to the enterprise IdP to log the user in, and after they are logged in they can authorize the client.

User logs in with OIDC

I'll spare you the complete expanded sequence diagram, since it looks a lot more complicated than it actually is. But I again want to stress that this is nothing new, this is already how things are commonly done today.

This all just becomes cleaner to understand when you separate the MCP Authorization Server from the MCP Resource Server.

We can push all the complexity of user login, token minting, and more onto the MCP Authorization Server, keeping the MCP Resource Server free to do the much simpler task of validating access tokens and serving resources.

Future Improvements of Enterprise IdP Integration

There are two things I want to call out about how enterprise IdP integration could be improved. Both of these are entire topics on their own, so I will only touch on the problems and link out to other places where work is happening to solve them.

There are two points of friction with the current state of enterprise login for SaaS apps.

  • IdP discovery
  • User consent

IdP Discovery

When a user logs in to a SaaS app, they need to tell the app how to find their enterprise IdP. This is commonly done by either asking the user to enter their work email, or asking the user to enter their tenant URL at the service.

Sign in with SSO

Neither of these is really a great user experience. It would be a lot better if the browser already knew which enterprise IdP the user should be sent to. This is one of my goals with the work happening in FedCM. With this new browser API, the browser can mediate the login, telling the SaaS app which enterprise IdP to use automatically only needing the user to click their account icon rather than type anything in.

User Consent

Another point of friction in the enterprise happens when a user starts connecting multiple applications to each other within the company. For example, if you drop in a Google Docs link into Slack, Slack will prompt you to connect your Google account to preview the link. Multiply this by N number of applications that can preview links, and M number of applications you might drop links to, and you end up sending the user through a huge number of OAuth consent flows.

The problem is only made worse with the explosion of AI tools. Every AI tool will need access to data in every other application in the enterprise. That is a lot of OAuth consent flows for the user to manage. Plus, the user shouldn't really be the one granting consent for Slack to access the company Google Docs account anyway. That consent should ideally be managed by the enterprise IT admin.

What we actually need is a way to enable the IT admin to grant consent for apps to talk to each other company-wide, removing the need for users to be sent through an OAuth flow at all.

This is the basis of another OAuth spec I've been working on, the Identity Assertion Authorization Grant.

The same problem applies to MCP Servers, and with the separation of concerns laid out above, it becomes straightforward to add this extension to move the consent to the enterprise and streamline the user experience.

Get in touch!

If these sound like interesting problems, please get in touch! You can find me on LinkedIn or reach me via email at aaron@parecki.com.

2025-03-07T00:00:00+00:00 Fullscreen Open in Tab
Standards for ANSI escape codes

Hello! Today I want to talk about ANSI escape codes.

For a long time I was vaguely aware of ANSI escape codes (“that’s how you make text red in the terminal and stuff”) but I had no real understanding of where they were supposed to be defined or whether or not there were standards for them. I just had a kind of vague “there be dragons” feeling around them. While learning about the terminal this year, I’ve learned that:

  1. ANSI escape codes are responsible for a lot of usability improvements in the terminal (did you know there’s a way to copy to your system clipboard when SSHed into a remote machine?? It’s an escape code called OSC 52!)
  2. They aren’t completely standardized, and because of that they don’t always work reliably. And because they’re also invisible, it’s extremely frustrating to troubleshoot escape code issues.

So I wanted to put together a list for myself of some standards that exist around escape codes, because I want to know if they have to feel unreliable and frustrating, or if there’s a future where we could all rely on them with more confidence.

what’s an escape code?

Have you ever pressed the left arrow key in your terminal and seen ^[[D? That’s an escape code! It’s called an “escape code” because the first character is the “escape” character, which is usually written as ESC, \x1b, \E, \033, or ^[.

Escape codes are how your terminal emulator communicates various kinds of information (colours, mouse movement, etc) with programs running in the terminal. There are two kind of escape codes:

  1. input codes which your terminal emulator sends for keypresses or mouse movements that don’t fit into Unicode. For example “left arrow key” is ESC[D, “Ctrl+left arrow” might be ESC[1;5D, and clicking the mouse might be something like ESC[M :3.
  2. output codes which programs can print out to colour text, move the cursor around, clear the screen, hide the cursor, copy text to the clipboard, enable mouse reporting, set the window title, etc.

Now let’s talk about standards!

ECMA-48

The first standard I found relating to escape codes was ECMA-48, which was originally published in 1976.

ECMA-48 does two things:

  1. Define some general formats for escape codes (like “CSI” codes, which are ESC[ + something and “OSC” codes, which are ESC] + something)
  2. Define some specific escape codes, like how “move the cursor to the left” is ESC[D, or “turn text red” is ESC[31m. In the spec, the “cursor left” one is called CURSOR LEFT and the one for changing colours is called SELECT GRAPHIC RENDITION.

The formats are extensible, so there’s room for others to define more escape codes in the future. Lots of escape codes that are popular today aren’t defined in ECMA-48: for example it’s pretty common for terminal applications (like vim, htop, or tmux) to support using the mouse, but ECMA-48 doesn’t define escape codes for the mouse.

xterm control sequences

There are a bunch of escape codes that aren’t defined in ECMA-48, for example:

  • enabling mouse reporting (where did you click in your terminal?)
  • bracketed paste (did you paste that text or type it in?)
  • OSC 52 (which terminal applications can use to copy text to your system clipboard)

I believe (correct me if I’m wrong!) that these and some others came from xterm, are documented in XTerm Control Sequences, and have been widely implemented by other terminal emulators.

This list of “what xterm supports” is not a standard exactly, but xterm is extremely influential and so it seems like an important document.

terminfo

In the 80s (and to some extent today, but my understanding is that it was MUCH more dramatic in the 80s) there was a huge amount of variation in what escape codes terminals actually supported.

To deal with this, there’s a database of escape codes for various terminals called “terminfo”.

It looks like the standard for terminfo is called X/Open Curses, though you need to create an account to view that standard for some reason. It defines the database format as well as a C library interface (“curses”) for accessing the database.

For example you can run this bash snippet to see every possible escape code for “clear screen” for all of the different terminals your system knows about:

for term in $(toe -a | awk '{print $1}')
do
  echo $term
  infocmp -1 -T "$term" 2>/dev/null | grep 'clear=' | sed 's/clear=//g;s/,//g'
done

On my system (and probably every system I’ve ever used?), the terminfo database is managed by ncurses.

should programs use terminfo?

I think it’s interesting that there are two main approaches that applications take to handling ANSI escape codes:

  1. Use the terminfo database to figure out which escape codes to use, depending on what’s in the TERM environment variable. Fish does this, for example.
  2. Identify a “single common set” of escape codes which works in “enough” terminal emulators and just hardcode those.

Some examples of programs/libraries that take approach #2 (“don’t use terminfo”) include:

I got curious about why folks might be moving away from terminfo and I found this very interesting and extremely detailed rant about terminfo from one of the fish maintainers, which argues that:

[the terminfo authors] have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is.

I’m not going to do it justice so I’m not going to summarize it, I think it’s worth reading.

is there a “single common set” of escape codes?

I was just talking about the idea that you can use a “common set” of escape codes that will work for most people. But what is that set? Is there any agreement?

I really do not know the answer to this at all, but from doing some reading it seems like it’s some combination of:

  • The codes that the VT100 supported (though some aren’t relevant on modern terminals)
  • what’s in ECMA-48 (which I think also has some things that are no longer relevant)
  • What xterm supports (though I’d guess that not everything in there is actually widely supported enough)

and maybe ultimately “identify the terminal emulators you think your users are going to use most frequently and test in those”, the same way web developers do when deciding which CSS features are okay to use

I don’t think there are any resources like Can I use…? or Baseline for the terminal though. (in theory terminfo is supposed to be the “caniuse” for the terminal but it seems like it often takes 10+ years to add new terminal features when people invent them which makes it very limited)

some reasons to use terminfo

I also asked on Mastodon why people found terminfo valuable in 2025 and got a few reasons that made sense to me:

  • some people expect to be able to use the TERM environment variable to control how programs behave (for example with TERM=dumb), and there’s no standard for how that should work in a post-terminfo world
  • even though there’s less variation between terminal emulators than there was in the 80s, there’s far from zero variation: there are graphical terminals, the Linux framebuffer console, the situation you’re in when connecting to a server via its serial console, Emacs shell mode, and probably more that I’m missing
  • there is no one standard for what the “single common set” of escape codes is, and sometimes programs use escape codes which aren’t actually widely supported enough

terminfo & user agent detection

The way that ncurses uses the TERM environment variable to decide which escape codes to use reminds me of how webservers used to sometimes use the browser user agent to decide which version of a website to serve.

It also seems like it’s had some of the same results – the way iTerm2 reports itself as being “xterm-256color” feels similar to how Safari’s user agent is “Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15”. In both cases the terminal emulator / browser ends up changing its user agent to get around user agent detection that isn’t working well.

On the web we ended up deciding that user agent detection was not a good practice and to instead focus on standardization so we can serve the same HTML/CSS to all browsers. I don’t know if the same approach is the future in the terminal though – I think the terminal landscape today is much more fragmented than the web ever was as well as being much less well funded.

some more documents/standards

A few more documents and standards related to escape codes, in no particular order:

why I think this is interesting

I sometimes see people saying that the unix terminal is “outdated”, and since I love the terminal so much I’m always curious about what incremental changes might make it feel less “outdated”.

Maybe if we had a clearer standards landscape (like we do on the web!) it would be easier for terminal emulator developers to build new features and for authors of terminal applications to more confidently adopt those features so that we can all benefit from them and have a richer experience in the terminal.

Obviously standardizing ANSI escape codes is not easy (ECMA-48 was first published almost 50 years ago and we’re still not there!). I don’t even know what all of the challenges are. But the situation with HTML/CSS/JS used to be extremely bad too and now it’s MUCH better, so maybe there’s hope.

2025-02-13T12:27:56+00:00 Fullscreen Open in Tab
How to add a directory to your PATH

I was talking to a friend about how to add a directory to your PATH today. It’s something that feels “obvious” to me since I’ve been using the terminal for a long time, but when I searched for instructions for how to do it, I actually couldn’t find something that explained all of the steps – a lot of them just said “add this to ~/.bashrc”, but what if you’re not using bash? What if your bash config is actually in a different file? And how are you supposed to figure out which directory to add anyway?

So I wanted to try to write down some more complete directions and mention some of the gotchas I’ve run into over the years.

Here’s a table of contents:

step 1: what shell are you using?

If you’re not sure what shell you’re using, here’s a way to find out. Run this:

ps -p $$ -o pid,comm=
  • if you’re using bash, it’ll print out 97295 bash
  • if you’re using zsh, it’ll print out 97295 zsh
  • if you’re using fish, it’ll print out an error like “In fish, please use $fish_pid” ($$ isn’t valid syntax in fish, but in any case the error message tells you that you’re using fish, which you probably already knew)

Also bash is the default on Linux and zsh is the default on Mac OS (as of 2024). I’ll only cover bash, zsh, and fish in these directions.

step 2: find your shell’s config file

  • in zsh, it’s probably ~/.zshrc
  • in bash, it might be ~/.bashrc, but it’s complicated, see the note in the next section
  • in fish, it’s probably ~/.config/fish/config.fish (you can run echo $__fish_config_dir if you want to be 100% sure)

a note on bash’s config file

Bash has three possible config files: ~/.bashrc, ~/.bash_profile, and ~/.profile.

If you’re not sure which one your system is set up to use, I’d recommend testing this way:

  1. add echo hi there to your ~/.bashrc
  2. Restart your terminal
  3. If you see “hi there”, that means ~/.bashrc is being used! Hooray!
  4. Otherwise remove it and try the same thing with ~/.bash_profile
  5. You can also try ~/.profile if the first two options don’t work.

(there are a lot of elaborate flow charts out there that explain how bash decides which config file to use but IMO it’s not worth it to internalize them and just testing is the fastest way to be sure)

step 3: figure out which directory to add

Let’s say that you’re trying to install and run a program called http-server and it doesn’t work, like this:

$ npm install -g http-server
$ http-server
bash: http-server: command not found

How do you find what directory http-server is in? Honestly in general this is not that easy – often the answer is something like “it depends on how npm is configured”. A few ideas:

  • Often when setting up a new installer (like cargo, npm, homebrew, etc), when you first set it up it’ll print out some directions about how to update your PATH. So if you’re paying attention you can get the directions then.
  • Sometimes installers will automatically update your shell’s config file to update your PATH for you
  • Sometimes just Googling “where does npm install things?” will turn up the answer
  • Some tools have a subcommand that tells you where they’re configured to install things, like:
    • Node/npm: npm config get prefix (then append /bin/)
    • Go: go env GOPATH (then append /bin/)
    • asdf: asdf info | grep ASDF_DIR (then append /bin/ and /shims/)

step 3.1: double check it’s the right directory

Once you’ve found a directory you think might be the right one, make sure it’s actually correct! For example, I found out that on my machine, http-server is in ~/.npm-global/bin. I can make sure that it’s the right directory by trying to run the program http-server in that directory like this:

$ ~/.npm-global/bin/http-server
Starting up http-server, serving ./public

It worked! Now that you know what directory you need to add to your PATH, let’s move to the next step!

step 4: edit your shell config

Now we have the 2 critical pieces of information we need:

  1. Which directory you’re trying to add to your PATH (like ~/.npm-global/bin/)
  2. Where your shell’s config is (like ~/.bashrc, ~/.zshrc, or ~/.config/fish/config.fish)

Now what you need to add depends on your shell:

bash instructions:

Open your shell’s config file, and add a line like this:

export PATH=$PATH:~/.npm-global/bin/

(obviously replace ~/.npm-global/bin with the actual directory you’re trying to add)

zsh instructions:

You can do the same thing as in bash, but zsh also has some slightly fancier syntax you can use if you prefer:

path=(
  $path
  ~/.npm-global/bin
)

fish instructions:

In fish, the syntax is different:

set PATH $PATH ~/.npm-global/bin

(in fish you can also use fish_add_path, some notes on that further down)

step 5: restart your shell

Now, an extremely important step: updating your shell’s config won’t take effect if you don’t restart it!

Two ways to do this:

  1. open a new terminal (or terminal tab), and maybe close the old one so you don’t get confused
  2. Run bash to start a new shell (or zsh if you’re using zsh, or fish if you’re using fish)

I’ve found that both of these usually work fine.

And you should be done! Try running the program you were trying to run and hopefully it works now.

If not, here are a couple of problems that you might run into:

problem 1: it ran the wrong program

If the wrong version of a program is running, you might need to add the directory to the beginning of your PATH instead of the end.

For example, on my system I have two versions of python3 installed, which I can see by running which -a:

$ which -a python3
/usr/bin/python3
/opt/homebrew/bin/python3

The one your shell will use is the first one listed.

If you want to use the Homebrew version, you need to add that directory (/opt/homebrew/bin) to the beginning of your PATH instead, by putting this in your shell’s config file (it’s /opt/homebrew/bin/:$PATH instead of the usual $PATH:/opt/homebrew/bin/)

export PATH=/opt/homebrew/bin/:$PATH

or in fish:

set PATH ~/.cargo/bin $PATH

problem 2: the program isn’t being run from your shell

All of these directions only work if you’re running the program from your shell. If you’re running the program from an IDE, from a GUI, in a cron job, or some other way, you’ll need to add the directory to your PATH in a different way, and the exact details might depend on the situation.

in a cron job

Some options:

  • use the full path to the program you’re running, like /home/bork/bin/my-program
  • put the full PATH you want as the first line of your crontab (something like PATH=/bin:/usr/bin:/usr/local/bin:….). You can get the full PATH you’re using in your shell by running echo "PATH=$PATH".

I’m honestly not sure how to handle it in an IDE/GUI because I haven’t run into that in a long time, will add directions here if someone points me in the right direction.

problem 3: duplicate PATH entries making it harder to debug

If you edit your path and start a new shell by running bash (or zsh, or fish), you’ll often end up with duplicate PATH entries, because the shell keeps adding new things to your PATH every time you start your shell.

Personally I don’t think I’ve run into a situation where this kind of duplication breaks anything, but the duplicates can make it harder to debug what’s going on with your PATH if you’re trying to understand its contents.

Some ways you could deal with this:

  1. If you’re debugging your PATH, open a new terminal to do it in so you get a “fresh” state. This should avoid the duplication.
  2. Deduplicate your PATH at the end of your shell’s config (for example in zsh apparently you can do this with typeset -U path)
  3. Check that the directory isn’t already in your PATH when adding it (for example in fish I believe you can do this with fish_add_path --path /some/directory)

How to deduplicate your PATH is shell-specific and there isn’t always a built in way to do it so you’ll need to look up how to accomplish it in your shell.

problem 4: losing your history after updating your PATH

Here’s a situation that’s easy to get into in bash or zsh:

  1. Run a command (it fails)
  2. Update your PATH
  3. Run bash to reload your config
  4. Press the up arrow a couple of times to rerun the failed command (or open a new terminal)
  5. The failed command isn’t in your history! Why not?

This happens because in bash, by default, history is not saved until you exit the shell.

Some options for fixing this:

  • Instead of running bash to reload your config, run source ~/.bashrc (or source ~/.zshrc in zsh). This will reload the config inside your current session.
  • Configure your shell to continuously save your history instead of only saving the history when the shell exits. (How to do this depends on whether you’re using bash or zsh, the history options in zsh are a bit complicated and I’m not exactly sure what the best way is)

a note on source

When you install cargo (Rust’s installer) for the first time, it gives you these instructions for how to set up your PATH, which don’t mention a specific directory at all.

This is usually done by running one of the following (note the leading DOT):

. "$HOME/.cargo/env"        	# For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish

The idea is that you add that line to your shell’s config, and their script automatically sets up your PATH (and potentially other things) for you.

This is pretty common (for example Homebrew suggests you eval brew shellenv), and there are two ways to approach this:

  1. Just do what the tool suggests (like adding . "$HOME/.cargo/env" to your shell’s config)
  2. Figure out which directories the script they’re telling you to run would add to your PATH, and then add those manually. Here’s how I’d do that:
    • Run . "$HOME/.cargo/env" in my shell (or the fish version if using fish)
    • Run echo "$PATH" | tr ':' '\n' | grep cargo to figure out which directories it added
    • See that it says /Users/bork/.cargo/bin and shorten that to ~/.cargo/bin
    • Add the directory ~/.cargo/bin to PATH (with the directions in this post)

I don’t think there’s anything wrong with doing what the tool suggests (it might be the “best way”!), but personally I usually use the second approach because I prefer knowing exactly what configuration I’m changing.

a note on fish_add_path

fish has a handy function called fish_add_path that you can run to add a directory to your PATH like this:

fish_add_path /some/directory

This is cool (it’s such a simple command!) but I’ve stopped using it for a couple of reasons:

  1. Sometimes fish_add_path will update the PATH for every session in the future (with a “universal variable”) and sometimes it will update the PATH just for the current session and it’s hard for me to tell which one it will do. In theory the docs explain this but I could not understand them.
  2. If you ever need to remove the directory from your PATH a few weeks or months later because maybe you made a mistake, it’s kind of hard to do (there are instructions in this comments of this github issue though).

that’s all

Hopefully this will help some people. Let me know (on Mastodon or Bluesky) if you there are other major gotchas that have tripped you up when adding a directory to your PATH, or if you have questions about this post!

2025-02-05T16:57:00+00:00 Fullscreen Open in Tab
Some terminal frustrations

A few weeks ago I ran a terminal survey (you can read the results here) and at the end I asked:

What’s the most frustrating thing about using the terminal for you?

1600 people answered, and I decided to spend a few days categorizing all the responses. Along the way I learned that classifying qualitative data is not easy but I gave it my best shot. I ended up building a custom tool to make it faster to categorize everything.

As with all of my surveys the methodology isn’t particularly scientific. I just posted the survey to Mastodon and Twitter, ran it for a couple of days, and got answers from whoever happened to see it and felt like responding.

Here are the top categories of frustrations!

I think it’s worth keeping in mind while reading these comments that

  • 40% of people answering this survey have been using the terminal for 21+ years
  • 95% of people answering the survey have been using the terminal for at least 4 years

These comments aren’t coming from total beginners.

Here are the categories of frustrations! The number in brackets is the number of people with that frustration. I’m mostly writing this up for myself because I’m trying to write a zine about the terminal and I wanted to get a sense for what people are having trouble with.

remembering syntax (115)

People talked about struggles remembering:

  • the syntax for CLI tools like awk, jq, sed, etc
  • the syntax for redirects
  • keyboard shortcuts for tmux, text editing, etc

One example comment:

There are just so many little “trivia” details to remember for full functionality. Even after all these years I’ll sometimes forget where it’s 2 or 1 for stderr, or forget which is which for > and >>.

switching terminals is hard (91)

People talked about struggling with switching systems (for example home/work computer or when SSHing) and running into:

  • OS differences in keyboard shortcuts (like Linux vs Mac)
  • systems which don’t have their preferred text editor (“no vim” or “only vim”)
  • different versions of the same command (like Mac OS grep vs GNU grep)
  • no tab completion
  • a shell they aren’t used to (“the subtle differences between zsh and bash”)

as well as differences inside the same system like pagers being not consistent with each other (git diff pagers, other pagers).

One example comment:

I got used to fish and vi mode which are not available when I ssh into servers, containers.

color (85)

Lots of problems with color, like:

  • programs setting colors that are unreadable with a light background color
  • finding a colorscheme they like (and getting it to work consistently across different apps)
  • color not working inside several layers of SSH/tmux/etc
  • not liking the defaults
  • not wanting color at all and struggling to turn it off

This comment felt relatable to me:

Getting my terminal theme configured in a reasonable way between the terminal emulator and fish (I did this years ago and remember it being tedious and fiddly and now feel like I’m locked into my current theme because it works and I dread touching any of that configuration ever again).

keyboard shortcuts (84)

Half of the comments on keyboard shortcuts were about how on Linux/Windows, the keyboard shortcut to copy/paste in the terminal is different from in the rest of the OS.

Some other issues with keyboard shortcuts other than copy/paste:

  • using Ctrl-W in a browser-based terminal and closing the window
  • the terminal only supports a limited set of keyboard shortcuts (no Ctrl-Shift-, no Super, no Hyper, lots of ctrl- shortcuts aren’t possible like Ctrl-,)
  • the OS stopping you from using a terminal keyboard shortcut (like by default Mac OS uses Ctrl+left arrow for something else)
  • issues using emacs in the terminal
  • backspace not working (2)

other copy and paste issues (75)

Aside from “the keyboard shortcut for copy and paste is different”, there were a lot of OTHER issues with copy and paste, like:

  • copying over SSH
  • how tmux and the terminal emulator both do copy/paste in different ways
  • dealing with many different clipboards (system clipboard, vim clipboard, the “middle click” clipboard on Linux, tmux’s clipboard, etc) and potentially synchronizing them
  • random spaces added when copying from the terminal
  • pasting multiline commands which automatically get run in a terrifying way
  • wanting a way to copy text without using the mouse

discoverability (55)

There were lots of comments about this, which all came down to the same basic complaint – it’s hard to discover useful tools or features! This comment kind of summed it all up:

How difficult it is to learn independently. Most of what I know is an assorted collection of stuff I’ve been told by random people over the years.

steep learning curve (44)

A lot of comments about it generally having a steep learning curve. A couple of example comments:

After 15 years of using it, I’m not much faster than using it than I was 5 or maybe even 10 years ago.

and

That I know I could make my life easier by learning more about the shortcuts and commands and configuring the terminal but I don’t spend the time because it feels overwhelming.

history (42)

Some issues with shell history:

  • history not being shared between terminal tabs (16)
  • limits that are too short (4)
  • history not being restored when terminal tabs are restored
  • losing history because the terminal crashed
  • not knowing how to search history

One example comment:

It wasted a lot of time until I figured it out and still annoys me that “history” on zsh has such a small buffer; I have to type “history 0” to get any useful length of history.

bad documentation (37)

People talked about:

  • documentation being generally opaque
  • lack of examples in man pages
  • programs which don’t have man pages

Here’s a representative comment:

Finding good examples and docs. Man pages often not enough, have to wade through stack overflow

scrollback (36)

A few issues with scrollback:

  • programs printing out too much data making you lose scrollback history
  • resizing the terminal messes up the scrollback
  • lack of timestamps
  • GUI programs that you start in the background printing stuff out that gets in the way of other programs’ outputs

One example comment:

When resizing the terminal (in particular: making it narrower) leads to broken rewrapping of the scrollback content because the commands formatted their output based on the terminal window width.

“it feels outdated” (33)

Lots of comments about how the terminal feels hampered by legacy decisions and how users often end up needing to learn implementation details that feel very esoteric. One example comment:

Most of the legacy cruft, it would be great to have a green field implementation of the CLI interface.

shell scripting (32)

Lots of complaints about POSIX shell scripting. There’s a general feeling that shell scripting is difficult but also that switching to a different less standard scripting language (fish, nushell, etc) brings its own problems.

Shell scripting. My tolerance to ditch a shell script and go to a scripting language is pretty low. It’s just too messy and powerful. Screwing up can be costly so I don’t even bother.

more issues

Some more issues that were mentioned at least 10 times:

  • (31) inconsistent command line arguments: is it -h or help or –help?
  • (24) keeping dotfiles in sync across different systems
  • (23) performance (e.g. “my shell takes too long to start”)
  • (20) window management (potentially with some combination of tmux tabs, terminal tabs, and multiple terminal windows. Where did that shell session go?)
  • (17) generally feeling scared/uneasy (“The debilitating fear that I’m going to do some mysterious Bad Thing with a command and I will have absolutely no idea how to fix or undo it or even really figure out what happened”)
  • (16) terminfo issues (“Having to learn about terminfo if/when I try a new terminal emulator and ssh elsewhere.”)
  • (16) lack of image support (sixel etc)
  • (15) SSH issues (like having to start over when you lose the SSH connection)
  • (15) various tmux/screen issues (for example lack of integration between tmux and the terminal emulator)
  • (15) typos & slow typing
  • (13) the terminal getting messed up for various reasons (pressing Ctrl-S, cating a binary, etc)
  • (12) quoting/escaping in the shell
  • (11) various Windows/PowerShell issues

n/a (122)

There were also 122 answers to the effect of “nothing really” or “only that I can’t do EVERYTHING in the terminal”

One example comment:

Think I’ve found work arounds for most/all frustrations

that’s all!

I’m not going to make a lot of commentary on these results, but here are a couple of categories that feel related to me:

  • remembering syntax & history (often the thing you need to remember is something you’ve run before!)
  • discoverability & the learning curve (the lack of discoverability is definitely a big part of what makes it hard to learn)
  • “switching systems is hard” & “it feels outdated” (tools that haven’t really changed in 30 or 40 years have many problems but they do tend to be always there no matter what system you’re on, which is very useful and makes them hard to stop using)

Trying to categorize all these results in a reasonable way really gave me an appreciation for social science researchers’ skills.

2025-01-11T09:46:01+00:00 Fullscreen Open in Tab
What's involved in getting a "modern" terminal setup?

Hello! Recently I ran a terminal survey and I asked people what frustrated them. One person commented:

There are so many pieces to having a modern terminal experience. I wish it all came out of the box.

My immediate reaction was “oh, getting a modern terminal experience isn’t that hard, you just need to….”, but the more I thought about it, the longer the “you just need to…” list got, and I kept thinking about more and more caveats.

So I thought I would write down some notes about what it means to me personally to have a “modern” terminal experience and what I think can make it hard for people to get there.

what is a “modern terminal experience”?

Here are a few things that are important to me, with which part of the system is responsible for them:

  • multiline support for copy and paste: if you paste 3 commands in your shell, it should not immediately run them all! That’s scary! (shell, terminal emulator)
  • infinite shell history: if I run a command in my shell, it should be saved forever, not deleted after 500 history entries or whatever. Also I want commands to be saved to the history immediately when I run them, not only when I exit the shell session (shell)
  • a useful prompt: I can’t live without having my current directory and current git branch in my prompt (shell)
  • 24-bit colour: this is important to me because I find it MUCH easier to theme neovim with 24-bit colour support than in a terminal with only 256 colours (terminal emulator)
  • clipboard integration between vim and my operating system so that when I copy in Firefox, I can just press p in vim to paste (text editor, maybe the OS/terminal emulator too)
  • good autocomplete: for example commands like git should have command-specific autocomplete (shell)
  • having colours in ls (shell config)
  • a terminal theme I like: I spend a lot of time in my terminal, I want it to look nice and I want its theme to match my terminal editor’s theme. (terminal emulator, text editor)
  • automatic terminal fixing: If a programs prints out some weird escape codes that mess up my terminal, I want that to automatically get reset so that my terminal doesn’t get messed up (shell)
  • keybindings: I want Ctrl+left arrow to work (shell or application)
  • being able to use the scroll wheel in programs like less: (terminal emulator and applications)

There are a million other terminal conveniences out there and different people value different things, but those are the ones that I would be really unhappy without.

how I achieve a “modern experience”

My basic approach is:

  1. use the fish shell. Mostly don’t configure it, except to:
    • set the EDITOR environment variable to my favourite terminal editor
    • alias ls to ls --color=auto
  2. use any terminal emulator with 24-bit colour support. In the past I’ve used GNOME Terminal, Terminator, and iTerm, but I’m not picky about this. I don’t really configure it other than to choose a font.
  3. use neovim, with a configuration that I’ve been very slowly building over the last 9 years or so (the last time I deleted my vim config and started from scratch was 9 years ago)
  4. use the base16 framework to theme everything

A few things that affect my approach:

  • I don’t spend a lot of time SSHed into other machines
  • I’d rather use the mouse a little than come up with keyboard-based ways to do everything
  • I work on a lot of small projects, not one big project

some “out of the box” options for a “modern” experience

What if you want a nice experience, but don’t want to spend a lot of time on configuration? Figuring out how to configure vim in a way that I was satisfied with really did take me like ten years, which is a long time!

My best ideas for how to get a reasonable terminal experience with minimal config are:

  • shell: either fish or zsh with oh-my-zsh
  • terminal emulator: almost anything with 24-bit colour support, for example all of these are popular:
    • linux: GNOME Terminal, Konsole, Terminator, xfce4-terminal
    • mac: iTerm (Terminal.app doesn’t have 256-colour support)
    • cross-platform: kitty, alacritty, wezterm, or ghostty
  • shell config:
    • set the EDITOR environment variable to your favourite terminal text editor
    • maybe alias ls to ls --color=auto
  • text editor: this is a tough one, maybe micro or helix? I haven’t used either of them seriously but they both seem like very cool projects and I think it’s amazing that you can just use all the usual GUI editor commands (Ctrl-C to copy, Ctrl-V to paste, Ctrl-A to select all) in micro and they do what you’d expect. I would probably try switching to helix except that retraining my vim muscle memory seems way too hard. Also helix doesn’t have a GUI or plugin system yet.

Personally I wouldn’t use xterm, rxvt, or Terminal.app as a terminal emulator, because I’ve found in the past that they’re missing core features (like 24-bit colour in Terminal.app’s case) that make the terminal harder to use for me.

I don’t want to pretend that getting a “modern” terminal experience is easier than it is though – I think there are two issues that make it hard. Let’s talk about them!

issue 1 with getting to a “modern” experience: the shell

bash and zsh are by far the two most popular shells, and neither of them provide a default experience that I would be happy using out of the box, for example:

  • you need to customize your prompt
  • they don’t come with git completions by default, you have to set them up
  • by default, bash only stores 500 (!) lines of history and (at least on Mac OS) zsh is only configured to store 2000 lines, which is still not a lot
  • I find bash’s tab completion very frustrating, if there’s more than one match then you can’t tab through them

And even though I love fish, the fact that it isn’t POSIX does make it hard for a lot of folks to make the switch.

Of course it’s totally possible to learn how to customize your prompt in bash or whatever, and it doesn’t even need to be that complicated (in bash I’d probably start with something like export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ ', or maybe use starship). But each of these “not complicated” things really does add up and it’s especially tough if you need to keep your config in sync across several systems.

An extremely popular solution to getting a “modern” shell experience is oh-my-zsh. It seems like a great project and I know a lot of people use it very happily, but I’ve struggled with configuration systems like that in the past – it looks like right now the base oh-my-zsh adds about 3000 lines of config, and often I find that having an extra configuration system makes it harder to debug what’s happening when things go wrong. I personally have a tendency to use the system to add a lot of extra plugins, make my system slow, get frustrated that it’s slow, and then delete it completely and write a new config from scratch.

issue 2 with getting to a “modern” experience: the text editor

In the terminal survey I ran recently, the most popular terminal text editors by far were vim, emacs, and nano.

I think the main options for terminal text editors are:

  • use vim or emacs and configure it to your liking, you can probably have any feature you want if you put in the work
  • use nano and accept that you’re going to have a pretty limited experience (for example I don’t think you can select text with the mouse and then “cut” it in nano)
  • use micro or helix which seem to offer a pretty good out-of-the-box experience, potentially occasionally run into issues with using a less mainstream text editor
  • just avoid using a terminal text editor as much as possible, maybe use VSCode, use VSCode’s terminal for all your terminal needs, and mostly never edit files in the terminal. Or I know a lot of people use code as their EDITOR in the terminal.

issue 3: individual applications

The last issue is that sometimes individual programs that I use are kind of annoying. For example on my Mac OS machine, /usr/bin/sqlite3 doesn’t support the Ctrl+Left Arrow keyboard shortcut. Fixing this to get a reasonable terminal experience in SQLite was a little complicated, I had to:

  • realize why this is happening (Mac OS won’t ship GNU tools, and “Ctrl-Left arrow” support comes from GNU readline)
  • find a workaround (install sqlite from homebrew, which does have readline support)
  • adjust my environment (put Homebrew’s sqlite3 in my PATH)

I find that debugging application-specific issues like this is really not easy and often it doesn’t feel “worth it” – often I’ll end up just dealing with various minor inconveniences because I don’t want to spend hours investigating them. The only reason I was even able to figure this one out at all is that I’ve been spending a huge amount of time thinking about the terminal recently.

A big part of having a “modern” experience using terminal programs is just using newer terminal programs, for example I can’t be bothered to learn a keyboard shortcut to sort the columns in top, but in htop I can just click on a column heading with my mouse to sort it. So I use htop instead! But discovering new more “modern” command line tools isn’t easy (though I made a list here), finding ones that I actually like using in practice takes time, and if you’re SSHed into another machine, they won’t always be there.

everything affects everything else

Something I find tricky about configuring my terminal to make everything “nice” is that changing one seemingly small thing about my workflow can really affect everything else. For example right now I don’t use tmux. But if I needed to use tmux again (for example because I was doing a lot of work SSHed into another machine), I’d need to think about a few things, like:

  • if I wanted tmux’s copy to synchronize with my system clipboard over SSH, I’d need to make sure that my terminal emulator has OSC 52 support
  • if I wanted to use iTerm’s tmux integration (which makes tmux tabs into iTerm tabs), I’d need to change how I configure colours – right now I set them with a shell script that I run when my shell starts, but that means the colours get lost when restoring a tmux session.

and probably more things I haven’t thought of. “Using tmux means that I have to change how I manage my colours” sounds unlikely, but that really did happen to me and I decided “well, I don’t want to change how I manage colours right now, so I guess I’m not using that feature!”.

It’s also hard to remember which features I’m relying on – for example maybe my current terminal does have OSC 52 support and because copying from tmux over SSH has always Just Worked I don’t even realize that that’s something I need, and then it mysteriously stops working when I switch terminals.

change things slowly

Personally even though I think my setup is not that complicated, it’s taken me 20 years to get to this point! Because terminal config changes are so likely to have unexpected and hard-to-understand consequences, I’ve found that if I change a lot of terminal configuration all at once it makes it much harder to understand what went wrong if there’s a problem, which can be really disorienting.

So I usually prefer to make pretty small changes, and accept that changes can might take me a REALLY long time to get used to. For example I switched from using ls to eza a year or two ago and while I like it (because eza -l prints human-readable file sizes by default) I’m still not quite sure about it. But also sometimes it’s worth it to make a big change, like I made the switch to fish (from bash) 10 years ago and I’m very happy I did.

getting a “modern” terminal is not that easy

Trying to explain how “easy” it is to configure your terminal really just made me think that it’s kind of hard and that I still sometimes get confused.

I’ve found that there’s never one perfect way to configure things in the terminal that will be compatible with every single other thing. I just need to try stuff, figure out some kind of locally stable state that works for me, and accept that if I start using a new tool it might disrupt the system and I might need to rethink things.

2024-12-12T09:28:22+00:00 Fullscreen Open in Tab
"Rules" that terminal programs follow

Recently I’ve been thinking about how everything that happens in the terminal is some combination of:

  1. Your operating system’s job
  2. Your shell’s job
  3. Your terminal emulator’s job
  4. The job of whatever program you happen to be running (like top or vim or cat)

The first three (your operating system, shell, and terminal emulator) are all kind of known quantities – if you’re using bash in GNOME Terminal on Linux, you can more or less reason about how how all of those things interact, and some of their behaviour is standardized by POSIX.

But the fourth one (“whatever program you happen to be running”) feels like it could do ANYTHING. How are you supposed to know how a program is going to behave?

This post is kind of long so here’s a quick table of contents:

programs behave surprisingly consistently

As far as I know, there are no real standards for how programs in the terminal should behave – the closest things I know of are:

  • POSIX, which mostly dictates how your terminal emulator / OS / shell should work together. I think it does specify a few things about how core utilities like cp should work but AFAIK it doesn’t have anything to say about how for example htop should behave.
  • these command line interface guidelines

But even though there are no standards, in my experience programs in the terminal behave in a pretty consistent way. So I wanted to write down a list of “rules” that in my experience programs mostly follow.

these are meant to be descriptive, not prescriptive

My goal here isn’t to convince authors of terminal programs that they should follow any of these rules. There are lots of exceptions to these and often there’s a good reason for those exceptions.

But it’s very useful for me to know what behaviour to expect from a random new terminal program that I’m using. Instead of “uh, programs could do literally anything”, it’s “ok, here are the basic rules I expect, and then I can keep a short mental list of exceptions”.

So I’m just writing down what I’ve observed about how programs behave in my 20 years of using the terminal, why I think they behave that way, and some examples of cases where that rule is “broken”.

it’s not always obvious which “rules” are the program’s responsibility to implement

There are a bunch of common conventions that I think are pretty clearly the program’s responsibility to implement, like:

  • config files should go in ~/.BLAHrc or ~/.config/BLAH/FILE or /etc/BLAH/ or something
  • --help should print help text
  • programs should print “regular” output to stdout and errors to stderr

But in this post I’m going to focus on things that it’s not 100% obvious are the program’s responsibility. For example it feels to me like a “law of nature” that pressing Ctrl-D should quit a REPL, but programs often need to explicitly implement support for it – even though cat doesn’t need to implement Ctrl-D support, ipython does. (more about that in “rule 3” below)

Understanding which things are the program’s responsibility makes it much less surprising when different programs’ implementations are slightly different.

rule 1: noninteractive programs should quit when you press Ctrl-C

The main reason for this rule is that noninteractive programs will quit by default on Ctrl-C if they don’t set up a SIGINT signal handler, so this is kind of a “you should act like the default” rule.

Something that trips a lot of people up is that this doesn’t apply to interactive programs like python3 or bc or less. This is because in an interactive program, Ctrl-C has a different job – if the program is running an operation (like for example a search in less or some Python code in python3), then Ctrl-C will interrupt that operation but not stop the program.

As an example of how this works in an interactive program: here’s the code in prompt-toolkit (the library that iPython uses for handling input) that aborts a search when you press Ctrl-C.

rule 2: TUIs should quit when you press q

TUI programs (like less or htop) will usually quit when you press q.

This rule doesn’t apply to any program where pressing q to quit wouldn’t make sense, like tmux or text editors.

rule 3: REPLs should quit when you press Ctrl-D on an empty line

REPLs (like python3 or ed) will usually quit when you press Ctrl-D on an empty line. This rule is similar to the Ctrl-C rule – the reason for this is that by default if you’re running a program (like cat) in “cooked mode”, then the operating system will return an EOF when you press Ctrl-D on an empty line.

Most of the REPLs I use (sqlite3, python3, fish, bash, etc) don’t actually use cooked mode, but they all implement this keyboard shortcut anyway to mimic the default behaviour.

For example, here’s the code in prompt-toolkit that quits when you press Ctrl-D, and here’s the same code in readline.

I actually thought that this one was a “Law of Terminal Physics” until very recently because I’ve basically never seen it broken, but you can see that it’s just something that each individual input library has to implement in the links above.

Someone pointed out that the Erlang REPL does not quit when you press Ctrl-D, so I guess not every REPL follows this “rule”.

rule 4: don’t use more than 16 colours

Terminal programs rarely use colours other than the base 16 ANSI colours. This is because if you specify colours with a hex code, it’s very likely to clash with some users’ background colour. For example if I print out some text as #EEEEEE, it would be almost invisible on a white background, though it would look fine on a dark background.

But if you stick to the default 16 base colours, you have a much better chance that the user has configured those colours in their terminal emulator so that they work reasonably well with their background color. Another reason to stick to the default base 16 colours is that it makes less assumptions about what colours the terminal emulator supports.

The only programs I usually see breaking this “rule” are text editors, for example Helix by default will use a purple background which is not a default ANSI colour. It seems fine for Helix to break this rule since Helix isn’t a “core” program and I assume any Helix user who doesn’t like that colorscheme will just change the theme.

rule 5: vaguely support readline keybindings

Almost every program I use supports readline keybindings if it would make sense to do so. For example, here are a bunch of different programs and a link to where they define Ctrl-E to go to the end of the line:

None of those programs actually uses readline directly, they just sort of mimic emacs/readline keybindings. They don’t always mimic them exactly: for example atuin seems to use Ctrl-A as a prefix, so Ctrl-A doesn’t go to the beginning of the line.

Also all of these programs seem to implement their own internal cut and paste buffers so you can delete a line with Ctrl-U and then paste it with Ctrl-Y.

The exceptions to this are:

  • some programs (like git, cat, and nc) don’t have any line editing support at all (except for backspace, Ctrl-W, and Ctrl-U)
  • as usual text editors are an exception, every text editor has its own approach to editing text

I wrote more about this “what keybindings does a program support?” question in entering text in the terminal is complicated.

rule 5.1: Ctrl-W should delete the last word

I’ve never seen a program (other than a text editor) where Ctrl-W doesn’t delete the last word. This is similar to the Ctrl-C rule – by default if a program is in “cooked mode”, the OS will delete the last word if you press Ctrl-W, and delete the whole line if you press Ctrl-U. So usually programs will imitate that behaviour.

I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!

rule 6: disable colours when writing to a pipe

Most programs will disable colours when writing to a pipe. For example:

  • rg blah will highlight all occurrences of blah in the output, but if the output is to a pipe or a file, it’ll turn off the highlighting.
  • ls --color=auto will use colour when writing to a terminal, but not when writing to a pipe

Both of those programs will also format their output differently when writing to the terminal: ls will organize files into columns, and ripgrep will group matches with headings.

If you want to force the program to use colour (for example because you want to look at the colour), you can use unbuffer to force the program’s output to be a tty like this:

unbuffer rg blah |  less -R

I’m sure that there are some programs that “break” this rule but I can’t think of any examples right now. Some programs have an --color flag that you can use to force colour to be on, in the example above you could also do rg --color=always | less -R.

rule 7: - means stdin/stdout

Usually if you pass - to a program instead of a filename, it’ll read from stdin or write to stdout (whichever is appropriate). For example, if you want to format the Python code that’s on your clipboard with black and then copy it, you could run:

pbpaste | black - | pbcopy

(pbpaste is a Mac program, you can do something similar on Linux with xclip)

My impression is that most programs implement this if it would make sense and I can’t think of any exceptions right now, but I’m sure there are many exceptions.

these “rules” take a long time to learn

These rules took me a long time for me to learn because I had to:

  1. learn that the rule applied anywhere at all ("Ctrl-C will exit programs")
  2. notice some exceptions (“okay, Ctrl-C will exit find but not less”)
  3. subconsciously figure out what the pattern is ("Ctrl-C will generally quit noninteractive programs, but in interactive programs it might interrupt the current operation instead of quitting the program")
  4. eventually maybe formulate it into an explicit rule that I know

A lot of my understanding of the terminal is honestly still in the “subconscious pattern recognition” stage. The only reason I’ve been taking the time to make things explicit at all is because I’ve been trying to explain how it works to others. Hopefully writing down these “rules” explicitly will make learning some of this stuff a little bit faster for others.

2024-11-29T08:23:31+00:00 Fullscreen Open in Tab
Why pipes sometimes get "stuck": buffering

Here’s a niche terminal problem that has bothered me for years but that I never really understood until a few weeks ago. Let’s say you’re running this command to watch for some specific output in a log file:

tail -f /some/log/file | grep thing1 | grep thing2

If log lines are being added to the file relatively slowly, the result I’d see is… nothing! It doesn’t matter if there were matches in the log file or not, there just wouldn’t be any output.

I internalized this as “uh, I guess pipes just get stuck sometimes and don’t show me the output, that’s weird”, and I’d handle it by just running grep thing1 /some/log/file | grep thing2 instead, which would work.

So as I’ve been doing a terminal deep dive over the last few months I was really excited to finally learn exactly why this happens.

why this happens: buffering

The reason why “pipes get stuck” sometimes is that it’s VERY common for programs to buffer their output before writing it to a pipe or file. So the pipe is working fine, the problem is that the program never even wrote the data to the pipe!

This is for performance reasons: writing all output immediately as soon as you can uses more system calls, so it’s more efficient to save up data until you have 8KB or so of data to write (or until the program exits) and THEN write it to the pipe.

In this example:

tail -f /some/log/file | grep thing1 | grep thing2

the problem is that grep thing1 is saving up all of its matches until it has 8KB of data to write, which might literally never happen.

programs don’t buffer when writing to a terminal

Part of why I found this so disorienting is that tail -f file | grep thing will work totally fine, but then when you add the second grep, it stops working!! The reason for this is that the way grep handles buffering depends on whether it’s writing to a terminal or not.

Here’s how grep (and many other programs) decides to buffer its output:

  • Check if stdout is a terminal or not using the isatty function
    • If it’s a terminal, use line buffering (print every line immediately as soon as you have it)
    • Otherwise, use “block buffering” – only print data if you have at least 8KB or so of data to print

So if grep is writing directly to your terminal then you’ll see the line as soon as it’s printed, but if it’s writing to a pipe, you won’t.

Of course the buffer size isn’t always 8KB for every program, it depends on the implementation. For grep the buffering is handled by libc, and libc’s buffer size is defined in the BUFSIZ variable. Here’s where that’s defined in glibc.

(as an aside: “programs do not use 8KB output buffers when writing to a terminal” isn’t, like, a law of terminal physics, a program COULD use an 8KB buffer when writing output to a terminal if it wanted, it would just be extremely weird if it did that, I can’t think of any program that behaves that way)

commands that buffer & commands that don’t

One annoying thing about this buffering behaviour is that you kind of need to remember which commands buffer their output when writing to a pipe.

Some commands that don’t buffer their output:

  • tail
  • cat
  • tee

I think almost everything else will buffer output, especially if it’s a command where you’re likely to be using it for batch processing. Here’s a list of some common commands that buffer their output when writing to a pipe, along with the flag that disables block buffering.

  • grep (--line-buffered)
  • sed (-u)
  • awk (there’s a fflush() function)
  • tcpdump (-l)
  • jq (-u)
  • tr (-u)
  • cut (can’t disable buffering)

Those are all the ones I can think of, lots of unix commands (like sort) may or may not buffer their output but it doesn’t matter because sort can’t do anything until it finishes receiving input anyway.

Also I did my best to test both the Mac OS and GNU versions of these but there are a lot of variations and I might have made some mistakes.

programming languages where the default “print” statement buffers

Also, here are a few programming language where the default print statement will buffer output when writing to a pipe, and some ways to disable buffering if you want:

  • C (disable with setvbuf)
  • Python (disable with python -u, or PYTHONUNBUFFERED=1, or sys.stdout.reconfigure(line_buffering=False), or print(x, flush=True))
  • Ruby (disable with STDOUT.sync = true)
  • Perl (disable with $| = 1)

I assume that these languages are designed this way so that the default print function will be fast when you’re doing batch processing.

Also whether output is buffered or not might depend on how you print, for example in C++ cout << "hello\n" buffers when writing to a pipe but cout << "hello" << endl will flush its output.

when you press Ctrl-C on a pipe, the contents of the buffer are lost

Let’s say you’re running this command as a hacky way to watch for DNS requests to example.com, and you forgot to pass -l to tcpdump:

sudo tcpdump -ni any port 53 | grep example.com

When you press Ctrl-C, what happens? In a magical perfect world, what I would want to happen is for tcpdump to flush its buffer, grep would search for example.com, and I would see all the output I missed.

But in the real world, what happens is that all the programs get killed and the output in tcpdump’s buffer is lost.

I think this problem is probably unavoidable – I spent a little time with strace to see how this works and grep receives the SIGINT before tcpdump anyway so even if tcpdump tried to flush its buffer grep would already be dead.

After a little more investigation, there is a workaround: if you find tcpdump’s PID and kill -TERM $PID, then tcpdump will flush the buffer so you can see the output. That’s kind of a pain but I tested it and it seems to work.

redirecting to a file also buffers

It’s not just pipes, this will also buffer:

sudo tcpdump -ni any port 53 > output.txt

Redirecting to a file doesn’t have the same “Ctrl-C will totally destroy the contents of the buffer” problem though – in my experience it usually behaves more like you’d want, where the contents of the buffer get written to the file before the program exits. I’m not 100% sure whether this is something you can always rely on or not.

a bunch of potential ways to avoid buffering

Okay, let’s talk solutions. Let’s say you’ve run this command:

tail -f /some/log/file | grep thing1 | grep thing2

I asked people on Mastodon how they would solve this in practice and there were 5 basic approaches. Here they are:

solution 1: run a program that finishes quickly

Historically my solution to this has been to just avoid the “command writing to pipe slowly” situation completely and instead run a program that will finish quickly like this:

cat /some/log/file | grep thing1 | grep thing2 | tail

This doesn’t do the same thing as the original command but it does mean that you get to avoid thinking about these weird buffering issues.

(you could also do grep thing1 /some/log/file but I often prefer to use an “unnecessary” cat)

solution 2: remember the “line buffer” flag to grep

You could remember that grep has a flag to avoid buffering and pass it like this:

tail -f /some/log/file | grep --line-buffered thing1 | grep thing2

solution 3: use awk

Some people said that if they’re specifically dealing with a multiple greps situation, they’ll rewrite it to use a single awk instead, like this:

tail -f /some/log/file |  awk '/thing1/ && /thing2/'

Or you would write a more complicated grep, like this:

tail -f /some/log/file |  grep -E 'thing1.*thing2'

(awk also buffers, so for this to work you’ll want awk to be the last command in the pipeline)

solution 4: use stdbuf

stdbuf uses LD_PRELOAD to turn off libc’s buffering, and you can use it to turn off output buffering like this:

tail -f /some/log/file | stdbuf -o0 grep thing1 | grep thing2

Like any LD_PRELOAD solution it’s a bit unreliable – it doesn’t work on static binaries, I think won’t work if the program isn’t using libc’s buffering, and doesn’t always work on Mac OS. Harry Marr has a really nice How stdbuf works post.

solution 5: use unbuffer

unbuffer program will force the program’s output to be a TTY, which means that it’ll behave the way it normally would on a TTY (less buffering, colour output, etc). You could use it in this example like this:

tail -f /some/log/file | unbuffer grep thing1 | grep thing2

Unlike stdbuf it will always work, though it might have unwanted side effects, for example grep thing1’s will also colour matches.

If you want to install unbuffer, it’s in the expect package.

that’s all the solutions I know about!

It’s a bit hard for me to say which one is “best”, I think personally I’m mostly likely to use unbuffer because I know it’s always going to work.

If I learn about more solutions I’ll try to add them to this post.

I’m not really sure how often this comes up

I think it’s not very common for me to have a program that slowly trickles data into a pipe like this, normally if I’m using a pipe a bunch of data gets written very quickly, processed by everything in the pipeline, and then everything exits. The only examples I can come up with right now are:

  • tcpdump
  • tail -f
  • watching log files in a different way like with kubectl logs
  • the output of a slow computation

what if there were an environment variable to disable buffering?

I think it would be cool if there were a standard environment variable to turn off buffering, like PYTHONUNBUFFERED in Python. I got this idea from a couple of blog posts by Mark Dominus in 2018. Maybe NO_BUFFER like NO_COLOR?

The design seems tricky to get right; Mark points out that NETBSD has environment variables called STDBUF, STDBUF1, etc which gives you a ton of control over buffering but I imagine most developers don’t want to implement many different environment variables to handle a relatively minor edge case.

I’m also curious about whether there are any programs that just automatically flush their output buffers after some period of time (like 1 second). It feels like it would be nice in theory but I can’t think of any program that does that so I imagine there are some downsides.

stuff I left out

Some things I didn’t talk about in this post since these posts have been getting pretty long recently and seriously does anyone REALLY want to read 3000 words about buffering?

  • the difference between line buffering and having totally unbuffered output
  • how buffering to stderr is different from buffering to stdout
  • this post is only about buffering that happens inside the program, your operating system’s TTY driver also does a little bit of buffering sometimes
  • other reasons you might need to flush your output other than “you’re writing to a pipe”
2024-11-18T09:35:42+00:00 Fullscreen Open in Tab
Importing a frontend Javascript library without a build system

I like writing Javascript without a build system and for the millionth time yesterday I ran into a problem where I needed to figure out how to import a Javascript library in my code without using a build system, and it took FOREVER to figure out how to import it because the library’s setup instructions assume that you’re using a build system.

Luckily at this point I’ve mostly learned how to navigate this situation and either successfully use the library or decide it’s too difficult and switch to a different library, so here’s the guide I wish I had to importing Javascript libraries years ago.

I’m only going to talk about using Javacript libraries on the frontend, and only about how to use them in a no-build-system setup.

In this post I’m going to talk about:

  1. the three main types of Javascript files a library might provide (ES Modules, the “classic” global variable kind, and CommonJS)
  2. how to figure out which types of files a Javascript library includes in its build
  3. ways to import each type of file in your code

the three kinds of Javascript files

There are 3 basic types of Javascript files a library can provide:

  1. the “classic” type of file that defines a global variable. This is the kind of file that you can just <script src> and it’ll Just Work. Great if you can get it but not always available
  2. an ES module (which may or may not depend on other files, we’ll get to that)
  3. a “CommonJS” module. This is for Node, you can’t use it in a browser at all without using a build system.

I’m not sure if there’s a better name for the “classic” type but I’m just going to call it “classic”. Also there’s a type called “AMD” but I’m not sure how relevant it is in 2024.

Now that we know the 3 types of files, let’s talk about how to figure out which of these the library actually provides!

where to find the files: the NPM build

Every Javascript library has a build which it uploads to NPM. You might be thinking (like I did originally) – Julia! The whole POINT is that we’re not using Node to build our library! Why are we talking about NPM?

But if you’re using a link from a CDN like https://cdnjs.cloudflare.com/ajax/libs/Chart.js/4.4.1/chart.umd.min.js, you’re still using the NPM build! All the files on the CDNs originally come from NPM.

Because of this, I sometimes like to npm install the library even if I’m not planning to use Node to build my library at all – I’ll just create a new temp folder, npm install there, and then delete it when I’m done. I like being able to poke around in the files in the NPM build on my filesystem, because then I can be 100% sure that I’m seeing everything that the library is making available in its build and that the CDN isn’t hiding something from me.

So let’s npm install a few libraries and try to figure out what types of Javascript files they provide in their builds!

example library 1: chart.js

First let’s look inside Chart.js, a plotting library.

$ cd /tmp/whatever
$ npm install chart.js
$ cd node_modules/chart.js/dist
$ ls *.*js
chart.cjs  chart.js  chart.umd.js  helpers.cjs  helpers.js

This library seems to have 3 basic options:

option 1: chart.cjs. The .cjs suffix tells me that this is a CommonJS file, for using in Node. This means it’s impossible to use it directly in the browser without some kind of build step.

option 2:chart.js. The .js suffix by itself doesn’t tell us what kind of file it is, but if I open it up, I see import '@kurkle/color'; which is an immediate sign that this is an ES module – the import ... syntax is ES module syntax.

option 3: chart.umd.js. “UMD” stands for “Universal Module Definition”, which I think means that you can use this file either with a basic <script src>, CommonJS, or some third thing called AMD that I don’t understand.

how to use a UMD file

When I was using Chart.js I picked Option 3. I just needed to add this to my code:

<script src="./chart.umd.js"> </script>

and then I could use the library with the global Chart environment variable. Couldn’t be easier. I just copied chart.umd.js into my Git repository so that I didn’t have to worry about using NPM or the CDNs going down or anything.

the build files aren’t always in the dist directory

A lot of libraries will put their build in the dist directory, but not always! The build files’ location is specified in the library’s package.json.

For example here’s an excerpt from Chart.js’s package.json.

  "jsdelivr": "./dist/chart.umd.js",
  "unpkg": "./dist/chart.umd.js",
  "main": "./dist/chart.cjs",
  "module": "./dist/chart.js",

I think this is saying that if you want to use an ES Module (module) you should use dist/chart.js, but the jsDelivr and unpkg CDNs should use ./dist/chart.umd.js. I guess main is for Node.

chart.js’s package.json also says "type": "module", which according to this documentation tells Node to treat files as ES modules by default. I think it doesn’t tell us specifically which files are ES modules and which ones aren’t but it does tell us that something in there is an ES module.

example library 2: @atcute/oauth-browser-client

@atcute/oauth-browser-client is a library for logging into Bluesky with OAuth in the browser.

Let’s see what kinds of Javascript files it provides in its build!

$ npm install @atcute/oauth-browser-client
$ cd node_modules/@atcute/oauth-browser-client/dist
$ ls *js
constants.js  dpop.js  environment.js  errors.js  index.js  resolvers.js

It seems like the only plausible root file in here is index.js, which looks something like this:

export { configureOAuth } from './environment.js';
export * from './errors.js';
export * from './resolvers.js';

This export syntax means it’s an ES module. That means we can use it in the browser without a build step! Let’s see how to do that.

how to use an ES module with importmaps

Using an ES module isn’t an easy as just adding a <script src="whatever.js">. Instead, if the ES module has dependencies (like @atcute/oauth-browser-client does) the steps are:

  1. Set up an import map in your HTML
  2. Put import statements like import { configureOAuth } from '@atcute/oauth-browser-client'; in your JS code
  3. Include your JS code in your HTML like this: <script type="module" src="YOURSCRIPT.js"></script>

The reason we need an import map instead of just doing something like import { BrowserOAuthClient } from "./oauth-client-browser.js" is that internally the module has more import statements like import {something} from @atcute/client, and we need to tell the browser where to get the code for @atcute/client and all of its other dependencies.

Here’s what the importmap I used looks like for @atcute/oauth-browser-client:

<script type="importmap">
{
  "imports": {
    "nanoid": "./node_modules/nanoid/bin/dist/index.js",
    "nanoid/non-secure": "./node_modules/nanoid/non-secure/index.js",
    "nanoid/url-alphabet": "./node_modules/nanoid/url-alphabet/dist/index.js",
    "@atcute/oauth-browser-client": "./node_modules/@atcute/oauth-browser-client/dist/index.js",
    "@atcute/client": "./node_modules/@atcute/client/dist/index.js",
    "@atcute/client/utils/did": "./node_modules/@atcute/client/dist/utils/did.js"
  }
}
</script>

Getting these import maps to work is pretty fiddly, I feel like there must be a tool to generate them automatically but I haven’t found one yet. It’s definitely possible to write a script that automatically generates the importmaps using esbuild’s metafile but I haven’t done that and maybe there’s a better way.

I decided to set up importmaps yesterday to get github.com/jvns/bsky-oauth-example to work, so there’s some example code in that repo.

Also someone pointed me to Simon Willison’s download-esm, which will download an ES module and rewrite the imports to point to the JS files directly so that you don’t need importmaps. I haven’t tried it yet but it seems like a great idea.

problems with importmaps: too many files

I did run into some problems with using importmaps in the browser though – it needed to download dozens of Javascript files to load my site, and my webserver in development couldn’t keep up for some reason. I kept seeing files fail to load randomly and then had to reload the page and hope that they would succeed this time.

It wasn’t an issue anymore when I deployed my site to production, so I guess it was a problem with my local dev environment.

Also one slightly annoying thing about ES modules in general is that you need to be running a webserver to use them, I’m sure this is for a good reason but it’s easier when you can just open your index.html file without starting a webserver.

Because of the “too many files” thing I think actually using ES modules with importmaps in this way isn’t actually that appealing to me, but it’s good to know it’s possible.

how to use an ES module without importmaps

If the ES module doesn’t have dependencies then it’s even easier – you don’t need the importmaps! You can just:

  • put <script type="module" src="YOURCODE.js"></script> in your HTML. The type="module" is important.
  • put import {whatever} from "https://example.com/whatever.js" in YOURCODE.js

alternative: use esbuild

If you don’t want to use importmaps, you can also use a build system like esbuild. I talked about how to do that in Some notes on using esbuild, but this blog post is about ways to avoid build systems completely so I’m not going to talk about that option here. I do still like esbuild though and I think it’s a good option in this case.

what’s the browser support for importmaps?

CanIUse says that importmaps are in “Baseline 2023: newly available across major browsers” so my sense is that in 2024 that’s still maybe a little bit too new? I think I would use importmaps for some fun experimental code that I only wanted like myself and 12 people to use, but if I wanted my code to be more widely usable I’d use esbuild instead.

example library 3: @atproto/oauth-client-browser

Let’s look at one final example library! This is a different Bluesky auth library than @atcute/oauth-browser-client.

$ npm install @atproto/oauth-client-browser
$ cd node_modules/@atproto/oauth-client-browser/dist
$ ls *js
browser-oauth-client.js  browser-oauth-database.js  browser-runtime-implementation.js  errors.js  index.js  indexed-db-store.js  util.js

Again, it seems like only real candidate file here is index.js. But this is a different situation from the previous example library! Let’s take a look at index.js:

There’s a bunch of stuff like this in index.js:

__exportStar(require("@atproto/oauth-client"), exports);
__exportStar(require("./browser-oauth-client.js"), exports);
__exportStar(require("./errors.js"), exports);
var util_js_1 = require("./util.js");

This require() syntax is CommonJS syntax, which means that we can’t use this file in the browser at all, we need to use some kind of build step, and ESBuild won’t work either.

Also in this library’s package.json it says "type": "commonjs" which is another way to tell it’s CommonJS.

how to use a CommonJS module with esm.sh

Originally I thought it was impossible to use CommonJS modules without learning a build system, but then someone Bluesky told me about esm.sh! It’s a CDN that will translate anything into an ES Module. skypack.dev does something similar, I’m not sure what the difference is but one person mentioned that if one doesn’t work sometimes they’ll try the other one.

For @atproto/oauth-client-browser using it seems pretty simple, I just need to put this in my HTML:

<script type="module" src="script.js"> </script>

and then put this in script.js.

import { BrowserOAuthClient } from "https://esm.sh/@atproto/oauth-client-browser@0.3.0"

It seems to Just Work, which is cool! Of course this is still sort of using a build system – it’s just that esm.sh is running the build instead of me. My main concerns with this approach are:

  • I don’t really trust CDNs to keep working forever – usually I like to copy dependencies into my repository so that they don’t go away for some reason in the future.
  • I’ve heard of some issues with CDNs having security compromises which scares me.
  • I don’t really understand what esm.sh is doing.

esbuild can also convert CommonJS modules into ES modules

I also learned that you can also use esbuild to convert a CommonJS module into an ES module, though there are some limitations – the import { BrowserOAuthClient } from syntax doesn’t work. Here’s a github issue about that.

I think the esbuild approach is probably more appealing to me than the esm.sh approach because it’s a tool that I already have on my computer so I trust it more. I haven’t experimented with this much yet though.

summary of the three types of files

Here’s a summary of the three types of JS files you might encounter, options for how to use them, and how to identify them.

Unhelpfully a .js or .min.js file extension could be any of these 3 options, so if the file is something.js you need to do more detective work to figure out what you’re dealing with.

  1. “classic” JS files
    • How to use it:: <script src="whatever.js"></script>
    • Ways to identify it:
      • The website has a big friendly banner in its setup instructions saying “Use this with a CDN!” or something
      • A .umd.js extension
      • Just try to put it in a <script src=... tag and see if it works
  2. ES Modules
    • Ways to use it:
      • If there are no dependencies, just import {whatever} from "./my-module.js" directly in your code
      • If there are dependencies, create an importmap and import {whatever} from "my-module"
      • Use esbuild or any ES Module bundler
    • Ways to identify it:
      • Look for an import or export statement. (not module.exports = ..., that’s CommonJS)
      • An .mjs extension
      • maybe "type": "module" in package.json (though it’s not clear to me which file exactly this refers to)
  3. CommonJS Modules
    • Ways to use it:
      • Use https://esm.sh to convert it into an ES module, like https://esm.sh/@atproto/oauth-client-browser@0.3.0
      • Use a build somehow (??)
    • Ways to identify it:
      • Look for require() or module.exports = ... in the code
      • A .cjs extension
      • maybe "type": "commonjs" in package.json (though it’s not clear to me which file exactly this refers to)

it’s really nice to have ES modules standardized

The main difference between CommonJS modules and ES modules from my perspective is that ES modules are actually a standard. This makes me feel a lot more confident using them, because browsers commit to backwards compatibility for web standards forever – if I write some code using ES modules today, I can feel sure that it’ll still work the same way in 15 years.

It also makes me feel better about using tooling like esbuild because even if the esbuild project dies, because it’s implementing a standard it feels likely that there will be another similar tool in the future that I can replace it with.

the JS community has built a lot of very cool tools

A lot of the time when I talk about this stuff I get responses like “I hate javascript!!! it’s the worst!!!”. But my experience is that there are a lot of great tools for Javascript (I just learned about https://esm.sh yesterday which seems great! I love esbuild!), and that if I take the time to learn how things works I can take advantage of some of those tools and make my life a lot easier.

So the goal of this post is definitely not to complain about Javascript, it’s to understand the landscape so I can use the tooling in a way that feels good to me.

questions I still have

Here are some questions I still have, I’ll add the answers into the post if I learn the answer.

  • Is there a tool that automatically generates importmaps for an ES Module that I have set up locally? (apparently yes: jspm)
  • How can I convert a CommonJS module into an ES module on my computer, the way https://esm.sh does? (apparently esbuild can sort of do this, though named exports don’t work)
  • When people normally build CommonJS modules into regular JS code, what’s code is doing that? Obviously there are tools like webpack, rollup, esbuild, etc, but do those tools all implement their own JS parsers/static analysis? How many JS parsers are there out there?
  • Is there any way to bundle an ES module into a single file (like atcute-client.js), but so that in the browser I can still import multiple different paths from that file (like both @atcute/client/lexicons and @atcute/client)?

all the tools

Here’s a list of every tool we talked about in this post:

Writing this post has made me think that even though I usually don’t want to have a build that I run every time I update the project, I might be willing to have a build step (using download-esm or something) that I run only once when setting up the project and never run again except maybe if I’m updating my dependency versions.

that’s all!

Thanks to Marco Rogers who taught me a lot of the things in this post. I’ve probably made some mistakes in this post and I’d love to know what they are – let me know on Bluesky or Mastodon!

2024-11-09T09:24:29+00:00 Fullscreen Open in Tab
New microblog with TILs

I added a new section to this site a couple weeks ago called TIL (“today I learned”).

the goal: save interesting tools & facts I posted on social media

One kind of thing I like to post on Mastodon/Bluesky is “hey, here’s a cool thing”, like the great SQLite repl litecli, or the fact that cross compiling in Go Just Works and it’s amazing, or cryptographic right answers, or this great diff tool. Usually I don’t want to write a whole blog post about those things because I really don’t have much more to say than “hey this is useful!”

It started to bother me that I didn’t have anywhere to put those things: for example recently I wanted to use diffdiff and I just could not remember what it was called.

the solution: make a new section of this blog

So I quickly made a new folder called /til/, added some custom styling (I wanted to style the posts to look a little bit like a tweet), made a little Rake task to help me create new posts quickly (rake new_til), and set up a separate RSS Feed for it.

I think this new section of the blog might be more for myself than anything, now when I forget the link to Cryptographic Right Answers I can hopefully look it up on the TIL page. (you might think “julia, why not use bookmarks??” but I have been failing to use bookmarks for my whole life and I don’t see that changing ever, putting things in public is for whatever reason much easier for me)

So far it’s been working, often I can actually just make a quick post in 2 minutes which was the goal.

inspired by Simon Willison’s TIL blog

My page is inspired by Simon Willison’s great TIL blog, though my TIL posts are a lot shorter.

I don’t necessarily want everything to be archived

This came about because I spent a lot of time on Twitter, so I’ve been thinking about what I want to do about all of my tweets.

I keep reading the advice to “POSSE” (“post on your own site, syndicate elsewhere”), and while I find the idea appealing in principle, for me part of the appeal of social media is that it’s a little bit ephemeral. I can post polls or questions or observations or jokes and then they can just kind of fade away as they become less relevant.

I find it a lot easier to identify specific categories of things that I actually want to have on a Real Website That I Own:

and then let everything else be kind of ephemeral.

I really believe in the advice to make email lists though – the first two (blog posts & comics) both have email lists and RSS feeds that people can subscribe to if they want. I might add a quick summary of any TIL posts from that week to the “blog posts from this week” mailing list.

2024-11-04T09:18:03+00:00 Fullscreen Open in Tab
My IETF 121 Agenda

Here's where you can find me at IETF 121 in Dublin!

Monday

Tuesday

  • 9:30 - 11:30 • oauth
  • 13:00 - 14:30 • spice
  • 16:30 - 17:30 • scim

Thursday

Get in Touch

My Current Drafts

2024-10-31T08:00:10+00:00 Fullscreen Open in Tab
ASCII control characters in my terminal

Hello! I’ve been thinking about the terminal a lot and yesterday I got curious about all these “control codes”, like Ctrl-A, Ctrl-C, Ctrl-W, etc. What’s the deal with all of them?

a table of ASCII control characters

Here’s a table of all 33 ASCII control characters, and what they do on my machine (on Mac OS), more or less. There are about a million caveats, but I’ll talk about what it means and all the problems with this diagram that I know about.

You can also view it as an HTML page (I just made it an image so it would show up in RSS).

different kinds of codes are mixed together

The first surprising thing about this diagram to me is that there are 33 control codes, split into (very roughly speaking) these categories:

  1. Codes that are handled by the operating system’s terminal driver, for example when the OS sees a 3 (Ctrl-C), it’ll send a SIGINT signal to the current program
  2. Everything else is passed through to the application as-is and the application can do whatever it wants with them. Some subcategories of those:
    • Codes that correspond to a literal keypress of a key on your keyboard (Enter, Tab, Backspace). For example when you press Enter, your terminal gets sent 13.
    • Codes used by readline: “the application can do whatever it wants” often means “it’ll do more or less what the readline library does, whether the application actually uses readline or not”, so I’ve labelled a bunch of the codes that readline uses
    • Other codes, for example I think Ctrl-X has no standard meaning in the terminal in general but emacs uses it very heavily

There’s no real structure to which codes are in which categories, they’re all just kind of randomly scattered because this evolved organically.

(If you’re curious about readline, I wrote more about readline in entering text in the terminal is complicated, and there are a lot of cheat sheets out there)

there are only 33 control codes

Something else that I find a little surprising is that are only 33 control codes – A to Z, plus 7 more (@, [, \, ], ^, _, ?). This means that if you want to have for example Ctrl-1 as a keyboard shortcut in a terminal application, that’s not really meaningful – on my machine at least Ctrl-1 is exactly the same thing as just pressing 1, Ctrl-3 is the same as Ctrl-[, etc.

Also Ctrl+Shift+C isn’t a control code – what it does depends on your terminal emulator. On Linux Ctrl-Shift-X is often used by the terminal emulator to copy or open a new tab or paste for example, it’s not sent to the TTY at all.

Also I use Ctrl+Left Arrow all the time, but that isn’t a control code, instead it sends an ANSI escape sequence (ctrl-[[1;5D) which is a different thing which we absolutely do not have space for in this post.

This “there are only 33 codes” thing is totally different from how keyboard shortcuts work in a GUI where you can have Ctrl+KEY for any key you want.

the official ASCII names aren’t very meaningful to me

Each of these 33 control codes has a name in ASCII (for example 3 is ETX). When all of these control codes were originally defined, they weren’t being used for computers or terminals at all, they were used for the telegraph machine. Telegraph machines aren’t the same as UNIX terminals so a lot of the codes were repurposed to mean something else.

Personally I don’t find these ASCII names very useful, because 50% of the time the name in ASCII has no actual relationship to what that code does on UNIX systems today. So it feels easier to just ignore the ASCII names completely instead of trying to figure which ones still match their original meaning.

It’s hard to use Ctrl-M as a keyboard shortcut

Another thing that’s a bit weird is that Ctrl-M is literally the same as Enter, and Ctrl-I is the same as Tab, which makes it hard to use those two as keyboard shortcuts.

From some quick research, it seems like some folks do still use Ctrl-I and Ctrl-M as keyboard shortcuts (here’s an example), but to do that you need to configure your terminal emulator to treat them differently than the default.

For me the main takeaway is that if I ever write a terminal application I should avoid Ctrl-I and Ctrl-M as keyboard shortcuts in it.

how to identify what control codes get sent

While writing this I needed to do a bunch of experimenting to figure out what various key combinations did, so I wrote this Python script echo-key.py that will print them out.

There’s probably a more official way but I appreciated having a script I could customize.

caveat: on canonical vs noncanonical mode

Two of these codes (Ctrl-W and Ctrl-U) are labelled in the table as “handled by the OS”, but actually they’re not always handled by the OS, it depends on whether the terminal is in “canonical” mode or in “noncanonical mode”.

In canonical mode, programs only get input when you press Enter (and the OS is in charge of deleting characters when you press Backspace or Ctrl-W). But in noncanonical mode the program gets input immediately when you press a key, and the Ctrl-W and Ctrl-U codes are passed through to the program to handle any way it wants.

Generally in noncanonical mode the program will handle Ctrl-W and Ctrl-U similarly to how the OS does, but there are some small differences.

Some examples of programs that use canonical mode:

  • probably pretty much any noninteractive program, like grep or cat
  • git, I think

Examples of programs that use noncanonical mode:

  • python3, irb and other REPLs
  • your shell
  • any full screen TUI like less or vim

caveat: all of the “OS terminal driver” codes are configurable with stty

I said that Ctrl-C sends SIGINT but technically this is not necessarily true, if you really want to you can remap all of the codes labelled “OS terminal driver”, plus Backspace, using a tool called stty, and you can view the mappings with stty -a.

Here are the mappings on my machine right now:

$ stty -a
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
	eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
	min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
	stop = ^S; susp = ^Z; time = 0; werase = ^W;

I have personally never remapped any of these and I cannot imagine a reason I would (I think it would be a recipe for confusion and disaster for me), but I asked on Mastodon and people said the most common reasons they used stty were:

  • fix a broken terminal with stty sane
  • set stty erase ^H to change how Backspace works
  • set stty ixoff
  • some people even map SIGINT to a different key, like their DELETE key

caveat: on signals

Two signals caveats:

  1. If the ISIG terminal mode is turned off, then the OS won’t send signals. For example vim turns off ISIG
  2. Apparently on BSDs, there’s an extra control code (Ctrl-T) which sends SIGINFO

You can see which terminal modes a program is setting using strace like this, terminal modes are set with the ioctl system call:

$ strace -tt -o out  vim
$ grep ioctl out | grep SET

here are the modes vim sets when it starts (ISIG and ICANON are missing!):

17:43:36.670636 ioctl(0, TCSETS, {c_iflag=IXANY|IMAXBEL|IUTF8,
c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST, c_cflag=B38400|CS8|CREAD,
c_lflag=ECHOK|ECHOCTL|ECHOKE|PENDIN, ...}) = 0

and it resets the modes when it exits:

17:43:38.027284 ioctl(0, TCSETS, {c_iflag=ICRNL|IXANY|IMAXBEL|IUTF8,
c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD,
c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE|PENDIN, ...}) = 0

I think the specific combination of modes vim is using here might be called “raw mode”, man cfmakeraw talks about that.

there are a lot of conflicts

Related to “there are only 33 codes”, there are a lot of conflicts where different parts of the system want to use the same code for different things, for example by default Ctrl-S will freeze your screen, but if you turn that off then readline will use Ctrl-S to do a forward search.

Another example is that on my machine sometimes Ctrl-T will send SIGINFO and sometimes it’ll transpose 2 characters and sometimes it’ll do something completely different depending on:

  • whether the program has ISIG set
  • whether the program uses readline / imitates readline’s behaviour

caveat: on “backspace” and “other backspace”

In this diagram I’ve labelled code 127 as “backspace” and 8 as “other backspace”. Uh, what?

I think this was the single biggest topic of discussion in the replies on Mastodon – apparently there’s a LOT of history to this and I’d never heard of any of it before.

First, here’s how it works on my machine:

  1. I press the Backspace key
  2. The TTY gets sent the byte 127, which is called DEL in ASCII
  3. the OS terminal driver and readline both have 127 mapped to “backspace” (so it works both in canonical mode and noncanonical mode)
  4. The previous character gets deleted

If I press Ctrl+H, it has the same effect as Backspace if I’m using readline, but in a program without readline support (like cat for instance), it just prints out ^H.

Apparently Step 2 above is different for some folks – their Backspace key sends the byte 8 instead of 127, and so if they want Backspace to work then they need to configure the OS (using stty) to set erase = ^H.

There’s an incredible section of the Debian Policy Manual on keyboard configuration that describes how Delete and Backspace should work according to Debian policy, which seems very similar to how it works on my Mac today. My understanding (via this mastodon post) is that this policy was written in the 90s because there was a lot of confusion about what Backspace should do in the 90s and there needed to be a standard to get everything to work.

There’s a bunch more historical terminal stuff here but that’s all I’ll say for now.

there’s probably a lot more diversity in how this works

I’ve probably missed a bunch more ways that “how it works on my machine” might be different from how it works on other people’s machines, and I’ve probably made some mistakes about how it works on my machine too. But that’s all I’ve got for today.

Some more stuff I know that I’ve left out: according to stty -a Ctrl-O is “discard”, Ctrl-R is “reprint”, and Ctrl-Y is “dsusp”. I have no idea how to make those actually do anything (pressing them does not do anything obvious, and some people have told me what they used to do historically but it’s not clear to me if they have a use in 2024), and a lot of the time in practice they seem to just be passed through to the application anyway so I just labelled Ctrl-R and Ctrl-Y as readline.

not all of this is that useful to know

Also I want to say that I think the contents of this post are kind of interesting but I don’t think they’re necessarily that useful. I’ve used the terminal pretty successfully every day for the last 20 years without knowing literally any of this – I just knew what Ctrl-C, Ctrl-D, Ctrl-Z, Ctrl-R, Ctrl-L did in practice (plus maybe Ctrl-A, Ctrl-E and Ctrl-W) and did not worry about the details for the most part, and that was almost always totally fine except when I was trying to use xterm.js.

But I had fun learning about it so maybe it’ll be interesting to you too.

2024-10-27T07:47:04+00:00 Fullscreen Open in Tab
Using less memory to look up IP addresses in Mess With DNS

I’ve been having problems for the last 3 years or so where Mess With DNS periodically runs out of memory and gets OOM killed.

This hasn’t been a big priority for me: usually it just goes down for a few minutes while it restarts, and it only happens once a day at most, so I’ve just been ignoring. But last week it started actually causing a problem so I decided to look into it.

This was kind of winding road where I learned a lot so here’s a table of contents:

there’s about 100MB of memory available

I run Mess With DNS on a VM without about 465MB of RAM, which according to ps aux (the RSS column) is split up something like:

  • 100MB for PowerDNS
  • 200MB for Mess With DNS
  • 40MB for hallpass

That leaves about 110MB of memory free.

A while back I set GOMEMLIMIT to 250MB to try to make sure the garbage collector ran if Mess With DNS used more than 250MB of memory, and I think this helped but it didn’t solve everything.

the problem: OOM killing the backup script

A few weeks ago I started backing up Mess With DNS’s database for the first time using restic.

This has been working okay, but since Mess With DNS operates without much extra memory I think restic sometimes needed more memory than was available on the system, and so the backup script sometimes got OOM killed.

This was a problem because

  1. backups might be corrupted sometimes
  2. more importantly, restic takes out a lock when it runs, and so I’d have to manually do an unlock if I wanted the backups to continue working. Doing manual work like this is the #1 thing I try to avoid with all my web services (who has time for that!) so I really wanted to do something about it.

There’s probably more than one solution to this, but I decided to try to make Mess With DNS use less memory so that there was more available memory on the system, mostly because it seemed like a fun problem to try to solve.

what’s using memory: IP addresses

I’d run a memory profile of Mess With DNS a bunch of times in the past, so I knew exactly what was using most of Mess With DNS’s memory: IP addresses.

When it starts, Mess With DNS loads this database where you can look up the ASN of every IP address into memory, so that when it receives a DNS query it can take the source IP address like 74.125.16.248 and tell you that IP address belongs to GOOGLE.

This database by itself used about 117MB of memory, and a simple du told me that was too much – the original text files were only 37MB!

$ du -sh *.tsv
26M	ip2asn-v4.tsv
11M	ip2asn-v6.tsv

The way it worked originally is that I had an array of these:

type IPRange struct {
	StartIP net.IP
	EndIP   net.IP
	Num     int
	Name    string
	Country string
}

and I searched through it with a binary search to figure out if any of the ranges contained the IP I was looking for. Basically the simplest possible thing and it’s super fast, my machine can do about 9 million lookups per second.

attempt 1: use SQLite

I’ve been using SQLite recently, so my first thought was – maybe I can store all of this data on disk in an SQLite database, give the tables an index, and that’ll use less memory.

So I:

  • wrote a quick Python script using sqlite-utils to import the TSV files into an SQLite database
  • adjusted my code to select from the database instead

This did solve the initial memory goal (after a GC it now hardly used any memory at all because the table was on disk!), though I’m not sure how much GC churn this solution would cause if we needed to do a lot of queries at once. I did a quick memory profile and it seemed to allocate about 1KB of memory per lookup.

Let’s talk about the issues I ran into with using SQLite though.

problem: how to store IPv6 addresses

SQLite doesn’t have support for big integers and IPv6 addresses are 128 bits, so I decided to store them as text. I think BLOB might have been better, I originally thought BLOBs couldn’t be compared but the sqlite docs say they can.

I ended up with this schema:

CREATE TABLE ipv4_ranges (
   start_ip INTEGER NOT NULL,
   end_ip INTEGER NOT NULL,
   asn INTEGER NOT NULL,
   country TEXT NOT NULL,
   name TEXT NOT NULL
);
CREATE TABLE ipv6_ranges (
   start_ip TEXT NOT NULL,
   end_ip TEXT NOT NULL,
   asn INTEGER,
   country TEXT,
   name TEXT
);
CREATE INDEX idx_ipv4_ranges_start_ip ON ipv4_ranges (start_ip);
CREATE INDEX idx_ipv6_ranges_start_ip ON ipv6_ranges (start_ip);
CREATE INDEX idx_ipv4_ranges_end_ip ON ipv4_ranges (end_ip);
CREATE INDEX idx_ipv6_ranges_end_ip ON ipv6_ranges (end_ip);

Also I learned that Python has an ipaddress module, so I could use ipaddress.ip_address(s).exploded to make sure that the IPv6 addresses were expanded so that a string comparison would compare them properly.

problem: it’s 500x slower

I ran a quick microbenchmark, something like this. It printed out that it could look up 17,000 IPv6 addresses per second, and similarly for IPv4 addresses.

This was pretty discouraging – being able to look up 17k addresses per section is kind of fine (Mess With DNS does not get a lot of traffic), but I compared it to the original binary search code and the original code could do 9 million per second.

	ips := []net.IP{}
	count := 20000
	for i := 0; i < count; i++ {
		// create a random IPv6 address
		bytes := randomBytes()
		ip := net.IP(bytes[:])
		ips = append(ips, ip)
	}
	now := time.Now()
	success := 0
	for _, ip := range ips {
		_, err := ranges.FindASN(ip)
		if err == nil {
			success++
		}
	}
	fmt.Println(success)
	elapsed := time.Since(now)
	fmt.Println("number per second", float64(count)/elapsed.Seconds())

time for EXPLAIN QUERY PLAN

I’d never really done an EXPLAIN in sqlite, so I thought it would be a fun opportunity to see what the query plan was doing.

sqlite> explain query plan select * from ipv6_ranges where '2607:f8b0:4006:0824:0000:0000:0000:200e' BETWEEN start_ip and end_ip;
QUERY PLAN
`--SEARCH ipv6_ranges USING INDEX idx_ipv6_ranges_end_ip (end_ip>?)

It looks like it’s just using the end_ip index and not the start_ip index, so maybe it makes sense that it’s slower than the binary search.

I tried to figure out if there was a way to make SQLite use both indexes, but I couldn’t find one and maybe it knows best anyway.

At this point I gave up on the SQLite solution, I didn’t love that it was slower and also it’s a lot more complex than just doing a binary search. I felt like I’d rather keep something much more similar to the binary search.

A few things I tried with SQLite that did not cause it to use both indexes:

  • using a compound index instead of two separate indexes
  • running ANALYZE
  • using INTERSECT to intersect the results of start_ip < ? and ? < end_ip. This did make it use both indexes, but it also seemed to make the query literally 1000x slower, probably because it needed to create the results of both subqueries in memory and intersect them.

attempt 2: use a trie

My next idea was to use a trie, because I had some vague idea that maybe a trie would use less memory, and I found this library called ipaddress-go that lets you look up IP addresses using a trie.

I tried using it here’s the code, but I think I was doing something wildly wrong because, compared to my naive array + binary search:

  • it used WAY more memory (800MB to store just the IPv4 addresses)
  • it was a lot slower to do the lookups (it could do only 100K/second instead of 9 million/second)

I’m not really sure what went wrong here but I gave up on this approach and decided to just try to make my array use less memory and stick to a simple binary search.

some notes on memory profiling

One thing I learned about memory profiling is that you can use runtime package to see how much memory is currently allocated in the program. That’s how I got all the memory numbers in this post. Here’s the code:

func memusage() {
	runtime.GC()
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("Alloc = %v MiB\n", m.Alloc/1024/1024)
	// write mem.prof
	f, err := os.Create("mem.prof")
	if err != nil {
		log.Fatal(err)
	}
	pprof.WriteHeapProfile(f)
	f.Close()
}

Also I learned that if you use pprof to analyze a heap profile there are two ways to analyze it: you can pass either --alloc-space or --inuse-space to go tool pprof. I don’t know how I didn’t realize this before but alloc-space will tell you about everything that was allocated, and inuse-space will just include memory that’s currently in use.

Anyway I ran go tool pprof -pdf --inuse_space mem.prof > mem.pdf a lot. Also every time I use pprof I find myself referring to my own intro to pprof, it’s probably the blog post I wrote that I use the most often. I should add --alloc-space and --inuse-space to it.

attempt 3: make my array use less memory

I was storing my ip2asn entries like this:

type IPRange struct {
	StartIP net.IP
	EndIP   net.IP
	Num     int
	Name    string
	Country string
}

I had 3 ideas for ways to improve this:

  1. There was a lot of repetition of Name and the Country, because a lot of IP ranges belong to the same ASN
  2. net.IP is an []byte under the hood, which felt like it involved an unnecessary pointer, was there a way to inline it into the struct?
  3. Maybe I didn’t need both the start IP and the end IP, often the ranges were consecutive so maybe I could rearrange things so that I only had the start IP

idea 3.1: deduplicate the Name and Country

I figured I could store the ASN info in an array, and then just store the index into the array in my IPRange struct. Here are the structs so you can see what I mean:

type IPRange struct {
	StartIP netip.Addr
	EndIP   netip.Addr
	ASN     uint32
	Idx     uint32
}

type ASNInfo struct {
	Country string
	Name    string
}

type ASNPool struct {
	asns   []ASNInfo
	lookup map[ASNInfo]uint32
}

This worked! It brought memory usage from 117MB to 65MB – a 50MB savings. I felt good about this.

Here’s all of the code for that part.

how big are ASNs?

As an aside – I’m storing the ASN in a uint32, is that right? I looked in the ip2asn file and the biggest one seems to be 401307, though there are a few lines that say 4294901931 which is much bigger, but also are just inside the range of a uint32. So I can definitely use a uint32.

59.101.179.0	59.101.179.255	4294901931	Unknown	AS4294901931

idea 3.2: use netip.Addr instead of net.IP

It turns out that I’m not the only one who felt that net.IP was using an unnecessary amount of memory – in 2021 the folks at Tailscale released a new IP address library for Go which solves this and many other issues. They wrote a great blog post about it.

I discovered (to my delight) that not only does this new IP address library exist and do exactly what I want, it’s also now in the Go standard library as netip.Addr. Switching to netip.Addr was very easy and saved another 20MB of memory, bringing us to 46MB.

I didn’t try my third idea (remove the end IP from the struct) because I’d already been programming for long enough on a Saturday morning and I was happy with my progress.

It’s always such a great feeling when I think “hey, I don’t like this, there must be a better way” and then immediately discover that someone has already made the exact thing I want, thought about it a lot more than me, and implemented it much better than I would have.

all of this was messier in real life

Even though I tried to explain this in a simple linear way “I tried X, then I tried Y, then I tried Z”, that’s kind of a lie – I always try to take my actual debugging process (total chaos) and make it seem more linear and understandable because the reality is just too annoying to write down. It’s more like:

  • try sqlite
  • try a trie
  • second guess everything that I concluded about sqlite, go back and look at the results again
  • wait what about indexes
  • very very belatedly realize that I can use runtime to check how much memory everything is using, start doing that
  • look at the trie again, maybe I misunderstood everything
  • give up and go back to binary search
  • look at all of the numbers for tries/sqlite again to make sure I didn’t misunderstand

A note on using 512MB of memory

Someone asked why I don’t just give the VM more memory. I could very easily afford to pay for a VM with 1GB of memory, but I feel like 512MB really should be enough (and really that 256MB should be enough!) so I’d rather stay inside that constraint. It’s kind of a fun puzzle.

a few ideas from the replies

Folks had a lot of good ideas I hadn’t thought of. Recording them as inspiration if I feel like having another Fun Performance Day at some point.

  • Try Go’s unique package for the ASNPool. Someone tried this and it uses more memory, probably because Go’s pointers are 64 bits
  • Try compiling with GOARCH=386 to use 32-bit pointers to sace space (maybe in combination with using unique!)
  • It should be possible to store all of the IPv6 addresses in just 64 bits, because only the first 64 bits of the address are public
  • Interpolation search might be faster than binary search since IP addresses are numeric
  • Try the MaxMind db format with mmdbwriter or mmdbctl
  • Tailscale’s art routing table package

the result: saved 70MB of memory!

I deployed the new version and now Mess With DNS is using less memory! Hooray!

A few other notes:

  • lookups are a little slower – in my microbenchmark they went from 9 million lookups/second to 6 million, maybe because I added a little indirection. Using less memory and a little more CPU seemed like a good tradeoff though.
  • it’s still using more memory than the raw text files do (46MB vs 37MB), I guess pointers take up space and that’s okay.

I’m honestly not sure if this will solve all my memory problems, probably not! But I had fun, I learned a few things about SQLite, I still don’t know what to think about tries, and it made me love binary search even more than I already did.

2024-10-07T09:19:57+00:00 Fullscreen Open in Tab
Some notes on upgrading Hugo

Warning: this is a post about very boring yakshaving, probably only of interest to people who are trying to upgrade Hugo from a very old version to a new version. But what are blogs for if not documenting one’s very boring yakshaves from time to time?

So yesterday I decided to try to upgrade Hugo. There’s no real reason to do this – I’ve been using Hugo version 0.40 to generate this blog since 2018, it works fine, and I don’t have any problems with it. But I thought – maybe it won’t be as hard as I think, and I kind of like a tedious computer task sometimes!

I thought I’d document what I learned along the way in case it’s useful to anyone else doing this very specific migration. I upgraded from Hugo v0.40 (from 2018) to v0.135 (from 2024).

Here are most of the changes I had to make:

change 1: template "theme/partials/thing.html is now partial thing.html

I had to replace a bunch of instances of {{ template "theme/partials/header.html" . }} with {{ partial "header.html" . }}.

This happened in v0.42:

We have now virtualized the filesystems for project and theme files. This makes everything simpler, faster and more powerful. But it also means that template lookups on the form {{ template “theme/partials/pagination.html” . }} will not work anymore. That syntax has never been documented, so it’s not expected to be in wide use.

change 2: .Data.Pages is now site.RegularPages

This seems to be discussed in the release notes for 0.57.2

I just needed to replace .Data.Pages with site.RegularPages in the template on the homepage as well as in my RSS feed template.

change 3: .Next and .Prev got flipped

I had this comment in the part of my theme where I link to the next/previous blog post:

“next” and “previous” in hugo apparently mean the opposite of what I’d think they’d mean intuitively. I’d expect “next” to mean “in the future” and “previous” to mean “in the past” but it’s the opposite

It looks they changed this in ad705aac064 so that “next” actually is in the future and “prev” actually is in the past. I definitely find the new behaviour more intuitive.

downloading the Hugo changelogs with a script

Figuring out why/when all of these changes happened was a little difficult. I ended up hacking together a bash script to download all of the changelogs from github as text files, which I could then grep to try to figure out what happened. It turns out it’s pretty easy to get all of the changelogs from the GitHub API.

So far everything was not so bad – there was also a change around taxonomies that’s I can’t quite explain, but it was all pretty manageable, but then we got to the really tough one: the markdown renderer.

change 4: the markdown renderer (blackfriday -> goldmark)

The blackfriday markdown renderer (which was previously the default) was removed in v0.100.0. This seems pretty reasonable:

It has been deprecated for a long time, its v1 version is not maintained anymore, and there are many known issues. Goldmark should be a mature replacement by now.

Fixing all my Markdown changes was a huge pain – I ended up having to update 80 different Markdown files (out of 700) so that they would render properly, and I’m not totally sure

why bother switching renderers?

The obvious question here is – why bother even trying to upgrade Hugo at all if I have to switch Markdown renderers? My old site was running totally fine and I think it wasn’t necessarily a good use of time, but the one reason I think it might be useful in the future is that the new renderer (goldmark) uses the CommonMark markdown standard, which I’m hoping will be somewhat more futureproof. So maybe I won’t have to go through this again? We’ll see.

Also it turned out that the new Goldmark renderer does fix some problems I had (but didn’t know that I had) with smart quotes and how lists/blockquotes interact.

finding all the Markdown problems: the process

The hard part of this Markdown change was even figuring out what changed. Almost all of the problems (including #2 and #3 above) just silently broke the site, they didn’t cause any errors or anything. So I had to diff the HTML to hunt them down.

Here’s what I ended up doing:

  1. Generate the site with the old version, put it in public_old
  2. Generate the new version, put it in public
  3. Diff every single HTML file in public/ and public_old with this diff.sh script and put the results in a diffs/ folder
  4. Run variations on find diffs -type f | xargs cat | grep -C 5 '(31m|32m)' | less -r over and over again to look at every single change until I found something that seemed wrong
  5. Update the Markdown to fix the problem
  6. Repeat until everything seemed okay

(the grep 31m|32m thing is searching for red/green text in the diff)

This was very time consuming but it was a little bit fun for some reason so I kept doing it until it seemed like nothing too horrible was left.

the new markdown rules

Here’s a list of every type of Markdown change I had to make. It’s very possible these are all extremely specific to me but it took me a long time to figure them all out so maybe this will be helpful to one other person who finds this in the future.

4.1: mixing HTML and markdown

This doesn’t work anymore (it doesn’t expand the link):

<small>
[a link](https://example.com)
</small>

I need to do this instead:

<small>

[a link](https://example.com)

</small>

This works too:

<small> [a link](https://example.com) </small>

4.2: << is changed into «

I didn’t want this so I needed to configure:

markup:
  goldmark:
    extensions:
      typographer:
        leftAngleQuote: '&lt;&lt;'
        rightAngleQuote: '&gt;&gt;'

4.3: nested lists sometimes need 4 space indents

This doesn’t render as a nested list anymore if I only indent by 2 spaces, I need to put 4 spaces.

1. a
  * b
  * c
2. b

The problem is that the amount of indent needed depends on the size of the list markers. Here’s a reference in CommonMark for this.

4.4: blockquotes inside lists work better

Previously the > quote here didn’t render as a blockquote, and with the new renderer it does.

* something
> quote
* something else

I found a bunch of Markdown that had been kind of broken (which I hadn’t noticed) that works better with the new renderer, and this is an example of that.

Lists inside blockquotes also seem to work better.

4.5: headings inside lists

Previously this didn’t render as a heading, but now it does. So I needed to replace the # with &num;.

* # passengers: 20

4.6: + or 1) at the beginning of the line makes it a list

I had something which looked like this:

`1 / (1
+ exp(-1)) = 0.73`

With Blackfriday it rendered like this:

<p><code>1 / (1
+ exp(-1)) = 0.73</code></p>

and with Goldmark it rendered like this:

<p>`1 / (1</p>
<ul>
<li>exp(-1)) = 0.73`</li>
</ul>

Same thing if there was an accidental 1) at the beginning of a line, like in this Markdown snippet

I set up a small Hadoop cluster (1 master, 2 workers, replication set to 
1) on 

To fix this I just had to rewrap the line so that the + wasn’t the first character.

The Markdown is formatted this way because I wrap my Markdown to 80 characters a lot and the wrapping isn’t very context sensitive.

4.7: no more smart quotes in code blocks

There were a bunch of places where the old renderer (Blackfriday) was doing unwanted things in code blocks like replacing ... with or replacing quotes with smart quotes. I hadn’t realized this was happening and I was very happy to have it fixed.

4.8: better quote management

The way this gets rendered got better:

"Oh, *interesting*!"
  • old: “Oh, interesting!“
  • new: “Oh, interesting!”

Before there were two left smart quotes, now the quotes match.

4.9: images are no longer wrapped in a p tag

Previously if I had an image like this:

<img src="https://jvns.ca/images/rustboot1.png">

it would get wrapped in a <p> tag, now it doesn’t anymore. I dealt with this just by adding a margin-bottom: 0.75em to images in the CSS, hopefully that’ll make them display well enough.

4.10: <br> is now wrapped in a p tag

Previously this wouldn’t get wrapped in a p tag, but now it seems to:

<br><br>

I just gave up on fixing this though and resigned myself to maybe having some extra space in some cases. Maybe I’ll try to fix it later if I feel like another yakshave.

4.11: some more goldmark settings

I also needed to

  • turn off code highlighting (because it wasn’t working properly and I didn’t have it before anyway)
  • use the old “blackfriday” method to generate heading IDs so they didn’t change
  • allow raw HTML in my markdown

Here’s what I needed to add to my config.yaml to do all that:

markup:
  highlight:
    codeFences: false
  goldmark:
    renderer:
      unsafe: true
    parser:
      autoHeadingIDType: blackfriday

Maybe I’ll try to get syntax highlighting working one day, who knows. I might prefer having it off though.

a little script to compare blackfriday and goldmark

I also wrote a little program to compare the Blackfriday and Goldmark output for various markdown snippets, here it is in a gist.

It’s not really configured the exact same way Blackfriday and Goldmark were in my Hugo versions, but it was still helpful to have to help me understand what was going on.

a quick note on maintaining themes

My approach to themes in Hugo has been:

  1. pay someone to make a nice design for the site (for example wizardzines.com was designed by Melody Starling)
  2. use a totally custom theme
  3. commit that theme to the same Github repo as the site

So I just need to edit the theme files to fix any problems. Also I wrote a lot of the theme myself so I’m pretty familiar with how it works.

Relying on someone else to keep a theme updated feels kind of scary to me, I think if I were using a third-party theme I’d just copy the code into my site’s github repo and then maintain it myself.

which static site generators have better backwards compatibility?

I asked on Mastodon if anyone had used a static site generator with good backwards compatibility.

The main answers seemed to be Jekyll and 11ty. Several people said they’d been using Jekyll for 10 years without any issues, and 11ty says it has stability as a core goal.

I think a big factor in how appealing Jekyll/11ty are is how easy it is for you to maintain a working Ruby / Node environment on your computer: part of the reason I stopped using Jekyll was that I got tired of having to maintain a working Ruby installation. But I imagine this wouldn’t be a problem for a Ruby or Node developer.

Several people said that they don’t build their Jekyll site locally at all – they just use GitHub Pages to build it.

that’s it!

Overall I’ve been happy with Hugo – I started using it because it had fast build times and it was a static binary, and both of those things are still extremely useful to me. I might have spent 10 hours on this upgrade, but I’ve probably spent 1000+ hours writing blog posts without thinking about Hugo at all so that seems like an extremely reasonable ratio.

I find it hard to be too mad about the backwards incompatible changes, most of them were quite a long time ago, Hugo does a great job of making their old releases available so you can use the old release if you want, and the most difficult one is removing support for the blackfriday Markdown renderer in favour of using something CommonMark-compliant which seems pretty reasonable to me even if it is a huge pain.

But it did take a long time and I don’t think I’d particularly recommend moving 700 blog posts to a new Markdown renderer unless you’re really in the mood for a lot of computer suffering for some reason.

The new renderer did fix a bunch of problems so I think overall it might be a good thing, even if I’ll have to remember to make 2 changes to how I write Markdown (4.1 and 4.3).

Also I’m still using Hugo 0.54 for https://wizardzines.com so maybe these notes will be useful to Future Me if I ever feel like upgrading Hugo for that site.

Hopefully I didn’t break too many things on the blog by doing this, let me know if you see anything broken!

2024-10-01T10:01:44+00:00 Fullscreen Open in Tab
Terminal colours are tricky

Yesterday I was thinking about how long it took me to get a colorscheme in my terminal that I was mostly happy with (SO MANY YEARS), and it made me wonder what about terminal colours made it so hard.

So I asked people on Mastodon what problems they’ve run into with colours in the terminal, and I got a ton of interesting responses! Let’s talk about some of the problems and a few possible ways to fix them.

problem 1: blue on black

One of the top complaints was “blue on black is hard to read”. Here’s an example of that: if I open Terminal.app, set the background to black, and run ls, the directories are displayed in a blue that isn’t that easy to read:

To understand why we’re seeing this blue, let’s talk about ANSI colours!

the 16 ANSI colours

Your terminal has 16 numbered colours – black, red, green, yellow, blue, magenta, cyan, white, and “bright” version of each of those.

Programs can use them by printing out an “ANSI escape code” – for example if you want to see each of the 16 colours in your terminal, you can run this Python program:

def color(num, text):
    return f"\033[38;5;{num}m{text}\033[0m"

for i in range(16):
    print(color(i, f"number {i:02}"))

what are the ANSI colours?

This made me wonder – if blue is colour number 5, who decides what hex color that should correspond to?

The answer seems to be “there’s no standard, terminal emulators just choose colours and it’s not very consistent”. Here’s a screenshot of a table from Wikipedia, where you can see that there’s a lot of variation:

problem 1.5: bright yellow on white

Bright yellow on white is even worse than blue on black, here’s what I get in a terminal with the default settings:

That’s almost impossible to read (and some other colours like light green cause similar issues), so let’s talk about solutions!

two ways to reconfigure your colours

If you’re annoyed by these colour contrast issues (or maybe you just think the default ANSI colours are ugly), you might think – well, I’ll just choose a different “blue” and pick something I like better!

There are two ways you can do this:

Way 1: Configure your terminal emulator: I think most modern terminal emulators have a way to reconfigure the colours, and some of them even come with some preinstalled themes that you might like better than the defaults.

Way 2: Run a shell script: There are ANSI escape codes that you can print out to tell your terminal emulator to reconfigure its colours. Here’s a shell script that does that, from the base16-shell project. You can see that it has a few different conventions for changing the colours – I guess different terminal emulators have different escape codes for changing their colour palette, and so the script is trying to pick the right style of escape code based on the TERM environment variable.

what are the pros and cons of the 2 ways of configuring your colours?

I prefer to use the “shell script” method, because:

  • if I switch terminal emulators for some reason, I don’t need to a different configuration system, my colours still Just Work
  • I use base16-shell with base16-vim to make my vim colours match my terminal colours, which is convenient

some advantages of configuring colours in your terminal emulator:

  • if you use a popular terminal emulator, there are probably a lot more nice terminal themes out there that you can choose from
  • not all terminal emulators support the “shell script method”, and even if they do, the results can be a little inconsistent

This is what my shell has looked like for probably the last 5 years (using the solarized light base16 theme), and I’m pretty happy with it. Here’s htop:

Okay, so let’s say you’ve found a terminal colorscheme that you like. What else can go wrong?

problem 2: programs using 256 colours

Here’s what some output of fd, a find alternative, looks like in my colorscheme:

The contrast is pretty bad here, and I definitely don’t have that lime green in my normal colorscheme. What’s going on?

We can see what color codes fd is using using the unbuffer program to capture its output including the color codes:

$ unbuffer fd . > out
$ vim out
^[[38;5;48mbad-again.sh^[[0m
^[[38;5;48mbad.sh^[[0m
^[[38;5;48mbetter.sh^[[0m
out

^[[38;5;48 means “set the foreground color to color 48”. Terminals don’t only have 16 colours – many terminals these days actually have 3 ways of specifying colours:

  1. the 16 ANSI colours we already talked about
  2. an extended set of 256 colours
  3. a further extended set of 24-bit hex colours, like #ffea03

So fd is using one of the colours from the extended 256-color set. bat (a cat alternative) does something similar – here’s what it looks like by default in my terminal.

This looks fine though and it really seems like it’s trying to work well with a variety of terminal themes.

some newer tools seem to have theme support

I think it’s interesting that some of these newer terminal tools (fd, cat, delta, and probably more) have support for arbitrary custom themes. I guess the downside of this approach is that the default theme might clash with your terminal’s background, but the upside is that it gives you a lot more control over theming the tool’s output than just choosing 16 ANSI colours.

I don’t really use bat, but if I did I’d probably use bat --theme ansi to just use the ANSI colours that I have set in my normal terminal colorscheme.

problem 3: the grays in Solarized

A bunch of people on Mastodon mentioned a specific issue with grays in the Solarized theme: when I list a directory, the base16 Solarized Light theme looks like this:

but iTerm’s default Solarized Light theme looks like this:

This is because in the iTerm theme (which is the original Solarized design), colors 9-14 (the “bright blue”, “bright red”, etc) are mapped to a series of grays, and when I run ls, it’s trying to use those “bright” colours to color my directories and executables.

My best guess for why the original Solarized theme is designed this way is to make the grays available to the vim Solarized colorscheme.

I’m pretty sure I prefer the modified base16 version I use where the “bright” colours are actually colours instead of all being shades of gray though. (I didn’t actually realize the version I was using wasn’t the “original” Solarized theme until I wrote this post)

In any case I really love Solarized and I’m very happy it exists so that I can use a modified version of it.

problem 4: a vim theme that doesn’t match the terminal background

If I my vim theme has a different background colour than my terminal theme, I get this ugly border, like this:

This one is a pretty minor issue though and I think making your terminal background match your vim background is pretty straightforward.

problem 5: programs setting a background color

A few people mentioned problems with terminal applications setting an unwanted background colour, so let’s look at an example of that.

Here ngrok has set the background to color #16 (“black”), but the base16-shell script I use sets color 16 to be bright orange, so I get this, which is pretty bad:

I think the intention is for ngrok to look something like this:

I think base16-shell sets color #16 to orange (instead of black) so that it can provide extra colours for use by base16-vim. This feels reasonable to me – I use base16-vim in the terminal, so I guess I’m using that feature and it’s probably more important to me than ngrok (which I rarely use) behaving a bit weirdly.

This particular issue is a maybe obscure clash between ngrok and my colorschem, but I think this kind of clash is pretty common when a program sets an ANSI background color that the user has remapped for some reason.

a nice solution to contrast issues: “minimum contrast”

A bunch of terminals (iTerm2, tabby, kitty’s text_fg_override_threshold, and folks tell me also Ghostty and Windows Terminal) have a “minimum contrast” feature that will automatically adjust colours to make sure they have enough contrast.

Here’s an example from iTerm. This ngrok accident from before has pretty bad contrast, I find it pretty difficult to read:

With “minimum contrast” set to 40 in iTerm, it looks like this instead:

I didn’t have minimum contrast turned on before but I just turned it on today because it makes such a big difference when something goes wrong with colours in the terminal.

problem 6: TERM being set to the wrong thing

A few people mentioned that they’ll SSH into a system that doesn’t support the TERM environment variable that they have set locally, and then the colours won’t work.

I think the way TERM works is that systems have a terminfo database, so if the value of the TERM environment variable isn’t in the system’s terminfo database, then it won’t know how to output colours for that terminal. I don’t know too much about terminfo, but someone linked me to this terminfo rant that talks about a few other issues with terminfo.

I don’t have a system on hand to reproduce this one so I can’t say for sure how to fix it, but this stackoverflow question suggests running something like TERM=xterm ssh instead of ssh.

problem 7: picking “good” colours is hard

A couple of problems people mentioned with designing / finding terminal colorschemes:

  • some folks are colorblind and have trouble finding an appropriate colorscheme
  • accidentally making the background color too close to the cursor or selection color, so they’re hard to find
  • generally finding colours that work with every program is a struggle (for example you can see me having a problem with this with ngrok above!)

problem 8: making nethack/mc look right

Another problem people mentioned is using a program like nethack or midnight commander which you might expect to have a specific colourscheme based on the default ANSI terminal colours.

For example, midnight commander has a really specific classic look:

But in my Solarized theme, midnight commander looks like this:

The Solarized version feels like it could be disorienting if you’re very used to the “classic” look.

One solution Simon Tatham mentioned to this is using some palette customization ANSI codes (like the ones base16 uses that I talked about earlier) to change the color palette right before starting the program, for example remapping yellow to a brighter yellow before starting Nethack so that the yellow characters look better.

problem 9: commands disabling colours when writing to a pipe

If I run fd | less, I see something like this, with the colours disabled.

In general I find this useful – if I pipe a command to grep, I don’t want it to print out all those color escape codes, I just want the plain text. But what if you want to see the colours?

To see the colours, you can run unbuffer fd | less -r! I just learned about unbuffer recently and I think it’s really cool, unbuffer opens a tty for the command to write to so that it thinks it’s writing to a TTY. It also fixes issues with programs buffering their output when writing to a pipe, which is why it’s called unbuffer.

Here’s what the output of unbuffer fd | less -r looks like for me:

Also some commands (including fd) support a --color=always flag which will force them to always print out the colours.

problem 10: unwanted colour in ls and other commands

Some people mentioned that they don’t want ls to use colour at all, perhaps because ls uses blue, it’s hard to read on black, and maybe they don’t feel like customizing their terminal’s colourscheme to make the blue more readable or just don’t find the use of colour helpful.

Some possible solutions to this one:

  • you can run ls --color=never, which is probably easiest
  • you can also set LS_COLORS to customize the colours used by ls. I think some other programs other than ls support the LS_COLORS environment variable too.
  • also some programs support setting NO_COLOR=true (there’s a list here)

Here’s an example of running LS_COLORS="fi=0:di=0:ln=0:pi=0:so=0:bd=0:cd=0:or=0:ex=0" ls:

problem 11: the colours in vim

I used to have a lot of problems with configuring my colours in vim – I’d set up my terminal colours in a way that I thought was okay, and then I’d start vim and it would just be a disaster.

I think what was going on here is that today, there are two ways to set up a vim colorscheme in the terminal:

  1. using your ANSI terminal colours – you tell vim which ANSI colour number to use for the background, for functions, etc.
  2. using 24-bit hex colours – instead of ANSI terminal colours, the vim colorscheme can use hex codes like #faea99 directly

20 years ago when I started using vim, terminals with 24-bit hex color support were a lot less common (or maybe they didn’t exist at all), and vim certainly didn’t have support for using 24-bit colour in the terminal. From some quick searching through git, it looks like vim added support for 24-bit colour in 2016 – just 8 years ago!

So to get colours to work properly in vim before 2016, you needed to synchronize your terminal colorscheme and your vim colorscheme. Here’s what that looked like, the colorscheme needed to map the vim color classes like cterm05 to ANSI colour numbers.

But in 2024, the story is really different! Vim (and Neovim, which I use now) support 24-bit colours, and as of Neovim 0.10 (released in May 2024), the termguicolors setting (which tells Vim to use 24-bit hex colours for colorschemes) is turned on by default in any terminal with 24-bit color support.

So this “you need to synchronize your terminal colorscheme and your vim colorscheme” problem is not an issue anymore for me in 2024, since I don’t plan to use terminals without 24-bit color support in the future.

The biggest consequence for me of this whole thing is that I don’t need base16 to set colors 16-21 to weird stuff anymore to integrate with vim – I can just use a terminal theme and a vim theme, and as long as the two themes use similar colours (so it’s not jarring for me to switch between them) there’s no problem. I think I can just remove those parts from my base16 shell script and totally avoid the problem with ngrok and the weird orange background I talked about above.

some more problems I left out

I think there are a lot of issues around the intersection of multiple programs, like using some combination tmux/ssh/vim that I couldn’t figure out how to reproduce well enough to talk about them. Also I’m sure I missed a lot of other things too.

base16 has really worked for me

I’ve personally had a lot of success with using base16-shell with base16-vim – I just need to add a couple of lines to my fish config to set it up (+ a few .vimrc lines) and then I can move on and accept any remaining problems that that doesn’t solve.

I don’t think base16 is for everyone though, some limitations I’m aware of with base16 that might make it not work for you:

  • it comes with a limited set of builtin themes and you might not like any of them
  • the Solarized base16 theme (and maybe all of the themes?) sets the “bright” ANSI colours to be exactly the same as the normal colours, which might cause a problem if you’re relying on the “bright” colours to be different from the regular ones
  • it sets colours 16-21 in order to give the vim colorschemes from base16-vim access to more colours, which might not be relevant if you always use a terminal with 24-bit color support, and can cause problems like the ngrok issue above
  • also the way it sets colours 16-21 could be a problem in terminals that don’t have 256-color support, like the linux framebuffer terminal

Apparently there’s a community fork of base16 called tinted-theming, which I haven’t looked into much yet.

some other colorscheme tools

Just one so far but I’ll link more if people tell me about them:

okay, that was a lot

We talked about a lot in this post and while I think learning about all these details is kind of fun if I’m in the mood to do a deep dive, I find it SO FRUSTRATING to deal with it when I just want my colours to work! Being surprised by unreadable text and having to find a workaround is just not my idea of a good day.

Personally I’m a zero-configuration kind of person and it’s not that appealing to me to have to put together a lot of custom configuration just to make my colours in the terminal look acceptable. I’d much rather just have some reasonable defaults that I don’t have to change.

minimum contrast seems like an amazing feature

My one big takeaway from writing this was to turn on “minimum contrast” in my terminal, I think it’s going to fix most of the occasional accidental unreadable text issues I run into and I’m pretty excited about it.

2024-09-27T11:16:00+00:00 Fullscreen Open in Tab
Some Go web dev notes

I spent a lot of time in the past couple of weeks working on a website in Go that may or may not ever see the light of day, but I learned a couple of things along the way I wanted to write down. Here they are:

go 1.22 now has better routing

I’ve never felt motivated to learn any of the Go routing libraries (gorilla/mux, chi, etc), so I’ve been doing all my routing by hand, like this.

	// DELETE /records:
	case r.Method == "DELETE" && n == 1 && p[0] == "records":
		if !requireLogin(username, r.URL.Path, r, w) {
			return
		}
		deleteAllRecords(ctx, username, rs, w, r)
	// POST /records/<ID>
	case r.Method == "POST" && n == 2 && p[0] == "records" && len(p[1]) > 0:
		if !requireLogin(username, r.URL.Path, r, w) {
			return
		}
		updateRecord(ctx, username, p[1], rs, w, r)

But apparently as of Go 1.22, Go now has better support for routing in the standard library, so that code can be rewritten something like this:

	mux.HandleFunc("DELETE /records/", app.deleteAllRecords)
	mux.HandleFunc("POST /records/{record_id}", app.updateRecord)

Though it would also need a login middleware, so maybe something more like this, with a requireLogin middleware.

	mux.Handle("DELETE /records/", requireLogin(http.HandlerFunc(app.deleteAllRecords)))

a gotcha with the built-in router: redirects with trailing slashes

One annoying gotcha I ran into was: if I make a route for /records/, then a request for /records will be redirected to /records/.

I ran into an issue with this where sending a POST request to /records redirected to a GET request for /records/, which broke the POST request because it removed the request body. Thankfully Xe Iaso wrote a blog post about the exact same issue which made it easier to debug.

I think the solution to this is just to use API endpoints like POST /records instead of POST /records/, which seems like a more normal design anyway.

sqlc automatically generates code for my db queries

I got a little bit tired of writing so much boilerplate for my SQL queries, but I didn’t really feel like learning an ORM, because I know what SQL queries I want to write, and I didn’t feel like learning the ORM’s conventions for translating things into SQL queries.

But then I found sqlc, which will compile a query like this:


-- name: GetVariant :one
SELECT *
FROM variants
WHERE id = ?;

into Go code like this:

const getVariant = `-- name: GetVariant :one
SELECT id, created_at, updated_at, disabled, product_name, variant_name
FROM variants
WHERE id = ?
`

func (q *Queries) GetVariant(ctx context.Context, id int64) (Variant, error) {
	row := q.db.QueryRowContext(ctx, getVariant, id)
	var i Variant
	err := row.Scan(
		&i.ID,
		&i.CreatedAt,
		&i.UpdatedAt,
		&i.Disabled,
		&i.ProductName,
		&i.VariantName,
	)
	return i, err
}

What I like about this is that if I’m ever unsure about what Go code to write for a given SQL query, I can just write the query I want, read the generated function and it’ll tell me exactly what to do to call it. It feels much easier to me than trying to dig through the ORM’s documentation to figure out how to construct the SQL query I want.

Reading Brandur’s sqlc notes from 2024 also gave me some confidence that this is a workable path for my tiny programs. That post gives a really helpful example of how to conditionally update fields in a table using CASE statements (for example if you have a table with 20 columns and you only want to update 3 of them).

sqlite tips

Someone on Mastodon linked me to this post called Optimizing sqlite for servers. My projects are small and I’m not so concerned about performance, but my main takeaways were:

  • have a dedicated object for writing to the database, and run db.SetMaxOpenConns(1) on it. I learned the hard way that if I don’t do this then I’ll get SQLITE_BUSY errors from two threads trying to write to the db at the same time.
  • if I want to make reads faster, I could have 2 separate db objects, one for writing and one for reading

There are a more tips in that post that seem useful (like “COUNT queries are slow” and “Use STRICT tables”), but I haven’t done those yet.

Also sometimes if I have two tables where I know I’ll never need to do a JOIN beteween them, I’ll just put them in separate databases so that I can connect to them independently.

Go 1.19 introduced a way to set a GC memory limit

I run all of my Go projects in VMs with relatively little memory, like 256MB or 512MB. I ran into an issue where my application kept getting OOM killed and it was confusing – did I have a memory leak? What?

After some Googling, I realized that maybe I didn’t have a memory leak, maybe I just needed to reconfigure the garbage collector! It turns out that by default (according to A Guide to the Go Garbage Collector), Go’s garbage collector will let the application allocate memory up to 2x the current heap size.

Mess With DNS’s base heap size is around 170MB and the amount of memory free on the VM is around 160MB right now, so if its memory doubled, it’ll get OOM killed.

In Go 1.19, they added a way to tell Go “hey, if the application starts using this much memory, run a GC”. So I set the GC memory limit to 250MB and it seems to have resulted in the application getting OOM killed less often:

export GOMEMLIMIT=250MiB

some reasons I like making websites in Go

I’ve been making tiny websites (like the nginx playground) in Go on and off for the last 4 years or so and it’s really been working for me. I think I like it because:

  • there’s just 1 static binary, all I need to do to deploy it is copy the binary. If there are static files I can just embed them in the binary with embed.
  • there’s a built-in webserver that’s okay to use in production, so I don’t need to configure WSGI or whatever to get it to work. I can just put it behind Caddy or run it on fly.io or whatever.
  • Go’s toolchain is very easy to install, I can just do apt-get install golang-go or whatever and then a go build will build my project
  • it feels like there’s very little to remember to start sending HTTP responses – basically all there is are functions like Serve(w http.ResponseWriter, r *http.Request) which read the request and send a response. If I need to remember some detail of how exactly that’s accomplished, I just have to read the function!
  • also net/http is in the standard library, so you can start making websites without installing any libraries at all. I really appreciate this one.
  • Go is a pretty systems-y language, so if I need to run an ioctl or something that’s easy to do

In general everything about it feels like it makes projects easy to work on for 5 days, abandon for 2 years, and then get back into writing code without a lot of problems.

For contrast, I’ve tried to learn Rails a couple of times and I really want to love Rails – I’ve made a couple of toy websites in Rails and it’s always felt like a really magical experience. But ultimately when I come back to those projects I can’t remember how anything works and I just end up giving up. It feels easier to me to come back to my Go projects that are full of a lot of repetitive boilerplate, because at least I can read the code and figure out how it works.

things I haven’t figured out yet

some things I haven’t done much of yet in Go:

  • rendering HTML templates: usually my Go servers are just APIs and I make the frontend a single-page app with Vue. I’ve used html/template a lot in Hugo (which I’ve used for this blog for the last 8 years) but I’m still not sure how I feel about it.
  • I’ve never made a real login system, usually my servers don’t have users at all.
  • I’ve never tried to implement CSRF

In general I’m not sure how to implement security-sensitive features so I don’t start projects which need login/CSRF/etc. I imagine this is where a framework would help.

it’s cool to see the new features Go has been adding

Both of the Go features I mentioned in this post (GOMEMLIMIT and the routing) are new in the last couple of years and I didn’t notice when they came out. It makes me think I should pay closer attention to the release notes for new Go versions.

2024-09-12T15:09:12+00:00 Fullscreen Open in Tab
Reasons I still love the fish shell

I wrote about how much I love fish in this blog post from 2017 and, 7 years of using it every day later, I’ve found even more reasons to love it. So I thought I’d write a new post with both the old reasons I loved it and some reasons.

This came up today because I was trying to figure out why my terminal doesn’t break anymore when I cat a binary to my terminal, the answer was “fish fixes the terminal!”, and I just thought that was really nice.

1. no configuration

In 10 years of using fish I have never found a single thing I wanted to configure. It just works the way I want. My fish config file just has:

  • environment variables
  • aliases (alias ls eza, alias vim nvim, etc)
  • the occasional direnv hook fish | source to integrate a tool like direnv
  • a script I run to set up my terminal colours

I’ve been told that configuring things in fish is really easy if you ever do want to configure something though.

2. autosuggestions from my shell history

My absolute favourite thing about fish is that I type, it’ll automatically suggest (in light grey) a matching command that I ran recently. I can press the right arrow key to accept the completion, or keep typing to ignore it.

Here’s what that looks like. In this example I just typed the “v” key and it guessed that I want to run the previous vim command again.

2.5 “smart” shell autosuggestions

One of my favourite subtle autocomplete features is how fish handles autocompleting commands that contain paths in them. For example, if I run:

$ ls blah.txt

that command will only be autocompleted in directories that contain blah.txt – it won’t show up in a different directory. (here’s a short comment about how it works)

As an example, if in this directory I type bash scripts/, it’ll only suggest history commands including files that actually exist in my blog’s scripts folder, and not the dozens of other irrelevant scripts/ commands I’ve run in other folders.

I didn’t understand exactly how this worked until last week, it just felt like fish was magically able to suggest the right commands. It still feels a little like magic and I love it.

3. pasting multiline commands

If I copy and paste multiple lines, bash will run them all, like this:

[bork@grapefruit linux-playground (main)]$ echo hi
hi
[bork@grapefruit linux-playground (main)]$ touch blah
[bork@grapefruit linux-playground (main)]$ echo hi
hi

This is a bit alarming – what if I didn’t actually want to run all those commands?

Fish will paste them all at a single prompt, so that I can press Enter if I actually want to run them. Much less scary.

bork@grapefruit ~/work/> echo hi

                         touch blah
                         echo hi

4. nice tab completion

If I run ls and press tab, it’ll display all the filenames in a nice grid. I can use either Tab, Shift+Tab, or the arrow keys to navigate the grid.

Also, I can tab complete from the middle of a filename – if the filename starts with a weird character (or if it’s just not very unique), I can type some characters from the middle and press tab.

Here’s what the tab completion looks like:

bork@grapefruit ~/work/> ls 
api/  blah.py     fly.toml   README.md
blah  Dockerfile  frontend/  test_websocket.sh

I honestly don’t complete things other than filenames very much so I can’t speak to that, but I’ve found the experience of tab completing filenames to be very good.

5. nice default prompt (including git integration)

Fish’s default prompt includes everything I want:

  • username
  • hostname
  • current folder
  • git integration
  • status of last command exit (if the last command failed)

Here’s a screenshot with a few different variations on the default prompt, including if the last command was interrupted (the SIGINT) or failed.

6. nice history defaults

In bash, the maximum history size is 500 by default, presumably because computers used to be slow and not have a lot of disk space. Also, by default, commands don’t get added to your history until you end your session. So if your computer crashes, you lose some history.

In fish:

  1. the default history size is 256,000 commands. I don’t see any reason I’d ever need more.
  2. if you open a new tab, everything you’ve ever run (including commands in open sessions) is immediately available to you
  3. in an existing session, the history search will only include commands from the current session, plus everything that was in history at the time that you started the shell

I’m not sure how clearly I’m explaining how fish’s history system works here, but it feels really good to me in practice. My impression is that the way it’s implemented is the commands are continually added to the history file, but fish only loads the history file once, on startup.

I’ll mention here that if you want to have a fancier history system in another shell it might be worth checking out atuin or fzf.

7. press up arrow to search history

I also like fish’s interface for searching history: for example if I want to edit my fish config file, I can just type:

$ config.fish

and then press the up arrow to go back the last command that included config.fish. That’ll complete to:

$ vim ~/.config/fish/config.fish

and I’m done. This isn’t so different from using Ctrl+R in bash to search your history but I think I like it a little better over all, maybe because Ctrl+R has some behaviours that I find confusing (for example you can end up accidentally editing your history which I don’t like).

8. the terminal doesn’t break

I used to run into issues with bash where I’d accidentally cat a binary to the terminal, and it would break the terminal.

Every time fish displays a prompt, it’ll try to fix up your terminal so that you don’t end up in weird situations like this. I think this is some of the code in fish to prevent broken terminals.

Some things that it does are:

  • turn on echo so that you can see the characters you type
  • make sure that newlines work properly so that you don’t get that weird staircase effect
  • reset your terminal background colour, etc

I don’t think I’ve run into any of these “my terminal is broken” issues in a very long time, and I actually didn’t even realize that this was because of fish – I thought that things somehow magically just got better, or maybe I wasn’t making as many mistakes. But I think it was mostly fish saving me from myself, and I really appreciate that.

9. Ctrl+S is disabled

Also related to terminals breaking: fish disables Ctrl+S (which freezes your terminal and then you need to remember to press Ctrl+Q to unfreeze it). It’s a feature that I’ve never wanted and I’m happy to not have it.

Apparently you can disable Ctrl+S in other shells with stty -ixon.

10. nice syntax highlighting

By default commands that don’t exist are highlighted in red, like this.

11. easier loops

I find the loop syntax in fish a lot easier to type than the bash syntax. It looks like this:

for i in *.yaml
  echo $i
end

Also it’ll add indentation in your loops which is nice.

12. easier multiline editing

Related to loops: you can edit multiline commands much more easily than in bash (just use the arrow keys to navigate the multiline command!). Also when you use the up arrow to get a multiline command from your history, it’ll show you the whole command the exact same way you typed it instead of squishing it all onto one line like bash does:

$ bash
$ for i in *.png
> do
> echo $i
> done
$ # press up arrow
$ for i in *.png; do echo $i; done ink

13. Ctrl+left arrow

This might just be me, but I really appreciate that fish has the Ctrl+left arrow / Ctrl+right arrow keyboard shortcut for moving between words when writing a command.

I’m honestly a bit confused about where this keyboard shortcut is coming from (the only documented keyboard shortcut for this I can find in fish is Alt+left arrow / Alt + right arrow which seems to do the same thing), but I’m pretty sure this is a fish shortcut.

A couple of notes about getting this shortcut to work / where it comes from:

  • one person said they needed to switch their terminal emulator from the “Linux console” keybindings to “Default (XFree 4)” to get it to work in fish
  • on Mac OS, Ctrl+left arrow switches workspaces by default, so I had to turn that off.
  • Also apparently Ubuntu configures libreadline in /etc/inputrc to make Ctrl+left/right arrow go back/forward a word, so it’ll work in bash on Ubuntu and maybe other Linux distros too. Here’s a stack overflow question talking about that

a downside: not everything has a fish integration

Sometimes tools don’t have instructions for integrating them with fish. That’s annoying, but:

  • I’ve found this has gotten better over the last 10 years as fish has gotten more popular. For example Python’s virtualenv has had a fish integration for a long time now.
  • If I need to run a POSIX shell command real quick, I can always just run bash or zsh
  • I’ve gotten much better over the years at translating simple commands to fish syntax when I need to

My biggest day-to-day to annoyance is probably that for whatever reason I’m still not used to fish’s syntax for setting environment variables, I get confused about set vs set -x.

another downside: fish_add_path

fish has a function called fish_add_path that you can run to add a directory to your PATH like this:

fish_add_path /some/directory

I love the idea of it and I used to use it all the time, but I’ve stopped using it for two reasons:

  1. Sometimes fish_add_path will update the PATH for every session in the future (with a “universal variable”) and sometimes it will update the PATH just for the current session. It’s hard for me to tell which one it will do: in theory the docs explain this but I could not understand them.
  2. If you ever need to remove the directory from your PATH a few weeks or months later because maybe you made a mistake, that’s also kind of hard to do (there are instructions in this comments of this github issue though).

Instead I just update my PATH like this, similarly to how I’d do it in bash:

set PATH $PATH /some/directory/bin

on POSIX compatibility

When I started using fish, you couldn’t do things like cmd1 && cmd2 – it would complain “no, you need to run cmd1; and cmd2” instead.

It seems like over the years fish has started accepting a little more POSIX-style syntax than it used to, like:

  • cmd1 && cmd2
  • export a=b to set an environment variable (though this seems a bit limited, you can’t do export PATH=$PATH:/whatever so I think it’s probably better to learn set instead)

on fish as a default shell

Changing my default shell to fish is always a little annoying, I occasionally get myself into a situation where

  1. I install fish somewhere like maybe /home/bork/.nix-stuff/bin/fish
  2. I add the new fish location to /etc/shells as an allowed shell
  3. I change my shell with chsh
  4. at some point months/years later I reinstall fish in a different location for some reason and remove the old one
  5. oh no!!! I have no valid shell! I can’t open a new terminal tab anymore!

This has never been a major issue because I always have a terminal open somewhere where I can fix the problem and rescue myself, but it’s a bit alarming.

If you don’t want to use chsh to change your shell to fish (which is very reasonable, maybe I shouldn’t be doing that), the Arch wiki page has a couple of good suggestions – either configure your terminal emulator to run fish or add an exec fish to your .bashrc.

I’ve never really learned the scripting language

Other than occasionally writing a for loop interactively on the command line, I’ve never really learned the fish scripting language. I still do all of my shell scripting in bash.

I don’t think I’ve ever written a fish function or if statement.

I ran a highly unscientific poll on Mastodon asking people what shell they use interactively. The results were (of 2600 responses):

  • 46% bash
  • 49% zsh
  • 16% fish
  • 5% other

I think 16% for fish is pretty remarkable, since (as far as I know) there isn’t any system where fish is the default shell, and my sense is that it’s very common to just stick to whatever your system’s default shell is.

It feels like a big achievement for the fish project, even if maybe my Mastodon followers are more likely than the average shell user to use fish for some reason.

who might fish be right for?

Fish definitely isn’t for everyone. I think I like it because:

  1. I really dislike configuring my shell (and honestly my dev environment in general), I want things to “just work” with the default settings
  2. fish’s defaults feel good to me
  3. I don’t spend that much time logged into random servers using other shells so there’s not too much context switching
  4. I liked its features so much that I was willing to relearn how to do a few “basic” shell things, like using parentheses (seq 1 10) to run a command instead of backticks or using set instead of export

Maybe you’re also a person who would like fish! I hope a few more of the people who fish is for can find it, because I spend so much of my time in the terminal and it’s made that time much more pleasant.

2024-08-31T18:36:50-07:00 Fullscreen Open in Tab
Thoughts on the Resiliency of Web Projects

I just did a massive spring cleaning of one of my servers, trying to clean up what has become quite the mess of clutter. For every website on the server, I either:

  • Documented what it is, who is using it, and what version of language and framework it uses
  • Archived it as static HTML flat files
  • Moved the source code from GitHub to a private git server
  • Deleted the files

It feels good to get rid of old code, and to turn previously dynamic sites (with all of the risk they come with) into plain HTML.

This is also making me seriously reconsider the value of spinning up any new projects. Several of these are now 10 years old, still churning along fine, but difficult to do any maintenance on because of versions and dependencies. For example:

  • indieauth.com - this has been on the chopping block for years, but I haven't managed to build a replacement yet, and is still used by a lot of people
  • webmention.io - this is a pretty popular service, and I don't want to shut it down, but there's a lot of problems with how it's currently built and no easy way to make changes
  • switchboard.p3k.io - this is a public WebSub (PubSubHubbub) hub, like Superfeedr, and has weirdly gained a lot of popularity in the podcast feed space in the last few years

One that I'm particularly happy with, despite it being an ugly pile of PHP, is oauth.net. I inherited this site in 2012, and it hasn't needed any framework upgrades since it's just using PHP templates. My ham radio website w7apk.com is similarly a small amount of templated PHP, and it is low stress to maintain, and actually fun to quickly jot some notes down when I want. I like not having to go through the whole ceremony of setting up a dev environment, installing dependencies, upgrading things to the latest version, checking for backwards incompatible changes, git commit, deploy, etc. I can just sftp some changes up to the server and they're live.

Some questions for myself for the future, before starting a new project:

  • Could this actually just be a tag page on my website, like #100DaysOfMusic or #BikeTheEclipse?
  • If it really needs to be a new project, then:
  • Can I create it in PHP without using any frameworks or libraries? Plain PHP ages far better than pulling in any dependencies which inevitably stop working with a version 2-3 EOL cycles back, so every library brought in means signing up for annual maintenance of the whole project. Frameworks can save time in the short term, but have a huge cost in the long term.
  • Is it possible to avoid using a database? Databases aren't inherently bad, but using one does make the project slightly more fragile, since it requires plans for migrations and backups, and 
  • If a database is required, is it possible to create it in a way that does not result in ever-growing storage needs?
  • Is this going to store data or be a service that other people are going to use? If so, plan on a registration form so that I have a way to contact people eventually when I need to change it or shut it down.
  • If I've got this far with the questions, am I really ready to commit to supporting this code base for the next 10 years?

One project I've been committed to maintaining and doing regular (ok fine, "semi-regular") updates for is Meetable, the open source events website that I run on a few domains:

I started this project in October 2019, excited for all the IndieWebCamps we were going to run in 2020. Somehow that is already 5 years ago now. Well that didn't exactly pan out, but I did quickly pivot it to add a bunch of features that are helpful for virtual events, so it worked out ok in the end. We've continued to use it for posting IndieWeb events, and I also run an instance for two IETF working groups. I'd love to see more instances pop up, I've only encountered one or two other ones in the wild. I even spent a significant amount of time on the onboarding flow so that it's relatively easy to install and configure. I even added passkeys for the admin login so you don't need any external dependencies on auth providers. It's a cool project if I may say so myself.

Anyway, this is not a particularly well thought out blog post, I just wanted to get my thoughts down after spending all day combing through the filesystem of my web server and uncovering a lot of ancient history.

2024-08-29T12:59:53-07:00 Fullscreen Open in Tab
OAuth Oh Yeah!

The first law of OAuth states that
the total number of authorized access tokens
in an isolated system
must remain constant over time. Over time.

In the world of OAuth, where the sun always shines,
Tokens like treasures, in digital lines.
Security's a breeze, with every law so fine,
OAuth, oh yeah, tonight we dance online!

The second law of OAuth states that
the overall security of the system
must always remain constant over time.
Over time. Over time. Over time.

In the world of OAuth, where the sun always shines,
Tokens like treasures, in digital lines.
Security's a breeze, with every law so fine,
OAuth, oh yeah, tonight we dance online!

The third law of OAuth states that
as the security of the system approaches absolute,
the ability to grant authorized access approaches zero. Zero!

In the world of OAuth, where the sun always shines,
Tokens like treasures, in digital lines.
Security's a breeze, with every law so fine,
OAuth, oh yeah, tonight we dance online!

Tonight we dance online!
OAuth, oh yeah!
Lyrics and music by AI, prompted and edited by Aaron Parecki
2024-08-19T08:15:28+00:00 Fullscreen Open in Tab
Migrating Mess With DNS to use PowerDNS

About 3 years ago, I announced Mess With DNS in this blog post, a playground where you can learn how DNS works by messing around and creating records.

I wasn’t very careful with the DNS implementation though (to quote the release blog post: “following the DNS RFCs? not exactly”), and people started reporting problems that eventually I decided that I wanted to fix.

the problems

Some of the problems people have reported were:

  • domain names with underscores weren’t allowed, even though they should be
  • If there was a CNAME record for a domain name, it allowed you to create other records for that domain name, even if it shouldn’t
  • you could create 2 different CNAME records for the same domain name, which shouldn’t be allowed
  • no support for the SVCB or HTTPS record types, which seemed a little complex to implement
  • no support for upgrading from UDP to TCP for big responses

And there are certainly more issues that nobody got around to reporting, for example that if you added an NS record for a subdomain to delegate it, Mess With DNS wouldn’t handle the delegation properly.

the solution: PowerDNS

I wasn’t sure how to fix these problems for a long time – technically I could have started addressing them individually, but it felt like there were a million edge cases and I’d never get there.

But then one day I was chatting with someone else who was working on a DNS server and they said they were using PowerDNS: an open source DNS server with an HTTP API!

This seemed like an obvious solution to my problems – I could just swap out my own crappy DNS implementation for PowerDNS.

There were a couple of challenges I ran into when setting up PowerDNS that I’ll talk about here. I really don’t do a lot of web development and I think I’ve never built a website that depends on a relatively complex API before, so it was a bit of a learning experience.

challenge 1: getting every query made to the DNS server

One of the main things Mess With DNS does is give you a live view of every DNS query it receives for your subdomain, using a websocket. To make this work, it needs to intercept every DNS query before they it gets sent to the PowerDNS DNS server:

There were 2 options I could think of for how to intercept the DNS queries:

  1. dnstap: dnsdist (a DNS load balancer from the PowerDNS project) has support for logging all DNS queries it receives using dnstap, so I could put dnsdist in front of PowerDNS and then log queries that way
  2. Have my Go server listen on port 53 and proxy the queries myself

I originally implemented option #1, but for some reason there was a 1 second delay before every query got logged. I couldn’t figure out why, so I implemented my own very simple proxy instead.

challenge 2: should the frontend have direct access to the PowerDNS API?

The frontend used to have a lot of DNS logic in it – it converted emoji domain names to ASCII using punycode, had a lookup table to convert numeric DNS query types (like 1) to their human-readable names (like A), did a little bit of validation, and more.

Originally I considered keeping this pattern and just giving the frontend (more or less) direct access to the PowerDNS API to create and delete, but writing even more complex code in Javascript didn’t feel that appealing to me – I don’t really know how to write tests in Javascript and it seemed like it wouldn’t end well.

So I decided to take all of the DNS logic out of the frontend and write a new DNS API for managing records, shaped something like this:

  • GET /records
  • DELETE /records/<ID>
  • DELETE /records/ (delete all records for a user)
  • POST /records/ (create record)
  • POST /records/<ID> (update record)

This meant that I could actually write tests for my code, since the backend is in Go and I do know how to write tests in Go.

what I learned: it’s okay for an API to duplicate information

I had this idea that APIs shouldn’t return duplicate information – for example if I get a DNS record, it should only include a given piece of information once.

But I ran into a problem with that idea when displaying MX records: an MX record has 2 fields, “preference”, and “mail server”. And I needed to display that information in 2 different ways on the frontend:

  1. In a form, where “Preference” and “Mail Server” are 2 different form fields (like 10 and mail.example.com)
  2. In a summary view, where I wanted to just show the record (10 mail.example.com)

This is kind of a small problem, but it came up in a few different places.

I talked to my friend Marco Rogers about this, and based on some advice from him I realized that I could return the same information in the API in 2 different ways! Then the frontend just has to display it. So I started just returning duplicate information in the API, something like this:

{
  values: {'Preference': 10, 'Server': 'mail.example.com'},
  content: '10 mail.example.com',
  ...
}

I ended up using this pattern in a couple of other places where I needed to display the same information in 2 different ways and it was SO much easier.

I think what I learned from this is that if I’m making an API that isn’t intended for external use (there are no users of this API other than the frontend!), I can tailor it very specifically to the frontend’s needs and that’s okay.

challenge 3: what’s a record’s ID?

In Mess With DNS (and I think in most DNS user interfaces!), you create, add, and delete records.

But that’s not how the PowerDNS API works. In PowerDNS, you create a zone, which is made of record sets. Records don’t have any ID in the API at all.

I ended up solving this by generate a fake ID for each records which is made of:

  • its name
  • its type
  • and its content (base64-encoded)

For example one record’s ID is brooch225.messwithdns.com.|NS|bnMxLm1lc3N3aXRoZG5zLmNvbS4=

Then I can search through the zone and find the appropriate record to update it.

This means that if you update a record then its ID will change which isn’t usually what I want in an ID, but that seems fine.

challenge 4: making clear error messages

I think the error messages that the PowerDNS API returns aren’t really intended to be shown to end users, for example:

  • Name 'new\032site.island358.messwithdns.com.' contains unsupported characters (this error encodes the space as \032, which is a bit disorienting if you don’t know that the space character is 32 in ASCII)
  • RRset test.pear5.messwithdns.com. IN CNAME: Conflicts with pre-existing RRset (this talks about RRsets, which aren’t a concept that the Mess With DNS UI has at all)
  • Record orange.beryl5.messwithdns.com./A '1.2.3.4$': Parsing record content (try 'pdnsutil check-zone'): unable to parse IP address, strange character: $ (mentions “pdnsutil”, a utility which Mess With DNS’s users don’t have access to in this context)

I ended up handling this in two ways:

  1. Do some initial basic validation of values that users enter (like IP addresses), so I can just return errors like Invalid IPv4 address: "1.2.3.4$
  2. If that goes well, send the request to PowerDNS and if we get an error back, then do some hacky translation of those messages to make them clearer.

Sometimes users will still get errors from PowerDNS directly, but I added some logging of all the errors that users see, so hopefully I can review them and add extra translations if there are other common errors that come up.

I think what I learned from this is that if I’m building a user-facing application on top of an API, I need to be pretty thoughtful about how I resurface those errors to users.

challenge 5: setting up SQLite

Previously Mess With DNS was using a Postgres database. This was problematic because I only gave the Postgres machine 256MB of RAM, which meant that the database got OOM killed almost every single day. I never really worked out exactly why it got OOM killed every day, but that’s how it was. I spent some time trying to tune Postgres’ memory usage by setting the max connections / work-mem / maintenance-work-mem and it helped a bit but didn’t solve the problem.

So for this refactor I decided to use SQLite instead, because the website doesn’t really get that much traffic. There are some choices involved with using SQLite, and I decided to:

  1. Run db.SetMaxOpenConns(1) to make sure that we only open 1 connection to the database at a time, to prevent SQLITE_BUSY errors from two threads trying to access the database at the same time (just setting WAL mode didn’t work)
  2. Use separate databases for each of the 3 tables (users, records, and requests) to reduce contention. This maybe isn’t really necessary, but there was no reason I needed the tables to be in the same database so I figured I’d set up separate databases to be safe.
  3. Use the cgo-free modernc.org/sqlite, which translates SQLite’s source code to Go. I might switch to a more “normal” sqlite implementation instead at some point and use cgo though. I think the main reason I prefer to avoid cgo is that cgo has landed me with difficult-to-debug errors in the past.
  4. use WAL mode

I still haven’t set up backups, though I don’t think my Postgres database had backups either. I think I’m unlikely to use litestream for backups – Mess With DNS is very far from a critical application, and I think daily backups that I could recover from in case of a disaster are more than good enough.

challenge 6: upgrading Vue & managing forms

This has nothing to do with PowerDNS but I decided to upgrade Vue.js from version 2 to 3 as part of this refresh. The main problem with that is that the form validation library I was using (FormKit) completely changed its API between Vue 2 and Vue 3, so I decided to just stop using it instead of learning the new API.

I ended up switching to some form validation tools that are built into the browser like required and oninvalid (here’s the code). I think it could use some of improvement, I still don’t understand forms very well.

challenge 7: managing state in the frontend

This also has nothing to do with PowerDNS, but when modifying the frontend I realized that my state management in the frontend was a mess – in every place where I made an API request to the backend, I had to try to remember to add a “refresh records” call after that in every place that I’d modified the state and I wasn’t always consistent about it.

With some more advice from Marco, I ended up implementing a single global state management store which stores all the state for the application, and which lets me create/update/delete records.

Then my components can just call store.createRecord(record), and the store will automatically resynchronize all of the state as needed.

challenge 8: sequencing the project

This project ended up having several steps because I reworked the whole integration between the frontend and the backend. I ended up splitting it into a few different phases:

  1. Upgrade Vue from v2 to v3
  2. Make the state management store
  3. Implement a different backend API, move a lot of DNS logic out of the frontend, and add tests for the backend
  4. Integrate PowerDNS

I made sure that the website was (more or less) 100% working and then deployed it in between phases, so that the amount of changes I was managing at a time stayed somewhat under control.

the new website is up now!

I released the upgraded website a few days ago and it seems to work! The PowerDNS API has been great to work on top of, and I’m relieved that there’s a whole class of problems that I now don’t have to think about at all, other than potentially trying to make the error messages from PowerDNS a little clearer. Using PowerDNS has fixed a lot of the DNS issues that folks have reported in the last few years and it feels great.

If you run into problems with the new Mess With DNS I’d love to hear about them here.

2024-08-06T08:38:35+00:00 Fullscreen Open in Tab
Go structs are copied on assignment (and other things about Go I'd missed)

I’ve been writing Go pretty casually for years – the backends for all of my playgrounds (nginx, dns, memory, more DNS) are written in Go, but many of those projects are just a few hundred lines and I don’t come back to those codebases much.

I thought I more or less understood the basics of the language, but this week I’ve been writing a lot more Go than usual while working on some upgrades to Mess with DNS, and ran into a bug that revealed I was missing a very basic concept!

Then I posted about this on Mastodon and someone linked me to this very cool site (and book) called 100 Go Mistakes and How To Avoid Them by Teiva Harsanyi. It just came out in 2022 so it’s relatively new.

I decided to read through the site to see what else I was missing, and found a couple of other misconceptions I had about Go. I’ll talk about some of the mistakes that jumped out to me the most, but really the whole 100 Go Mistakes site is great and I’d recommend reading it.

Here’s the initial mistake that started me on this journey:

mistake 1: not understanding that structs are copied on assignment

Let’s say we have a struct:

type Thing struct {
    Name string
}

and this code:

thing := Thing{"record"}
other_thing := thing
other_thing.Name = "banana"
fmt.Println(thing)

This prints “record” and not “banana” (play.go.dev link), because thing is copied when you assign it to other_thing.

the problem this caused me: ranges

The bug I spent 2 hours of my life debugging last week was effectively this code (play.go.dev link):

type Thing struct {
  Name string
}
func findThing(things []Thing, name string) *Thing {
  for _, thing := range things {
    if thing.Name == name {
      return &thing
    }
  }
  return nil
}

func main() {
  things := []Thing{Thing{"record"}, Thing{"banana"}}
  thing := findThing(things, "record")
  thing.Name = "gramaphone"
  fmt.Println(things)
}

This prints out [{record} {banana}] – because findThing returned a copy, we didn’t change the name in the original array.

This mistake is #30 in 100 Go Mistakes.

I fixed the bug by changing it to something like this (play.go.dev link), which returns a reference to the item in the array we’re looking for instead of a copy.

func findThing(things []Thing, name string) *Thing {
  for i := range things {
    if things[i].Name == name {
      return &things[i]
    }
  }
  return nil
}

why didn’t I realize this?

When I learned that I was mistaken about how assignment worked in Go I was really taken aback, like – it’s such a basic fact about the language works! If I was wrong about that then what ELSE am I wrong about in Go????

My best guess for what happened is:

  1. I’ve heard for my whole life that when you define a function, you need to think about whether its arguments are passed by reference or by value
  2. So I’d thought about this in Go, and I knew that if you pass a struct as a value to a function, it gets copied – if you want to pass a reference then you have to pass a pointer
  3. But somehow it never occurred to me that you need to think about the same thing for assignments, perhaps because in most of the other languages I use (Python, JS, Java) I think everything is a reference anyway. Except for in Rust, where you do have values that you make copies of but I think most of the time I had to run .clone() explicitly. (though apparently structs will be automatically copied on assignment if the struct implements the Copy trait)
  4. Also obviously I just don’t write that much Go so I guess it’s never come up.

mistake 2: side effects appending slices (#25)

When you subset a slice with x[2:3], the original slice and the sub-slice share the same backing array, so if you append to the new slice, it can unintentionally change the old slice:

For example, this code prints [1 2 3 555 5] (code on play.go.dev)

x := []int{1, 2, 3, 4, 5}
y := x[2:3]
y = append(y, 555)
fmt.Println(x)

I don’t think this has ever actually happened to me, but it’s alarming and I’m very happy to know about it.

Apparently you can avoid this problem by changing y := x[2:3] to y := x[2:3:3], which restricts the new slice’s capacity so that appending to it will re-allocate a new slice. Here’s some code on play.go.dev that does that.

mistake 3: not understanding the different types of method receivers (#42)

This one isn’t a “mistake” exactly, but it’s been a source of confusion for me and it’s pretty simple so I’m glad to have it cleared up.

In Go you can declare methods in 2 different ways:

  1. func (t Thing) Function() (a “value receiver”)
  2. func (t *Thing) Function() (a “pointer receiver”)

My understanding now is that basically:

  • If you want the method to mutate the struct t, you need a pointer receiver.
  • If you want to make sure the method doesn’t mutate the struct t, use a value receiver.

Explanation #42 has a bunch of other interesting details though. There’s definitely still something I’m missing about value vs pointer receivers (I got a compile error related to them a couple of times in the last week that I still don’t understand), but hopefully I’ll run into that error again soon and I can figure it out.

more interesting things I noticed

Some more notes from 100 Go Mistakes:

Also there are some things that have tripped me up in the past, like:

this “100 common mistakes” format is great

I really appreciated this “100 common mistakes” format – it made it really easy for me to skim through the mistakes and very quickly mentally classify them into:

  1. yep, I know that
  2. not interested in that one right now
  3. WOW WAIT I DID NOT KNOW THAT, THAT IS VERY USEFUL!!!!

It looks like “100 Common Mistakes” is a series of books from Manning and they also have “100 Java Mistakes” and an upcoming “100 SQL Server Mistakes”.

Also I enjoyed what I’ve read of Effective Python by Brett Slatkin, which has a similar “here are a bunch of short Python style tips” structure where you can quickly skim it and take what’s useful to you. There’s also Effective C++, Effective Java, and probably more.

some other Go resources

other resources I’ve appreciated:

2024-07-21T12:54:40-07:00 Fullscreen Open in Tab
My IETF 120 Agenda

Here's where you can find me at IETF 120 in Vancouver!

Monday

  • 9:30 - 11:30 • alldispatch • Regency C/D
  • 13:00 - 15:00 • oauth • Plaza B
  • 18:30 - 19:30 • Hackdemo Happy Hour • Regency Hallway

Tuesday

  • 15:30 - 17:00 • oauth • Georgia A
  • 17:30 - 18:30 • oauth • Plaza B

Wednesday

  • 9:30 - 11:30 • wimse • Georgia A
  • 11:45 - 12:45 • Chairs Forum • Regency C/D
  • 17:30 - 19:30 • IETF Plenary • Regency A/B/C/D

Thursday

  • 17:00 - 18:00 • spice • Regency A/B
  • 18:30 - 19:30 • spice • Regency A/B

Friday

  • 13:00 - 15:00 • oauth • Regency A/B

My Current Drafts

2024-07-08T13:00:15+00:00 Fullscreen Open in Tab
Entering text in the terminal is complicated

The other day I asked what folks on Mastodon find confusing about working in the terminal, and one thing that stood out to me was “editing a command you already typed in”.

This really resonated with me: even though entering some text and editing it is a very “basic” task, it took me maybe 15 years of using the terminal every single day to get used to using Ctrl+A to go to the beginning of the line (or Ctrl+E for the end – I think I used Home/End instead).

So let’s talk about why entering text might be hard! I’ll also share a few tips that I wish I’d learned earlier.

it’s very inconsistent between programs

A big part of what makes entering text in the terminal hard is the inconsistency between how different programs handle entering text. For example:

  1. some programs (cat, nc, git commit --interactive, etc) don’t support using arrow keys at all: if you press arrow keys, you’ll just see ^[[D^[[D^[[C^[[C^
  2. many programs (like irb, python3 on a Linux machine and many many more) use the readline library, which gives you a lot of basic functionality (history, arrow keys, etc)
  3. some programs (like /usr/bin/python3 on my Mac) do support very basic features like arrow keys, but not other features like Ctrl+left or reverse searching with Ctrl+R
  4. some programs (like the fish shell or ipython3 or micro or vim) have their own fancy system for accepting input which is totally custom

So there’s a lot of variation! Let’s talk about each of those a little more.

mode 1: the baseline

First, there’s “the baseline” – what happens if a program just accepts text by calling fgets() or whatever and doing absolutely nothing else to provide a nicer experience. Here’s what using these tools typically looks for me – If I start the version of dash installed on my machine (a pretty minimal shell) press the left arrow keys, it just prints ^[[D to the terminal.

$ ls l-^[[D^[[D^[[D

At first it doesn’t seem like all of these “baseline” tools have much in common, but there are actually a few features that you get for free just from your terminal, without the program needing to do anything special at all.

The things you get for free are:

  1. typing in text, obviously
  2. backspace
  3. Ctrl+W, to delete the previous word
  4. Ctrl+U, to delete the whole line
  5. a few other things unrelated to text editing (like Ctrl+C to interrupt the process, Ctrl+Z to suspend, etc)

This is not great, but it means that if you want to delete a word you generally can do it with Ctrl+W instead of pressing backspace 15 times, even if you’re in an environment which is offering you absolutely zero features.

You can get a list of all the ctrl codes that your terminal supports with stty -a.

mode 2: tools that use readline

The next group is tools that use readline! Readline is a GNU library to make entering text more pleasant, and it’s very widely used.

My favourite readline keyboard shortcuts are:

  1. Ctrl+E (or End) to go to the end of the line
  2. Ctrl+A (or Home) to go to the beginning of the line
  3. Ctrl+left/right arrow to go back/forward 1 word
  4. up arrow to go back to the previous command
  5. Ctrl+R to search your history

And you can use Ctrl+W / Ctrl+U from the “baseline” list, though Ctrl+U deletes from the cursor to the beginning of the line instead of deleting the whole line. I think Ctrl+W might also have a slightly different definition of what a “word” is.

There are a lot more (here’s a full list), but those are the only ones that I personally use.

The bash shell is probably the most famous readline user (when you use Ctrl+R to search your history in bash, that feature actually comes from readline), but there are TONS of programs that use it – for example psql, irb, python3, etc.

tip: you can make ANYTHING use readline with rlwrap

One of my absolute favourite things is that if you have a program like nc without readline support, you can just run rlwrap nc to turn it into a program with readline support!

This is incredible and makes a lot of tools that are borderline unusable MUCH more pleasant to use. You can even apparently set up rlwrap to include your own custom autocompletions, though I’ve never tried that.

some reasons tools might not use readline

I think reasons tools might not use readline might include:

  • the program is very simple (like cat or nc) and maybe the maintainers don’t want to bring in a relatively large dependency
  • license reasons, if the program’s license is not GPL-compatible – readline is GPL-licensed, not LGPL
  • only a very small part of the program is interactive, and maybe readline support isn’t seen as important. For example git has a few interactive features (like git add -p), but not very many, and usually you’re just typing a single character like y or n – most of the time you need to really type something significant in git, it’ll drop you into a text editor instead.

For example idris2 says they don’t use readline to keep dependencies minimal and suggest using rlwrap to get better interactive features.

how to know if you’re using readline

The simplest test I can think of is to press Ctrl+R, and if you see:

(reverse-i-search)`':

then you’re probably using readline. This obviously isn’t a guarantee (some other library could use the term reverse-i-search too!), but I don’t know of another system that uses that specific term to refer to searching history.

the readline keybindings come from Emacs

Because I’m a vim user, It took me a very long time to understand where these keybindings come from (why Ctrl+A to go to the beginning of a line??? so weird!)

My understanding is these keybindings actually come from Emacs – Ctrl+A and Ctrl+E do the same thing in Emacs as they do in Readline and I assume the other keyboard shortcuts mostly do as well, though I tried out Ctrl+W and Ctrl+U in Emacs and they don’t do the same thing as they do in the terminal so I guess there are some differences.

There’s some more history of the Readline project here.

mode 3: another input library (like libedit)

On my Mac laptop, /usr/bin/python3 is in a weird middle ground where it supports some readline features (for example the arrow keys), but not the other ones. For example when I press Ctrl+left arrow, it prints out ;5D, like this:

$ python3
>>> importt subprocess;5D

Folks on Mastodon helped me figure out that this is because in the default Python install on Mac OS, the Python readline module is actually backed by libedit, which is a similar library which has fewer features, presumably because Readline is GPL licensed.

Here’s how I was eventually able to figure out that Python was using libedit on my system:

$ python3 -c "import readline; print(readline.__doc__)"
Importing this module enables command line editing using libedit readline.

Generally Python uses readline though if you install it on Linux or through Homebrew. It’s just that the specific version that Apple includes on their systems doesn’t have readline. Also Python 3.13 is going to remove the readline dependency in favour of a custom library, so “Python uses readline” won’t be true in the future.

I assume that there are more programs on my Mac that use libedit but I haven’t looked into it.

mode 4: something custom

The last group of programs is programs that have their own custom (and sometimes much fancier!) system for editing text. This includes:

  • most terminal text editors (nano, micro, vim, emacs, etc)
  • some shells (like fish), for example it seems like fish supports Ctrl+Z for undo when typing in a command. Zsh’s line editor is called zle.
  • some REPLs (like ipython), for example IPython uses the prompt_toolkit library instead of readline
  • lots of other programs (like atuin)

Some features you might see are:

  • better autocomplete which is more customized to the tool
  • nicer history management (for example with syntax highlighting) than the default you get from readline
  • more keyboard shortcuts

custom input systems are often readline-inspired

I went looking at how Atuin (a wonderful tool for searching your shell history that I started using recently) handles text input. Looking at the code and some of the discussion around it, their implementation is custom but it’s inspired by readline, which makes sense to me – a lot of users are used to those keybindings, and it’s convenient for them to work even though atuin doesn’t use readline.

prompt_toolkit (the library IPython uses) is similar – it actually supports a lot of options (including vi-like keybindings), but the default is to support the readline-style keybindings.

This is like how you see a lot of programs which support very basic vim keybindings (like j for down and k for up). For example Fastmail supports j and k even though most of its other keybindings don’t have much relationship to vim.

I assume that most “readline-inspired” custom input systems have various subtle incompatibilities with readline, but this doesn’t really bother me at all personally because I’m extremely ignorant of most of readline’s features. I only use maybe 5 keyboard shortcuts, so as long as they support the 5 basic commands I know (which they always do!) I feel pretty comfortable. And usually these custom systems have much better autocomplete than you’d get from just using readline, so generally I prefer them over readline.

lots of shells support vi keybindings

Bash, zsh, and fish all have a “vi mode” for entering text. In a very unscientific poll I ran on Mastodon, 12% of people said they use it, so it seems pretty popular.

Readline also has a “vi mode” (which is how Bash’s support for it works), so by extension lots of other programs have it too.

I’ve always thought that vi mode seems really cool, but for some reason even though I’m a vim user it’s never stuck for me.

understanding what situation you’re in really helps

I’ve spent a lot of my life being confused about why a command line application I was using wasn’t behaving the way I wanted, and it feels good to be able to more or less understand what’s going on.

I think this is roughly my mental flowchart when I’m entering text at a command line prompt:

  1. Do the arrow keys not work? Probably there’s no input system at all, but at least I can use Ctrl+W and Ctrl+U, and I can rlwrap the tool if I want more features.
  2. Does Ctrl+R print reverse-i-search? Probably it’s readline, so I can use all of the readline shortcuts I’m used to, and I know I can get some basic history and press up arrow to get the previous command.
  3. Does Ctrl+R do something else? This is probably some custom input library: it’ll probably act more or less like readline, and I can check the documentation if I really want to know how it works.

Being able to diagnose what’s going on like this makes the command line feel a more predictable and less chaotic.

some things this post left out

There are lots more complications related to entering text that we didn’t talk about at all here, like:

  • issues related to ssh / tmux / etc
  • the TERM environment variable
  • how different terminals (gnome terminal, iTerm, xterm, etc) have different kinds of support for copying/pasting text
  • unicode
  • probably a lot more
2024-07-03T08:00:20+00:00 Fullscreen Open in Tab
Reasons to use your shell's job control

Hello! Today someone on Mastodon asked about job control (fg, bg, Ctrl+z, wait, etc). It made me think about how I don’t use my shell’s job control interactively very often: usually I prefer to just open a new terminal tab if I want to run multiple terminal programs, or use tmux if it’s over ssh. But I was curious about whether other people used job control more often than me.

So I asked on Mastodon for reasons people use job control. There were a lot of great responses, and it even made me want to consider using job control a little more!

In this post I’m only going to talk about using job control interactively (not in scripts) – the post is already long enough just talking about interactive use.

what’s job control?

First: what’s job control? Well – in a terminal, your processes can be in one of 3 states:

  1. in the foreground. This is the normal state when you start a process.
  2. in the background. This is what happens when you run some_process &: the process is still running, but you can’t interact with it anymore unless you bring it back to the foreground.
  3. stopped. This is what happens when you start a process and then press Ctrl+Z. This pauses the process: it won’t keep using the CPU, but you can restart it if you want.

“Job control” is a set of commands for seeing which processes are running in a terminal and moving processes between these 3 states

how to use job control

  • fg brings a process to the foreground. It works on both stopped processes and background processes. For example, if you start a background process with cat < /dev/zero &, you can bring it back to the foreground by running fg
  • bg restarts a stopped process and puts it in the background.
  • Pressing Ctrl+z stops the current foreground process.
  • jobs lists all processes that are active in your terminal
  • kill sends a signal (like SIGKILL) to a job (this is the shell builtin kill, not /bin/kill)
  • disown removes the job from the list of running jobs, so that it doesn’t get killed when you close the terminal
  • wait waits for all background processes to complete. I only use this in scripts though.
  • apparently in bash/zsh you can also just type %2 instead of fg %2

I might have forgotten some other job control commands but I think those are all the ones I’ve ever used.

You can also give fg or bg a specific job to foreground/background. For example if I see this in the output of jobs:

$ jobs
Job Group State   Command
1   3161  running cat < /dev/zero &
2   3264  stopped nvim -w ~/.vimkeys $argv

then I can foreground nvim with fg %2. You can also kill it with kill -9 %2, or just kill %2 if you want to be more gentle.

how is kill %2 implemented?

I was curious about how kill %2 works – does %2 just get replaced with the PID of the relevant process when you run the command, the way environment variables are? Some quick experimentation shows that it isn’t:

$ echo kill %2
kill %2
$ type kill
kill is a function with definition
# Defined in /nix/store/vicfrai6lhnl8xw6azq5dzaizx56gw4m-fish-3.7.0/share/fish/config.fish

So kill is a fish builtin that knows how to interpret %2. Looking at the source code (which is very easy in fish!), it uses jobs -p %2 to expand %2 into a PID, and then runs the regular kill command.

on differences between shells

Job control is implemented by your shell. I use fish, but my sense is that the basics of job control work pretty similarly in bash, fish, and zsh.

There are definitely some shells which don’t have job control at all, but I’ve only used bash/fish/zsh so I don’t know much about that.

Now let’s get into a few reasons people use job control!

reason 1: kill a command that’s not responding to Ctrl+C

I run into processes that don’t respond to Ctrl+C pretty regularly, and it’s always a little annoying – I usually switch terminal tabs to find and kill and the process. A bunch of people pointed out that you can do this in a faster way using job control!

How to do this: Press Ctrl+Z, then kill %1 (or the appropriate job number if there’s more than one stopped/background job, which you can get from jobs). You can also kill -9 if it’s really not responding.

reason 2: background a GUI app so it’s not using up a terminal tab

Sometimes I start a GUI program from the command line (for example with wireshark some_file.pcap), forget to start it in the background, and don’t want it eating up my terminal tab.

How to do this:

  • move the GUI program to the background by pressing Ctrl+Z and then running bg.
  • you can also run disown to remove it from the list of jobs, to make sure that the GUI program won’t get closed when you close your terminal tab.

Personally I try to avoid starting GUI programs from the terminal if possible because I don’t like how their stdout pollutes my terminal (on a Mac I use open -a Wireshark instead because I find it works better but sometimes you don’t have another choice.

reason 2.5: accidentally started a long-running job without tmux

This is basically the same as the GUI app thing – you can move the job to the background and disown it.

I was also curious about if there are ways to redirect a process’s output to a file after it’s already started. A quick search turned up this Linux-only tool which is based on nelhage’s reptyr (which lets you for example move a process that you started outside of tmux to tmux) but I haven’t tried either of those.

reason 3: running a command while using vim

A lot of people mentioned that if they want to quickly test something while editing code in vim or another terminal editor, they like to use Ctrl+Z to stop vim, run the command, and then run fg to go back to their editor.

You can also use this to check the output of a command that you ran before starting vim.

I’ve never gotten in the habit of this, probably because I mostly use a GUI version of vim. I feel like I’d also be likely to switch terminal tabs and end up wondering “wait… where did I put my editor???” and have to go searching for it.

reason 4: preferring interleaved output

A few people said that they prefer to the output of all of their commands being interleaved in the terminal. This really surprised me because I usually think of having the output of lots of different commands interleaved as being a bad thing, but one person said that they like to do this with tcpdump specifically and I think that actually sounds extremely useful. Here’s what it looks like:

# start tcpdump
$ sudo tcpdump -ni any port 1234 &
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes

# run curl
$ curl google.com:1234
13:13:29.881018 IP 192.168.1.173.49626 > 142.251.41.78.1234: Flags [S], seq 613574185, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2730440518 ecr 0,sackOK,eol], length 0
13:13:30.881963 IP 192.168.1.173.49626 > 142.251.41.78.1234: Flags [S], seq 613574185, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2730441519 ecr 0,sackOK,eol], length 0
13:13:31.882587 IP 192.168.1.173.49626 > 142.251.41.78.1234: Flags [S], seq 613574185, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2730442520 ecr 0,sackOK,eol], length 0
 
# when you're done, kill the tcpdump in the background
$ kill %1 

I think it’s really nice here that you can see the output of tcpdump inline in your terminal – when I’m using tcpdump I’m always switching back and forth and I always get confused trying to match up the timestamps, so keeping everything in one terminal seems like it might be a lot clearer. I’m going to try it.

reason 5: suspend a CPU-hungry program

One person said that sometimes they’re running a very CPU-intensive program, for example converting a video with ffmpeg, and they need to use the CPU for something else, but don’t want to lose the work that ffmpeg already did.

You can do this by pressing Ctrl+Z to pause the process, and then run fg when you want to start it again.

reason 6: you accidentally ran Ctrl+Z

Many people replied that they didn’t use job control intentionally, but that they sometimes accidentally ran Ctrl+Z, which stopped whatever program was running, so they needed to learn how to use fg to bring it back to the foreground.

The were also some mentions of accidentally running Ctrl+S too (which stops your terminal and I think can be undone with Ctrl+Q). My terminal totally ignores Ctrl+S so I guess I’m safe from that one though.

reason 7: already set up a bunch of environment variables

Some folks mentioned that they already set up a bunch of environment variables that they need to run various commands, so it’s easier to use job control to run multiple commands in the same terminal than to redo that work in another tab.

reason 8: it’s your only option

Probably the most obvious reason to use job control to manage multiple processes is “because you have to” – maybe you’re in single-user mode, or on a very restricted computer, or SSH’d into a machine that doesn’t have tmux or screen and you don’t want to create multiple SSH sessions.

reason 9: some people just like it better

Some people also said that they just don’t like using terminal tabs: for instance a few folks mentioned that they prefer to be able to see all of their terminals on the screen at the same time, so they’d rather have 4 terminals on the screen and then use job control if they need to run more than 4 programs.

I learned a few new tricks!

I think my two main takeaways from thos post is I’ll probably try out job control a little more for:

  1. killing processes that don’t respond to Ctrl+C
  2. running tcpdump in the background with whatever network command I’m running, so I can see both of their output in the same place
2024-06-03T09:45:11+00:00 Fullscreen Open in Tab
New zine: How Git Works!

Hello! I’ve been writing about git on here nonstop for months, and the git zine is FINALLY done! It came out on Friday!

You can get it for $12 here: https://wizardzines.com/zines/git, or get an 14-pack of all my zines here.

Here’s the cover:

the table of contents

Here’s the table of contents:

who is this zine for?

I wrote this zine for people who have been using git for years and are still afraid of it. As always – I think it sucks to be afraid of the tools that you use in your work every day! I want folks to feel confident using git.

My goals are:

  • To explain how some parts of git that initially seem scary (like “detached HEAD state”) are pretty straightforward to deal with once you understand what’s going on
  • To show some parts of git you probably should be careful around. For example, the stash is one of the places in git where it’s easiest to lose your work in a way that’s incredibly annoying to recover form, and I avoid using it heavily because of that.
  • To clear up a few common misconceptions about how the core parts of git (like commits, branches, and merging) work

what’s the difference between this and Oh Shit, Git!

You might be wondering – Julia! You already have a zine about git! What’s going on? Oh Shit, Git! is a set of tricks for fixing git messes. “How Git Works” explains how Git actually works.

Also, Oh Shit, Git! is the amazing Katie Sylor Miller’s concept: we made it into a zine because I was such a huge fan of her work on it.

I think they go really well together.

what’s so confusing about git, anyway?

This zine was really hard for me to write because when I started writing it, I’d been using git pretty confidently for 10 years. I had no real memory of what it was like to struggle with git.

But thanks to a huge amount of help from Marie as well as everyone who talked to me about git on Mastodon, eventually I was able to see that there are a lot of things about git that are counterintuitive, misleading, or just plain confusing. These include:

  • confusing terminology (for example “fast-forward”, “reference”, or “remote-tracking branch”)
  • misleading messages (for example how Your branch is up to date with 'origin/main' doesn’t necessary mean that your branch is up to date with the main branch on the origin)
  • uninformative output (for example how I STILL can’t reliably figure out which code comes from which branch when I’m looking at a merge conflict)
  • a lack of guidance around handling diverged branches (for example how when you run git pull and your branch has diverged from the origin, it doesn’t give you great guidance how to handle the situation)
  • inconsistent behaviour (for example how git’s reflogs are almost always append-only, EXCEPT for the stash, where git will delete entries when you run git stash drop)

The more I heard from people how about how confusing they find git, the more it became clear that git really does not make it easy to figure out what its internal logic is just by using it.

handling git’s weirdnesses becomes pretty routine

The previous section made git sound really bad, like “how can anyone possibly use this thing?”.

But my experience is that after I learned what git actually means by all of its weird error messages, dealing with it became pretty routine! I’ll see an error: failed to push some refs to 'github.com:jvns/wizard-zines-site', realize “oh right, probably a coworker made some changes to main since I last ran git pull”, run git pull --rebase to incorporate their changes, and move on with my day. The whole thing takes about 10 seconds.

Or if I see a You are in 'detached HEAD' state warning, I’ll just make sure to run git checkout mybranch before continuing to write code. No big deal.

For me (and for a lot of folks I talk to about git!), dealing with git’s weird language can become so normal that you totally forget why anybody would even find it weird.

a little bit of internals

One of my biggest questions when writing this zine was how much to focus on what’s in the .git directory. We ended up deciding to include a couple of pages about internals (“inside .git”, pages 14-15), but otherwise focus more on git’s behaviour when you use it and why sometimes git behaves in unexpected ways.

This is partly because there are lots of great guides to git’s internals out there already (1, 2), and partly because I think even if you have read one of these guides to git’s internals, it isn’t totally obvious how to connect that information to what you actually see in git’s user interface.

For example: it’s easy to find documentation about remotes in git – for example this page says:

Remote-tracking branches […] remind you where the branches in your remote repositories were the last time you connected to them.

But even if you’ve read that, you might not realize that the statement Your branch is up to date with 'origin/main'" in git status doesn’t necessarily mean that you’re actually up to date with the remote main branch.

So in general in the zine we focus on the behaviour you see in Git’s UI, and then explain how that relates to what’s happening internally in Git.

the cheat sheet

The zine also comes with a free printable cheat sheet: (click to get a PDF version)

it comes with an HTML transcript!

The zine also comes with an HTML transcript, to (hopefully) make it easier to read on a screen reader! Our Operations Manager, Lee, transcribed all of the pages and wrote image descriptions. I’d love feedback about the experience of reading the zine on a screen reader if you try it.

I really do love git

I’ve been pretty critical about git in this post, but I only write zines about technologies I love, and git is no exception.

Some reasons I love git:

  • it’s fast!
  • it’s backwards compatible! I learned how to use it 10 years ago and everything I learned then is still true
  • there’s tons of great free Git hosting available out there (GitHub! Gitlab! a million more!), so I can easily back up all my code
  • simple workflows are REALLY simple (if I’m working on a project on my own, I can just run git commit -am 'whatever' and git push over and over again and it works perfectly)
  • Almost every internal file in git is a pretty simple text file (or has a version which is a text file), which makes me feel like I can always understand exactly what’s going on under the hood if I want to.

I hope this zine helps some of you love it too.

people who helped with this zine

I don’t make these zines by myself!

I worked with Marie Claire LeBlanc Flanagan every morning for 8 months to write clear explanations of git.

The cover is by Vladimir Kašiković, Gersande La Flèche did copy editing, James Coglan (of the great Building Git) did technical review, our Operations Manager Lee did the transcription as well as a million other things, my partner Kamal read the zine and told me which parts were off (as he always does), and I had a million great conversations with Marco Rogers about git.

And finally, I want to thank all the beta readers! There were 66 this time which is a record! They left hundreds of comments about what was confusing, what they learned, and which of my jokes were funny. It’s always hard to hear from beta readers that a page I thought made sense is actually extremely confusing, and fixing those problems before the final version makes the zine so much better.

get the zine

Here are some links to get the zine again:

As always, you can get either a PDF version to print at home or a print version shipped to your house. The only caveat is print orders will ship in July – I need to wait for orders to come in to get an idea of how many I should print before sending it to the printer.

thank you

As always: if you’ve bought zines in the past, thank you for all your support over the years. And thanks to all of you (1000+ people!!!) who have already bought the zine in the first 3 days. It’s already set a record for most zines sold in a single day and I’ve been really blown away.

2024-05-12T07:39:30-07:00 Fullscreen Open in Tab
FedCM for IndieAuth

IndieWebCamp Düsseldorf took place this weekend, and I was inspired to work on a quick hack for demo day to show off a new feature I've been working on for IndieAuth.

Since I do actually use my website to log in to different websites on a regular basis, I am often presented with the login screen asking for my domain name, which is admittedly an annoying part of the process. I don't even like having to enter my email address when I log in to a site, and entering my domain isn't any better.

So instead, I'd like to get rid of this prompt, and let the browser handle it for you! Here's a quick video of logging in to a website using my domain with the new browser API:

So how does this work?

For the last couple of years, there has been an ongoing effort at the Federated Identity Community Group at the W3C to build a new API in browsers that can sit in the middle of login flows. It's primarily being driven by Google for their use case of letting websites show a Google login popup dialog without needing 3rd party cookies and doing so in a privacy-preserving way. There's a lot to unpack here, more than I want to go into in this blog post. You can check out Tim Cappalli's slides from the OAuth Security Workshop for a good explainer on the background and how it works.

However, there are a few experimental features that are being considered for the API to accommodate use cases beyond the "Sign in with Google" case. The one that's particularly interesting to the IndieAuth use case is the IdP Registration API. This API allows any website to register itself as an identity provider that can appear in the account chooser popup, so that a relying party website doesn't have to list out all the IdPs it supports, it can just say it supports "any" IdP. This maps to how IndieAuth is already used today, where a website can accept any user's IndieAuth server without any prior relationship with the user. For more background, check out my previous blog post "OAuth for the Open Web".

So now, with the IdP Registration API in FedCM, your website can tell your browser that it is an IdP, then when a website wants to log you in, it asks your browser to prompt you. You choose your account from the list, the negotiation happens behind the scenes, and you're logged in!

One of the nice things about combining FedCM with IndieAuth is it lends itself nicely to running the FedCM IdP as a separate service from your actual website. I could run an IndieAuth IdP service that you could sign up for and link your website to. Since your identity is your website, your website would be the thing ultimately sent to the relying party that you're signing in to, even though it was brokered through the IdP service. Ultimately this means much faster adoption is possible, since all it takes to turn your website into a FedCM-supported site is adding a single <link> tag to your home page.

So if this sounds interesting to you, leave a comment below! The IdP registration API is currently an early experiment, and Google needs to see actual interest in it in order to keep it around! In particular, they are looking for Relying Parties who would be interested in actually using this to log users in. I am planning on launching this on webmention.io as an experiment. If you have a website where users can sign in with IndieAuth, feel free to get in touch and I'd be happy to help you set up FedCM support as well!

2024-05-02T15:06:00-07:00 Fullscreen Open in Tab
OAuth for Browser-Based Apps Working Group Last Call!

The draft specification OAuth for Browser-Based Applications has just entered Working Group Last Call!

https://datatracker.ietf.org/doc/html/draft-ietf-oauth-browser-based-apps

This begins a two-week period to collect final comments on the draft. Please review the draft and reply on the OAuth mailing list if you have any comments or concerns. And if you've reviewed the document and are happy with the current state, it is also extremely helpful if you can reply on the list to just say "looks good to me"!

If joining the mailing list is too much work, you're also welcome to comment on the Last Call issue on GitHub.

In case you were wondering, yes your comments matter! Even just a small indication of support goes a long way in these discussions!

I am extremely happy with how this draft has turned out, and would like to again give a huge thanks to Philippe De Ryck for the massive amount of work he's put in to the latest few versions to help get this over the finish line!

2024-04-10T12:43:14+00:00 Fullscreen Open in Tab
Notes on git's error messages

While writing about Git, I’ve noticed that a lot of folks struggle with Git’s error messages. I’ve had many years to get used to these error messages so it took me a really long time to understand why folks were confused, but having thought about it much more, I’ve realized that:

  1. sometimes I actually am confused by the error messages, I’m just used to being confused
  2. I have a bunch of strategies for getting more information when the error message git gives me isn’t very informative

So in this post, I’m going to go through a bunch of Git’s error messages, list a few things that I think are confusing about them for each one, and talk about what I do when I’m confused by the message.

improving error messages isn’t easy

Before we start, I want to say that trying to think about why these error messages are confusing has given me a lot of respect for how difficult maintaining Git is. I’ve been thinking about Git for months, and for some of these messages I really have no idea how to improve them.

Some things that seem hard to me about improving error messages:

  • if you come up with an idea for a new message, it’s hard to tell if it’s actually better!
  • work like improving error messages often isn’t funded
  • the error messages have to be translated (git’s error messages are translated into 19 languages!)

That said, if you find these messages confusing, hopefully some of these notes will help clarify them a bit.

error: git push on a diverged branch

$ git push
To github.com:jvns/int-exposed
! [rejected]        main -> main (non-fast-forward)
error: failed to push some refs to 'github.com:jvns/int-exposed'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

$ git status
On branch main
Your branch and 'origin/main' have diverged,
and have 2 and 1 different commits each, respectively.

Some things I find confusing about this:

  1. You get the exact same error message whether the branch is just behind or the branch has diverged. There’s no way to tell which it is from this message: you need to run git status or git pull to find out.
  2. It says failed to push some refs, but it’s not totally clear which references it failed to push. I believe everything that failed to push is listed with ! [rejected] on the previous line– in this case just the main branch.

What I like to do if I’m confused:

  • I’ll run git status to figure out what the state of my current branch is.
  • I think I almost never try to push more than one branch at a time, so I usually totally ignore git’s notes about which specific branch failed to push – I just assume that it’s my current branch

error: git pull on a diverged branch

$ git pull
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

The main thing I think is confusing here is that git is presenting you with a kind of overwhelming number of options: it’s saying that you can either:

  1. configure pull.rebase false, pull.rebase true, or pull.ff only locally
  2. or configure them globally
  3. or run git pull --rebase or git pull --no-rebase

It’s very hard to imagine how a beginner to git could easily use this hint to sort through all these options on their own.

If I were explaining this to a friend, I’d say something like “you can use git pull --rebase or git pull --no-rebase to resolve this with a rebase or merge right now, and if you want to set a permanent preference, you can do that with git config pull.rebase false or git config pull.rebase true.

git config pull.ff only feels a little redundant to me because that’s git’s default behaviour anyway (though it wasn’t always).

What I like to do here:

  • run git status to see the state of my current branch
  • maybe run git log origin/main or git log to see what the diverged commits are
  • usually run git pull --rebase to resolve it
  • sometimes I’ll run git push --force or git reset --hard origin/main if I want to throw away my local work or remote work (for example because I accidentally commited to the wrong branch, or because I ran git commit --amend on a personal branch that only I’m using and want to force push)

error: git checkout asdf (a branch that doesn't exist)

$ git checkout asdf
error: pathspec 'asdf' did not match any file(s) known to git

This is a little weird because we my intention was to check out a branch, but git checkout is complaining about a path that doesn’t exist.

This is happening because git checkout’s first argument can be either a branch or a path, and git has no way of knowing which one you intended. This seems tricky to improve, but I might expect something like “No such branch, commit, or path: asdf”.

What I like to do here:

  • in theory it would be good to use git switch instead, but I keep using git checkout anyway
  • generally I just remember that I need to decode this as “branch asdf doesn’t exist”

error: git switch asdf (a branch that doesn't exist)

$ git switch asdf
fatal: invalid reference: asdf

git switch only accepts a branch as an argument (unless you pass -d), so why is it saying invalid reference: asdf instead of invalid branch: asdf?

I think the reason is that internally, git switch is trying to be helpful in its error messages: if you run git switch v0.1 to switch to a tag, it’ll say:

$ git switch v0.1
fatal: a branch is expected, got tag 'v0.1'`

So what git is trying to communicate with fatal: invalid reference: asdf is “asdf isn’t a branch, but it’s not a tag either, or any other reference”. From my various git polls my impression is that a lot of git users have literally no idea what a “reference” is in git, so I’m not sure if that’s coming across.

What I like to do here:

90% of the time when a git error message says reference I just mentally replace it with branch in my head.

error: git checkout HEAD^

$ git checkout HEAD^
Note: switching to 'HEAD^'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c 

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 182cd3f add "swap byte order" button

This is a tough one. Definitely a lot of people are confused about this message, but obviously there's been a lot of effort to improve it too. I don't have anything smart to say about this one.

What I like to do here:

  • my shell prompt tells me if I’m in detached HEAD state, and generally I can remember not to make new commits while in that state
  • when I’m done looking at whatever old commits I wanted to look at, I’ll run git checkout main or something to go back to a branch

message: git status when a rebase is in progress

This isn’t an error message, but I still find it a little confusing on its own:

$ git status
interactive rebase in progress; onto c694cf8
Last command done (1 command done):
   pick 0a9964d wip
No commands remaining.
You are currently rebasing branch 'main' on 'c694cf8'.
  (fix conflicts and then run "git rebase --continue")
  (use "git rebase --skip" to skip this patch)
  (use "git rebase --abort" to check out the original branch)

Unmerged paths:
  (use "git restore --staged ..." to unstage)
  (use "git add ..." to mark resolution)
  both modified:   index.html

no changes added to commit (use "git add" and/or "git commit -a")

Two things I think could be clearer here:

  1. I think it would be nice if You are currently rebasing branch 'main' on 'c694cf8'. were on the first line instead of the 5th line – right now the first line doesn’t say which branch you’re rebasing.
  2. In this case, c694cf8 is actually origin/main, so I feel like You are currently rebasing branch 'main' on 'origin/main' might be even clearer.

What I like to do here:

My shell prompt includes the branch that I’m currently rebasing, so I rely on that instead of the output of git status.

error: git rebase when a file has been deleted

$ git rebase main
CONFLICT (modify/delete): index.html deleted in 0ce151e (wip) and modified in HEAD.  Version HEAD of index.html left in tree.
error: could not apply 0ce151e... wip

The thing I still find confusing about this is – index.html was modified in HEAD. But what is HEAD? Is it the commit I was working on when I started the merge/rebase, or is it the commit from the other branch? (the answer is “HEAD is your branch if you’re doing a merge, and it’s the “other branch” if you’re doing a rebase, but I always find that hard to remember)

I think I would personally find it easier to understand if the message listed the branch names if possible, something like this:

CONFLICT (modify/delete): index.html deleted on `main` and modified on `mybranch`

error: git status during a merge or rebase (who is "them"?)

$ git status 
On branch master
You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Unmerged paths: (use “git add/rm …” as appropriate to mark resolution) deleted by them: the_file

no changes added to commit (use “git add” and/or “git commit -a”)

I find this one confusing in exactly the same way as the previous message: it says deleted by them:, but what “them” refers to depends on whether you did a merge or rebase or cherry-pick.

  • for a merge, them is the other branch you merged in
  • for a rebase, them is the branch that you were on when you ran git rebase
  • for a cherry-pick, I guess it’s the commit you cherry-picked

What I like to do if I’m confused:

  • try to remember what I did
  • run git show main --stat or something to see what I did on the main branch if I can’t remember

error: git clean

$ git clean
fatal: clean.requireForce defaults to true and neither -i, -n, nor -f given; refusing to clean

I just find it a bit confusing that you need to look up what -i, -n and -f are to be able to understand this error message. I’m personally way too lazy to do that so even though I’ve probably been using git clean for 10 years I still had no idea what -i stood for (interactive) until I was writing this down.

What I like to do if I’m confused:

Usually I just chaotically run git clean -f to delete all my untracked files and hope for the best, though I might actually switch to git clean -i now that I know what -i stands for. Seems a lot safer.

that’s all!

Hopefully some of this is helpful!

2024-03-29T08:15:24-07:00 Fullscreen Open in Tab
OAuth: "grant" vs "flow" vs "grant type"
Is it called an OAuth "grant" or a "flow"? What about "grant type"?

These are common questions when writing documentation for OAuth-related things. While these terms are all used in RFC 6749 and many extensions, the differences between the terminology is never actually explained.

I wanted to finally write down a definition of the terms, along with examples of when each is appropriate.

  • flow - use "flow" when referring to the end-to-end process, for example:
    • "the client initiates the flow by..."
    • "the flow ends with the successful issuance of an access token"
    • This can also be combined with the type of flow, for example:
    • "The Authorization Code flow starts by..."
  • grant - use "grant" when referring to the specific POST request to the token endpoint, for example:
    • "The authorization code grant includes the PKCE code verifier..."
    • "The refresh token grant can be used with or without client authentication..."
    • "Grant" also refers to the abstract concept of the user having granted authorization, which is expressed as the authorization code, or implicitly with the client credentials grant. This is a bit of an academic definition of the term, and is used much less frequently in normal conversation around OAuth.
  • grant type - use "grant type" when referring to the definition of the flow in the spec itself, for example:
    • "there are several drawbacks to the Implicit grant type"
    • "the Authorization Code grant type enables the use of..."

Let me know if you have any suggestions for clarifying any of this, or any other helpful examples to add! I'm planning on adding this summary to OAuth 2.1 so that we have a formal reference for it in the future!

2023-12-01T19:38:05-08:00 Fullscreen Open in Tab
I took the High-Speed Brightline Train from Miami to Orlando with only two hours notice

It was 11am at the Fort Lauderdale airport, an hour after my non-stop flight to Portland was supposed to have boarded. As I had been watching our estimated departure get pushed back in 15 minute increments, I finally received the dreaded news over the loudspeaker - the flight was cancelled entirely. As hordes of people started lining up to rebook their flights with the gate agent, I found a quiet spot in the corner and opened up my laptop to look at my options.

The other Alaska Airlines flight options were pretty terrible. There was a Fort Lauderdale to Seattle to Portland option that would have me landing at midnight. A flight on a partner airline had a 1-hour connection through Dallas, and there were only middle seats available on both legs. So I started to get creative, and searched for flights from Orlando, about 200 miles north. There was a non-stop on Alaska Airlines at 7pm, with plenty of available seats, so I called up customer service and asked them to change my booking. Since the delay was their fault, there were no change fees even though the flight was leaving from a different airport.

So now it was my responsibility to get myself from Miami to Orlando by 7pm. I could have booked a flight on a budget airline for $150, but it wouldn't have been a very nice experience, and I'd have a lot of time to kill in the Orlando airport. Then I remembered the Brightline train recently opened new service from Miami to Orlando, supposedly taking less time than driving there.

Brightline Station Fort Lauderdale

Brightline Station

Never having tried to take that train before, I didn't realize they run a shuttle service from the Fort Lauderdale airport to the train station, so I jumped in an Uber headed to the station. On the way there, I booked a ticket on my phone. The price from Miami to Orlando was $144 for Coach, or $229 for Premium class. Since this will probably be the only time I take this train for the foreseeable future, I splurged for the Premium class ticket to see what that experience is like.

Astute readers will have noticed that I mentioned I booked a ticket from Miami rather than Fort Lauderdale. We'll come back to that in a bit. Once I arrived at the station, I began my Brightline experience.

Walking in to the station felt like something between an airport and a car rental center.

Brightline Station entrance

There was a small ticket counter in the lobby, but I already had a ticket on my phone so I went up the escalators.

Brightline Station escalator

At the top of the escalators was an electronic gate where you scan your QR code to go through. Mine didn't work (again, more on that later), but it was relatively empty and a staff member was able to look at my ticket on my phone and let me through anyway. There was a small X-Ray machine, I tossed my roller bag and backpack onto the belt, but kept my phone and wallet in my pocket, and walked through the security checkpoint.

Once through the minimal security checkpoint, I was up in the waiting area above the platform with a variety of different sections. There was a small bar with drinks and snacks, a couple large seating areas, an automated mini mart, some tall tables...

Stool seating

More seating

Even more seating

Shop entrance

... and the entrance to the Premium lounge.

Brightline Station Premium Lounge

Premium Lounge entrance

The Premium Lounge entrance had another electronic gate with a QR code scanner. I tried getting in but it also rejected my boarding pass. My first thought was I booked my ticket just 10 minutes earlier so it hadn't synced up yet, so I went back to the the security checkpoint and asked what was wrong. They looked at my boarding pass and had no idea what was wrong, and let me in to the lounge via the back employee-only entrance instead.

Once inside the lounge, I did a quick loop to see what kind of food and drink options there were. The lounge was entirely un-attended, the only staff I saw were at the security checkpoint, and someone occasionally coming through to take out dirty dishes.

The first thing you're presented with after entering the lounge is the beverage station. There are 6 taps with beer and wine, and you use a touch screen to make your selection and pour what you want.

Beverages

On the other side of the wall is the food. I arrived at the tail end of the breakfast service, so there were pretty slim pickings by the end.

Breakfast

There were yogurts, granola, a bowl of bacon and egg mix, several kinds of pastries, and a bowl of fruit that nobody seemed to have touched. I don't know if this was just because this was the end of the morning, but if you were vegan or gluten free there was really nothing you could eat there.

There was also a coffee and tea station with some minimal options.

Coffee station

Shortly after I arrived, it rolled over to lunch time, so the staff came out to swap out the food at the food station. The lunch options were also minimal, but there was a bit more selection.

Lunch

There was a good size meat and cheese spread. I'm not a big fan of when they mix the meat and cheese on the same plate, but there was enough of a cheese island in the middle I was reasonably confident I wasn't eating meat juice off the side of the cheeses. The pasta dish also had meat so I didn't investigate further. Two of the three wraps had meat and I wasn't confident about which were which so I skipped those. There was a pretty good spinach and feta salad, and some hummus as well as artichoke dip, and a variety of crackers. If you like desserts, there was an even better selection of small desserts as well.

At this point I was starting to listen for my train's boarding announcement. There was really barely any staff visible anywhere, but the few people I saw had made it clear they would clearly announce the train over the loudspeakers when it was time. There was also a sign at the escalators to the platform that said boarding opens 10 minutes before the train departs.

Ten minute warning

The trains run northbound and southbound every 1-2 hours, so it's likely that you'll only hear one announcement for a train other than yours the entire time you're there.

Departure board

The one train announcement I heard was a good demonstration of how quickly the whole process actually is once the train shows up. The train pulls up, they call everyone down to the platform, and you have ten minutes to get onto the train. Ten minutes isn't much, but you're sitting literally right on top of the train platform so it takes no time to get down there.

View from the lounge

Once your train is called, it's time to head down the escalator to the train platform!

Boarding the Train

Escalators

Escalators

But wait, I mentioned my barcode had failed to be scanned a couple times at this point. Let me explain. Apparently, in my haste in the back of the Uber, I had actually booked a ticket from Miami to Orlando, but since I was already at the Fort Lauderdale airport, I had gone to the Fort Lauderdale Brightline station since it was the closest. So the departure time I saw on my ticket didn't match the time the train arrived at Fort Lauderdale, and the ticket gates refused to let me in because the ticket didn't depart from that station. I don't know why none of the employees who looked at my ticket mentioned this ever. It didn't end up being a big deal because thankfully Miami was earlier in the route, so I essentially just got on my scheduled train 2 stops late.

Brightline Route

So anyway, I made my way down to the platform to board the train. I should also mention at this point that I was on a conference call from my phone. I had previously connected my phone to the free wifi at the station, and it was plenty good enough for the call. As I went down the escalator to the platform, it broke up a bit in the middle of the escalator, but picked back up once I was on the platform outside.

Platform

There were some signs on the platform to indicate "Coach 1", "Coach 2" and "Coach 3" cars. However my ticket was a "Premium" ticket, so I walked to where I assumed the front of the train would be when it pulled up.

Train approach

I got on the train on the front car marked "SMART" and "3", seats 9-17. It wasn't clear what "SMART" was since I didn't see that option when booking online. My seat was seat 9A, so I wasn't entirely sure I was in the right spot, but I figured better to be on the train than on the platform, so I just went in. We started moving shortly after. As soon as I walked in, I had to walk past the train attendant pushing a beverage cart through the aisles. I made it to seat 9, but it was occupied. I asked the attendant where my seat was, and she said it was in car 1 at the "front", and motioned to the back of the train. I don't know why their cars are in the opposite order you'd expect. So I took my bags back to car 1 where I was finally greeted with the "Premium" sign I was looking for.

Premium

I was quickly able to find my seat, which was not in fact occupied. The Premium car was configured with 2 seats on one side and 1 seat on the other side.

The Brightline Premium Car

Premium Seats

Some of the seats are configured to face each other, so there is a nice variety of seating options. You could all be sitting around a table if you booked a ticket for 4 people, or you could book 2 tickets and sit either next to each other or across from each other.

Seating across

Since I had booked my ticket so last minute, I had basically the last available seat in the car so I was sitting next to someone. As soon as I sat down, the beverage cart came by with drinks. The cart looked like the same type you'd find on an airplane, and even had some identical warning stickers on it such as the "must be secured for takeoff and landing" sign. The drink options were also similar to what you'd get on a Premium Economy flight service. I opted for a glass of prosecco, and made myself comfortable.

The tray table at the seat had two configurations. You could either drop down a small flap or the whole tray.

Small tray table

Large tray table

The small tray was big enough to hold a drink or an iPad or phone, but not much else. The large tray was big enough for my laptop with a drink next to it as well as an empty glass or bottle behind it.

Under the seat there was a single power outlet for the 2 seats with 120v power as well as two USB-C ports.

Power outlets

Shortly after I had settled in, the crew came back with a snack tray and handed me these four snacks without really giving me the option of refusing any of them.

Snacks

At this point I wasn't really hungry since I had just eaten at the airport, so I stuffed the snacks in my bag, except for the prosciutto, which I offered to my seat mate but he refused.

By this point we were well on our way to the Boca Raton stop. A few people got off and on there, and we continued on. I should add here that I always feel a bit unsettled when there is that much movement of people getting on and off all the time. These stops were about 20-30 minutes away from each other, which meant the beginning of the ride I never really felt completely settled in. This is the same reason I prefer a 6 hour flight over two 3 hour flights. I like to be able to settle in and just not think about anything until we arrive.

We finally left the last of the South Florida stops, West Palm Beach, and started the rest of the trip to Orlando. A bunch of people got off at West Palm Beach, enough that the Premium cabin was nearly empty at that point. I was able to move to the seat across the aisle which was a window/aisle seat all to myself!

My own seat

Finally I could settle in for the long haul. Shortly before 3, the crew came by with the lunch cart. The options were either a vegetarian or non-vegetarian option, so that made the choice easy for me.

Lunch

The vegetarian option was a tomato basil mozzarella sandwich, a side of fruit salad, and some vegetables with hummus. The hummus was surprisingly good, not like the little plastic tubs you get at the airport. The sandwich was okay, but did have a nice pesto spread on it.

After lunch, I opened up my computer to start writing this post and worked on it for most of the rest of the trip.

As the train started making a left turn to head west, the conductor came on the loudspeaker and made an announcement along the lines of "we're about to head west onto the newest tracks that have been built in the US in 100 years. We'll be reaching 120 miles per hour, so feel free to feel smug as we whiz by the cars on the highway." And sure enough, we really picked up the speed on that stretch! While we had reached 100-120mph briefly during the trip north, that last stretch was a solid 120mph sustained for about 20 minutes!

Orlando Station

Orlando Station

We finally slowed down and pulled into the Orlando station at the airport.

Disembarking the train was simple enough. This was the last stop of the train so there wasn't quite as much of a rush to get off before the train started again. There's no need to mind the gap as you get off since there's a little platform that extends from the train car.

Don't mind the gap

At the Orlando station there was a short escalator up and then you exit through the automated gates.

Exit gates

I assumed I would have to scan my ticket when exiting but that ended up not being the case. Which actually meant that the only time my ticket was ever checked was when entering the station. I never saw anyone come through to check tickets on the train.

At this point I was already in the airport, and it was a short walk around the corner to the tram that goes directly to the airport security checkpoint.

The whole trip took 176 minutes for 210 miles, which is an average speed of 71 miles per hour. When moving, we were typically moving at anywhere from 80-120 miles per hour.

Summary

  • The whole experience was way nicer than an airplane, I would take this over a short flight from Miami to Orlando any day.
  • It felt similar to a European train, but with service closer to an airline.
  • The service needs to be better timed with the stops when people are boarding.
  • The only ticket check was when entering the station, nobody came to check my ticket or seat on the train, or even when I left the destination station.
  • While the Premium car food and drinks were free, I'm not sure it was worth the $85 extra ticket price over just buying the food you want.
  • Unfortunately the ticket cost was similar to that of budget airlines, I would have preferred the cost to be slightly lower. But even still, I would definitely take this train over a budget airline at the same cost.

We need more high speed trains in the US! I go from Portland to Seattle often enough that a train running every 90 minutes that was faster than a car and easier and more comfortable than an airplane would be so nice!

2023-10-23T09:12:55-07:00 Fullscreen Open in Tab
OAuth for Browser-Based Apps Draft 15

After a lot of discussion on the mailing list over the last few months, and after some excellent discussions at the OAuth Security Workshop, we've been working on revising the draft to provide clearer guidance and clearer discussion of the threats and consequences of the various architectural patterns in the draft.

I would like to give a huge thanks to Philippe De Ryck for stepping up to work on this draft as a co-author!

This version is a huge restructuring of the draft and now starts with a concrete description of possible threats of malicious JavaScript as well as the consequences of each. The architectural patterns have been updated to reference which of each threat is mitigated by the pattern. This restructuring should help readers make a better informed decision by being able to evaluate the risks and benefits of each solution.

https://datatracker.ietf.org/doc/html/draft-ietf-oauth-browser-based-apps

https://www.ietf.org/archive/id/draft-ietf-oauth-browser-based-apps-15.html

Please give this a read, I am confident that this is a major improvement to the draft!