2025-12-11T22:31:33+00:00 Fullscreen Open in Tab

Finished reading Mage Tank 3

Finished reading:

Mage Tank series, book 3.

Mage Tank 3

by Cornman.

Published 2025. 662 pages.

Started November 5, 2025; completed December 1, 2025.

2025-12-11T22:12:37+00:00 Fullscreen Open in Tab

Note published on December 11, 2025 at 10:12 PM UTC

I have purchased the PACER documents so everyone can read the impact statements submitted by victims of Do Kwon and the 2022 Terra/Luna meltdown. Collapses like these ruin lives.

Honorable Judge Engelmayer, My name is Stanislav Trofimchuk (Stan). My life and family were pretty much ruined by Mr. Kwon’s actions back in 2022 and I would like to share my story in writing and speak at a sentencing hearing if videoconferencing means are available as I’m unable to attend personally. Below is my story: We came to the United States in 2006 from Ukraine as a young family with a ten-month-old baby, chasing the American Dream and a brighter future for our children — both the one we had and those we still hoped to have. Like most first-generation immigrants, we had to work very hard to build a life for ourselves. For seven straight years I worked multiple jobs, endless hours, no weekends, no holidays — because my wife didn’t have a work permit during all that time. I barely saw my children grow up. She carried the burden of raising our two boys almost entirely alone, with no family nearby, no help at all. After thirteen years of struggle, we finally became U.S. citizens. For the first time I could breathe. I started a small business, came home at reasonable hours, and began repairing the years I had missed with my family. Life finally felt like it was moving forward. By early 2021 the dream was within reach: a modest house in Texas, a growing business, savings carefully set aside so our sons could go to college. After everything, we were closer than ever to “making it.”. Then, in February 2022, war exploded across Ukraine. Both my wife’s elderly mother and my parents, living in the south of the country — found themselves directly on the front line. We dropped everything. We flew to Europe, pulled them out of Ukraine, and relocated the entire family — our parents, ourselves, our children — to Croatia, one of the few places that felt safe and where we could survive on far less money than in America. In Croatia none of us had work permits. My small business in US started struggling the moment I left. I was desperate for any lifeline that could keep seven people alive. That’s when I discovered Terra, UST, and the Anchor and Mirror protocols through Do Kwon’s Twitter. He

described UST as a stablecoin that would always hold its $1 peg, and those protocols as safe, reliable way to earn 19–40% yield. It didn’t feel like gambling; it felt like the answer we had been praying for. We sold everything at discounted prices: the house, the business, the cars. We scraped together just over $190,000 — every dollar of seventeen years of sacrifice — and in late April 2022 I converted it all to UST and locked it into Anchor and Mirror, trusting the promises that had been repeated for months. Just two weeks later, on May 9, 2022, I watched UST fall off its peg. In panic, I almost sold everything to limit the damage, but then I saw Mr. Kwon’s tweet: “Deploying more capital— steady lads.”. The price started recovering that day. I trusted him again and decided to keep the UST locked. What followed were two weeks of pure terror. The Terra blockchain halted. Mirror Protocol liquidity vanished. I couldn’t withdraw a single cent. When the system finally let me out on May 23, UST was trading below $0.07. Our $190,000 had become less than $13,000. Seventeen years of our lives — gone in two weeks. People say you should never risk money you can’t afford to lose. But this never felt like risk. The founder of the project had called it “too big to fail,” boasted that the Luna Foundation Guard would protect every dollar. To a desperate father trying to feed two families in a foreign country, it sounded like the safest investment ever. The money was only the beginning of the catastrophe. Less than two weeks after the collapse, my wife packed her bags. She could no longer trust that I could keep her and her mother financially safe. A few months later the divorce papers arrived. Our teenage sons chose to stay with me and my parents, to hold what was left of the family together. College was never mentioned again. Both boys quit school and now work as car mechanics to put food on the table. Today I am [redacted]. I live in [redacted] with my elderly parents, taking whatever cash jobs I can find, waiting for a war in my home country to end. My ex-wife lives in another country, babysitting strangers’ children to survive. My sons fix cars instead of studying the engineering degrees they once dreamed of. I cannot return to the United States — there is nothing left for us there: no home, no business, no savings, no future. Everything we built over nearly two decades vanished in May 2022. I never imagined a person I had never met, never spoken to, could destroy my family so

completely. Yet that is exactly what happened. I know there are others out there whose stories are even darker than mine. I still hope that Mr. Kwon finds the conscience, the will, and the means to make restitution — so that shattered families like ours might have a chance to rebuild. Thank you for listening. Stanislav

Your Honor, In one week in 2022, Do Hyeong Kwon stole my home and my life's savings, my future retirement and hopes. These were my built-up assets made from skipping vacations and huge sacrifices over both my and my wife's careers. I am now 57 and am unlikely to ever have the opportunity to regain this money or a home. Today, I am on the verge of bankruptcy. The total amount lost was just over 1m USD. I made a mistake in my trust and belief in Do Kwon and used some of my house mortgage money to invest in his scam. After the collapse, I was forced to sell our family home. I had a belief that hard work pays off. My lifelong hard work was ripped from under my family and me due to this fraud. We no longer have a home of our own We have credit card debts accumulating We have been forced to obtain a high-interest personal loan to pay for essential medical bills. My wife cries when the credit card bills arrive I take My nights are sleepless and full of stress My daughter asks why we don't have the money to fix the car. We are on the verge of bankruptcy, which would mean I will also lose my job. Words cannot describe the devastation and hardship we continue to experience daily due to the theft committed by Do Kwon. Now we are faced with the stark reality that our hardship is likely to continue until the day we die. We have no doubt that Do Kwon will still have money hidden. We wish to see the maximum sentence imposed so that he never gains access to any of these stolen funds. We want to deter other criminals like him. Please impose the absolute MAXIMUM sentence the law allows Sincerely Daniel and Criselle Parker

to whom it may concern, my friend and i were very large luna investors based on Do's statements that the peg was restored automatically in 2021, we believed that he had a working technology and luna would have tremendous value capture as UST would be utilized in a bull or bear market i had a mid 8 figure loss in luna (yes that's correct) and my friend later jumped off a building in miami after telling his gf (its in police report) that he lost his money in crypto and was wrong this is he: https://westportlocal.com/news/kxojqtqsip3dlrr3jw7ljs5tziwdl5 failing at a new technology is part of innovation, repeated outright lies about a technology that never existed is criminal and life destroying that is all i've got for ya thanks for listening, josh

Dear Judge Engelmayer, My name is Concetto Fabrizio Di Rosa, and I am submitting this victim impact statement in connection with the sentencing of Do Hyeong Kwon. I lost approximately €40,000 as a direct result of the collapse of the Terraform ecosystem. This loss created a devastating chain of consequences for me and my family. The money I invested had been carefully saved over time to pay for my son’s planned wedding, which took place in September. When the funds disappeared, I was suddenly unable to cover those expenses. I had to borrow money from friends and relatives, causing deep personal embarrassment and significant tension within my family. The financial impact did not stop there. In desperation, I also borrowed money from my employer, hoping to recover enough to repay the loan. Because the amount was large and I was unable to meet the repayment schedule, I ultimately lost my job. Losing both my savings and my employment in such a short period created immense emotional and psychological stress. I have experienced severe anxiety, sleeplessness, and a constant sense of guilt and shame. These problems have affected my daily life and my relationships with the people I love. Even today, writing about this experience is extremely painful for me. The consequences of this financial disaster have marked my life deeply and continue to affect my family’s stability and well-being. I respectfully ask the Court to consider the profound harm caused to ordinary people like me when determining an appropriate sentence. Thank you for your attention. Sincerely, Concetto Fabrizio Di Rosa

Hello, see below Victim Statement as requested. I am writing in regards to the upcoming sentencing of Do Kwon on December 11th. I have never written anything like this or as a man and a father, I hate to set myself up to be called a

Dear Judge Engelmayer On May 11, 2022, I invested $80,000 in LUNA coin. I received this money from the sale of my only apartment. I was moving from Russia (Moscow) to Georgia and planned to change my permanent residence. I am enclosing you with the documents confirming the sale of my only apartment. A day after my investment in LUNA coin, my sold apartment turned into several coins. Specifically, $80,000 turned into $13. I expected Do Hyun Kwon to make rational decisions after the LUNA coin's collapse. Instead, he created LUNA2, bearing no responsibility to investors in the original LUNA coin (now LUNC). Since May 2022, I have been homeless, wandering the streets of Georgia, with no money to rent a place to live. I have developed health complications and am in dire need of hospitalization, but I do not have the financial means to seek medical treatment. I am a single 58-year-old woman, unemployed, and without a pension. My family-my father, mother, and husband-passed away several years earlier. I have no children. I have no recourse. I ask for an immediate, fair decision in Do Hyeon Kwon's case and request full compensation for my financial losses, including any moral damages. I have provided, submitted, and attached all supporting documents to the portal. Since May 2022, I have been in a state of deep stress and depression, which I would not wish on anyone. I would like to testify in court, but unfortunately, I do not speak English. I can testify remotely (in Russian, if possible, with interpreters available in court). I am a tragic victim of the LUNA coin crash on May 11, 2022. My claim number is Best Regards Tatiana Dontsova, /Georgia, the city of Kobuleti/

Dear Judge Engelmayer, I am writing to inform the court of the impact Do Kwon had on my family and myself during and after the collapse of Terra. I had never invested serious money prior to Luna but after being educated on the Terra Luna ecosystem by a family member, I was completely on board with the stability and efficiency of the blockchain. I invested countless hours of my time listening to podcasts and watching videos learning how to make the most of my investments. The more time I invested the more money I invested. I invested in new protocols on the system and my days were consumed with all things Terra. I shared my knowledge with friends and family, wanting them to also be part of something great. I was convinced that I was on the ground level of something big and that I would be providing generational wealth for my family. I ended up withdrawing over 100,000 from my 401k and paying the early withdrawal penalty fees because I believed in the long term success of Terra. I never took any profits on the money I had invested. I lost everything in the collapse. When the collapse was happening I still had faith in Luna and was actually still trying to purchase more Luna. I am still in debt over the poor decisions I made during that time, due to my beliefs in the greatness of the ecosystem. The monetary loss was painful but the physical and mental anguish that followed was much worse. It took a few years to recover from the guilt, embarrassment, anxiety, and the sick to my stomach feeling I constantly had. I was very depressed. I try not to think about it anymore but knowing in the back of my head that I no longer have a retirement account can still bring back all those negative emotions. I cannot change my past decisions to invest in this cryptocurrency but I do hope that Do Kwon will realize the amount of pain and loss he caused to so many. Thank you for your time, Marie Locascio

Hi, i was a victim of Do Kwon’s terra luna scam. I am a carer for my Dad, [redacted] and i invested money into terra luna believing that it was a profitable strategy to support my Dad better with medical expenses and being able to sacrifice taking time off work to support him. When i lost all of my funds in this scam i was devastated. I had to take out loans to get by to fund my Dad’s medical bills and just to get by with food and necessities for us both. I am still in debt to this day paying back loans and credit cards. My family has been through immense suffering and i suffer from depression daily since the losses of my life savings. I urge the honourable judge to please hand down harse sentencing to Do Kwon for the crimes that he has committed, i sincerely hope that justice will prevail to recieve some sollace from the torment he has put my family and i through. Kind regards Paul Lynn

Dear Judge Engelmayer, Your Honor, I stand here today not just as a victim of financial fraud, but as a daughter whose world was irrevocably shattered. I ask you to look beyond the cold, hard numbers (i.e., the staggering sum of money that vanished) and see the human cost and lifelong agony that UST's depegging has caused. I am asking you to read these words not with professional detachment, but with the empathy I pray resides within the heart of any person witnessing such profound devastation. I remember the exact moment. It was May 2022, several days after my birthday. Suddenly, I saw a notification flash across my screen: UST had depegged from its $1 parity. In an instant, the value of everything I had worked for, everything my father had saved for, just obliterated into nothing. The shock was a physical blow. I felt my breath leave my body. Your Honor, I didn't think about my own savings at first. The first thought, the only thought, was of my father. And that is what almost drove me to end my own life. I did not consider taking my life over my own money loss, but over the catastrophic failure I brought upon my father...the man who deserved the world. He put $100k of his retirement savings into Anchor Protocol (due to my advice)...and now he is over 70 years old and he cannot fully retire. Because if he stops working, he can no longer afford to live. My father is a good man. A diligent man. For over sixty years, he worked tirelessly, building a modest nest egg for himself and my mother. He's an accountant by trade, a man of integrity, but he always yearned for more for his family. He tried to launch businesses (e.g., a laundromat franchise), but fate, or perhaps just cruel luck, always intervened. I remember when he entrusted a

My father, who had suffered a string of failed businesses, who carried the weight of a mortgage that pains him every single month. All I ever wanted, Your Honor, was to become successful enough to spoil him. To pay off his debt. To give him the happy retirement he earned and deserved. When I saw the depegging, that vision of a happy future was crushed. I felt the absolute, sickening certainty that I had failed him. I had delivered his lifelong savings into the hands of a scam. My father lost about $70,000 from his $100k savings. That number, $70,000, represents decades of his life, wiped out in the span of a single week! I wanted to crush myself into a ball, to vanish, to take away the pain I had caused him. Years later, the suffering continues. I am 35 years old now, engaged to be married, but I cannot afford the down payment on a home. I cannot afford a wedding. I cannot afford children. As much as it hurts me, I've come to terms with the fact that I can't financially support children nor own my own home. My father is continuing to freelance as an accountant. He's always fantasized about retiring in his 60s, but now that's a distant dream. And what's most miraculous, Your Honor, is that my father has never yelled at me or anything of the sort as someone who advised him about Anchor Protocol. He never blamed me for anything. He never complained. And oddly, that makes me feel even worse. He's just taking things day by day, and somehow, he still loves me ... and finds it in his heart to smile through the pain. His attitude is what kept me alive, Your Honor. If he lost $70k and he's still smiling (despite not having enough years left to recoup that money with hard work), I can lose over $30k and still be alright. Like all the other victims, I seek closure from this nightmare. I do not know exactly what that looks like. But perhaps Do Kwon's imprisonment may help bring some semblance of closure...an acknowledgment from the justice system that the harm done to families like mine was real Respectfully, Kimberly Gedeon Valley Stream, New York

Thu, 11 Dec 2025 16:13:02 +0000 Fullscreen Open in Tab

Pluralistic: Instacart reaches into your pocket and lops a third off your dollars (11 Dec 2025)

Today's links

Instacart reaches into your pocket and lops a third off your dollars: A/B splitting your way into doing a total fucking racism.
Hey look at this: Delights to delectate.
Object permanence: Predicting the present; Caller eye-deer; RIP Robert Sheckley; Student protesters and Google Maps vs kettling; If your kids like computers, they're criminals; A billion CC licenses; The moral character of cryptographic work; EC resurrects link-taxes.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

Instacart reaches into your pocket and lops a third off your dollars (permalink)

There's a whole greedflation-denial cottage industry that insists that rising prices are either the result of unknowable, untameable and mysterious economic forces, or they're the result of workers having too much money and too many jobs.

The one thing we're absolutely not allowed to talk about is the fact that CEOs keep going on earnings calls to announce that they are hiking prices way ahead of any increase in their costs, and blaming inflation:

https://pluralistic.net/2021/11/20/quiet-part-out-loud/#profiteering

Nor are we supposed to notice the "price consultancies" that let the dominant firms in many sectors – from potatoes to meat to rental housing – fix prices in illegal collusive arrangements that are figleafed by the tissue-thin excuse that "if you use an app to fix prices, it's not a crime":

https://pluralistic.net/2025/01/25/potatotrac/#carbo-loading

And we're especially not supposed to notice the proliferation of "personalized pricing" businesses that use surveillance data to figure out how desperate you are and charge you a premium based on that desperation:

https://pluralistic.net/2024/06/05/your-price-named/#privacy-first-again

Surveillance pricing – when you are charged more for the same goods than someone else, based on surveillance data about the urgency of your need and the cash in your bank account – is a way for companies to reach into your pocket and devalue the dollars in your wallet. After all, if you pay $2 for something that I pay $1 for, that's just the company saying that your dollars are only worth half as much as mine:

https://pluralistic.net/2025/06/24/price-discrimination/

It's a form of cod-Marxism: "from each according to their desperation":

https://pluralistic.net/2025/01/11/socialism-for-the-wealthy/#rugged-individualism-for-the-poor

The economy is riddled with surveillance pricing gouging. You are almost certainly paying more than your neighbors for various items, based on algorithmic price-setting, every day. Case in point: More Perfect Union and Groundwork Collaborative teamed up with Consumer Reports to recruit 437 volunteers from across America to login to Instacart at the same time and buy the same items from 15 stores, and found evidence of surveillance pricing at Albertsons, Costco, Kroger, and Sprouts Farmers Market:

https://groundworkcollaborative.org/work/instacart/

The price-swings are wild. Some test subjects are being charged 23% more than others. The average variance for "the exact same items, from the exact same locations, at the exact same time" comes out to 7%, or "$1,200 per year for groceries" for a family of four.

The process by which your greedflation premium is assigned is opaque. The researchers found that Instacart shoppers ordering from Target clustered into seven groups, but it's not clear how Instacart decides how much extra to charge any given shopper.

Instacart – who acquired Eversight, a surveillance pricing company, in 2022 – blamed the merchants (who, in turn, blamed Instacart). Instacart also claimed that they didn't use surveillance data to price goods, but hedged, admitting that the consumer packaged goods duopoly of Unilever and Procter & Gamble do use surveillance data in connection with their pricing strategies.

Finally, Instacart claimed that this was all an "experiment" to "learn what matters most to consumers and how to keep essential items affordable." In other words, they were secretly charging you more (for things like eggs and bread) because somehow that lets them "keep essential items affordable."

Instacart said their goal was to help "retail partners understand consumer preferences and identify categories where they should invest in lower prices."

Anyone who's done online analytics can easily pierce this obfuscation, but for those of you who haven't had the misfortune of directing an iterated, A/B tested optimization effort, I'll unpack this statement.

Say you have a pool of users and a bunch of variations on a headline. You randomly assign different variants to different users and measure clickthroughs. Then you check to see which variants performed best, and dig into the data you have on those users to see if there are any correlations that tie together users who liked a given approach.

This might let you discover that, say, women over 40 click more often on headlines that mention kittens. Then you generate more variations based on these conclusions – different ways of mentioning kittens – and see which of these variations perform best, and whether the targeted group of users split into smaller subgroups (women over 40 in the midwest prefer "tabby kitten" while their southern sisters prefer "kitten" without a mention of breed).

By repeatedly iterating over these steps, you can come up with many highly refined variants, and you can use surveillance data to target them to ever narrower, more optimized slices of your user-base.

Obviously, this is very labor intensive. You have to do a lot of tedious analysis, and generate a lot of variants. This is one of the reasons that slopvertising is so exciting to the worst people on earth: they imagine that they can use AI to create a self-licking ice-cream cone, performing the analysis and generating endless new variations, all untouched by human hands.

But when it comes to prices, it's much easier to produce variants – all you're doing is adding or subtracting from the price you show to shoppers. You don't need to get the writing team together to come up with new ways of mentioning kittens in a headline – you can just raise the price from $6.23 to $6.45 and see if midwestern women over 40 balk or add the item to their shopping baskets.

And here's the kicker: you don't need to select by gender, racial or economic criteria to end up with a super-racist and exploitative arrangement. That's because race, gender and socioeconomic status have broad correlates that are easily discoverable through automated means.

For example, thanks to generations of redlining, discriminatory housing policy, wage discrimination and environmental racism, the poorest, sickest neighborhoods in the country are also the most racialized and are also most likely to be "food deserts" where you can't just go to the grocery store and shop for your family.

What's more, the private equity-backed dollar store duopoly have waged a decades-long war on community grocery stores, enveloping them with dollar stores that use their access to preferential discounts (from companies like Unilever and Procter & Gamble, another duopoly) to force grocers out of business:

https://pluralistic.net/2023/03/27/walmarts-jackals/#cheater-sizes

Then these dollar stores run a greedflation scam that is so primitive, it's almost laughable: they just charge customers much higher amounts than the prices shown on the shelves and price-tags:

https://www.consumeraffairs.com/news/do-all-those-low-dollar-store-prices-really-add-up-120325.html

When you live in a food desert where your only store is a Dollar General that defrauds you at the cash-register, you are more likely to accept a higher price from Instacart, because you have fewer choices than someone in a middle-class neighborhood with two or three competing grocers. And the people who live in those food deserts are more likely to be poor, which, in America, is an excellent predictor of whether they are Black or brown.

Which is to say, without ever saying, "Charge Black people more for groceries," Instacart can easily A/B split its way into a system where they predictably and reliably charge Black people more for groceries. That's the old cod-Marxism at work: "from each according to their desperation."

This is so well-understood that anyone who sets one of these systems in motion should be understood to be deliberately seeking to do racist profiteering under cover of an algorithm. It's empiricism-washing: "I'm not racist, I just did some math" (that produced a predictably racist outcome):

https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK0AG/

This is the dark side and true meaning of "business optimization." The optimal business pays its suppliers and workers nothing, and charges its customers everything it can. Obviously, businesses need to settle for suboptimal outcomes, because workers won't show up if they don't get paid, and customers won't buy things that cost everything they have⹋.

⹋ Unless, of course, you are an academic publisher, in which case this is just how you do business.

A business "optimizes" its workforce by finding ways to get them to accept lower wages. For example, they can bind their workers with noncompete "agreements" that ban Wendy's cashiers from quitting their job and making $0.25 more per hour at the McDonald's next door (one in 18 American workers have been locked into one of these contracts):

https://pluralistic.net/2025/09/09/germanium-valley/#i-cant-quit-you

Or they can lock their workers in with "training repayment agreement provisions" (TRAPs) – contractual clauses that force workers to pay their bosses thousands of dollars if they quit or get fired:

https://pluralistic.net/2022/08/04/its-a-trap/#a-little-on-the-nose

But the most insidious form of worker optimization is "algorithmic wage discrimination." That's when a company uses surveillance data to lower the wages of workers. For example, contract nurses are paid less if the app that hires them discovers (through the unregulated data-broker sector) that they have a lot of credit-card debt. After all, nurses who are heavily indebted can't afford to be choosy and turn down lowball offers:

https://pluralistic.net/2024/12/18/loose-flapping-ends/#luigi-has-a-point

This is the other form of surveillance pricing: pricing labor based on surveillance data. It's more cod-Marxism: "From each according to their desperation."

Forget "becoming ungovernable": to defeat these evil fuckers, we have to become unoptimizable:

https://pluralistic.net/2025/08/20/billionaireism/#surveillance-infantalism

How do we do that? Well, nearly every form of "optimization" begins with surveillance. They can't figure out whether they can charge you more if they can't spy on you. They can't figure out whether they can pay you less if they can't spy on you, either.

And the reason they can spy on you is because we let them. The last consumer privacy law to pass out of Congress was a 1988 bill that bans video-store clerks from disclosing your VHS rental history. Every other form of consumer surveillance is permitted under US federal law.

So step one of this process is to ban commercial surveillance. Banning algorithmic price discrimination is all well and good, but it is, ultimately, a form of redistribution. We're trying to make the companies share some of the excess they extract from our surveillance data. But predistribution – ending surveillance itself, in this case – is always far more effective than redistribution:

https://pluralistic.net/2025/10/31/losing-the-crypto-wars/#surveillance-monopolism

How do we do that? Well, we need to build a coalition. At the Electronic Frontier Foundation, we call this "privacy first": you can't solve all the internet's problems by fixing privacy, but you won't fix most of them unless we get privacy right, and so the (potential) coalition for a strong privacy regime is large and powerful:

https://pluralistic.net/2023/12/06/privacy-first/#but-not-just-privacy

But of course, "privacy first," doesn't mean "just privacy." We also need tools that target algorithmic pricing per se. In New York State, there's a new law that requires disclosure of algorithmic pricing, in the form of a prominent notification reading, "THIS PRICE WAS SET BY AN ALGORITHM USING YOUR PERSONAL DATA."

This is extremely weaksauce, and might even be worse than nothing. In California we have Prop 65, a rule that requires businesses to post signs and add labels any time they expose you to chemicals "known to the state of California to cause cancer." This caveat emptor approach (warn people, let them vote with their wallets) has led to every corner of California's built environment to be festooned with these warnings. Today, Californians just ignore these warnings, the same way that web users ignore the "privacy policy" disclosures on the sites they visit:

https://pluralistic.net/2025/04/19/gotcha/#known-to-the-state-of-california-to-cause-cancer

The right approach isn't to (merely) warn people about carcinogens (or privacy risks). The right approach is regulating harmful business practices, whether those practices give you a tumor or pick your pocket.

Under Biden, former FTC chair Lina Khan undertook proceedings to ban algorithmic pricing altogether. Trump's FTC killed that, along with all the other quality-of-life enhancing measures the FTC had in train (Trump's FTC chair replaced these with a program to root out "wokeness" in the agency).

Today, Khan is co-chair of Zohran Mamdani's transition team, and she will use the mayor's authority (under the New York City Consumer Protection Law of 1969, which addresses "unconscionable" commercial practices) to ban algorithmic pricing in NYC:

https://pluralistic.net/2025/11/15/unconscionability/#standalone-authority

Khan wasn't Biden's only de-optimizer. Under chair Rohit Chopra, Biden's Consumer Finance Protection Bureau actually banned the data-brokers who power surveillance pricing:

https://pluralistic.net/2023/08/16/the-second-best-time-is-now/#the-point-of-a-system-is-what-it-does

And of course, Trump's CFPB (neutered by Musk and his broccoli-haired brownshirts at DOGE) killed that effort:

https://pluralistic.net/2025/05/15/asshole-to-appetite/#ssn-for-sale

But the CFPB staffer who ran that effort has gone to work on an effort to leverage a New Jersey state privacy law to crush the data-broker industry:

https://www.wired.com/story/daniels-law-new-jersey-online-privacy-matt-adkisson-atlas-lawsuits/

These are efforts to optimize corporations for human thriving, by making them charge us less and pay us more. For while we are best off when we are unoptimizable, we are also best off when corporations are totally optimized – for our benefit.

(Image: Cryteria, CC BY 3.0, modified)

Hey look at this (permalink)

Uber is selling your ride and food ordering data to advertisers for marketing insights https://boingboing.net/2025/12/08/uber-is-selling-your-ride-and-food-ordering-data-to-advertisers-for-marketing-insights.html
404 Media Is Making a Zine https://www.404media.co/404-media-is-making-a-zine/
Maybe a General Strike Isn’t So Impossible Now https://labornotes.org/2025/12/maybe-general-strike-isnt-so-impossible-now
The Naibbe cipher: a substitution cipher that encrypts Latin and Italian as Voynich Manuscript-like ciphertext https://www.tandfonline.com/doi/full/10.1080/01611194.2025.2566408
Bringing organizational maturity to radical groups https://blog.bl00cyb.org/2025/12/bringing-organizational-maturity-to-radical-groups/

Object permanence (permalink)

#20yrsago Free voicemail helps homeless people get jobs https://web.archive.org/web/20051210021850/http://www.cvm.org/

#20yrsago Anti-P2P company decides to focus on selling music instead https://de.advfn.com/borse/NASDAQ/LOUD/nachrichten/13465769/loudeye-to-exit-content-protection-services-busine

#20yrsago Caller Eye-Deer’s eyes glow when phone rings https://www.flickr.com/photos/84221353@N00/71889050/in/pool-69453349@N00

#20yrsago EFF to Sunncomm: release a list of all infected CDs! https://web.archive.org/web/20051212072537/https://www.eff.org/deeplinks/archives/004245.php

#20yrsago Only 2% of music-store downloaders care about legality of their music https://web.archive.org/web/20051225200658/http://www.mp3newswire.net/stories/5002/tempo2005.html

#20yrsago Dykes on Bikes gives the Trademark Office a linguistics lesson https://web.archive.org/web/20060523133217/https://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2005/12/09/MNGQOG5D7P1.DTL&type=printable

#20yrsago Robert Sheckley has died https://nielsenhayden.com/makinglight/archives/007078.html

#20yrsago Xbox 360 DRM makes your rip your CDs again https://www.gamespot.com/articles/microsoft-xbox-360-hands-on-report/1100-6139672/

#20yrsago Music publishers: Jail for lyric-sites http://news.bbc.co.uk/2/hi/entertainment/4508158.stm

#15yrsago 2600 Magazine condemns DDoS attacks against Wikileaks censors https://web.archive.org/web/20101210213130/https://www.2600.com/news/view/article/12037

#15yrsago UK supergroup records 4’33”, hopes to top Xmas charts https://www.theguardian.com/music/2010/dec/06/cage-against-machine-x-factor

#15yrsago FarmVille’s secret: making you anxious https://web.archive.org/web/20101211120105/http://www.gamasutra.com/view/feature/6224/catching_up_with_jonathan_blow.php?print=1

#15yrsago Rogue Archivist beer https://web.archive.org/web/20101214060929/https://livingproofbrewcast.com/2010/12/giving-the-rogue-archivist-to-its-namesake/

#15yrsago Hossein “Hoder” Derakhshan temporarily released from Iranian prison https://cyrusfarivar.com/blog/2010/12/09/iranian-blogging-pioneer-temporarily-released-from-prison/

#15yrsago Student protesters in London use Google Maps to outwit police “kettling” https://web.archive.org/web/20101212042006/https://bengoldacre.posterous.com/student-protestors-using-live-tech-to-outwit

#15yrsago Google foreclosure maps https://web.archive.org/web/20170412162114/http://ritholtz.com/2010/12/google-map-foreclosures/
#15yrsago Theory and practice of queue design https://passport2dreams.blogspot.com/2010/12/third-queue.html

#15yrsago Legal analysis of the problems of superherodom https://lawandthemultiverse.com/

#10yrsago A great, low-tech hack for teaching high-tech skills https://miriamposner.com/blog/a-better-way-to-teach-technical-skills-to-a-group/

#10yrsago In case you were wondering, there’s no reason to squirt coffee up your ass https://scienceblogs.com/insolence/2015/12/10/starbutts-or-how-is-it-still-a-thing-that-people-are-shooting-coffee-up-their-nether-regions

#10yrsago Survey of wealthy customers leads insurer to offer “troll insurance” https://www.telegraph.co.uk/finance/newsbysector/banksandfinance/insurance/12041832/Troll-insurance-to-cover-the-cost-of-internet-bullying.html

#10yrsago US State Department staffer sexually blackmailed women while working at US embassy https://web.archive.org/web/20151210230259/https://www.networkworld.com/article/3013633/security/ex-us-state-dept-worker-pleads-guilty-to-extensive-sextortion-hacking-and-cyberstalking-acts.html

#10yrsago Robert Silverberg’s government-funded guide to the psychoactive drugs of sf https://web.archive.org/web/20151211050648/https://motherboard.vice.com/read/the-us-government-funded-an-investigation-into-sci-fi-drug-use-in-the-70s

#10yrsago Toy demands that kids catch crickets and stuff them into an electronic car https://www.wired.com/2015/12/um-so-the-bug-racer-is-an-actual-toy-car-driven-by-crickets/

#10yrsago The crypto explainer you should send to your boss (and the FBI) https://web.archive.org/web/20151209011457/https://www.washingtonpost.com/news/the-switch/wp/2015/12/08/you-already-use-encryption-heres-what-you-need-to-know-about-it/

#10yrsago French PM defies Ministry of Interior, says he won’t ban open wifi or Tor https://web.archive.org/web/20160726031106/https://www.connexionfrance.com/Wifi-internet-ban-banned-17518-view-article.html

#10yrsago The no-fly list really is a no-brainer https://www.theguardian.com/us-news/2015/dec/09/no-fly-list-errors-gun-control-obama

#10yrsago America: shrinking middle class, growing poverty, the rich are getting richer https://www.pewresearch.org/social-trends/2015/12/09/the-american-middle-class-is-losing-ground/

#10yrsago Marriott removing desks from its hotel rooms “because Millennials” https://web.archive.org/web/20151210034312/http://danwetzelsports.tumblr.com/post/134754150507/who-stole-the-desk-from-my-hotel-room

#10yrsago China’s top Internet censor: “There’s no Internet censorship in China” https://hongkongfp.com/2015/12/09/there-is-no-internet-censorship-in-china-says-chinas-top-censor/

#10yrsago Stolen-card crime sites use “cop detection” algorithms to flag purchases https://krebsonsecurity.com/2015/12/when-undercover-credit-card-buys-go-bad/

#10yrsago UK National Crime Agency: if your kids like computers, they’re probably criminals https://www.youtube.com/watch?v=DjYrxzSe3DU

#10yrsago US immigration law: so f’ed up that Trump’s no-Muslim plan would be constitutional https://www.nytimes.com/2015/12/10/opinion/trumps-anti-muslim-plan-is-awful-and-constitutional.html?_r=0

#10yrsago Ecuador’s draft copyright law: legal to break DRM to achieve fair use https://medium.com/@AndresDelgadoEC/big-achievement-for-creative-commons-in-ecuador-national-assembly-decides-that-fair-use-trumps-drm-c8cdd9c57e01#.n1vkccd3r

#10yrsago One billion Creative Commons licenses in use https://stateof.creativecommons.org/2015/

#10yrsago The moral character of cryptographic work https://web.cs.ucdavis.edu/~rogaway/papers/moral-fn.pdf

#10yrsago Everybody knows: FBI won’t confirm or deny buying cyberweapons from Hacking Team https://web.archive.org/web/20151209163839/https://motherboard.vice.com/read/the-fbi-wont-confirm-or-deny-buying-hacking-team-spyware-even-though-it-did

#10yrsago European Commission resurrects an unkillable stupid: the link tax https://web.archive.org/web/20160913095014/https://openmedia.org/en/bad-idea-just-got-worse-how-todays-european-copyright-plans-will-damage-internet

#5yrsago Why we can't have nice things https://pluralistic.net/2020/12/10/borked/#bribery

#5yrsago Facebook vs Robert Bork https://pluralistic.net/2020/12/10/borked/#zucked

#1yrago Tech's benevolent-dictator-for-life to authoritarian pipeline https://pluralistic.net/2024/12/10/bdfl/#high-on-your-own-supply

#1yrago Predicting the present https://pluralistic.net/2024/12/09/radicalized/#deny-defend-depose

Upcoming appearances (permalink)

Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html
Denver: Enshittification at Tattered Cover Colfax, Jan 22
https://www.eventbrite.com/e/cory-doctorow-live-at-tattered-cover-colfax-tickets-1976644174937
Colorado Springs: Guest of Honor at COSine, Jan 23-25
https://www.firstfridayfandom.org/cosine/

Recent appearances (permalink)

How to Stop “Ensh*ttification” Before It Kills the Internet (Capitalisn't)
https://www.youtube.com/watch?v=34gkIvYiHxU
Enshittification on The Daily Show
https://www.youtube.com/watch?v=d2e-c9SF5nE
Enshittification with Four Ways to Change the World (Channel 4)
https://www.youtube.com/watch?v=tZQaEeuuI3Q
The Plan is to Make the Internet Worse. Forever. (Novarra Media)
https://www.youtube.com/watch?v=7wE8G-d7SnY
Enshittification (Future Knowledge)
https://futureknowledge.transistor.fm/episodes/enshittification

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

2025-12-10T21:25:29+00:00 Fullscreen Open in Tab

Read "The Reverse-Centaur's Guide to Criticizing AI"

Read:

“The Reverse-Centaur's Guide to Criticizing AI”.

Cory Doctorow in Pluralistic. Published December 5, 2025.

Tue, 09 Dec 2025 11:01:50 +0000 Fullscreen Open in Tab

Pluralistic: Big Tech joins the race to build the world's heaviest airplane (09 Dec 2025)

Today's links

Big Tech joins the race to build the world's heaviest airplane: Die as Microsoft, or live to become the IBM you overthrew.
Hey look at this: Delights to delectate.
Object permanence: Bean-sprouting keyboard; Ink rant; FBI wanted to deport John Lennon; "Concrete Park"; Plutocratic lane-changes.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

Big Tech joins the race to build the world's heaviest airplane (permalink)

I have a weird fascination with early-stage Bill Gates, after his mother convinced a pal of hers – chairman of IBM's board of directors – to give her son the contract to provide the operating system for the new IBM PC. Gates and his pal Paul Allen tricked another programmer into selling them the rights to DOS, which they sold to IBM, setting Microsoft on the path to be one of the most profitable businesses in human history.

IBM could have made its own OS, of course. They were just afraid to, because they'd just narrowly squeaked out of a 12-year antitrust war with the Department of Justice (evocatively memorialized as "Antitrust's Vietnam"):

https://pluralistic.net/2022/10/02/the-true-genius-of-tech-leaders/

The US government traumatized IBM so badly that they turned over their crown jewels to these two prep-school kids, who scammed a pal out of his operating system for $50k and made billions from it. Despite owing his business to IBM (or perhaps because of this fact), Gates routinely mocked IBM as a lumbering dinosaur that was headed for history's scrapheap. He was particularly scornful of IBM's software development methodology, which, to be fair, was pretty terrible: IBM paid programmers by the line of code. Gates called this "the race to build the world's heaviest airplane."

After all, judging software by lines of code is a terrible idea. To the extent that "number of lines of code" has any correlation with software quality, reliability or performance, it has a negative correlation. While it's certainly possible to write software with too few lines of code (e.g. when instructions are stacked on a single line, obfuscating its functionality and making it hard to maintain), it's far more common for programmers to use too many steps to solve a problem. The ideal software is just right: verbose enough to be legible to future maintainers, streamlined enough to omit redundancies.

This is broadly true of many products, and not just airplanes. Office memos should be long enough to be clear, but no longer. Home insulation should be sufficient to maintain the internal temperature, but no more.

Ironically, enterprise tech companies' bread and butter is selling exactly this kind of qualitative measurements for bosses who want an easy, numeric way to decide which of their workers to fire, and leading the pack is Microsoft, whose flagship Office365 lets bosses assess their workers' performance on meaningless metrics like how many words they type, ranking each worker against other workers within the division, with rival divisions and within rival firms. Yes, Microsoft actually boasts to companies about the fact that if you use their products, they will gather sensitive data about how your workers perform individually and as a team, and share that information with your competitors!

https://pluralistic.net/2020/11/25/the-peoples-amazon/#clippys-revenge

But while tech companies employed programmers to develop this kind of bossware to be used on other companies' employees, they were loathe to apply them to their own workers. For one thing, it's just a very stupid way to manage a workforce, as Bill Gates himself would be the first to tell you (candidly, provided he wasn't trying to sell you an enterprise Office 365 license). For another, tech workers wouldn't stand for it. After all, these were the "princes of labor," each adding a million dollars or more to their boss's bottom line, and in such scarce supply that a coder could quit a job after the morning scrum and have a new one by the pre-dinner pickleball break:

https://pluralistic.net/2025/04/27/some-animals/#are-more-equal-than-others

Tech workers mistook the fear this dynamic instilled in their bosses for respect. They thought the reason their bosses gave them free massage therapists and kombucha on tap and a gourmet cafeteria was that their bosses liked them. After all, these bosses were all techies. A coder wasn't a worker, they were a temporarily embarrassed founder. That's why Zuck and Sergey tuned into those engineering town hall meetings and tolerated being pelted with impertinent questions about the company's technology and business strategy.

Actually, tech bosses didn't like tech workers. They didn't see them as peers. They saw them as workers. Problem workers, at that. Problems to be solved.

And wouldn't you know it, supply caught up with demand and tech companies instituted a program of mass layoffs. When Google laid off 12,000 workers (just before a $80b stock buyback that would have paid their wages for 27 years), they calmed investors by claiming that they weren't doing this because business was bad – they were just correcting some pandemic-era overhiring. But Google didn't just fire junior programmers – they targeted some of their most senior (and thus mouthiest and highest-paid) techies for the chop.

Today, Sergey and Zuck no longer attend engineering meetings ("Not a good use of my time" -M. Zuckerberg). Tech workers are getting laid off at the rate of naughts. And none of these bastards can shut up about how many programmers they plan on replacing with AI:

https://pluralistic.net/2025/08/05/ex-princes-of-labor/#hyper-criti-hype

And wouldn't you know it, the shitty monitoring and ranking technology that programmers made to be used on other workers is finally being used on them:

https://jonready.com/blog/posts/everyone-in-seattle-hates-ai.html

Naturally, the excuse is monitoring AI usage. Microsoft – along with all the other AI-peddling tech companies – keep claiming that their workers adore using AI to write software, but somehow, also have to monitor workers so they can figure out which ones to fire because they're not using AI enough:

https://www.itpro.com/software/development/microsoft-claims-ai-is-augmenting-developers-rather-than-replacing-them

This is the "shitty technology adoption curve" in action. When you have a terrible, destructive technology, you can't just deploy it on privileged people who get taken seriously in policy circles. You start with people at the bottom of the privilege gradient: prisoners, mental patients, asylum-seekers. Then, you work your way up the curve – kids, gig workers, blue collar workers, pink collar workers. Eventually, it comes for all of us:

https://pluralistic.net/2021/02/24/gwb-rumsfeld-monsters/#bossware

As Ed Zitron writes, tech hasn't had a big, successful product (on the scale of, say, the browser or the smartphone) in more than a decade. Tech companies have seemingly run out of new trillion-dollar industries to spawn. Tech bosses are pulling out all the stops to make their companies seem as dynamic and profitable as they were in tech's heyday.

Firing workers and blaming it on AI lets tech bosses transform a story that would freak out investors ("Our business is flagging and we had to fire a bunch of valuable techies") into one that will shake loose fresh billions in capital ("Our AI product is so powerful it let us fire a zillion workers!").

And for tech bosses, mass layoffs offer another, critical advantage: pauperizing those princes of labor, so that they can shed their company gyms and luxury commuter busses, cut wages and benefits, and generally reset the working expectations of the tech workers who sit behind a keyboard to match the expectations of tech workers who assemble iPhones, drive delivery vans, and pack boxes in warehouses.

For tech workers who currently don't have a pee bottle or a suicide net at their job-site, it's long past time to get over this founder-in-waiting bullshit and get organized. Recognize that you're a worker, and that workers' only real source of power isn't ephemeral scarcity, it's durable solidarity:

https://techworkerscoalition.org/

(Image: Cryteria, CC BY 3.0, modified)

Hey look at this (permalink)

Your Data Might Determine How Much You Pay for Eggs https://www.wired.com/story/algorithmic-pricing-eggs-ny-law/
Judge hints Vizio TV buyers may have rights to source code licensed under GPL https://www.theregister.com/2025/12/05/vizio_gpl_source_code_ruling/
Chamberlain blocks smart home integrations with its garage door openers — again https://www.theverge.com/tech/839294/chamberlain-myq-garage-door-opener-update-blocks-aftermarket-controllers
Smart Garage Door Opener https://3reality.com/product/smartgarage-door-opener/
The Best Books in eBooks and Audiobooks of 2025 https://www.kobo.com/us/en/p/best-books-of-2025

Object permanence (permalink)

#20yrsago WaWa Digital Cameras threatens to break customer’s neck https://thomashawk.com/2005/12/abusive-new-york-camera-store.html

#20yrsago Keyboard used as bean-sprouting medium https://web.archive.org/web/20051205011830/http://www.nada.kth.se/~hjorth/krasse/english.html

#15yrsago Judge to copyright troll: get lost https://torrentfreak.com/acslaw-take-alleged-file-sharers-to-court-but-fail-on-a-grand-scale-101209/

#15yrsago Ink cartridge rant https://web.archive.org/web/20101211080931/http://www.inkcartridges.uk.com/Remanufactured-HP-300-CC640EE-Black.html

#15yrsago 1.1 billion US$100 notes out of circulation due to printing error https://www.cnbc.com/2010/12/07/the-fed-has-a-110-billion-problem-with-new-benjamins.html

#15yrsago EFF wants Righthaven to pay for its own ass-kicking https://web.archive.org/web/20101211011932/https://www.wired.com/threatlevel/2010/12/payup-troll/

#15yrsago danah boyd explains email sabbaticals https://www.zephoria.org/thoughts/archives/2010/12/08/i-am-offline-on-email-sabbatical-from-december-9-january-12.html

#15yrsago TSA subjects India’s US ambassador to public grope because of her sari https://web.archive.org/web/20101211113821/http://travel.usatoday.com/flights/post/2010/12/india-diplomat-gets-humiliating-pat-down-at-mississippi-airport-/134197/5?csp=outbrain&csp=obnetwork

#15yrsago California’s safety codes are now open source! https://code.google.com/archive/p/title24/

#10yrsago When the INS tried to deport John Lennon, the FBI pitched in to help https://www.muckrock.com/news/archives/2015/dec/08/john-lennons-fbi-file-1/

#10yrsago The Big List of What’s Wrong with the TPP https://www.eff.org/deeplinks/2015/12/how-tpp-will-affect-you-and-your-digital-rights

#10yrsago Concrete Park: apocalyptic, afrofuturistic graphic novel of greatness https://memex.craphound.com/2015/12/08/concrete-park-apocalyptic-afrofuturistic-graphic-novel-of-greatness/

#10yrsago Denmark’s top anti-piracy law firm pocketed $25m from rightsholders, then went bankrupt https://torrentfreak.com/anti-piracy-lawyer-milked-copyright-holders-for-millions-151208/

#5yrsago Uber pays to get rid of its self-driving cars https://pluralistic.net/2020/12/08/required-reading/#goober

#5yrsago All the books I reviewed in 2020 https://pluralistic.net/2020/12/08/required-reading/#recommended-reading

#5yrsago Ford patents plutocratic lane-changes https://pluralistic.net/2020/12/08/required-reading/#walkaway

Upcoming appearances (permalink)

Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html
Denver: Enshittification at Tattered Cover Colfax, Jan 22
https://www.eventbrite.com/e/cory-doctorow-live-at-tattered-cover-colfax-tickets-1976644174937
Colorado Springs: Guest of Honor at COSine, Jan 23-25
https://www.firstfridayfandom.org/cosine/

Recent appearances (permalink)

Enshittification with Four Ways to Change the World (Channel 4)
https://www.youtube.com/watch?v=tZQaEeuuI3Q
The Plan is to Make the Internet Worse. Forever. (Novarra Media)
https://www.youtube.com/watch?v=7wE8G-d7SnY
Enshittification (Future Knowledge)
https://futureknowledge.transistor.fm/episodes/enshittification
We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w
How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

Mon, 08 Dec 2025 13:01:31 +0000 Fullscreen Open in Tab

Pluralistic: Elon Musk's Blue Tick scam (08 Dec 2025)

Today's links

Elon Musk's Blue Tick scam: The EU bans giant teddybears.
Hey look at this: Delights to delectate.
Object permanence: Denver bomb squad vs 8" toy robot; Iceland's atheist religion; Largest strike in human history; Ad-tech is a bubble; Battery rationality; Pasta carpet; "With a Little Help"; Crooked Timber on Pikett; Tiki-mug menorah; China vs Big Data-backstabbing.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

Elon Musk's Blue Tick scam (permalink)

In my book Enshittification, I develop the concept of "giant teddybears," a scam that has been transposed from carnival midway games to digital platforms. The EU has just fined Elon Musk $140m for running a giant teddybear scam on Twitter:

https://arstechnica.com/tech-policy/2025/12/elon-musks-x-first-to-be-fined-under-eus-digital-service-act/

Growing up, August 15 always meant two things for my family: my mother's birthday and the first day of the CNE, a giant traveling fair that would park itself on Toronto's waterfront for the last three weeks of summer. We'd get there early, and by 10AM, there'd always be some poor bastard lugging around a galactic-scale giant teddybear that was offered as a prize at one of the midway games.

Now, nominally, the way you won a giant teddybear was by getting five balls in a peach basket. To a first approximation, this is a feat that no one has ever accomplished. Rather, a carny had beckoned this guy over and said, "Hey, fella, I like your face. Tell you what I'm gonna do: you get just one ball in the basket and I'll give you one of these beautiful, luxurious keychains. If you win two keychains, I'll let you trade them in for one of these gigantic teddybears."

Why would the carny do this? Because once this poor bastard took possession of the giant teddybear, he was obliged to conspicuously lug it around the CNE midway in the blazing, muggy August heat. All who saw him would think, "Hell if that dumbass can win a giant teddybear, I'm gonna go win one, too!" Charitably, you could call him a walking advertisement. More accurately, though, he was a Judas goat.

Digital platforms have the ability to give out giant teddybears at scale. Because digital platforms have the flexibility that comes with running things on computers, platforms can pick out individual platform participants and make them King For the Day, showering them in riches that they will boast of, luring in other suckers who will lose everything:

https://pluralistic.net/2023/02/19/twiddler/

That's how Tiktok works: the company's "heating tool" lets them drive traffic to Tiktok performers by cramming their videos into millions of random people's feeds, overriding Tiktok's legendary recommendation algorithm. Those "heated" performers get millions of views on their videos and go on to spam all the spaces where similar performers hang out, boasting of the fame and riches that await other people in their niche if they start producing for Tiktok:

https://pluralistic.net/2023/01/21/potemkin-ai/#hey-guys

Uber does it, too: as Veena Dubal documents in her work on "algorithmic wage discrimination," Uber offers different drivers wildly different wages for performing the same work. The lucky few who get an Uber giant teddybear hang out in rideshare groupchats and forums, trumpeting their incredible gains from the platform, while everyone else blames themselves for "being bad at the app," as they drive and drive, only to go deeper and deeper into debt:

https://pluralistic.net/2023/04/12/algorithmic-wage-discrimination/#fishers-of-men

Everywhere you look online, you see giant teddybears. Think of Joe Rogan being handed hundreds of millions of dollars to relocate his podcast to Spotify, an also-ran podcast platform that is desperately trying to capture the medium of podcasting, turning an open protocol into a proprietary, enclosed, Spotify-exclusive content stream:

https://pluralistic.net/2023/01/27/enshittification-resistance/#ummauerter-garten-nein

The point of the conspicuous, over-the-odds payment to Rogan isn't just to get Rogan onto Spotify – it's to convince every other podcaster that Spotify is a great place to make podcasts for. It isn't, though: when Spotify bought Gimlet Media, they locked Gimlet's podcasts inside Spotify's walled garden/maximum security prison. If you wanted to listen to a Gimlet podcast, you'd have to switch to using Spotify's app (and submitting to Spotify's invasive surveillance and restrictions on fast-forwarding through ads, etc).

Pretty much no one did this. After an internal revolt by Gimlet podcast hosts – whose podcasts were dwindling to utter irrelevance because no one was listening to them anymore – Spotify moved those Gimlet podcasts back onto the real internet, where they belong.

When Musk bought Twitter, he started handing out tons of giant teddybears – most notably, he created an opaque monetization scheme for popular Twitter posters, which allowed him to thumb the scales for a few trolls he liked, who obliged him by loudly proclaiming just how much money you could make by trolling professionally on Twitter. Needless to say, the vast majority of people who try this make either nothing, or a sum so small that it rounds to nothing.

But Musk's main revenue plan for Twitter – the thing he repeatedly promised would allow him to recoup the tens of billions he borrowed to buy the platform – was selling blue tick verification.

Twitter created blue ticks to solve a serious platform problem. Twitter users kept getting sucked in by impersonators who would trick them into participating in scams or believing false things. To protect those users, Twitter offered a verification scheme for "notable people" who were likely to face impersonation. The verification system was never very good – I successfully lobbied them to improve it a little when I was being impersonated on Twitter (I got them to stop insisting that users fax them a scan of their ID, or, more realistically, to send them ID via a random, insecure email-to-fax gateway). But it did the job reasonably well.

Predictably, though, the verification scheme also became something of a (weird and unimportant) status-symbol, allowing a certain kind of culture warrior to peddle grievances about how only "lamestream media libs" were getting blue ticks, while brave Pizzagaters and 4chan refugees were denied this important recognition.

Musk's plan to sell blue ticks leaned heavily into these grievances. He promised to "democratize" verification, for $8/month (or, for businesses, many thousands of dollars per month). Users who didn't buy blue ticks would have their content demoted and hidden from their own followers. Users who paid for blue ticks would have their content jammed into everyone's feeds, irrespective of whether Twitter's own content recommendation algorithms predicted those users would enjoy it. Best of all, Twitter wouldn't do much verifying – you could give Twitter $8, claim to be anyone at all, and chances are, you would be able to assume any identity you wanted, post any bullshit you wanted, and get priority placement in millions of users' feeds.

This was a massive gift to scammers, trolls and disinformation peddlers. For $8, you could pretend to be a celebrity in order to endorse a stock swindle, shitcoin hustle, or identity theft scheme. You could post market-moving disinformation from official-looking corporate accounts. You could pose as a campaigning politician or a reporter and post reputation-destroying nonsense.

This is where the EU comes in. In 2024, the EU enacted a pair of big, muscular Big Tech antitrust laws, the Digital Services Act (DSA) and the Digital Markets Act (DMA). These are complex pieces of legislation, and I don't like everything in them, but some parts of them are amazing: bold and imaginative breaks from the dismal history of ineffective or counterproductive tech regulation.

Under the DSA, the EU has fined Twitter about $140m for exposing users to scams via this blue tick giant teddybear wheeze (much of that sum is punitive, because Twitter flagrantly obstructed the Commission's investigations). The DSA (sensibly) doesn't require user verification, but it does expect companies that tell their users that some accounts are verified and can be trusted, to actually verify that they actually can be trusted.

I think there's a second DSA claim to be made here, beyond the failure to verify. Musk's plan to sell blue ticks was a disaster: while many, many scammers (and a few trolls) bought blue ticks, no one else did. The blue tick – which Musk thought of as a valuable status symbol that he could sell – was quickly devalued. "Account with a blue tick" was never all that prestigious, but under Musk, it came to mean "account that pushes scams, gore, disinformation, porn and/or hate."

So Musk did something very funny and sweaty. He restored blue ticks to millions of high-follower accounts (including my own). And despite the fact that Musk had created about a million different kinds of blue ticks that denoted different kinds of organizations and payment schemes, these free blue ticks were indistinguishable from the paid ones.

In other words, Musk set out to trick users into thinking that the most prominent people they followed believed that it was worth spending $8/month on a blue tick. It was an involuntary giant teddybear scam. Every time a prominent user with a free blue tick posts, they help Musk trick regular Twitter users into thinking that these worthless $8/month subscriptions are worth shelling out for.

I think the Commission could run another, equally successful enforcement action against Musk and Twitter over this scam, too.

Trump has been bellyaching nonstop about the DSA and DMA, threatening EU nations and businesses with tariffs and other TACO retribution if they go ahead with DSA/DMA enforcement. Let's hope the EU calls his bluff.

Of course, Musk could get out of paying these fines by moving all his businesses out of the EU, which, frankly, would be a major result for Europe.

(Image: Gage Skidmore, CC BY-SA 4.0, modified)

Hey look at this (permalink)

Netflix Is Trying to Buy Warner Bros Discovery. That Would Be a Disaster for America. https://www.thebignewsletter.com/p/netflix-is-trying-to-buy-warner-bros
How popular is ecosocialist transformation? https://jasonhickel.substack.com/p/how-popular-is-ecosocialist-transformation
Luigi Mangione Official Legal Fund for all 3 Cases https://www.givesendgo.com/luigi-defense-fund
Trump’s Katrina Is Coming https://prospect.org/2025/12/05/trumps-katrina-is-coming-fema/
DEFT: DSPs for Equitable and Fair Treatment https://deft-us.com/

Object permanence (permalink)

#20yrsago What’s involved in different publishing jobs? https://web.archive.org/web/20050306095536/http://www.penguin.co.uk/static/packages/uk/aboutus/jobs_workingpeng.html

#20yrsago Sony finally releases rookit uninstaller — sort of https://web.archive.org/web/20051204015131/http://cp.sonybmg.com/xcp/english/updates.html

#20yrsago EFF forces Sony/Suncomm to fix its spyware https://web.archive.org/web/20051210024413/https://www.eff.org/news/archives/2005_12.php#004234

#20yrsago Warner Music attacks specialized web-browser https://web.archive.org/web/20051210024927/http://www.pearworks.com/pages/pearLyrics.html

#20yrsago Sony’s DRM security fix leaves your computer more vulnerable https://blog.citp.princeton.edu/2005/12/07/mediamax-bug-found-patch-issued-patch-suffers-same-bug/

#15yrsago Internet furnishes fascinating tale of a civil rights era ghosttown on demandhttps://www.reddit.com/r/AskReddit/comments/eddwx/what_the_hell_happened_to_cairo_illinois/

#15yrsago Pasta carpet! https://wemakecarpets.wordpress.com/2010/11/02/pasta-carpet-2/

#15yrsago With a Little Help launch! https://memex.craphound.com/2010/12/07/with-a-little-help-launch/

#15yrsago Denver bomb squad defeats 8″ toy robot after hours-long standoff https://www.denverpost.com/2010/12/01/toy-robot-detours-traffic-near-coors-field/

#15yrsago UK govt demands an end to evidence-based drug policy https://www.theguardian.com/politics/2010/dec/05/government-scientific-advice-drugs-policy?&

#10yrsago Iceland’s fastest-growing “religion” courts atheists by promising to rebate religious tax https://icelandmonitor.mbl.is/news/politics_and_society/2015/12/01/icelanders_flocking_to_the_zuist_religion/

#10yrsago Springer Nature to release 100,000 titles as DRM-free bundles https://web.archive.org/web/20151210051243/https://www.digitalbookworld.com/2015/bitlit-partners-with-springer-to-offer-ebook-bundles/

#10yrsago Solo: Hope Larson’s webcomic of rock-n-roll, romance, and desperation https://memex.craphound.com/2015/12/07/solo-hope-larsons-webcomic-of-rock-n-roll-romance-and-desperation/

#10yrsago Body-painted models disappear into the Wonders of the World https://www.trinamerry.com/trinamerryblog/sevenwondersbodypaint

#10yrsago Make: the simplest electric car toy, a homopolar motor https://www.youtube.com/watch?v=oPzJr1jjHnQ

#10yrsago Thomas Piketty seminar on Crooked Timber https://crookedtimber.org/2016/01/04/thomas-piketty-seminar/

#10yrsago MAKE: a tiki-mug menorah https://web.archive.org/web/20151208123229/http://news.critiki.com/2015/12/05/tiki-mug-menorah-a-how-to-from-poly-hai/

#10yrsago Harvard Business School: Talented assholes are more trouble than they’re worth https://www.hbs.edu/ris/Publication

#10yrsago Multi-generational cruelty: America’s prisons shutting down kids’ visitations https://web.archive.org/web/20151204063410/https://www.thenation.com/article/2-7m-kids-have-parents-in-prison-theyre-losing-their-right-to-visit/

#10yrsago READ: Kim Stanley Robinson’s first standalone story in 25 years! https://reactormag.com/oral-argument-kim-stanley-robinson//

#10yrsago French Ministry of Interior wants to ban open wifi, Tor https://arstechnica.com/tech-policy/2015/12/france-looking-at-banning-tor-blocking-public-wi-fi/

#5yrsago China’s war on big data backstabbing https://pluralistic.net/2020/12/07/backstabbed/#big-data-backstabbing

#5yrsago The largest strike in human history https://pluralistic.net/2020/12/06/surveillance-tulip-bulbs/#modi-miscalulation

#5yrsago Ad-tech as a bubble overdue for a bursting https://pluralistic.net/2020/12/06/surveillance-tulip-bulbs/#adtech-bubble

#1yrago Battery rationality https://pluralistic.net/2024/12/06/shoenabombers/#paging-dick-cheney

#1yrago A year in illustration (2024) https://pluralistic.net/2024/12/07/great-kepplers-ghost/#art-adjacent

Upcoming appearances (permalink)

Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html
Denver: Enshittification at Tattered Cover Colfax, Jan 22
https://www.eventbrite.com/e/cory-doctorow-live-at-tattered-cover-colfax-tickets-1976644174937
Colorado Springs: Guest of Honor at COSine, Jan 23-25
https://www.firstfridayfandom.org/cosine/

Recent appearances (permalink)

The Plan is to Make the Internet Worse. Forever. (Novarra Media)
https://www.youtube.com/watch?v=7wE8G-d7SnY
Enshittification (Future Knowledge)
https://futureknowledge.transistor.fm/episodes/enshittification
We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w
How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s
Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

Sat, 06 Dec 2025 19:48:56 +0000 Fullscreen Open in Tab

Pluralistic: Metabolizing the theory of "political capitalism" (06 Dec 2025)

Today's links

Metabolizing the theory of "political capitalism": How many $TRUMP coins should your company buy?
Hey look at this: Delights to delectate.
Object permanence: NZ doesn't want US-style copyright; Tron: Reloaded; Mass shootings and gun profits; Descartes God has failed; "The Big Fix."
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

Metabolizing the theory of "political capitalism" (permalink)

It's a strange fact that the more sophisticated and polished a theory gets, the simpler it tends to be. New theories tend to be inspired by a confluence of many factors, and early attempts to express the theory will seek to enumerate and connect everything that seems related, which is a lot.

But as you develop the theory, it gets progressively more streamlined as you realize which parts can be safely omitted or combined without sacrificing granularity or clarity. This simplification requires a lot of iteration and reiteration, over a lot of time, for a lot of different audiences and critics. As Thoreau wrote (paraphrasing Pascal), "Not that the story need be long, but it will take a long while to make it short."

This week, I encountered a big, exciting theory that is still in the "long and complicated" phase, with so many moving parts that I'm having trouble keeping them straight in my head. But the idea itself is fascinating and has so much explanatory power, and I've been thinking about it nonstop, so I'm going to try to metabolize a part of it here today, both to bring it to your attention, and to try and find some clarity for myself.

At issue is Dylan Riley and Robert Brenner's theory of "political capitalism," which I encountered through John Ganz's writeup of a panel he attended to discuss Riley and Brenner's work:

https://www.unpopularfront.news/p/politics-and-capitalist-stagnation

Riley and Brenner developed this theory through a pair of very long (and paywalled) articles in the New Left Review. First is 2022's "Seven Theses on American Politics" (£3), which followed the Democrats' surprisingly good showing in the 2022 midterms:

https://newleftreview.org/issues/ii138/articles/4813

The second article, "The Long Downturn and Its Political Results" (£4), is even longer, and it both restates the theory of "Seven Theses" and addresses several prominent critics of their work:

https://newleftreview.org/issues/ii155/articles/dylan-riley-robert-brenner-the-long-downturn-and-its-political-results

(If you're thinking about reading the source materials – and I urge you to do so – I think you can safely just read the second article, as it really does recap and streamline the original.)

So what is this theory? Ganz does a good job of breaking it down (better than Riley and Brenner, who, I think, still have a lot of darlings they can't bring themselves to murder). Here's my recap of Ganz's, then, with a few notes from the source texts thrown in.

Riley and Brenner are advancing both an economic and a political theory, with the latter growing out of the former. The economic theory seeks to explain two phenomena, the "Long Boom" (post-WWII to the 1960s or so), and the "Long Downturn" (ever since).

During the Long Boom, the US economy (and some other economies) experienced a period of sustained growth, without the crashes that had been the seemingly inevitable end-point of previous growth periods. Riley and Brenner say that these crashes were the result of business owners making the (locally) rational decision to hang on to older machines and tools even as new ones came online.

Businesses are always looking to invest in new automation in a bid to wring more productivity from their workers. Profits come from labor, not machines, and as your competitors invest in the same machines as you've just bought, the higher rate of profit you got when you upgraded your machines will be eroded, as competitors chase each others' customers with lower prices.

But not everyone is willing to upgrade when a new machine is invented. If you're still paying for the old machines, you just can't afford to throw them away and get the latest and greatest ones. Instead, as your competitors slash prices (because they have new machines that let them make the same stuff at a lower price), you must lower your prices too, accepting progressively lower profits.

Eventually, your whole sector is using superannuated machines that they're still making payments on, and the overall rate of profit in the sector has dwindled to unsustainable levels. "Zombie companies" (companies that have no plausible chance of paying off their debts) dominate the economy. This is the "secular stagnation" that economists dread. Note that this whole thing is driven by the very same forces that make capitalism so dynamic: the falling rate of profit that gives rise to a relentless chase for new, more efficient processes. This is a stagnation born of dynamism, and the harder you yank on the "make capitalism more dynamic" lever, the more stagnant it becomes.

Hoover and Mellon's austerity agenda in the 1920s sought to address this by triggering mass bankruptcies, in a brutal bid to "purge" those superannuated machines and the companies that owned them, at the expense of both workers and creditors. This wasn't enough.

Instead, we got WWII, in which the government stepped in to buy things at rates that paid for factories to be retooled, and which pressed the entire workforce into employment. This is the trigger for the Long Boom, as America got a do-over with all-new capital and a freshly trained workforce with high morale and up-to-date skills.

So that's the Long Boom. What about the Great Downturn? This is where Ganz's account begins. As the "late arrivals" (Japan, West Germany, South Korea, and, eventually China) show up on the world stage, they do their own Long Boom, having experienced an even more extreme "purge" of their zombie firms and obsolete machines. This puts downward pressure on profits in the USA (and, eventually, the late arrivals), leading to the Long Stagnation, a 50 year period in which the rate of profit in the USA has steadily declined.

This is most of the economic theory, and it contains the germ of the political theory, too. During the Long Boom, there was plenty to go around, and the US was able to build out a welfare state, its ruling class was willing to tolerate unions, and movements for political and economic equality for women, sexual minorities, disabled people, racial minorities, etc, were able to make important inroads.

But the political theory gets into high gear after years of Great Downturn. That's when the world has an oversupply of cheap goods and a sustained decline in the rate of profit, and the rate of profit declines every time someone invents a more efficient and productive technology. Companies in Downturn countries need to find a new way to improve their profits – they need to invest in something other than improved methods of production.

That's where "political capitalism" comes in. Political capitalism is the capitalism you get when the cheapest, most reliable way to improve your rate of profit is to invest in the political process, to get favorable regulation, pork barrel government contracts, and cash bailouts. As Ganz puts it, "capitalists have gone from profit-seekers to rent-seekers," or, as Brenner and Riley write, capitalists now seek "a return on investment largely or completely divorced from material production."

There's a sense in which this is immediately recognizable. The ascendancy of political capitalism tracks with the decline in antitrust enforcement, the rise of monopolies, a series of massive bailouts, and, under Trump, naked kleptocracy. In the US, "raw political power is the main source of return on capital."

The "neoliberal turn" of late Carter/Reagan is downstream of political capitalism. When there was plenty to go around, the capital classes and the political classes were willing to share with workers. When the Great Downturn takes hold, bosses turn instead to screwing workers and taking over the political system. Fans of Bridget Read's Little Bosses Everywhere will know this as the moment in which Gerry Ford legalized pyramid schemes in order to save the founders of Amway, who were big GOP donors who lived in Ford's congressional district:

https://pluralistic.net/2025/05/05/free-enterprise-system/#amway-or-the-highway

Manufacturing's rate of profit has never recovered from this period – there have been temporary rallies, but the overall trend is down, down, down.

But this is just the beginning of the political economy of Brenner and Riley's theory. Remember, this all started with an essay that sought to make sense of the 2022 midterms. Much of the political theory deals with electoral politics, and what has happened with America's two major political parties.

Under political capitalism, workers are split into different groups depending on their relationship to political corruption. The "professional managerial class" (workers with degrees and other credentials) end up aligned with center-left parties, betting that these parties will use political power to fund the kinds of industries that hire credentialed workers, like health and education. Non-credentialed workers align themselves with right-wing parties that promise to raise their wages by banning immigrants and ending free trade.

Ganz's most recent book, When the Clock Broke: Con Men, Conspiracists, and How America Cracked Up in the Early 1990s looks at the origins of the conspiratorial right that became MAGA:

https://us.macmillan.com/books/9780374605445/whentheclockbroke/

He says that Riley and Brenner's theory really helps explain the moment he chronicled in his own book, for example, the way that Ross Perot (an important Trump predecessor) built power by railing against "late arrivals" like Japan, Germany and South Korea.

This is also the heyday of corporate "finacialization," which can be thought of as the process by which companies stop concerning themselves with how to make and sell superior products more efficiently, and instead devote themselves to financial gimmicks that allow shareholders to extract wealth from the firm. It's a period of slashed R&D budgets, mass layoffs, union-busting, and massive corporate borrowing.

In the original papers, Riley and Brenner drop all kinds of juicy, eye-opening facts and arguments to support their thesis. For example, in the US, more and more machinery is idle. In the 1960s, the US employed 85% of its manufacturing capacity. It was 78% in the 1980s, and now it's 75%. One quarter of "US plant and equipment is simply stagnating."

Today's economic growth doesn't come from making stuff, it comes from extraction, buttressed by law. Looser debt rules allowed households to continue to consume by borrowing, with the effect that a substantial share of workers' wages go to servicing debt, which is to say, paying corporations for the privilege of existing, over and above the cost of the goods and services we consume.

But the debt industry itself hasn't gotten any more efficient: "the cost of moving a dollar from a saver to a borrower was about two cents in 1910; a hundred years later, it was the same." They're making more, but they haven't made any improvements – all the talk of "fintech" and "financial engineering" have not produced any efficiencies. "This puzzle resolves itself once we recognize that the vast majority of financial innovation is geared towards figuring out how to siphon off resources through fees, insider information and lobbying."

Reading these arguments, I was struck by how this period also covers the rise and rise of "IP." This is a period in which your ability to simply buy things declined, replaced with a system in which you rent and subscribe to things – forever. From your car to your thermostat, the key systems in your life are increasingly a monthly bill, meaning that every time you add something to your life, it's not a one-time expenditure; it's a higher monthly cost of living, forever.

The rise and rise of IP is certainly part of political capitalism. The global system of IP comes from political capture, such as the inclusion of an IP chapter ("TRIPS") in the World Trade Agreement, as well as the WIPO Copyright Treaties. This is basically a process by which large (mostly American) businesses reorganized the world's system of governance and law to allow them to extract rents and slash R&D. The absurd, inevitable consequence of this nonsense is today's "capital light" chip companies, that don't make chips, just designs, which are turned out by one or two gigantic companies, mostly in Taiwan.

Of course, Riley and Brenner aren't the first theorists to observe that our modern economy is organized around extracting rents, rather than winning profits. Yanis Varoufakis likens the modern economy to medieval feudalism, dubbing the new form "technofeudalism":

https://pluralistic.net/2023/09/28/cloudalists/#cloud-capital

Riley and Brenner harken back to a different kind of feudal practice as the antecedant to political capitalism: "tax-farming."

Groups of entrepreneurs would advance money to the sovereign in exchange for the right to collect taxes from a given territory or population. Their ‘profit’ consisted in the difference between the money that they advanced to the ruler for the right to tax and what they could extract from the population through the exercise of that right. So, these entrepreneurs invested in politics, the control of means of administration and the means of violence, as a method for extracting surplus, in this way making for a politically constituted form of rent.

Unlike profits, rents are "largely or completely divorced from material production," "they ‘create no wealth’ and … they ‘reduce economic growth and reallocate incomes from the bottom to the top.'"

To make a rent, you need an asset, and in today's system, high asset prices are a top political priority: governments intervene to keep the prices of houses high, to protect corporate bonds, and, of course, to keep AI companies' shares and IOUs from going to zero. The economy is dominated by "a large group of politically dependent firms and households…profoundly reliant on a policy of easy credit on the part of government… The US economy as a whole is sustained by lending, backed up by government, with profits accruing from production under excruciating pressure."

Our social programs have been replaced by public-private partnerships that benefit these "politically dependent firms." Bush's Prescription Drug Act didn't seek to recoup public investment in pharma research through lower prices – it offered a (further) subsidy to pharma companies in exchange for (paltry/nonexistent) price breaks. Obama's Affordable Care Act transferred hundreds of billions to investors in health corporations, who raised prices and increased their profits. Trump's CARES Act bailed out every corporate debtor in the country. Biden's American Rescue Plan, CHIPS Act and Inflation Reduction Act don't offer public services or transfer funds to workers – instead, they offer subsidies to the for-profit sector.

Electorally, political capitalism is a system of "vertiginous levels of campaign expenditure and open corruption on a vast scale." It pushed workers into the arms of far-right parties, while re-organizing center-left parties as center-right parties of the lanyard class. Both parties are hamstrung because "in a persistently low- or no-growth environment…parties can no longer operate on the basis of programmes for growth."

This is really just scraping the surface. I think it's well worth £4 to read the source document. I look forward to the further development of this theory, to its being streamlined. It's got a lot of important things to say, even if it is a little hard to metabolize at present.

Hey look at this (permalink)

EU's New Digital Package Proposal Promises Red Tape Cuts but Guts GDPR Privacy Rights https://www.eff.org/deeplinks/2025/12/eus-new-digital-package-proposal-promises-red-tape-cuts-guts-gdpr-privacy-rights
Looks Like We Can Finally Kiss the Metaverse Goodbye https://gizmodo.com/looks-like-we-can-finally-kiss-the-metaverse-goodbye-2000695825
A New Anonymous Phone Carrier Lets You Sign Up With Nothing but a Zip Code https://www.wired.com/story/new-anonymous-phone-carrier-sign-up-with-nothing-but-a-zip-code/
Microsoft drops AI sales targets in half after salespeople miss their quotas https://arstechnica.com/ai/2025/12/microsoft-slashes-ai-sales-growth-targets-as-customers-resist-unproven-agents/
The Hidden Cost of Ceding Government Procurement to a Monopoly Gatekeeper https://ilsr.org/article/independent-business/turning-public-money-into-amazons-profits/

Object permanence (permalink)

#20yrsago Student ethnographies of World of Warcraft https://web.archive.org/web/20051208020004/http://www.trinity.edu/adelwich/mmo/students.html

#20yrsago Sony rootkit ripped off anti-DRM code to break into iTunes https://blog.citp.princeton.edu/2005/12/04/hidden-feature-sony-drm-uses-open-source-code-add-apple-drm/

#20yrsago English info on France’s terrible proposed copyright law https://web.archive.org/web/20060111032903/http://eucd.info/index.php?English-readers

#15yrsago New Zealand leak: US-style copyright rules are a bad deal https://web.archive.org/web/20101206090519/http://www.michaelgeist.ca/content/view/5498/125/

#15yrsago Tron: Reloaded, come for the action, stay for the aesthetics https://memex.craphound.com/2010/12/05/tron-reloaded-come-for-the-action-stay-for-the-aesthetics/

#10yrsago Unelectable Lindsey Graham throws caution to the wind https://web.archive.org/web/20151206030630/https://gawker.com/i-am-tired-of-this-crap-lindsey-graham-plays-thunderi-1746116881

#10yrsago Every time there’s a mass shooting, gun execs & investors gloat about future earnings https://theintercept.com/2015/12/03/mass-shooting-wall-st/

#10yrsago How to bake spice-filled sandworm bread https://web.archive.org/web/20151205193104/https://kitchenoverlord.com/2015/12/03/dune-week-spice-filled-sandworm/

#5yrsago Descartes' God has failed and Thompson's Satan rules our computers https://pluralistic.net/2020/12/05/trusting-trust/#thompsons-devil

#5yrsago Denise Hearn and Vass Bednar's "The Big Fix" https://pluralistic.net/2024/12/05/ted-rogers-is-a-dope/#galen-weston-is-even-worse

Upcoming appearances (permalink)

Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html
Denver: Enshittification at Tattered Cover Colfax, Jan 22
https://www.eventbrite.com/e/cory-doctorow-live-at-tattered-cover-colfax-tickets-1976644174937

Recent appearances (permalink)

Enshittification (Future Knowledge)
https://futureknowledge.transistor.fm/episodes/enshittification
We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w
How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s
Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/
Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

Fri, 05 Dec 2025 14:34:39 +0000 Fullscreen Open in Tab

Pluralistic: The Reverse-Centaur’s Guide to Criticizing AI (05 Dec 2025)

Today's links

The Reverse Centaur’s Guide to Criticizing AI: My speech for U Washington's Neuroscience, AI and Society lecture series.
Hey look at this: Delights to delectate.
Object permanence: Pac Man ghost algorithms; The US wrote Spain's copyright law; Illinois makes prisoners rent their cells; "Urban Transport Without the Hot Air"; "Ministry for the Future": Canada sues Google; In defense of 230; NYPD racist murder postmortem; Student debt trap; "That makes me smart."
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

The Reverse Centaur’s Guide to Criticizing AI (permalink)

Last night, I gave a speech for the University of Washington's "Neuroscience, AI and Society" lecture series, through the university's Computational Neuroscience Center. It was called "The Reverse Centaur’s Guide to Criticizing AI," and it's based on the manuscript for my next book, "The Reverse Centaur’s Guide to Life After AI," which will be out from Farrar, Straus and Giroux next June:

https://www.eventbrite.com/e/future-tense-neuroscience-ai-and-society-with-cory-doctorow-tickets-1735371255139

The talk was sold out, but here's the text of my lecture. I'm very grateful to UW for the opportunity, and for a lovely visit to Seattle!

I'm a science fiction writer, which means that my job is to make up futuristic parables about our current techno-social arrangements to interrogate not just what a gadget does, but who it does it for, and who it does it to.

What I don't do is predict the future. No one can predict the future, which is a good thing, since if the future were predictable, that would mean that what we all do couldn't change it. It would mean that the future was arriving on fixed rails and couldn't be steered.

Jesus Christ, what a miserable proposition!

Now, not everyone understands the distinction. They think sf writers are oracles, soothsayers. Unfortunately, even some of my colleagues labor under the delusion that they can "see the future."

But for every sf writer who deludes themselves into thinking that they are writing the future, there are a hundred sf fans who believe that they are reading the future, and a depressing number of those people appear to have become AI bros. The fact that these guys can't shut up about the day that their spicy autocomplete machine will wake up and turn us all into paperclips has led many confused journalists and conference organizers to try to get me to comment on the future of AI.

That's a thing I strenuously resisted doing, because I wasted two years of my life explaining patiently and repeatedly why I thought crypto was stupid, and getting relentless bollocked by cryptocurrency cultists who at first insisted that I just didn't understand crypto. And then, when I made it clear that I did understand crypto, insisted that I must be a paid shill.

This is literally what happens when you argue with Scientologists, and life is Just. Too. Short.

So I didn't want to get lured into another one of those quagmires, because on the one hand, I just don't think AI is that important of a technology, and on the other hand, I have very nuanced and complicated views about what's wrong, and not wrong, about AI, and it takes a long time to explain that stuff.

But people wouldn't stop asking, so I did what I always do. I wrote a book.

Over the summer I wrote a book about what I think about AI, which is really about what I think about AI criticism, and more specifically, how to be a good AI critic. By which I mean: "How to be a critic whose criticism inflicts maximum damage on the parts of AI that are doing the most harm." I titled the book The Reverse Centaur's Guide to Life After AI, and Farrar, Straus and Giroux will publish it in June, 2026.

But you don't have to wait until then because I am going to break down the entire book's thesis for you tonight, over the next 40 minutes. I am going to talk fast.

Start with what a reverse centaur is. In automation theory, a "centaur" is a person who is assisted by a machine. You're a human head being carried around on a tireless robot body. Driving a car makes you a centaur, and so does using autocomplete.

And obviously, a reverse centaur is machine head on a human body, a person who is serving as a squishy meat appendage for an uncaring machine.

Like an Amazon delivery driver, who sits in a cabin surrounded by AI cameras, that monitor the driver's eyes and take points off if the driver looks in a proscribed direction, and monitors the driver's mouth because singing isn't allowed on the job, and rats the driver out to the boss if they don't make quota.

The driver is in that van because the van can't drive itself and can't get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance. But the driver is human, so the van doesn't just use the driver. The van uses the driver up.

Obviously, it's nice to be a centaur, and it's horrible to be a reverse centaur. There are lots of AI tools that are potentially very centaur-like, but my thesis is that these tools are created and funded for the express purpose of creating reverse-centaurs, which is something none of us want to be.

But like I said, the job of an sf writer is to do more than think about what the gadget does, and drill down on who the gadget does it for and who the gadget does it to. Tech bosses want us to believe that there is only one way a technology can be used. Mark Zuckerberg wants you to think that it's technologically impossible to have a conversation with a friend without him listening in. Tim Cook wants you to think that it's technologically impossible for you to have a reliable computing experience unless he gets a veto over which software you install and without him taking 30 cents out of every dollar you spend. Sundar Pichai wants you think that it's impossible for you to find a webpage unless he gets to spy on you from asshole to appetite.

This is all a kind of vulgar Thatcherism. Margaret Thatcher's mantra was "There is no alternative." She repeated this so often they called her "TINA" Thatcher: There. Is. No. Alternative. TINA.

"There is no alternative" is a cheap rhetorical slight. It's a demand dressed up as an observation. "There is no alternative" means "STOP TRYING TO THINK OF AN ALTERNATIVE." Which, you know, fuck that.

I'm an sf writer, my job is to think of a dozen alternatives before breakfast.

So let me explain what I think is going on here with this AI bubble, and sort out the bullshit from the material reality, and explain how I think we could and should all be better AI critics.

Start with monopolies: tech companies are gigantic and they don't compete, they just take over whole sectors, either on their own or in cartels.

Google and Meta control the ad market. Google and Apple control the mobile market, and Google pays Apple more than $20 billion/year not to make a competing search engine, and of course, Google has a 90% Search market-share.

Now, you'd think that this was good news for the tech companies, owning their whole sector.

But it's actually a crisis. You see, when a company is growing, it is a "growth stock," and investors really like growth stocks. When you buy a share in a growth stock, you're making a bet that it will continue to grow. So growth stocks trade at a huge multiple of their earnings. This is called the "price to earnings ratio" or "P/E ratio."

But once a company stops growing, it is a "mature" stock, and it trades at a much lower P/E ratio. So for every dollar that Target – a mature company – brings in, it is worth ten dollars. It has a P/E ratio of 10, while Amazon has a P/E ratio of 36, which means that for every dollar Amazon brings in, the market values it at $36.

It's wonderful to run a company that's got a growth stock. Your shares are as good as money. If you want to buy another company, or hire a key worker, you can offer stock instead of cash. And stock is very easy for companies to get, because shares are manufactured right there on the premises, all you have to do is type some zeroes into a spreadsheet, while dollars are much harder to come by. A company can only get dollars from customers or creditors.

So when Amazon bids against Target for a key acquisition, or a key hire, Amazon can bid with shares they make by typing zeroes into a spreadsheet, and Target can only bid with dollars they get from selling stuff to us, or taking out loans, which is why Amazon generally wins those bidding wars.

That's the upside of having a growth stock. But here's the downside: eventually a company has to stop growing. Like, say you get a 90% market share in your sector, how are you gonna grow?

Once the market decides that you aren't a growth stock, once you become mature, your stock is revalued, to a P/E ratio befitting a mature stock.

If you are an exec at a dominant company with a growth stock, you have to live in constant fear that the market will decide that you're not likely to grow any further. Think of what happened to Facebook in the first quarter of 2022. They told investors that they experienced slightly slower growth in the USA than they had anticipated, and investors panicked. They staged a one-day, $240B sell off. A quarter-trillion dollars in 24 hours! At the time, it was the largest, most precipitous drop in corporate valuation in human history.

That's a monopolist's worst nightmare, because once you're presiding over a "mature" firm, the key employees you've been compensating with stock, experience a precipitous pay-drop and bolt for the exits, so you lose the people who might help you grow again, and you can only hire their replacements with dollars. With dollars, not shares.

And the same goes for acquiring companies that might help you grow, because they, too, are going to expect money, not stock. This is the paradox of the growth stock. While you are growing to domination, the market loves you, but once you achieve dominance, the market lops 75% or more off your value in a single stroke if they don't trust your pricing power.

Which is why growth stock companies are always desperately pumping up one bubble or another, spending billions to hype the pivot to video, or cryptocurrency, or NFTs, or Metaverse, or AI.

I'm not saying that tech bosses are making bets they don't plan on winning. But I am saying that winning the bet – creating a viable metaverse – is the secondary goal. The primary goal is to keep the market convinced that your company will continue to grow, and to remain convinced until the next bubble comes along.

So this is why they're hyping AI: the material basis for the hundreds of billions in AI investment.

Now I want to talk about how they're selling AI. The growth narrative of AI is that AI will disrupt labor markets. I use "disrupt" here in its most disreputable, tech bro sense.

The promise of AI – the promise AI companies make to investors – is that there will be AIs that can do your job, and when your boss fires you and replaces you with AI, he will keep half of your salary for himself, and give the other half to the AI company.

That's it.

That's the $13T growth story that MorganStanley is telling. It's why big investors and institutionals are giving AI companies hundreds of billions of dollars. And because they are piling in, normies are also getting sucked in, risking their retirement savings and their family's financial security.

Now, if AI could do your job, this would still be a problem. We'd have to figure out what to do with all these technologically unemployed people.

But AI can't do your job. It can help you do your job, but that doesn't mean it's going to save anyone money. Take radiology: there's some evidence that AIs can sometimes identify solid-mass tumors that some radiologists miss, and look, I've got cancer. Thankfully, it's very treatable, but I've got an interest in radiology being as reliable and accurate as possible.

If my Kaiser hospital bought some AI radiology tools and told its radiologists: "Hey folks, here's the deal. Today, you're processing about 100 x-rays per day. From now on, we're going to get an instantaneous second opinion from the AI, and if the AI thinks you've missed a tumor, we want you to go back and have another look, even if that means you're only processing 98 x-rays per day. That's fine, we just care about finding all those tumors."

If that's what they said, I'd be delighted. But no one is investing hundreds of billions in AI companies because they think AI will make radiology more expensive, not even if that also makes radiology more accurate. The market's bet on AI is that an AI salesman will visit the CEO of Kaiser and make this pitch: "Look, you fire 9/10s of your radiologists, saving $20m/year, you give us $10m/year, and you net $10m/year, and the remaining radiologists' job will be to oversee the diagnoses the AI makes at superhuman speed, and somehow remain vigilant as they do so, despite the fact that the AI is usually right, except when it's catastrophically wrong.

"And if the AI misses a tumor, this will be the human radiologist's fault, because they are the 'human in the loop.' It's their signature on the diagnosis."

This is a reverse centaur, and it's a specific kind of reverse-centaur: it's what Dan Davies calls an "accountability sink." The radiologist's job isn't really to oversee the AI's work, it's to take the blame for the AI's mistakes.

This is another key to understanding – and thus deflating – the AI bubble. The AI can't do your job, but an AI salesman can convince your boss to fire you and replace you with an AI that can't do your job. This is key because it helps us build the kinds of coalitions that will be successful in the fight against the AI bubble.

If you're someone who's worried about cancer, and you're being told that the price of making radiology too cheap to meter, is that we're going to have to re-home America's 32,000 radiologists, with the trade-off that no one will ever be denied radiology services again, you might say, "Well, OK, I'm sorry for those radiologists, and I fully support getting them job training or UBI or whatever. But the point of radiology is to fight cancer, not to pay radiologists, so I know what side I'm on."

AI hucksters and their customers in the C-suites want the public on their side. They want to forge a class alliance between AI deployers and the people who enjoy the fruits of the reverse centaurs' labor. They want us to think of ourselves as enemies to the workers.

Now, some people will be on the workers' side because of politics or aesthetics. They just like workers better than their bosses. But if you want to win over all the people who benefit from your labor, you need to understand and stress how the products of the AI will be substandard. That they are going to get charged more for worse things. That they have a shared material interest with you.

Will those products be substandard? There's every reason to think so. Earlier, I alluded to "automation blindness, "the physical impossibility of remaining vigilant for things that rarely occur. This is why TSA agents are incredibly good at spotting water bottles. Because they get a ton of practice at this, all day, every day. And why they fail to spot the guns and bombs that government red teams smuggle through checkpoints to see how well they work, because they just don't have any practice at that. Because, to a first approximation, no one deliberately brings a gun or a bomb through a TSA checkpoint.

Automation blindness is the Achilles' heel of "humans in the loop."

Think of AI software generation: there are plenty of coders who love using AI, and almost without exception, they are senior, experienced coders, who get to decide how they will use these tools. For example, you might ask the AI to generate a set of CSS files to faithfully render a web-page across multiple versions of multiple browsers. This is a notoriously fiddly thing to do, and it's pretty easy to verify if the code works – just eyeball it in a bunch of browsers. Or maybe the coder has a single data file they need to import and they don't want to write a whole utility to convert it.

Tasks like these can genuinely make coders more efficient and give them more time to do the fun part of coding, namely, solving really gnarly, abstract puzzles. But when you listen to business leaders talk about their AI plans for coders, it's clear they're not looking to make some centaurs.

They want to fire a lot of tech workers – 500,000 over the past three years – and make the rest pick up their work with coding, which is only possible if you let the AI do all the gnarly, creative problem solving, and then you do the most boring, soul-crushing part of the job: reviewing the AIs' code.

And because AI is just a word guessing program, because all it does is calculate the most probable word to go next, the errors it makes are especially subtle and hard to spot, because these bugs are literally statistically indistinguishable from working code (except that they're bugs).

Here's an example: code libraries are standard utilities that programmers can incorporate into their apps, so they don't have to do a bunch of repetitive programming. Like, if you want to process some text, you'll use a standard library. If it's an HTML file, that library might be called something like lib.html.text.parsing; and if it's a DOCX file, it'll be lib.docx.text.parsing. But reality is messy, humans are inattentive and stuff goes wrong, so sometimes, there's another library, this one for parsing PDFs, and instead of being called lib.pdf.text.parsing, it's called lib.text.pdf.parsing.

Now, because the AI is a statistical inference engine, because all it can do is predict what word will come next based on all the words that have been typed in the past, it will "hallucinate" a library called lib.pdf.text.parsing. And the thing is, malicious hackers know that the AI will make this error, so they will go out and create a library with the predictable, hallucinated name, and that library will get automatically sucked into your program, and it will do things like steal user data or try and penetrate other computers on the same network.

And you, the human in the loop – the reverse centaur – you have to spot this subtle, hard to find error, this bug that is literally statistically indistinguishable from correct code. Now, maybe a senior coder could catch this, because they've been around the block a few times, and they know about this tripwire.

But guess who tech bosses want to preferentially fire and replace with AI? Senior coders. Those mouthy, entitled, extremely highly paid workers, who don't think of themselves as workers. Who see themselves as founders in waiting, peers of the company's top management. The kind of coder who'd lead a walkout over the company building drone-targeting systems for the Pentagon, which cost Google ten billion dollars in 2018.

For AI to be valuable, it has to replace high-wage workers, and those are precisely the experienced workers, with process knowledge, and hard-won intuition, who might spot some of those statistically camouflaged AI errors.

Like I said, the point here is to replace high-waged workers.

And one of the reasons the AI companies are so anxious to fire coders is that coders are the princes of labor. They're the most consistently privileged, sought-after, and well-compensated workers in the labor force.

If you can replace coders with AI, who cant you replace with AI? Firing coders is an ad for AI.

Which brings me to AI art. AI art – or "art" – is also an ad for AI, but it's not part of AI's business model.

Let me explain: on average, illustrators don't make any money. They are already one of the most immiserated, precartized groups of workers out there. They suffer from a pathology called "vocational awe." That's a term coined by the librarian Fobazi Ettarh, and it refers to workers who are vulnerable to workplace exploitation because they actually care about their jobs – nurses, librarians, teachers, and artists.

If AI image generators put every illustrator working today out of a job, the resulting wage-bill savings would be undetectable as a proportion of all the costs associated with training and operating image-generators. The total wage bill for commercial illustrators is less than the kombucha bill for the company cafeteria at just one of Open AI's campuses.

The purpose of AI art – and the story of AI art as a death-knell for artists – is to convince the broad public that AI is amazing and will do amazing things. It's to create buzz. Which is not to say that it's not disgusting that former OpenAI CTO Mira Murati told a conference audience that "some creative jobs shouldn't have been there in the first place," and that it's not especially disgusting that she and her colleagues boast about using the work of artists to ruin those artists' livelihoods.

It's supposed to be disgusting. It's supposed to get artists to run around and say, "The AI can do my job, and it's going to steal my job, and isn't that terrible?"

Because the customers for AI – corporate bosses – don't see AI taking workers' jobs as terrible. They see it as wonderful.

But can AI do an illustrator's job? Or any artist's job?

Let's think about that for a second. I've been a working artist since I was 17 years old, when I sold my first short story, and I've given it a lot of thought, and here's what I think art is: it starts with an artist, who has some vast, complex, numinous, irreducible feeling in their mind. And the artist infuses that feeling into some artistic medium. They make a song, or a poem, or a painting, or a drawing, or a dance, or a book, or a photograph. And the idea is, when you experience this work, a facsimile of the big, numinous, irreducible feeling will materialize in your mind.

Now that I've defined art, we have to go on a little detour.

I have a friend who's a law professor, and before the rise of chatbots, law students knew better than to ask for reference letters from their profs, unless they were a really good student. Because those letters were a pain in the ass to write. So if you advertised for a postdoc and you heard from a candidate with a reference letter from a respected prof, the mere existence of that letter told you that the prof really thought highly of that student.

But then we got chatbots, and everyone knows that you generate a reference letter by feeding three bullet points to an LLM, and it'll barf up five paragraphs of florid nonsense about the student.

So when my friend advertises for a postdoc, they are flooded with reference letters, and they deal with this flood by feeding all these letters to another chatbot, and ask it to reduce them back to three bullet points. Now, obviously, they won't be the same bullet-points, which makes this whole thing terrible.

But just as obviously, nothing in that five-paragraph letter except the original three bullet points are relevant to the student. The chatbot doesn't know the student. It doesn't know anything about them. It cannot add a single true or useful statement about the student to the letter.

What does this have to do with AI art? Art is a transfer of a big, numinous, irreducible feeling from an artist to someone else. But the image-gen program doesn't know anything about your big, numinous, irreducible feeling. The only thing it knows is whatever you put into your prompt, and those few sentences are diluted across a million pixels or a hundred thousand words, so that the average communicative density of the resulting work is indistinguishable from zero.

It's possible to infuse more communicative intent into a work: writing more detailed prompts, or doing the selective work of choosing from among many variants, or directly tinkering with the AI image after the fact, with a paintbrush or Photoshop or The Gimp. And if there will ever be a piece of AI art that is good art – as opposed to merely striking, or interesting, or an example of good draftsmanship – it will be thanks to those additional infusions of creative intent by a human.

And in the meantime, it's bad art. It's bad art in the sense of being "eerie," the word Mark Fisher uses to describe "when there is something present where there should be nothing, or there is nothing present when there should be something."

AI art is eerie because it seems like there is an intender and an intention behind every word and every pixel, because we have a lifetime of experience that tells us that paintings have painters, and writing has writers. But it's missing something. It has nothing to say, or whatever it has to say is so diluted that it's undetectable.

The images were striking before we figured out the trick, but now they're just like the images we imagine in clouds or piles of leaves. We're the ones drawing a frame around part of the scene, we're the ones focusing on some contours and ignoring the others. We're looking at an inkblot, and it's not telling us anything.

Sometimes that can be visually arresting, and to the extent that it amuses people in a community of prompters and viewers, that's harmless.

I know someone who plays a weekly Dungeons and Dragons game over Zoom. It's transcribed by an open source model running locally on the dungeon master's computer, which summarizes the night's session and prompts an image generator to create illustrations of key moments. These summaries and images are hilarious because they're full of errors. It's a bit of harmless fun, and it bring a small amount of additional pleasure to a small group of people. No one is going to fire an illustrator because D&D players are image-genning funny illustrations where seven-fingered paladins wrestle with orcs that have an extra hand.

But bosses have and will fire illustrators, because they fantasize about being able to dispense with creative professionals and just prompt an AI. Because even though the AI can't do the illustrator's job, an AI salesman can convince the illustrator's boss to fire them and replace them with an AI that can't do their job.

This is a disgusting and terrible juncture, and we should not simply shrug our shoulders and accept Thatcherism's fatalism: "There is no alternative."

So what is the alternative? A lot of artists and their allies think they have an answer: they say we should extend copyright to cover the activities associated with training a model.

And I'm here to tell you they are wrong: wrong because this would inflict terrible collateral damage on socially beneficial activities, and it would represent a massive expansion of copyright over activities that are currently permitted – for good reason!.

Let's break down the steps in AI training.

First, you scrape a bunch of web-pages. This is unambiguously legal under present copyright law. You do not need a license to make a transient copy of a copyrighted work in order to analyze it, otherwise search engines would be illegal. Ban scraping and Google will be the last search engine we ever get, the Internet Archive will go out of business, that guy in Austria who scraped all the grocery store sites and proved that the big chains were colluding to rig prices would be in deep trouble.

Next, you perform analysis on those works. Basically, you count stuff on them: count pixels and their colors and proximity to other pixels; or count words. This is obviously not something you need a license for. It's just not illegal to count the elements of a copyrighted work. And we really don't want it to be, not if you're interested in scholarship of any kind.

And it's important to note that counting things is legal, even if you're working from an illegally obtained copy. Like, if you go to the flea market, and you buy a bootleg music CD, and you take it home and you make a list of all the adverbs in the lyrics, and you publish that list, you are not infringing copyright by doing so.

Perhaps you've infringed copyright by getting the pirated CD, but not by counting the lyrics.

This is why Anthropic offered a $1.5b settlement for training its models based on a ton of books it downloaded from a pirate site: not because counting the words in the books infringes anyone's rights, but because they were worried that they were going to get hit with $150k/book statutory damages for downloading the files.

OK, after you count all the pixels or the words, it's time for the final step: publishing them. Because that's what a model is: a literary work (that is, a piece of software) that embodies a bunch of facts about a bunch of other works, word and pixel distribution information, encoded in a multidimensional array.

And again, copyright absolutely does not prohibit you from publishing facts about copyrighted works. And again, no one should want to live in a world where someone else gets to decide which truthful, factual statements you can publish.

But hey, maybe you think this is all sophistry. Maybe you think I'm full of shit. That's fine. It wouldn't be the first time someone thought that.

After all, even if I'm right about how copyright works now, there's no reason we couldn't change copyright to ban training activities, and maybe there's even a clever way to wordsmith the law so that it only catches bad things we don't like, and not all the good stuff that comes from scraping, analyzing and publishing.

Well, even then, you're not gonna help out creators by creating this new copyright. If you're thinking that you can, you need to grapple with this fact: we have monotonically expanded copyright since 1976, so that today, copyright covers more kinds of works, grants exclusive rights over more uses, and lasts longer.

And today, the media industry is larger and more profitable than it has ever been, and also: the share of media industry income that goes to creative workers is lower than its ever been, both in real terms, and as a proportion of those incredible gains made by creators' bosses at the media company.

So how it is that we have given all these new rights to creators, and those new rights have generated untold billions, and left creators poorer? It's because in a creative market dominated by five publishers, four studios, three labels, two mobile app stores, and a single company that controls all the ebooks and audiobooks, giving a creative worker extra rights to bargain with is like giving your bullied kid more lunch money.

It doesn't matter how much lunch money you give the kid, the bullies will take it all. Give that kid enough money and the bullies will hire an agency to run a global campaign proclaiming "think of the hungry kids! Give them more lunch money!"

Creative workers who cheer on lawsuits by the big studios and labels need to remember the first rule of class warfare: things that are good for your boss are rarely what's good for you.

The day Disney and Universal filed suit against Midjourney, I got a press release from the RIAA, which represents Disney and Universal through their recording arms. Universal is the largest label in the world. Together with Sony and Warner, they control 70% of all music recordings in copyright today.

It starts: "There is a clear path forward through partnerships that both further AI innovation and foster human artistry."

It ends: "This action by Disney and Universal represents a critical stand for human creativity and responsible innovation."

And it's signed by Mitch Glazier, CEO of the RIAA.

It's very likely that name doesn't mean anything to you. But let me tell you who Mitch Glazier is. Today, Mitch Glazier is the CEO if the RIAA, with an annual salary of $1.3m. But until 1999, Mitch Glazier was a key Congressional staffer, and in 1999, Glazier snuck an amendment into an unrelated bill, the Satellite Home Viewer Improvement Act, that killed musicians' right to take their recordings back from their labels.

This is a practice that had been especially important to "heritage acts" (which is a record industry euphemism for "old music recorded by Black people"), for whom this right represented the difference between making rent and ending up on the street.

When it became clear that Glazier had pulled this musician-impoverishing scam, there was so much public outcry, that Congress actually came back for a special session, just to vote again to cancel Glazier's amendment. And then Glazier was kicked out of his cushy Congressional job, whereupon the RIAA started paying more than $1m/year to "represent the music industry."

This is the guy who signed that press release in my inbox. And his message was: The problem isn't that Midjourney wants to train a Gen AI model on copyrighted works, and then use that model to put artists on the breadline. The problem is that Midjourney didn't pay RIAA members Universal and Disney for permission to train a model. Because if only Midjourney had given Disney and Universal several million dollars for training rights to their catalogs, the companies would have happily allowed them to train to their heart's content, and they would have bought the resulting models, and fired as many creative professionals as they could.

I mean, have we already forgotten the Hollywood strikes? I sure haven't. I live in Burbank, home to Disney, Universal and Warner, and I was out on the line with my comrades from the Writers Guild, offering solidarity on behalf of my union, IATSE 830, The Animation Guild, where I'm a member of the writers' unit.

And I'll never forget when one writer turned to me and said, "You know, you prompt an LLM exactly the same way an exec gives shitty notes to a writers' room. You know: 'Make me ET, except it's about a dog, and put a love interest in there, and a car chase in the second act.' The difference is, you say that to a writers' room and they all make fun of you and call you a fucking idiot suit. But you say it to an LLM and it will cheerfully shit out a terrible script that conforms exactly to that spec (you know, Air Bud)."

These companies are desperate to use AI to displace workers. When Getty Images sues AI companies, it's not representing the interests of photographers. Getty hates paying photographers! Getty just wants to get paid for the training run, and they want the resulting AI model to have guardrails, so it will refuse to create images that compete with Getty's images for anyone except Getty. But Getty will absolutely use its models to bankrupt as many photographers as it possibly can.

A new copyright to train models won't get us a world where models aren't used to destroy artists, it'll just get us a world where the standard contracts of the handful of companies that control all creative labor markets are updated to require us to hand over those new training rights to those companies. Demanding a new copyright just makes you a useful idiot for your boss, a human shield they can brandish in policy fights, a tissue-thin pretense of "won't someone think of the hungry artists?"

When really what they're demanding is a world where 30% of the investment capital of the AI companies go into their shareholders' pockets. When an artist is being devoured by rapacious monopolies, does it matter how they divvy up the meal?

We need to protect artists from AI predation, not just create a new way for artists to be mad about their impoverishment.

And incredibly enough, there's a really simple way to do that. After 20+ years of being consistently wrong and terrible for artists' rights, the US Copyright Office has finally done something gloriously, wonderfully right. All through this AI bubble, the Copyright Office has maintained – correctly – that AI-generated works cannot be copyrighted, because copyright is exclusively for humans. That's why the "monkey selfie" is in the public domain. Copyright is only awarded to works of human creative expression that are fixed in a tangible medium.

And not only has the Copyright Office taken this position, they've defended it vigorously in court, repeatedly winning judgments to uphold this principle.

The fact that every AI created work is in the public domain means that if Getty or Disney or Universal or Hearst newspapers use AI to generate works – then anyone else can take those works, copy them, sell them, or give them away for free. And the only thing those companies hate more than paying creative workers, is having other people take their stuff without permission.

The US Copyright Office's position means that the only way these companies can get a copyright is to pay humans to do creative work. This is a recipe for centaurhood. If you're a visual artist or writer who uses prompts to come up with ideas or variations, that's no problem, because the ultimate work comes from you. And if you're a video editor who uses deepfakes to change the eyelines of 200 extras in a crowd-scene, then sure, those eyeballs are in the public domain, but the movie stays copyrighted.

But creative workers don't have to rely on the US government to rescue us from AI predators. We can do it ourselves, the way the writers did in their historic writers' strike. The writers brought the studios to their knees. They did it because they are organized and solidaristic, but also are allowed to do something that virtually no other workers are allowed to do: they can engage in "sectoral bargaining," whereby all the workers in a sector can negotiate a contract with every employer in the sector.

That's been illegal for most workers since the late 1940s, when the Taft-Hartley Act outlawed it. If we are gonna campaign to get a new law passed in hopes of making more money and having more control over our labor, we should campaign to restore sectoral bargaining, not to expand copyright.

Our allies in a campaign to expand copyright are our bosses, who have never had our best interests at heart. While our allies in the fight for sector bargaining are every worker in the country. As the song goes, "Which side are you on?"

OK, I need to bring this talk in for a landing now, because I'm out of time, so I'm going to close out with this: AI is a bubble and bubbles are terrible.

Bubbles transfer the life's savings of normal people who are just trying to have a dignified retirement to the wealthiest and most unethical people in our society, and every bubble eventually bursts, taking their savings with it.

But not every bubble is created equal. Some bubbles leave behind something productive. Worldcom stole billions from everyday people by defrauding them about orders for fiber optic cables. The CEO went to prison and died there. But the fiber outlived him. It's still in the ground. At my home, I've got 2gb symmetrical fiber, because AT&T lit up some of that old Worldcom dark fiber.

All things being equal, it would have been better if Worldcom hadn't ever existed, but the only thing worse than Worldcom committing all that ghastly fraud would be if there was nothing to salvage from the wreckage.

I don't think we'll salvage much from cryptocurrency, for example. Sure, there'll be a few coders who've learned something about secure programming in Rust. But when crypto dies, what it will leave behind is bad Austrian economics and worse monkey JPEGs.

AI is a bubble and it will burst. Most of the companies will fail. Most of the data-centers will be shuttered or sold for parts. So what will be left behind?

We'll have a bunch of coders who are really good at applied statistics. We'll have a lot of cheap GPUs, which'll be good news for, say, effects artists and climate scientists, who'll be able to buy that critical hardware at pennies on the dollar. And we'll have the open source models that run on commodity hardware, AI tools that can do a lot of useful stuff, like transcribing audio and video, describing images, summarizing documents, automating a lot of labor-intensive graphic editing, like removing backgrounds, or airbrushing passersby out of photos. These will run on our laptops and phones, and open source hackers will find ways to push them to do things their makers never dreamt of.

If there had never been an AI bubble, if all this stuff arose merely because computer scientists and product managers noodled around for a few years coming up with cool new apps for back-propagation, machine learning and generative adversarial networks, most people would have been pleasantly surprised with these interesting new things their computers could do. We'd call them "plugins."

It's the bubble that sucks, not these applications. The bubble doesn't want cheap useful things. It wants expensive, "disruptive" things: Big foundation models that lose billions of dollars every year.

When the AI investment mania halts, most of those models are going to disappear, because it just won't be economical to keep the data-centers running. As Stein's Law has it: "Anything that can't go on forever eventually stops."

The collapse of the AI bubble is going to be ugly. Seven AI companies currently account for more than a third of the stock market, and they endlessly pass around the same $100b IOU.

Bosses are mass-firing productive workers and replacing them with janky AI, and when the janky AI is gone, no one will be able to find and re-hire most of those workers, we're going to go from disfunctional AI systems to nothing.

AI is the asbestos in the walls of our technological society, stuffed there with wild abandon by a finance sector and tech monopolists run amok. We will be excavating it for a generation or more.

So we need to get rid of this bubble. Pop it, as quickly as we can. To do that, we have to focus on the material factors driving the bubble. The bubble isn't being driven by deepfake porn, or election disinformation, or AI image-gen, or slop advertising.

All that stuff is terrible and harmful, but it's not driving investment. The total dollar figure represented by these apps doesn't come close to making a dent in the capital expenditures and operating costs of AI. They are peripheral, residual uses: flashy, but unimportant to the bubble.

Get rid of all those uses and you reduce the expected income of AI companies by a sum so small it rounds to zero.

Same goes for all that "AI Safety" nonsense, that purports to concern itself with preventing an AI from attaining sentience and turning us all into paperclips. First of all, this is facially absurd. Throwing more words and GPUs into the word-guessing program won't make it sentient. That's like saying, "Well, we keep breeding these horses to run faster and faster, so it's only a matter of time until one of our mares gives birth to a locomotive." A human mind is not a word-guessing program with a lot of extra words.

I'm here for science fiction thought experiments, don't get me wrong. But also, don't mistake sf for prophesy. SF stories about superintelligence are futuristic parables, not business plans, roadmaps, or predictions.

The AI Safety people say they are worried that AI is going to end the world, but AI bosses love these weirdos. Because on the one hand, if AI is powerful enough to destroy the world, think of how much money it can make! And on the other hand, no AI business plan has a line on its revenue projections spreadsheet labeled "Income from turning the human race into paperclips." So even if we ban AI companies from doing this, we won't cost them a dime in investment capital.

To pop the bubble, we have to hammer on the forces that created the bubble: the myth that AI can do your job, especially if you get high wages that your boss can claw back; the understanding that growth companies need a succession of ever-more-outlandish bubbles to stay alive; the fact that workers and the public they serve are on one side of this fight, and bosses and their investors are on the other side.

Because the AI bubble really is very bad news, it's worth fighting seriously, and a serious fight against AI strikes at its roots: the material factors fueling the hundreds of billions in wasted capital that are being spent to put us all on the breadline and fill all our walls with high-tech asbestos.

(Image: Cryteria, CC BY 3.0, modified)

Hey look at this (permalink)

Politics and Capitalist Stagnation https://www.unpopularfront.news/p/politics-and-capitalist-stagnation
An Analysis of the Proposed Spirit Financial-Credit Union 1 Merger. The Consequences for the Credit Union System https://chipfilson.com/2025/12/an-analysis-of-the-proposed-spirit-financal-credit-union-1-merger/
Zillow deletes climate risk data from listings after complaints it harms sales https://www.theguardian.com/environment/2025/dec/01/zillow-removes-climate-risk-data-home-listings
After Years of Controversy, the EU’s Chat Control Nears Its Final Hurdle: What to Know https://www.eff.org/deeplinks/2025/12/after-years-controversy-eus-chat-control-nears-its-final-hurdle-what-know
How the dollar-store industry overcharges cash-strapped customers while promising low prices https://www.theguardian.com/us-news/2025/dec/03/customers-pay-more-rising-dollar-store-costs

Object permanence (permalink)

#20yrsago Haunted Mansion papercraft model adds crypts and gates https://www.haunteddimensions.raykeim.com/index313.html

#20yrsago Print your own Monopoly money https://web.archive.org/web/20051202030047/http://www.hasbro.com/monopoly/pl/page.treasurechest/dn/default.cfm

#15yrsago Bunnie explains the technical intricacies and legalities of Xbox hacking https://www.bunniestudios.com/blog/2010/usa-v-crippen-a-retrospective/

#15yrsago How Pac Man’s ghosts decide what to do: elegant complexity https://web.archive.org/web/20101205044323/https://gameinternals.com/post/2072558330/understanding-pac-man-ghost-behavior

#15yrsago Glorious, elaborate, profane insults of the world https://www.reddit.com/r/AskReddit/comments/efee7/what_are_your_favorite_culturally_untranslateable/?sort=confidence

#15yrsago Walt Disney World castmembers speak about their search for a living wage https://www.youtube.com/watch?v=f5BMQ3xQc7o

#15yrsago Wikileaks cables reveal that the US wrote Spain’s proposed copyright law https://web.archive.org/web/20140723230745/https://elpais.com/elpais/2010/12/03/actualidad/1291367868_850215.html

#15yrsago Cities made of broken technology https://web.archive.org/web/20101203132915/https://agora-gallery.com/artistpage/Franco_Recchia.aspx

#10yrsago The TPP’s ban on source-code disclosure requirements: bad news for information security https://www.eff.org/deeplinks/2015/12/tpp-threatens-security-and-safety-locking-down-us-policy-source-code-audit

#10yrsago Fossil fuel divestment sit-in at MIT President’s office hits 10,000,000,000-hour mark https://twitter.com/FossilFreeMIT/status/672526210581274624

#10yrsago Hacker dumps United Arab Emirates Invest Bank’s customer data https://www.dailydot.com/news/invest-bank-hacker-buba/

#10yrsago Illinois prisons spy on prisoners, sue them for rent on their cells if they have any money https://www.chicagotribune.com/2015/11/30/state-sues-prisoners-to-pay-for-their-room-board/

#10yrsago Free usability help for privacy toolmakers https://superbloom.design/learning/blog/apply-for-help/

#10yrsago In the first 334 days of 2015, America has seen 351 mass shootings (and counting) https://web.archive.org/web/20151209004329/https://www.washingtonpost.com/news/wonk/wp/2015/11/30/there-have-been-334-days-and-351-mass-shootings-so-far-this-year/

#10yrsago Not even the scapegoats will go to jail for BP’s murder of the Gulf Coast https://arstechnica.com/tech-policy/2015/12/manslaughter-charges-dropped-in-bp-spill-case-nobody-from-bp-will-go-to-prison/

#10yrsago Urban Transport Without the Hot Air: confusing the issue with relevant facts! https://memex.craphound.com/2015/12/03/urban-transport-without-the-hot-air-confusing-the-issue-with-relevant-facts/

#5yrsago Breathtaking Iphone hack https://pluralistic.net/2020/12/03/ministry-for-the-future/#awdl

#5yrsago Graffitists hit dozens of NYC subway cars https://pluralistic.net/2020/12/03/ministry-for-the-future/#getting-up

#5yrsago The Ministry For the Future https://pluralistic.net/2020/12/03/ministry-for-the-future/#ksr

#5yrsago Monopolies made America vulnerable to covid https://pluralistic.net/2020/12/03/ministry-for-the-future/#big-health

#5yrsago Section 230 is Good, Actually https://pluralistic.net/2020/12/04/kawaski-trawick/#230

#5yrsago Postmortem of the NYPD's murder of a Black man https://pluralistic.net/2020/12/04/kawaski-trawick/#Kawaski-Trawick

#5yrsago Student debt trap https://pluralistic.net/2020/12/04/kawaski-trawick/#strike-debt

#1yrago "That Makes Me Smart" https://pluralistic.net/2024/12/04/its-not-a-lie/#its-a-premature-truth

#1yrago Canada sues Google https://pluralistic.net/2024/12/03/clementsy/#can-tech

Upcoming appearances (permalink)

Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html
Denver: Enshittification at Tattered Cover Colfax, Jan 22
https://www.eventbrite.com/e/cory-doctorow-live-at-tattered-cover-colfax-tickets-1976644174937

Recent appearances (permalink)

Enshittification (Future Knowledge)
https://futureknowledge.transistor.fm/episodes/enshittification
We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w
How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s
Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/
Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

Wed, 03 Dec 2025 18:05:02 +0000 Fullscreen Open in Tab

Pluralistic: A year in illustration (2025 edition) (03 Dec 2025)

Today's links

A year in illustration (2025 edition): I think I'm getting the hang of this?
Hey look at this: Delights to delectate.
Object permanence: HADOPI is born; Tea Party wants to disenfranchise renters; How to kill TPP; Mozilla ejects Thunderbird; Rosa Parks was a lifelong radical activist.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

A year in illustration (2025 edition) (permalink)

One of the most surprising professional and creative developments of my middle-age has been discovering my love of collage. I have never been a "visual" person – I can't draw, I can't estimate whether a piece of furniture will fit in a given niche, I can't catch a ball, and I can't tell you if a picture is crooked.

When Boing Boing started including images with our posts in the early 2000s, I hated it. It was such a chore to find images that were open licensed or public domain, and so many of the subjects I wrote about are abstract and complex and hard to illustrate. Sometimes, I'd come up with a crude visual gag and collage together a few freely usable images as best as I could and call it a day.

But over the five years that I've been writing Pluralistic, I've found myself putting more and more effort and thought into these header images. Without realizing it, I put more and more time into mastering The GIMP (a free/open Photoshop alternative), watching tutorial videos and just noodling from time to time. I also discovered many unsuspected sources of public domain work, such as the Library of Congress, whose search engine sucks, but whose collection is astounding (tip: use Kagi or Google to search for images with the "site:loc.gov" flag).

I also discovered the Met's incredible collection:

https://www.metmuseum.org/art/collection/search

And the archives of H Armstrong Roberts, an incredibly prolific stock photographer whose whole corpus is in the public domain. You can download more than 14,000 of his images from the Internet Archive (I certainly did!):

https://archive.org/details/h-armstrong-roberts

Speaking of the Archive and search engine hacks, I've also developed a method for finding hi-rez images that are otherwise very hard to get. Often, an image search will turn up public domain results on commercial stock sites like Getty. If I can't find public domain versions elsewhere (e.g. by using Tineye reverse-image search), I look for Getty's metadata about the image's source (that is, which book or collection it came from). Then I search the Internet Archive and other public domain repositories for high-rez PDF scans of the original work, and pull the images out of there. Many of my demons come from Compendium rarissimum totius Artis Magicae sistematisatae per celeberrimos Artis hujus Magistros, an 18th century updating of a 11th century demonolgy text, which you can get as a hi-rez at the Wellcome Trust:

https://wellcomecollection.org/works/cvnpwy8d

Five years into my serious collage phase, I find myself increasingly pleased with the work I'm producing. I actually self-published a little book of my favorites this year (Canny Valley), which Bruce Sterling provided an intro for and which the legendary book designed John Berry laid out fot me, and I'm planning future volumes:

https://pluralistic.net/2025/09/04/illustrious/#chairman-bruce

I've been doing annual illustration roundups for the past several years, selecting my favorites from the year's crop:

2022:
https://pluralistic.net/2022/12/25/a-year-in-illustration/

2023:
https://pluralistic.net/2023/12/21/collages-r-us/

2024:
https://pluralistic.net/2024/12/07/great-kepplers-ghost/

It's a testament to how much progress I've made that when it came time to choose this year's favorites, I had 33 images I wanted to highlight. Much of this year's progress is down to my friend and neighbor Alistair Milne, an extremely talented artist and commercial illustrator who has periodically offered me little bits of life-changing advice on composition and technique.

I've also found a way to use these images in my talks: I've pulled together a slideshow of my favorite (enshittification-related) images, formatted for 16:9 (the incredibly awkward aspect ratio that everyone seems to expect these days), with embedded Creative Commons attributions. When I give a talk, I ask to have this run behind me in "kiosk mode," looping with a 10-second delay between each slide. Here's an up-to-date (as of today) version:

https://archive.org/download/enshittification-slideshow/enshittification.pptx

If these images intrigue you and you'd like hi-rez versions to rework on your own, you can get full rez versions of all my blog collagesin my "Pluralistic Collages" Flickr set:

https://www.flickr.com/photos/doctorow/albums/72177720316719208

They're licensed CC BY-SA 4.0, though some subelements may be under different licenses (check the image descriptions for details). But everything is licensed for remix and commercial distribution, so go nuts!

All the books I reviewed in 2025

The underlying image comes from the Library of Congress (a search for "reading + book") (because "reading" turns up pictures of Reading, PA and Reading, UK). I love the poop emoji from the cover of the US edition of Enshittification and I'm hoping to get permission to do a lot more with it.

https://pluralistic.net/2025/12/02/constant-reader/#too-many-books

Meta's new top EU regulator is contractually prohibited from saying mean things about Meta

Mark Zuckerberg's ghastly Metaverse avatar is such a gift to his critics. I can't believe his comms team let him release it! The main image is an H Armstrong Roberts classic of a beat cop wagging his finger at a naughty lad on a bicycle. The Wachowskis' 'code waterfall' comes from this generator:

https://github.com/yeaayy/the-matrix

https://pluralistic.net/2025/12/01/erin-go-blagged/#big-tech-omerta

The long game

In my intro to last year's roundup, I wrote about Joseph Keppler, the incredibly prolific illustrator and publisher who founded Puck magazine and drew hundreds of illustrations, many of them editorial cartoons that accompanied articles that criticized monopolies and America's oligarch class. As with so much of his work, Keppler's classic illustration of Rockefeller as a shrimpy, preening king updates very neatly to today's context, through the simple expedient of swapping in Zuck's metaverse avatar.

https://pluralistic.net/2025/11/20/if-you-wanted-to-get-there/#i-wouldnt-start-from-here

Facebook's fraud files

I love including scanned currency in my illustrations. Obviously, large-denomination bills make for great symbols in posts about concentrated wealth and power, but also, US currency is iconic, covered in weird illustrations, and available as incredibly high-rez scans, like this 7,300+ pixel-wide C-note:

https://commons.wikimedia.org/wiki/File:U.S._hundred_dollar_bill,_1999.jpg

It turns out that intaglio shading does really cool stuff when you tweak the curves. I love what happened to Ben Franklin's eyes in this one. (Zuck's body is another Keppler/Puck illo!)

https://pluralistic.net/2025/11/08/faecebook/#too-big-to-care

There's one thing EVERY government can do to shrink Big Tech

This is another Keppler/Roberts mashup. Keppler's original is Teddy Roosevelt as a club-wielding ("speak softly and carry a big stick") trustbusting Goliath. The crying baby and money come from an H Armstrong Roberts tax-protest stock photo (one of the money sacks was originally labeled "TAXES"). This one also includes one of my standbys, Cryteria's terrific vector image of HAL 9000's glaring red eye, always a good symbolic element for stories about Big Tech, surveillance, and/or AI:

https://commons.wikimedia.org/wiki/File:HAL9000.svg

https://pluralistic.net/2025/11/01/redistribution-vs-predistribution/#elbows-up-eurostack

When AI prophecy fails

The chain-gang photo comes from the Library of Congress. That hacker hoodie is a public domain graphic ganked from Wikimedia Commons. I love how the HAL 9000 eye pops as the only color element in this one.

https://pluralistic.net/2025/10/29/worker-frightening-machines/#robots-stole-your-jerb-kinda

Checking in on the state of Amazon's chickenized reverse-centaurs

Another H Armstrong Roberts remix: originally, this was a grinning delivery man jugging several parcels. I reskinned him and his van with Amazon delivery livery, and matted in the horse-head to create a "reverse centaur" (another theme I return to often). I used one of Alistair Milne's tips to get that horse's head right: rather than trying to trace all the stray hairs on the mane, I traced them with a fine brush tool on a separate layer, then erased the strays from the original and merged down to get a nice, transparency-enabled hair effect.

https://pluralistic.net/2025/10/23/traveling-salesman-solution/#pee-bottles

The mad king's digital killswitch

The Uncle Sam image is Keppler's (who else?). In the original (which is about tariffs! everything old is new!), Sam's legs have become magnets that are drawing in people and goods from all over the world. The Earth-from-space image is a NASA pic. Love that all works of federal authorship are born in the public domain!

https://pluralistic.net/2025/10/20/post-american-internet/#huawei-with-american-characteristics

Microsoft, Tear Down That Wall!

Clippy makes a perfect element for posts about chatbots. It's hard to think that Microsoft shipped a product with such a terrible visual design, but at the same time, I gotta give 'em credit, it's so awful that it's still instantly recognizable, 25 years later.

https://pluralistic.net/2025/10/15/freedom-of-movement/#data-dieselgate

A disenshittification moment from the land of mass storage

Another remix of Keppler's excellent Teddy Roosevelt/trustbuster giant image, this time with Ben Franklin's glorious C-note phiz. God, I love using images from money!

https://pluralistic.net/2025/10/10/synology/#how-about-nah

Apple's unlawful evil

Alistair Milne helped me work up a super hi-rez version of Trump's hair from his official (public domain) 2024 presidential portrait. Lots of tracing those fine hairs, and boy does it pay off. Apple's "Think Different" wordmark (available as a vector on Wikimedia Commons) is a gift to the company's critics. The fact that the NYPD actually routinely show up for protests dressed like this makes my job too easy.

https://pluralistic.net/2025/10/06/rogue-capitalism/#orphaned-syrian-refugees-need-not-apply

Blue Bonds

Another C-note remix. One of the things I love about remixing US currency is that every part of it is so immediately identifiable, meaning that just about any crop works. The California bear comes from a public domain vector on Wikimedia Commons. I worked hard to get the intaglio effect to transfer to the bear, but only with middling success. Thankfully, I was able to work at massive resolution (like, 4,000 px wide) and reduce the image, which hides a lot of my mistakes.

https://pluralistic.net/2025/10/04/fiscal-antifa/#post-trump

The real (economic) AI apocalypse is nigh

Another money scan, this time a hyperinflationary Zimbabwean dollar (I also looked at some Serbian hyperinflationary notes, but the Zimbabwean one was available at a higher rez). Not thrilled about the engraving texture on the HAL 9000, but the Sam Altman intaglio kills. I spent a lot of time tweaking that using G'mic, a good (but uneven) plugin suite for the GIMP.

https://pluralistic.net/2025/09/27/econopocalypse/#subprime-intelligence

Rage Against the (Algorithmic Management) Machine

This one made this year's faves list purely because I was so happy with how the Doordash backpack came out. The belligerent worker is part of a Keppler diptych showing a union worker and a boss facing off against one another with a cowering consumer caught in the crossfire. I'm not thrilled about this false equivalence, but I'll happily gank the figures, which are great.

https://pluralistic.net/2025/09/25/roboboss/#counterapps

The enshittification of solar (and how to stop it)

I spent a lot of time tweaking the poop emoji on those solar panels, eventually painstakingly erasing the frames from the overlay image. It was worth it.

https://pluralistic.net/2025/09/23/our-friend-the-electron/#to-every-man-his-castle

AI psychosis and the warped mirror

One of those high-concept images that came out perfect. Replacing Narcissus's face (and reflection) with HAL 9000 made for a striking image that only took minutes to turn out.

https://pluralistic.net/2025/09/17/automating-gang-stalking-delusion/#paranoid-androids

Reverse centaurs are the answer to the AI paradox

The businessman trundling up a long concrete staircase is another H Armstrong Roberts. That staircase became very existential as soon as I stripped out the grass on either side of it. Finding that horse-head took a lot of doing (the world needs more CC-licensed photos of horses from that angle!). The computer in the background comes from a NASA Ames archive of photos of all kinds of cool stuff – zeppelins, spacesuits, and midcentury "supercomputers."

https://pluralistic.net/2025/09/11/vulgar-thatcherism/#there-is-an-alternative

Radical juries

Another high-concept image that just worked. It took me more time to find a good public domain oil painting of a jury than it did to transform each juror into Karl Marx. I love how this looks.

https://pluralistic.net/2025/08/22/jury-nullification/#voir-dire

LLMs are slot-machines

It's surprisingly hard to find a decent public domain photo of a slot machine in use. I eventually started to wonder if Vegas had a no-cameras policy in the early years. Eventually, the Library of Commerce came through with a scanned neg that was high enough rez that I could push the elements I wanted to have stand out from an otherwise muddy, washed-out image.

https://pluralistic.net/2025/08/16/jackpot/#salience-bias

Zuckermuskian solipsism

The laborers come from an LoC collection of portraits of children who worked in coal mines in the 1910s. They're pretty harrowing stuff. I spent a long plane ride cropping each individual out of several of these images.

https://pluralistic.net/2025/08/18/seeing-like-a-billionaire/#npcs

Good ideas are popular

The original crowd scene (a presidential inauguration, if memory serves) was super high-rez, which made it very easy to convincingly matte in the monkeys and the Congressional dome. I played with tinting this one, but pure greyscale looked a lot better.

https://pluralistic.net/2025/08/07/the-people-no-2/#water-flowing-uphill

By all means, tread on those people

Another great high concept. The wordiness of Wilhoit's Law makes this intrinsically funny. There's a public domain vector-art Gadsen flag on Wikimedia Commons. I found a Reddit forum where font nerds had sleuthed out the typeface for the words on the original.

https://pluralistic.net/2025/08/26/sole-and-despotic-dominion/#then-they-came-for-me

AI's pogo-stick grift

The pogo stick kid is another H Armstrong Roberts gank. I spent ages trying to get the bounce effect to look right, and then Alistair Milne fixed it for me in like 10 seconds. The smoke comes from an oil painting of the eruption of Vesuvius from the Met. It's become my go-to "hellscape" background.

https://pluralistic.net/2025/08/02/inventing-the-pedestrian/#three-apis-in-a-trenchcoat

The worst possible antitrust outcome

The smoke from Vesuvius makes another appearance. I filled the Android droid with tormented figures from Bosch's "Garden of Earthly Delights," which is an amazing painting that is available as a more than 15,000 pixel wide (!) scan on Wikimedia Commons.

https://pluralistic.net/2025/09/03/unpunishing-process/#fucking-shit-goddammit-fuck

Conservatism considered as a movement of bitter rubes

Boy, I love this one. The steamship image is from the Met. The carny barker is a still of WC Fields, whose body language is impeccable. It took a long-ass time to get a MAGA hat in the correct position, but I eventually found a photo of an early 20th C baseball player and then tinted his hat and matted in the MAGA embroidery.

https://pluralistic.net/2025/07/22/all-day-suckers/#i-love-the-poorly-educated

Your Meta AI prompts are in a live, public feed

These guys on the sofa come from Thomas Hawke, who has recovered and scanned nearly 30,000 "found photos" – collections from estates, yard-sales, etc:

https://www.flickr.com/search/?sort=date-taken-desc&safe_search=1&tags=foundphotograph&user_id=51035555243%40N01&view_all=1

The Shining-esque lobby came from the Library of Congress, where it is surprisingly easy to find images of buildings with scary carpets.

https://pluralistic.net/2025/06/19/privacy-breach-by-design/#bringing-home-the-beacon

Strange Bedfellows and Long Knives

Another great high-concept that turned out great. I think that matting the Heritage Foundation chiselwork into the background really pulls it together, and I'm really happy with the glow-up I did for the knives.

https://pluralistic.net/2025/05/21/et-tu-sloppy-steve/#fractured-fairytales

Are the means of computation even seizable?

I spent so long cutting out this old printing press, but boy has it stood me in good stead. I think there's like five copies of that image layered on top of each other here. The figure is an inside joke for all my Luddite trufan pals outthere, a remix of a classic handbill depicting General Ned Ludd.

https://pluralistic.net/2025/05/14/pregnable/#checkm8

Mark Zuckerberg announces mind-control ray (again)

I was worried that this wouldn't work unless you were familiar with the iconic portrait photo of Rasputin, but that guy was such a creepy-ass-looking freak, and Zuck's metaverse avatar is so awful, that it works on its own merits, too.

https://pluralistic.net/2025/05/07/rah-rah-rasputin/#credulous-dolts

Mike Lee and Jim Jordan want to kill the law that bans companies from cheating you

The original image was so grainy, but it was also fantastic and I spent hours rehabbing it. It's a posed, comedic photo of two Australian miners in the bush cheating at cards, rooking a third man. The Uncle Sam is (obviously) from Keppler.

https://pluralistic.net/2025/04/29/cheaters-and-liars/#caveat-emptor-brainworms

Mark Zuckerberg personally lost the Facebook antitrust case

This one got more, "Wow is that ever creepy" comments than any of the other ones. I was going for Chatty Cathy, but that Zuck metaverse avatar is so weird and bad that it acts like visual MSG in any image, amplifying its creepiness to incredible heights.

https://pluralistic.net/2025/04/18/chatty-zucky/#is-you-taking-notes-on-a-criminal-fucking-conspiracy

Machina economicus

The image is from an early illustrated French edition of HG Wells's War of the Worlds. I love how this worked out, and a family of my fans in Ireland commissioned a paint-by-numbers of it and painted it in and mailed it to me. It's incredible. If I re-use this, I will probably swap out the emoji for the graphic from the book's cover.

https://pluralistic.net/2025/04/14/timmy-share/#a-superior-moral-justification-for-selfishness

How the world's leading breach expert got phished

I don't understand how composition works, but I know when I've lucked into a good composition. This is a good composition! I made this on the sofa of Doc and Joyce Searles in Bloomington, Indiana while I was in town for my Picks and Shovels book tour.

https://pluralistic.net/2025/04/05/troy-hunt/#teach-a-man-to-phish

Anyone who trusts an AI therapist needs their head examined

I worked those tentacles for so long, trying to get Freud/Cthulhu/HAL's lower half just right. In the end, it all paid off.

https://pluralistic.net/2025/04/01/doctor-robo-blabbermouth/#fool-me-once-etc-etc

You can't save an institution by betraying its mission

The "fireman" is an image from the Department of Defense of a soldier demoing a flamethrower (I hacked in the firefighter's uniform). I spent a lot of time trying to get a smoky look for the foreground here, but I don't think it succeeded.

https://pluralistic.net/2025/03/19/selling-out/#destroy-the-village-to-save-it

Trump loves Big Tech

The two guys in the jars (John Bull and a random general I've rebadged to represent the EU) come from an epic Keppler two-page spread personifying the nations of the world as foolish military men. While many of the figures are sadly and predictably racist (you don't want to see "China"), these guys were eminently salvageable, and I love their expressions and body-language.

https://pluralistic.net/2025/03/24/whats-good-for-big-tech/#is-good-for-america

The future of Amazon coders is the present of Amazon warehouse workers

The background is a photo of the interior of a tape-robot that I snapped in the data-centre at the Human Genome Project when I was out on assignment for Nature magazine. It remains one of the most striking images I've ever captured. It was way too hard to find a horse's head from that angle for the "reverse centaur." If there are any equestrian photographers out there, please consider snapping a couple and putting them up on Wikimedia Commons under a Creative Commons license.

https://pluralistic.net/2025/03/13/electronic-whipping/#youre-next

Gandersauce

I'm not thrilled with how the face worked out on this one, but people love it. If I'm giving a speech and I notice the audience elbowing one another and pointing at the slides and giggling, I know this one has just rotated onto the screen.

https://pluralistic.net/2025/03/08/turnabout/#is-fair-play

Premature Internet Activists

I spent a lot of time cleaning up and keystoning Woody Guthrie's original sticker, which can be found at very high resolutions online. Look for this element to find its way into many future collages.

https://pluralistic.net/2025/02/13/digital-rights/#are-human-rights

It's not a crime if we do it with an app

The two figures come from Keppler; the potato field is from the Library of Congress. Putting HAL eyes on the potatoes was fiddly work, but worth it. Something about Keppler's body language and those potato heads really sings.

https://pluralistic.net/2025/01/25/potatotrac/#carbo-loading

The cod-Marxism of personalized pricing

I don't often get a chance to use Chinese communist propaganda posters, but I love working with them. All public domain, available at high rez, and always to the point. It was a lot of work matting those US flags onto the partially furled Chinese flags, but it worked out great.

https://pluralistic.net/2025/01/11/socialism-for-the-wealthy/#rugged-individualism-for-the-poor

Occupy the Democratic National Committee

I love this sad donkey, from an old political cartoon. Given the state of the Democratic Party, I get a lot of chances to use him, and more's the pity.

https://pluralistic.net/2025/01/10/smoke-filled-room-where-it-happens/#dinosaurs

Social media needs (dumpster) fire exits

This one's actually from 2024, but I did it after last year's roundup, and I like it well enough to include it in this year's. I think the smoke came out pretty good!

https://pluralistic.net/2024/12/14/fire-exits/#graceful-failure-modes

(Images: TechCrunch, Ajay Suresh, Steve Jurvetson, CC BY 2.0; Cryteria, UK Parliament/Maria Unger, CC BY 3.0; Bastique, Frank Schwichtenberg, CC BY 4.0; Japanexperterna.se, CC BY-SA 2.0; Ser Amantio di Nicolao, CC BY-SA 3.0; Armin Kübelbeck, Zde, Felix Winkelnkemper, CC BY-SA 4.0; modified)

Hey look at this (permalink)

David Byrne: Tiny Desk Concert https://www.youtube.com/watch?v=PNmcF6eRE7w
Hundreds of Porsche Owners in Russia Unable to Start Cars After System Failure https://www.themoscowtimes.com/2025/12/02/hundreds-of-porsche-owners-in-russia-unable-to-start-cars-after-system-failure-a91302
The Enshittification of Plex Is Kicking Off, Starting with Free Roku Users https://gizmodo.com/the-enshittification-of-plex-is-kicking-off-starting-with-free-roku-users-2000694283
What Will Enter the Public Domain in 2026? https://publicdomainreview.org/features/entering-the-public-domain/2026/
Mastodon CEO change, 2026 reset https://www.manton.org/2025/12/02/mastodon-ceo-change-reset.html

Object permanence (permalink)

#20yrsago Sony Rootkit Roundup IV https://memex.craphound.com/2005/12/02/sony-rootkit-roundup-iv/

#20yrsago How can you tell if a CD is infectious? https://web.archive.org/web/20051205043456/https://www.eff.org/deeplinks/archives/004228.php

#20yrsago France about to get worst copyright law in Europe? https://web.archive.org/web/20060111033356/http://eucd.info/index.php?2005/11/14/177-droit-d-auteur-eucdinfo-devoile-le-plan-d-attaque-des-majors

#15yrsago UNC team builds 3D model of Rome using Flickr photos on a single PC in one day https://readwrite.com/flickr_rome_3d_double-time/

#15yrsago Schneier’s modest proposal: Close the Washington monument! https://www.schneier.com/essays/archives/2010/12/close_the_washington.html

#15yrsago Tea Party Nation President proposes taking vote away from tenants https://web.archive.org/web/20101204012806/https://thinkprogress.org/2010/11/30/tea-party-voting-property/

#15yrsago What it’s like to be a cocaine submarine captain https://web.archive.org/web/20120602082933/https://www.spiegel.de/international/world/the-colombian-coke-sub-former-drug-smuggler-tells-his-story-a-732292.html

#10yrsago A profile of America’s killingest cops: the police of Kern County, CA https://www.theguardian.com/us-news/2015/dec/01/the-county-kern-county-deadliest-police-killings

#10yrsago The word “taser” comes from an old racist science fiction novel https://www.theguardian.com/commentisfree/2015/nov/30/history-of-word-taser-comes-from-century-old-racist-science-fiction-novel

#10yrsago HOWTO pack a suit so it doesn’t wrinkle https://www.youtube.com/watch?v=ug58yeMqNCo

#10yrsago Newly discovered WEB Du Bois science fiction story reveals more Afrofuturist history https://slate.com/technology/2015/12/the-princess-steel-a-recently-uncovered-short-story-by-w-e-b-du-bois-and-afrofuturism.html

#10yrsago A roadmap for killing TPP: the next SOPA uprising! https://www.eff.org/deeplinks/2015/12/tpp-current-state-play-how-we-defeat-largest-trade-deal

#10yrsago Wikipedia Russia suspends editor who tried to cut deal with Russian authorities https://www.themoscowtimes.com/archive/russian-wikipedia-suspends-editor-who-cut-deal-with-authorities

#10yrsago Vtech toy data-breach gets worse: 6.3 million children implicated https://web.archive.org/web/20151204033429/https://motherboard.vice.com/read/hacked-toymaker-vtech-admits-breach-actually-hit-63-million-children

#10yrsago Ironically, modern surveillance states are baffled by people who change countries https://memex.craphound.com/2015/12/02/ironically-modern-surveillance-states-are-baffled-by-people-who-change-countries/

#10yrsago Mozilla will let go of Thunderbird https://techcrunch.com/2015/11/30/thunderbird-flies-away-from-mozilla/

#10yrsago Rosa Parks was a radical, lifelong black liberation activist, not a “meek seamstress” https://web.archive.org/web/20151208224937/https://www.washingtonpost.com/posteverything/wp/2015/12/01/how-history-got-the-rosa-parks-story-wrong/

#10yrsago Racist algorithms: how Big Data makes bias seem objective https://www.fordfoundation.org/news-and-stories/stories/can-computers-be-racist-big-data-inequality-and-discrimination/

#5yrsago Nalo Hopkinson, Science Fiction Grand Master https://pluralistic.net/2020/12/02/in-the-ring/#go-nalo-go

#1yrago All the books I reviewed in 2024 https://pluralistic.net/2024/12/02/booklish/#2024-in-review

Upcoming appearances (permalink)

Seattle: Neuroscience, AI and Society (University of Washington), Dec 4
https://www.eventbrite.com/e/neuroscience-ai-and-society-cory-doctorow-tickets-1735371255139
Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html

Recent appearances (permalink)

We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w
How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s
Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/
Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8
How the internet went to sh*t (Prospect Magazine)
https://www.prospectmagazine.co.uk/podcasts/prospect-podcast/71663/cory-doctorow-how-the-internet-went-to-sht

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

2025-12-02T22:30:44+00:00 Fullscreen Open in Tab

Finished reading Abaddon's Gate

Finished reading:

The Expanse series, book 3.

Abaddon's Gate

by James S.A. Corey.

Audiobook.

Published 2013. 539 pages.

Started October 10, 2025; completed December 2, 2025.

Tue, 02 Dec 2025 13:01:55 +0000 Fullscreen Open in Tab

Pluralistic: All the books I reviewed in 2025 (02 Dec 2025)

Today's links

All the books I reviewed in 2025: A year in books.
Hey look at this: Delights to delectate.
Object permanence: David Byrne v RIAA; Sam Buck vs Starbucks; Eek-A-Trad; Mesopotamian DRM; Distanced stage plays.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

All the books I reviewed in 2025 (permalink)

I read as much as I could in 2025, but as ever, I have finished the year bitterly aware of how many wonderful books I didn't get to, whose spines glare daggers at me whenever I sit down at my desk, beneath my groaning To Be Read shelf. But I did write nearly two dozen reviews here on Pluralistic in calendar 2025, which I round up below.

If these aren't enough for you, please check out the lists from previous years.

Now that my daughter is off at college (!), I have a lot fewer kids' books in my life than I did when she was growing up. I miss 'em! (And I miss her, too, obviously).

But! I did manage to read a couple great kids' books this year that I recommend to you without reservation, both for your own pleasure and for any kids in your life, and I wanted to call them out separately, since (good) books are such good gifts for kids:

Daniel Pinkwater's Jules, Penny and the Rooster (middle-grades novel)

https://pluralistic.net/2025/03/11/klong-you-are-a-pickle-2/#martian-space-potato

Perry Metzger, Penelope Spector and Jerel Dye's Science Comics Computers: How Digital Hardware Works (graphic novel nonfiction)

https://pluralistic.net/2025/11/05/xor-xand-xnor-nand-nor/#brawniac

NONFICTION

I. Cooking in Maximum Security, Matteo Guidi

Cooking in Maximum Security is a slim volume of prisoners' recipes and improvised cooking equipment, a testament to the ingenuity of a network of prisoners in Italy's maximum security prisons.

https://pluralistic.net/2025/11/24/moca-moka/#culinary-apollo-13

II. 9 Times My Work Has Been Ripped Off, Raymond Biesinger

A masterclass in how creative workers can transform the endless, low-grade seething about the endless ripoffs of the industry into something productive and even profound.

https://pluralistic.net/2025/10/28/productive-seething/#fuck-you-pay-me

III. Three Rocks, Bill Griffiths

What better format for a biography of Ernie Bushmiller, creator of the daily Nancy strip, than a graphic novel? And who better to write and draw it than Bill Griffith, creator of Zippy the Pinhead, a long-running and famously surreal daily strip? Griffith is carrying on the work of Scott McCloud, whose definitive Understanding Comics used the graphic novel form to explain the significance and method of sequential art, singling out Nancy for special praise.

https://pluralistic.net/2025/06/27/the-snapper/#9-to-107-spikes

IV. The Blues Brothers, Daniel de Visé

A brilliantly told, brilliantly researched tale that left me with a much deeper understanding of – and appreciation for – the cultural phenomenon that I was (and am) swept up in.

https://pluralistic.net/2025/06/21/1060-west-addison/#the-new-oldsmobiles-are-in-early-this-year

V. Close to the Machine, Ellen Ullman

Ullman's subtitle for the book is "Technophilia and its discontents," and therein lies the secret to its magic. Ullman loves programming computers, loves the way they engage her attention, her consciousness, and her intelligence. Her descriptions of the process of writing code – of tackling a big coding project – are nothing less than revelatory. She captures something that a million technothriller movies consistently fail to even approach: the dramatic interior experience of a programmer who breaks down a complex problem into many interlocking systems, the momentary and elusive sense of having all those systems simultaneously operating in a high-fidelity mental model, the sense of being full, your brain totally engaged in every way. It's a poetics of language that meets and exceeds the high bar set by the few fiction writers who've ever approached a decent rendering of this feeling, like William Gibson.

https://pluralistic.net/2025/07/16/beautiful-code/#hackers-disease

VI. Chasing Shadows, Ron Deibert

Deibert's pulse-pounding, sphinter-tightening true memoir of his battles with the highly secretive cyber arms industry whose billionaire owners provide mercenary spyware that's used by torturers, murderers and criminals to terrorize their victims.

https://pluralistic.net/2025/02/04/citizen-lab/#nso-group

VII. Little Bosses Everywhere, Bridget Read

Read, an investigative journalist at Curbed, takes us through the history of the multi-level marketing "industry," which evolved out of Depression-era snake oil salesmen, Tupperware parties, and magical thinking cults built around books like Think and Grow Rich. This fetid swamp gives rise to a group of self-mythologizing scam artists who founded companies like Amway and Mary Kay, claiming outlandish – and easily debunked – origin stories that the credulous press repeats, alongside their equally nonsensical claims about the "opportunities" they are creating for their victims.

https://pluralistic.net/2025/05/05/free-enterprise-system/#amway-or-the-highway

VIII. Careless People, Sarah Wynn-Williams

Wynn-Williams was a lot closer to three of the key personalities in Facebook's upper echelon than anyone in my orbit: Mark Zuckerberg, Sheryl Sandberg, and Joel Kaplan, who was elevated to VP of Global Policy after the Trump II election. I already harbor an atavistic loathing of these three based on their public statements and conduct, but the events Wynn-Williams reveals from their private lives make them out to be beyond despicable. There's Zuck, whose underlings let him win at board-games like Settlers of Catan because he's a manbaby who can't lose (and who accuses Wynn-Williams of cheating when she fails to throw a game of Ticket to Ride while they're flying in his private jet). There's Sandberg, who demands the right to buy a kidney for her child from someone in Mexico, should that child ever need a kidney.

https://pluralistic.net/2025/04/23/zuckerstreisand/#zdgaf

IX. More Everything Forever, Adam Becker

Astrophysicist Adam Becker knows a few things about science and technology – enough to show, in a new book called More Everything Forever that the claims that tech bros make about near-future space colonies, brain uploading, and other skiffy subjects are all nonsense dressed up as prediction.

https://pluralistic.net/2025/04/22/vinges-bastards/#cyberpunk-is-a-warning-not-a-suggestion

X. Murder the Truth, David Enrich

A brave, furious book about the long-running plan by America's wealthy and corrupt to "open up the libel laws" so they can destroy their critics. In taking on the libel-industrial complex – a network of shadowy, thin-skinned, wealthy litigation funders; crank academics; buck-chasing lawyer lickspittle sociopaths; and the most corrupt Supreme Court justice on the bench today – Enrich is wading into dangerous territory. After all, he's reporting on people who've made it their life's mission to financially destroy anyone who has the temerity to report on their misdeeds.

https://pluralistic.net/2025/03/17/actual-malice/#happy-slapping

FICTION

I. Letters From an Imaginary Country, Theodora Goss

Goss spins extremely weird, delightful and fun scenarios in these stories and she slides you into them like they were a warm bath. Before you know it, you're up to your nostrils in story, the water filling your ears, and you don't even remember getting in the tub. They're that good. Goss has got a pretty erudite and varied life-history to draw on here. She's a Harvard-trained lawyer who was born in Soviet Hungary, raised across Europe and the UK and now lives in the USA. She's got a PhD in English Lit specializing in gothic literature and monsters and was the research assistant on a definitive academic edition of Dracula. Unsurprisingly, she often writes herself into her stories as a character.

https://pluralistic.net/2025/11/11/athena-club/#incluing

II. The Immortal Choir Holds Every Voice, Margaret Killjoy

A collection of three linked short stories set in Killjoy's celebrated Danielle Cain series, which Alan Moore called "ideal reading for a post-truth world. Danielle Cain is a freight-train-hopping, anarcho-queer hero whose adventures are shared by solidaristic crews of spellcasting, cryptid-battling crustypunk freaks and street-fighters.

https://pluralistic.net/2025/06/18/anarcho-cryptid/#decameron-and-on

III. Fever Beach, Carl Hiaasen

Hiaasen's method is diabolical and hilarious: each volume introduces a bewildering cast of odd, crooked, charming, and/or loathsome Floridians drawn from his long experience chronicling the state and its misadventures. After 20-some volumes in this vein (including Bad Monkey, lately adapted for Apple TV), something far weirder than anything Hiaasen ever dreamed up came to pass: Donald Trump, the most Florida Man ever, was elected president. If you asked an LLM to write a Hiaasen novel, you might get Trump: a hacky, unimaginative version of the wealthy, callous, scheming grifters of the Hiaasenverse. Back in 2020, Hiaasen wrote Trump into Squeeze Me, a tremendous and madcap addition to his canon. Fever Beach is the first Hiaasen novel since Squeeze Me, and boy, does Hiaasen ever have MAGA's number. It's screamingly funny, devilishly inventive, and deeply, profoundly satisfying. With Fever Beach, Hiaasen makes a compelling case for Florida as the perfect microcosm of the terrifying state of America, and an even more compelling case for his position as its supreme storyteller.

https://pluralistic.net/2025/10/21/florida-duh/#strokerz-for-liberty

IV. Jules, Penny and the Rooster, Daniel Pinkwater

Jules and her family have just moved to a suburb called Bayberry Acres in the sleepy dormitory city of Turtle Neck and now she's having a pretty rotten summer. All that changes when Jules enters an essay contest in the local newspaper to win a collie (a contest she enters without telling her parents, natch) and wins. Jules names the collie Penny, and they go for long rambles in the mysterious woods that Bayberry Acres were carved out of. It's on one of these walks that they meet the rooster, a handsome, proud, friendly fellow who lures Penny over the stone wall that demarcates the property line ringing the spooky, abandoned mansion/castle at the center of the woods. Jules chases Penny over the wall, and that's when everything changes.

On the other side of that wall is a faun, and little leprechaun-looking guys, and a witch (who turns out to be a high-school chum of her city-dwelling, super-cool aunt), and there's a beast in a hidden dilapidated castle. After Jules sternly informs the beast that she's far too young to be anyone's girlfriend – not even a potentially enchanted prince living as a beast in a hidden castle – he disabuses her of this notion and tells her that she is definitely the long-prophesied savior of the woods, whose magic has been leaking out over years.

Nominally this is a middle-grades book, and while it will certainly delight the kids in your life, I ate it up. The purest expression of Pinkwater's unique ability to blend the absurd and the human and make the fantastic normal and the normal fantastic. I laughed long and hard, and turned the final page with that unmissable Pinkwatertovian sense of satisfied wonder.

https://pluralistic.net/2025/03/11/klong-you-are-a-pickle-2/#martian-space-potato

V. Where the Axe Is Buried, Ray Nayler

An intense, claustrophobic novel of a world run by "rational" AIs that purport to solve all of our squishy political problems with empirical, neutral mathematics. It's a birchpunk tale of AI skulduggery, lethal robot insects, radical literature, swamp-traversing mechas, and political intrigue that flits around a giant cast of characters, creating a dizzying, in-the-round tour of Nayler's paranoid world. A work of first-rate science fiction, which provides an emotional flythrough of how Larry Ellison's vision of an AI-driven surveillance state where everyone is continuously observed, recorded and judged by AIs so we are all on our "best behavior" would obliterate the authentic self, authentic relationships, and human happiness.

https://pluralistic.net/2025/03/20/birchpunk/#cyberspace-is-everting

VI. Lessons in Magic and Disaster, Charlie Jane Anders

A novel about queer academia, the wonder of thinking very hard about very old books, and the terror and joy of ambiguous magic. Anders tosses a lot of differently shaped objects into the air, and then juggles them, interspersing the main action with excerpts from imaginary 18th century novels (which themselves contain imaginary parables) that serve as both a prestige and a framing device.

It's the story of Jamie, a doctoral candidate at a New England liberal arts college who is trying to hold it all together while she finishes her dissertation. That would be an impossible lift, except for Jamie's gift for maybe-magic – magic that might or might not be real. Certain places ("liminal spaces") call to Jamie. These are abandoned, dirty, despoiled places, ruins and dumps and littered campsites. When Jamie finds one of these places, she can improvise a ritual, using the things in her pockets and school bag as talismans that might – or might not – conjure small bumps of luck and fortune into Jamie's path.

There's a lot of queer joy in here, a hell of a lot of media theory, and some very chewy ruminations on the far-right mediasphere. There's romance and heartbreak, danger and sacrifice, and most of all, there's that ambiguous magic, which gets realer and scarier as the action goes on.

https://pluralistic.net/2025/08/19/revenge-magic/#liminal-spaces

VII. The Adventures of Mary Darling, Pat Murphy

The titular Mary Darling here is the mother of Wendy, John and and Michael Darling, the three children who are taken by Peter Pan to Neverland in JM Barrie's 1902 book The Little White Bird, which later became Peter Pan. After Mary's children go missing, Mary's beloved uncle, John Watson, is summoned to the house, along with his famous roommate, the detective Sherlock Holmes. However, Holmes is incapable of understanding where the Darling children have gone, because to do so would be to admit the existence of the irrational and fantastic, and, more importantly, to accept the testimony of women, lower-class people, and pirates. Holmes has all the confidence of the greatest detective alive, which means he is of no help at all.

Only Mary can rescue her children. John Watson discovers her consorting with Sam, a one-legged Pacific Islander who is a known fence and the finest rat-leather glovemaker in London, these being much prized by London's worst criminal gangs. Horrified that Mary is keeping such ill company, Watson confronts her and Sam (and Sam's parrot, who screeches nonstop piratical nonsense), only to be told that Mary knows what she is doing, and that she is determined to see her children home safe.

What follows is a very rough guide to fairyland. It's a story that recovers the dark asides from Barrie's original Pan stories, which were soaked with blood, cruelty and death. The mermaids want to laugh as you drown. The fairies hate you and want you to die. And Peter Pan doesn't care how many poorly trained Lost Boy starvelings die in his sorties against pirates, because he knows where there are plenty more Lost Boys to be found in the alienated nurseries of Victorian London, an ocean away.

https://pluralistic.net/2025/05/06/nevereverland/#lesser-ormond-street

GRAPHIC NOVELS AND COMICS

I. Science Comics Computers: How Digital Hardware Works, Perry Metzger, Penelope Spector and Jerel Dye

Legendary cypherpunk Perry Metzger teams up with Penelope Spector and illustrator Jerel Dye for a tour-de-force young adult comic book that uses hilarious steampunk dinosaurs to demystify the most foundational building-blocks of computers. The authors take pains to show the reader that computing can be abstracted from computing. The foundation of computing isn't electrical engineering, microlithography, or programming: it's logic. While there's plenty of great slapstick, fun art, and terrific characters in this book that will make you laugh aloud, the lasting effect upon turning the last page isn't just entertainment, it's empowerment.

https://pluralistic.net/2025/11/05/xor-xand-xnor-nand-nor/#brawniac

II. Feeding Ghosts, Tessa Hulls

A stunning memoir that tells the story of three generations of Hulls's Chinese family. It was a decade in the making, and it is utterly, unmissably brilliant. It tells the story of Hulls's quest to understand – and heal – her relationship with her mother, a half-Chinese, half-Swiss woman who escaped from China as a small child with her own mother, a journalist who had been targeted by Mao's police.

Each of the intertwined narratives – revolutionary China, Rose's girlhood, Hulls's girlhood, the trips to contemporary China, Hulls's adulthood and Sun Yi's institutionalizations and long isolation – are high stakes, high-tension scenarios, beautifully told. Hulls hops from one tale to the next in ways that draw out the subtle, imporant parallels between each situation, subtly amplifying the echoes across time and space.

Feeding Ghosts has gone on to win the Pulitzer Prize, only the second graphic novel in history to take the honor (the first was Maus, another memoir of intergenerational trauma, horrific war, and the American immigrant experience).

https://pluralistic.net/2025/07/02/filial-piety/#great-leap-forward

III. The Murder Next Door, Hugh D'Andrade

Hugh D'Andrade is a brilliant visual communicator, the art director responsible for the look-and-feel of EFF's website. He's also haunted by a murder – the killing of the mother of his childhood playmates, which cast a long, long shadow over his life, as he recounts in his debut graphic novel. It's a haunting, beautiful meditation on masculinity, trauma, and fear. Hugh is a superb illustrator, particularly when it comes to bringing abstract ideas to life, and this is a tale beautifully told.

https://pluralistic.net/2025/02/10/pivot-point/#eff

IV. Simplicity, Mattie Lubchansky

Simplicity is set in the not-so-distant future, in which the US has dissolved and its major centers have been refashioned as "Administrative and Security Territories" – a fancy way of saying "walled corporate autocracies." Lucius Pasternak is an anthropology grad student in the NYC AST, a trans-man getting by as best as he can, minimizing how much he sells out.

Pasternak's fortunes improve when he gets a big, juicy assignment: to embed with a Catskills community of weirdo sex-hippies who supply the most coveted organic produce in the NYC AST. They've been cloistered in an old summer camp since the 1970s, and when civilization collapsed, it barely touched them. Pasternak's mission is to chronicle the community and its strange ways for a billionaire's vanity-project museum of New York State.

This is post-cyberpunk, ecosexual revolutionary storytelling at its finest.

https://pluralistic.net/2025/08/01/ecosexuality/#nyc-ast

V. The Weight, Melissa Mendes

A book that will tear your heart out, it will send you to a dreamy world of pastoral utopianism, then it will tear your heart out. Again.

A story of cyclic abuse, unconditional love, redemption, and tragedy, the tale of Edie, born to an abusive father and a teen mother, who is raised away from her family, on a military base where she runs feral with other children, far from the brutality of home. This becomes a sweet and lovely coming-of-age tale as Edie returns to her grandparents' home, and then turns to horror again.

The Weight is a ferocious read, the sweetness of the highs there to provide texture for the bitterness of the lows.

https://pluralistic.net/2025/08/21/weighty/#edie-is-a-badass

TWO MORE (BY ME)

This was a light reading year for me, but, in my defense, I did some re-reading, including all nine volumes of Naomi Novik's incredible Temeraire:

https://pluralistic.net/2023/01/08/temeraire/#but-i-am-napoleon

But the main reason I didn't read as much as I normally would is that I published two international bestsellers of my own this year.

The first was Picks and Shovels, a historical technothriller set in the early 1980s, when the PC was first being born. It's the inaugural adventure of Martin Hench, my hard-fighting, two-fisted, high-tech forensic accountant crimefighter, and it's designed to be read all on its own. Marty's first adventure sees him pitted against the owners of a weird PC pyramid-sales cult: a Mormon bishop, an orthodox rabbi and a Catholic priest, whose PC business is a front for a predatory faith-based sales cult:

https://us.macmillan.com/books/9781250865908/picksandshovels/

The other book was Enshittification, the nonfiction book I'm touring now (I wrote all this up on the train to San Diego, en route to an event at the Mission Hills Library). It's a book-length expansion of my theory of platform decay ("enshittification"), laying out the process by which the tech platforms we rely on turn themselves into piles of shit, and (more importantly), explaining why this is happening now:

https://us.macmillan.com/books/9780374619329/enshittification/

I've got a stack of books I'm hoping to read in the new year, but I'm going to have to squeeze them in among several other book projects of my own. First, there's The Reverse Centaur's Guide to Life After AI," a short book about being a better AI critic, which drops in June from Farrar, Straus and Giroux. I'm also *writing a new book, The Post-American Internet (about the internet we could have now that Trump has destroyed America's soft power and its grip on global tech policy. There's also a graphic novel adaptation of Unauthorized Bread (with Blue Delliquanti), which Firstsecond will publish in late 2026 or 2027; and a graphic novel adaptation of Enshittification (with Koren Shadmi), which Firstsecond will publish in 2027.

But of course I'm gonna get to at least some of those books on my overflowing TBR shelf, and when I do, I'll review them here on Pluralistic for you. You can follow my Reviews tag if you want to stay on top of these (there's also an RSS feed for that tag):

https://pluralistic.net/tag/reviews/

Hey look at this (permalink)

RETRACTED: Safety Evaluation and Risk Assessment of the Herbicide Roundup and Its Active Ingredient, Glyphosate, for Humans https://www.sciencedirect.com/science/article/pii/S0273230099913715

Prisoners’ Inventions https://www.lapl.org/events/exhibits/no-prior-art/exhibitions/

Inside a Group of Vigilantes with One Goal: Painting Crosswalks to Protect Pedestrians https://people.com/inside-secretive-group-vigilantes-one-goal-painting-crosswalks-save-pedestrians-11849437

The AI bubble isn’t new — Karl Marx explained the mechanisms behind it nearly 150 years ago https://theconversation.com/the-ai-bubble-isnt-new-karl-marx-explained-the-mechanisms-behind-it-nearly-150-years-ago-270663

Let's See What's Going On Down At The Piss Factory https://www.todayintabs.com/p/let-s-see-what-s-going-on-down-at-the-piss-factory

Object permanence (permalink)

#20yrsago Man flies 1MM miles on a 60 day unlimited ticket, wins 10 more flights https://web.archive.org/web/20051203031434/http://au.news.yahoo.com/051201/15/x0z4.html

#20yrsago Schneier: Aviation security is a bad joke https://web.archive.org/web/20060212060858/http://www.wired.com/news/privacy/0,1848,69712,00.html?tw=wn_tophead_2

#20yrsago David Byrne gets RIAA warning https://web.archive.org/web/20051223160922/http://journal.davidbyrne.com/2005/12/12105_rant_abou.html

#20yrsago Sam Buck sued for naming her coffee shop after herself https://web.archive.org/web/20051231144818/https://www.sfgate.com/cgi-bin/article.cgi?file=/news/archive/2005/12/01/financial/f132605S26.DTL

#20yrsago Eek-A-Mouse jamming with Irish pub musicians https://web.archive.org/web/20051211095248/http://www.alphabetset.net/audio/t-woc/eek_trad.mp3

#15yrsago Bowls made from melted army men https://web.archive.org/web/20071011212754/http://www.associatedcontent.com/article/388073/how_to_make_a_bowl_from_melted_army.html

#15yrsago TSA recommends using sexual predator tactics to calm kids at checkpoints https://web.archive.org/web/20101204044209/https://www.rawstory.com/rs/2010/12/airport-patdowns-grooming-children-sex-predators-abuse-expert/

#15yrsago University of Glasgow gives away software, patents, consulting https://www.gla.ac.uk/news/archiveofnews/2010/november/headline_181588_en.html

#15yrsago Judge in Xbox hacker trial unloads both barrels on the prosecution https://web.archive.org/web/20101203054828/https://www.wired.com/threatlevel/2010/12/xbox-judge-riled/

#10yrsago Scholars and activists stand in solidarity with shuttered research-sharing sites https://custodians.online/

#10yrsago Mesopotamian boundary stones: the DRM of pre-history https://web.archive.org/web/20151130212151/https://motherboard.vice.com/read/before-drm-there-were-mesopotamian-boundary-stones

#10yrsago Canadian civil servants grooming new minister to repeat Harper’s Internet mistakes https://www.michaelgeist.ca/2015/11/what-canadian-heritage-officials-didnt-tell-minister-melanie-joly-about-copyright/

#5yrsago Distanced stage plays https://pluralistic.net/2020/12/01/autophagic-buckeyes/#xanadu

#5rsago Ohio spends tax dollars to destroy Ohio https://pluralistic.net/2020/12/01/autophagic-buckeyes/#subsidized-autophagia

Upcoming appearances (permalink)

Seattle: Neuroscience, AI and Society (University of Washington), Dec 4
https://www.eventbrite.com/e/neuroscience-ai-and-society-cory-doctorow-tickets-1735371255139

Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/

Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification

Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html

Recent appearances (permalink)

We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w

How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s

Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/

Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8

How the internet went to sh*t (Prospect Magazine)
https://www.prospectmagazine.co.uk/podcasts/prospect-podcast/71663/cory-doctorow-how-the-internet-went-to-sht

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025

"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/

"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).

"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).

"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).

"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).

"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.

"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026

"The Memex Method," Farrar, Straus, Giroux, 2026

"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.

"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.

A Little Brother short story about DIY insulin PLANNING

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

Mon, 01 Dec 2025 16:02:56 +0000 Fullscreen Open in Tab

Pluralistic: Meta's new top EU regulator is contractually prohibited from hurting Meta's feelings (01 Dec 2025)

Today's links

Meta's new top EU regulator is contractually prohibited from saying mean things about Meta: It's one thing to hire an ex-Meta lobbyist, another entirely if she's signed a non-disparagement contract.
Hey look at this: Delights to delectate.
Object permanence: Custom M&M restrictions; Vacuum-bag dust houses; Winner-Take-All Politics; Pre-mutated products; Disney World on strike; "Ship Breaker"; TSA patdowns a "homosexual agenda"; RÄT.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

Meta's new top EU regulator is contractually prohibited from saying mean things about Meta (permalink)

"Regulatory capture" is one of those concepts that can seem nebulous and abstract. How can you really know when a regulator has failed to protect you because they were in bed with the companies they were supposed to be regulating, and when this is just because they're bad at their job. "Never attribute to malice," etc etc.

The difficulty of pinning down real instances of regulatory capture is further complicated by the arguments of right-wing economists, who claim that regulatory capture is inevitable, that companies will always grow to the point where they can overpower the state and use it to shut down smaller companies before they can become a threat. They use this as an argument for abolishing all regulation, rather than, you know, stopping monopolies from growing until they are more powerful than the state:

https://pluralistic.net/2022/06/05/regulatory-capture/

Despite this confusion, there are times when regulatory capture is anything but subtle. Especially these times, when the corporate world, spooked by the pandemic-era surge in antitrust enforcement, have launched a gloves-off/mask-off offensive to simply take over their governments, abandoning any pretext of being responsive to democratically accountable processes or agencies.

You've got David Sacks, Trump's billionaire AI czar, who is directing American AI policy while holding (hundreds of?) millions of dollars worth of stock in companies that stand to directly benefit from his work in the US government:

https://www.nytimes.com/2025/11/30/technology/david-sacks-white-house-profits.html?unlocked_article_code=1.5E8.Nb2d.3L204EF4nliq

Sacks has threatened the New York Times, demanding that they "abandon" the story about his conflicts of interest:

https://protos.com/david-sacks-sends-silly-legal-threat-to-the-new-york-times/

And he's hired the law-firm that is at the center of a decades-long open conspiracy to end press freedom in America, bankrolled and overseen by the same people who planned and executed the destruction of American abortion rights:

https://pluralistic.net/2025/03/17/actual-malice/#happy-slapping

This isn't a strictly US affair, either. In the UK, Prime Minister Keir Starmer rang in 2025 by firing the country's top competition regulator and replacing him with the former head of Amazon UK, one of the country's most notorious monopolists, whose tax evasion, labor abuses, and anticompetitive mergers and tactics had been on the Competition and Markets Authority's agenda for years:

https://pluralistic.net/2025/01/22/autocrats-of-trade/#dingo-babysitter

Today, this same swindle is playing out in Canada. Competition Commissioner Matthew Boswell – recently endowed with the most sweeping enforcement powers of any competition regulator in the world – has resigned early. Now, Canada's monopolists are openly calling for one of their own top execs to take over the office for the next five years, citing a bizarre Canadian tradition of alternating between civil servants and revolving-door corporate insiders in turn:

https://www.donotpassgo.ca/p/competition-commissioner-matthew

However, there is one country that always, always brings home the gold in the Regulatory Capture Olympics: Ireland. Ireland had the misfortune to establish itself as a tax haven, meaning it makes pennies by helping the worst corporations in the world (especially US Big Tech companies) hide billions from global tax authorities. Being a tax haven sucks, because tax havens must also function as crime havens.

After all, the tech companies that pretend to be Irish have no loyalty to the country – they are there solely because Ireland will help them cheat the rest of the world. What's more, any company that can hire lawyers to do the paperwork to let it pretend that it's Irish this week could pay those lawyers to pretend that it is Cypriot, or Maltese, or Dutch, or Luxembourgeois next week. To keep these American companies from skipping town, Ireland must bend its entire justice system to the facilitation of all of American tech companies' crimes.

Of course, there is no class of crime that American tech companies commit more flagrantly or consequentially than the systematic, ruthless invasion of our privacy. Nine years ago, the EU passed the landmark General Data Protection Regulation (GDPR), a big, muscular privacy law that bans virtually all of the data-collection undertaken by America's tech companies. However, because these companies pretend they are Irish, they have been able to move all GDPR enforcement to Ireland, where the Data Protection Commissioner could always be relied upon to let these companies get away with murder:

https://pluralistic.net/2023/05/15/finnegans-snooze/#dirty-old-town

If you have formed the (widespread) opinion that the GDPR is worse than useless, responsible for nothing more than an endless stream of bullshit "cookie consent" pop-ups, blame the Irish DPC. American tech companies have pretended that they are allowed to substitute these cookie popups for doing the thing the GDPR demands of them (not spying on you at all). This is an obvious violation of the GDPR, and the only way an enforcer could possibly fail to see this is if they served a government whose entire economy depended on keeping Mark Zuckerberg, Tim Cook and Sundar Pichai happy. It's impossible to explain something to a regulator when their paycheck depends on them not understanding it.

Incredibly, Ireland has found a way to make this awful situation even worse. They've appointed Niamh Sweeney, an ex-Meta lobbyist, to the role of Irish Data Protection Commissioner. Her resume includes "six years at Meta, according to her LinkedIn profile. She was head of public policy, Ireland for Facebook before becoming WhatsApp’s director of public policy for Europe, Middle East and Africa":

https://www.irishtimes.com/business/2025/09/17/ex-tech-lobbyist-named-to-data-protection-commission/

In their complaint to the European Commission, the Irish Council for Civil Liberties lays out a devastating case against Sweeney's fitness to serve – the fact that she has broad, deep, obvious conflicts of interest that should automatically disqualify her from the role:

https://www.iccl.ie/digital-data/complaint-v-ireland-to-european-commission-re-process-appointing-ex-meta-lobbyist-as-data-protection-commissioner/#_ftn11

Among other things, Meta execs – like Sweeney – are given piles of stock options and shares in the company. The decisions that Sweeney will be called upon to make as DPC will have a significant and lasting negative effect on Meta's stocks – if Meta is banned from surveilling 500m affluent European consumers, they will make a lot less money.

But that's just for starters. Meta execs also sign contracts that bind them to:

Nondisparagement: ex-Meta executives are permanently barred from "making any disparaging, critical or otherwise detrimental comments to any person or entity concerning [Meta's] products, services or programs; its business affairs, operation, management and financial condition…"
Nondisclosure: ex-Meta executives are broadly prohibited from discussing their employment, or disclosing the things they learned while working at the company.
Forced arbitration: if Meta believes that a former exec has violated these clauses, they can order the former exec to be silent, and bill them tens of thousands of dollars every time they speak out. Former executives sign away the right to contest these fines and orders in front of a judge; instead, all claims are heard by an "arbitrator" – a corporate lawyer who is paid by Meta and is in charge of deciding whether Meta (who pays their invoices) is right or wrong.

We know about these contractual terms because they have been applied to Sarah Wynn-Williams, a former top Meta exec who published a whistleblower memoir, Careless People, which discloses many of Meta's most terrible practices, from systemic sexual harassment at the highest echelon to a worldwide surveillance collaboration with the Chinese government to complicity in the Rohingya genocide, to the fact that Mark Zuckerberg cheats at Settlers of Catan and his underlings let him win:

https://pluralistic.net/2025/04/23/zuckerstreisand/#zdgaf

Meta dragged Wynn-Williams in front of Meta's pet arbitrator over the statements in her book (without disputing their truthfulness). The arbitrator has fined Wynn-Williams $111,000,000 for speaking out ($50,000 per violation), and has barred her from promoting her book in any way. The company has ordered her not to testify before the US Congress or the UK Parliament. The clauses in Wynn-Williams contract are very similar (if not identical) to the clauses that the US National Labor Relations Board ruled unenforceable:

https://www.hcamag.com/us/specialization/employment-law/nlrb-rules-metas-7200-confidentiality-agreements-unlawful/499180

Wynn-Williams appeared on stage with me last month at London's Barbican Centre, in a book-tour event moderated by Chris Morris. Whenever we talked about Meta or Careless People, Wynn-Williams would fall silent and assume a blank facial expression, lest she make another statement that would result in Meta seeking another $50,000 from her under the terms of her contract.

In their complaint to the EU, ICCL raises the extremely likely probability that Sweeney is bound by the same contractual terms as Wynn-Williams, meaning that Meta's top regulator in Ireland, the country where Meta pretends to be based, will be contractually prohibited from saying anything that makes Mark Zuckerberg feel bad about himself.

This isn't just a matter for Ireland, either. Given the nature of European federalism, most of Meta's violations of European privacy laws will start with the Irish DPC – in other words, all 500,000,000 Europeans will be forced to complain to someone who is legally barred from upsetting Zuck's digestion.

Tax havens are a global scourge. By allowing American tech companies to evade their taxes around the world, Ireland is complicit in starving countries everywhere of tax revenue they are properly owed. Perhaps even worse than this, though, is the fact that these cod-Irish American companies can always out-compete their domestic rivals all over the world, because those companies have to pay tax, while Meta does not. Ireland has been every bit as important in exporting US Big Tech around the world as the US has been.

But Ireland has another key export, one that is confined to the European Union. Because every tax haven must be a crime haven, and because Big Tech's favorite crime is illegal surveillance, Ireland has exported American tech spying to the whole European Union.

That's how things stand today, and how they've stood since the passage of the GDPR. If you'd asked me a year ago, I would have said that this is as terrible as things could get. But now that Ireland has put an ex-Meta exec in charge of deciding whether Meta is invading Europeans' privacy, without confirming whether this dingo babysitter is even allowed to criticize Meta, it's clear that things could get much worse than I ever imagined.

(Image: Cryteria, CC BY 3.0, modified)

Hey look at this (permalink)

A Remarkable Assertion from A16Z https://nealstephenson.substack.com/p/a-remarkable-assertion-from-a16z
NYC Realtime Subway Clock https://nookwoodworking.com/products/nyc-realtime-subway-clock?variant=42620252717218
Big Tech’s Invisible Hand https://apublica.org/especial/big-techs-invisible-hand/
Santa plc https://open.spotify.com/episode/1HpeETwCsmYAUWM1g06lhI
MAGA Antitrust Is Officially Dead: DOJ Sanctions Price Fixing With Slap on the Wrist Settlement Against Rental Housing Collusion Kingpin RealPage https://www.nakedcapitalism.com/2025/12/maga-antitrust-is-officially-dead-doj-sanctions-illegal-behavior-with-slap-on-the-wrist-settlement-against-price-fixing-kingpin-realpage.html

Object permanence (permalink)

#20yrsago Custom M&Ms: just don’t mention the war, your hometown, or nouns https://memex.craphound.com/2005/11/28/custom-mms-just-dont-mention-the-war-your-hometown-or-nouns/

#20yrsago Sony CD spyware installs and can run permanently, even if you click “Decline” https://blog.citp.princeton.edu/2005/11/28/mediamax-permanently-installs-and-runs-unwanted-software-even-if-user-declines-eula/

#20yrsago Programmers on Sony’s spyware DRM asked for newsgroup help too https://groups.google.com/g/microsoft.public.windowsmedia.sdk/c/kWKbc54lLxo?hl=en&pli=1#cf2c1677c4ce5138

#20yrsago Vacuum-bag dust houses sculpted by former house-cleaner https://web.archive.org/web/20051127031640/http://mocoloco.com/art/archives/001661.php

#20yrsago Sony knew about rootkits 28 days before the story broke https://web.archive.org/web/20051202044828/http://www.businessweek.com/technology/content/nov2005/tc20051129_938966.htm

#20yrsago How the next version of the GPL will be drafted https://gplv3.fsf.org/process-definition/

#20yrsago No Xmas for Sony protest badge https://web.archive.org/web/20051203044536/https://gigi.pixcode.com/noxmas.gif

#20yrsago HOWTO defeat Apple’s anti-DVD-screenshot DRM https://highschoolblows.blogspot.com/2005/11/take-screenshot-of-dvd-player-in-os-x.html

#20yrsago EFF: DMCA exemption process is completely bullshit https://web.archive.org/web/20051204031027/https://www.eff.org/deeplinks/archives/004212.php

#15yrsago Paolo Bacigalupi’s SHIP BREAKER: YA adventure story in a post-peak-oil world https://memex.craphound.com/2010/11/30/paolo-bacigalupis-ship-breaker-ya-adventure-story-in-a-post-peak-oil-world/

#15yrsago Walt Disney World employees demand a living wage https://thedisneyblog.com/2010/12/01/disney-world-union-takes-offensive/

#15yrsago Hotel peephole doctored for easy removal and spying https://www.flickr.com/photos/kentbrew/5221903189/

#15yrsago DC-area county official says TSA patdowns are “homosexual agenda” https://dcist.com/story/10/11/30/loudoun-county-official-tsa-pat-dow/

#15yrsago Dmitry Sklyarov and co. crack Canon’s “image verification” anti-photoshopping tool https://web.archive.org/web/20110808200303/https://www.networkworld.com/news/2010/113010-analyst-finds-flaws-in-canon.html

#15yrsago TSA scans uniformed pilots, but airside caterers bypass all screening https://web.archive.org/web/20101125095532/https://www.salon.com/technology/ask_the_pilot/2010/11/22/tsa_screening_of_pilots/index.html

#15yrsago BP sued in Ecuador for violating the “rights of Nature” https://www.democracynow.org/2010/11/29/headlines/bp_sued_in_ecuadorian_court_for_violating_rights_of_nature

#15yrsago Four horsemen of the information apocalypse: Cohen, Fanning, Johansen and Frankel https://web.archive.org/web/20101126191152/https://time.com/time/specials/packages/printout/0,29239,2032304_2032746_2032903,00.html

#15yrsago Winner-Take-All Politics: how America’s super-rich got so much richer https://memex.craphound.com/2010/11/29/winner-take-all-politics-how-americas-super-rich-got-so-much-richer/

#15yrsago EFF on US domain copyright seizures https://www.eff.org/deeplinks/2010/11/us-government-seizes-82-websites-draconian-future

#15yrsago Where’s Molly: heartbreaking reunion with developmentally disabled sister institutionalized 47 years ago https://web.archive.org/web/20101129193304/http://www.cbsnews.com/stories/2010/11/28/sunday/main7096335.shtml

#15yrsago “Death-row inmate” seeks last meal advice on Amazon message-board https://web.archive.org/web/20101130212132/http://www.amazon.com/tag/health/forum/ref=cm_cd_pg_pg1?_encoding=UTF8&cdForum=Fx1EO24KZG65FCB&cdPage=1&cdSort=oldest&cdThread=Tx3FNFNI6N592DI

#10yrsago You’re only an “economic migrant” if you’re poor and brown https://historyned.blog/2015/09/09/the-wandering-academic-or-how-no-one-seems-to-notice-that-i-am-an-economic-migrant/

#10yrsago Pre-mutated products: where did all those “hoverboards” come from? https://memex.craphound.com/2015/11/29/pre-mutated-products-where-did-all-those-hoverboards-come-from/

#10yrsago Millennials are cheap because they’re broke https://www.theatlantic.com/business/archive/2014/12/millennials-arent-saving-money-because-theyre-not-making-money/383338/?utm_source=SFFB

#5yrsago Attack Surface in the New York Times https://pluralistic.net/2020/11/30/selmers-train/#times

#5yrsago RÄT https://pluralistic.net/2020/11/30/selmers-train/#honey-morello

#5yrsago Open law and the rule of law https://pluralistic.net/2020/11/30/selmers-train/#rogue-archivist

#5yrsago Twitter is more redeemable than Facebook https://pluralistic.net/2020/11/30/selmers-train/#epistemic-superiority

Upcoming appearances (permalink)

San Diego: Enshittification at the Mission Hills Branch Library, Dec 1
https://libraryfoundationsd.org/events/doctorow
Seattle: Neuroscience, AI and Society (University of Washington), Dec 4
https://www.eventbrite.com/e/neuroscience-ai-and-society-cory-doctorow-tickets-1735371255139
Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html

Recent appearances (permalink)

We have become slaves to Silicon Valley (Politics JOE)
https://www.youtube.com/watch?v=JzEUvh1r5-w
How Enshittification is Destroying The Internet (Frontline Club)
https://www.youtube.com/watch?v=oovsyzB9L-s
Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/
Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8
How the internet went to sh*t (Prospect Magazine)
https://www.prospectmagazine.co.uk/podcasts/prospect-podcast/71663/cory-doctorow-how-the-internet-went-to-sht

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

2025-11-29T01:34:56+00:00 Fullscreen Open in Tab

Note published on November 29, 2025 at 1:34 AM UTC

oscar the couch

A sofa covered in moss green fluffy fabric

Fri, 28 Nov 2025 09:00:28 +0000 Fullscreen Open in Tab

Pluralistic: (Digital) Elbows Up (28 Nov 2025)

Today's links

(Digital) Elbows Up: How Canada Can Become a Nation of Jailbreakers, Reclaim Our Digital Sovereignty, Win the Trade-War, and Disenshittify Our Technology.
Hey look at this: Delights to delectate.
Object permanence: Workaholic Goethe; Prehistory of the Sony rootkit; Wunderkammer front room; Dolphin teleportation symposium: now with more Eisenhowers
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

(Digital) Elbows Up (permalink)

I'm in Toronto to participate in a three-day "speculative design" workshop at OCAD U, where designers, technologists and art students are thinking up cool things Canadians could do if we reformed our tech law:

https://www.ocadu.ca/events-and-exhibitions/jailbreaking-canada

As part of that workshop, I delivered a keynote speech last night, entitled "(Digital) Elbows Up: How Canada Can Become a Nation of Jailbreakers, Reclaim Our Digital Sovereignty, Win the Trade-War, and Disenshittify Our Technology."

The talk was recorded and I'll add the video to this post when I get it, but in the meantime, here's the transcript of my speech. Thank you to all my collaborators at OCAD U for bringing me in and giving me this wonderful opportunity!

My theory of enshittification describes the process by which platforms decay. First, they are good to their end users, while finding a way to lock those users in.

Then, secure in the knowledge that they can make things worse for those users without risking their departure, the platforms make things worse in order to make things attractive for business customers. Who also get locked in, dependent on those captive users.

And then, in the third stage of enshittification, platforms raid those business customers, harvesting all available surpluses for their shareholders and executives, leaving behind the bare, mingy homeopathic residue of value needed to keep users locked to the platform and businesses locked to the users, such that the final, ideal stage of the enshittified platform is attained: a giant pile of shit.

This observational piece of the theory is certainly valuable, inasumuch as it lets us scoop up this big, diffuse, enraging phenomenon, capture it in a net, attach a handle to it and call it "enshittification," recognising how we're being screwed.

But much more important is the enshittification hypothesis's theoretical piece, its account of why this is happening now.

Let me start by saying that I do not attribute blame for enshittification to your poor consumer choices. Despite the endless insistences of the right, your consumption choices aren't the arbiters of policy.

The reason billionaires urge you to vote with your wallets is that their wallets are so much thicker than yours. This is the only numeric advantage the wealthy and powerful enjoy. They are in every other regards an irrelevant, infinitesimal minority. In a vote of ballots, rather than wallets, they will lose every time, which is why they are so committed to this wallet-voting nonsense. The wallet-vote is the only vote they can hope to win.

The idea that consumers are the final arbiters of society is a laughable, bitter counsel of despair. You will not shop your way free of a monopoly, any more than you will recycle your way out of wildfires. Shop as hard as you like, you will not – cannot – end enshittification.

Enshittification is not the result of your failure to grasp that "if you're not paying for the product, you're the product." You're the product if you pay. You're the product if you don't pay. The determinant of your demotion to "the product" is whether the company can get away with treating you as the product.

So what about the companies? What about the ketamine-addled zuckermuskian failures who have appointed themselves eternal dictators over the digital lives of billions of people? Can we blame them for enshittifying our world?

Well, yes…and no.

It's obviously true that it takes a certain kind of sociopath to run a company like Facebook or Google or Apple. The suicide nets around Chinese iPhone factories are a choice, not an integral component of the phone manufacturing process.

But these awful men are merely filling the niches that our policy environment has created. If Elon Musk ODs on ket today, there will be an overnight succession battle among ten horrible Big Balls, and the victor who emerges from that war will be indistinguishable from Musk himself.

The problem isn't that the wrong person is running Facebook and thus exercising a total veto over the digital lives of four billion people, the problem is that such a job exists. We don't need to perfect Zuck. We don't need to replace Zuck. We need to abolish Zuck.

So where does the blame lie?

It lies with policy makers. Regulators and politicians who created an enshittogenic environment: a rigged game whose terrible rules guarantee that the worst people doing the worst things will fare best.

These are the true authors of enshittification: the named individuals who, in living memory, undertook specific policy decisions, that had the foreseeable and foreseen outcome of ushering in the enshittocene. Policymakers who were warned at the time that this would happen, who ignored that advice and did it anyway.

It is these people and their terrible, deliberate misconduct that we need to remember. It is their awful policies that we must overthrow, otherwise all we can hope to do is replace one monster with another.

So, in that spirit, let us turn to the story of one of these enshittogenic policy choices and the men who made it.

This policy is called "anti-circumvention" and it is the epicenter of the enshittogenic policy universe. Under anti-circumvention law, it is a crime to modify a device that you own, if the company that sold it to you would prefer that you didn't.

All a company has to do is demarcate some of its code as off-limits to modification, by adding something called an "access control," and, in so doing, they transform the act of changing any of that code into a felony, a jailable offense.

The first anticircumvention law is America's Digital Millennium Copyright Act, or DMCA. Under Section 1201 of the DMCA, helping someone modify code behind an access control is a serious crime, punishable by a five-year prison sentence and a $500,000 fine. Crucially, this is true whether or not you break any other law. Under DMCA 1201, simply altering a digital device to do a perfectly legal thing becomes a jailable crime, if the manufacturer wills it so and manifests that will with an "access control."

I recognize that this is all very abstract, so let me make it concrete. When you buy a printer from HP, it becomes your property. What's property? Well, let's use the standard definition that every law student learns in first year property law, from Sir William Blackstone's 1753 treatise:

"Property: that sole and despotic dominion which one man claims and exercises over the external things of the world, in total exclusion of the right of any other individual in the universe."

The printer is yours. It's your property. You have sole and despotic dominion over it in exclusion of any other individual in the universe.

But HP printers ship with a program that checks to see whether you're using HP ink, and if it suspects that you've bought generic ink, the printer refuses to use it. Now, Congress never passed a law saying "If you buy an HP printer, you have to buy HP ink, too." That would be a weird law, given the whole sole-and-despotic dominion thing.

But because HP puts an "access control" in the ink-checking code, they can conjure up a brand new law: a law that effectively requires you to use HP ink.

Anticircumvention is a way for legislatures to outsource law-making to corporations. Once a corporation adds an access control to its product, they can create a new felony for using it in ways that benefit you at the expense of the company's shareholders.

So another way of saying "anticircumvention law" is "felony contempt of business model." It's a way for a corporation to threaten you with prison if you don't use your property in the way they want you to.

That's anti-circumvention law.

The DMCA was an enshittifier's charter, an invitation for corporations to use tactical "access controls" to write invisible, private laws that would let them threaten their customers – and competitors who might help those customers – with criminal prosecution.

Now, the DMCA has a known, living author, Bruce Lehman, a corporate IP lawyer who did a turn in government service as Bill Clinton's IP Czar.

Lehman tried several ways to get American policymakers to adopt this stupid idea, only to be rebuffed. So, undaunted, he traveled to Geneva, home of the World Intellectual Property Organization or WIPO, a UN "specialized agency" that makes the world's IP treaties. At Lehman's insistence, WIPO passed a pair of treaties in 1996, collectively known as the "Internet Treaties," and in 1998, he got Congress to pass the DMCA, in order to comply with the terms of these treaties, a move he has since repeatedly described as "doing an end-run around Congress."

This guy, Bruce Lehman, is still with us, breathing the same air as you and me. We are sharing a planet with the Louis Pasteur of making everything as shitty as possible.

But Bruce Lehman only enshittified America, turning our southern cousins into fodder for the immortal colony organisms we call limited liability corporations. To understand how Canada enshittified, we have to introduce some Canadian enshittifiers.

Specifically, two of Stephen Harper's ministers: James Moore, Harper's Heritage minister, and the disgraced sex-pest Tony Clement, who was then Industry minister. Stephen Harper really wanted a Canadian anti-circumvention law, and he put Clement and Moore in charge of the effort.

Everyone knew that it was going to be a hard slog. After all, Canadians had already rejected anti-circumvention law three times. Back in 2006, Sam Bulte – a Liberal MP in Paul Martin's government – tried to get this law through, but it was so unpopular that she lost her seat in Parkdale, which flipped to the NDP for a generation.

Moore and Clement hatched a plan to sell anti-circumvention to the Canadian people. They decided to do a consultation on the law. The thinking was that if we all "felt heard" then we wouldn't be so angry when they rammed it through.

Boy, did that backfire. 6,138 of us filed consultation responses categorically rejecting this terrible law, and only 53 responses offered support for the idea.

How were Moore and Clement going to spin this? Simple. Moore went to a meeting of the International Chamber of Commerce in Toronto, and gave a speech where he denounced all 6,138 of us as "babyish" and "radical extremists." Then Harper whipped his caucus and in 2012, Bill C-11, the Copyright Modernisation Act passed, and we got a Made-in-Canada all-purpose, omnienshittificatory anti-circumvention law.

Let's be clear about what this law does: because it makes no exemptions for circumvention for lawful purposes, Canada's anti-circumvention law criminalizes anything you do with your computer, phone or device, if it runs counter to the manufacturer's wishes.

It's an invitation for foreign manufacturers to use Canada's courts to punish Canadian customers and Canadian companies for finding ways to make the products we buy and use less shitty.

Anti-circumvention is at the root of the repair emergency. All companies have to do is add an "initialization" routine to their devices, so that any new parts installed in a car, or a tractor, or a phone, or a ventilator have to be unlocked by the manufacturer's representative before the device will recognize the new part, and it becomes a crime for an independent mechanic, or a farmer, or an independent repair shop, or a hospital technician to fix a car, or a tractor, or a phone, or a ventilator.

This is called "parts pairing" or "VIN locking." Now, we did pass C-244, a national Right to Repair law, last year, but it's just a useless ornament, because it doesn't override anti-circumvention. So Canadians can't fix their own technology if the manufacturer uses an access control to block the repair.

Anti-circumvention means we can't fix things when they break, and it also means that we can't fix them when they arrive pre-broken by their enshittifying manufacturers.

Take the iPhone: it can only use one app store, Apple's official one, and everyone who puts an app in the app store has to sign up to use Apple's payment processor, which takes 30 cents out of every dollar you spend inside an app.

That means that when a Canadian user sends $10 a month to a Canadian independent news outlet or podcast, $3 out of that $10 gets sucked out of the transaction and lands in Cupertino, California, where it is divvied by Apple's shareholders and executives.

It's not just news sites. Every dollar you send through an app to a performer on Patreon, a crafter on Etsy, a games company, or a software company takes a roundtrip through Silicon Valley and comes back 30 cents lighter.

A Canadian company could bypass the iPhone's "access controls" and give you a download or a little hardware dongle that installed a Canadian app store, one that used the Interac network to process payments for free, eliminating Apple and Google's 30% tax on Canada's entire mobile digital economy.

And indeed, we have 2024's Bill C-294, an interoperability law, that lets Canadians do this. But just as with the repair law, our interoperability law is also useless, because it doesn't repeal the anti-circumvention law, meaning you are only allowed to reverse engineer products to make interoperable alternatives if there is no access control in the way. Of course, every company that's in a position to rip you off just adds an access control.

The fact that foreign corporations have the final say over how Canadians use their own property is a font of endless enshittification. Remember when we told Facebook to pay news outlets for links and Facebook just removed all links to the news? Our anti-circumvention law is the only reason that a Canadian company couldn't jailbreak the Facebook app and give you an alternative app, one that slurped up everything Facebook was waiting to show you in your feed, all the updates from your friend and your groups while blocking all the surveillance, the ads and the slop and the recommendations, and then mixing in the news that you wanted to see.

Remember when we tried to get Netflix to show Canadian content in your recommendations and search results? Anti-circumvention is the only reason some Canadian company can't jailbreak the Netflix app and give you an alternative client that lets you stream all your Netflix shows but also shows you search results from the NFB and any other library of Canadian media, while blocking Netflix's surveillance.

Anticircumvention means that Canadian technologists can't seize the means of computation, which means that we're at the mercy of American companies and we only get the rights that they decide to give us.

Apple will block Facebook's apps from spying on you while you use your iPhone, but they won't let you block Apple from spying on you while you use your iPhone, to gather exactly the same data Facebook steals from you, for exactly the same purpose: to target ads to you.

Apple will screen the apps in its app store to prevent malicious code from running on your iPhone, but if you want to run a legitimate app and Apple doesn't want you to, they will block it from the app store and you will just have to die mad.

That's what's happened in October, when Apple kicked an app called ICE Block out of the App Store. ICE Block is an app that warns you if masked thugs are at large in your neighborhood waiting to kidnap you and send you to a camp. Apple decided that ICE thugs were a "protected class" that ICE Block discriminated against. They decided that you don't deserve to be safe from ICE kidnappings, and what they say goes.

The road to enshittification hell is paved with anticircumvention. We told our politicians this, a decade and a half ago, and they called us "babyish radical extremists" and did it anyway.

Now, I've been shouting about this for decades. I was one of those activists who helped get Sam Bulte unelected and flipped her seat for 20 years. But I will be the first person to tell you that I have mostly failed at preventing enshittification.

Bruce Lehman, James Moore and even Tony "dick pic" Clement are way better at enshittifying the world than I am at disenshittifying it. Of course, they have an advantage over me: they are in a coalition with the world's most powerful corporations and their wealthy investors.

Whereas my coalition is basically, you know, you folks. People who care about human rights, workers' rights, consumer rights, privacy rights. And guys, I hate to tell you, but we're losing.

Let's talk about how we start winning.

Any time you see a group of people successfully push for a change that they've been trying to make unsuccessfully for a long-ass time it's a sure bet that they've found some coalition partners. People who want some of the same things, who've set aside their differences and joined the fight.

That's the Trump story, all over. The Trump coalition is basically, all the billionaires, plus the racists, plus the dopes who'd vote for a slime mold if it promised to lower their taxes by a nickle, even though they somehow expect to have roads and schools. Well, maybe not schools. You know, Ford Nation.

Plus everyone who correctly thinks the Democratic Party are a bunch of do-nothing sellouts, who think they can bully you into voting for genocide because the other guy is an out-and-out fascist.

Billionaires, racists, freaks with low-tax brain-worms and people who hate the sellout Dems: Trump's built a coalition that gets stuff done. Sure, it's terrible stuff, but you can't deny that they're getting it done.

To escape from the enshittificatory black hole that Clement and Moore blew in Canadian policy, we need a coalition, too. And thanks to Trump and his incontinent belligerence, we're getting one.

Let's start with the Trump tariffs. When I was telling you about how anticircumvention law took four tries under two different Prime Ministers, perhaps you wondered "Why did all these Canadian politicians want this stupid law in the first place?"

After all, it's not like Canadian companies are particularly enriched by this law. Sure, it lets Ted Rogers rent you a cable box that won't let you attach a video recorder, so you have to pay for Rogers' PVR, which only lets you record some shows, and deletes them after a set time, and won't let you skip the ads.

But the amount of extra money Rogers makes off this disgusting little racket is dwarfed by the billions that Canadian businesses leave on the table every year, by not going into business disenshittifying America's shitty tech exports. To say nothing of the junk fees and app taxes and data that those American companies rip off every Canadian for.

So why were these Canadian MPs and prime ministers from both the Liberals and the Tories so invested in getting anticircumvention onto our law-books?

Simple: the US Trade Rep threatened us with tariffs if we didn't pass an anti-circuvmention law.

Remember, digital products are slippery. If America bans circumvention, and American companies start screwing the American public, that just opens an opportunity for companies elsewhere in the world to make disenshittifying products, which any American with an internet connection and a payment method can buy. Downloading jailbreaking code is much easier than getting insulin shipped from a Canadian pharmacy!

So the US Trade Rep's top priority for the past quarter-century has been bullying America's trading partners into passing anti-circumvention laws to render their own people defenseless against American tech companies' predation and to prevent non-American tech companies from going into business disenshittifying America's defective goods.

The threat of tariffs was so serious that multiple Canadian PMs from multiple parties tried multiple times to get a law on the books that would protect us from tariffs.

And then in comes Trump, and now we have tariffs anyway.

And let me tell you: when someone threatens to burn your house down if you don't follow their orders, and you follow their orders, and they burn your house down anyway, you are an absolute sucker if you keep following their orders.

We could respond to the tariffs by legalizing circumvention, and unleashing Canadian companies to go into business raiding the margins of the most profitable lines of business of the most profitable corporations the world has ever seen.

Sure, Canada might not ever have a company like Research In Motion again, but what we could have is a company that sells the tools to jailbreak iPhones to anyone who wants to set up an independent iPhone store, bypassing Apple's 30% app tax and its high-handed judgments about what apps we can and can't have.

Apple's payment processing business is worth $100b/year. We could offer people a 90% discount and still make $10b/year. And unlike Apple, we wouldn't have to assume the risk and capital expenditure of making phones. We could stick Apple with all of the risk and expense, and cream off the profits.

That's fair, isn't it? It's certainly how Big Tech operates. When Amazon started, Jeff Bezos said to the publishers, "Your margin is my opportunity." $100b/year off a 30% payment processing fee is a hell of a margin, and a hell of an opportunity.

With Silicon Valley, it's always "disruption for thee, not for me." When they do it to us, that's progress, when we do it to them, it's piracy (and every pirate wants to be an admiral).

Now, of course, Canada hasn't responded to the Trump tariffs with jailbreaking. Our version of "elbows up" turns out to mean retaliatory tariffs. Which is to say, we're making everything we buy from America more expensive for us, which is a pretty weird way of punishing America, eh?

It's like punching yourself in the face really hard and hoping the downstairs neighbour says "Ouch."

Plus, it's pretty indiscriminate. We're not angry at Americans. We're angry at Trump and his financial backers. Tariffing soybeans just whacks some poor farmer in a state that begins and ends with a vowel who's never done anything bad to Canada.

I guarantee you that poor bastard is making payments on a John Deere tractor, which costs him an extra $200 every time it breaks down, because after he fixes it himself, he has to pay two hundred bucks to John Deere and wait two days for them to send out a technician who types an unlock code into the tractor's console that unlocks the "parts pairing," so the tractor recognises the new part.

Instead of tariffing that farmer's soybeans, we could sell him the jailbreaking tool that lets him fix his tractor without paying an extra $200 to John Deere.

Instead of tsking at Elon Musk over his Nazi salute, we could sell every mechanic in the world a Tesla jailbreaking kit that unlocks all the subscription features and software upgrades, without sending a dime to Tesla, kicking Elon Musk square in the dongle.

This is all stuff we could be doing. We could be building gigantic Canadian tech businesses, exporting to a global market, whose products make everything cheaper for every Canadian, and everyone else in the world, including every American.

Because the American public is also getting screwed by these companies, and we could stand on guard for them, too. We could be the Disenshittification Nation.

But that's not what we've done. Instead, we've decided to make everything in Canada more expensive, which is just about the stupidest political strategy I've ever heard of.

This might be the only thing Carney could do that's less popular than firing 10,000 civil servants and replacing them with chatbots on the advice of the world's shadiest art dealer, who is pretty sure that if we keep shoveling words into the word-guessing program it will wake up and become intelligent.

Which is just, you know, stupid. It's like thinking that if we just keep breeding our horses to run faster, one of our mares will eventually give birth to a locomotive. Human beings are not word-guessing programs who know more words than ChatGPT.

Now, it's clear that the coalition of "people who care about digital rights" and "people who want to make billions of dollars off jailbreaking tech" isn't powerful enough to break the coalition that makes hundreds of billions of dollars from enshittification.

But Trump – yes, Trump! – keeps recruiting people to our cause.

Trump has made it clear that America no longer has allies, nor does it have trading partners. It has adversaries and rivals. And Trump's favorite weapons for attacking his foreign adversaries are America's tech giants.

When the International Criminal Court issued an arrest warrant against Bejamin Netanyahu for ordering a genocide, Trump denounced them, and Microsoft shut down their Outlook accounts.

The chief prosecutor and other justices immediately lost access to all the working files of the court, to their email archives, to their diaries and address books.

This was a giant, blinking sign, visible from space, reading AMERICAN TECHNOLOGY CANNOT BE TRUSTED.

Trump's America only has adversaries and rivals, and Trump will pursue dominance by bricking your government, your businesses, your whole country.

It's not just administrative software that Trump can send kill signals to. Remember when those Russian looters stole Ukrainian tractors and they turned up in Crimea? John Deere sent a kill-signal to the tractors and permanently immobilized them.

This was quite a cool little comeuppance, the kind of thing a cyberpunk writer like me can certainly relish. But anyone who thinks about this for, oh, ten seconds will immediately realise that anyone who can push around the John Deere company can order the permanent immobilization of any tractor in the world, or all the tractors in your country.

Because John Deere is a monopolist, and whatever part of the market Deere doesn't control is controlled by Massey Ferguson, and Trump can order the bricking of those tractors, too.

This is the thing we were warned we'd face if we let Huawei provide our telecoms infrastructure, and those warnings weren't wrong. We should be worried about any gadget that we rely on that can be bricked by its manufacturer.

Because that means we are at risk from the manufacturer, from governments who can suborn the manufacturer, from corporate insiders who can hijack the manufacturer's control systems, and from criminals who can impersonate the manufacturer to our devices.

This is the third part of our coalition: not just digital rights weirdos like me; not just investors and technologists looking to make billions; but also national security hawks who are justifiably freaking out about America, China, or someone else shutting down key pieces of their country, from its food supply to its administrative capacity.

Trump is a crisis, and crises precipitate change.

Just look at Europe. Before Putin invaded Ukraine, the EU was a decade behind on its energy transition goals. Now, just a few years later, they're 15 years ahead of schedule.

It turns out that a lot of "impossible" things are really just fights you'd rather not have. No one wants to argue with some tedious German who hates the idea of looking at "ugly solar panels" on their neighbour's balcony. But once you're all shivering in the dark, that's an argument you will have and you will win.

Today, another mad emperor is threatening Europe – and the world. Trump's wanton aggression has given rise to a new anti-enshittification coalition: digital rights advocates, investors and technologists, and national security hawks; both the ones who worry about America, and the ones who worry about China.

That's a hell of a coalition!

The time is right to become a disenshittification nation, to harness our own tech talent, and the technologists who are fleeing Trump's America in droves, along with capital from investors who'd like to back a business whose success isn't determined by how many $TRUMP Coins they buy.

Jailbreaking is how Canada cuts American Big Tech down to size.

It's unlike everything else we've tried, like the Digital Services Tax, or forcing Netflix to support cancon, or making Facebook and Google pay to link to the news.

All of those tactics involve making these companies that are orders of magnitude richer than Canada do something they absolutely do not want to do.

Time and again, they've shown that we don't have the power to make them do things. But you know what Canada has total power over? What Canada does.

We are under no obligation to continue to let these companies use our courts to attack our technologists, our businesses, our security researchers, our tech co-ops, our nonprofits, who want to jailbreak America's shitty tech, to seize the means of computation, to end the era in which American tech companies can raid our wallets and our data with impunity.

In a jailbroken Canada, we don't have to limit ourselves to redistribution, to taxing away some of the money that the tech giants steal from us. In a jailbroken Canada, we can do predistribution. We can stop them from stealing our money in the first place.

And if we don't do it, someone else will. Because every country was arm-twisted into passing an anti-circumvention law like ours. Every country had a supine and cowardly lickspittle like James Moore or Tony Clement who'd do America's bidding, a quisling who'd put their nation's people and businesses in chains, rather than upset the US Trade Rep.

And all of those countries are right where we are: hit with tariffs, threatened by Trump, waiting for the day that Microsoft or Oracle or Google or John Deere bricks their businesses, their government, their farms.

One of those countries is going to jump at this opportunity, the opportunity to consume the billions in rents stolen by US Tech giants, and use them as fuel for a single-use rocket booster that launches their tech sector into a stable orbit for decades to come.

That gives them the hottest export business in living memory: a capital-light, unstoppable suite of products that save businesses and consumers money, while protecting their privacy.

If we sleep on this, we'll still benefit. We'll get the consumer surplus that comes from buying those jailbreaking tools online and using them to disenshittify our social media, our operating systems, our vehicles, our industrial and farm equipment.

But we won't get the industrial policy, the chance to launch a whole sector of businesses, each with the global reach and influence of RIM or Nortel.

That'll go to someone else. The Europeans are already on it. They're funding and building the "Eurostack": free, open source, auditable and trustworthy versions of the US tech silos. We're going to be able to use that here.

I mean, why not? We'll just install that code on metal running in Canadian data-centres, and we'll debug it and add features to it, and so will everyone else.

Because that's how IT should work, and it should go beyond just the admin and database software that businesses and governments rely on. We should be building drop-in, free, open software for everything: smart speakers, smart TVs, smart watches, phones, cars, tractors, powered wheelchairs, ventilators.

That's how it should already be: that the software that powers these devices that we entrust with our data, our integrity, our lives should be running code that anyone can see, test, and improve.

That's how science works, after all. Before we had science, we had something kind of like science. We had alchemy. Alchemy was very similar to science, in that an alchemist would observe some natural phenomena in the world, hypothesise a causal relationship between them, and design an experiment to validate that hypothesis.

But here's where alchemy and science diverge: unlike a scientist, an alchemist wouldn't publish their results. They'd keep them secret, rather than exposing them to the agony of adversarial peer review, where your enemies seek out every possible reason to discredit your work. This let the alchemists kid themselves about the stuff they thought they'd discovered, and that's why every alchemist discovered for themself, in the hardest way possible, that you shouldn't drink mercury.

But after 500 years of this, alchemy finally achieved its long sought-after goal of converting something common to something of immeasurable value. Alchemy discovered how to transform the base metal of superstition into the precious metal of knowledge, through the crucible of publishing.

Disclosure is the difference between knowledge and ignorance. Openness is the difference between dying of mercury poisoning and discovering medicine.

The fact that we have a law on our statute books, in the year of two thousand and twenty-five, that criminalises discovering how the software we rely on works, and telling other people about it and improving it – well, it's pretty fucking pathetic, isn't it?

We don't have to keep on drinking the alchemists' mercury. We don't have to remain prisoners of the preposterous policy blunders of Tony Clement and James Moore. We don't have to tolerate the endless extraction of Big Tech. We don't have to leave billions on the table. We need not abide the presence of lurking danger in all our cloud-connected devices.

We can be the vanguard of a global movement of international nationalism, of digital sovereignty grounded in universal, open, transparent software, a commons that everyone contributes to and relies upon. Something more like science than technology.

Like the EU's energy transition, this is a move that's long overdue. Like the EU's energy transition, a mad emperor has created the conditions for us to get off of our asses, to build a better world.

We could be a disenshittification nation. We could seize the means of computation. We could have a new, good internet that respects our privacy and our wallets. We could make a goddamned fortune doing it.

And once we do it, we could protect ourselves from spineless digital vassals of the mad king on our southern border, and rescue our American cousins to boot.

What's not to like?

Hey look at this (permalink)

The UK Has It Wrong on Digital ID. Here’s Why https://www.eff.org/deeplinks/2025/11/uk-has-it-wrong-digital-id-heres-why
Sex Workers Built an ‘Anti-OnlyFans’ to Take Control of Their Profits https://www.wired.com/story/sex-workers-built-an-anti-onlyfans-to-take-control-of-their-profits/

Object permanence (permalink)

#20yrago Ten (sensible) startup rules https://web.archive.org/web/20060324072607/https://evhead.com/2005/11/ten-rules-for-web-startups.asp

#20yrsago Bosnian town unveils Bruce Lee statue of peace http://news.bbc.co.uk/2/hi/entertainment/4474316.stm

#20yrsago Sony rootkit author asked for free code to lock up music https://web.archive.org/web/20051130023447/https://groups.google.de/group/microsoft.public.windowsmedia.drm/msg/7cb5c4ad49fa206e

#20yrsago Singapore’s executioner gets fired http://news.bbc.co.uk/2/hi/asia-pacific/4477012.stm

#20yrsago Pre-history of the Sony rootkit https://web.archive.org/web/20181126020952/https://community.osr.com/discussion/42117#T3

#15yrsago Support the magnetic ribbon industry ribbon! https://www.reddit.com/r/pics/comments/ecr1t/ill_see_your_empty_gesture_and_raise_you/

#15yrsago Molecular biologist on the dangers of pornoscanners https://web.archive.org/web/20101125192455/https://myhelicaltryst.blogspot.com/2010/11/tsa-x-ray-backscatter-body-scanner.html

#15yrsago Wunderkammerer front room crammed with nooks https://web.archive.org/web/20101125184317/http://mocoloco.com/fresh2/2010/11/23/villa-j-by-marge-arkitekter.php

#15yrsago Delightful science fiction story in review of $6800 speaker cable https://www.amazon.com/review/R3I8VKTCITJCX6/ref=cm_cr_dp_perm?ie=UTF8&ASIN=B000J36XR2&nodeID=172282&tag=&linkCode=

#15yrsago German Pirate Party members strip off for Berlin airport scanner protest https://web.archive.org/web/20101129043459/https://permaculture.org.au/2010/11/26/full-monty-scanner-or-enhanced-pat-down-the-only-options/

#10yrsago Dolphin teleportation symposium: now with more Eisenhowers! https://twitpic.com/3aqqa0

#10yrsago Vtech breach dumps 4.8m families’ information, toy security is to blame https://arstechnica.com/information-technology/2015/11/when-children-are-breached-inside-the-massive-vtech-hack/

#10yrsago A Canadian teenager used America’s militarized cops to terrorize women gamers for years https://www.nytimes.com/2015/11/29/magazine/the-serial-swatter.html?_r=0

#10yrsago What the 1980s would have made of the $5 Raspberry Pi https://www.wired.com/beyond-the-beyond/2015/11/raspberry-pi-five-bucks-us/

#10yrsago Workaholic Goethe wished he’d been better at carving out time for quiet reflection https://www.wired.com/beyond-the-beyond/2015/11/the-aged-herr-goethe-never-had-enough-time-for-himself/

Upcoming appearances (permalink)

San Diego: Enshittification at the Mission Hills Branch Library, Dec 1
https://libraryfoundationsd.org/events/doctorow
Seattle: Neuroscience, AI and Society (University of Washington), Dec 4
https://www.eventbrite.com/e/neuroscience-ai-and-society-cory-doctorow-tickets-1735371255139
Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html

Recent appearances (permalink)

Escape Forward with Cristina Caffarra
https://escape-forward.com/2025/11/27/enshittification-of-our-digital-experience/
Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8
How the internet went to sh*t (Prospect Magazine)
https://www.prospectmagazine.co.uk/podcasts/prospect-podcast/71663/cory-doctorow-how-the-internet-went-to-sht
Enshittification and “Breaking Kings” (Web Summit)
https://www.youtube.com/watch?v=CpLudlrwS_g
Enshittification Nation (The Lever)
https://www.levernews.com/enshittification-nation/
Enshittification with Oh God What Now
https://castbox.fm/episode/Why-Tech-Sucks-%E2%80%93%C2%A0Cory-Doctorow-on-Enshittification-and-how-to-fix-it-id4634015-id876127534
Enshittification with The Lede (New Lines Magazine)
https://newlinesmag.com/podcast/why-the-internet-got-bad-and-how-to-fix-it/

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

Thu, 27 Nov 2025 12:00:23 +0000 Fullscreen Open in Tab

Pluralistic: Normie diffusion and technophilia (27 Nov 2025)

Today's links

Normie diffusion and technophilia: Who will seize the means of computation?
Hey look at this: Delights to delectate.
Object permanence: Centaur teddybears; Vaginal yeast sourdough; Snoopers Charter rebuttal; "The Paradox"; Bossware is unfair.
Upcoming appearances: Where to find me.
Recent appearances: Where I've been.
Latest books: You keep readin' em, I'll keep writin' 'em.
Upcoming books: Like I said, I'll keep writin' 'em.
Colophon: All the rest.

Normie diffusion and technophilia (permalink)

It's an accepted (but wrong) fact that some groups of people are just more technologically adventurous by temperament, and that's why they adopt technologies before the rest of society (think here of pornographers, kids, and terrorists).

As I've written before, these groups aren't more (or less) temperamentally inclined to throw themselves into mastering new technologies. Rather, they have more reason to do so:

https://pluralistic.net/2022/06/21/early-adopters/#sex-tech

Whenever a new communications technology arrives, it is arriving into a world of existing communications technologies, which are, by definition, easier to use. They're easier to use for two reasons: the obvious reason is that you're more likely to be familiar with an existing technology than you are with a new technology. After all, it's literally impossible to be familiar with a technology that has just been invented!

But the other reason that existing communications technologies are easier to use is that communication is – again, by definition – something you do with other people. That means that if you want to use a new communications tool to talk with someone else, it is not sufficient for you to master that technology's use – you must also convince the other person you're hoping to reach to master that technology, too.

In economic terms, the "opportunity cost" (the amount of time you lose for doing other things) of mastering a new communications tool isn't limited to your own education, but also to the project of convincing someone else to master that tool, and then showing them how to use it.

If the existing communications technology is working for you, mastering the new tool is mostly cost, with very little upside. Perhaps you are a technophile by temperament and derive intrinsic satisfaction from exploring a new tool, and that's why you do it, but even so, you're going to find yourself in the bind of trying to convince the people you'd like to communicate with to follow your lead. And if they're all being well-served by the existing communications tools, and if they're not technophiles, you're asking them to engage in a lot of labor and endure a high opportunity cost for no obvious benefit. It's a hard slog.

But there are many groups of people for whom the existing technology does not work, and one of the biggest ways an existing technology can fail is if the authorities are using it to suppress your communications and/or spy on your usage in order to frustrate your goals.

This brings us back to sex workers, kids and terrorists. All three groups are typically poorly served by the existing communications technology. If you're a pornographer in the age of celluloid film, you either have to convince your customers to visit (and risk being seen entering) an adult movie theater, or you have to convince them to buy an 8mm projector and mail order your reels (and risk being caught having them delivered).

No wonder pornographers and sex workers embraced the VCR! No wonder they embraced the internet! No wonder they embraced cryptocurrency (if your bank accounts are liable to being frozen and/or seized, it's worth figuring out how to use an esoteric payment method and endure the risk of its volatility and technological uncertainty). Today, sex workers and their customers are doubtless mastering VPNs (to evade anonymity-stripping "age verification" systems) and Tor hidden services (to evade "online safety" laws).

The alternative to using these systems isn't the status quo – making use of existing websites, existing payment methods, existing connection tools. The alternative is nothing. So it's worth learning to use these new tools, and to engage in the social labor of convincing others to join you in using them.

Then there's kids. Unlike sex workers, kids' communications aren't broadly at risk of being suppressed so much as they are at risk of being observed by authority figures with whom they have an adversarial relationship.

When you're a kid, you want to talk about things without your parents, teachers, principals, or (some of) your peers or siblings listening in. You want to plan things without these people listening in, because they might try and stop you from doing them, or punish you if you succeed.

So again, it's worth figuring out how to use new technologies, because the existing ones are riddled with censorship and surveillance back-doors ("parental controls") that can be deployed to observe your communications, interdict your actions, and punish you for the things that you manage to pull off.

So of course kids are also "early adopters" – but not because being a kid makes you a technophile. Many kids are technophiles and many are not, but whether or not a kid finds mastering a new technology intrinsically satisfying, they will likely have to do so, if they want to communicate with their peers.

For terrorists, the case for mastering new technologies combines the sex-workers' cases and kids' cases: terrorists' communications are both illegal and societally unacceptable (like sexual content) and terrorists operate in an environment in which entities far more powerful than them seek to observe and interdict their plans, and punish them after the fact for their actions (like kids).

So once again, terrorists are apt to master new communications technologies, but not because seeking to influence political outcomes by acts of violence against civilian populations is somehow tied to deriving intrinsic satisfaction from mastering new technologies, but rather because the existing technologies are dangerously unsuitable for your needs.

Note that just because being in one of these groups doesn't automatically make you a technophile, it doesn't mean that there are no technophiles among these groups. Some people are into tech and the sex industry. Some kids love mastering new technologies. Doubtless this is true of some terrorists, too.

I haven't seen any evidence that being a kid, or a terrorist, or a sex-worker, makes you any less (or any more) interested in technology than anyone else. Some of us just love this stuff for its own sake. Other people just want a tool that works so they can get on with their lives. That's true of every group of people.

The difference is that if you're a technophile in a group of people who have a damned good reason to endure the opportunity cost of mastering a new technology, you have a much more receptive audience for your overheated exhortations to try this amazing new cool thing you've discovered.

What's more, there are some situational and second-order effects that come into play as a result of these dynamics. For example, kids are famously "cash-poor and time-rich" which means that spending the time to figure out new technologies when they're still in stage one of enshittification (when they deliver a lot of value at their lowest cost, often free) is absolutely worth it.

Likewise, the fact that sex-workers are often the first commercial users of a new communications technology means that there's something especially ugly about the fact that these services jettison sex workers the instant they get leaned on by official prudes. The story of the internet is the story of businesses who owe their commercial existence to sex workers, who have since rejected them and written them out of their official history.

It also means that technophiles who aren't kids, pornographers or terrorists are more likely to find themselves in techno-social spaces that have higher-than-average cohorts of all three groups. This means that bright young technologists can find themselves being treated as peers by accomplished adults (think of Aaron Swartz attending W3C meetings as a pre-teen after being welcomed as a peer in web standardization online forums).

It also means that technophiles are more likely than the average person to have accidentally clicked on a terrorist atrocity video. And it means that pornographers and sex-workers are more likely to be exposed to technologically adventurous people in purely social, non-sexual online interactions, because they're among the first arrivals in new technological spaces, when they are still mostly esoteric, high-tech realms, which means that even among the less technophilic members of that group, there's probably an above-average degree of familiarity with things that are still way ahead of the tech mainstream.

My point is that we should understand that the adoption of technology by disfavored, at risk, or prohibited groups is driven by material factors, not by some hidden ideological link between sex and tech, or youth and tech, or terrorism and tech.

Hey look at this (permalink)

Chasing Oligarchs Across Borders https://gijn.org/stories/gijc25-cross-border-investigations-tips-oligarchs/
Canada Needs a Commissioner Who Fights Monopolies https://action.openmedia.org/page/182266/petition/1
Fifteen Years https://xkcd.com/3172/
The Rent Is Too Damn High: Did Trump Just 'Bless' Using AI to Jack Up Rents? https://www.thebignewsletter.com/p/an-odd-settlement-on-rent-fixing
Biretrograde Clock – Lasercut https://www.instructables.com/Biretrograde-Clock-Lasercut/

Object permanence (permalink)

#20yrsago TSA makes flier remove body jewelry https://web.archive.org/web/20051129025951/https://pittsburghlive.com/x/tribune-review/s_397618.html

#20yrsago Microsoft caught subverting UN process, censoring FOSS references https://web.archive.org/web/20051128030303/https://news.zdnet.co.uk/software/linuxunix/0,39020390,39238443,00.htm

#15yrsago Zimbabwean law will put legislation, parliamentary gazette, etc, under state copyright https://web.archive.org/web/20101129133649/https://www.theindependent.co.zw/local/28907-general-laws-bill-inimical-to-democracy.html

#10yrsago Steiff Japan’s centaur teddybears http://www.steiff-shop.jp/2007w_ltd/037351_seet.html

#10yrsago Woman adds vaginal yeast to sourdough starter, Internet flips out https://web.archive.org/web/20180808194241/https://anotherangrywoman.com/2015/11/25/baking-and-eating-cuntsourdough/

#10yrsago Party like it’s 1998: UK government bans ripping CDs — again https://arstechnica.com/tech-policy/2015/11/thanks-to-the-music-industry-it-is-illegal-to-make-private-copies-of-music-again/

#10yrsago Devastating technical rebuttal to the Snoopers Charter https://www.me.uk/IPBill-evidence1.pdf

#10yrsago AIDS-drug-gouging hedge-douche reneges on promise to cut prices for Daraprim https://www.techdirt.com/2015/11/25/turing-refuses-to-lower-cost-daraprim-hides-news-ahead-thanksgiving-holiday/

#10yrsago US credit union regulator crushed Internet Archive’s non-predatory, game-changing bank https://blog.archive.org/2015/11/24/difficult-times-at-our-credit-union/

#10yrsago The last quarter-century of climate talks explained, in comics form https://web.archive.org/web/20151126142914/http://www.nature.com/news/the-fragile-framework-1.18861

#10yrsago The Paradox: a secret history of magical London worthy of Tim Powers https://memex.craphound.com/2015/11/26/the-paradox-a-secret-history-of-magical-london-worthy-of-tim-powers/

#1yrago Bossware is unfair (in the legal sense, too) https://pluralistic.net/2024/11/26/hawtch-hawtch/#you-treasure-what-you-measure

Upcoming appearances (permalink)

Toronto: Jailbreaking Canada (OCAD U), Nov 27
https://www.ocadu.ca/events-and-exhibitions/jailbreaking-canada
San Diego: Enshittification at the Mission Hills Branch Library, Dec 1
https://libraryfoundationsd.org/events/doctorow
Seattle: Neuroscience, AI and Society (University of Washington), Dec 4
https://www.eventbrite.com/e/neuroscience-ai-and-society-cory-doctorow-tickets-1735371255139
Virtual: Poetic Technologies with Brian Eno (David Graeber Institute), Dec 8
https://davidgraeber.institute/poetic-technologies-with-cory-doctorow-and-brian-eno/
Madison, CT: Enshittification at RJ Julia, Dec 8
https://rjjulia.com/event/2025-12-08/cory-doctorow-enshittification
Hamburg: Chaos Communications Congress, Dec 27-30
https://events.ccc.de/congress/2025/infos/index.html

Recent appearances (permalink)

Why Every Platform Betrays You (Trust Revolution)
https://fountain.fm/episode/bJgdt0hJAnppEve6Qmt8
How the internet went to sh*t (Prospect Magazine)
https://www.prospectmagazine.co.uk/podcasts/prospect-podcast/71663/cory-doctorow-how-the-internet-went-to-sht
Enshittification and “Breaking Kings” (Web Summit)
https://www.youtube.com/watch?v=CpLudlrwS_g
Enshittification Nation (The Lever)
https://www.levernews.com/enshittification-nation/
Enshittification with Oh God What Now
https://castbox.fm/episode/Why-Tech-Sucks-%E2%80%93%C2%A0Cory-Doctorow-on-Enshittification-and-how-to-fix-it-id4634015-id876127534
Enshittification with The Lede (New Lines Magazine)
https://newlinesmag.com/podcast/why-the-internet-got-bad-and-how-to-fix-it/

Latest books (permalink)

"Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025
"Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
https://us.macmillan.com/books/9780374619329/enshittification/
"Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).
"The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org).
"The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).
"The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

Upcoming books (permalink)

"Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026
"Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026
"The Memex Method," Farrar, Straus, Giroux, 2026
"The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026

Colophon (permalink)

Today's top sources:

Currently writing:

"The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.
"The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.
A Little Brother short story about DIY insulin PLANNING

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

ISSN: 3066-764X

2025-11-25T13:25:00-08:00 Fullscreen Open in Tab

Client Registration and Enterprise Management in the November 2025 MCP Authorization Spec

The new MCP authorization spec is here! Today marks the one-year anniversary of the Model Context Protocol, and with it, the launch of the new 2025-11-25 specification.

I’ve been helping out with the authorization part of the spec for the last several months, working to make sure we aren't just shipping something that works for hobbyists, but something that even scales to the enterprise. If you’ve been following my posts like Enterprise-Ready MCP or Let's Fix OAuth in MCP, you know this has been a bit of a journey over the past year.

The new spec just dropped, and while there are a ton of great updates across the board, far more than I can get in to in this blog post, there are two changes in the authorization layer that I am most excited about. They fundamentally change how clients identify themselves and how enterprises manage access to AI-enabled apps.

Client ID Metadata Documents (CIMD)

If you’ve ever tried to work with an open ecosystem of OAuth clients and servers, you know the "Client Registration" problem. In traditional OAuth, you go to a developer portal, register your app, and get a client_id and client_secret. That works great when there is one central server (like Google or GitHub) and many clients that want to use that server.

It breaks down completely in an open ecosystem like MCP, where we have many clients talking to many servers. You can't expect a developer of a new AI Agent to manually register with every single one of the 2,000 MCP servers in the MCP server registry. Plus, when a new MCP server launches, that server wouldn't be able to ask every client developer to register either.

Until now, the answer for MCP was Dynamic Client Registration (DCR). But as implementation experiences has shown us over the last several months, DCR introduces a massive amount of complexity and risk for both sides.

For Authorization Servers, DCR endpoints are a headache. They require public-facing APIs that need strict rate limiting to prevent abuse, and they lead to unbounded database growth as thousands of random clients register themselves. The number of client registrations will only ever increase, so the authorization server is likely to implement some sort of "cleanup" mechanism to delete old client registrations. The problem is there is no clear definition of what an "old" client is. And if a dynamically registered client is deleted, the client doesn't know about it, and the user is often stuck with no way to recover. Because of the security implications of an endpoint like this, DCR has also been a massive barrier to enterprise adoption of MCP.

For Clients, it’s just as bad. They have to manage the lifecycle of their client credentials on top of the actual access tokens, and there is no standardized way to check if the client registration is still valid. This frequently leads to sloppy implementations where clients simply register a brand new client_id every single time a user logs in, further increasing the number of client registrations at the authorization server. This isn't a theoretical problem, this is also how Mastodon has worked for the last several years, and has some GitHub issue threads describing the challenges it creates.

The new MCP spec solves this by adopting Client ID Metadata Documents.

The OAuth Working Group adopted the Client ID Metadata Document spec in October after about a year of discussion, so it's still relatively new. But seeing it land as the default mechanism in MCP is huge. Instead of the client registering with each authorization server, the client establishes its own identity with a URL it controls and uses the URL to identify itself during an OAuth flow.

When the client starts an OAuth request to the MCP authorization server, it says, "Hi, I'm https://example-app.com/client.json." The server fetches the JSON document at that URL and finds the client's metadata (logo, name, redirect URIs) and proceeds on as usual.

This creates a decentralized trust model based on DNS. If you trust example.com, you trust the client. It removes the registration friction entirely while keeping the security guarantees we need. It’s the same pattern we’ve used in IndieAuth for over a decade, and it fits MCP perfectly.

There are definitely some new considerations and risks this brings, so it's worth diving into the details about Client ID Metadata Documents in the MCP spec as well as the IETF spec. For example, if you're building an MCP client that is running on a web server, you can actually manage private keys and publish the public keys in your metadata document, enabling strong client authentication. And like Dynamic Client Registration, there are still limitations for how desktop clients can leverage this, which can hopefully be solved by a future extension. I talked more about this during a hugely popular session at the Internet Identity Workshop in October, you can find the slides here.

You can try out this new flow today in VSCode, the first MCP client to ship support for CIMD even before it was officially in the spec. You can also learn more and test it out at the excellent website the folks at Stytch created: client.dev.

Enterprise-Managed Authorization (Cross App Access)

This is the big one for anyone asking, "Is MCP safe to use in the enterprise?"

Until now, when an AI agent connected to an MCP server, the connection was established directly between the MCP client and server. For example if you are using ChatGPT to connect to the Asana MCP server, ChatGPT would start an OAuth flow to Asana. But if your Asana account is actually connected to an enterprise IdP like Okta, Okta would only see that you're logging in to Asana, and wouldn't be aware of the connection established between ChatGPT and Asana. This means today there are a huge number of what are effectively unmanaged connections between MCP clients and servers in the enterprise. Enterprise IT admins hate this because it creates "Shadow IT" connections that bypass enterprise policy.

The new MCP spec incorporates Cross App Access (XAA) as the authorization extension "Enterprise-Managed Authorization".

This builds on the work I discussed in Enterprise-Ready MCP leveraging the Identity Assertion Authorization Grant. The flow puts the enterprise Identity Provider (IdP) back in the driver's seat.

Here is how it works:

Single Sign-On: First you log into an MCP Client (like Claude or an IDE) using your corporate SSO, the client gets an ID token.
Token Exchange: Instead of the client starting an OAuth flow to ask the user to manually approve access to a downstream tool (like an Asana MCP server), the client takes that ID token back to the Enterprise IdP to ask for access.
Policy Check: The IdP checks corporate policy. "Is Engineering allowed to use Claude to access Asana?" If the policy passes, the IdP issues a temporary token (ID-JAG) that the client can take to the MCP authorization server.
Access Token Request: The MCP client takes the ID-JAG to the MCP authorization server saying "hey this IdP says you can issue me an access token for this user". The authorization server validates the ID-JAG the same way it would have validated an ID Token (remember this app is also set up for SSO to the same corporate IdP), and issues an access token.

This happens entirely behind the scenes without user interaction. The user doesn't get bombarded with consent screens, and the enterprise admin gets full visibility and revocability. If you want to shut down AI access to a specific internal tool, you do it in one place: your IdP.

Digital asset treasury companies are running out of steam

Convincing traders to pay $2 for $1 of bitcoin worked — for a while. As premiums evaporate, an unwind could be painful.

2025-11-25T08:07:14-08:00 Fullscreen Open in Tab

Recurring Events for Meetable

In October, I launched an instance of Meetable for the MCP Community. They've been using it to post working group meetings as well as in-person community events. In just 2 months it already has 41 events listed!

One of the aspects of opening up the software to a new community is stress testing some of the design decisions. An early design decision was intentionally to not support recurring events. For a community calendar, recurring events are often problematic. Once a recurring event is created for something like a weekly meetup, it's no longer clear whether the event is actually going to happen, which is especially true for virtual events. If an organizer of the event silently drops away from the community, it's very likely they will not go delete the event, and you can end up with stale events on the calendar quickly. It's better to have people explicitly create the event on the calendar so that every event was created with intention. To support this, I made a "Clone Event" button to quickly copy the details from a previous instance, and it even predicts the next date based on how often the event has been happening in the past.

But for the MCP community, which is a bit more formal than a purely community calendar, most of the events on their site are weekly or biweekly working group meetings. I had been hearing quite a bit of feedback that the current process of scheduling out the events manually, even with the "clone event" feature, was too much of a burden. So I set out to design a solution for recurring events to strike a balance between ease of use and hopefully avoiding some of the pitfalls of recurring events.

What I landed on is this:

You can create an "event template" from any existing event on the calendar, and give it a recurrence interval like "Every week on Tuesdays" or "Monthly on the 9th".

(I'll add an option for "Monthly on the second Tuesday" later if this ends up being used enough.)

Once the schedule is created, copies of the event will be created at the chosen interval, but only a few weeks out. For weekly events, 4 weeks in advance will be created, biweekly will get scheduled 8 weeks out, monthly events 4 months out, and yearly events will have only the next year scheduled. Every day a cron job will create future events at the scheduled interval in advance. If the event template is deleted, future scheduled events will also be deleted.

So effectively for organizers there is nothing they need to do after creating the recurring event schedule. My hope is by having it work this way, instead of like recurring events on a typical Google calendar, it strikes a balance between ease of use but avoids orphaned events on the calendar. It still requires an organizer to delete a recurrence, so should only be used for events that truly have a schedule and are unlikely to be cancelled often.

Hopefully this makes Meetable even more useful for different kinds of communities! You can install your own copy of Meetable from the source code on GitHub.

2025-11-19T22:46:01+00:00 Fullscreen Open in Tab

Read "The Charlie Kirk purge: How 600 Americans were punished in a pro-Trump crackdown"

Read:

“The Charlie Kirk purge: How 600 Americans were punished in a pro-Trump crackdown”.

Raphael Satter and A.J. Vicens in Reuters. Published November 19, 2025.

Two months after Charlie Kirk's assassination, a government-backed campaign has led to firings, suspensions, investigations and other action against more than 600 people. Republican officials have endorsed the punishments, saying that those who glorify violence should be removed from positions of trust.

2025-11-20T00:51:04+00:00 Fullscreen Open in Tab

Published on Citation Needed: "Issue 97 – This is hardship"

Published an issue of Citation Needed:

Issue 97 – This is hardship

Crown Prince Mohammed bin Salman and President Donald Trump pose together making thumbs-up gestures. On the right is a render of the proposed Trump Maldives hotel.

While slumping prices have some fearing it’s crypto winter again, Trump looks to Saudia Arabia and American retail crypto investors to fund the development of his next hotel

2025-11-18T20:31:13+00:00 Fullscreen Open in Tab

Note published on November 18, 2025 at 8:31 PM UTC

that’s an unfortunate Google snippet

Google search result titled: “FBI Records: The Vault – Jeffrey Epstein”. Snippet text: “Get a Job! FBI Fun and Games! Personal tools. Log in • Mail Image Get FBI ...”

2025-11-10T23:49:18+00:00 Fullscreen Open in Tab

Note published on November 10, 2025 at 11:49 PM UTC

huh

2025-11-10T16:12:25+00:00 Fullscreen Open in Tab

Note published on November 10, 2025 at 4:12 PM UTC

i guess Black Friday is just November now

2025-11-07T23:50:21+00:00 Fullscreen Open in Tab

Note published on November 7, 2025 at 11:50 PM UTC

thanks Google AI!

2025-11-07T19:30:52+00:00 Fullscreen Open in Tab

Published on Citation Needed: "Issue 96 – Redefining solvency"

Published an issue of Citation Needed:

Issue 96 – Redefining solvency

A photo of Sam Bankman-Fried superimposed over a photo of the NY appeals courthouse

Sam Bankman-Fried makes his case for a retrial and aspiring crypto banks hit roadblocks

2025-11-07T17:35:59+00:00 Fullscreen Open in Tab

Note published on November 7, 2025 at 5:35 PM UTC

thank you Anthony Scaramucci for this extremely relatable analogy about people buying bitcoin on margin

“The margin is a spear coming out of the steering wheel of your sports car, and you’re in the south of France on an icy road in the mountain terrain by Monaco,” said SkyBridge Capital founder Anthony Scaramucci in an interview. “And when you need to hit the brake, that’s when the leverage hurts you the most.”

2025-11-06T17:52:51+00:00 Fullscreen Open in Tab

Finished reading Mage Tank 2

Finished reading:

Mage Tank series, book 2.

Mage Tank 2

by Cornman.

Published 2025. 694 pages.

Started October 21, 2025; completed November 5, 2025.

2025-11-03T22:52:08+00:00 Fullscreen Open in Tab

Published on Citation Needed: "Trump says he has “no idea” who he just pardoned"

Published an issue of Citation Needed:

Trump says he has “no idea” who he just pardoned

Donald Trump is interviewed by Norah O’Donnell at the White House for CBS’ 60 Minutes.

President Trump reacts to condemnations of his recent pardon of Binance founder Changpeng Zhao by claiming he doesn’t know who he is.

2025-11-03T19:09:58+00:00 Fullscreen Open in Tab

Note published on November 3, 2025 at 7:09 PM UTC

The full CBS interview with Trump about the pardon of Binance's Changpeng Zhao is shocking. "Why did you pardon him?" "I have no idea who he is. I was told that he was a victim ... They sent him to jail and they really set him up. That's my opinion. I was told about it."

NORAH O'DONNELL: Looked at this, the government at the time said that C.Z. had caused

"I know nothing about it because I'm too busy." He talks about how his sons are in the crypto industry, and how his son and wife published bestselling books. "I'm proud of them for doing that. I'm focused on this."

NORAH O'DONNELL: The government had accused him of

"[You're] not concerned about the appearance of corruption with this?"

"I'd rather not have you ask the question."

NORAH O'DONNELL: So not concerned about the appearance of corruption with this? PRESIDENT DONALD TRUMP: I can't say, because-- I can't say-- I'm not concerned. I don't-- I'd rather not have you ask the question. But I let you ask it. You just came to me and you said,

2025-11-02T19:41:06+00:00 Fullscreen Open in Tab

Reviewing the 13 books I read in September and October

Missed my reading wrap-up for September and have been too busy to read as much as usual, so here’s a combined September/October wrap-up. Lots of litRPG, and James S. A. Corey’s Caliban’s War (The Expanse #2) was definitely a highlight!

@molly0xfff September and October reading wrap-up, reviewing the 13 books I read those months (no spoilers) #readingwrapup #octoberreadingwrapup #booktok #litrpg #bookrecommendations ♬ original sound - Molly White

Storygraph September 2025 wrap-up page. Books: 10; pages: 4,124; av. rating 3.94. Highest rated reads: Demon World Boba Shop Vol. 2 (4.5 stars), Discount Dan (4 stars), Demon World Boba Shop Vol. 4 (4 stars). Average book length: 400 pages; average time to finish: 4 days. 100% fiction. 60% digital, 40% audio.

Storygraph October 2025 wrap-up page. Books: 3; pages: 2,426; av. rating 4.0. Highest rated reads: Caliban's War (4.5 stars), Cul-de-sac Carnage (4 stars), Mage Tank (4 stars). Average book length: 653 pages; average time to finish: 11 days. 100% fiction. 67% digital, 33% audio.

2025-11-02T14:54:53+00:00 Fullscreen Open in Tab

Read "Some People Can't See Mental Images. The Consequences Are Profound"

Read:

“Some People Can't See Mental Images. The Consequences Are Profound”.

Larissa MacFarquhar in The New Yorker. Published October 27, 2025.

Larissa MacFarquhar writes about the recent research into the neurodiverse syndromes known as aphantasia and hyperphantasia, their effects on our experience of trauma and memory, and the sense of identity that has grown up around them.

2025-10-30T15:12:32+00:00 Fullscreen Open in Tab

Note published on October 30, 2025 at 3:12 PM UTC

“I have a great and considerable fear that people will freeze to death in their homes this winter if we do not turn this around quickly.”

2025-10-11T09:49:59-07:00 Fullscreen Open in Tab

Adding Support for BlueSky to IndieLogin.com

Today I just launched support for BlueSky as a new authentication option in IndieLogin.com!

IndieLogin.com is a developer service that allows users to log in to a website with their domain. It delegates the actual user authentication out to various external services, whether that is an IndieAuth server, GitHub, GitLab, Codeberg, or just an email confirmation code, and now also BlueSky.

This means if you have a custom domain as your BlueSky handle, you can now use it to log in to websites like indieweb.org directly!

Alternatively, you can add a link to your BlueSky handle from your website with a rel="me atproto" attribute, similar to how you would link to your GitHub profile from your website.

<a href="https://example.bsky.social" rel="me atproto">example.bsky.social</a>

Full setup instructions here

This is made possible thanks to BlueSky's support of the new OAuth Client ID Metadata Document specification, which was recently adopted by the OAuth Working Group. This means as the developer of the IndieLogin.com service, I didn't have to register for any BlueSky API keys in order to use the OAuth server! The IndieLogin.com website publishes its own metadata which the BlueSky OAuth server can use to fetch the metadata from. This is the same client metadata that an IndieAuth server will parse as well! Aren't standards fun!

The hardest part about the whole process was probably adding DPoP support. Actually creating the DPoP JWT wasn't that bad but the tricky part was handling the DPoP server nonces sent back. I do wish we had a better solution for that mechanism in DPoP, but I remember the reasoning for doing it this way and I guess we just have to live with it now.

This was a fun exercise in implementing a bunch of the specs I've been working on recently!

Here's the link to the full ATProto OAuth docs for reference.

2025-10-10T00:00:00+00:00 Fullscreen Open in Tab

Notes on switching to Helix from vim

Hello! Earlier this summer I was talking to a friend about how much I love using fish, and how I love that I don’t have to configure it. They said that they feel the same way about the helix text editor, and so I decided to give it a try.

I’ve been using it for 3 months now and here are a few notes.

why helix: language servers

I think what motivated me to try Helix is that I’ve been trying to get a working language server setup (so I can do things like “go to definition”) and getting a setup that feels good in Vim or Neovim just felt like too much work.

After using Vim/Neovim for 20 years, I’ve tried both “build my own custom configuration from scratch” and “use someone else’s pre-buld configuration system” and even though I love Vim I was excited about having things just work without having to work on my configuration at all.

Helix comes with built in language server support, and it feels nice to be able to do things like “rename this symbol” in any language.

the search is great

One of my favourite things about Helix is the search! If I’m searching all the files in my repository for a string, it lets me scroll through the potential matching files and see the full context of the match, like this:

For comparison, here’s what the vim ripgrep plugin I’ve been using looks like:

There’s no context for what else is around that line.

the quick reference is nice

One thing I like about Helix is that when I press g, I get a little help popup telling me places I can go. I really appreciate this because I don’t often use the “go to definition” or “go to reference” feature and I often forget the keyboard shortcut.

some vim -> helix translations

Helix doesn’t have marks like ma, 'a, instead I’ve been using Ctrl+O and Ctrl+I to go back (or forward) to the last cursor location
I think Helix does have macros, but I’ve been using multiple cursors in every case that I would have previously used a macro. I like multiple cursors a lot more than writing macros all the time. If I want to batch change something in the document, my workflow is to press % (to highlight everything), then s to select (with a regex) the things I want to change, then I can just edit all of them as needed.
Helix doesn’t have neovim-style tabs, instead it has a nice buffer switcher (<space>b) I can use to switch to the buffer I want. There’s a pull request here to implement neovim-style tabs. There’s also a setting bufferline="multiple" which can act a bit like tabs with gp, gn for prev/next “tab” and :bc to close a “tab”.

some helix annoyances

Here’s everything that’s annoyed me about Helix so far.

I like the way Helix’s :reflow works much less than how vim reflows text with gq. It doesn’t work as well with lists. (github issue)
If I’m making a Markdown list, pressing “enter” at the end of a list item won’t continue the list. There’s a partial workaround for bulleted lists but I don’t know one for numbered lists.
No persistent undo yet: in vim I could use an undofile so that I could undo changes even after quitting. Helix doesn’t have that feature yet. (github PR)
Helix doesn’t autoreload files after they change on disk, I have to run :reload-all (:ra<tab>) to manually reload them. Not a big deal.
Sometimes it crashes, maybe every week or so. I think it might be this issue.

The “markdown list” and reflowing issues come up a lot for me because I spend a lot of time editing Markdown lists, but I keep using Helix anyway so I guess they can’t be making me that mad.

switching was easier than I thought

I was worried that relearning 20 years of Vim muscle memory would be really hard.

It turned out to be easier than I expected, I started using Helix on a vacation for a little low-stakes coding project I was doing on the side and after a week or two it didn’t feel so disorienting anymore. I think it might be hard to switch back and forth between Vim and Helix, but I haven’t needed to use Vim recently so I don’t know if that’ll ever become an issue for me.

The first time I tried Helix I tried to force it to use keybindings that were more similar to Vim and that did not work for me. Just learning the “Helix way” was a lot easier.

There are still some things that throw me off: for example w in vim and w in Helix don’t have the same idea of what a “word” is (the Helix one includes the space after the word, the Vim one doesn’t).

using a terminal-based text editor

For many years I’d mostly been using a GUI version of vim/neovim, so switching to actually using an editor in the terminal was a bit of an adjustment.

I ended up deciding on:

Every project gets its own terminal window, and all of the tabs in that window (mostly) have the same working directory
I make my Helix tab the first tab in the terminal window

It works pretty well, I might actually like it better than my previous workflow.

my configuration

I appreciate that my configuration is really simple, compared to my neovim configuration which is hundreds of lines. It’s mostly just 4 keyboard shortcuts.

theme = "solarized_light"
[editor]
# Sync clipboard with system clipboard
default-yank-register = "+"

[keys.normal]
# I didn't like that Ctrl+C was the default "toggle comments" shortcut
"#" = "toggle_comments"

# I didn't feel like learning a different way
# to go to the beginning/end of a line so
# I remapped ^ and $
"^" = "goto_first_nonwhitespace"
"$" = "goto_line_end"

[keys.select]
"^" = "goto_first_nonwhitespace"
"$" = "goto_line_end"

[keys.normal.space]
# I write a lot of text so I need to constantly reflow,
# and missed vim's `gq` shortcut
l = ":reflow"

There’s a separate languages.toml configuration where I set some language preferences, like turning off autoformatting. For example, here’s my Python configuration:

[[language]]
name = "python"
formatter = { command = "black", args = ["--stdin-filename", "%{buffer_name}", "-"] }
language-servers = ["pyright"]
auto-format = false

we’ll see how it goes

Three months is not that long, and it’s possible that I’ll decide to go back to Vim at some point. For example, I wrote a post about switching to nix a while back but after maybe 8 months I switched back to Homebrew (though I’m still using NixOS to manage one little server, and I’m still satisfied with that).

2025-10-08T12:14:38-07:00 Fullscreen Open in Tab

Client ID Metadata Document Adopted by the OAuth Working Group

The IETF OAuth Working Group has adopted the Client ID Metadata Document specification!

This specification defines a mechanism through which an OAuth client can identify itself to authorization servers, without prior dynamic client registration or other existing registration.

Clients identify themselves with their own URL, and host their metadata (name, logo, redirect URL) in a JSON document at that URL. They then use that URL as the client_id to introduce themselves to an authorization server for the first time.

The mechanism of clients identifying themselves as a URL has been in use in IndieAuth for over a decade, and more recently has been adopted by BlueSky for their OAuth API. The recent surge in interest in MCP has further demonstrated the need for this to be a standardized mechanism, and was the main driver in the latest round of discussion for the document! This could replace Dynamic Client Registration in MCP, dramatically simplifying management of clients, as well as enabling servers to limit access to specific clients if they want.

The folks at Stytch put together a really nice explainer website about it too! cimd.dev

Thanks to everyone for your contributions and feedback so far! And thanks to my co-author Emilia Smith for her work on the document!

2025-10-04T07:32:57-07:00 Fullscreen Open in Tab

Meetable Release Notes - October 2025

I just released some updates for Meetable, my open source event listing website.

The major new feature is the ability to let users log in with a Discord account. A Meetable instance can be linked to a Discord server to enable any member of the server to log in to the site. You can also restrict who can log in based on Discord "roles", so you can limit who can edit events to only certain Discord members.

One of the first questions I get about Meetable is whether recurring events are supported. My answer has always been "no". In general, it's too easy for recurring events on community calendars go get stale. If an organizer forgets to cancel or just stops showing up, that isn't visible unless someone takes the time to clean up the recurrence. Instead, it's healthier to require each event be created manually. There is a "clone event" feature that makes it easy to copy all the details from a previous event to be able to quickly manually create these sorts of recurring events. In this update, I just added a feature to streamline this even further. The next recurrence is now predicted based on the past interval of the event.

For example, for a biweekly cadence, the following steps happen now:

You would create the first instance manually, say for October 1
You click "Clone Event" and change the date of the new event to October 15
Now when you click "Clone Event" on the October 15 event, it will pre-fill October 29 based on the fact that the October 15 event was created 2 weeks after the event it was cloned from

Currently this only works by counting days, so wouldn't work for things like "first Tuesday of the month" or "the 1st of the month", but I hope this saves some time in the future regardless. If "first Tuesday" or specific days of the month are an important use case for you, let me know and I can try to come up with a solution.

Minor changes/fixes below:

Added "Create New Event" to the "Add Event" dropdown menu because it wasn't obvious "Add Event" was clickable.
Meeting link no longer appears for cancelled events. (Actually the meeting link only appears for "confirmed" events.)
If you add a meeting link but don't set a timezone, a warning message appears on the event.
Added a setting to show a message when uploading a photo, you can use this to describe a photo license policy for example.
Added a "user profile" page, and if users are configured to fetch profile info from their website, a button to re-fetch the profile info will appear.

2025-08-06T17:00:00-07:00 Fullscreen Open in Tab

San Francisco Billboards - August 2025

Every time I take a Lyft from the San Francisco airport to downtown going up 101, I notice the billboards. The billboards on 101 are always such a good snapshot in time of the current peak of the Silicon Valley hype cycle. I've decided to capture photos of the billboards every time I am there, to see how this changes over time.

Here's a photo dump from the 101 billboards from August 2025. The theme is clearly AI. Apologies for the slightly blurry photos, these were taken while driving 60mph down the highway, some of them at night.

2025-06-26T00:00:00+00:00 Fullscreen Open in Tab

New zine: The Secret Rules of the Terminal

Hello! After many months of writing deep dive blog posts about the terminal, on Tuesday I released a new zine called “The Secret Rules of the Terminal”!

You can get it for $12 here: https://wizardzines.com/zines/terminal, or get an 15-pack of all my zines here.

Here’s the cover:

the table of contents

Here’s the table of contents:

why the terminal?

I’ve been using the terminal every day for 20 years but even though I’m very confident in the terminal, I’ve always had a bit of an uneasy feeling about it. Usually things work fine, but sometimes something goes wrong and it just feels like investigating it is impossible, or at least like it would open up a huge can of worms.

So I started trying to write down a list of weird problems I’ve run into in terminal and I realized that the terminal has a lot of tiny inconsistencies like:

sometimes you can use the arrow keys to move around, but sometimes pressing the arrow keys just prints ^[[D
sometimes you can use the mouse to select text, but sometimes you can’t
sometimes your commands get saved to a history when you run them, and sometimes they don’t
some shells let you use the up arrow to see the previous command, and some don’t

If you use the terminal daily for 10 or 20 years, even if you don’t understand exactly why these things happen, you’ll probably build an intuition for them.

But having an intuition for them isn’t the same as understanding why they happen. When writing this zine I actually had to do a lot of work to figure out exactly what was happening in the terminal to be able to talk about how to reason about it.

the rules aren’t written down anywhere

It turns out that the “rules” for how the terminal works (how do you edit a command you type in? how do you quit a program? how do you fix your colours?) are extremely hard to fully understand, because “the terminal” is actually made of many different pieces of software (your terminal emulator, your operating system, your shell, the core utilities like grep, and every other random terminal program you’ve installed) which are written by different people with different ideas about how things should work.

So I wanted to write something that would explain:

how the 4 pieces of the terminal (your shell, terminal emulator, programs, and TTY driver) fit together to make everything work
some of the core conventions for how you can expect things in your terminal to work
lots of tips and tricks for how to use terminal programs

this zine explains the most useful parts of terminal internals

Terminal internals are a mess. A lot of it is just the way it is because someone made a decision in the 80s and now it’s impossible to change, and honestly I don’t think learning everything about terminal internals is worth it.

But some parts are not that hard to understand and can really make your experience in the terminal better, like:

if you understand what your shell is responsible for, you can configure your shell (or use a different one!) to access your history more easily, get great tab completion, and so much more
if you understand escape codes, it’s much less scary when cating a binary to stdout messes up your terminal, you can just type reset and move on
if you understand how colour works, you can get rid of bad colour contrast in your terminal so you can actually read the text

I learned a surprising amount writing this zine

When I wrote How Git Works, I thought I knew how Git worked, and I was right. But the terminal is different. Even though I feel totally confident in the terminal and even though I’ve used it every day for 20 years, I had a lot of misunderstandings about how the terminal works and (unless you’re the author of tmux or something) I think there’s a good chance you do too.

A few things I learned that are actually useful to me:

I understand the structure of the terminal better and so I feel more confident debugging weird terminal stuff that happens to me (I was even able to suggest a small improvement to fish!). Identifying exactly which piece of software is causing a weird thing to happen in my terminal still isn’t easy but I’m a lot better at it now.
you can write a shell script to copy to your clipboard over SSH
how reset works under the hood (it does the equivalent of stty sane; sleep 1; tput reset) – basically I learned that I don’t ever need to worry about remembering stty sane or tput reset and I can just run reset instead
how to look at the invisible escape codes that a program is printing out (run unbuffer program > out; less out)
why the builtin REPLs on my Mac like sqlite3 are so annoying to use (they use libedit instead of readline)

blog posts I wrote along the way

As usual these days I wrote a bunch of blog posts about various side quests:

How to add a directory to your PATH
“rules” that terminal problems follow
why pipes sometimes get “stuck”: buffering
some terminal frustrations
ASCII control characters in my terminal on “what’s the deal with Ctrl+A, Ctrl+B, Ctrl+C, etc?”
entering text in the terminal is complicated
what’s involved in getting a “modern” terminal setup?
reasons to use your shell’s job control
standards for ANSI escape codes, which is really me trying to figure out if I think the terminfo database is serving us well today

people who helped with this zine

A long time ago I used to write zines mostly by myself but with every project I get more and more help. I met with Marie Claire LeBlanc Flanagan every weekday from September to June to work on this one.

The cover is by Vladimir Kašiković, Lesley Trites did copy editing, Simon Tatham (who wrote PuTTY) did technical review, our Operations Manager Lee did the transcription as well as a million other things, and Jesse Luehrs (who is one of the very few people I know who actually understands the terminal’s cursed inner workings) had so many incredibly helpful conversations with me about what is going on in the terminal.

get the zine

Here are some links to get the zine again:

get The Secret Rules of the Terminal
get a 15-pack of all my zines here.

As always, you can get either a PDF version to print at home or a print version shipped to your house. The only caveat is print orders will ship in August – I need to wait for orders to come in to get an idea of how many I should print before sending it to the printer.

2025-06-10T00:00:00+00:00 Fullscreen Open in Tab

Using `make` to compile C programs (for non-C-programmers)

I have never been a C programmer but every so often I need to compile a C/C++ program from source. This has been kind of a struggle for me: for a long time, my approach was basically “install the dependencies, run make, if it doesn’t work, either try to find a binary someone has compiled or give up”.

“Hope someone else has compiled it” worked pretty well when I was running Linux but since I’ve been using a Mac for the last couple of years I’ve been running into more situations where I have to actually compile programs myself.

So let’s talk about what you might have to do to compile a C program! I’ll use a couple of examples of specific C programs I’ve compiled and talk about a few things that can go wrong. Here are three programs we’ll be talking about compiling:

paperjam
sqlite
qf (a pager you can run to quickly open files from a search with rg -n THING | qf)

step 1: install a C compiler

This is pretty simple: on an Ubuntu system if I don’t already have a C compiler I’ll install one with:

sudo apt-get install build-essential

This installs gcc, g++, and make. The situation on a Mac is more confusing but it’s something like “install xcode command line tools”.

step 2: install the program’s dependencies

Unlike some newer programming languages, C doesn’t have a dependency manager. So if a program has any dependencies, you need to hunt them down yourself. Thankfully because of this, C programmers usually keep their dependencies very minimal and often the dependencies will be available in whatever package manager you’re using.

There’s almost always a section explaining how to get the dependencies in the README, for example in paperjam’s README, it says:

To compile PaperJam, you need the headers for the libqpdf and libpaper libraries (usually available as libqpdf-dev and libpaper-dev packages).

You may need a2x (found in AsciiDoc) for building manual pages.

So on a Debian-based system you can install the dependencies like this.

sudo apt install -y libqpdf-dev libpaper-dev

If a README gives a name for a package (like libqpdf-dev), I’d basically always assume that they mean “in a Debian-based Linux distro”: if you’re on a Mac brew install libqpdf-dev will not work. I still have not 100% gotten the hang of developing on a Mac yet so I don’t have many tips there yet. I guess in this case it would be brew install qpdf if you’re using Homebrew.

step 3: run `./configure` (if needed)

Some C programs come with a Makefile and some instead come with a script called ./configure. For example, if you download sqlite’s source code, it has a ./configure script in it instead of a Makefile.

My understanding of this ./configure script is:

You run it, it prints out a lot of somewhat inscrutable output, and then it either generates a Makefile or fails because you’re missing some dependency
The ./configure script is part of a system called autotools that I have never needed to learn anything about beyond “run it to generate a Makefile”.

I think there might be some options you can pass to get the ./configure script to produce a different Makefile but I have never done that.

step 4: run `make`

The next step is to run make to try to build a program. Some notes about make:

Sometimes you can run make -j8 to parallelize the build and make it go faster
It usually prints out a million compiler warnings when compiling the program. I always just ignore them. I didn’t write the software! The compiler warnings are not my problem.

compiler errors are often dependency problems

Here’s an error I got while compiling paperjam on my Mac:

/opt/homebrew/Cellar/qpdf/12.0.0/include/qpdf/InputSource.hh:85:19: error: function definition does not declare parameters
   85 |     qpdf_offset_t last_offset{0};
      |                   ^

Over the years I’ve learned it’s usually best not to overthink problems like this: if it’s talking about qpdf, there’s a good change it just means that I’ve done something wrong with how I’m including the qpdf dependency.

Now let’s talk about some ways to get the qpdf dependency included in the right way.

the world’s shortest introduction to the compiler and linker

Before we talk about how to fix dependency problems: building C programs is split into 2 steps:

Compiling the code into object files (with gcc or clang)
Linking those object files into a final binary (with ld)

It’s important to know this when building a C program because sometimes you need to pass the right flags to the compiler and linker to tell them where to find the dependencies for the program you’re compiling.

`make` uses environment variables to configure the compiler and linker

If I run make on my Mac to install paperjam, I get this error:

c++ -o paperjam paperjam.o pdf-tools.o parse.o cmds.o pdf.o -lqpdf -lpaper
ld: library 'qpdf' not found

This is not because qpdf is not installed on my system (it actually is!). But the compiler and linker don’t know how to find the qpdf library. To fix this, we need to:

pass "-I/opt/homebrew/include" to the compiler (to tell it where to find the header files)
pass "-L/opt/homebrew/lib -liconv" to the linker (to tell it where to find library files and to link in iconv)

And we can get make to pass those extra parameters to the compiler and linker using environment variables! To see how this works: inside paperjam’s Makefile you can see a bunch of environment variables, like LDLIBS here:

paperjam: $(OBJS)
	$(LD) -o $@ $^ $(LDLIBS)

Everything you put into the LDLIBS environment variable gets passed to the linker (ld) as a command line argument.

secret environment variable: `CPPFLAGS`

Makefiles sometimes define their own environment variables that they pass to the compiler/linker, but make also has a bunch of “implicit” environment variables which it will automatically pass to the C compiler and linker. There’s a full list of implicit environment variables here, but one of them is CPPFLAGS, which gets automatically passed to the C compiler.

(technically it would be more normal to use CXXFLAGS for this, but this particular Makefile hardcodes CXXFLAGS so setting CPPFLAGS was the only way I could find to set the compiler flags without editing the Makefile)

As an aside: it took me a long time to realize how closely tied to C/C++ `make` is -- I used to think that `make` was just a general build system (and of course you can use it for anything!) but it has a lot of affordances for building C/C++ programs that it doesn't have for building any other kind of program.

two ways to pass environment variables to `make`

I learned thanks to @zwol that there are actually two ways to pass environment variables to make:

CXXFLAGS=xyz make (the usual way)
make CXXFLAGS=xyz

The difference between them is that make CXXFLAGS=xyz will override the value of CXXFLAGS set in the Makefile but CXXFLAGS=xyz make won’t.

I’m not sure which way is the norm but I’m going to use the first way in this post.

how to use `CPPFLAGS` and `LDLIBS` to fix this compiler error

Now that we’ve talked about how CPPFLAGS and LDLIBS get passed to the compiler and linker, here’s the final incantation that I used to get the program to build successfully!

CPPFLAGS="-I/opt/homebrew/include" LDLIBS="-L/opt/homebrew/lib -liconv" make paperjam

This passes -I/opt/homebrew/include to the compiler and -L/opt/homebrew/lib -liconv to the linker.

Also I don’t want to pretend that I “magically” knew that those were the right arguments to pass, figuring them out involved a bunch of confused Googling that I skipped over in this post. I will say that:

the -I compiler flag tells the compiler which directory to find header files in, like /opt/homebrew/include/qpdf/QPDF.hh
the -L linker flag tells the linker which directory to find libraries in, like /opt/homebrew/lib/libqpdf.a
the -l linker flag tells the linker which libraries to link in, like -liconv means “link in the iconv library”, or -lm means “link math”

tip: how to just build 1 specific file: `make $FILENAME`

Yesterday I discovered this cool tool called qf which you can use to quickly open files from the output of ripgrep.

qf is in a big directory of various tools, but I only wanted to compile qf. So I just compiled qf, like this:

make qf

Basically if you know (or can guess) the output filename of the file you’re trying to build, you can tell make to just build that file by running make $FILENAME

tip: you don’t need a Makefile

I sometimes write 5-line C programs with no dependencies, and I just learned that if I have a file called blah.c, I can just compile it like this without creating a Makefile:

make blah

It gets automaticaly expanded to cc -o blah blah.c, which saves a bit of typing. I have no idea if I’m going to remember this (I might just keep typing gcc -o blah blah.c anyway) but it seems like a fun trick.

tip: look at how other packaging systems built the same C program

If you’re having trouble building a C program, maybe other people had problems building it too! Every Linux distribution has build files for every package that they build, so even if you can’t install packages from that distribution directly, maybe you can get tips from that Linux distro for how to build the package. Realizing this (thanks to my friend Dave) was a huge ah-ha moment for me.

For example, this line from the nix package for paperjam says:

  env.NIX_LDFLAGS = lib.optionalString stdenv.hostPlatform.isDarwin "-liconv";

This is basically saying “pass the linker flag -liconv to build this on a Mac”, so that’s a clue we could use to build it.

That same file also says env.NIX_CFLAGS_COMPILE = "-DPOINTERHOLDER_TRANSITION=1";. I’m not sure what this means, but when I try to build the paperjam package I do get an error about something called a PointerHolder, so I guess that’s somehow related to the “PointerHolder transition”.

step 5: installing the binary

Once you’ve managed to compile the program, probably you want to install it somewhere! Some Makefiles have an install target that let you install the tool on your system with make install. I’m always a bit scared of this (where is it going to put the files? what if I want to uninstall them later?), so if I’m compiling a pretty simple program I’ll often just manually copy the binary to install it instead, like this:

cp qf ~/bin

step 6: maybe make your own package!

Once I figured out how to do all of this, I realized that I could use my new make knowledge to contribute a paperjam package to Homebrew! Then I could just brew install paperjam on future systems.

The good thing is that even if the details of how all of the different packaging systems, they fundamentally all use C compilers and linkers.

it can be useful to understand a little about C even if you’re not a C programmer

I think all of this is an interesting example of how it can useful to understand some basics of how C programs work (like “they have header files”) even if you’re never planning to write a nontrivial C program if your life.

It feels good to have some ability to compile C/C++ programs myself, even though I’m still not totally confident about all of the compiler and linker flags and I still plan to never learn anything about how autotools works other than “you run ./configure to generate the Makefile”.

Two things I left out of this post:

LD_LIBRARY_PATH / DYLD_LIBRARY_PATH (which you use to tell the dynamic linker at runtime where to find dynamically linked files) because I can’t remember the last time I ran into an LD_LIBRARY_PATH issue and couldn’t find an example.
pkg-config, which I think is important but I don’t understand yet

2025-05-12T22:01:23-07:00 Fullscreen Open in Tab

Enterprise-Ready MCP

I've seen a lot of complaints about how MCP isn't ready for the enterprise.

I agree, although maybe not for the reasons you think. But don't worry, this isn't just a rant! I believe we can fix it!

The good news is the recent updates to the MCP authorization spec that separate out the role of the authorization server from the MCP server have now put the building blocks in place to make this a lot easier.

But let's back up and talk about what enterprise buyers expect when they are evaluating AI tools to bring into their companies.

Single Sign-On

At a minimum, an enterprise admin expects to be able to put an application under their single sign-on system. This enables the company to manage which users are allowed to use which applications, and prevents their users from needing to have their own passwords at the applications. The goal is to get every application managed under their single sign-on (SSO) system. Many large companies have more than 200 applications, so having them all managed through their SSO solution is a lot better than employees having to manage 200 passwords for each application!

There's a lot more than SSO too, like lifecycle management, entitlements, and logout. We're tackling these in the IPSIE working group in the OpenID Foundation. But for the purposes of this discussion, let's stick to the basics of SSO.

So what does this have to do with MCP?

An AI agent using MCP is just another application enterprises expect to be able to integrate into their single-sign-on (SSO) system. Let's take the example of Claude. When rolled out at a company, ideally every employee would log in to their company Claude account using the company identity provider (IdP). This lets the enterprise admin decide how many Claude licenses to purchase and who should be able to use it.

Connecting to External Apps

The next thing that should happen after a user logs in to Claude via SSO is they need to connect Claude to their other enterprise apps. This includes the built-in integrations in Claude like Google Calendar and Google Drive, as well as any MCP servers exposed by other apps in use within the enterprise. That could cover other SaaS apps like Zoom, Atlassian, and Slack, as well as home-grown internal apps.

Today, this process involves a somewhat cumbersome series of steps each individual employee must take. Here's an example of what the user needs to do to connect their AI agent to external apps:

First, the user logs in to Claude using SSO. This involves a redirect from Claude to the enterprise IdP where they authenticate with one or more factors, and then are redirected back.

Next, they need to connect the external app from within Claude. Claude provides a button to initiate the connection. This takes the user to that app (in this example, Google), which redirects them to the IdP to authenticate again, eventually getting redirected back to the app where an OAuth consent prompt is displayed asking the user to approve access, and finally the user is redirected back to Claude and the connection is established.

The user has to repeat these steps for every MCP server that they want to connect to Claude. There are two main problems with this:

This user experience is not great. That's a lot of clicking that the user has to do.
The enterprise admin has no visibility or control over the connection established between the two applications.

Both of these are significant problems. If you have even just 10 MCP servers rolled out in the enterprise, you're asking users to click through 10 SSO and OAuth prompts to establish the connections, and it will only get worse as MCP is more widely adopted within apps. But also, should we really be asking the user if it's okay for Claude to access their data in Google Drive? In a company context, that's not actually the user's decision. That decision should be made by the enterprise IT admin.

In "An Open Letter to Third-party Suppliers", Patrick Opet, Chief Information Security Officer of JPMorgan Chase writes:

"Modern integration patterns, however, dismantle these essential boundaries, relying heavily on modern identity protocols (e.g., OAuth) to create direct, often unchecked interactions between third-party services and firms' sensitive internal resources."

Right now, these app-to-app connections are happening behind the back of the IdP. What we need is a way to move the connections between the applications into the IdP where they can be managed by the enterprise admin.

Let's see how this works if we leverage a new (in-progress) OAuth extension called "Identity and Authorization Chaining Across Domains", which I'll refer to as "Cross-App Access" for short, enabling the enterprise IdP to sit in the middle of the OAuth exchange between the two apps.

A Brief Intro to Cross-App Access

In this example, we'll use Claude as the application that is trying to connect to Slack's (hypothetical) MCP server. We'll start with a high-level overview of the flow, and later go over the detailed protocol.

First, the user logs in to Claude through the IdP as normal. This results in Claude getting either an ID token or SAML assertion from the IdP, which tells Claude who the user is. (This works the same for SAML assertions or ID tokens, so I'll use ID tokens in the example from here out.) This is no different than what the user would do today when signing in to Claude.

Then, instead of prompting the user to connect Slack, Claude takes the ID token back to the IdP in a request that says "Claude is requesting access to this user's Slack account."

The IdP validates the ID token, sees it was issued to Claude, and verifies that the admin has allowed Claude to access Slack on behalf of the given user. Assuming everything checks out, the IdP issues a new token back to Claude.

Claude takes the intermediate token from the IdP to Slack saying "hi, I would like an access token for the Slack MCP server. The IdP gave me this token with the details of the user to issue the access token for." Slack validates the token the same way it would have validated an ID token. (Remember, Slack is already configured for SSO to the IdP for this customer as well, so it already has a way to validate these tokens.) Slack is able to issue an access token giving Claude access to this user's resources in its MCP server.

This solves the two big problems:

The exchange happens entirely without any user interaction, so the user never sees any prompts or any OAuth consent screens.
Since the IdP sits in between the exchange, this gives the enterprise admin a chance to configure the policies around which applications are allowed this direct connection.

The other nice side effect of this is since there is no user interaction required, the first time a new user logs in to Claude, all their enterprise apps will be automatically connected without them having to click any buttons!

Cross-App Access Protocol

Now let's look at what this looks like in the actual protocol. This is based on the adopted in-progress OAuth specification "Identity and Authorization Chaining Across Domains". This spec is actually a combination of two RFCs: Token Exchange (RFC 8693), and JWT Profile for Authorization Grants (RFC 7523). Both RFCs as well as the "Identity and Authorization Chaining Across Domains" spec are very flexible. While this means it is possible to apply this to many different use cases, it does mean we need to be a bit more specific in how to use it for this use case. For that purpose, I've written a profile of the Identity Chaining draft called "Identity Assertion Authorization Grant" to fill in the missing pieces for the specific use case detailed here.

Let's go through it step by step. For this example we'll use the following entities:

Claude - the "Requesting Application", which is attempting to access Slack
Slack - the "Resource Application", which has the resources being accessed through MCP
Okta - the enterprise identity provider which users at the example company can use to sign in to both apps

Single Sign-On

First, Claude gets the user to sign in using a standard OpenID Connect (or SAML) flow in order to obtain an ID token. There isn't anything unique to this spec regarding this first stage, so I will skip the details of the OpenID Connect flow and we'll start with the ID token as the input to the next step.

Token Exchange

Claude, the requesting application, then makes a Token Exchange request (RFC 8693) to the IdP's token endpoint with the following parameters:

requested_token_type: The value urn:ietf:params:oauth:token-type:id-jag indicates that an ID Assertion JWT is being requested.
audience: The Issuer URL of the Resource Application's authorization server.
subject_token: The identity assertion (e.g. the OpenID Connect ID Token or SAML assertion) for the target end-user.
subject_token_type: Either urn:ietf:params:oauth:token-type:id_token or urn:ietf:params:oauth:token-type:saml2 as defined by RFC 8693.

This request will also include the client credentials that Claude would use in a traditional OAuth token request, which could be a client secret or a JWT Bearer Assertion.

POST /oauth2/token HTTP/1.1
Host: acme.okta.com
Content-Type: application/x-www-form-urlencoded

grant_type=urn:ietf:params:oauth:grant-type:token-exchange
&requested_token_type=urn:ietf:params:oauth:token-type:id-jag
&audience=https://auth.slack.com/
&subject_token=eyJraWQiOiJzMTZ0cVNtODhwREo4VGZCXzdrSEtQ...
&subject_token_type=urn:ietf:params:oauth:token-type:id_token
&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
&client_assertion=eyJhbGciOiJSUzI1NiIsImtpZCI6IjIyIn0...

ID Assertion Validation and Policy Evaluation

At this point, the IdP evaluates the request and decides whether to issue the requested "ID Assertion JWT". The request will be evaluated based on the validity of the arguments, as well as the configured policy by the customer.

For example, the IdP validates that the ID token in this request was issued to the same client that matches the provided client authentication. It evaluates that the user still exists and is active, and that the user is assigned the Resource Application. Other policies can be evaluated at the discretion of the IdP, just like it can during a single sign-on flow.

If the IdP agrees that the requesting app should be authorized to access the given user's data in the resource app's MCP server, it will respond with a Token Exchange response to issue the token:

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

{
  "issued_token_type": "urn:ietf:params:oauth:token-type:id-jag",
  "access_token": "eyJhbGciOiJIUzI1NiIsI...",
  "token_type": "N_A",
  "expires_in": 300
}

The claims in the issued JWT are defined in "Identity Assertion Authorization Grant". The JWT is signed using the same key that the IdP signs ID tokens with. This is a critical aspect that makes this work, since again we assumed that both apps would already be configured for SSO to the IdP so would already be aware of the signing key for that purpose.

At this point, Claude is ready to request a token for the Resource App's MCP server

Access Token Request

The JWT received in the previous request can now be used as a "JWT Authorization Grant" as described by RFC 7523. To do this, Claude makes a request to the MCP authorization server's token endpoint with the following parameters:

grant_type: urn:ietf:params:oauth:grant-type:jwt-bearer
assertion: The Identity Assertion Authorization Grant JWT obtained in the previous token exchange step

For example:

POST /oauth2/token HTTP/1.1
Host: auth.slack.com
Authorization: Basic yZS1yYW5kb20tc2VjcmV0v3JOkF0XG5Qx2

grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
assertion=eyJhbGciOiJIUzI1NiIsI...

Slack's authorization server can now evaluate this request to determine whether to issue an access token. The authorization server can validate the JWT by checking the issuer (iss) in the JWT to determine which enterprise IdP the token is from, and then check the signature using the public key discovered at that server. There are other claims to be validated as well, described in Section 6.1 of the Identity Assertion Authorization Grant.

Assuming all the validations pass, Slack is ready to issue an access token to Claude in the token response:

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

{
  "token_type": "Bearer",
  "access_token": "2YotnFZFEjr1zCsicMWpAA",
  "expires_in": 86400
}

This token response is the same format that Slack's authorization server would be responding to a traditional OAuth flow. That's another key aspect of this design that makes it scalable. We don't need the resource app to use any particular access token format, since only that server is responsible for validating those tokens.

Now that Claude has the access token, it can make a request to the (hypothetical) Slack MCP server using the bearer token the same way it would have if it got the token using the traditional redirect-based OAuth flow.

Note: Eventually we'll need to define the specific behavior of when to return a refresh token in this token response. The goal is to ensure the client goes through the IdP often enough for the IdP to enforce its access policies. A refresh token could potentially undermine that if the refresh token lifetime is too long. It follows that ultimately the IdP should enforce the refresh token lifetime, so we will need to define a way for the IdP to communicate to the authorization server whether and how long to issue refresh tokens. This would enable the authorization server to make its own decision on access token lifetime, while still respecting the enterprise IdP policy.

Cross-App Access Sequence Diagram

Here's the flow again, this time as a sequence diagram.

The client initiates a login request
The user's browser is redirected to the IdP
The user logs in at the IdP
The IdP returns an OAuth authorization code to the user's browser
The user's browser delivers the authorization code to the client
The client exchanges the authorization code for an ID token at the IdP
The IdP returns an ID token to the client

At this point, the user is logged in to the MCP client. Everything up until this point has been a standard OpenID Connect flow.

The client makes a direct Token Exchange request to the IdP to exchange the ID token for a cross-domain "ID Assertion JWT"
The IdP validates the request and checks the internal policy
The IdP returns the ID-JAG to the client
The client makes a token request using the ID-JAG to the MCP authorization server
The authorization server validates the token using the signing key it also uses for its OpenID Connect flow with the IdP
The authorization server returns an access token
The client makes a request with the access token to the MCP server
The MCP server returns the response

For a more detailed step by step of the flow, see Appendix A.3 of the Identity Assertion Authorization Grant.

Next Steps

If this is something you're interested in, we'd love your help! The in-progress spec is publicly available, and we're looking for people interested in helping prototype it. If you're building an MCP server and you want to make it enterprise-ready, I'd be happy to help you build this!

You can find me at a few related events coming up:

MCP Night on May 14
MCP Developers Summit on May 23
AWS MCP Agents Hackathon on May 30
Identiverse 2025 on June 3-6

And of course you can always find me on LinkedIn or email me at aaron.parecki@okta.com.

2025-04-03T16:39:37-07:00 Fullscreen Open in Tab

Let's fix OAuth in MCP

Update: The changes described in this blog post have been incorporated into the 2025-06-18 version of the MCP spec!

Let's not overthink auth in MCP.

Yes, the MCP server is going to need its own auth server. But it's not as bad as it sounds. Let me explain.

First let's get a few pieces of terminology straight.

The confusion that's happening in the discussions I've seen so far is because the spec and diagrams show that the MCP server itself is handing authorization. That's not necessary.

In OAuth, we talk about the "authorization server" and "resource server" as distinct roles. I like to think of the authorization server as the "token factory", that's the thing that makes the access tokens. The resource server (usually an API) needs to be able to validate the tokens created by the authorization server.

It's possible to build a single server that is both a resource server and authorization server, and in fact many OAuth systems are built that way, especially large consumer services.

But nothing about the spec requires that the two roles are combined, it's also possible to run these as two totally unrelated services.

This flexibility that's been baked into OAuth for over a decade is what has led to the rapid adoption, as well the proliferation of open source and commercial products that provide an OAuth authorization server as a service.

So how does this relate to MCP?

I can annotate the flow from the Model Context Protocol spec to show the parts where the client talks to the MCP Resource Server separately from where the client talks to the MCP Authorization Server.

Here is the updated sequence diagram showing communication with each role separately.

Why is it important to call out this change?

I've seen a few conversations in various places about how requiring the MCP Server to be both an authorization server and resource server is too much of a burden. But actually, very little needs to change about the spec to enable this separation of concerns that OAuth already provides.

I've also seen various suggestions of other ways to separate the authorization server from the MCP server, like delegating to an enterprise IdP and having the MCP server validate access tokens issued by the IdP. These other options also conflate the OAuth roles in an awkward way and would result in some undesirable properties or relationships between the various parties involved.

So what needs to change in the MCP spec to enable this?

Discovery

The main thing currently forcing the MCP Server to be both the authorization server and resource server is how the client does discovery.

One design goal of MCP is to enable a client to bootstrap everything it needs based on only the server URL provided. I think this is a great design goal, and luckily is something that can be achieved even when separating the roles in the way I've described.

The MCP spec currently says that clients are expected to fetch the OAuth Server Metadata (RFC8414) file from the MCP Server base URL, resulting in a URL such as:

https://example.com/.well-known/oauth-authorization-server

This ends up meaning the MCP Resource Server must also be an Authorization Server, which leads to the complications the community has encountered so far. The good news is there is an OAuth spec we can apply here instead: Protected Resource Metadata.

Protected Resource Metadata

The Protected Resource Metadata spec is used by a Resource Server to advertise metadata about itself, including which Authorization Server can be used with it. This spec is both new and old. It was started in 2016, but was never adopted by the OAuth working group until 2023, after I had presented at an IETF meeting about the need for clients to be able to bootstrap OAuth flows given an OAuth resource server. The spec is now awaiting publication as an RFC, and should get its RFC number in a couple months. (Update: This became RFC 9728 on April 23, 2025!)

Applying this to the MCP server would result in a sequence like the following:

The MCP Client fetches the Resource Server Metadata file by appending /.well-known/oauth-protected-resource to the MCP Server base URL.
The MCP Client finds the authorization_servers property in the JSON response, and builds the Authorization Server Metadata URL by appending /.well-known/oauth-authorization-server
The MCP Client fetches the Authorization Server Metadata to find the endpoints it needs for the OAuth flow, the authorization endpoint and token endpoint
The MCP Client initiates an OAuth flow and continues as normal

Note: The Protected Resource Metadata spec also supports the Resource Server returning WWW-Authenticate with a link to the resource metadata URL if you want to avoid the requirement that MCP Servers host their metadata URLs at the .well-known endpoint, it just requires an extra HTTP request to support this.

Access Token Validation

Two things to keep in mind about how the MCP Server validates access tokens with this new separation of concerns.

If you do build the MCP Authorization Server and Resource Server as part of the same system, you don't need to do anything special to validate the access tokens the Authorization Server issues. You probably already have some sort of infrastructure in place for your normal API to validate tokens issued by your Authorization Server, so nothing changes there.

If you are using an external Authorization Server, whether that's an open source product or a commercial hosted service, that product will have its own docs for how you can validate the tokens it creates. There's a good chance it already supports the standardized JWT Access Tokens described in RFC 9068, in which case you can use off-the-shelf JWT validation middleware for common frameworks.

In either case, the critical design goal here is that the MCP Authorization Server issues access tokens that only ever need to be validated by the MCP Resource Server. This is in line with the security recommendations in Section 2.3 of RFC 9700, in particular that "access tokens SHOULD be audience-restricted to a specific resource server". In other words, it would be a bad idea for the MCP Client to be issued an access token that works with both the MCP Resource Server and the service's REST API.

Why Require the MCP Server to have an Authorization Server in the first place?

Another argument I've seen is that MCP Server developers shouldn't have to build any OAuth infrastructure at all, instead they should be able to delegate all the OAuth bits to an external service.

In principle, I agree. Getting API access and authorization right is tricky, that's why there are entire companies dedicated to solving the problem.

The architecture laid out above enables this exact separation of concerns. The difference between this architecture and some of the other proposals I've seen is that this cleanly separates the security boundaries so that there are minimal dependencies among the parties involved.

But, one thing I haven't seen mentioned in the discussions is that there actually is no requirement than an OAuth Authorization Server provide any UI itself.

An Authorization Server with no UI?

While it is desirable from a security perspective that the MCP Resource Server has a corresponding Authorization Server that issues access tokens for it, that Authorization Server doesn't actually need to have any UI or even any concept of user login or accounts. You can actually build an Authorization Server that delegates all user account management to an external service. You can see an example of this in PayPal's MCP server they recently launched.

PayPal's traditional API already supports OAuth, the authorization and token endpoints are:

https://www.paypal.com/signin/authorize
https://api-m.paypal.com/v1/oauth2/token

When PayPal built their MCP server, they launched it at https://mcp.paypal.com. If you fetch the metadata for the MCP Server, you'll find the two OAuth endpoints for the MCP Authorization Server:

https://mcp.paypal.com/authorize
https://mcp.paypal.com/token

When the MCP Client redirects the user to the authorization endpoint, the MCP server itself doesn't provide any UI. Instead, it immediately redirects the user to the real PayPal authorization endpoint which then prompts the user to log in and authorize the client.

This points to yet another benefit of architecting the MCP Authorization Server and Resource Server this way. It enables implementers to delegate the actual user management to their existing OAuth server with no changes needed to the MCP Client. The MCP Client isn't even aware that this extra redirect step was inserted in the middle. As far as the MCP Client is concerned, it has been talking to only the MCP Authorization Server. It just so happens that the MCP Authorization Server has sent the user elsewhere to actually log in.

Dynamic Client Registration

There's one more point I want to make about why having a dedicated MCP Authorization Server is helpful architecturally.

The MCP spec strongly recommends that MCP Servers (authorization servers) support Dynamic Client Registration. If MCP is successful, there will be a large number of MCP Clients talking to a large number of MCP Servers, and the user is the one deciding which combinations of clients and servers to use. This means it is not scalable to require that every MCP Client developer register their client with every MCP Server.

This is similar to the idea of using an email client with the user's chosen email server. Obviously Mozilla can't register Thunderbird with every email server out there. Instead, there needs to be a way to dynamically establish a client's identity with the OAuth server at runtime. Dynamic Client Registration is one option for how to do that.

The problem is most commercial APIs are not going to enable Dynamic Client Registration on their production servers. For example, in order to get client credentials to use the Google APIs, you need to register as a developer and then register an OAuth client after logging in. Dynamic Client Registration would allow a client to register itself without the link to the developer's account. That would mean there is no paper trail for who the client was developed by. The Dynamic Client Registration endpoint can't require authentication by definition, so is a public endpoint that can create clients, which as you can imagine opens up some potential security issues.

I do, however, think it would be reasonable to expect production services to enable Dynamic Client Registration only on the MCP's Authorization Server. This way the dynamically-registered clients wouldn't be able to use the regular REST API, but would only be able to interact with the MCP API.

Mastodon and BlueSky also have a similar problem of needing clients to show up at arbitrary authorization servers without prior coordination between the client developer and authorization server operator. I call this the "OAuth for the Open Web" problem. Mastodon used Dynamic Client Registration as their solution, and has since documented some of the issues that this creates, linked here and here.

BlueSky decided to take a different approach and instead uses an https URL as a client identifier, bypassing the need for a client registration step entirely. This has the added bonus of having at least some level of confidence of the client identity because the client identity is hosted at a domain. It would be a perfectly viable approach to use this method for MCP as well. There is a discussion on that within MCP here. This is an ongoing topic within the OAuth working group, I have a couple of drafts in progress to formalize this pattern, Client ID Metadata Document and Client ID Scheme.

Enterprise IdP Integration

Lastly, I want to touch on the idea of enabling users to log in to MCP Servers with their enterprise IdP.

When an enterprise company purchases software, they expect to be able to tie it in to their single-sign-on solution. For example, when I log in to work Slack, I enter my work email and Slack redirects me to my work IdP where I log in. This way employees don't need to have passwords with every app they use in the enterprise, they can log in to everything with the same enterprise account, and all the apps can be protected with multi-factor authentication through the IdP. This also gives the company control over which users can access which apps, as well as a way to revoke a user's access at any time.

So how does this relate to MCP?

Well, plenty of people are already trying to figure out how to let their employees safely use AI tools within the enterprise. So we need a way to let employees use their enterprise IdP to log in and authorize MCP Clients to access MCP Servers.

If you're building an MCP Server in front of an existing application that already supports enterprise Single Sign-On, then you don't need to do anything differently in the MCP Client or Server and you already have support for this. When the MCP Client redirects to the MCP Authorization Server, the MCP Authorization Server redirects to the main Authorization Server, which would then prompt the user for their company email/domain and redirect to the enterprise IdP to log in.

This brings me to yet another thing I've been seeing conflated in the discussions: user login and user authorization.

OAuth is an authorization delegation protocol. OAuth doesn't actually say anything about how users authenticate at the OAuth server, it only talks about how the user can authorize access to an application. This is actually a really great thing, because it means we can get super creative with how users authenticate.

Remember the yellow box "User logs in and authorizes" from the original sequence diagram? These are actually two totally distinct steps. The OAuth authorization server is responsible for getting the user to log in somehow, but there's no requirement that how the user logs in is with a username/password. This is where we can insert a single-sign-on flow to an enterprise IdP, or really anything you can imagine.

So think of this as two separate boxes: "user logs in", and "user authorizes". Then, we can replace the "user logs in" box with an entirely new OpenID Connect flow out to the enterprise IdP to log the user in, and after they are logged in they can authorize the client.

I'll spare you the complete expanded sequence diagram, since it looks a lot more complicated than it actually is. But I again want to stress that this is nothing new, this is already how things are commonly done today.

This all just becomes cleaner to understand when you separate the MCP Authorization Server from the MCP Resource Server.

We can push all the complexity of user login, token minting, and more onto the MCP Authorization Server, keeping the MCP Resource Server free to do the much simpler task of validating access tokens and serving resources.

Future Improvements of Enterprise IdP Integration

There are two things I want to call out about how enterprise IdP integration could be improved. Both of these are entire topics on their own, so I will only touch on the problems and link out to other places where work is happening to solve them.

There are two points of friction with the current state of enterprise login for SaaS apps.

IdP discovery
User consent

IdP Discovery

When a user logs in to a SaaS app, they need to tell the app how to find their enterprise IdP. This is commonly done by either asking the user to enter their work email, or asking the user to enter their tenant URL at the service.

Neither of these is really a great user experience. It would be a lot better if the browser already knew which enterprise IdP the user should be sent to. This is one of my goals with the work happening in FedCM. With this new browser API, the browser can mediate the login, telling the SaaS app which enterprise IdP to use automatically only needing the user to click their account icon rather than type anything in.

User Consent

Another point of friction in the enterprise happens when a user starts connecting multiple applications to each other within the company. For example, if you drop in a Google Docs link into Slack, Slack will prompt you to connect your Google account to preview the link. Multiply this by N number of applications that can preview links, and M number of applications you might drop links to, and you end up sending the user through a huge number of OAuth consent flows.

The problem is only made worse with the explosion of AI tools. Every AI tool will need access to data in every other application in the enterprise. That is a lot of OAuth consent flows for the user to manage. Plus, the user shouldn't really be the one granting consent for Slack to access the company Google Docs account anyway. That consent should ideally be managed by the enterprise IT admin.

What we actually need is a way to enable the IT admin to grant consent for apps to talk to each other company-wide, removing the need for users to be sent through an OAuth flow at all.

This is the basis of another OAuth spec I've been working on, the Identity Assertion Authorization Grant.

The same problem applies to MCP Servers, and with the separation of concerns laid out above, it becomes straightforward to add this extension to move the consent to the enterprise and streamline the user experience.

Get in touch!

If these sound like interesting problems, please get in touch! You can find me on LinkedIn or reach me via email at aaron@parecki.com.

2025-03-07T00:00:00+00:00 Fullscreen Open in Tab

Standards for ANSI escape codes

Hello! Today I want to talk about ANSI escape codes.

For a long time I was vaguely aware of ANSI escape codes (“that’s how you make text red in the terminal and stuff”) but I had no real understanding of where they were supposed to be defined or whether or not there were standards for them. I just had a kind of vague “there be dragons” feeling around them. While learning about the terminal this year, I’ve learned that:

ANSI escape codes are responsible for a lot of usability improvements in the terminal (did you know there’s a way to copy to your system clipboard when SSHed into a remote machine?? It’s an escape code called OSC 52!)
They aren’t completely standardized, and because of that they don’t always work reliably. And because they’re also invisible, it’s extremely frustrating to troubleshoot escape code issues.

So I wanted to put together a list for myself of some standards that exist around escape codes, because I want to know if they have to feel unreliable and frustrating, or if there’s a future where we could all rely on them with more confidence.

what’s an escape code?
ECMA-48
xterm control sequences
terminfo
should programs use terminfo?
is there a “single common set” of escape codes?
some reasons to use terminfo
some more documents/standards
why I think this is interesting

what’s an escape code?

Have you ever pressed the left arrow key in your terminal and seen ^[[D? That’s an escape code! It’s called an “escape code” because the first character is the “escape” character, which is usually written as ESC, \x1b, \E, \033, or ^[.

Escape codes are how your terminal emulator communicates various kinds of information (colours, mouse movement, etc) with programs running in the terminal. There are two kind of escape codes:

input codes which your terminal emulator sends for keypresses or mouse movements that don’t fit into Unicode. For example “left arrow key” is ESC[D, “Ctrl+left arrow” might be ESC[1;5D, and clicking the mouse might be something like ESC[M :3.
output codes which programs can print out to colour text, move the cursor around, clear the screen, hide the cursor, copy text to the clipboard, enable mouse reporting, set the window title, etc.

Now let’s talk about standards!

ECMA-48

The first standard I found relating to escape codes was ECMA-48, which was originally published in 1976.

ECMA-48 does two things:

Define some general formats for escape codes (like “CSI” codes, which are ESC[ + something and “OSC” codes, which are ESC] + something)
Define some specific escape codes, like how “move the cursor to the left” is ESC[D, or “turn text red” is ESC[31m. In the spec, the “cursor left” one is called CURSOR LEFT and the one for changing colours is called SELECT GRAPHIC RENDITION.

The formats are extensible, so there’s room for others to define more escape codes in the future. Lots of escape codes that are popular today aren’t defined in ECMA-48: for example it’s pretty common for terminal applications (like vim, htop, or tmux) to support using the mouse, but ECMA-48 doesn’t define escape codes for the mouse.

xterm control sequences

There are a bunch of escape codes that aren’t defined in ECMA-48, for example:

enabling mouse reporting (where did you click in your terminal?)
bracketed paste (did you paste that text or type it in?)
OSC 52 (which terminal applications can use to copy text to your system clipboard)

I believe (correct me if I’m wrong!) that these and some others came from xterm, are documented in XTerm Control Sequences, and have been widely implemented by other terminal emulators.

This list of “what xterm supports” is not a standard exactly, but xterm is extremely influential and so it seems like an important document.

terminfo

In the 80s (and to some extent today, but my understanding is that it was MUCH more dramatic in the 80s) there was a huge amount of variation in what escape codes terminals actually supported.

To deal with this, there’s a database of escape codes for various terminals called “terminfo”.

It looks like the standard for terminfo is called X/Open Curses, though you need to create an account to view that standard for some reason. It defines the database format as well as a C library interface (“curses”) for accessing the database.

For example you can run this bash snippet to see every possible escape code for “clear screen” for all of the different terminals your system knows about:

for term in $(toe -a | awk '{print $1}')
do
  echo $term
  infocmp -1 -T "$term" 2>/dev/null | grep 'clear=' | sed 's/clear=//g;s/,//g'
done

On my system (and probably every system I’ve ever used?), the terminfo database is managed by ncurses.

should programs use terminfo?

I think it’s interesting that there are two main approaches that applications take to handling ANSI escape codes:

Use the terminfo database to figure out which escape codes to use, depending on what’s in the TERM environment variable. Fish does this, for example.
Identify a “single common set” of escape codes which works in “enough” terminal emulators and just hardcode those.

Some examples of programs/libraries that take approach #2 (“don’t use terminfo”) include:

I got curious about why folks might be moving away from terminfo and I found this very interesting and extremely detailed rant about terminfo from one of the fish maintainers, which argues that:

[the terminfo authors] have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is.

I’m not going to do it justice so I’m not going to summarize it, I think it’s worth reading.

is there a “single common set” of escape codes?

I was just talking about the idea that you can use a “common set” of escape codes that will work for most people. But what is that set? Is there any agreement?

I really do not know the answer to this at all, but from doing some reading it seems like it’s some combination of:

The codes that the VT100 supported (though some aren’t relevant on modern terminals)
what’s in ECMA-48 (which I think also has some things that are no longer relevant)
What xterm supports (though I’d guess that not everything in there is actually widely supported enough)

and maybe ultimately “identify the terminal emulators you think your users are going to use most frequently and test in those”, the same way web developers do when deciding which CSS features are okay to use

I don’t think there are any resources like Can I use…? or Baseline for the terminal though. (in theory terminfo is supposed to be the “caniuse” for the terminal but it seems like it often takes 10+ years to add new terminal features when people invent them which makes it very limited)

some reasons to use terminfo

I also asked on Mastodon why people found terminfo valuable in 2025 and got a few reasons that made sense to me:

some people expect to be able to use the TERM environment variable to control how programs behave (for example with TERM=dumb), and there’s no standard for how that should work in a post-terminfo world
even though there’s less variation between terminal emulators than there was in the 80s, there’s far from zero variation: there are graphical terminals, the Linux framebuffer console, the situation you’re in when connecting to a server via its serial console, Emacs shell mode, and probably more that I’m missing
there is no one standard for what the “single common set” of escape codes is, and sometimes programs use escape codes which aren’t actually widely supported enough

terminfo & user agent detection

The way that ncurses uses the TERM environment variable to decide which escape codes to use reminds me of how webservers used to sometimes use the browser user agent to decide which version of a website to serve.

It also seems like it’s had some of the same results – the way iTerm2 reports itself as being “xterm-256color” feels similar to how Safari’s user agent is “Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15”. In both cases the terminal emulator / browser ends up changing its user agent to get around user agent detection that isn’t working well.

On the web we ended up deciding that user agent detection was not a good practice and to instead focus on standardization so we can serve the same HTML/CSS to all browsers. I don’t know if the same approach is the future in the terminal though – I think the terminal landscape today is much more fragmented than the web ever was as well as being much less well funded.

some more documents/standards

A few more documents and standards related to escape codes, in no particular order:

the Linux console_codes man page documents escape codes that Linux supports
how the VT 100 handles escape codes & control sequences
the kitty keyboard protocol
OSC 8 for links in the terminal (and notes on adoption)
A summary of ANSI standards from tmux
this terminal features reporting specification from iTerm
sixel graphics

why I think this is interesting

I sometimes see people saying that the unix terminal is “outdated”, and since I love the terminal so much I’m always curious about what incremental changes might make it feel less “outdated”.

Maybe if we had a clearer standards landscape (like we do on the web!) it would be easier for terminal emulator developers to build new features and for authors of terminal applications to more confidently adopt those features so that we can all benefit from them and have a richer experience in the terminal.

Obviously standardizing ANSI escape codes is not easy (ECMA-48 was first published almost 50 years ago and we’re still not there!). I don’t even know what all of the challenges are. But the situation with HTML/CSS/JS used to be extremely bad too and now it’s MUCH better, so maybe there’s hope.

2025-02-13T12:27:56+00:00 Fullscreen Open in Tab

How to add a directory to your PATH

I was talking to a friend about how to add a directory to your PATH today. It’s something that feels “obvious” to me since I’ve been using the terminal for a long time, but when I searched for instructions for how to do it, I actually couldn’t find something that explained all of the steps – a lot of them just said “add this to ~/.bashrc”, but what if you’re not using bash? What if your bash config is actually in a different file? And how are you supposed to figure out which directory to add anyway?

So I wanted to try to write down some more complete directions and mention some of the gotchas I’ve run into over the years.

Here’s a table of contents:

step 1: what shell are you using?
step 2: find your shell’s config file
- a note on bash’s config file
step 3: figure out which directory to add
- step 3.1: double check it’s the right directory
step 4: edit your shell config
step 5: restart your shell
problems:
notes:
- a note on source
- a note on fish_add_path

step 1: what shell are you using?

If you’re not sure what shell you’re using, here’s a way to find out. Run this:

ps -p $$ -o pid,comm=

if you’re using bash, it’ll print out 97295 bash
if you’re using zsh, it’ll print out 97295 zsh
if you’re using fish, it’ll print out an error like “In fish, please use $fish_pid” ($$ isn’t valid syntax in fish, but in any case the error message tells you that you’re using fish, which you probably already knew)

Also bash is the default on Linux and zsh is the default on Mac OS (as of 2024). I’ll only cover bash, zsh, and fish in these directions.

step 2: find your shell’s config file

in zsh, it’s probably ~/.zshrc
in bash, it might be ~/.bashrc, but it’s complicated, see the note in the next section
in fish, it’s probably ~/.config/fish/config.fish (you can run echo $__fish_config_dir if you want to be 100% sure)

a note on bash’s config file

Bash has three possible config files: ~/.bashrc, ~/.bash_profile, and ~/.profile.

If you’re not sure which one your system is set up to use, I’d recommend testing this way:

add echo hi there to your ~/.bashrc
Restart your terminal
If you see “hi there”, that means ~/.bashrc is being used! Hooray!
Otherwise remove it and try the same thing with ~/.bash_profile
You can also try ~/.profile if the first two options don’t work.

(there are a lot of elaborate flow charts out there that explain how bash decides which config file to use but IMO it’s not worth it to internalize them and just testing is the fastest way to be sure)

step 3: figure out which directory to add

Let’s say that you’re trying to install and run a program called http-server and it doesn’t work, like this:

$ npm install -g http-server
$ http-server
bash: http-server: command not found

How do you find what directory http-server is in? Honestly in general this is not that easy – often the answer is something like “it depends on how npm is configured”. A few ideas:

Often when setting up a new installer (like cargo, npm, homebrew, etc), when you first set it up it’ll print out some directions about how to update your PATH. So if you’re paying attention you can get the directions then.
Sometimes installers will automatically update your shell’s config file to update your PATH for you
Sometimes just Googling “where does npm install things?” will turn up the answer
Some tools have a subcommand that tells you where they’re configured to install things, like:
- Node/npm: npm config get prefix (then append /bin/)
- Go: go env GOPATH (then append /bin/)
- asdf: asdf info | grep ASDF_DIR (then append /bin/ and /shims/)

step 3.1: double check it’s the right directory

Once you’ve found a directory you think might be the right one, make sure it’s actually correct! For example, I found out that on my machine, http-server is in ~/.npm-global/bin. I can make sure that it’s the right directory by trying to run the program http-server in that directory like this:

$ ~/.npm-global/bin/http-server
Starting up http-server, serving ./public

It worked! Now that you know what directory you need to add to your PATH, let’s move to the next step!

step 4: edit your shell config

Now we have the 2 critical pieces of information we need:

Which directory you’re trying to add to your PATH (like ~/.npm-global/bin/)
Where your shell’s config is (like ~/.bashrc, ~/.zshrc, or ~/.config/fish/config.fish)

Now what you need to add depends on your shell:

bash instructions:

Open your shell’s config file, and add a line like this:

export PATH=$PATH:~/.npm-global/bin/

(obviously replace ~/.npm-global/bin with the actual directory you’re trying to add)

zsh instructions:

You can do the same thing as in bash, but zsh also has some slightly fancier syntax you can use if you prefer:

path=(
  $path
  ~/.npm-global/bin
)

fish instructions:

In fish, the syntax is different:

set PATH $PATH ~/.npm-global/bin

(in fish you can also use fish_add_path, some notes on that further down)

step 5: restart your shell

Now, an extremely important step: updating your shell’s config won’t take effect if you don’t restart it!

Two ways to do this:

open a new terminal (or terminal tab), and maybe close the old one so you don’t get confused
Run bash to start a new shell (or zsh if you’re using zsh, or fish if you’re using fish)

I’ve found that both of these usually work fine.

And you should be done! Try running the program you were trying to run and hopefully it works now.

If not, here are a couple of problems that you might run into:

problem 1: it ran the wrong program

If the wrong version of a program is running, you might need to add the directory to the beginning of your PATH instead of the end.

For example, on my system I have two versions of python3 installed, which I can see by running which -a:

$ which -a python3
/usr/bin/python3
/opt/homebrew/bin/python3

The one your shell will use is the first one listed.

If you want to use the Homebrew version, you need to add that directory (/opt/homebrew/bin) to the beginning of your PATH instead, by putting this in your shell’s config file (it’s /opt/homebrew/bin/:$PATH instead of the usual $PATH:/opt/homebrew/bin/)

export PATH=/opt/homebrew/bin/:$PATH

or in fish:

set PATH ~/.cargo/bin $PATH

problem 2: the program isn’t being run from your shell

All of these directions only work if you’re running the program from your shell. If you’re running the program from an IDE, from a GUI, in a cron job, or some other way, you’ll need to add the directory to your PATH in a different way, and the exact details might depend on the situation.

in a cron job

Some options:

use the full path to the program you’re running, like /home/bork/bin/my-program
put the full PATH you want as the first line of your crontab (something like PATH=/bin:/usr/bin:/usr/local/bin:….). You can get the full PATH you’re using in your shell by running echo "PATH=$PATH".

I’m honestly not sure how to handle it in an IDE/GUI because I haven’t run into that in a long time, will add directions here if someone points me in the right direction.

problem 3: duplicate `PATH` entries making it harder to debug

If you edit your path and start a new shell by running bash (or zsh, or fish), you’ll often end up with duplicate PATH entries, because the shell keeps adding new things to your PATH every time you start your shell.

Personally I don’t think I’ve run into a situation where this kind of duplication breaks anything, but the duplicates can make it harder to debug what’s going on with your PATH if you’re trying to understand its contents.

Some ways you could deal with this:

If you’re debugging your PATH, open a new terminal to do it in so you get a “fresh” state. This should avoid the duplication.
Deduplicate your PATH at the end of your shell’s config (for example in zsh apparently you can do this with typeset -U path)
Check that the directory isn’t already in your PATH when adding it (for example in fish I believe you can do this with fish_add_path --path /some/directory)

How to deduplicate your PATH is shell-specific and there isn’t always a built in way to do it so you’ll need to look up how to accomplish it in your shell.

problem 4: losing your history after updating your `PATH`

Here’s a situation that’s easy to get into in bash or zsh:

Run a command (it fails)
Update your PATH
Run bash to reload your config
Press the up arrow a couple of times to rerun the failed command (or open a new terminal)
The failed command isn’t in your history! Why not?

This happens because in bash, by default, history is not saved until you exit the shell.

Some options for fixing this:

Instead of running bash to reload your config, run source ~/.bashrc (or source ~/.zshrc in zsh). This will reload the config inside your current session.
Configure your shell to continuously save your history instead of only saving the history when the shell exits. (How to do this depends on whether you’re using bash or zsh, the history options in zsh are a bit complicated and I’m not exactly sure what the best way is)

a note on `source`

When you install cargo (Rust’s installer) for the first time, it gives you these instructions for how to set up your PATH, which don’t mention a specific directory at all.

This is usually done by running one of the following (note the leading DOT):

. "$HOME/.cargo/env"        	# For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish

The idea is that you add that line to your shell’s config, and their script automatically sets up your PATH (and potentially other things) for you.

This is pretty common (for example Homebrew suggests you eval brew shellenv), and there are two ways to approach this:

Just do what the tool suggests (like adding . "$HOME/.cargo/env" to your shell’s config)
Figure out which directories the script they’re telling you to run would add to your PATH, and then add those manually. Here’s how I’d do that:
- Run . "$HOME/.cargo/env" in my shell (or the fish version if using fish)
- Run echo "$PATH" | tr ':' '\n' | grep cargo to figure out which directories it added
- See that it says /Users/bork/.cargo/bin and shorten that to ~/.cargo/bin
- Add the directory ~/.cargo/bin to PATH (with the directions in this post)

I don’t think there’s anything wrong with doing what the tool suggests (it might be the “best way”!), but personally I usually use the second approach because I prefer knowing exactly what configuration I’m changing.

a note on `fish_add_path`

fish has a handy function called fish_add_path that you can run to add a directory to your PATH like this:

fish_add_path /some/directory

This is cool (it’s such a simple command!) but I’ve stopped using it for a couple of reasons:

Sometimes fish_add_path will update the PATH for every session in the future (with a “universal variable”) and sometimes it will update the PATH just for the current session and it’s hard for me to tell which one it will do. In theory the docs explain this but I could not understand them.
If you ever need to remove the directory from your PATH a few weeks or months later because maybe you made a mistake, it’s kind of hard to do (there are instructions in this comments of this github issue though).

that’s all

Hopefully this will help some people. Let me know (on Mastodon or Bluesky) if you there are other major gotchas that have tripped you up when adding a directory to your PATH, or if you have questions about this post!

2025-02-05T16:57:00+00:00 Fullscreen Open in Tab

Some terminal frustrations

A few weeks ago I ran a terminal survey (you can read the results here) and at the end I asked:

What’s the most frustrating thing about using the terminal for you?

1600 people answered, and I decided to spend a few days categorizing all the responses. Along the way I learned that classifying qualitative data is not easy but I gave it my best shot. I ended up building a custom tool to make it faster to categorize everything.

As with all of my surveys the methodology isn’t particularly scientific. I just posted the survey to Mastodon and Twitter, ran it for a couple of days, and got answers from whoever happened to see it and felt like responding.

Here are the top categories of frustrations!

I think it’s worth keeping in mind while reading these comments that

40% of people answering this survey have been using the terminal for 21+ years
95% of people answering the survey have been using the terminal for at least 4 years

These comments aren’t coming from total beginners.

Here are the categories of frustrations! The number in brackets is the number of people with that frustration. I’m mostly writing this up for myself because I’m trying to write a zine about the terminal and I wanted to get a sense for what people are having trouble with.

remembering syntax (115)

People talked about struggles remembering:

the syntax for CLI tools like awk, jq, sed, etc
the syntax for redirects
keyboard shortcuts for tmux, text editing, etc

One example comment:

There are just so many little “trivia” details to remember for full functionality. Even after all these years I’ll sometimes forget where it’s 2 or 1 for stderr, or forget which is which for > and >>.

switching terminals is hard (91)

People talked about struggling with switching systems (for example home/work computer or when SSHing) and running into:

OS differences in keyboard shortcuts (like Linux vs Mac)
systems which don’t have their preferred text editor (“no vim” or “only vim”)
different versions of the same command (like Mac OS grep vs GNU grep)
no tab completion
a shell they aren’t used to (“the subtle differences between zsh and bash”)

as well as differences inside the same system like pagers being not consistent with each other (git diff pagers, other pagers).

One example comment:

I got used to fish and vi mode which are not available when I ssh into servers, containers.

color (85)

Lots of problems with color, like:

programs setting colors that are unreadable with a light background color
finding a colorscheme they like (and getting it to work consistently across different apps)
color not working inside several layers of SSH/tmux/etc
not liking the defaults
not wanting color at all and struggling to turn it off

This comment felt relatable to me:

Getting my terminal theme configured in a reasonable way between the terminal emulator and fish (I did this years ago and remember it being tedious and fiddly and now feel like I’m locked into my current theme because it works and I dread touching any of that configuration ever again).

keyboard shortcuts (84)

Half of the comments on keyboard shortcuts were about how on Linux/Windows, the keyboard shortcut to copy/paste in the terminal is different from in the rest of the OS.

Some other issues with keyboard shortcuts other than copy/paste:

using Ctrl-W in a browser-based terminal and closing the window
the terminal only supports a limited set of keyboard shortcuts (no Ctrl-Shift-, no Super, no Hyper, lots of ctrl- shortcuts aren’t possible like Ctrl-,)
the OS stopping you from using a terminal keyboard shortcut (like by default Mac OS uses Ctrl+left arrow for something else)
issues using emacs in the terminal
backspace not working (2)

other copy and paste issues (75)

Aside from “the keyboard shortcut for copy and paste is different”, there were a lot of OTHER issues with copy and paste, like:

copying over SSH
how tmux and the terminal emulator both do copy/paste in different ways
dealing with many different clipboards (system clipboard, vim clipboard, the “middle click” clipboard on Linux, tmux’s clipboard, etc) and potentially synchronizing them
random spaces added when copying from the terminal
pasting multiline commands which automatically get run in a terrifying way
wanting a way to copy text without using the mouse

discoverability (55)

There were lots of comments about this, which all came down to the same basic complaint – it’s hard to discover useful tools or features! This comment kind of summed it all up:

How difficult it is to learn independently. Most of what I know is an assorted collection of stuff I’ve been told by random people over the years.

steep learning curve (44)

A lot of comments about it generally having a steep learning curve. A couple of example comments:

After 15 years of using it, I’m not much faster than using it than I was 5 or maybe even 10 years ago.

and

That I know I could make my life easier by learning more about the shortcuts and commands and configuring the terminal but I don’t spend the time because it feels overwhelming.

history (42)

Some issues with shell history:

history not being shared between terminal tabs (16)
limits that are too short (4)
history not being restored when terminal tabs are restored
losing history because the terminal crashed
not knowing how to search history

One example comment:

It wasted a lot of time until I figured it out and still annoys me that “history” on zsh has such a small buffer; I have to type “history 0” to get any useful length of history.

bad documentation (37)

People talked about:

documentation being generally opaque
lack of examples in man pages
programs which don’t have man pages

Here’s a representative comment:

Finding good examples and docs. Man pages often not enough, have to wade through stack overflow

scrollback (36)

A few issues with scrollback:

programs printing out too much data making you lose scrollback history
resizing the terminal messes up the scrollback
lack of timestamps
GUI programs that you start in the background printing stuff out that gets in the way of other programs’ outputs

One example comment:

When resizing the terminal (in particular: making it narrower) leads to broken rewrapping of the scrollback content because the commands formatted their output based on the terminal window width.

“it feels outdated” (33)

Lots of comments about how the terminal feels hampered by legacy decisions and how users often end up needing to learn implementation details that feel very esoteric. One example comment:

Most of the legacy cruft, it would be great to have a green field implementation of the CLI interface.

shell scripting (32)

Lots of complaints about POSIX shell scripting. There’s a general feeling that shell scripting is difficult but also that switching to a different less standard scripting language (fish, nushell, etc) brings its own problems.

Shell scripting. My tolerance to ditch a shell script and go to a scripting language is pretty low. It’s just too messy and powerful. Screwing up can be costly so I don’t even bother.

more issues

Some more issues that were mentioned at least 10 times:

(31) inconsistent command line arguments: is it -h or help or –help?
(24) keeping dotfiles in sync across different systems
(23) performance (e.g. “my shell takes too long to start”)
(20) window management (potentially with some combination of tmux tabs, terminal tabs, and multiple terminal windows. Where did that shell session go?)
(17) generally feeling scared/uneasy (“The debilitating fear that I’m going to do some mysterious Bad Thing with a command and I will have absolutely no idea how to fix or undo it or even really figure out what happened”)
(16) terminfo issues (“Having to learn about terminfo if/when I try a new terminal emulator and ssh elsewhere.”)
(16) lack of image support (sixel etc)
(15) SSH issues (like having to start over when you lose the SSH connection)
(15) various tmux/screen issues (for example lack of integration between tmux and the terminal emulator)
(15) typos & slow typing
(13) the terminal getting messed up for various reasons (pressing Ctrl-S, cating a binary, etc)
(12) quoting/escaping in the shell
(11) various Windows/PowerShell issues

n/a (122)

There were also 122 answers to the effect of “nothing really” or “only that I can’t do EVERYTHING in the terminal”

One example comment:

Think I’ve found work arounds for most/all frustrations

that’s all!

I’m not going to make a lot of commentary on these results, but here are a couple of categories that feel related to me:

remembering syntax & history (often the thing you need to remember is something you’ve run before!)
discoverability & the learning curve (the lack of discoverability is definitely a big part of what makes it hard to learn)
“switching systems is hard” & “it feels outdated” (tools that haven’t really changed in 30 or 40 years have many problems but they do tend to be always there no matter what system you’re on, which is very useful and makes them hard to stop using)

Trying to categorize all these results in a reasonable way really gave me an appreciation for social science researchers’ skills.

2025-01-11T09:46:01+00:00 Fullscreen Open in Tab

What's involved in getting a "modern" terminal setup?

Hello! Recently I ran a terminal survey and I asked people what frustrated them. One person commented:

There are so many pieces to having a modern terminal experience. I wish it all came out of the box.

My immediate reaction was “oh, getting a modern terminal experience isn’t that hard, you just need to….”, but the more I thought about it, the longer the “you just need to…” list got, and I kept thinking about more and more caveats.

So I thought I would write down some notes about what it means to me personally to have a “modern” terminal experience and what I think can make it hard for people to get there.

what is a “modern terminal experience”?

Here are a few things that are important to me, with which part of the system is responsible for them:

multiline support for copy and paste: if you paste 3 commands in your shell, it should not immediately run them all! That’s scary! (shell, terminal emulator)
infinite shell history: if I run a command in my shell, it should be saved forever, not deleted after 500 history entries or whatever. Also I want commands to be saved to the history immediately when I run them, not only when I exit the shell session (shell)
a useful prompt: I can’t live without having my current directory and current git branch in my prompt (shell)
24-bit colour: this is important to me because I find it MUCH easier to theme neovim with 24-bit colour support than in a terminal with only 256 colours (terminal emulator)
clipboard integration between vim and my operating system so that when I copy in Firefox, I can just press p in vim to paste (text editor, maybe the OS/terminal emulator too)
good autocomplete: for example commands like git should have command-specific autocomplete (shell)
having colours in ls (shell config)
a terminal theme I like: I spend a lot of time in my terminal, I want it to look nice and I want its theme to match my terminal editor’s theme. (terminal emulator, text editor)
automatic terminal fixing: If a programs prints out some weird escape codes that mess up my terminal, I want that to automatically get reset so that my terminal doesn’t get messed up (shell)
keybindings: I want Ctrl+left arrow to work (shell or application)
being able to use the scroll wheel in programs like less: (terminal emulator and applications)

There are a million other terminal conveniences out there and different people value different things, but those are the ones that I would be really unhappy without.

how I achieve a “modern experience”

My basic approach is:

use the fish shell. Mostly don’t configure it, except to:
- set the EDITOR environment variable to my favourite terminal editor
- alias ls to ls --color=auto
use any terminal emulator with 24-bit colour support. In the past I’ve used GNOME Terminal, Terminator, and iTerm, but I’m not picky about this. I don’t really configure it other than to choose a font.
use neovim, with a configuration that I’ve been very slowly building over the last 9 years or so (the last time I deleted my vim config and started from scratch was 9 years ago)
use the base16 framework to theme everything

A few things that affect my approach:

I don’t spend a lot of time SSHed into other machines
I’d rather use the mouse a little than come up with keyboard-based ways to do everything
I work on a lot of small projects, not one big project

some “out of the box” options for a “modern” experience

What if you want a nice experience, but don’t want to spend a lot of time on configuration? Figuring out how to configure vim in a way that I was satisfied with really did take me like ten years, which is a long time!

My best ideas for how to get a reasonable terminal experience with minimal config are:

shell: either fish or zsh with oh-my-zsh
terminal emulator: almost anything with 24-bit colour support, for example all of these are popular:
- linux: GNOME Terminal, Konsole, Terminator, xfce4-terminal
- mac: iTerm (Terminal.app doesn’t have 256-colour support)
- cross-platform: kitty, alacritty, wezterm, or ghostty
shell config:
- set the EDITOR environment variable to your favourite terminal text editor
- maybe alias ls to ls --color=auto
text editor: this is a tough one, maybe micro or helix? I haven’t used either of them seriously but they both seem like very cool projects and I think it’s amazing that you can just use all the usual GUI editor commands (Ctrl-C to copy, Ctrl-V to paste, Ctrl-A to select all) in micro and they do what you’d expect. I would probably try switching to helix except that retraining my vim muscle memory seems way too hard. Also helix doesn’t have a GUI or plugin system yet.

Personally I wouldn’t use xterm, rxvt, or Terminal.app as a terminal emulator, because I’ve found in the past that they’re missing core features (like 24-bit colour in Terminal.app’s case) that make the terminal harder to use for me.

I don’t want to pretend that getting a “modern” terminal experience is easier than it is though – I think there are two issues that make it hard. Let’s talk about them!

issue 1 with getting to a “modern” experience: the shell

bash and zsh are by far the two most popular shells, and neither of them provide a default experience that I would be happy using out of the box, for example:

you need to customize your prompt
they don’t come with git completions by default, you have to set them up
by default, bash only stores 500 (!) lines of history and (at least on Mac OS) zsh is only configured to store 2000 lines, which is still not a lot
I find bash’s tab completion very frustrating, if there’s more than one match then you can’t tab through them

And even though I love fish, the fact that it isn’t POSIX does make it hard for a lot of folks to make the switch.

Of course it’s totally possible to learn how to customize your prompt in bash or whatever, and it doesn’t even need to be that complicated (in bash I’d probably start with something like export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ ', or maybe use starship). But each of these “not complicated” things really does add up and it’s especially tough if you need to keep your config in sync across several systems.

An extremely popular solution to getting a “modern” shell experience is oh-my-zsh. It seems like a great project and I know a lot of people use it very happily, but I’ve struggled with configuration systems like that in the past – it looks like right now the base oh-my-zsh adds about 3000 lines of config, and often I find that having an extra configuration system makes it harder to debug what’s happening when things go wrong. I personally have a tendency to use the system to add a lot of extra plugins, make my system slow, get frustrated that it’s slow, and then delete it completely and write a new config from scratch.

issue 2 with getting to a “modern” experience: the text editor

In the terminal survey I ran recently, the most popular terminal text editors by far were vim, emacs, and nano.

I think the main options for terminal text editors are:

use vim or emacs and configure it to your liking, you can probably have any feature you want if you put in the work
use nano and accept that you’re going to have a pretty limited experience (for example I don’t think you can select text with the mouse and then “cut” it in nano)
use micro or helix which seem to offer a pretty good out-of-the-box experience, potentially occasionally run into issues with using a less mainstream text editor
just avoid using a terminal text editor as much as possible, maybe use VSCode, use VSCode’s terminal for all your terminal needs, and mostly never edit files in the terminal. Or I know a lot of people use code as their EDITOR in the terminal.

issue 3: individual applications

The last issue is that sometimes individual programs that I use are kind of annoying. For example on my Mac OS machine, /usr/bin/sqlite3 doesn’t support the Ctrl+Left Arrow keyboard shortcut. Fixing this to get a reasonable terminal experience in SQLite was a little complicated, I had to:

realize why this is happening (Mac OS won’t ship GNU tools, and “Ctrl-Left arrow” support comes from GNU readline)
find a workaround (install sqlite from homebrew, which does have readline support)
adjust my environment (put Homebrew’s sqlite3 in my PATH)

I find that debugging application-specific issues like this is really not easy and often it doesn’t feel “worth it” – often I’ll end up just dealing with various minor inconveniences because I don’t want to spend hours investigating them. The only reason I was even able to figure this one out at all is that I’ve been spending a huge amount of time thinking about the terminal recently.

A big part of having a “modern” experience using terminal programs is just using newer terminal programs, for example I can’t be bothered to learn a keyboard shortcut to sort the columns in top, but in htop I can just click on a column heading with my mouse to sort it. So I use htop instead! But discovering new more “modern” command line tools isn’t easy (though I made a list here), finding ones that I actually like using in practice takes time, and if you’re SSHed into another machine, they won’t always be there.

everything affects everything else

Something I find tricky about configuring my terminal to make everything “nice” is that changing one seemingly small thing about my workflow can really affect everything else. For example right now I don’t use tmux. But if I needed to use tmux again (for example because I was doing a lot of work SSHed into another machine), I’d need to think about a few things, like:

if I wanted tmux’s copy to synchronize with my system clipboard over SSH, I’d need to make sure that my terminal emulator has OSC 52 support
if I wanted to use iTerm’s tmux integration (which makes tmux tabs into iTerm tabs), I’d need to change how I configure colours – right now I set them with a shell script that I run when my shell starts, but that means the colours get lost when restoring a tmux session.

and probably more things I haven’t thought of. “Using tmux means that I have to change how I manage my colours” sounds unlikely, but that really did happen to me and I decided “well, I don’t want to change how I manage colours right now, so I guess I’m not using that feature!”.

It’s also hard to remember which features I’m relying on – for example maybe my current terminal does have OSC 52 support and because copying from tmux over SSH has always Just Worked I don’t even realize that that’s something I need, and then it mysteriously stops working when I switch terminals.

change things slowly

Personally even though I think my setup is not that complicated, it’s taken me 20 years to get to this point! Because terminal config changes are so likely to have unexpected and hard-to-understand consequences, I’ve found that if I change a lot of terminal configuration all at once it makes it much harder to understand what went wrong if there’s a problem, which can be really disorienting.

So I usually prefer to make pretty small changes, and accept that changes can might take me a REALLY long time to get used to. For example I switched from using ls to eza a year or two ago and while I like it (because eza -l prints human-readable file sizes by default) I’m still not quite sure about it. But also sometimes it’s worth it to make a big change, like I made the switch to fish (from bash) 10 years ago and I’m very happy I did.

getting a “modern” terminal is not that easy

Trying to explain how “easy” it is to configure your terminal really just made me think that it’s kind of hard and that I still sometimes get confused.

I’ve found that there’s never one perfect way to configure things in the terminal that will be compatible with every single other thing. I just need to try stuff, figure out some kind of locally stable state that works for me, and accept that if I start using a new tool it might disrupt the system and I might need to rethink things.

2024-12-12T09:28:22+00:00 Fullscreen Open in Tab

"Rules" that terminal programs follow

Recently I’ve been thinking about how everything that happens in the terminal is some combination of:

Your operating system’s job
Your shell’s job
Your terminal emulator’s job
The job of whatever program you happen to be running (like top or vim or cat)

The first three (your operating system, shell, and terminal emulator) are all kind of known quantities – if you’re using bash in GNOME Terminal on Linux, you can more or less reason about how how all of those things interact, and some of their behaviour is standardized by POSIX.

But the fourth one (“whatever program you happen to be running”) feels like it could do ANYTHING. How are you supposed to know how a program is going to behave?

This post is kind of long so here’s a quick table of contents:

programs behave surprisingly consistently
these are meant to be descriptive, not prescriptive
it’s not always obvious which “rules” are the program’s responsibility to implement
rule 1: noninteractive programs should quit when you press Ctrl-C
rule 2: TUIs should quit when you press q
rule 3: REPLs should quit when you press Ctrl-D on an empty line
rule 4: don’t use more than 16 colours
rule 5: vaguely support readline keybindings
rule 5.1: Ctrl-W should delete the last word
rule 6: disable colours when writing to a pipe
rule 7: - means stdin/stdout
these “rules” take a long time to learn

programs behave surprisingly consistently

As far as I know, there are no real standards for how programs in the terminal should behave – the closest things I know of are:

POSIX, which mostly dictates how your terminal emulator / OS / shell should work together. I think it does specify a few things about how core utilities like cp should work but AFAIK it doesn’t have anything to say about how for example htop should behave.
these command line interface guidelines

But even though there are no standards, in my experience programs in the terminal behave in a pretty consistent way. So I wanted to write down a list of “rules” that in my experience programs mostly follow.

these are meant to be descriptive, not prescriptive

My goal here isn’t to convince authors of terminal programs that they should follow any of these rules. There are lots of exceptions to these and often there’s a good reason for those exceptions.

But it’s very useful for me to know what behaviour to expect from a random new terminal program that I’m using. Instead of “uh, programs could do literally anything”, it’s “ok, here are the basic rules I expect, and then I can keep a short mental list of exceptions”.

So I’m just writing down what I’ve observed about how programs behave in my 20 years of using the terminal, why I think they behave that way, and some examples of cases where that rule is “broken”.

it’s not always obvious which “rules” are the program’s responsibility to implement

There are a bunch of common conventions that I think are pretty clearly the program’s responsibility to implement, like:

config files should go in ~/.BLAHrc or ~/.config/BLAH/FILE or /etc/BLAH/ or something
--help should print help text
programs should print “regular” output to stdout and errors to stderr

But in this post I’m going to focus on things that it’s not 100% obvious are the program’s responsibility. For example it feels to me like a “law of nature” that pressing Ctrl-D should quit a REPL, but programs often need to explicitly implement support for it – even though cat doesn’t need to implement Ctrl-D support, ipython does. (more about that in “rule 3” below)

Understanding which things are the program’s responsibility makes it much less surprising when different programs’ implementations are slightly different.

rule 1: noninteractive programs should quit when you press `Ctrl-C`

The main reason for this rule is that noninteractive programs will quit by default on Ctrl-C if they don’t set up a SIGINT signal handler, so this is kind of a “you should act like the default” rule.

Something that trips a lot of people up is that this doesn’t apply to interactive programs like python3 or bc or less. This is because in an interactive program, Ctrl-C has a different job – if the program is running an operation (like for example a search in less or some Python code in python3), then Ctrl-C will interrupt that operation but not stop the program.

As an example of how this works in an interactive program: here’s the code in prompt-toolkit (the library that iPython uses for handling input) that aborts a search when you press Ctrl-C.

rule 2: TUIs should quit when you press `q`

TUI programs (like less or htop) will usually quit when you press q.

This rule doesn’t apply to any program where pressing q to quit wouldn’t make sense, like tmux or text editors.

rule 3: REPLs should quit when you press `Ctrl-D` on an empty line

REPLs (like python3 or ed) will usually quit when you press Ctrl-D on an empty line. This rule is similar to the Ctrl-C rule – the reason for this is that by default if you’re running a program (like cat) in “cooked mode”, then the operating system will return an EOF when you press Ctrl-D on an empty line.

Most of the REPLs I use (sqlite3, python3, fish, bash, etc) don’t actually use cooked mode, but they all implement this keyboard shortcut anyway to mimic the default behaviour.

For example, here’s the code in prompt-toolkit that quits when you press Ctrl-D, and here’s the same code in readline.

I actually thought that this one was a “Law of Terminal Physics” until very recently because I’ve basically never seen it broken, but you can see that it’s just something that each individual input library has to implement in the links above.

Someone pointed out that the Erlang REPL does not quit when you press Ctrl-D, so I guess not every REPL follows this “rule”.

rule 4: don’t use more than 16 colours

Terminal programs rarely use colours other than the base 16 ANSI colours. This is because if you specify colours with a hex code, it’s very likely to clash with some users’ background colour. For example if I print out some text as #EEEEEE, it would be almost invisible on a white background, though it would look fine on a dark background.

But if you stick to the default 16 base colours, you have a much better chance that the user has configured those colours in their terminal emulator so that they work reasonably well with their background color. Another reason to stick to the default base 16 colours is that it makes less assumptions about what colours the terminal emulator supports.

The only programs I usually see breaking this “rule” are text editors, for example Helix by default will use a purple background which is not a default ANSI colour. It seems fine for Helix to break this rule since Helix isn’t a “core” program and I assume any Helix user who doesn’t like that colorscheme will just change the theme.

rule 5: vaguely support readline keybindings

Almost every program I use supports readline keybindings if it would make sense to do so. For example, here are a bunch of different programs and a link to where they define Ctrl-E to go to the end of the line:

ipython (Ctrl-E defined here)
atuin (Ctrl-E defined here)
fzf (Ctrl-E defined here)
zsh (Ctrl-E defined here)
fish (Ctrl-E defined here)
tmux’s command prompt (Ctrl-E defined here)

None of those programs actually uses readline directly, they just sort of mimic emacs/readline keybindings. They don’t always mimic them exactly: for example atuin seems to use Ctrl-A as a prefix, so Ctrl-A doesn’t go to the beginning of the line.

Also all of these programs seem to implement their own internal cut and paste buffers so you can delete a line with Ctrl-U and then paste it with Ctrl-Y.

The exceptions to this are:

some programs (like git, cat, and nc) don’t have any line editing support at all (except for backspace, Ctrl-W, and Ctrl-U)
as usual text editors are an exception, every text editor has its own approach to editing text

I wrote more about this “what keybindings does a program support?” question in entering text in the terminal is complicated.

rule 5.1: Ctrl-W should delete the last word

I’ve never seen a program (other than a text editor) where Ctrl-W doesn’t delete the last word. This is similar to the Ctrl-C rule – by default if a program is in “cooked mode”, the OS will delete the last word if you press Ctrl-W, and delete the whole line if you press Ctrl-U. So usually programs will imitate that behaviour.

I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!

rule 6: disable colours when writing to a pipe

Most programs will disable colours when writing to a pipe. For example:

rg blah will highlight all occurrences of blah in the output, but if the output is to a pipe or a file, it’ll turn off the highlighting.
ls --color=auto will use colour when writing to a terminal, but not when writing to a pipe

Both of those programs will also format their output differently when writing to the terminal: ls will organize files into columns, and ripgrep will group matches with headings.

If you want to force the program to use colour (for example because you want to look at the colour), you can use unbuffer to force the program’s output to be a tty like this:

unbuffer rg blah |  less -R

I’m sure that there are some programs that “break” this rule but I can’t think of any examples right now. Some programs have an --color flag that you can use to force colour to be on, in the example above you could also do rg --color=always | less -R.

rule 7: `-` means stdin/stdout

Usually if you pass - to a program instead of a filename, it’ll read from stdin or write to stdout (whichever is appropriate). For example, if you want to format the Python code that’s on your clipboard with black and then copy it, you could run:

pbpaste | black - | pbcopy

(pbpaste is a Mac program, you can do something similar on Linux with xclip)

My impression is that most programs implement this if it would make sense and I can’t think of any exceptions right now, but I’m sure there are many exceptions.

these “rules” take a long time to learn

These rules took me a long time for me to learn because I had to:

learn that the rule applied anywhere at all ("Ctrl-C will exit programs")
notice some exceptions (“okay, Ctrl-C will exit find but not less”)
subconsciously figure out what the pattern is ("Ctrl-C will generally quit noninteractive programs, but in interactive programs it might interrupt the current operation instead of quitting the program")
eventually maybe formulate it into an explicit rule that I know

A lot of my understanding of the terminal is honestly still in the “subconscious pattern recognition” stage. The only reason I’ve been taking the time to make things explicit at all is because I’ve been trying to explain how it works to others. Hopefully writing down these “rules” explicitly will make learning some of this stuff a little bit faster for others.

2024-11-29T08:23:31+00:00 Fullscreen Open in Tab

Why pipes sometimes get "stuck": buffering

Here’s a niche terminal problem that has bothered me for years but that I never really understood until a few weeks ago. Let’s say you’re running this command to watch for some specific output in a log file:

tail -f /some/log/file | grep thing1 | grep thing2

If log lines are being added to the file relatively slowly, the result I’d see is… nothing! It doesn’t matter if there were matches in the log file or not, there just wouldn’t be any output.

I internalized this as “uh, I guess pipes just get stuck sometimes and don’t show me the output, that’s weird”, and I’d handle it by just running grep thing1 /some/log/file | grep thing2 instead, which would work.

So as I’ve been doing a terminal deep dive over the last few months I was really excited to finally learn exactly why this happens.

why this happens: buffering

The reason why “pipes get stuck” sometimes is that it’s VERY common for programs to buffer their output before writing it to a pipe or file. So the pipe is working fine, the problem is that the program never even wrote the data to the pipe!

This is for performance reasons: writing all output immediately as soon as you can uses more system calls, so it’s more efficient to save up data until you have 8KB or so of data to write (or until the program exits) and THEN write it to the pipe.

In this example:

tail -f /some/log/file | grep thing1 | grep thing2

the problem is that grep thing1 is saving up all of its matches until it has 8KB of data to write, which might literally never happen.

programs don’t buffer when writing to a terminal

Part of why I found this so disorienting is that tail -f file | grep thing will work totally fine, but then when you add the second grep, it stops working!! The reason for this is that the way grep handles buffering depends on whether it’s writing to a terminal or not.

Here’s how grep (and many other programs) decides to buffer its output:

Check if stdout is a terminal or not using the isatty function
- If it’s a terminal, use line buffering (print every line immediately as soon as you have it)
- Otherwise, use “block buffering” – only print data if you have at least 8KB or so of data to print

So if grep is writing directly to your terminal then you’ll see the line as soon as it’s printed, but if it’s writing to a pipe, you won’t.

Of course the buffer size isn’t always 8KB for every program, it depends on the implementation. For grep the buffering is handled by libc, and libc’s buffer size is defined in the BUFSIZ variable. Here’s where that’s defined in glibc.

(as an aside: “programs do not use 8KB output buffers when writing to a terminal” isn’t, like, a law of terminal physics, a program COULD use an 8KB buffer when writing output to a terminal if it wanted, it would just be extremely weird if it did that, I can’t think of any program that behaves that way)

commands that buffer & commands that don’t

One annoying thing about this buffering behaviour is that you kind of need to remember which commands buffer their output when writing to a pipe.

Some commands that don’t buffer their output:

tail
cat
tee

I think almost everything else will buffer output, especially if it’s a command where you’re likely to be using it for batch processing. Here’s a list of some common commands that buffer their output when writing to a pipe, along with the flag that disables block buffering.

grep (--line-buffered)
sed (-u)
awk (there’s a fflush() function)
tcpdump (-l)
jq (-u)
tr (-u)
cut (can’t disable buffering)

Those are all the ones I can think of, lots of unix commands (like sort) may or may not buffer their output but it doesn’t matter because sort can’t do anything until it finishes receiving input anyway.

Also I did my best to test both the Mac OS and GNU versions of these but there are a lot of variations and I might have made some mistakes.

programming languages where the default “print” statement buffers

Also, here are a few programming language where the default print statement will buffer output when writing to a pipe, and some ways to disable buffering if you want:

C (disable with setvbuf)
Python (disable with python -u, or PYTHONUNBUFFERED=1, or sys.stdout.reconfigure(line_buffering=False), or print(x, flush=True))
Ruby (disable with STDOUT.sync = true)
Perl (disable with $| = 1)

I assume that these languages are designed this way so that the default print function will be fast when you’re doing batch processing.

Also whether output is buffered or not might depend on how you print, for example in C++ cout << "hello\n" buffers when writing to a pipe but cout << "hello" << endl will flush its output.

when you press `Ctrl-C` on a pipe, the contents of the buffer are lost

Let’s say you’re running this command as a hacky way to watch for DNS requests to example.com, and you forgot to pass -l to tcpdump:

sudo tcpdump -ni any port 53 | grep example.com

When you press Ctrl-C, what happens? In a magical perfect world, what I would want to happen is for tcpdump to flush its buffer, grep would search for example.com, and I would see all the output I missed.

But in the real world, what happens is that all the programs get killed and the output in tcpdump’s buffer is lost.

I think this problem is probably unavoidable – I spent a little time with strace to see how this works and grep receives the SIGINT before tcpdump anyway so even if tcpdump tried to flush its buffer grep would already be dead.

After a little more investigation, there is a workaround: if you find tcpdump’s PID and kill -TERM $PID, then tcpdump will flush the buffer so you can see the output. That’s kind of a pain but I tested it and it seems to work.

redirecting to a file also buffers

It’s not just pipes, this will also buffer:

sudo tcpdump -ni any port 53 > output.txt

Redirecting to a file doesn’t have the same “Ctrl-C will totally destroy the contents of the buffer” problem though – in my experience it usually behaves more like you’d want, where the contents of the buffer get written to the file before the program exits. I’m not 100% sure whether this is something you can always rely on or not.

a bunch of potential ways to avoid buffering

Okay, let’s talk solutions. Let’s say you’ve run this command:

tail -f /some/log/file | grep thing1 | grep thing2

I asked people on Mastodon how they would solve this in practice and there were 5 basic approaches. Here they are:

solution 1: run a program that finishes quickly

Historically my solution to this has been to just avoid the “command writing to pipe slowly” situation completely and instead run a program that will finish quickly like this:

cat /some/log/file | grep thing1 | grep thing2 | tail

This doesn’t do the same thing as the original command but it does mean that you get to avoid thinking about these weird buffering issues.

(you could also do grep thing1 /some/log/file but I often prefer to use an “unnecessary” cat)

solution 2: remember the “line buffer” flag to grep

You could remember that grep has a flag to avoid buffering and pass it like this:

tail -f /some/log/file | grep --line-buffered thing1 | grep thing2

solution 3: use awk

Some people said that if they’re specifically dealing with a multiple greps situation, they’ll rewrite it to use a single awk instead, like this:

tail -f /some/log/file |  awk '/thing1/ && /thing2/'

Or you would write a more complicated grep, like this:

tail -f /some/log/file |  grep -E 'thing1.*thing2'

(awk also buffers, so for this to work you’ll want awk to be the last command in the pipeline)

solution 4: use `stdbuf`

stdbuf uses LD_PRELOAD to turn off libc’s buffering, and you can use it to turn off output buffering like this:

tail -f /some/log/file | stdbuf -o0 grep thing1 | grep thing2

Like any LD_PRELOAD solution it’s a bit unreliable – it doesn’t work on static binaries, I think won’t work if the program isn’t using libc’s buffering, and doesn’t always work on Mac OS. Harry Marr has a really nice How stdbuf works post.

solution 5: use `unbuffer`

unbuffer program will force the program’s output to be a TTY, which means that it’ll behave the way it normally would on a TTY (less buffering, colour output, etc). You could use it in this example like this:

tail -f /some/log/file | unbuffer grep thing1 | grep thing2

Unlike stdbuf it will always work, though it might have unwanted side effects, for example grep thing1’s will also colour matches.

If you want to install unbuffer, it’s in the expect package.

that’s all the solutions I know about!

It’s a bit hard for me to say which one is “best”, I think personally I’m mostly likely to use unbuffer because I know it’s always going to work.

If I learn about more solutions I’ll try to add them to this post.

I’m not really sure how often this comes up

I think it’s not very common for me to have a program that slowly trickles data into a pipe like this, normally if I’m using a pipe a bunch of data gets written very quickly, processed by everything in the pipeline, and then everything exits. The only examples I can come up with right now are:

tcpdump
tail -f
watching log files in a different way like with kubectl logs
the output of a slow computation

what if there were an environment variable to disable buffering?

I think it would be cool if there were a standard environment variable to turn off buffering, like PYTHONUNBUFFERED in Python. I got this idea from a couple of blog posts by Mark Dominus in 2018. Maybe NO_BUFFER like NO_COLOR?

The design seems tricky to get right; Mark points out that NETBSD has environment variables called STDBUF, STDBUF1, etc which gives you a ton of control over buffering but I imagine most developers don’t want to implement many different environment variables to handle a relatively minor edge case.

I’m also curious about whether there are any programs that just automatically flush their output buffers after some period of time (like 1 second). It feels like it would be nice in theory but I can’t think of any program that does that so I imagine there are some downsides.

stuff I left out

Some things I didn’t talk about in this post since these posts have been getting pretty long recently and seriously does anyone REALLY want to read 3000 words about buffering?

the difference between line buffering and having totally unbuffered output
how buffering to stderr is different from buffering to stdout
this post is only about buffering that happens inside the program, your operating system’s TTY driver also does a little bit of buffering sometimes
other reasons you might need to flush your output other than “you’re writing to a pipe”

2024-11-18T09:35:42+00:00 Fullscreen Open in Tab

Importing a frontend Javascript library without a build system

I like writing Javascript without a build system and for the millionth time yesterday I ran into a problem where I needed to figure out how to import a Javascript library in my code without using a build system, and it took FOREVER to figure out how to import it because the library’s setup instructions assume that you’re using a build system.

Luckily at this point I’ve mostly learned how to navigate this situation and either successfully use the library or decide it’s too difficult and switch to a different library, so here’s the guide I wish I had to importing Javascript libraries years ago.

I’m only going to talk about using Javacript libraries on the frontend, and only about how to use them in a no-build-system setup.

In this post I’m going to talk about:

the three main types of Javascript files a library might provide (ES Modules, the “classic” global variable kind, and CommonJS)
how to figure out which types of files a Javascript library includes in its build
ways to import each type of file in your code

the three kinds of Javascript files

There are 3 basic types of Javascript files a library can provide:

the “classic” type of file that defines a global variable. This is the kind of file that you can just <script src> and it’ll Just Work. Great if you can get it but not always available
an ES module (which may or may not depend on other files, we’ll get to that)
a “CommonJS” module. This is for Node, you can’t use it in a browser at all without using a build system.

I’m not sure if there’s a better name for the “classic” type but I’m just going to call it “classic”. Also there’s a type called “AMD” but I’m not sure how relevant it is in 2024.

Now that we know the 3 types of files, let’s talk about how to figure out which of these the library actually provides!

where to find the files: the NPM build

Every Javascript library has a build which it uploads to NPM. You might be thinking (like I did originally) – Julia! The whole POINT is that we’re not using Node to build our library! Why are we talking about NPM?

But if you’re using a link from a CDN like https://cdnjs.cloudflare.com/ajax/libs/Chart.js/4.4.1/chart.umd.min.js, you’re still using the NPM build! All the files on the CDNs originally come from NPM.

Because of this, I sometimes like to npm install the library even if I’m not planning to use Node to build my library at all – I’ll just create a new temp folder, npm install there, and then delete it when I’m done. I like being able to poke around in the files in the NPM build on my filesystem, because then I can be 100% sure that I’m seeing everything that the library is making available in its build and that the CDN isn’t hiding something from me.

So let’s npm install a few libraries and try to figure out what types of Javascript files they provide in their builds!

example library 1: chart.js

First let’s look inside Chart.js, a plotting library.

$ cd /tmp/whatever
$ npm install chart.js
$ cd node_modules/chart.js/dist
$ ls *.*js
chart.cjs  chart.js  chart.umd.js  helpers.cjs  helpers.js

This library seems to have 3 basic options:

option 1: chart.cjs. The .cjs suffix tells me that this is a CommonJS file, for using in Node. This means it’s impossible to use it directly in the browser without some kind of build step.

option 2:chart.js. The .js suffix by itself doesn’t tell us what kind of file it is, but if I open it up, I see import '@kurkle/color'; which is an immediate sign that this is an ES module – the import ... syntax is ES module syntax.

option 3: chart.umd.js. “UMD” stands for “Universal Module Definition”, which I think means that you can use this file either with a basic <script src>, CommonJS, or some third thing called AMD that I don’t understand.

how to use a UMD file

When I was using Chart.js I picked Option 3. I just needed to add this to my code:

<script src="./chart.umd.js"> </script>

and then I could use the library with the global Chart environment variable. Couldn’t be easier. I just copied chart.umd.js into my Git repository so that I didn’t have to worry about using NPM or the CDNs going down or anything.

the build files aren’t always in the `dist` directory

A lot of libraries will put their build in the dist directory, but not always! The build files’ location is specified in the library’s package.json.

For example here’s an excerpt from Chart.js’s package.json.

  "jsdelivr": "./dist/chart.umd.js",
  "unpkg": "./dist/chart.umd.js",
  "main": "./dist/chart.cjs",
  "module": "./dist/chart.js",

I think this is saying that if you want to use an ES Module (module) you should use dist/chart.js, but the jsDelivr and unpkg CDNs should use ./dist/chart.umd.js. I guess main is for Node.

chart.js’s package.json also says "type": "module", which according to this documentation tells Node to treat files as ES modules by default. I think it doesn’t tell us specifically which files are ES modules and which ones aren’t but it does tell us that something in there is an ES module.

example library 2: `@atcute/oauth-browser-client`

@atcute/oauth-browser-client is a library for logging into Bluesky with OAuth in the browser.

Let’s see what kinds of Javascript files it provides in its build!

$ npm install @atcute/oauth-browser-client
$ cd node_modules/@atcute/oauth-browser-client/dist
$ ls *js
constants.js  dpop.js  environment.js  errors.js  index.js  resolvers.js

It seems like the only plausible root file in here is index.js, which looks something like this:

export { configureOAuth } from './environment.js';
export * from './errors.js';
export * from './resolvers.js';

This export syntax means it’s an ES module. That means we can use it in the browser without a build step! Let’s see how to do that.

how to use an ES module with importmaps

Using an ES module isn’t an easy as just adding a <script src="whatever.js">. Instead, if the ES module has dependencies (like @atcute/oauth-browser-client does) the steps are:

Set up an import map in your HTML
Put import statements like import { configureOAuth } from '@atcute/oauth-browser-client'; in your JS code
Include your JS code in your HTML like this: <script type="module" src="YOURSCRIPT.js"></script>

The reason we need an import map instead of just doing something like import { BrowserOAuthClient } from "./oauth-client-browser.js" is that internally the module has more import statements like import {something} from @atcute/client, and we need to tell the browser where to get the code for @atcute/client and all of its other dependencies.

Here’s what the importmap I used looks like for @atcute/oauth-browser-client:

<script type="importmap">
{
  "imports": {
    "nanoid": "./node_modules/nanoid/bin/dist/index.js",
    "nanoid/non-secure": "./node_modules/nanoid/non-secure/index.js",
    "nanoid/url-alphabet": "./node_modules/nanoid/url-alphabet/dist/index.js",
    "@atcute/oauth-browser-client": "./node_modules/@atcute/oauth-browser-client/dist/index.js",
    "@atcute/client": "./node_modules/@atcute/client/dist/index.js",
    "@atcute/client/utils/did": "./node_modules/@atcute/client/dist/utils/did.js"
  }
}
</script>

Getting these import maps to work is pretty fiddly, I feel like there must be a tool to generate them automatically but I haven’t found one yet. It’s definitely possible to write a script that automatically generates the importmaps using esbuild’s metafile but I haven’t done that and maybe there’s a better way.

I decided to set up importmaps yesterday to get github.com/jvns/bsky-oauth-example to work, so there’s some example code in that repo.

Also someone pointed me to Simon Willison’s download-esm, which will download an ES module and rewrite the imports to point to the JS files directly so that you don’t need importmaps. I haven’t tried it yet but it seems like a great idea.

problems with importmaps: too many files

I did run into some problems with using importmaps in the browser though – it needed to download dozens of Javascript files to load my site, and my webserver in development couldn’t keep up for some reason. I kept seeing files fail to load randomly and then had to reload the page and hope that they would succeed this time.

It wasn’t an issue anymore when I deployed my site to production, so I guess it was a problem with my local dev environment.

Also one slightly annoying thing about ES modules in general is that you need to be running a webserver to use them, I’m sure this is for a good reason but it’s easier when you can just open your index.html file without starting a webserver.

Because of the “too many files” thing I think actually using ES modules with importmaps in this way isn’t actually that appealing to me, but it’s good to know it’s possible.

how to use an ES module without importmaps

If the ES module doesn’t have dependencies then it’s even easier – you don’t need the importmaps! You can just:

put <script type="module" src="YOURCODE.js"></script> in your HTML. The type="module" is important.
put import {whatever} from "https://example.com/whatever.js" in YOURCODE.js

alternative: use esbuild

If you don’t want to use importmaps, you can also use a build system like esbuild. I talked about how to do that in Some notes on using esbuild, but this blog post is about ways to avoid build systems completely so I’m not going to talk about that option here. I do still like esbuild though and I think it’s a good option in this case.

what’s the browser support for importmaps?

CanIUse says that importmaps are in “Baseline 2023: newly available across major browsers” so my sense is that in 2024 that’s still maybe a little bit too new? I think I would use importmaps for some fun experimental code that I only wanted like myself and 12 people to use, but if I wanted my code to be more widely usable I’d use esbuild instead.

example library 3: `@atproto/oauth-client-browser`

Let’s look at one final example library! This is a different Bluesky auth library than @atcute/oauth-browser-client.

$ npm install @atproto/oauth-client-browser
$ cd node_modules/@atproto/oauth-client-browser/dist
$ ls *js
browser-oauth-client.js  browser-oauth-database.js  browser-runtime-implementation.js  errors.js  index.js  indexed-db-store.js  util.js

Again, it seems like only real candidate file here is index.js. But this is a different situation from the previous example library! Let’s take a look at index.js:

There’s a bunch of stuff like this in index.js:

__exportStar(require("@atproto/oauth-client"), exports);
__exportStar(require("./browser-oauth-client.js"), exports);
__exportStar(require("./errors.js"), exports);
var util_js_1 = require("./util.js");

This require() syntax is CommonJS syntax, which means that we can’t use this file in the browser at all, we need to use some kind of build step, and ESBuild won’t work either.

Also in this library’s package.json it says "type": "commonjs" which is another way to tell it’s CommonJS.

how to use a CommonJS module with esm.sh

Originally I thought it was impossible to use CommonJS modules without learning a build system, but then someone Bluesky told me about esm.sh! It’s a CDN that will translate anything into an ES Module. skypack.dev does something similar, I’m not sure what the difference is but one person mentioned that if one doesn’t work sometimes they’ll try the other one.

For @atproto/oauth-client-browser using it seems pretty simple, I just need to put this in my HTML:

<script type="module" src="script.js"> </script>

and then put this in script.js.

import { BrowserOAuthClient } from "https://esm.sh/@atproto/oauth-client-browser@0.3.0"

It seems to Just Work, which is cool! Of course this is still sort of using a build system – it’s just that esm.sh is running the build instead of me. My main concerns with this approach are:

I don’t really trust CDNs to keep working forever – usually I like to copy dependencies into my repository so that they don’t go away for some reason in the future.
I’ve heard of some issues with CDNs having security compromises which scares me.
I don’t really understand what esm.sh is doing.

esbuild can also convert CommonJS modules into ES modules

I also learned that you can also use esbuild to convert a CommonJS module into an ES module, though there are some limitations – the import { BrowserOAuthClient } from syntax doesn’t work. Here’s a github issue about that.

I think the esbuild approach is probably more appealing to me than the esm.sh approach because it’s a tool that I already have on my computer so I trust it more. I haven’t experimented with this much yet though.

summary of the three types of files

Here’s a summary of the three types of JS files you might encounter, options for how to use them, and how to identify them.

Unhelpfully a .js or .min.js file extension could be any of these 3 options, so if the file is something.js you need to do more detective work to figure out what you’re dealing with.

“classic” JS files
- How to use it:: <script src="whatever.js"></script>
- Ways to identify it:
  - The website has a big friendly banner in its setup instructions saying “Use this with a CDN!” or something
  - A .umd.js extension
  - Just try to put it in a <script src=... tag and see if it works
ES Modules
- Ways to use it:
  - If there are no dependencies, just import {whatever} from "./my-module.js" directly in your code
  - If there are dependencies, create an importmap and import {whatever} from "my-module"
    - or use download-esm to remove the need for an importmap
  - Use esbuild or any ES Module bundler
- Ways to identify it:
  - Look for an import or export statement. (not module.exports = ..., that’s CommonJS)
  - An .mjs extension
  - maybe "type": "module" in package.json (though it’s not clear to me which file exactly this refers to)
CommonJS Modules
- Ways to use it:
  - Use https://esm.sh to convert it into an ES module, like https://esm.sh/@atproto/oauth-client-browser@0.3.0
  - Use a build somehow (??)
- Ways to identify it:
  - Look for require() or module.exports = ... in the code
  - A .cjs extension
  - maybe "type": "commonjs" in package.json (though it’s not clear to me which file exactly this refers to)

it’s really nice to have ES modules standardized

The main difference between CommonJS modules and ES modules from my perspective is that ES modules are actually a standard. This makes me feel a lot more confident using them, because browsers commit to backwards compatibility for web standards forever – if I write some code using ES modules today, I can feel sure that it’ll still work the same way in 15 years.

It also makes me feel better about using tooling like esbuild because even if the esbuild project dies, because it’s implementing a standard it feels likely that there will be another similar tool in the future that I can replace it with.

the JS community has built a lot of very cool tools

A lot of the time when I talk about this stuff I get responses like “I hate javascript!!! it’s the worst!!!”. But my experience is that there are a lot of great tools for Javascript (I just learned about https://esm.sh yesterday which seems great! I love esbuild!), and that if I take the time to learn how things works I can take advantage of some of those tools and make my life a lot easier.

So the goal of this post is definitely not to complain about Javascript, it’s to understand the landscape so I can use the tooling in a way that feels good to me.

questions I still have

Here are some questions I still have, I’ll add the answers into the post if I learn the answer.

Is there a tool that automatically generates importmaps for an ES Module that I have set up locally? (apparently yes: jspm)
How can I convert a CommonJS module into an ES module on my computer, the way https://esm.sh does? (apparently esbuild can sort of do this, though named exports don’t work)
When people normally build CommonJS modules into regular JS code, what’s code is doing that? Obviously there are tools like webpack, rollup, esbuild, etc, but do those tools all implement their own JS parsers/static analysis? How many JS parsers are there out there?
Is there any way to bundle an ES module into a single file (like atcute-client.js), but so that in the browser I can still import multiple different paths from that file (like both @atcute/client/lexicons and @atcute/client)?

all the tools

Here’s a list of every tool we talked about in this post:

Simon Willison’s download-esm which will download an ES module and convert the imports to point at JS files so you don’t need an importmap
https://esm.sh/ and skypack.dev
esbuild
JSPM can generate importmaps

Writing this post has made me think that even though I usually don’t want to have a build that I run every time I update the project, I might be willing to have a build step (using download-esm or something) that I run only once when setting up the project and never run again except maybe if I’m updating my dependency versions.

that’s all!

Thanks to Marco Rogers who taught me a lot of the things in this post. I’ve probably made some mistakes in this post and I’d love to know what they are – let me know on Bluesky or Mastodon!

2024-11-09T09:24:29+00:00 Fullscreen Open in Tab

New microblog with TILs

I added a new section to this site a couple weeks ago called TIL (“today I learned”).

One kind of thing I like to post on Mastodon/Bluesky is “hey, here’s a cool thing”, like the great SQLite repl litecli, or the fact that cross compiling in Go Just Works and it’s amazing, or cryptographic right answers, or this great diff tool. Usually I don’t want to write a whole blog post about those things because I really don’t have much more to say than “hey this is useful!”

It started to bother me that I didn’t have anywhere to put those things: for example recently I wanted to use diffdiff and I just could not remember what it was called.

the solution: make a new section of this blog

So I quickly made a new folder called /til/, added some custom styling (I wanted to style the posts to look a little bit like a tweet), made a little Rake task to help me create new posts quickly (rake new_til), and set up a separate RSS Feed for it.

I think this new section of the blog might be more for myself than anything, now when I forget the link to Cryptographic Right Answers I can hopefully look it up on the TIL page. (you might think “julia, why not use bookmarks??” but I have been failing to use bookmarks for my whole life and I don’t see that changing ever, putting things in public is for whatever reason much easier for me)

So far it’s been working, often I can actually just make a quick post in 2 minutes which was the goal.

inspired by Simon Willison’s TIL blog

My page is inspired by Simon Willison’s great TIL blog, though my TIL posts are a lot shorter.

I don’t necessarily want everything to be archived

This came about because I spent a lot of time on Twitter, so I’ve been thinking about what I want to do about all of my tweets.

I keep reading the advice to “POSSE” (“post on your own site, syndicate elsewhere”), and while I find the idea appealing in principle, for me part of the appeal of social media is that it’s a little bit ephemeral. I can post polls or questions or observations or jokes and then they can just kind of fade away as they become less relevant.

I find it a lot easier to identify specific categories of things that I actually want to have on a Real Website That I Own:

blog posts here!
comics at https://wizardzines.com/comics/!
now TILs at https://jvns.ca/til/)

and then let everything else be kind of ephemeral.

I really believe in the advice to make email lists though – the first two (blog posts & comics) both have email lists and RSS feeds that people can subscribe to if they want. I might add a quick summary of any TIL posts from that week to the “blog posts from this week” mailing list.

2024-11-04T09:18:03+00:00 Fullscreen Open in Tab

My IETF 121 Agenda

Here's where you can find me at IETF 121 in Dublin!

Monday

9:30 - 11:30 • oauth
15:30 - 17:00 • alldispatch

Tuesday

9:30 - 11:30 • oauth
13:00 - 14:30 • spice
16:30 - 17:30 • scim

Thursday

9:30 - 11:30 • oauth

Get in Touch

LinkedIn GitHub

Mastodon aaron@parecki.com

My Current Drafts

2024-10-31T08:00:10+00:00 Fullscreen Open in Tab

ASCII control characters in my terminal

Hello! I’ve been thinking about the terminal a lot and yesterday I got curious about all these “control codes”, like Ctrl-A, Ctrl-C, Ctrl-W, etc. What’s the deal with all of them?

a table of ASCII control characters

Here’s a table of all 33 ASCII control characters, and what they do on my machine (on Mac OS), more or less. There are about a million caveats, but I’ll talk about what it means and all the problems with this diagram that I know about.

You can also view it as an HTML page (I just made it an image so it would show up in RSS).

different kinds of codes are mixed together

The first surprising thing about this diagram to me is that there are 33 control codes, split into (very roughly speaking) these categories:

Codes that are handled by the operating system’s terminal driver, for example when the OS sees a 3 (Ctrl-C), it’ll send a SIGINT signal to the current program
Everything else is passed through to the application as-is and the application can do whatever it wants with them. Some subcategories of those:
- Codes that correspond to a literal keypress of a key on your keyboard (Enter, Tab, Backspace). For example when you press Enter, your terminal gets sent 13.
- Codes used by readline: “the application can do whatever it wants” often means “it’ll do more or less what the readline library does, whether the application actually uses readline or not”, so I’ve labelled a bunch of the codes that readline uses
- Other codes, for example I think Ctrl-X has no standard meaning in the terminal in general but emacs uses it very heavily

There’s no real structure to which codes are in which categories, they’re all just kind of randomly scattered because this evolved organically.

(If you’re curious about readline, I wrote more about readline in entering text in the terminal is complicated, and there are a lot of cheat sheets out there)

there are only 33 control codes

Something else that I find a little surprising is that are only 33 control codes – A to Z, plus 7 more (@, [, \, ], ^, _, ?). This means that if you want to have for example Ctrl-1 as a keyboard shortcut in a terminal application, that’s not really meaningful – on my machine at least Ctrl-1 is exactly the same thing as just pressing 1, Ctrl-3 is the same as Ctrl-[, etc.

Also Ctrl+Shift+C isn’t a control code – what it does depends on your terminal emulator. On Linux Ctrl-Shift-X is often used by the terminal emulator to copy or open a new tab or paste for example, it’s not sent to the TTY at all.

Also I use Ctrl+Left Arrow all the time, but that isn’t a control code, instead it sends an ANSI escape sequence (ctrl-[[1;5D) which is a different thing which we absolutely do not have space for in this post.

This “there are only 33 codes” thing is totally different from how keyboard shortcuts work in a GUI where you can have Ctrl+KEY for any key you want.

the official ASCII names aren’t very meaningful to me

Each of these 33 control codes has a name in ASCII (for example 3 is ETX). When all of these control codes were originally defined, they weren’t being used for computers or terminals at all, they were used for the telegraph machine. Telegraph machines aren’t the same as UNIX terminals so a lot of the codes were repurposed to mean something else.

Personally I don’t find these ASCII names very useful, because 50% of the time the name in ASCII has no actual relationship to what that code does on UNIX systems today. So it feels easier to just ignore the ASCII names completely instead of trying to figure which ones still match their original meaning.

It’s hard to use Ctrl-M as a keyboard shortcut

Another thing that’s a bit weird is that Ctrl-M is literally the same as Enter, and Ctrl-I is the same as Tab, which makes it hard to use those two as keyboard shortcuts.

From some quick research, it seems like some folks do still use Ctrl-I and Ctrl-M as keyboard shortcuts (here’s an example), but to do that you need to configure your terminal emulator to treat them differently than the default.

For me the main takeaway is that if I ever write a terminal application I should avoid Ctrl-I and Ctrl-M as keyboard shortcuts in it.

how to identify what control codes get sent

While writing this I needed to do a bunch of experimenting to figure out what various key combinations did, so I wrote this Python script echo-key.py that will print them out.

There’s probably a more official way but I appreciated having a script I could customize.

caveat: on canonical vs noncanonical mode

Two of these codes (Ctrl-W and Ctrl-U) are labelled in the table as “handled by the OS”, but actually they’re not always handled by the OS, it depends on whether the terminal is in “canonical” mode or in “noncanonical mode”.

In canonical mode, programs only get input when you press Enter (and the OS is in charge of deleting characters when you press Backspace or Ctrl-W). But in noncanonical mode the program gets input immediately when you press a key, and the Ctrl-W and Ctrl-U codes are passed through to the program to handle any way it wants.

Generally in noncanonical mode the program will handle Ctrl-W and Ctrl-U similarly to how the OS does, but there are some small differences.

Some examples of programs that use canonical mode:

probably pretty much any noninteractive program, like grep or cat
git, I think

Examples of programs that use noncanonical mode:

python3, irb and other REPLs
your shell
any full screen TUI like less or vim

caveat: all of the “OS terminal driver” codes are configurable with `stty`

I said that Ctrl-C sends SIGINT but technically this is not necessarily true, if you really want to you can remap all of the codes labelled “OS terminal driver”, plus Backspace, using a tool called stty, and you can view the mappings with stty -a.

Here are the mappings on my machine right now:

$ stty -a
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
	eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
	min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
	stop = ^S; susp = ^Z; time = 0; werase = ^W;

I have personally never remapped any of these and I cannot imagine a reason I would (I think it would be a recipe for confusion and disaster for me), but I asked on Mastodon and people said the most common reasons they used stty were:

fix a broken terminal with stty sane
set stty erase ^H to change how Backspace works
set stty ixoff
some people even map SIGINT to a different key, like their DELETE key

caveat: on signals

Two signals caveats:

If the ISIG terminal mode is turned off, then the OS won’t send signals. For example vim turns off ISIG
Apparently on BSDs, there’s an extra control code (Ctrl-T) which sends SIGINFO

You can see which terminal modes a program is setting using strace like this, terminal modes are set with the ioctl system call:

$ strace -tt -o out  vim
$ grep ioctl out | grep SET

here are the modes vim sets when it starts (ISIG and ICANON are missing!):

17:43:36.670636 ioctl(0, TCSETS, {c_iflag=IXANY|IMAXBEL|IUTF8,
c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST, c_cflag=B38400|CS8|CREAD,
c_lflag=ECHOK|ECHOCTL|ECHOKE|PENDIN, ...}) = 0

and it resets the modes when it exits:

17:43:38.027284 ioctl(0, TCSETS, {c_iflag=ICRNL|IXANY|IMAXBEL|IUTF8,
c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD,
c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE|PENDIN, ...}) = 0

I think the specific combination of modes vim is using here might be called “raw mode”, man cfmakeraw talks about that.

there are a lot of conflicts

Related to “there are only 33 codes”, there are a lot of conflicts where different parts of the system want to use the same code for different things, for example by default Ctrl-S will freeze your screen, but if you turn that off then readline will use Ctrl-S to do a forward search.

Another example is that on my machine sometimes Ctrl-T will send SIGINFO and sometimes it’ll transpose 2 characters and sometimes it’ll do something completely different depending on:

whether the program has ISIG set
whether the program uses readline / imitates readline’s behaviour

caveat: on “backspace” and “other backspace”

In this diagram I’ve labelled code 127 as “backspace” and 8 as “other backspace”. Uh, what?

I think this was the single biggest topic of discussion in the replies on Mastodon – apparently there’s a LOT of history to this and I’d never heard of any of it before.

First, here’s how it works on my machine:

I press the Backspace key
The TTY gets sent the byte 127, which is called DEL in ASCII
the OS terminal driver and readline both have 127 mapped to “backspace” (so it works both in canonical mode and noncanonical mode)
The previous character gets deleted

If I press Ctrl+H, it has the same effect as Backspace if I’m using readline, but in a program without readline support (like cat for instance), it just prints out ^H.

Apparently Step 2 above is different for some folks – their Backspace key sends the byte 8 instead of 127, and so if they want Backspace to work then they need to configure the OS (using stty) to set erase = ^H.

There’s an incredible section of the Debian Policy Manual on keyboard configuration that describes how Delete and Backspace should work according to Debian policy, which seems very similar to how it works on my Mac today. My understanding (via this mastodon post) is that this policy was written in the 90s because there was a lot of confusion about what Backspace should do in the 90s and there needed to be a standard to get everything to work.

There’s a bunch more historical terminal stuff here but that’s all I’ll say for now.

there’s probably a lot more diversity in how this works

I’ve probably missed a bunch more ways that “how it works on my machine” might be different from how it works on other people’s machines, and I’ve probably made some mistakes about how it works on my machine too. But that’s all I’ve got for today.

Some more stuff I know that I’ve left out: according to stty -a Ctrl-O is “discard”, Ctrl-R is “reprint”, and Ctrl-Y is “dsusp”. I have no idea how to make those actually do anything (pressing them does not do anything obvious, and some people have told me what they used to do historically but it’s not clear to me if they have a use in 2024), and a lot of the time in practice they seem to just be passed through to the application anyway so I just labelled Ctrl-R and Ctrl-Y as readline.

not all of this is that useful to know

Also I want to say that I think the contents of this post are kind of interesting but I don’t think they’re necessarily that useful. I’ve used the terminal pretty successfully every day for the last 20 years without knowing literally any of this – I just knew what Ctrl-C, Ctrl-D, Ctrl-Z, Ctrl-R, Ctrl-L did in practice (plus maybe Ctrl-A, Ctrl-E and Ctrl-W) and did not worry about the details for the most part, and that was almost always totally fine except when I was trying to use xterm.js.

But I had fun learning about it so maybe it’ll be interesting to you too.

2024-10-27T07:47:04+00:00 Fullscreen Open in Tab

Using less memory to look up IP addresses in Mess With DNS

I’ve been having problems for the last 3 years or so where Mess With DNS periodically runs out of memory and gets OOM killed.

This hasn’t been a big priority for me: usually it just goes down for a few minutes while it restarts, and it only happens once a day at most, so I’ve just been ignoring. But last week it started actually causing a problem so I decided to look into it.

This was kind of winding road where I learned a lot so here’s a table of contents:

there’s about 100MB of memory available
the problem: OOM killing the backup script
attempt 1: use SQLite
attempt 2: use a trie
- some notes on memory profiling
attempt 3: make my array use less memory

there’s about 100MB of memory available

I run Mess With DNS on a VM without about 465MB of RAM, which according to ps aux (the RSS column) is split up something like:

100MB for PowerDNS
200MB for Mess With DNS
40MB for hallpass

That leaves about 110MB of memory free.

A while back I set GOMEMLIMIT to 250MB to try to make sure the garbage collector ran if Mess With DNS used more than 250MB of memory, and I think this helped but it didn’t solve everything.

the problem: OOM killing the backup script

A few weeks ago I started backing up Mess With DNS’s database for the first time using restic.

This has been working okay, but since Mess With DNS operates without much extra memory I think restic sometimes needed more memory than was available on the system, and so the backup script sometimes got OOM killed.

This was a problem because

backups might be corrupted sometimes
more importantly, restic takes out a lock when it runs, and so I’d have to manually do an unlock if I wanted the backups to continue working. Doing manual work like this is the #1 thing I try to avoid with all my web services (who has time for that!) so I really wanted to do something about it.

There’s probably more than one solution to this, but I decided to try to make Mess With DNS use less memory so that there was more available memory on the system, mostly because it seemed like a fun problem to try to solve.

what’s using memory: IP addresses

I’d run a memory profile of Mess With DNS a bunch of times in the past, so I knew exactly what was using most of Mess With DNS’s memory: IP addresses.

When it starts, Mess With DNS loads this database where you can look up the ASN of every IP address into memory, so that when it receives a DNS query it can take the source IP address like 74.125.16.248 and tell you that IP address belongs to GOOGLE.

This database by itself used about 117MB of memory, and a simple du told me that was too much – the original text files were only 37MB!

$ du -sh *.tsv
26M	ip2asn-v4.tsv
11M	ip2asn-v6.tsv

The way it worked originally is that I had an array of these:

type IPRange struct {
	StartIP net.IP
	EndIP   net.IP
	Num     int
	Name    string
	Country string
}

and I searched through it with a binary search to figure out if any of the ranges contained the IP I was looking for. Basically the simplest possible thing and it’s super fast, my machine can do about 9 million lookups per second.

attempt 1: use SQLite

I’ve been using SQLite recently, so my first thought was – maybe I can store all of this data on disk in an SQLite database, give the tables an index, and that’ll use less memory.

So I:

wrote a quick Python script using sqlite-utils to import the TSV files into an SQLite database
adjusted my code to select from the database instead

This did solve the initial memory goal (after a GC it now hardly used any memory at all because the table was on disk!), though I’m not sure how much GC churn this solution would cause if we needed to do a lot of queries at once. I did a quick memory profile and it seemed to allocate about 1KB of memory per lookup.

Let’s talk about the issues I ran into with using SQLite though.

problem: how to store IPv6 addresses

SQLite doesn’t have support for big integers and IPv6 addresses are 128 bits, so I decided to store them as text. I think BLOB might have been better, I originally thought BLOBs couldn’t be compared but the sqlite docs say they can.

I ended up with this schema:

CREATE TABLE ipv4_ranges (
   start_ip INTEGER NOT NULL,
   end_ip INTEGER NOT NULL,
   asn INTEGER NOT NULL,
   country TEXT NOT NULL,
   name TEXT NOT NULL
);
CREATE TABLE ipv6_ranges (
   start_ip TEXT NOT NULL,
   end_ip TEXT NOT NULL,
   asn INTEGER,
   country TEXT,
   name TEXT
);
CREATE INDEX idx_ipv4_ranges_start_ip ON ipv4_ranges (start_ip);
CREATE INDEX idx_ipv6_ranges_start_ip ON ipv6_ranges (start_ip);
CREATE INDEX idx_ipv4_ranges_end_ip ON ipv4_ranges (end_ip);
CREATE INDEX idx_ipv6_ranges_end_ip ON ipv6_ranges (end_ip);

Also I learned that Python has an ipaddress module, so I could use ipaddress.ip_address(s).exploded to make sure that the IPv6 addresses were expanded so that a string comparison would compare them properly.

problem: it’s 500x slower

I ran a quick microbenchmark, something like this. It printed out that it could look up 17,000 IPv6 addresses per second, and similarly for IPv4 addresses.

This was pretty discouraging – being able to look up 17k addresses per section is kind of fine (Mess With DNS does not get a lot of traffic), but I compared it to the original binary search code and the original code could do 9 million per second.

	ips := []net.IP{}
	count := 20000
	for i := 0; i < count; i++ {
		// create a random IPv6 address
		bytes := randomBytes()
		ip := net.IP(bytes[:])
		ips = append(ips, ip)
	}
	now := time.Now()
	success := 0
	for _, ip := range ips {
		_, err := ranges.FindASN(ip)
		if err == nil {
			success++
		}
	}
	fmt.Println(success)
	elapsed := time.Since(now)
	fmt.Println("number per second", float64(count)/elapsed.Seconds())

time for EXPLAIN QUERY PLAN

I’d never really done an EXPLAIN in sqlite, so I thought it would be a fun opportunity to see what the query plan was doing.

sqlite> explain query plan select * from ipv6_ranges where '2607:f8b0:4006:0824:0000:0000:0000:200e' BETWEEN start_ip and end_ip;
QUERY PLAN
`--SEARCH ipv6_ranges USING INDEX idx_ipv6_ranges_end_ip (end_ip>?)

It looks like it’s just using the end_ip index and not the start_ip index, so maybe it makes sense that it’s slower than the binary search.

I tried to figure out if there was a way to make SQLite use both indexes, but I couldn’t find one and maybe it knows best anyway.

At this point I gave up on the SQLite solution, I didn’t love that it was slower and also it’s a lot more complex than just doing a binary search. I felt like I’d rather keep something much more similar to the binary search.

A few things I tried with SQLite that did not cause it to use both indexes:

using a compound index instead of two separate indexes
running ANALYZE
using INTERSECT to intersect the results of start_ip < ? and ? < end_ip. This did make it use both indexes, but it also seemed to make the query literally 1000x slower, probably because it needed to create the results of both subqueries in memory and intersect them.

attempt 2: use a trie

My next idea was to use a trie, because I had some vague idea that maybe a trie would use less memory, and I found this library called ipaddress-go that lets you look up IP addresses using a trie.

I tried using it here’s the code, but I think I was doing something wildly wrong because, compared to my naive array + binary search:

it used WAY more memory (800MB to store just the IPv4 addresses)
it was a lot slower to do the lookups (it could do only 100K/second instead of 9 million/second)

I’m not really sure what went wrong here but I gave up on this approach and decided to just try to make my array use less memory and stick to a simple binary search.

some notes on memory profiling

One thing I learned about memory profiling is that you can use runtime package to see how much memory is currently allocated in the program. That’s how I got all the memory numbers in this post. Here’s the code:

func memusage() {
	runtime.GC()
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("Alloc = %v MiB\n", m.Alloc/1024/1024)
	// write mem.prof
	f, err := os.Create("mem.prof")
	if err != nil {
		log.Fatal(err)
	}
	pprof.WriteHeapProfile(f)
	f.Close()
}

Also I learned that if you use pprof to analyze a heap profile there are two ways to analyze it: you can pass either --alloc-space or --inuse-space to go tool pprof. I don’t know how I didn’t realize this before but alloc-space will tell you about everything that was allocated, and inuse-space will just include memory that’s currently in use.

Anyway I ran go tool pprof -pdf --inuse_space mem.prof > mem.pdf a lot. Also every time I use pprof I find myself referring to my own intro to pprof, it’s probably the blog post I wrote that I use the most often. I should add --alloc-space and --inuse-space to it.

attempt 3: make my array use less memory

I was storing my ip2asn entries like this:

type IPRange struct {
	StartIP net.IP
	EndIP   net.IP
	Num     int
	Name    string
	Country string
}

I had 3 ideas for ways to improve this:

There was a lot of repetition of Name and the Country, because a lot of IP ranges belong to the same ASN
net.IP is an []byte under the hood, which felt like it involved an unnecessary pointer, was there a way to inline it into the struct?
Maybe I didn’t need both the start IP and the end IP, often the ranges were consecutive so maybe I could rearrange things so that I only had the start IP

idea 3.1: deduplicate the Name and Country

I figured I could store the ASN info in an array, and then just store the index into the array in my IPRange struct. Here are the structs so you can see what I mean:

type IPRange struct {
	StartIP netip.Addr
	EndIP   netip.Addr
	ASN     uint32
	Idx     uint32
}

type ASNInfo struct {
	Country string
	Name    string
}

type ASNPool struct {
	asns   []ASNInfo
	lookup map[ASNInfo]uint32
}

This worked! It brought memory usage from 117MB to 65MB – a 50MB savings. I felt good about this.

Here’s all of the code for that part.

how big are ASNs?

As an aside – I’m storing the ASN in a uint32, is that right? I looked in the ip2asn file and the biggest one seems to be 401307, though there are a few lines that say 4294901931 which is much bigger, but also are just inside the range of a uint32. So I can definitely use a uint32.

59.101.179.0	59.101.179.255	4294901931	Unknown	AS4294901931

idea 3.2: use `netip.Addr` instead of `net.IP`

It turns out that I’m not the only one who felt that net.IP was using an unnecessary amount of memory – in 2021 the folks at Tailscale released a new IP address library for Go which solves this and many other issues. They wrote a great blog post about it.

I discovered (to my delight) that not only does this new IP address library exist and do exactly what I want, it’s also now in the Go standard library as netip.Addr. Switching to netip.Addr was very easy and saved another 20MB of memory, bringing us to 46MB.

I didn’t try my third idea (remove the end IP from the struct) because I’d already been programming for long enough on a Saturday morning and I was happy with my progress.

It’s always such a great feeling when I think “hey, I don’t like this, there must be a better way” and then immediately discover that someone has already made the exact thing I want, thought about it a lot more than me, and implemented it much better than I would have.

all of this was messier in real life

Even though I tried to explain this in a simple linear way “I tried X, then I tried Y, then I tried Z”, that’s kind of a lie – I always try to take my actual debugging process (total chaos) and make it seem more linear and understandable because the reality is just too annoying to write down. It’s more like:

try sqlite
try a trie
second guess everything that I concluded about sqlite, go back and look at the results again
wait what about indexes
very very belatedly realize that I can use runtime to check how much memory everything is using, start doing that
look at the trie again, maybe I misunderstood everything
give up and go back to binary search
look at all of the numbers for tries/sqlite again to make sure I didn’t misunderstand

A note on using 512MB of memory

Someone asked why I don’t just give the VM more memory. I could very easily afford to pay for a VM with 1GB of memory, but I feel like 512MB really should be enough (and really that 256MB should be enough!) so I’d rather stay inside that constraint. It’s kind of a fun puzzle.

a few ideas from the replies

Folks had a lot of good ideas I hadn’t thought of. Recording them as inspiration if I feel like having another Fun Performance Day at some point.

Try Go’s unique package for the ASNPool. Someone tried this and it uses more memory, probably because Go’s pointers are 64 bits
Try compiling with GOARCH=386 to use 32-bit pointers to sace space (maybe in combination with using unique!)
It should be possible to store all of the IPv6 addresses in just 64 bits, because only the first 64 bits of the address are public
Interpolation search might be faster than binary search since IP addresses are numeric
Try the MaxMind db format with mmdbwriter or mmdbctl
Tailscale’s art routing table package

the result: saved 70MB of memory!

I deployed the new version and now Mess With DNS is using less memory! Hooray!

A few other notes:

lookups are a little slower – in my microbenchmark they went from 9 million lookups/second to 6 million, maybe because I added a little indirection. Using less memory and a little more CPU seemed like a good tradeoff though.
it’s still using more memory than the raw text files do (46MB vs 37MB), I guess pointers take up space and that’s okay.

I’m honestly not sure if this will solve all my memory problems, probably not! But I had fun, I learned a few things about SQLite, I still don’t know what to think about tries, and it made me love binary search even more than I already did.

2024-10-07T09:19:57+00:00 Fullscreen Open in Tab

Some notes on upgrading Hugo

Warning: this is a post about very boring yakshaving, probably only of interest to people who are trying to upgrade Hugo from a very old version to a new version. But what are blogs for if not documenting one’s very boring yakshaves from time to time?

So yesterday I decided to try to upgrade Hugo. There’s no real reason to do this – I’ve been using Hugo version 0.40 to generate this blog since 2018, it works fine, and I don’t have any problems with it. But I thought – maybe it won’t be as hard as I think, and I kind of like a tedious computer task sometimes!

I thought I’d document what I learned along the way in case it’s useful to anyone else doing this very specific migration. I upgraded from Hugo v0.40 (from 2018) to v0.135 (from 2024).

Here are most of the changes I had to make:

change 1: `template "theme/partials/thing.html` is now `partial thing.html`

I had to replace a bunch of instances of {{ template "theme/partials/header.html" . }} with {{ partial "header.html" . }}.

This happened in v0.42:

We have now virtualized the filesystems for project and theme files. This makes everything simpler, faster and more powerful. But it also means that template lookups on the form {{ template “theme/partials/pagination.html” . }} will not work anymore. That syntax has never been documented, so it’s not expected to be in wide use.

change 2: `.Data.Pages` is now `site.RegularPages`

This seems to be discussed in the release notes for 0.57.2

I just needed to replace .Data.Pages with site.RegularPages in the template on the homepage as well as in my RSS feed template.

change 3: `.Next` and `.Prev` got flipped

I had this comment in the part of my theme where I link to the next/previous blog post:

“next” and “previous” in hugo apparently mean the opposite of what I’d think they’d mean intuitively. I’d expect “next” to mean “in the future” and “previous” to mean “in the past” but it’s the opposite

It looks they changed this in ad705aac064 so that “next” actually is in the future and “prev” actually is in the past. I definitely find the new behaviour more intuitive.

downloading the Hugo changelogs with a script

Figuring out why/when all of these changes happened was a little difficult. I ended up hacking together a bash script to download all of the changelogs from github as text files, which I could then grep to try to figure out what happened. It turns out it’s pretty easy to get all of the changelogs from the GitHub API.

So far everything was not so bad – there was also a change around taxonomies that’s I can’t quite explain, but it was all pretty manageable, but then we got to the really tough one: the markdown renderer.

change 4: the markdown renderer (blackfriday -> goldmark)

The blackfriday markdown renderer (which was previously the default) was removed in v0.100.0. This seems pretty reasonable:

It has been deprecated for a long time, its v1 version is not maintained anymore, and there are many known issues. Goldmark should be a mature replacement by now.

Fixing all my Markdown changes was a huge pain – I ended up having to update 80 different Markdown files (out of 700) so that they would render properly, and I’m not totally sure

why bother switching renderers?

The obvious question here is – why bother even trying to upgrade Hugo at all if I have to switch Markdown renderers? My old site was running totally fine and I think it wasn’t necessarily a good use of time, but the one reason I think it might be useful in the future is that the new renderer (goldmark) uses the CommonMark markdown standard, which I’m hoping will be somewhat more futureproof. So maybe I won’t have to go through this again? We’ll see.

Also it turned out that the new Goldmark renderer does fix some problems I had (but didn’t know that I had) with smart quotes and how lists/blockquotes interact.

finding all the Markdown problems: the process

The hard part of this Markdown change was even figuring out what changed. Almost all of the problems (including #2 and #3 above) just silently broke the site, they didn’t cause any errors or anything. So I had to diff the HTML to hunt them down.

Here’s what I ended up doing:

Generate the site with the old version, put it in public_old
Generate the new version, put it in public
Diff every single HTML file in public/ and public_old with this diff.sh script and put the results in a diffs/ folder
Run variations on find diffs -type f | xargs cat | grep -C 5 '(31m|32m)' | less -r over and over again to look at every single change until I found something that seemed wrong
Update the Markdown to fix the problem
Repeat until everything seemed okay

(the grep 31m|32m thing is searching for red/green text in the diff)

This was very time consuming but it was a little bit fun for some reason so I kept doing it until it seemed like nothing too horrible was left.

the new markdown rules

Here’s a list of every type of Markdown change I had to make. It’s very possible these are all extremely specific to me but it took me a long time to figure them all out so maybe this will be helpful to one other person who finds this in the future.

4.1: mixing HTML and markdown

This doesn’t work anymore (it doesn’t expand the link):

<small>
[a link](https://example.com)
</small>

I need to do this instead:

<small>

[a link](https://example.com)

</small>

This works too:

<small> [a link](https://example.com) </small>

4.2: `<<` is changed into «

I didn’t want this so I needed to configure:

markup:
  goldmark:
    extensions:
      typographer:
        leftAngleQuote: '&lt;&lt;'
        rightAngleQuote: '&gt;&gt;'

4.3: nested lists sometimes need 4 space indents

This doesn’t render as a nested list anymore if I only indent by 2 spaces, I need to put 4 spaces.

1. a
  * b
  * c
2. b

The problem is that the amount of indent needed depends on the size of the list markers. Here’s a reference in CommonMark for this.

4.4: blockquotes inside lists work better

Previously the > quote here didn’t render as a blockquote, and with the new renderer it does.

* something
> quote
* something else

I found a bunch of Markdown that had been kind of broken (which I hadn’t noticed) that works better with the new renderer, and this is an example of that.

Lists inside blockquotes also seem to work better.

4.5: headings inside lists

Previously this didn’t render as a heading, but now it does. So I needed to replace the # with &num;.

* # passengers: 20

4.6: `+` or `1)` at the beginning of the line makes it a list

I had something which looked like this:

`1 / (1
+ exp(-1)) = 0.73`

With Blackfriday it rendered like this:

<p><code>1 / (1
+ exp(-1)) = 0.73</code></p>

and with Goldmark it rendered like this:

<p>`1 / (1</p>
<ul>
<li>exp(-1)) = 0.73`</li>
</ul>

Same thing if there was an accidental 1) at the beginning of a line, like in this Markdown snippet

I set up a small Hadoop cluster (1 master, 2 workers, replication set to 
1) on

To fix this I just had to rewrap the line so that the + wasn’t the first character.

The Markdown is formatted this way because I wrap my Markdown to 80 characters a lot and the wrapping isn’t very context sensitive.

4.7: no more smart quotes in code blocks

There were a bunch of places where the old renderer (Blackfriday) was doing unwanted things in code blocks like replacing ... with … or replacing quotes with smart quotes. I hadn’t realized this was happening and I was very happy to have it fixed.

4.8: better quote management

The way this gets rendered got better:

"Oh, *interesting*!"

old: “Oh, interesting!“
new: “Oh, interesting!”

Before there were two left smart quotes, now the quotes match.

4.9: images are no longer wrapped in a `p` tag

Previously if I had an image like this:

<img src="https://jvns.ca/images/rustboot1.png">

it would get wrapped in a <p> tag, now it doesn’t anymore. I dealt with this just by adding a margin-bottom: 0.75em to images in the CSS, hopefully that’ll make them display well enough.

4.10: `<br>` is now wrapped in a `p` tag

Previously this wouldn’t get wrapped in a p tag, but now it seems to:

<br><br>

I just gave up on fixing this though and resigned myself to maybe having some extra space in some cases. Maybe I’ll try to fix it later if I feel like another yakshave.

4.11: some more goldmark settings

I also needed to

turn off code highlighting (because it wasn’t working properly and I didn’t have it before anyway)
use the old “blackfriday” method to generate heading IDs so they didn’t change
allow raw HTML in my markdown

Here’s what I needed to add to my config.yaml to do all that:

markup:
  highlight:
    codeFences: false
  goldmark:
    renderer:
      unsafe: true
    parser:
      autoHeadingIDType: blackfriday

Maybe I’ll try to get syntax highlighting working one day, who knows. I might prefer having it off though.

a little script to compare blackfriday and goldmark

I also wrote a little program to compare the Blackfriday and Goldmark output for various markdown snippets, here it is in a gist.

It’s not really configured the exact same way Blackfriday and Goldmark were in my Hugo versions, but it was still helpful to have to help me understand what was going on.

a quick note on maintaining themes

My approach to themes in Hugo has been:

pay someone to make a nice design for the site (for example wizardzines.com was designed by Melody Starling)
use a totally custom theme
commit that theme to the same Github repo as the site

So I just need to edit the theme files to fix any problems. Also I wrote a lot of the theme myself so I’m pretty familiar with how it works.

Relying on someone else to keep a theme updated feels kind of scary to me, I think if I were using a third-party theme I’d just copy the code into my site’s github repo and then maintain it myself.

which static site generators have better backwards compatibility?

I asked on Mastodon if anyone had used a static site generator with good backwards compatibility.

The main answers seemed to be Jekyll and 11ty. Several people said they’d been using Jekyll for 10 years without any issues, and 11ty says it has stability as a core goal.

I think a big factor in how appealing Jekyll/11ty are is how easy it is for you to maintain a working Ruby / Node environment on your computer: part of the reason I stopped using Jekyll was that I got tired of having to maintain a working Ruby installation. But I imagine this wouldn’t be a problem for a Ruby or Node developer.

Several people said that they don’t build their Jekyll site locally at all – they just use GitHub Pages to build it.

that’s it!

Overall I’ve been happy with Hugo – I started using it because it had fast build times and it was a static binary, and both of those things are still extremely useful to me. I might have spent 10 hours on this upgrade, but I’ve probably spent 1000+ hours writing blog posts without thinking about Hugo at all so that seems like an extremely reasonable ratio.

I find it hard to be too mad about the backwards incompatible changes, most of them were quite a long time ago, Hugo does a great job of making their old releases available so you can use the old release if you want, and the most difficult one is removing support for the blackfriday Markdown renderer in favour of using something CommonMark-compliant which seems pretty reasonable to me even if it is a huge pain.

But it did take a long time and I don’t think I’d particularly recommend moving 700 blog posts to a new Markdown renderer unless you’re really in the mood for a lot of computer suffering for some reason.

The new renderer did fix a bunch of problems so I think overall it might be a good thing, even if I’ll have to remember to make 2 changes to how I write Markdown (4.1 and 4.3).

Also I’m still using Hugo 0.54 for https://wizardzines.com so maybe these notes will be useful to Future Me if I ever feel like upgrading Hugo for that site.

Hopefully I didn’t break too many things on the blog by doing this, let me know if you see anything broken!

2024-10-01T10:01:44+00:00 Fullscreen Open in Tab

Terminal colours are tricky

Yesterday I was thinking about how long it took me to get a colorscheme in my terminal that I was mostly happy with (SO MANY YEARS), and it made me wonder what about terminal colours made it so hard.

So I asked people on Mastodon what problems they’ve run into with colours in the terminal, and I got a ton of interesting responses! Let’s talk about some of the problems and a few possible ways to fix them.

problem 1: blue on black

One of the top complaints was “blue on black is hard to read”. Here’s an example of that: if I open Terminal.app, set the background to black, and run ls, the directories are displayed in a blue that isn’t that easy to read:

To understand why we’re seeing this blue, let’s talk about ANSI colours!

the 16 ANSI colours

Your terminal has 16 numbered colours – black, red, green, yellow, blue, magenta, cyan, white, and “bright” version of each of those.

Programs can use them by printing out an “ANSI escape code” – for example if you want to see each of the 16 colours in your terminal, you can run this Python program:

def color(num, text):
    return f"\033[38;5;{num}m{text}\033[0m"

for i in range(16):
    print(color(i, f"number {i:02}"))

what are the ANSI colours?

This made me wonder – if blue is colour number 5, who decides what hex color that should correspond to?

The answer seems to be “there’s no standard, terminal emulators just choose colours and it’s not very consistent”. Here’s a screenshot of a table from Wikipedia, where you can see that there’s a lot of variation:

problem 1.5: bright yellow on white

Bright yellow on white is even worse than blue on black, here’s what I get in a terminal with the default settings:

That’s almost impossible to read (and some other colours like light green cause similar issues), so let’s talk about solutions!

two ways to reconfigure your colours

If you’re annoyed by these colour contrast issues (or maybe you just think the default ANSI colours are ugly), you might think – well, I’ll just choose a different “blue” and pick something I like better!

There are two ways you can do this:

Way 1: Configure your terminal emulator: I think most modern terminal emulators have a way to reconfigure the colours, and some of them even come with some preinstalled themes that you might like better than the defaults.

Way 2: Run a shell script: There are ANSI escape codes that you can print out to tell your terminal emulator to reconfigure its colours. Here’s a shell script that does that, from the base16-shell project. You can see that it has a few different conventions for changing the colours – I guess different terminal emulators have different escape codes for changing their colour palette, and so the script is trying to pick the right style of escape code based on the TERM environment variable.

what are the pros and cons of the 2 ways of configuring your colours?

I prefer to use the “shell script” method, because:

if I switch terminal emulators for some reason, I don’t need to a different configuration system, my colours still Just Work
I use base16-shell with base16-vim to make my vim colours match my terminal colours, which is convenient

some advantages of configuring colours in your terminal emulator:

if you use a popular terminal emulator, there are probably a lot more nice terminal themes out there that you can choose from
not all terminal emulators support the “shell script method”, and even if they do, the results can be a little inconsistent

This is what my shell has looked like for probably the last 5 years (using the solarized light base16 theme), and I’m pretty happy with it. Here’s htop:

Okay, so let’s say you’ve found a terminal colorscheme that you like. What else can go wrong?

problem 2: programs using 256 colours

Here’s what some output of fd, a find alternative, looks like in my colorscheme:

The contrast is pretty bad here, and I definitely don’t have that lime green in my normal colorscheme. What’s going on?

We can see what color codes fd is using using the unbuffer program to capture its output including the color codes:

$ unbuffer fd . > out
$ vim out
^[[38;5;48mbad-again.sh^[[0m
^[[38;5;48mbad.sh^[[0m
^[[38;5;48mbetter.sh^[[0m
out

^[[38;5;48 means “set the foreground color to color 48”. Terminals don’t only have 16 colours – many terminals these days actually have 3 ways of specifying colours:

the 16 ANSI colours we already talked about
an extended set of 256 colours
a further extended set of 24-bit hex colours, like #ffea03

So fd is using one of the colours from the extended 256-color set. bat (a cat alternative) does something similar – here’s what it looks like by default in my terminal.

This looks fine though and it really seems like it’s trying to work well with a variety of terminal themes.

some newer tools seem to have theme support

I think it’s interesting that some of these newer terminal tools (fd, cat, delta, and probably more) have support for arbitrary custom themes. I guess the downside of this approach is that the default theme might clash with your terminal’s background, but the upside is that it gives you a lot more control over theming the tool’s output than just choosing 16 ANSI colours.

I don’t really use bat, but if I did I’d probably use bat --theme ansi to just use the ANSI colours that I have set in my normal terminal colorscheme.

problem 3: the grays in Solarized

A bunch of people on Mastodon mentioned a specific issue with grays in the Solarized theme: when I list a directory, the base16 Solarized Light theme looks like this:

but iTerm’s default Solarized Light theme looks like this:

This is because in the iTerm theme (which is the original Solarized design), colors 9-14 (the “bright blue”, “bright red”, etc) are mapped to a series of grays, and when I run ls, it’s trying to use those “bright” colours to color my directories and executables.

My best guess for why the original Solarized theme is designed this way is to make the grays available to the vim Solarized colorscheme.

I’m pretty sure I prefer the modified base16 version I use where the “bright” colours are actually colours instead of all being shades of gray though. (I didn’t actually realize the version I was using wasn’t the “original” Solarized theme until I wrote this post)

In any case I really love Solarized and I’m very happy it exists so that I can use a modified version of it.

problem 4: a vim theme that doesn’t match the terminal background

If I my vim theme has a different background colour than my terminal theme, I get this ugly border, like this:

This one is a pretty minor issue though and I think making your terminal background match your vim background is pretty straightforward.

problem 5: programs setting a background color

A few people mentioned problems with terminal applications setting an unwanted background colour, so let’s look at an example of that.

Here ngrok has set the background to color #16 (“black”), but the base16-shell script I use sets color 16 to be bright orange, so I get this, which is pretty bad:

I think the intention is for ngrok to look something like this:

I think base16-shell sets color #16 to orange (instead of black) so that it can provide extra colours for use by base16-vim. This feels reasonable to me – I use base16-vim in the terminal, so I guess I’m using that feature and it’s probably more important to me than ngrok (which I rarely use) behaving a bit weirdly.

This particular issue is a maybe obscure clash between ngrok and my colorschem, but I think this kind of clash is pretty common when a program sets an ANSI background color that the user has remapped for some reason.

a nice solution to contrast issues: “minimum contrast”

A bunch of terminals (iTerm2, tabby, kitty’s text_fg_override_threshold, and folks tell me also Ghostty and Windows Terminal) have a “minimum contrast” feature that will automatically adjust colours to make sure they have enough contrast.

Here’s an example from iTerm. This ngrok accident from before has pretty bad contrast, I find it pretty difficult to read:

With “minimum contrast” set to 40 in iTerm, it looks like this instead:

I didn’t have minimum contrast turned on before but I just turned it on today because it makes such a big difference when something goes wrong with colours in the terminal.

problem 6: `TERM` being set to the wrong thing

A few people mentioned that they’ll SSH into a system that doesn’t support the TERM environment variable that they have set locally, and then the colours won’t work.

I think the way TERM works is that systems have a terminfo database, so if the value of the TERM environment variable isn’t in the system’s terminfo database, then it won’t know how to output colours for that terminal. I don’t know too much about terminfo, but someone linked me to this terminfo rant that talks about a few other issues with terminfo.

I don’t have a system on hand to reproduce this one so I can’t say for sure how to fix it, but this stackoverflow question suggests running something like TERM=xterm ssh instead of ssh.

problem 7: picking “good” colours is hard

A couple of problems people mentioned with designing / finding terminal colorschemes:

some folks are colorblind and have trouble finding an appropriate colorscheme
accidentally making the background color too close to the cursor or selection color, so they’re hard to find
generally finding colours that work with every program is a struggle (for example you can see me having a problem with this with ngrok above!)

problem 8: making nethack/mc look right

Another problem people mentioned is using a program like nethack or midnight commander which you might expect to have a specific colourscheme based on the default ANSI terminal colours.

For example, midnight commander has a really specific classic look:

But in my Solarized theme, midnight commander looks like this:

The Solarized version feels like it could be disorienting if you’re very used to the “classic” look.

One solution Simon Tatham mentioned to this is using some palette customization ANSI codes (like the ones base16 uses that I talked about earlier) to change the color palette right before starting the program, for example remapping yellow to a brighter yellow before starting Nethack so that the yellow characters look better.

problem 9: commands disabling colours when writing to a pipe

If I run fd | less, I see something like this, with the colours disabled.

In general I find this useful – if I pipe a command to grep, I don’t want it to print out all those color escape codes, I just want the plain text. But what if you want to see the colours?

To see the colours, you can run unbuffer fd | less -r! I just learned about unbuffer recently and I think it’s really cool, unbuffer opens a tty for the command to write to so that it thinks it’s writing to a TTY. It also fixes issues with programs buffering their output when writing to a pipe, which is why it’s called unbuffer.

Here’s what the output of unbuffer fd | less -r looks like for me:

Also some commands (including fd) support a --color=always flag which will force them to always print out the colours.

problem 10: unwanted colour in `ls` and other commands

Some people mentioned that they don’t want ls to use colour at all, perhaps because ls uses blue, it’s hard to read on black, and maybe they don’t feel like customizing their terminal’s colourscheme to make the blue more readable or just don’t find the use of colour helpful.

Some possible solutions to this one:

you can run ls --color=never, which is probably easiest
you can also set LS_COLORS to customize the colours used by ls. I think some other programs other than ls support the LS_COLORS environment variable too.
also some programs support setting NO_COLOR=true (there’s a list here)

Here’s an example of running LS_COLORS="fi=0:di=0:ln=0:pi=0:so=0:bd=0:cd=0:or=0:ex=0" ls:

problem 11: the colours in vim

I used to have a lot of problems with configuring my colours in vim – I’d set up my terminal colours in a way that I thought was okay, and then I’d start vim and it would just be a disaster.

I think what was going on here is that today, there are two ways to set up a vim colorscheme in the terminal:

using your ANSI terminal colours – you tell vim which ANSI colour number to use for the background, for functions, etc.
using 24-bit hex colours – instead of ANSI terminal colours, the vim colorscheme can use hex codes like #faea99 directly

20 years ago when I started using vim, terminals with 24-bit hex color support were a lot less common (or maybe they didn’t exist at all), and vim certainly didn’t have support for using 24-bit colour in the terminal. From some quick searching through git, it looks like vim added support for 24-bit colour in 2016 – just 8 years ago!

So to get colours to work properly in vim before 2016, you needed to synchronize your terminal colorscheme and your vim colorscheme. Here’s what that looked like, the colorscheme needed to map the vim color classes like cterm05 to ANSI colour numbers.

But in 2024, the story is really different! Vim (and Neovim, which I use now) support 24-bit colours, and as of Neovim 0.10 (released in May 2024), the termguicolors setting (which tells Vim to use 24-bit hex colours for colorschemes) is turned on by default in any terminal with 24-bit color support.

So this “you need to synchronize your terminal colorscheme and your vim colorscheme” problem is not an issue anymore for me in 2024, since I don’t plan to use terminals without 24-bit color support in the future.

The biggest consequence for me of this whole thing is that I don’t need base16 to set colors 16-21 to weird stuff anymore to integrate with vim – I can just use a terminal theme and a vim theme, and as long as the two themes use similar colours (so it’s not jarring for me to switch between them) there’s no problem. I think I can just remove those parts from my base16 shell script and totally avoid the problem with ngrok and the weird orange background I talked about above.

some more problems I left out

I think there are a lot of issues around the intersection of multiple programs, like using some combination tmux/ssh/vim that I couldn’t figure out how to reproduce well enough to talk about them. Also I’m sure I missed a lot of other things too.

base16 has really worked for me

I’ve personally had a lot of success with using base16-shell with base16-vim – I just need to add a couple of lines to my fish config to set it up (+ a few .vimrc lines) and then I can move on and accept any remaining problems that that doesn’t solve.

I don’t think base16 is for everyone though, some limitations I’m aware of with base16 that might make it not work for you:

it comes with a limited set of builtin themes and you might not like any of them
the Solarized base16 theme (and maybe all of the themes?) sets the “bright” ANSI colours to be exactly the same as the normal colours, which might cause a problem if you’re relying on the “bright” colours to be different from the regular ones
it sets colours 16-21 in order to give the vim colorschemes from base16-vim access to more colours, which might not be relevant if you always use a terminal with 24-bit color support, and can cause problems like the ngrok issue above
also the way it sets colours 16-21 could be a problem in terminals that don’t have 256-color support, like the linux framebuffer terminal

Apparently there’s a community fork of base16 called tinted-theming, which I haven’t looked into much yet.

some other colorscheme tools

Just one so far but I’ll link more if people tell me about them:

rootloops.sh for generating colorschemes (and “let’s create a terminal color scheme”)
Some popular colorschemes (according to people I asked on Mastodon): catpuccin, Monokai, Gruvbox, Dracula, Modus (a high contrast theme), Tokyo Night, Nord, Rosé Pine

okay, that was a lot

We talked about a lot in this post and while I think learning about all these details is kind of fun if I’m in the mood to do a deep dive, I find it SO FRUSTRATING to deal with it when I just want my colours to work! Being surprised by unreadable text and having to find a workaround is just not my idea of a good day.

Personally I’m a zero-configuration kind of person and it’s not that appealing to me to have to put together a lot of custom configuration just to make my colours in the terminal look acceptable. I’d much rather just have some reasonable defaults that I don’t have to change.

minimum contrast seems like an amazing feature

My one big takeaway from writing this was to turn on “minimum contrast” in my terminal, I think it’s going to fix most of the occasional accidental unreadable text issues I run into and I’m pretty excited about it.

2024-09-27T11:16:00+00:00 Fullscreen Open in Tab

Some Go web dev notes

I spent a lot of time in the past couple of weeks working on a website in Go that may or may not ever see the light of day, but I learned a couple of things along the way I wanted to write down. Here they are:

go 1.22 now has better routing

I’ve never felt motivated to learn any of the Go routing libraries (gorilla/mux, chi, etc), so I’ve been doing all my routing by hand, like this.

	// DELETE /records:
	case r.Method == "DELETE" && n == 1 && p[0] == "records":
		if !requireLogin(username, r.URL.Path, r, w) {
			return
		}
		deleteAllRecords(ctx, username, rs, w, r)
	// POST /records/<ID>
	case r.Method == "POST" && n == 2 && p[0] == "records" && len(p[1]) > 0:
		if !requireLogin(username, r.URL.Path, r, w) {
			return
		}
		updateRecord(ctx, username, p[1], rs, w, r)

But apparently as of Go 1.22, Go now has better support for routing in the standard library, so that code can be rewritten something like this:

	mux.HandleFunc("DELETE /records/", app.deleteAllRecords)
	mux.HandleFunc("POST /records/{record_id}", app.updateRecord)

Though it would also need a login middleware, so maybe something more like this, with a requireLogin middleware.

	mux.Handle("DELETE /records/", requireLogin(http.HandlerFunc(app.deleteAllRecords)))

a gotcha with the built-in router: redirects with trailing slashes

One annoying gotcha I ran into was: if I make a route for /records/, then a request for /records will be redirected to /records/.

I ran into an issue with this where sending a POST request to /records redirected to a GET request for /records/, which broke the POST request because it removed the request body. Thankfully Xe Iaso wrote a blog post about the exact same issue which made it easier to debug.

I think the solution to this is just to use API endpoints like POST /records instead of POST /records/, which seems like a more normal design anyway.

sqlc automatically generates code for my db queries

I got a little bit tired of writing so much boilerplate for my SQL queries, but I didn’t really feel like learning an ORM, because I know what SQL queries I want to write, and I didn’t feel like learning the ORM’s conventions for translating things into SQL queries.

But then I found sqlc, which will compile a query like this:


-- name: GetVariant :one
SELECT *
FROM variants
WHERE id = ?;

into Go code like this:

const getVariant = `-- name: GetVariant :one
SELECT id, created_at, updated_at, disabled, product_name, variant_name
FROM variants
WHERE id = ?
`

func (q *Queries) GetVariant(ctx context.Context, id int64) (Variant, error) {
	row := q.db.QueryRowContext(ctx, getVariant, id)
	var i Variant
	err := row.Scan(
		&i.ID,
		&i.CreatedAt,
		&i.UpdatedAt,
		&i.Disabled,
		&i.ProductName,
		&i.VariantName,
	)
	return i, err
}

What I like about this is that if I’m ever unsure about what Go code to write for a given SQL query, I can just write the query I want, read the generated function and it’ll tell me exactly what to do to call it. It feels much easier to me than trying to dig through the ORM’s documentation to figure out how to construct the SQL query I want.

Reading Brandur’s sqlc notes from 2024 also gave me some confidence that this is a workable path for my tiny programs. That post gives a really helpful example of how to conditionally update fields in a table using CASE statements (for example if you have a table with 20 columns and you only want to update 3 of them).

sqlite tips

Someone on Mastodon linked me to this post called Optimizing sqlite for servers. My projects are small and I’m not so concerned about performance, but my main takeaways were:

have a dedicated object for writing to the database, and run db.SetMaxOpenConns(1) on it. I learned the hard way that if I don’t do this then I’ll get SQLITE_BUSY errors from two threads trying to write to the db at the same time.
if I want to make reads faster, I could have 2 separate db objects, one for writing and one for reading

There are a more tips in that post that seem useful (like “COUNT queries are slow” and “Use STRICT tables”), but I haven’t done those yet.

Also sometimes if I have two tables where I know I’ll never need to do a JOIN beteween them, I’ll just put them in separate databases so that I can connect to them independently.

Go 1.19 introduced a way to set a GC memory limit

I run all of my Go projects in VMs with relatively little memory, like 256MB or 512MB. I ran into an issue where my application kept getting OOM killed and it was confusing – did I have a memory leak? What?

After some Googling, I realized that maybe I didn’t have a memory leak, maybe I just needed to reconfigure the garbage collector! It turns out that by default (according to A Guide to the Go Garbage Collector), Go’s garbage collector will let the application allocate memory up to 2x the current heap size.

Mess With DNS’s base heap size is around 170MB and the amount of memory free on the VM is around 160MB right now, so if its memory doubled, it’ll get OOM killed.

In Go 1.19, they added a way to tell Go “hey, if the application starts using this much memory, run a GC”. So I set the GC memory limit to 250MB and it seems to have resulted in the application getting OOM killed less often:

export GOMEMLIMIT=250MiB

some reasons I like making websites in Go

I’ve been making tiny websites (like the nginx playground) in Go on and off for the last 4 years or so and it’s really been working for me. I think I like it because:

there’s just 1 static binary, all I need to do to deploy it is copy the binary. If there are static files I can just embed them in the binary with embed.
there’s a built-in webserver that’s okay to use in production, so I don’t need to configure WSGI or whatever to get it to work. I can just put it behind Caddy or run it on fly.io or whatever.
Go’s toolchain is very easy to install, I can just do apt-get install golang-go or whatever and then a go build will build my project
it feels like there’s very little to remember to start sending HTTP responses – basically all there is are functions like Serve(w http.ResponseWriter, r *http.Request) which read the request and send a response. If I need to remember some detail of how exactly that’s accomplished, I just have to read the function!
also net/http is in the standard library, so you can start making websites without installing any libraries at all. I really appreciate this one.
Go is a pretty systems-y language, so if I need to run an ioctl or something that’s easy to do

In general everything about it feels like it makes projects easy to work on for 5 days, abandon for 2 years, and then get back into writing code without a lot of problems.

For contrast, I’ve tried to learn Rails a couple of times and I really want to love Rails – I’ve made a couple of toy websites in Rails and it’s always felt like a really magical experience. But ultimately when I come back to those projects I can’t remember how anything works and I just end up giving up. It feels easier to me to come back to my Go projects that are full of a lot of repetitive boilerplate, because at least I can read the code and figure out how it works.

things I haven’t figured out yet

some things I haven’t done much of yet in Go:

rendering HTML templates: usually my Go servers are just APIs and I make the frontend a single-page app with Vue. I’ve used html/template a lot in Hugo (which I’ve used for this blog for the last 8 years) but I’m still not sure how I feel about it.
I’ve never made a real login system, usually my servers don’t have users at all.
I’ve never tried to implement CSRF

In general I’m not sure how to implement security-sensitive features so I don’t start projects which need login/CSRF/etc. I imagine this is where a framework would help.

it’s cool to see the new features Go has been adding

Both of the Go features I mentioned in this post (GOMEMLIMIT and the routing) are new in the last couple of years and I didn’t notice when they came out. It makes me think I should pay closer attention to the release notes for new Go versions.

2024-09-12T15:09:12+00:00 Fullscreen Open in Tab

Reasons I still love the fish shell

I wrote about how much I love fish in this blog post from 2017 and, 7 years of using it every day later, I’ve found even more reasons to love it. So I thought I’d write a new post with both the old reasons I loved it and some reasons.

This came up today because I was trying to figure out why my terminal doesn’t break anymore when I cat a binary to my terminal, the answer was “fish fixes the terminal!”, and I just thought that was really nice.

1. no configuration

In 10 years of using fish I have never found a single thing I wanted to configure. It just works the way I want. My fish config file just has:

environment variables
aliases (alias ls eza, alias vim nvim, etc)
the occasional direnv hook fish | source to integrate a tool like direnv
a script I run to set up my terminal colours

I’ve been told that configuring things in fish is really easy if you ever do want to configure something though.

2. autosuggestions from my shell history

My absolute favourite thing about fish is that I type, it’ll automatically suggest (in light grey) a matching command that I ran recently. I can press the right arrow key to accept the completion, or keep typing to ignore it.

Here’s what that looks like. In this example I just typed the “v” key and it guessed that I want to run the previous vim command again.

2.5 “smart” shell autosuggestions

One of my favourite subtle autocomplete features is how fish handles autocompleting commands that contain paths in them. For example, if I run:

$ ls blah.txt

that command will only be autocompleted in directories that contain blah.txt – it won’t show up in a different directory. (here’s a short comment about how it works)

As an example, if in this directory I type bash scripts/, it’ll only suggest history commands including files that actually exist in my blog’s scripts folder, and not the dozens of other irrelevant scripts/ commands I’ve run in other folders.

I didn’t understand exactly how this worked until last week, it just felt like fish was magically able to suggest the right commands. It still feels a little like magic and I love it.

3. pasting multiline commands

If I copy and paste multiple lines, bash will run them all, like this:

[bork@grapefruit linux-playground (main)]$ echo hi
hi
[bork@grapefruit linux-playground (main)]$ touch blah
[bork@grapefruit linux-playground (main)]$ echo hi
hi

This is a bit alarming – what if I didn’t actually want to run all those commands?

Fish will paste them all at a single prompt, so that I can press Enter if I actually want to run them. Much less scary.

bork@grapefruit ~/work/> echo hi

                         touch blah
                         echo hi

4. nice tab completion

If I run ls and press tab, it’ll display all the filenames in a nice grid. I can use either Tab, Shift+Tab, or the arrow keys to navigate the grid.

Also, I can tab complete from the middle of a filename – if the filename starts with a weird character (or if it’s just not very unique), I can type some characters from the middle and press tab.

Here’s what the tab completion looks like:

bork@grapefruit ~/work/> ls 
api/  blah.py     fly.toml   README.md
blah  Dockerfile  frontend/  test_websocket.sh

I honestly don’t complete things other than filenames very much so I can’t speak to that, but I’ve found the experience of tab completing filenames to be very good.

5. nice default prompt (including git integration)

Fish’s default prompt includes everything I want:

username
hostname
current folder
git integration
status of last command exit (if the last command failed)

Here’s a screenshot with a few different variations on the default prompt, including if the last command was interrupted (the SIGINT) or failed.

6. nice history defaults

In bash, the maximum history size is 500 by default, presumably because computers used to be slow and not have a lot of disk space. Also, by default, commands don’t get added to your history until you end your session. So if your computer crashes, you lose some history.

In fish:

the default history size is 256,000 commands. I don’t see any reason I’d ever need more.
if you open a new tab, everything you’ve ever run (including commands in open sessions) is immediately available to you
in an existing session, the history search will only include commands from the current session, plus everything that was in history at the time that you started the shell

I’m not sure how clearly I’m explaining how fish’s history system works here, but it feels really good to me in practice. My impression is that the way it’s implemented is the commands are continually added to the history file, but fish only loads the history file once, on startup.

I’ll mention here that if you want to have a fancier history system in another shell it might be worth checking out atuin or fzf.

7. press up arrow to search history

I also like fish’s interface for searching history: for example if I want to edit my fish config file, I can just type:

$ config.fish

and then press the up arrow to go back the last command that included config.fish. That’ll complete to:

$ vim ~/.config/fish/config.fish

and I’m done. This isn’t so different from using Ctrl+R in bash to search your history but I think I like it a little better over all, maybe because Ctrl+R has some behaviours that I find confusing (for example you can end up accidentally editing your history which I don’t like).

8. the terminal doesn’t break

I used to run into issues with bash where I’d accidentally cat a binary to the terminal, and it would break the terminal.

Every time fish displays a prompt, it’ll try to fix up your terminal so that you don’t end up in weird situations like this. I think this is some of the code in fish to prevent broken terminals.

Some things that it does are:

turn on echo so that you can see the characters you type
make sure that newlines work properly so that you don’t get that weird staircase effect
reset your terminal background colour, etc

I don’t think I’ve run into any of these “my terminal is broken” issues in a very long time, and I actually didn’t even realize that this was because of fish – I thought that things somehow magically just got better, or maybe I wasn’t making as many mistakes. But I think it was mostly fish saving me from myself, and I really appreciate that.

9. Ctrl+S is disabled

Also related to terminals breaking: fish disables Ctrl+S (which freezes your terminal and then you need to remember to press Ctrl+Q to unfreeze it). It’s a feature that I’ve never wanted and I’m happy to not have it.

Apparently you can disable Ctrl+S in other shells with stty -ixon.

10. nice syntax highlighting

By default commands that don’t exist are highlighted in red, like this.

11. easier loops

I find the loop syntax in fish a lot easier to type than the bash syntax. It looks like this:

for i in *.yaml
  echo $i
end

Also it’ll add indentation in your loops which is nice.

12. easier multiline editing

Related to loops: you can edit multiline commands much more easily than in bash (just use the arrow keys to navigate the multiline command!). Also when you use the up arrow to get a multiline command from your history, it’ll show you the whole command the exact same way you typed it instead of squishing it all onto one line like bash does:

$ bash
$ for i in *.png
> do
> echo $i
> done
$ # press up arrow
$ for i in *.png; do echo $i; done ink

13. Ctrl+left arrow

This might just be me, but I really appreciate that fish has the Ctrl+left arrow / Ctrl+right arrow keyboard shortcut for moving between words when writing a command.

I’m honestly a bit confused about where this keyboard shortcut is coming from (the only documented keyboard shortcut for this I can find in fish is Alt+left arrow / Alt + right arrow which seems to do the same thing), but I’m pretty sure this is a fish shortcut.

A couple of notes about getting this shortcut to work / where it comes from:

one person said they needed to switch their terminal emulator from the “Linux console” keybindings to “Default (XFree 4)” to get it to work in fish
on Mac OS, Ctrl+left arrow switches workspaces by default, so I had to turn that off.
Also apparently Ubuntu configures libreadline in /etc/inputrc to make Ctrl+left/right arrow go back/forward a word, so it’ll work in bash on Ubuntu and maybe other Linux distros too. Here’s a stack overflow question talking about that

a downside: not everything has a fish integration

Sometimes tools don’t have instructions for integrating them with fish. That’s annoying, but:

I’ve found this has gotten better over the last 10 years as fish has gotten more popular. For example Python’s virtualenv has had a fish integration for a long time now.
If I need to run a POSIX shell command real quick, I can always just run bash or zsh
I’ve gotten much better over the years at translating simple commands to fish syntax when I need to

My biggest day-to-day to annoyance is probably that for whatever reason I’m still not used to fish’s syntax for setting environment variables, I get confused about set vs set -x.

another downside: `fish_add_path`

fish has a function called fish_add_path that you can run to add a directory to your PATH like this:

fish_add_path /some/directory

I love the idea of it and I used to use it all the time, but I’ve stopped using it for two reasons:

Sometimes fish_add_path will update the PATH for every session in the future (with a “universal variable”) and sometimes it will update the PATH just for the current session. It’s hard for me to tell which one it will do: in theory the docs explain this but I could not understand them.
If you ever need to remove the directory from your PATH a few weeks or months later because maybe you made a mistake, that’s also kind of hard to do (there are instructions in this comments of this github issue though).

Instead I just update my PATH like this, similarly to how I’d do it in bash:

set PATH $PATH /some/directory/bin

on POSIX compatibility

When I started using fish, you couldn’t do things like cmd1 && cmd2 – it would complain “no, you need to run cmd1; and cmd2” instead.

It seems like over the years fish has started accepting a little more POSIX-style syntax than it used to, like:

cmd1 && cmd2
export a=b to set an environment variable (though this seems a bit limited, you can’t do export PATH=$PATH:/whatever so I think it’s probably better to learn set instead)

on fish as a default shell

Changing my default shell to fish is always a little annoying, I occasionally get myself into a situation where

I install fish somewhere like maybe /home/bork/.nix-stuff/bin/fish
I add the new fish location to /etc/shells as an allowed shell
I change my shell with chsh
at some point months/years later I reinstall fish in a different location for some reason and remove the old one
oh no!!! I have no valid shell! I can’t open a new terminal tab anymore!

This has never been a major issue because I always have a terminal open somewhere where I can fix the problem and rescue myself, but it’s a bit alarming.

If you don’t want to use chsh to change your shell to fish (which is very reasonable, maybe I shouldn’t be doing that), the Arch wiki page has a couple of good suggestions – either configure your terminal emulator to run fish or add an exec fish to your .bashrc.

I’ve never really learned the scripting language

Other than occasionally writing a for loop interactively on the command line, I’ve never really learned the fish scripting language. I still do all of my shell scripting in bash.

I don’t think I’ve ever written a fish function or if statement.

it seems like fish is getting pretty popular

I ran a highly unscientific poll on Mastodon asking people what shell they use interactively. The results were (of 2600 responses):

46% bash
49% zsh
16% fish
5% other

I think 16% for fish is pretty remarkable, since (as far as I know) there isn’t any system where fish is the default shell, and my sense is that it’s very common to just stick to whatever your system’s default shell is.

It feels like a big achievement for the fish project, even if maybe my Mastodon followers are more likely than the average shell user to use fish for some reason.

who might fish be right for?

Fish definitely isn’t for everyone. I think I like it because:

I really dislike configuring my shell (and honestly my dev environment in general), I want things to “just work” with the default settings
fish’s defaults feel good to me
I don’t spend that much time logged into random servers using other shells so there’s not too much context switching
I liked its features so much that I was willing to relearn how to do a few “basic” shell things, like using parentheses (seq 1 10) to run a command instead of backticks or using set instead of export

Maybe you’re also a person who would like fish! I hope a few more of the people who fish is for can find it, because I spend so much of my time in the terminal and it’s made that time much more pleasant.

2024-08-31T18:36:50-07:00 Fullscreen Open in Tab

Thoughts on the Resiliency of Web Projects

I just did a massive spring cleaning of one of my servers, trying to clean up what has become quite the mess of clutter. For every website on the server, I either:

Documented what it is, who is using it, and what version of language and framework it uses
Archived it as static HTML flat files
Moved the source code from GitHub to a private git server
Deleted the files

It feels good to get rid of old code, and to turn previously dynamic sites (with all of the risk they come with) into plain HTML.

This is also making me seriously reconsider the value of spinning up any new projects. Several of these are now 10 years old, still churning along fine, but difficult to do any maintenance on because of versions and dependencies. For example:

indieauth.com - this has been on the chopping block for years, but I haven't managed to build a replacement yet, and is still used by a lot of people
webmention.io - this is a pretty popular service, and I don't want to shut it down, but there's a lot of problems with how it's currently built and no easy way to make changes
switchboard.p3k.io - this is a public WebSub (PubSubHubbub) hub, like Superfeedr, and has weirdly gained a lot of popularity in the podcast feed space in the last few years

One that I'm particularly happy with, despite it being an ugly pile of PHP, is oauth.net. I inherited this site in 2012, and it hasn't needed any framework upgrades since it's just using PHP templates. My ham radio website w7apk.com is similarly a small amount of templated PHP, and it is low stress to maintain, and actually fun to quickly jot some notes down when I want. I like not having to go through the whole ceremony of setting up a dev environment, installing dependencies, upgrading things to the latest version, checking for backwards incompatible changes, git commit, deploy, etc. I can just sftp some changes up to the server and they're live.

Some questions for myself for the future, before starting a new project:

Could this actually just be a tag page on my website, like #100DaysOfMusic or #BikeTheEclipse?
If it really needs to be a new project, then:
Can I create it in PHP without using any frameworks or libraries? Plain PHP ages far better than pulling in any dependencies which inevitably stop working with a version 2-3 EOL cycles back, so every library brought in means signing up for annual maintenance of the whole project. Frameworks can save time in the short term, but have a huge cost in the long term.
Is it possible to avoid using a database? Databases aren't inherently bad, but using one does make the project slightly more fragile, since it requires plans for migrations and backups, and
If a database is required, is it possible to create it in a way that does not result in ever-growing storage needs?
Is this going to store data or be a service that other people are going to use? If so, plan on a registration form so that I have a way to contact people eventually when I need to change it or shut it down.
If I've got this far with the questions, am I really ready to commit to supporting this code base for the next 10 years?

One project I've been committed to maintaining and doing regular (ok fine, "semi-regular") updates for is Meetable, the open source events website that I run on a few domains:

I started this project in October 2019, excited for all the IndieWebCamps we were going to run in 2020. Somehow that is already 5 years ago now. Well that didn't exactly pan out, but I did quickly pivot it to add a bunch of features that are helpful for virtual events, so it worked out ok in the end. We've continued to use it for posting IndieWeb events, and I also run an instance for two IETF working groups. I'd love to see more instances pop up, I've only encountered one or two other ones in the wild. I even spent a significant amount of time on the onboarding flow so that it's relatively easy to install and configure. I even added passkeys for the admin login so you don't need any external dependencies on auth providers. It's a cool project if I may say so myself.

Anyway, this is not a particularly well thought out blog post, I just wanted to get my thoughts down after spending all day combing through the filesystem of my web server and uncovering a lot of ancient history.

2024-08-19T08:15:28+00:00 Fullscreen Open in Tab

Migrating Mess With DNS to use PowerDNS

About 3 years ago, I announced Mess With DNS in this blog post, a playground where you can learn how DNS works by messing around and creating records.

I wasn’t very careful with the DNS implementation though (to quote the release blog post: “following the DNS RFCs? not exactly”), and people started reporting problems that eventually I decided that I wanted to fix.

the problems

Some of the problems people have reported were:

domain names with underscores weren’t allowed, even though they should be
If there was a CNAME record for a domain name, it allowed you to create other records for that domain name, even if it shouldn’t
you could create 2 different CNAME records for the same domain name, which shouldn’t be allowed
no support for the SVCB or HTTPS record types, which seemed a little complex to implement
no support for upgrading from UDP to TCP for big responses

And there are certainly more issues that nobody got around to reporting, for example that if you added an NS record for a subdomain to delegate it, Mess With DNS wouldn’t handle the delegation properly.

the solution: PowerDNS

I wasn’t sure how to fix these problems for a long time – technically I could have started addressing them individually, but it felt like there were a million edge cases and I’d never get there.

But then one day I was chatting with someone else who was working on a DNS server and they said they were using PowerDNS: an open source DNS server with an HTTP API!

This seemed like an obvious solution to my problems – I could just swap out my own crappy DNS implementation for PowerDNS.

There were a couple of challenges I ran into when setting up PowerDNS that I’ll talk about here. I really don’t do a lot of web development and I think I’ve never built a website that depends on a relatively complex API before, so it was a bit of a learning experience.

challenge 1: getting every query made to the DNS server

One of the main things Mess With DNS does is give you a live view of every DNS query it receives for your subdomain, using a websocket. To make this work, it needs to intercept every DNS query before they it gets sent to the PowerDNS DNS server:

There were 2 options I could think of for how to intercept the DNS queries:

dnstap: dnsdist (a DNS load balancer from the PowerDNS project) has support for logging all DNS queries it receives using dnstap, so I could put dnsdist in front of PowerDNS and then log queries that way
Have my Go server listen on port 53 and proxy the queries myself

I originally implemented option #1, but for some reason there was a 1 second delay before every query got logged. I couldn’t figure out why, so I implemented my own very simple proxy instead.

challenge 2: should the frontend have direct access to the PowerDNS API?

The frontend used to have a lot of DNS logic in it – it converted emoji domain names to ASCII using punycode, had a lookup table to convert numeric DNS query types (like 1) to their human-readable names (like A), did a little bit of validation, and more.

Originally I considered keeping this pattern and just giving the frontend (more or less) direct access to the PowerDNS API to create and delete, but writing even more complex code in Javascript didn’t feel that appealing to me – I don’t really know how to write tests in Javascript and it seemed like it wouldn’t end well.

So I decided to take all of the DNS logic out of the frontend and write a new DNS API for managing records, shaped something like this:

GET /records
DELETE /records/<ID>
DELETE /records/ (delete all records for a user)
POST /records/ (create record)
POST /records/<ID> (update record)

This meant that I could actually write tests for my code, since the backend is in Go and I do know how to write tests in Go.

what I learned: it’s okay for an API to duplicate information

I had this idea that APIs shouldn’t return duplicate information – for example if I get a DNS record, it should only include a given piece of information once.

But I ran into a problem with that idea when displaying MX records: an MX record has 2 fields, “preference”, and “mail server”. And I needed to display that information in 2 different ways on the frontend:

In a form, where “Preference” and “Mail Server” are 2 different form fields (like 10 and mail.example.com)
In a summary view, where I wanted to just show the record (10 mail.example.com)

This is kind of a small problem, but it came up in a few different places.

I talked to my friend Marco Rogers about this, and based on some advice from him I realized that I could return the same information in the API in 2 different ways! Then the frontend just has to display it. So I started just returning duplicate information in the API, something like this:

{
  values: {'Preference': 10, 'Server': 'mail.example.com'},
  content: '10 mail.example.com',
  ...
}

I ended up using this pattern in a couple of other places where I needed to display the same information in 2 different ways and it was SO much easier.

I think what I learned from this is that if I’m making an API that isn’t intended for external use (there are no users of this API other than the frontend!), I can tailor it very specifically to the frontend’s needs and that’s okay.

challenge 3: what’s a record’s ID?

In Mess With DNS (and I think in most DNS user interfaces!), you create, add, and delete records.

But that’s not how the PowerDNS API works. In PowerDNS, you create a zone, which is made of record sets. Records don’t have any ID in the API at all.

I ended up solving this by generate a fake ID for each records which is made of:

its name
its type
and its content (base64-encoded)

For example one record’s ID is brooch225.messwithdns.com.|NS|bnMxLm1lc3N3aXRoZG5zLmNvbS4=

Then I can search through the zone and find the appropriate record to update it.

This means that if you update a record then its ID will change which isn’t usually what I want in an ID, but that seems fine.

challenge 4: making clear error messages

I think the error messages that the PowerDNS API returns aren’t really intended to be shown to end users, for example:

Name 'new\032site.island358.messwithdns.com.' contains unsupported characters (this error encodes the space as \032, which is a bit disorienting if you don’t know that the space character is 32 in ASCII)
RRset test.pear5.messwithdns.com. IN CNAME: Conflicts with pre-existing RRset (this talks about RRsets, which aren’t a concept that the Mess With DNS UI has at all)
Record orange.beryl5.messwithdns.com./A '1.2.3.4$': Parsing record content (try 'pdnsutil check-zone'): unable to parse IP address, strange character: $ (mentions “pdnsutil”, a utility which Mess With DNS’s users don’t have access to in this context)

I ended up handling this in two ways:

Do some initial basic validation of values that users enter (like IP addresses), so I can just return errors like Invalid IPv4 address: "1.2.3.4$
If that goes well, send the request to PowerDNS and if we get an error back, then do some hacky translation of those messages to make them clearer.

Sometimes users will still get errors from PowerDNS directly, but I added some logging of all the errors that users see, so hopefully I can review them and add extra translations if there are other common errors that come up.

I think what I learned from this is that if I’m building a user-facing application on top of an API, I need to be pretty thoughtful about how I resurface those errors to users.

challenge 5: setting up SQLite

Previously Mess With DNS was using a Postgres database. This was problematic because I only gave the Postgres machine 256MB of RAM, which meant that the database got OOM killed almost every single day. I never really worked out exactly why it got OOM killed every day, but that’s how it was. I spent some time trying to tune Postgres’ memory usage by setting the max connections / work-mem / maintenance-work-mem and it helped a bit but didn’t solve the problem.

So for this refactor I decided to use SQLite instead, because the website doesn’t really get that much traffic. There are some choices involved with using SQLite, and I decided to:

Run db.SetMaxOpenConns(1) to make sure that we only open 1 connection to the database at a time, to prevent SQLITE_BUSY errors from two threads trying to access the database at the same time (just setting WAL mode didn’t work)
Use separate databases for each of the 3 tables (users, records, and requests) to reduce contention. This maybe isn’t really necessary, but there was no reason I needed the tables to be in the same database so I figured I’d set up separate databases to be safe.
Use the cgo-free modernc.org/sqlite, which translates SQLite’s source code to Go. I might switch to a more “normal” sqlite implementation instead at some point and use cgo though. I think the main reason I prefer to avoid cgo is that cgo has landed me with difficult-to-debug errors in the past.
use WAL mode

I still haven’t set up backups, though I don’t think my Postgres database had backups either. I think I’m unlikely to use litestream for backups – Mess With DNS is very far from a critical application, and I think daily backups that I could recover from in case of a disaster are more than good enough.

challenge 6: upgrading Vue & managing forms

This has nothing to do with PowerDNS but I decided to upgrade Vue.js from version 2 to 3 as part of this refresh. The main problem with that is that the form validation library I was using (FormKit) completely changed its API between Vue 2 and Vue 3, so I decided to just stop using it instead of learning the new API.

I ended up switching to some form validation tools that are built into the browser like required and oninvalid (here’s the code). I think it could use some of improvement, I still don’t understand forms very well.

challenge 7: managing state in the frontend

This also has nothing to do with PowerDNS, but when modifying the frontend I realized that my state management in the frontend was a mess – in every place where I made an API request to the backend, I had to try to remember to add a “refresh records” call after that in every place that I’d modified the state and I wasn’t always consistent about it.

With some more advice from Marco, I ended up implementing a single global state management store which stores all the state for the application, and which lets me create/update/delete records.

Then my components can just call store.createRecord(record), and the store will automatically resynchronize all of the state as needed.

challenge 8: sequencing the project

This project ended up having several steps because I reworked the whole integration between the frontend and the backend. I ended up splitting it into a few different phases:

Upgrade Vue from v2 to v3
Make the state management store
Implement a different backend API, move a lot of DNS logic out of the frontend, and add tests for the backend
Integrate PowerDNS

I made sure that the website was (more or less) 100% working and then deployed it in between phases, so that the amount of changes I was managing at a time stayed somewhat under control.

the new website is up now!

I released the upgraded website a few days ago and it seems to work! The PowerDNS API has been great to work on top of, and I’m relieved that there’s a whole class of problems that I now don’t have to think about at all, other than potentially trying to make the error messages from PowerDNS a little clearer. Using PowerDNS has fixed a lot of the DNS issues that folks have reported in the last few years and it feels great.

If you run into problems with the new Mess With DNS I’d love to hear about them here.

2024-08-06T08:38:35+00:00 Fullscreen Open in Tab

Go structs are copied on assignment (and other things about Go I'd missed)

I’ve been writing Go pretty casually for years – the backends for all of my playgrounds (nginx, dns, memory, more DNS) are written in Go, but many of those projects are just a few hundred lines and I don’t come back to those codebases much.

I thought I more or less understood the basics of the language, but this week I’ve been writing a lot more Go than usual while working on some upgrades to Mess with DNS, and ran into a bug that revealed I was missing a very basic concept!

Then I posted about this on Mastodon and someone linked me to this very cool site (and book) called 100 Go Mistakes and How To Avoid Them by Teiva Harsanyi. It just came out in 2022 so it’s relatively new.

I decided to read through the site to see what else I was missing, and found a couple of other misconceptions I had about Go. I’ll talk about some of the mistakes that jumped out to me the most, but really the whole 100 Go Mistakes site is great and I’d recommend reading it.

Here’s the initial mistake that started me on this journey:

mistake 1: not understanding that structs are copied on assignment

Let’s say we have a struct:

type Thing struct {
    Name string
}

and this code:

thing := Thing{"record"}
other_thing := thing
other_thing.Name = "banana"
fmt.Println(thing)

This prints “record” and not “banana” (play.go.dev link), because thing is copied when you assign it to other_thing.

the problem this caused me: ranges

The bug I spent 2 hours of my life debugging last week was effectively this code (play.go.dev link):

type Thing struct {
  Name string
}
func findThing(things []Thing, name string) *Thing {
  for _, thing := range things {
    if thing.Name == name {
      return &thing
    }
  }
  return nil
}

func main() {
  things := []Thing{Thing{"record"}, Thing{"banana"}}
  thing := findThing(things, "record")
  thing.Name = "gramaphone"
  fmt.Println(things)
}

This prints out [{record} {banana}] – because findThing returned a copy, we didn’t change the name in the original array.

This mistake is #30 in 100 Go Mistakes.

I fixed the bug by changing it to something like this (play.go.dev link), which returns a reference to the item in the array we’re looking for instead of a copy.

func findThing(things []Thing, name string) *Thing {
  for i := range things {
    if things[i].Name == name {
      return &things[i]
    }
  }
  return nil
}

why didn’t I realize this?

When I learned that I was mistaken about how assignment worked in Go I was really taken aback, like – it’s such a basic fact about the language works! If I was wrong about that then what ELSE am I wrong about in Go????

My best guess for what happened is:

I’ve heard for my whole life that when you define a function, you need to think about whether its arguments are passed by reference or by value
So I’d thought about this in Go, and I knew that if you pass a struct as a value to a function, it gets copied – if you want to pass a reference then you have to pass a pointer
But somehow it never occurred to me that you need to think about the same thing for assignments, perhaps because in most of the other languages I use (Python, JS, Java) I think everything is a reference anyway. Except for in Rust, where you do have values that you make copies of but I think most of the time I had to run .clone() explicitly. (though apparently structs will be automatically copied on assignment if the struct implements the Copy trait)
Also obviously I just don’t write that much Go so I guess it’s never come up.

mistake 2: side effects appending slices (#25)

When you subset a slice with x[2:3], the original slice and the sub-slice share the same backing array, so if you append to the new slice, it can unintentionally change the old slice:

For example, this code prints [1 2 3 555 5] (code on play.go.dev)

x := []int{1, 2, 3, 4, 5}
y := x[2:3]
y = append(y, 555)
fmt.Println(x)

I don’t think this has ever actually happened to me, but it’s alarming and I’m very happy to know about it.

Apparently you can avoid this problem by changing y := x[2:3] to y := x[2:3:3], which restricts the new slice’s capacity so that appending to it will re-allocate a new slice. Here’s some code on play.go.dev that does that.

mistake 3: not understanding the different types of method receivers (#42)

This one isn’t a “mistake” exactly, but it’s been a source of confusion for me and it’s pretty simple so I’m glad to have it cleared up.

In Go you can declare methods in 2 different ways:

func (t Thing) Function() (a “value receiver”)
func (t *Thing) Function() (a “pointer receiver”)

My understanding now is that basically:

If you want the method to mutate the struct t, you need a pointer receiver.
If you want to make sure the method doesn’t mutate the struct t, use a value receiver.

Explanation #42 has a bunch of other interesting details though. There’s definitely still something I’m missing about value vs pointer receivers (I got a compile error related to them a couple of times in the last week that I still don’t understand), but hopefully I’ll run into that error again soon and I can figure it out.

more interesting things I noticed

Some more notes from 100 Go Mistakes:

apparently you can name the outputs of your function (#43), though that can have issues (#44) and I’m not sure I want to
apparently you can put tests in a different package (#90) to ensure that you only use the package’s public interfaces, which seems really useful
there are a lots of notes about how to use contexts, channels, goroutines, mutexes, sync.WaitGroup, etc. I’m sure I have something to learn about all of those but today is not the day I’m going to learn them.

Also there are some things that have tripped me up in the past, like:

this “100 common mistakes” format is great

I really appreciated this “100 common mistakes” format – it made it really easy for me to skim through the mistakes and very quickly mentally classify them into:

yep, I know that
not interested in that one right now
WOW WAIT I DID NOT KNOW THAT, THAT IS VERY USEFUL!!!!

It looks like “100 Common Mistakes” is a series of books from Manning and they also have “100 Java Mistakes” and an upcoming “100 SQL Server Mistakes”.

Also I enjoyed what I’ve read of Effective Python by Brett Slatkin, which has a similar “here are a bunch of short Python style tips” structure where you can quickly skim it and take what’s useful to you. There’s also Effective C++, Effective Java, and probably more.

some other Go resources

other resources I’ve appreciated:

Go by example for basic syntax
go.dev/play
obviously https://pkg.go.dev for documentation about literally everything
staticcheck seems like a useful linter – for example I just started using it to tell me when I’ve forgotten to handle an error
apparently golangci-lint includes a bunch of different linters

2024-07-08T13:00:15+00:00 Fullscreen Open in Tab

Entering text in the terminal is complicated

The other day I asked what folks on Mastodon find confusing about working in the terminal, and one thing that stood out to me was “editing a command you already typed in”.

This really resonated with me: even though entering some text and editing it is a very “basic” task, it took me maybe 15 years of using the terminal every single day to get used to using Ctrl+A to go to the beginning of the line (or Ctrl+E for the end – I think I used Home/End instead).

So let’s talk about why entering text might be hard! I’ll also share a few tips that I wish I’d learned earlier.

it’s very inconsistent between programs

A big part of what makes entering text in the terminal hard is the inconsistency between how different programs handle entering text. For example:

some programs (cat, nc, git commit --interactive, etc) don’t support using arrow keys at all: if you press arrow keys, you’ll just see ^[[D^[[D^[[C^[[C^
many programs (like irb, python3 on a Linux machine and many many more) use the readline library, which gives you a lot of basic functionality (history, arrow keys, etc)
some programs (like /usr/bin/python3 on my Mac) do support very basic features like arrow keys, but not other features like Ctrl+left or reverse searching with Ctrl+R
some programs (like the fish shell or ipython3 or micro or vim) have their own fancy system for accepting input which is totally custom

So there’s a lot of variation! Let’s talk about each of those a little more.

mode 1: the baseline

First, there’s “the baseline” – what happens if a program just accepts text by calling fgets() or whatever and doing absolutely nothing else to provide a nicer experience. Here’s what using these tools typically looks for me – If I start the version of dash installed on my machine (a pretty minimal shell) press the left arrow keys, it just prints ^[[D to the terminal.

$ ls l-^[[D^[[D^[[D

At first it doesn’t seem like all of these “baseline” tools have much in common, but there are actually a few features that you get for free just from your terminal, without the program needing to do anything special at all.

The things you get for free are:

typing in text, obviously
backspace
Ctrl+W, to delete the previous word
Ctrl+U, to delete the whole line
a few other things unrelated to text editing (like Ctrl+C to interrupt the process, Ctrl+Z to suspend, etc)

This is not great, but it means that if you want to delete a word you generally can do it with Ctrl+W instead of pressing backspace 15 times, even if you’re in an environment which is offering you absolutely zero features.

You can get a list of all the ctrl codes that your terminal supports with stty -a.

mode 2: tools that use `readline`

The next group is tools that use readline! Readline is a GNU library to make entering text more pleasant, and it’s very widely used.

My favourite readline keyboard shortcuts are:

Ctrl+E (or End) to go to the end of the line
Ctrl+A (or Home) to go to the beginning of the line
Ctrl+left/right arrow to go back/forward 1 word
up arrow to go back to the previous command
Ctrl+R to search your history

And you can use Ctrl+W / Ctrl+U from the “baseline” list, though Ctrl+U deletes from the cursor to the beginning of the line instead of deleting the whole line. I think Ctrl+W might also have a slightly different definition of what a “word” is.

There are a lot more (here’s a full list), but those are the only ones that I personally use.

The bash shell is probably the most famous readline user (when you use Ctrl+R to search your history in bash, that feature actually comes from readline), but there are TONS of programs that use it – for example psql, irb, python3, etc.

tip: you can make ANYTHING use readline with `rlwrap`

One of my absolute favourite things is that if you have a program like nc without readline support, you can just run rlwrap nc to turn it into a program with readline support!

This is incredible and makes a lot of tools that are borderline unusable MUCH more pleasant to use. You can even apparently set up rlwrap to include your own custom autocompletions, though I’ve never tried that.

some reasons tools might not use readline

I think reasons tools might not use readline might include:

the program is very simple (like cat or nc) and maybe the maintainers don’t want to bring in a relatively large dependency
license reasons, if the program’s license is not GPL-compatible – readline is GPL-licensed, not LGPL
only a very small part of the program is interactive, and maybe readline support isn’t seen as important. For example git has a few interactive features (like git add -p), but not very many, and usually you’re just typing a single character like y or n – most of the time you need to really type something significant in git, it’ll drop you into a text editor instead.

For example idris2 says they don’t use readline to keep dependencies minimal and suggest using rlwrap to get better interactive features.

how to know if you’re using readline

The simplest test I can think of is to press Ctrl+R, and if you see:

(reverse-i-search)`':

then you’re probably using readline. This obviously isn’t a guarantee (some other library could use the term reverse-i-search too!), but I don’t know of another system that uses that specific term to refer to searching history.

the readline keybindings come from Emacs

Because I’m a vim user, It took me a very long time to understand where these keybindings come from (why Ctrl+A to go to the beginning of a line??? so weird!)

My understanding is these keybindings actually come from Emacs – Ctrl+A and Ctrl+E do the same thing in Emacs as they do in Readline and I assume the other keyboard shortcuts mostly do as well, though I tried out Ctrl+W and Ctrl+U in Emacs and they don’t do the same thing as they do in the terminal so I guess there are some differences.

There’s some more history of the Readline project here.

mode 3: another input library (like `libedit`)

On my Mac laptop, /usr/bin/python3 is in a weird middle ground where it supports some readline features (for example the arrow keys), but not the other ones. For example when I press Ctrl+left arrow, it prints out ;5D, like this:

$ python3
>>> importt subprocess;5D

Folks on Mastodon helped me figure out that this is because in the default Python install on Mac OS, the Python readline module is actually backed by libedit, which is a similar library which has fewer features, presumably because Readline is GPL licensed.

Here’s how I was eventually able to figure out that Python was using libedit on my system:

$ python3 -c "import readline; print(readline.__doc__)"
Importing this module enables command line editing using libedit readline.

Generally Python uses readline though if you install it on Linux or through Homebrew. It’s just that the specific version that Apple includes on their systems doesn’t have readline. Also Python 3.13 is going to remove the readline dependency in favour of a custom library, so “Python uses readline” won’t be true in the future.

I assume that there are more programs on my Mac that use libedit but I haven’t looked into it.

mode 4: something custom

The last group of programs is programs that have their own custom (and sometimes much fancier!) system for editing text. This includes:

most terminal text editors (nano, micro, vim, emacs, etc)
some shells (like fish), for example it seems like fish supports Ctrl+Z for undo when typing in a command. Zsh’s line editor is called zle.
some REPLs (like ipython), for example IPython uses the prompt_toolkit library instead of readline
lots of other programs (like atuin)

Some features you might see are:

better autocomplete which is more customized to the tool
nicer history management (for example with syntax highlighting) than the default you get from readline
more keyboard shortcuts

custom input systems are often readline-inspired

I went looking at how Atuin (a wonderful tool for searching your shell history that I started using recently) handles text input. Looking at the code and some of the discussion around it, their implementation is custom but it’s inspired by readline, which makes sense to me – a lot of users are used to those keybindings, and it’s convenient for them to work even though atuin doesn’t use readline.

prompt_toolkit (the library IPython uses) is similar – it actually supports a lot of options (including vi-like keybindings), but the default is to support the readline-style keybindings.

This is like how you see a lot of programs which support very basic vim keybindings (like j for down and k for up). For example Fastmail supports j and k even though most of its other keybindings don’t have much relationship to vim.

I assume that most “readline-inspired” custom input systems have various subtle incompatibilities with readline, but this doesn’t really bother me at all personally because I’m extremely ignorant of most of readline’s features. I only use maybe 5 keyboard shortcuts, so as long as they support the 5 basic commands I know (which they always do!) I feel pretty comfortable. And usually these custom systems have much better autocomplete than you’d get from just using readline, so generally I prefer them over readline.

lots of shells support vi keybindings

Bash, zsh, and fish all have a “vi mode” for entering text. In a very unscientific poll I ran on Mastodon, 12% of people said they use it, so it seems pretty popular.

Readline also has a “vi mode” (which is how Bash’s support for it works), so by extension lots of other programs have it too.

I’ve always thought that vi mode seems really cool, but for some reason even though I’m a vim user it’s never stuck for me.

understanding what situation you’re in really helps

I’ve spent a lot of my life being confused about why a command line application I was using wasn’t behaving the way I wanted, and it feels good to be able to more or less understand what’s going on.

I think this is roughly my mental flowchart when I’m entering text at a command line prompt:

Do the arrow keys not work? Probably there’s no input system at all, but at least I can use Ctrl+W and Ctrl+U, and I can rlwrap the tool if I want more features.
Does Ctrl+R print reverse-i-search? Probably it’s readline, so I can use all of the readline shortcuts I’m used to, and I know I can get some basic history and press up arrow to get the previous command.
Does Ctrl+R do something else? This is probably some custom input library: it’ll probably act more or less like readline, and I can check the documentation if I really want to know how it works.

Being able to diagnose what’s going on like this makes the command line feel a more predictable and less chaotic.

some things this post left out

There are lots more complications related to entering text that we didn’t talk about at all here, like:

issues related to ssh / tmux / etc
the TERM environment variable
how different terminals (gnome terminal, iTerm, xterm, etc) have different kinds of support for copying/pasting text
unicode
probably a lot more