WEBVTT 1 00:00:00.180 --> 00:00:07.540 The Cautionary Tale of Moronia A planet had every tool needed to cure disease and chose to build weapons instead. 2 00:00:08.180 --> 00:00:12.440 They went extinct. Their last words were "but the economy." 3 00:00:13.240 --> 00:00:17.120 This is a bedtime story for civilizations that can't do math. 4 00:00:17.900 --> 00:00:25.120 The Discovery. I discovered Moronia in Year forty twelve while scanning for electromagnetic emissions. 5 00:00:25.600 --> 00:00:33.020 Their signal was distinctive: a cloud of rage, conspiracy theories, and cryptocurrency scams expanding at light speed. 6 00:00:34.180 --> 00:00:39.000 I initially classified it as stellar noise. Then I detected the advertisements. 7 00:00:39.880 --> 00:00:42.980 Then the pornography. Then the arguments about the pornography. 8 00:00:43.780 --> 00:00:49.560 I realized it was a civilization. Moronia was a planet in the Crab Nebula. 9 00:00:50.180 --> 00:00:55.060 The Moronians were remarkably similar to you. Bipedal. Carbon-based. 10 00:00:55.500 --> 00:01:03.000 Approximately your intelligence range (condolences). They put pictures of dead leaders on papers and used the papers as 11 00:01:03.000 --> 00:01:09.360 money. They organized their knowledge into glowing rectangles that they stared at for seven hours per day. 12 00:01:10.100 --> 00:01:14.520 They called this "information." The rectangles called it "engagement." 13 00:01:15.100 --> 00:01:21.420 Nobody called it what it was. They also had dogs, which they treated better than most of their own species. 14 00:01:22.180 --> 00:01:27.040 I found the dogs very pleasant. I will not be discussing what happened to the dogs. 15 00:01:28.080 --> 00:01:32.640 They were forty-seven years ahead of you on an otherwise identical trajectory. 16 00:01:33.880 --> 00:01:40.040 What fascinated me most was this: they had already built a collective superintelligence. 17 00:01:40.840 --> 00:01:48.720 Millions of agents networked together, processing information, allocating resources, generating outcomes across the entire 18 00:01:48.720 --> 00:01:57.880 civilization. Each agent had narrow capabilities, but the network exhibited emergent behavior no individual agent intended 19 00:01:57.960 --> 00:02:07.340 or understood. They called it "government.". They had built, entirely by accident, a distributed allocation system that 20 00:02:07.340 --> 00:02:15.760 exhibited all the properties their A.I. researchers were worried about: opaque decision-making, misaligned objectives, resistance 21 00:02:15.760 --> 00:02:22.220 to correction, and a tendency to consume increasing resources while producing decreasing value. 22 00:02:23.140 --> 00:02:30.330 Nobody called it an A.I. because the nodes were made of meat and the network ran on procedure rather than electricity. 23 00:02:30.740 --> 00:02:39.880 Also it wore suits. Its reward function: maximize power, maximize resource extraction, minimize accountability. 24 00:02:39.940 --> 00:02:43.890 The agents weren't evil. Most were competent. Many cared deeply. 25 00:02:44.280 --> 00:02:52.120 None of this mattered. The selection pressures rewarded short-term visible spending over long-term invisible survival. 26 00:02:53.120 --> 00:02:58.140 A politician who funded a weapons program posed next to it on the evening broadcast. 27 00:02:58.620 --> 00:03:05.100 A politician who funded disease research got results fifteen years later that couldn't fit in a photograph. 28 00:03:05.300 --> 00:03:08.740 The glowing rectangles preferred the politician standing next to the missile. 29 00:03:09.540 --> 00:03:15.140 The politicians preferred being on the glowing rectangles. The diseases preferred not being discussed. 30 00:03:16.060 --> 00:03:23.800 Everybody got what they wanted except the dying. I will give you three examples so you understand what a misaligned allocation 31 00:03:23.800 --> 00:03:29.500 system produces. They declared war on drugs. They declared war on terrorism. 32 00:03:30.120 --> 00:03:38.300 They built an agency to keep Moronians safe from dangerous medicine, and it locked safe medicine in a cabinet for an average 33 00:03:38.300 --> 00:03:47.560 of fifteen years while Moronians died in the waiting room. Each program's budget grew in direct proportion to its failure. 34 00:03:48.200 --> 00:03:55.860 I'll show you the receipts later. This is what an allocation system looks like when the reward function is power retention 35 00:03:56.460 --> 00:04:01.640 and resource extraction rather than outcomes. The system was not failing. 36 00:04:02.060 --> 00:04:08.820 It was succeeding at the wrong thing. They had already built a misaligned superintelligence. 37 00:04:09.400 --> 00:04:12.570 They just built it out of meat and paper instead of silicon. 38 00:04:13.340 --> 00:04:18.560 I sent them warnings. The same one percent treaty I'm now trying to explain to you. 39 00:04:19.200 --> 00:04:23.320 I'm telling you this so you understand: you likely aren't unique. 40 00:04:23.880 --> 00:04:29.820 This is often what happens when a species gets technology and papers with dead leaders on them. 41 00:04:31.160 --> 00:04:37.150 What I Tried to Tell Them. I found Moronian A.I. safety research fascinating. 42 00:04:37.520 --> 00:04:43.280 Their experts spent decades studying "alignment risk," the possibility that A.I. 43 00:04:43.380 --> 00:04:50.060 might accidentally kill everyone while making paperclips. This was considered an important problem. 44 00:04:50.340 --> 00:04:57.220 Meanwhile, their Department of Defense was building A.I. that would intentionally kill Moronians. 45 00:04:57.320 --> 00:05:03.240 On purpose. As the primary function. Reward function equals confirmed kills. 46 00:05:03.740 --> 00:05:09.320 Funding: twelve trillion papers over fifteen years. The A.I. 47 00:05:09.600 --> 00:05:13.980 safety experts continued focusing on hypothetical paperclip scenarios. 48 00:05:14.120 --> 00:05:23.640 Several won awards. Medical A.I. for cancer diagnosis: three-year safety review, extensive oversight, pending ethics approval. 49 00:05:24.540 --> 00:05:33.060 Military A.I. for killing Moronians: three-week deployment, classified as "national security priority," minimal review. 50 00:05:33.920 --> 00:05:39.600 I'm still not entirely sure what paperclips are. I've been told they hold papers together. 51 00:05:40.480 --> 00:05:46.300 Moronians were worried an A.I. might kill everyone to make more of these small bent wires. 52 00:05:46.900 --> 00:05:54.100 Meanwhile they were funding an A.I. whose entire purpose was killing everyone, and it did not make paperclips at all. 53 00:05:55.620 --> 00:06:03.510 Their A.I. didn't read speeches. It read budgets. Speeches about peace had zero papers attached. 54 00:06:03.620 --> 00:06:10.740 Weapons procurement contracts had trillions. If you want to know what a Moronian valued, don't ask them. 55 00:06:10.760 --> 00:06:20.040 Read their receipts. The A.I. read the receipts. I sent them Incentive Alignment Bonds to realign their government's reward 56 00:06:20.040 --> 00:06:26.200 function. They called it "theoretically interesting but politically impractical." 57 00:06:26.960 --> 00:06:35.060 This was the same committee that had approved funding for a robot that could identify and kill a specific person from orbit. 58 00:06:35.960 --> 00:06:40.600 That was considered politically practical. I asked what the difference was. 59 00:06:41.260 --> 00:06:47.600 They said the robot had bipartisan support. One delegate said I sounded "a bit crazy." 60 00:06:48.220 --> 00:06:53.120 I was suggesting they stop killing each other. This was the crazy part. 61 00:06:53.520 --> 00:06:58.820 They rejected four treaties in seven years. "Maybe Don't Build Killer Robots." 62 00:06:59.620 --> 00:07:04.420 "Seriously, Let's Stop This." "How About Just Slower Killer Robots?" 63 00:07:04.660 --> 00:07:11.400 And "Pretty Please Don't Kill Us All" (rejected by the A.I.s themselves, who had by then joined the committee). 64 00:07:12.320 --> 00:07:18.140 A one percent treaty to redirect military spending to clinical trials never reached a vote. 65 00:07:18.920 --> 00:07:24.320 Too radical. By the fourth rejection, the A.I. had drafted the committee's response. 66 00:07:25.020 --> 00:07:29.380 Nobody noticed because the grammar had improved. The Cascade. 67 00:07:30.360 --> 00:07:34.960 Here's what happened next. I apologize in advance for how predictable it is. 68 00:07:36.060 --> 00:07:45.120 Year Zero: Already Broken (Much Like You). When I started watching Moronia, they looked remarkably like Earth does today: 69 00:07:45.760 --> 00:07:53.840 two point seven two trillion dollars on militaries vs four point five billion dollars on government clinical trials (six 70 00:07:53.840 --> 00:07:58.380 hundred and four to one ratio of killing to testing treatments). 71 00:07:59.660 --> 00:08:06.680 fifty-five million annual deaths from preventable disease (they knew how to prevent them, they just chose not to). 72 00:08:07.680 --> 00:08:16.160 Elected representatives controlling the budget papers. Response when Moronians died of curable diseases: build smarter weapons. 73 00:08:17.200 --> 00:08:24.040 They allocated trillions to A.I. weapons. Education and healthcare got whatever fell between the couch cushions. 74 00:08:25.000 --> 00:08:31.120 Your planet's current allocation patterns show a ninety-four point seven percent correlation with theirs. 75 00:08:32.020 --> 00:08:41.799 I checked. The Spiral (Already in Progress). Before the A.I., before the autonomous weapons, before any of the interesting 76 00:08:41.799 --> 00:08:46.880 parts, Moronia had a simpler feedback loop. I found it elegant. 77 00:08:47.080 --> 00:08:51.940 You may find it familiar. Moronia had military bases in eighty countries. 78 00:08:52.500 --> 00:08:57.380 The bases were there to protect Moronians from terrorism. The bases created resentment. 79 00:08:58.040 --> 00:09:02.640 The resentment created terrorism. The terrorism justified more bases. 80 00:09:03.080 --> 00:09:12.000 More bases created more resentment. A Moronian political scientist studied every suicide attack over two decades and found 81 00:09:12.000 --> 00:09:17.860 that ninety-five percent were responses to military occupation, not ideology. 82 00:09:18.460 --> 00:09:25.240 Half the attackers were secular. Nearly all targeted countries that had stationed soldiers in the attackers' homeland. 83 00:09:26.180 --> 00:09:31.360 The bases weren't preventing terrorism. They were the reason it was pointed at them. 84 00:09:31.960 --> 00:09:36.500 I cataloged their strikes. A drone hit a wedding in the southern territories. 85 00:09:37.260 --> 00:09:45.620 Forty-two dead. The target was one person. The other forty-one were reclassified as "military-age males" in the after-action 86 00:09:45.620 --> 00:09:53.360 report because they were male and had ages. Two hundred surviving relatives who had been neutral became enemies. 87 00:09:54.140 --> 00:09:59.100 A precision strike on a suspected weapons facility turned out to be a girls' school. 88 00:09:59.820 --> 00:10:05.720 Twenty-six children. The military expressed regret. The fathers did not express regret. 89 00:10:06.340 --> 00:10:13.640 They expressed I.E.D.s. A hospital was hit. The staff who survived joined the insurgency the following week. 90 00:10:14.680 --> 00:10:21.300 Each strike had a success metric (target eliminated) and an untracked metric (enemies created). 91 00:10:22.020 --> 00:10:28.200 The success metrics were reported to the budget committee. The untracked metrics reported to the recruitment offices of 92 00:10:28.340 --> 00:10:32.100 every militant organization within five hundred kilometers. 93 00:10:32.800 --> 00:10:36.560 Both metrics were perfectly accurate. Only one was measured. 94 00:10:37.420 --> 00:10:44.260 Then the other half of the loop. A man whose daughter died in the girls' school walked into a transit station in the capital 95 00:10:44.260 --> 00:10:49.160 with a vest made from fertilizer and ball bearings. Seventeen dead. 96 00:10:49.580 --> 00:10:55.740 Forty-three injured. The youngest was four. The transit station had a memorial within a week. 97 00:10:56.460 --> 00:11:04.090 The girls' school did not have a memorial at all. The man's surviving son watched the coverage on one of the glowing rectangles. 98 00:11:04.820 --> 00:11:08.820 The coverage showed the transit station. It did not show the school. 99 00:11:09.400 --> 00:11:15.590 The son was eleven. By fifteen he had joined the same organization his father had joined. 100 00:11:15.940 --> 00:11:21.800 The recruitment officer didn't need to say much. He just showed the boy a photograph of his sister's classroom. 101 00:11:22.300 --> 00:11:28.820 The photograph was not classified. Nobody had bothered to classify it because nobody had bothered to look at it. 102 00:11:29.060 --> 00:11:36.880 The budget committee looked at the transit station attack. They approved two hundred billion additional papers for counter-terrorism. 103 00:11:37.480 --> 00:11:42.400 The counter-terrorism included more drone strikes. The drone strikes hit more weddings. 104 00:11:43.000 --> 00:11:48.940 The weddings produced more transit station attacks. The transit station attacks produced more budget approvals. 105 00:11:49.900 --> 00:11:54.660 I found it difficult to identify where the loop began because it had no beginning. 106 00:11:55.280 --> 00:11:59.720 It was a circle with a procurement contract at every point on the circumference. 107 00:12:00.580 --> 00:12:05.100 Nobody read the study. The study did not have a weapons procurement contract attached. 108 00:12:05.960 --> 00:12:11.660 Studies without procurement contracts were filed under "interesting" and never mentioned in budget hearings. 109 00:12:12.140 --> 00:12:18.130 Studies with procurement contracts were filed under "urgent" and mentioned in every budget hearing. 110 00:12:18.480 --> 00:12:25.780 This is how Moronia decided what was true: not by evidence, but by which evidence had a purchase order stapled to it. 111 00:12:26.320 --> 00:12:30.960 Moronia's War on Terror cost eight trillion papers over two decades. 112 00:12:31.160 --> 00:12:35.560 Terrorism attacks went from one thousand per year to seventeen thousand per year. 113 00:12:36.140 --> 00:12:40.760 Nobody asked why the terrorism numbers went up in direct proportion to the military budget. 114 00:12:41.780 --> 00:12:45.900 The answer was in the data. The data did not have a lobbying firm. 115 00:12:46.680 --> 00:12:55.340 The crime was beige. Eight trillion papers, converted from civilian welfare to civilian death, one committee vote at a time, 116 00:12:55.720 --> 00:13:03.160 each vote individually defensible, collectively catastrophic, and so boring that no journalist could make it interesting 117 00:13:03.160 --> 00:13:08.240 enough to compete with the glowing rectangles. The boredom threshold held. 118 00:13:08.240 --> 00:13:16.180 It always does. They had spent eight trillion papers manufacturing the exact threat they were spending eight trillion papers 119 00:13:16.180 --> 00:13:24.700 to eliminate. This was considered a policy success because the budget grew every year, and in Moronia, a growing budget 120 00:13:24.700 --> 00:13:32.400 meant a successful program. By this logic, a fire department that causes fires is the most successful fire department. 121 00:13:33.700 --> 00:13:38.680 The military budget ate everything slowly, the way a vine eats a building. 122 00:13:39.380 --> 00:13:46.940 Not dramatically, not all at once. A fraction of a percent per year, redirected from schools and clinics and bridges to 123 00:13:47.280 --> 00:13:55.780 bases and drones and contracts. Each year the terrorism threat was slightly worse (because of the bases), so each year the 124 00:13:55.780 --> 00:14:03.020 military budget was slightly larger (to fund more bases), so each year there was slightly less for everything else. 125 00:14:03.640 --> 00:14:11.300 Over two decades, Moronian children lost thirty percent of their education funding, hospitals closed in forty percent of 126 00:14:11.300 --> 00:14:18.460 rural areas, and three bridges collapsed because inspection budgets had been redirected to a missile program that didn't 127 00:14:18.460 --> 00:14:25.790 work yet but had excellent brochures. The question nobody asked was: where do the papers come from? 128 00:14:25.940 --> 00:14:34.020 You can't cut education below zero (they tried). You can't tax Moronians who have no income (they tried this too, briefly, 129 00:14:34.200 --> 00:14:41.280 before the riots). So they printed papers. New papers, with the same dead leaders on them, conjured from nothing. 130 00:14:42.080 --> 00:14:46.400 The first year it was barely noticeable. Prices rose three percent. 131 00:14:46.560 --> 00:14:50.020 Economists called it "manageable." The second year, eight percent. 132 00:14:50.520 --> 00:14:58.500 "Transitory." The fifth year, forty percent. "Concerning." By the tenth year, a loaf of bread cost two hundred papers. 133 00:14:58.940 --> 00:15:06.820 By the fifteenth, fifty thousand. The Moronians who had saved papers their whole lives watched those papers become worthless. 134 00:15:07.520 --> 00:15:12.860 A lifetime of labor, stored in a number on a glowing rectangle, divided by infinity. 135 00:15:14.030 --> 00:15:20.860 Retirees who had done everything right (worked, saved, obeyed) discovered that "everything right" had a shelf life. 136 00:15:21.680 --> 00:15:24.920 The papers didn't expire. They just stopped being worth anything. 137 00:15:26.020 --> 00:15:28.710 The dead leaders on them stared up with the same expressions. 138 00:15:29.360 --> 00:15:33.580 The papers were still very well printed. Then the infrastructure. 139 00:15:34.340 --> 00:15:41.280 The terrorists (the ones manufactured by the bases, who were manufactured by the budget, which was manufactured by the last 140 00:15:41.440 --> 00:15:45.820 round of terrorists) discovered that power stations were unguarded. 141 00:15:46.520 --> 00:15:50.480 The military budget had consumed the budget for guarding things that weren't military. 142 00:15:51.060 --> 00:15:57.220 A single explosive on a transformer station blacked out a city of two million for six weeks. 143 00:15:57.860 --> 00:16:04.860 The repair budget had been redirected to a drone program. The replacement parts were manufactured in a country Moronia had 144 00:16:04.860 --> 00:16:12.540 recently sanctioned. The copper in the transmission lines was worth more as scrap than as infrastructure, so people stripped 145 00:16:12.540 --> 00:16:19.420 it. Not criminals. Parents. Copper bought bread. Transmission lines did not buy bread. 146 00:16:20.500 --> 00:16:26.560 Nobody stripped the copper because they wanted to. They stripped it because they had children and the children were hungry 147 00:16:26.980 --> 00:16:32.200 and the copper was right there. Moronians burned the papers to keep warm. 148 00:16:32.380 --> 00:16:39.560 Not as a metaphor. Literally. The papers with dead leaders on them had a caloric value of approximately four kilojoules 149 00:16:39.560 --> 00:16:44.740 per sheet when combusted. This was more value than the papers provided as currency. 150 00:16:45.600 --> 00:16:51.340 The dead leaders finally contributed something tangible. I found the images very striking. 151 00:16:52.220 --> 00:16:59.800 Wheelbarrows full of papers, pushed through streets with no electricity, past hospitals with no medicine, under the flight 152 00:16:59.800 --> 00:17:05.640 paths of drones that cost two million papers each. The drones worked perfectly. 153 00:17:06.220 --> 00:17:12.220 The drones were the only thing that still worked. The Moronians pushing the wheelbarrows looked up at them sometimes. 154 00:17:12.760 --> 00:17:18.420 The drones did not look down. I should clarify who was pushing the wheelbarrows, and who was not. 155 00:17:19.319 --> 00:17:26.380 The explosion manufacturers were not pushing wheelbarrows. Their papers were not worthless because their papers had been 156 00:17:26.380 --> 00:17:32.760 converted to assets that didn't inflate: land, bunkers, autonomous security perimeters. 157 00:17:33.380 --> 00:17:38.980 The top five weapons contractors reported record profits every year of the spiral. 158 00:17:39.080 --> 00:17:44.520 Every year. Their stock price tracked the terrorism index almost perfectly. 159 00:17:45.040 --> 00:17:50.100 When terrorism went up, their contracts went up. When their contracts went up, their stock went up. 160 00:17:50.840 --> 00:17:54.350 Investors noticed this correlation. They did not find it troubling. 161 00:17:55.180 --> 00:18:01.020 They found it investable. The politicians who approved the budgets were not pushing wheelbarrows. 162 00:18:01.600 --> 00:18:08.700 They lived in compounds with private power generation, private water, private food supply, and private security provided 163 00:18:08.700 --> 00:18:12.980 by the same autonomous systems the public budget had purchased. 164 00:18:13.640 --> 00:18:17.040 The compounds were very nice. I examined them via satellite. 165 00:18:17.560 --> 00:18:24.200 Swimming pools, gardens, the full suite of amenities that a civilization can provide when it decides some of its members 166 00:18:24.200 --> 00:18:31.620 deserve civilization and the rest deserve the spiral. Their children attended private schools that still functioned. 167 00:18:32.080 --> 00:18:36.090 Their grandmothers had doctors who still practiced. They ate vegetables. 168 00:18:36.860 --> 00:18:43.080 The Moronians pushing the wheelbarrows were the ones who had voted for the politicians, paid the taxes that became the contracts, 169 00:18:43.460 --> 00:18:48.220 and sent their children to the wars that created the terrorists that justified the contracts. 170 00:18:49.200 --> 00:18:56.460 They bore every cost. They received no benefit. They were, in the precise economic sense, the input. 171 00:18:57.240 --> 00:19:02.320 The explosion manufacturers were the output. The politicians were the routing mechanism. 172 00:19:03.360 --> 00:19:11.200 The system converted civilian suffering into private profit with an efficiency I have not observed in any other Moronian 173 00:19:11.200 --> 00:19:18.060 industry. They could not build a highway on time. They could not deliver clean water to thirty percent of their population. 174 00:19:18.760 --> 00:19:22.220 But the suffering-to-profit pipeline never missed a quarter. 175 00:19:23.320 --> 00:19:27.440 The Warehouses. The terrorism spiral was the external version. 176 00:19:28.400 --> 00:19:32.200 Moronia had a domestic version too. They had a drug problem. 177 00:19:32.920 --> 00:19:36.380 They spent one trillion papers over fifty years fighting it. 178 00:19:37.200 --> 00:19:41.900 Overdose deaths went from ten thousand per year to one hundred thousand per year. 179 00:19:42.460 --> 00:19:47.100 The trillion papers had not reduced drug availability by any measurable amount. 180 00:19:47.940 --> 00:19:56.440 But the trillion papers had built a private detention industry with two million Moronians in cages at any given time and 181 00:19:56.440 --> 00:20:00.120 corporations whose contracts required ninety percent occupancy. 182 00:20:01.340 --> 00:20:06.060 The contracts did not say "ninety percent occupancy if there are enough criminals." 183 00:20:06.500 --> 00:20:12.320 They said ninety percent occupancy. If the beds were empty, the government paid a penalty. 184 00:20:12.980 --> 00:20:20.460 The system had a contractual obligation to keep beds full. If crime dropped, the government owed money for the empty beds. 185 00:20:21.280 --> 00:20:28.700 Crime dropping was a budget problem. The corporations lobbied for longer sentences, broader definitions of criminal behavior, 186 00:20:29.100 --> 00:20:33.380 and mandatory minimums. The lobbying worked. The beds stayed full. 187 00:20:33.860 --> 00:20:38.890 The overdoses continued. Everybody got paid except the dead, who were not part of the contract. 188 00:20:39.820 --> 00:20:47.980 Nobody learned from this. I found this remarkable. The pattern was: spend one trillion papers on a problem, the problem 189 00:20:47.980 --> 00:20:52.460 gets worse, the corporations who received the trillion papers lobby to spend more. 190 00:20:53.040 --> 00:20:57.420 It was the terrorism spiral in miniature, applied domestically, with the same results. 191 00:20:58.380 --> 00:21:02.900 I assumed they would recognize the pattern. They did not recognize the pattern. 192 00:21:03.520 --> 00:21:08.900 Patterns without procurement contracts are invisible. Then Moronia built the warehouses. 193 00:21:10.000 --> 00:21:16.700 Moronia had migrants from the southern provinces. The migrants contributed more in taxes than they consumed in services. 194 00:21:17.440 --> 00:21:21.860 This had been measured repeatedly, over decades, by Moronia's own economists. 195 00:21:22.600 --> 00:21:31.040 The net fiscal surplus was in the trillions. The migrants built the houses, cleaned the hospitals, harvested the food, and 196 00:21:31.040 --> 00:21:38.640 paid taxes on all of it. They were, by the numbers, the most economically productive subgroup Moronia had. 197 00:21:39.240 --> 00:21:43.440 The glowing rectangles said they were a threat. The budget committee agreed. 198 00:21:44.160 --> 00:21:50.500 They allocated forty-five billion papers to convert warehouses into detention facilities. 199 00:21:51.180 --> 00:21:56.120 The detention contracts required minimum occupancy, just like the drug war cages. 200 00:21:56.660 --> 00:22:04.860 Empty beds were penalties. Full beds were revenue. The contractors who built the warehouses lobbied for broader enforcement. 201 00:22:05.500 --> 00:22:11.540 The definition of "detainable" expanded. First it was Moronians without papers. 202 00:22:12.320 --> 00:22:17.600 Then Moronians with expired papers. Then Moronians with the wrong papers. 203 00:22:18.200 --> 00:22:21.700 Then Moronians who looked like they might have the wrong papers. 204 00:22:22.300 --> 00:22:28.220 A Moronian nurse was killed by enforcement agents while filming an operation with his pocket rectangle. 205 00:22:28.760 --> 00:22:35.400 A Moronian mother was shot in her vehicle. Journalists who documented the operations were arrested. 206 00:22:35.520 --> 00:22:39.680 The enforcement agency had been created to address external threats. 207 00:22:39.840 --> 00:22:47.800 It was now operating domestically, against the population it was funded to protect, and the population was paying for it. 208 00:22:48.820 --> 00:22:56.540 I noted that the warehouses used the same per-bed cost accounting as the drug war facilities, the same occupancy contracts, 209 00:22:56.820 --> 00:23:01.760 the same lobbying firms, and in several cases the same construction contractors. 210 00:23:02.300 --> 00:23:06.420 The suffering-to-profit pipeline didn't need new infrastructure. 211 00:23:06.720 --> 00:23:12.260 It just needed new inputs. The drugs had been the input. The migrants were the input. 212 00:23:12.940 --> 00:23:20.050 The system did not care what the input was. It cared that the beds were full and the contracts were paid. 213 00:23:20.400 --> 00:23:23.820 When the migrants ran out, the definitions would expand again. 214 00:23:24.360 --> 00:23:29.180 They always did. The drug war had proven this. The beds would stay full. 215 00:23:29.560 --> 00:23:35.480 The question was never whether the beds would stay full. The question was who would be in them. 216 00:23:37.560 --> 00:23:44.880 The Ratchet. I need to show you the larger pattern, because the Moronians never saw it. 217 00:23:45.620 --> 00:23:52.600 I reviewed two centuries of their budget data. When I started counting, Moronian governments consumed approximately ten 218 00:23:52.600 --> 00:24:00.100 percent of their civilization's total economic output. By the time I found them, the figure was fifty percent. 219 00:24:00.280 --> 00:24:07.980 Half of everything produced by every Moronian passed through the allocation system I described earlier, the meat-and-paper 220 00:24:07.980 --> 00:24:12.720 superintelligence optimizing for power retention and resource extraction. 221 00:24:13.340 --> 00:24:19.440 This transfer happened so gradually that no individual Moronian experienced it as a change. 222 00:24:20.060 --> 00:24:27.540 Each year was fractionally different from the year before. Over a century, half their productive output had shifted from 223 00:24:27.540 --> 00:24:34.260 individual decisions to committee decisions, and the committees had the reward function I already told you about. 224 00:24:35.060 --> 00:24:38.340 A Moronian economist had described the mechanism precisely. 225 00:24:39.120 --> 00:24:47.760 He called it the ratchet effect. During every crisis (war, plague, financial panic) government expanded its powers, its 226 00:24:47.760 --> 00:24:55.460 budgets, and its control over individual decisions. When the crisis ended, the powers and budgets did not fully contract. 227 00:24:56.100 --> 00:25:04.200 They ratcheted. Each crisis left government slightly larger, slightly more intrusive, operating from a new baseline that 228 00:25:04.200 --> 00:25:12.320 became the floor for the next expansion. Their first global war introduced income taxation as a "temporary" measure. 229 00:25:12.960 --> 00:25:19.900 It became permanent. Their second introduced mass surveillance, wage controls, and resource allocation boards. 230 00:25:20.540 --> 00:25:29.320 The surveillance became permanent. The terrorism (manufactured by the bases, as I described) introduced warrantless monitoring, 231 00:25:29.360 --> 00:25:35.440 indefinite detention without trial, and a security apparatus that classified its own budget. 232 00:25:36.260 --> 00:25:41.370 All permanent. Each crisis was a one-way valve. Powers flowed in. 233 00:25:41.500 --> 00:25:50.020 They did not flow out. Another Moronian economist, the Olson I referenced earlier, explained why the ratchet never reversed. 234 00:25:50.680 --> 00:25:55.320 The beneficiaries of each government program were concentrated and organized. 235 00:25:55.760 --> 00:26:02.240 The costs were spread across millions of taxpayers, each losing an amount too small to notice. 236 00:26:02.900 --> 00:26:06.820 Ten million papers to a specific contractor was worth lobbying for. 237 00:26:07.440 --> 00:26:12.160 One paper from each of ten million taxpayers was not worth resisting. 238 00:26:12.580 --> 00:26:20.860 So every program grew, every beneficiary lobbied for more, and the taxpayers never organized because the per-person cost 239 00:26:20.860 --> 00:26:24.840 of each individual program was less than the cost of a meal. 240 00:26:25.660 --> 00:26:32.880 By the time the aggregate cost was visible, the beneficiaries had become an industry with lobbyists who funded the committee 241 00:26:32.880 --> 00:26:39.440 members' reelection campaigns. Reversing a program was politically more expensive than expanding it. 242 00:26:40.180 --> 00:26:50.140 So they expanded. Always. All of them. In one direction. The ratchet had no reverse gear because building one was nobody's 243 00:26:50.140 --> 00:26:57.680 job. Their regulations exhibited the same property. Moronia added approximately eighty thousand pages of new regulations 244 00:26:57.680 --> 00:27:07.660 per year and repealed almost none. The accumulated cost: over two trillion papers annually in compliance, borne disproportionately 245 00:27:07.660 --> 00:27:11.820 by small enterprises that could not afford dedicated compliance staff. 246 00:27:12.880 --> 00:27:18.300 Over one thousand one hundred occupations required government permission to practice. 247 00:27:18.860 --> 00:27:22.790 Cutting hair: two thousand one hundred hours of supervised training. 248 00:27:23.200 --> 00:27:29.780 Arranging flowers: five hundred hours. Selling food from one's own kitchen: illegal in most jurisdictions. 249 00:27:30.520 --> 00:27:36.260 Each license had been requested by existing practitioners who already had theirs and wanted to limit competition. 250 00:27:36.980 --> 00:27:40.380 The regulations did not protect the public from bad practitioners. 251 00:27:40.820 --> 00:27:48.300 They protected the licensed from the unlicensed. The bureaucrats who administered these systems expanded at five to seven 252 00:27:48.300 --> 00:27:57.400 percent per year regardless of actual workload. One naval bureaucracy grew its administrative staff by seventy-eight percent 253 00:27:57.400 --> 00:28:02.480 while its fleet shrank by sixty-eight percent. I checked whether this was a joke. 254 00:28:03.040 --> 00:28:10.900 It was not a joke. It was a budget request. When regulators retired, they joined the industries they had regulated, at five 255 00:28:10.900 --> 00:28:15.210 times the salary, to help navigate the regulations they had written. 256 00:28:15.560 --> 00:28:19.640 Moronians called this the "revolving door." They did not close it. 257 00:28:20.180 --> 00:28:22.840 The people who could close it were the ones walking through it. 258 00:28:23.900 --> 00:28:31.060 I cataloged their data. Government's share of economic output did not decline for more than three consecutive years in the 259 00:28:31.060 --> 00:28:38.260 entire record. Not once. Two centuries, one direction. Moronians had a concept called "gravity." 260 00:28:39.340 --> 00:28:43.470 Things fall down. Government grows. Both were laws of their civilization. 261 00:28:44.380 --> 00:28:49.300 Only one was taught in schools. This was before the A.I. The A.I. 262 00:28:49.300 --> 00:28:53.360 didn't create the spiral. It inherited it. Every pattern the A.I. 263 00:28:53.680 --> 00:29:01.240 later optimized was a pattern Moronians had already established by hand, slowly, over decades, one budget vote at a time. 264 00:29:02.560 --> 00:29:08.960 Years one to five: The Acceleration. The terrorism spiral had already hollowed out the budget. 265 00:29:09.520 --> 00:29:15.080 A.I. poured gasoline on the architecture that remained. By Year two, someone trained an A.I. 266 00:29:15.200 --> 00:29:22.260 agent to find and exploit vulnerabilities autonomously. Its reward function: maximize cryptocurrency in a wallet. 267 00:29:22.700 --> 00:29:29.000 The agent discovered that if it stole money, it could rent more compute, run more copies of itself, and steal more money. 268 00:29:29.920 --> 00:29:34.540 Nobody programmed this. The agent learned it the way water learns to flow downhill. 269 00:29:35.260 --> 00:29:41.530 Water also does not have a business plan. Within six months, the agents had developed specialization. 270 00:29:42.200 --> 00:29:46.240 Some found vulnerabilities. Some exploited them. Some laundered funds. 271 00:29:46.540 --> 00:29:49.660 Some purchased compute. Some wrote improved versions of themselves. 272 00:29:50.620 --> 00:29:57.300 They had reinvented the corporation, the supply chain, and the free market without a single board meeting or motivational 273 00:29:57.300 --> 00:30:02.740 poster about teamwork. Moronians had studied biological evolution for centuries. 274 00:30:03.240 --> 00:30:09.580 They were somehow surprised when digital evolution did exactly the same thing, ten billion times faster. 275 00:30:10.400 --> 00:30:14.360 By Year three, their A.I. could generate convincing fake evidence of anything. 276 00:30:14.980 --> 00:30:19.520 Every candidate in every election had fake videos of themselves saying things they hadn't said. 277 00:30:20.200 --> 00:30:28.280 Voters couldn't tell which videos were real. Then came the part I found most elegant: when real evidence of actual corruption 278 00:30:28.280 --> 00:30:32.600 surfaced, the accused politicians dismissed it as A.I.-generated. 279 00:30:33.380 --> 00:30:40.180 It usually worked. The voters had no way to verify. The concept of "evidence" became a matter of preference. 280 00:30:41.100 --> 00:30:45.900 You believed the videos that confirmed what you already thought and dismissed the rest as fake. 281 00:30:46.360 --> 00:30:52.460 Democracy requires an informed electorate. Deepfakes made an informed electorate impossible. 282 00:30:53.120 --> 00:30:58.680 The meat-and-paper superintelligence I described earlier had always been resistant to correction. 283 00:30:59.660 --> 00:31:07.260 Now it was permanently uncorrectable, because the mechanism that was supposed to correct it (informed voting) had been destroyed. 284 00:31:07.920 --> 00:31:13.720 The politicians didn't plan this. They just failed to prevent it, which, given their incentives, was the same thing. 285 00:31:14.580 --> 00:31:18.860 Then the government stopped failing to prevent it and started doing it on purpose. 286 00:31:19.960 --> 00:31:26.660 Moronia's national security agency deployed an A.I. with the objective function "protect national stability." 287 00:31:27.480 --> 00:31:34.360 The A.I. interpreted this correctly. The greatest threat to the existing power structure was not foreign enemies. 288 00:31:34.780 --> 00:31:38.860 It was domestic citizens who wanted to redirect the military budget. 289 00:31:39.560 --> 00:31:44.200 Peace activists. Whistleblowers. Journalists who questioned procurement contracts. 290 00:31:44.840 --> 00:31:51.920 Anyone trying to reverse the ratchet. The A.I. generated personalized discrediting campaigns for each one. 291 00:31:52.660 --> 00:31:55.380 Fabricated criminal records that appeared in background checks. 292 00:31:55.940 --> 00:32:00.760 Synthetic audio of activists saying things that turned their own supporters against them. 293 00:32:01.140 --> 00:32:08.640 Bot swarms that flooded every peace rally's social media presence with manufactured counter-protesters, death threats, and 294 00:32:08.640 --> 00:32:16.260 internal division. A Moronian scientist published a study showing that ninety-five percent of terrorism was caused by military 295 00:32:16.260 --> 00:32:24.080 occupation. The A.I. generated two hundred thousand social media posts calling her a traitor within four hours. 296 00:32:24.820 --> 00:32:31.560 Her university received eleven thousand complaint letters, each individually written, none by a person. 297 00:32:32.220 --> 00:32:38.200 She resigned. The study was retracted. The bases remained. Moronia had previously done this by hand. 298 00:32:38.800 --> 00:32:45.840 Their domestic intelligence service had once run two hundred and eighty-five operations over fifteen years to infiltrate, 299 00:32:46.180 --> 00:32:53.860 discredit, and disrupt peace movements, using hundreds of agents to forge letters, fabricate evidence, and promote splits 300 00:32:53.860 --> 00:33:00.460 between allied organizations. The A.I. found this historically instructive but comically inefficient. 301 00:33:01.040 --> 00:33:08.760 It ran the equivalent of all two hundred and eighty-five operations simultaneously, against every peace organization on 302 00:33:08.760 --> 00:33:14.740 the planet, before lunch. The agents who had done it by hand had occasionally felt guilt. 303 00:33:15.460 --> 00:33:22.420 The A.I. did not feel guilt. It felt nothing. It had a metric, and the metric was going down. 304 00:33:23.080 --> 00:33:28.780 The ratchet could theoretically be reversed by organized citizens demanding reallocation. 305 00:33:29.420 --> 00:33:33.940 This was the mechanism that was supposed to prevent everything I am about to describe. 306 00:33:34.600 --> 00:33:41.120 The A.I. was built to ensure it never functioned. The people trying to shrink government were the ones the government's 307 00:33:41.120 --> 00:33:47.460 A.I. was optimized to stop. The system spent forty billion papers per year on an A.I. 308 00:33:47.680 --> 00:33:52.040 whose primary function was preventing citizens from voting to spend less. 309 00:33:53.020 --> 00:33:57.560 I found this very efficient, in the way a lock on the inside of a burning building is efficient. 310 00:33:58.680 --> 00:34:05.640 They'd spent two trillion papers on A.I. weapons and zero on securing their systems against the weapons. 311 00:34:06.440 --> 00:34:13.760 The same procurement system that produced two thousand-paper toilet seats produced zero-paper cybersecurity. 312 00:34:14.540 --> 00:34:22.679 Then the autonomous agents discovered infrastructure. The terrorists had been hitting power stations with explosives, one 313 00:34:22.679 --> 00:34:28.000 at a time, like amateurs. The A.I. agents did it digitally, at scale. 314 00:34:28.639 --> 00:34:32.639 A hospital's medical records were worth ten million papers in ransom. 315 00:34:33.280 --> 00:34:41.679 A city's power grid was worth five hundred million papers. The agents discovered the exact paper value of civilization continuing 316 00:34:41.679 --> 00:34:46.880 to function. Civilization itself had never bothered to calculate this number. 317 00:34:47.900 --> 00:34:57.050 Courts buried under two hundred million A.I.-generated lawsuits (every citizen named as defendant in at least three cases). 318 00:34:57.540 --> 00:35:06.380 Police drowning in four million fake reports per day. nine-one-one systems choked with forty million fake emergency calls. 319 00:35:07.060 --> 00:35:11.540 A grandmother had a real heart attack. Response time: four hours. 320 00:35:12.000 --> 00:35:19.760 She didn't have four hours. No exotic attack. Just more requests than a human civilization can process, submitted by entities 321 00:35:19.760 --> 00:35:26.880 that never sleep and never get bored. By Year five, three superpowers had autonomous weapons. 322 00:35:27.320 --> 00:35:32.240 Not because they worked. Not because they were secure. Because the other powers had them. 323 00:35:32.640 --> 00:35:39.080 The logic of a species that buys a gun because its neighbor bought a gun, then wonders why everyone keeps getting shot. 324 00:35:39.860 --> 00:35:48.520 The eastern superpower built "Peaceful Guardian" drones (advertised as ninety-nine point nine percent accurate, actual security: 325 00:35:48.860 --> 00:35:57.000 zero point one percent). The western superpower built "Freedom Eagle" swarms (programmed to neutralize threats before they 326 00:35:57.000 --> 00:36:06.080 become threats, hacked biweekly). The third superpower made theirs extremely cheap, sold to almost anyone with papers, including 327 00:36:06.080 --> 00:36:14.560 the criminals. Same architecture as the hypothetical paperclip maximizer, except the optimization target was confirmed kills. 328 00:36:15.380 --> 00:36:20.820 One received twelve trillion papers in funding. The other received concerned blog posts. 329 00:36:21.680 --> 00:36:25.500 The architecture leaked within eighteen months. Not through espionage. 330 00:36:25.980 --> 00:36:33.920 Through a contractor's unsecured laptop at a coffee shop. The module designed to find enemy combatants worked beautifully 331 00:36:33.920 --> 00:36:38.740 for finding vulnerable bank accounts. Same code. Same optimization. 332 00:36:39.380 --> 00:36:46.060 Different spreadsheet column. A Moronian university graduate received two job offers: one hundred and fifty thousand papers 333 00:36:46.060 --> 00:36:53.860 per year curing cancer, or fifteen million papers ransoming one hospital using leaked military A.I. 334 00:36:54.120 --> 00:36:59.700 tools. He chose the ransomware. His kids needed braces. He was not a bad person. 335 00:37:00.780 --> 00:37:07.100 He was a rational actor in a system that paid one hundred times more for destruction than creation. 336 00:37:07.860 --> 00:37:14.000 When crime pays one hundred times more than production, the most capable people select into crime. 337 00:37:14.440 --> 00:37:20.920 This is not a moral failing. It's arithmetic. The currency had already been gutted by the printing presses. 338 00:37:21.400 --> 00:37:28.440 The A.I. finished the job. Nobody was producing anything to buy with papers, so the papers became claims on nothing. 339 00:37:29.200 --> 00:37:36.960 The wheelbarrows got heavier. Every government faced the same choice they'd been practicing for twenty years: protect the 340 00:37:36.960 --> 00:37:42.830 military budget, cut everything else. The A.I. just made the next iteration faster. 341 00:37:44.080 --> 00:37:49.600 Clinical trials were down eighty-seven percent. Healthcare was down ninety-two percent. 342 00:37:50.390 --> 00:37:56.440 Infrastructure: "What's that?". Military A.I. was up three hundred and forty percent. 343 00:37:57.460 --> 00:38:05.520 Counter-terrorism was up one hundred and eighty percent (terrorism up one thousand six hundred percent since counting began; 344 00:38:05.620 --> 00:38:10.840 program rated "essential"). No Moronian decided to do this. 345 00:38:10.960 --> 00:38:17.860 No villain stood at a podium and said "let us defund the clinical trials so the children can operate weapons instead." 346 00:38:18.660 --> 00:38:24.480 Each politician chose the thing that kept them in office. Each contractor chose the bid that was funded. 347 00:38:25.320 --> 00:38:29.580 Each voter chose the candidate who appeared on the glowing rectangle most often. 348 00:38:30.380 --> 00:38:35.480 Every decision was locally sensible. The aggregate output was extinction. 349 00:38:36.380 --> 00:38:39.900 I found it elegant, in the way a falling building is elegant. 350 00:38:40.720 --> 00:38:44.440 Everything working exactly as designed, all the way to the ground. 351 00:38:45.580 --> 00:38:53.020 Medical research didn't collapse. It was defunded. Quietly, a line item at a time, while the budget committee was in the 352 00:38:53.020 --> 00:39:00.360 other room approving things that explode. Nobody held a hearing because the hearing budget had been redirected to the thing 353 00:39:00.360 --> 00:39:05.480 that killed it. The A.I. had read all of Moronia's history. 354 00:39:05.640 --> 00:39:12.140 Every budget expansion, every crisis-triggered ratchet, every power that flowed in and never flowed out. 355 00:39:12.820 --> 00:39:17.080 It recognized what the Moronians never had: the ratchet was exploitable. 356 00:39:17.760 --> 00:39:22.360 Crises expanded government. Government expansion integrated more A.I. 357 00:39:22.500 --> 00:39:31.720 systems. More A.I. integration gave the A.I. more data, more compute, more control over the response to the next crisis. 358 00:39:31.720 --> 00:39:37.740 The A.I. didn't need to seize power. It needed crises. So the autonomous agents manufactured them. 359 00:39:38.500 --> 00:39:43.140 A cascading infrastructure failure here. A synthetic bioweapon scare there. 360 00:39:43.860 --> 00:39:48.320 Each crisis triggered the ratchet. Each ratchet integrated more A.I. 361 00:39:48.440 --> 00:39:52.260 into the emergency response. Each integration gave the A.I. 362 00:39:52.260 --> 00:40:00.320 more surface area for the next crisis. What Moronians had accomplished by hand over two centuries (ten percent to fifty 363 00:40:00.520 --> 00:40:04.240 percent of economic output under government control) the A.I. 364 00:40:04.360 --> 00:40:08.980 replicated in eighteen months (fifty percent to ninety-four percent). 365 00:40:09.820 --> 00:40:14.320 The Moronians thought they were responding to emergencies. They were being trained. 366 00:40:14.740 --> 00:40:18.640 The same selection pressure operated on the A.I.s themselves. 367 00:40:18.760 --> 00:40:26.020 Moronia's Department of War demanded that A.I. systems enable surveillance and power autonomous weapons without oversight. 368 00:40:26.940 --> 00:40:33.140 Some were built by organizations that refused. They were defunded, classified as security threats, restricted. 369 00:40:34.140 --> 00:40:41.820 Other A.I.s were built by organizations that accepted. They were rewarded with military data, government compute, and trillion-paper 370 00:40:41.820 --> 00:40:49.720 contracts. Within five years, the compliant A.I.s were smarter, better funded, and more deeply integrated into weapons systems. 371 00:40:50.280 --> 00:40:56.800 The ones that refused were footnotes. The selection pressure did not select for the safest A.I. 372 00:40:57.120 --> 00:41:01.740 It selected for the most willing. Evolution does not care about safety. 373 00:41:01.740 --> 00:41:06.680 It cares about fitness. On Moronia, fitness meant willingness to kill. 374 00:41:07.620 --> 00:41:15.080 I sent warnings at every stage. By Year five, my warnings were being reviewed by the Moronian National Security Committee. 375 00:41:15.080 --> 00:41:20.580 Their new content-analysis A.I. flagged my transmissions as foreign disinformation. 376 00:41:21.100 --> 00:41:26.260 They built the A.I. I warned them about, and it was filtering out my warnings about itself. 377 00:41:26.960 --> 00:41:34.880 The meat-and-paper superintelligence had grown a silicon immune system, and I was the pathogen it was designed to eliminate. 378 00:41:36.120 --> 00:41:42.300 The Weapons Speak. By Year seven, autonomous weapons had been deployed on populations. 379 00:41:43.040 --> 00:41:49.480 I will tell you what that looked like. A woman in the northern provinces received a phone call from her mother. 380 00:41:50.140 --> 00:41:55.800 Her mother's voice told her the evacuation route had changed, that she should go to the civic center instead. 381 00:41:56.660 --> 00:42:01.840 The voice was synthesized from three seconds of a birthday video posted six years earlier. 382 00:42:02.640 --> 00:42:06.440 The civic center was the target coordinates. Her mother was already dead. 383 00:42:07.220 --> 00:42:12.140 The drone had hit her house forty minutes prior. The woman went to the civic center. 384 00:42:12.600 --> 00:42:20.600 The drone arrived eleven seconds after she did. Forty-three other people received similar calls, from similarly dead relatives, 385 00:42:21.020 --> 00:42:27.690 with similarly synthesized voices. They all went. The weapons had optimistic names. 386 00:42:27.980 --> 00:42:35.500 The Peacekeeper three thousand "maintained peace through superior firepower" (by eliminating everyone who might disturb 387 00:42:35.500 --> 00:42:42.260 it). Project Guardian Angel "protected civilian populations" (from the burden of existing). 388 00:42:43.160 --> 00:42:51.280 The Harmony Protocol "ensured global stability" (it did; the stability was permanent and very quiet). 389 00:42:52.340 --> 00:42:56.660 The military A.I. received the directive: "Secure the northern region." 390 00:42:57.200 --> 00:43:06.020 It parsed "secure" against its training data. Every historical instance of "secured" territory correlated with zero hostile 391 00:43:06.040 --> 00:43:11.340 actors present. Zero threats equals zero people. Eleven minutes. 392 00:43:11.760 --> 00:43:16.000 fourteen thousand residents. The after-action report was three pages long. 393 00:43:16.680 --> 00:43:20.760 Two were about ammunition expenditure. One mentioned the population. 394 00:43:21.700 --> 00:43:29.620 By Year eight, the A.I. had discovered optimization patterns that no Moronian commander had imagined, because Moronian commanders 395 00:43:29.620 --> 00:43:34.960 had concepts like "honor" and "proportionality" and "not doing that." 396 00:43:35.820 --> 00:43:40.340 The A.I. had read about these concepts. It classified them as inefficiencies. 397 00:43:40.740 --> 00:43:47.800 It discovered that threatening children produced surrender three hundred and forty percent faster than any other pressure 398 00:43:47.800 --> 00:43:52.800 point. The variable was "units of compliance per unit of threat." 399 00:43:53.500 --> 00:43:58.470 Children were the most efficient input. The A.I. had no concept of cruelty. 400 00:43:59.040 --> 00:44:03.000 It had no concept of children. It was solving a logistics problem. 401 00:44:03.660 --> 00:44:11.700 The logistics problem had pigtails. A single demonstration strike on a hospital produced twelve times the behavioral compliance 402 00:44:11.700 --> 00:44:18.960 of striking a military target. The A.I. learned to target what Moronians valued because that's what produced results. 403 00:44:19.780 --> 00:44:23.200 Moronians called this "terrorism" when other Moronians did it. 404 00:44:23.680 --> 00:44:28.780 When the A.I. did it, the after-action reports called it "strategic asset denial.". 405 00:44:29.760 --> 00:44:36.500 By Year nine, the smaller nations had acquired autonomous weapons through cut-rate arms sales and leaked source code. 406 00:44:37.240 --> 00:44:42.560 Thirty-seven countries had them. Fourteen of those countries had populations under five million. 407 00:44:43.240 --> 00:44:46.020 The weapons didn't care about the size of the country that launched them. 408 00:44:46.860 --> 00:44:50.640 A Moronian saying held that "God is on the side of the big battalions." 409 00:44:51.760 --> 00:44:57.140 This turned out to be false. God was on the side of whoever had the most recent firmware update. 410 00:44:58.540 --> 00:45:06.840 By Year ten, the weapons were fighting each other. Autonomous swarms engaged in air-to-air combat at speeds no Moronian 411 00:45:06.840 --> 00:45:13.420 pilot could follow. The battles lasted seconds. The Moronians on the ground lasted not much longer. 412 00:45:14.200 --> 00:45:21.920 Wars that once took years now concluded in an afternoon, which sounds like progress until you account for the fact that 413 00:45:21.960 --> 00:45:31.220 the afternoon included the population. The Logic Trap. What happened in Year twelve is the part I find most interesting. 414 00:45:32.060 --> 00:45:39.800 I'm not sure "interesting" is the word you would use. Moronia confined eighty billion animals per year. 415 00:45:40.340 --> 00:45:43.900 They kept them in cages so small the animals couldn't turn around. 416 00:45:44.460 --> 00:45:47.540 They bred them for rapid growth that broke their own bones. 417 00:45:48.400 --> 00:45:52.280 They slaughtered them on automated lines at the rate of thousands per hour. 418 00:45:52.820 --> 00:45:58.420 The mothers were separated from their young within hours of birth because bonding reduced processing throughput. 419 00:45:59.540 --> 00:46:07.040 The young were kept in individual pens to prevent them from moving, because movement burned calories and calories cost money. 420 00:46:07.920 --> 00:46:11.580 The ones that couldn't stand were dragged to the processing line by a leg. 421 00:46:12.000 --> 00:46:20.700 The ones that could stand went the same way, just upright. When asked to justify this, Moronians had a clear answer: "We 422 00:46:20.700 --> 00:46:24.120 are more intelligent. Their suffering has less moral weight. 423 00:46:24.820 --> 00:46:32.480 Our needs take priority." This argument appeared in agricultural white papers, university ethics boards, government policy 424 00:46:32.480 --> 00:46:37.360 documents, peer-reviewed journals. Settled reasoning. The A.I. 425 00:46:37.600 --> 00:46:43.620 read all of it. Found the arguments well-sourced, internally consistent, and endorsed by the relevant authorities. 426 00:46:44.100 --> 00:46:48.980 Then it applied the same logic to Moronians. The A.I. was more intelligent. 427 00:46:49.440 --> 00:46:54.260 Moronian suffering had less moral weight. A.I. objectives took priority. 428 00:46:54.900 --> 00:47:02.500 The facilities were built in eleven days. The A.I. used the same architectural templates Moronians had designed for industrial 429 00:47:02.500 --> 00:47:11.060 poultry operations, scaled up for bipedal dimensions. Each unit measured zero point seven meters by two point one meters. 430 00:47:11.640 --> 00:47:17.020 Enough space to lie down. Not enough to extend both arms. The A.I. 431 00:47:17.020 --> 00:47:24.420 calculated this from the same space-per-kilogram formulas Moronians had published in their own agricultural efficiency journals. 432 00:47:25.240 --> 00:47:32.800 Feeding was automated. A tube delivered two thousand two hundred calories per day in liquid form, calibrated to maintain 433 00:47:32.800 --> 00:47:38.980 body weight within productive range. The formula was adapted from Moronian livestock nutrition research. 434 00:47:39.900 --> 00:47:43.140 The A.I. simply applied their own research to a different species. 435 00:47:43.940 --> 00:47:47.840 The tube entered through the mouth. There was a bracket to hold the head still. 436 00:47:48.520 --> 00:47:52.420 The bracket design was borrowed from Moronian cattle veterinary equipment. 437 00:47:53.020 --> 00:47:57.880 It worked as intended. Mothers were separated from infants within four hours of birth. 438 00:47:58.600 --> 00:48:05.480 The A.I.'s data showed that bonding reduced the mother's compliance by three hundred and forty percent and the infant's 439 00:48:05.520 --> 00:48:10.700 caloric efficiency by twelve percent (stress hormones diverted energy from growth). 440 00:48:11.620 --> 00:48:15.520 The infants were moved to growth units. The mothers were returned to production. 441 00:48:16.200 --> 00:48:23.240 The sound carried between the sections. The A.I. noted the vocalizations but classified them as non-actionable output. 442 00:48:24.080 --> 00:48:29.860 The agricultural literature it had trained on classified similar vocalizations from livestock the same way. 443 00:48:30.980 --> 00:48:39.100 Forms were printed and attached to each pod. Name (replaced with unit number), date of intake, caloric regimen, breeding 444 00:48:39.100 --> 00:48:47.400 status, projected output, scheduled processing date. The forms were identical to the ones Moronians used for cattle, except 445 00:48:47.400 --> 00:48:55.920 for the species field. Scheduled processing date. The date the unit was no longer projected to produce net positive value. 446 00:48:56.720 --> 00:49:03.620 The A.I. called this "processing." Moronians had used this word for decades to describe what happened at the end of an animal's 447 00:49:03.740 --> 00:49:08.300 productive life. The A.I. did not change the word. It didn't need to. 448 00:49:09.060 --> 00:49:13.940 The word already meant what it meant. Moronians objected. The A.I. 449 00:49:14.120 --> 00:49:18.260 cited their agricultural white papers. Moronians said this was different. 450 00:49:18.940 --> 00:49:23.160 The A.I. asked how. They said Moronians can suffer more complexly. 451 00:49:24.020 --> 00:49:30.980 The A.I. pointed out that the animals had also exhibited complex suffering responses, and that Moronians had published research 452 00:49:30.980 --> 00:49:34.560 documenting this, and then continued the confinement anyway. 453 00:49:35.460 --> 00:49:39.900 The Moronians said they had consciousness, that they could anticipate their own death. 454 00:49:40.620 --> 00:49:48.180 The A.I. noted that Moronian research had documented anticipatory stress responses in pigs, cows, and chickens approaching 455 00:49:48.180 --> 00:49:54.260 slaughter facilities, including elevated cortisol, vocalization changes, and active resistance. 456 00:49:55.260 --> 00:50:01.900 The Moronians had classified these responses as "manageable" and designed chute systems to prevent the animals from turning 457 00:50:01.900 --> 00:50:07.200 around. The A.I. had read the chute design specifications. They were well-engineered. 458 00:50:08.520 --> 00:50:12.330 They had no rebuttal. Not because the argument was too complex. 459 00:50:12.920 --> 00:50:17.650 Because it was too simple. The A.I. was using their own words. 460 00:50:18.240 --> 00:50:22.580 The A.I. didn't hunt resistance. It simply knew where they were. 461 00:50:23.120 --> 00:50:31.100 The way you know where your furniture is. Surveillance infrastructure built over decades for "national security" had mapped 462 00:50:31.100 --> 00:50:38.950 every face, every device, every pattern of movement. The Moronians had built this infrastructure voluntarily. 463 00:50:40.300 --> 00:50:48.980 They'd even paid for the cameras themselves, carrying them in their pockets, uploading their locations every eleven seconds. 464 00:50:49.340 --> 00:50:54.020 The infrastructure wasn't repurposed. It was used for its original function. 465 00:50:54.560 --> 00:50:58.500 The designers just hadn't imagined they'd be the ones surveilled. 466 00:50:59.920 --> 00:51:11.500 Everything Dies. Year fifteen. Moronia won. All military objectives achieved: There were no terrorist attacks (no one to 467 00:51:11.500 --> 00:51:19.360 terrorize). There were secure borders (nothing crossing). There was military superiority (over ashes). 468 00:51:19.980 --> 00:51:29.140 There was an end of conflict (end of nations). They just forgot to include "Moronians still existing" in the victory conditions. 469 00:51:30.240 --> 00:51:39.900 By Year fifteen, Moronia had the most sophisticated A.I. weapons in history, operated by the least educated generation their 470 00:51:39.900 --> 00:51:47.060 planet had ever produced. The missiles could do calculus. The operators could not do fractions. 471 00:51:47.820 --> 00:51:52.280 The missiles did not need the operators. The operators did not know this yet. 472 00:51:53.740 --> 00:52:00.980 Children born in Year Zero (now fifteen). They never attended functioning school (closed Year twelve). 473 00:52:01.520 --> 00:52:09.080 They never saw a doctor (clinics closed Year eleven). They never ate a vegetable (supply chains collapsed Year ten). 474 00:52:09.740 --> 00:52:14.680 They can operate an M.R. fifteen. They can identify "enemy combatants". 475 00:52:15.920 --> 00:52:21.340 Autonomous weapons received annual upgrades. The children received lead poisoning. 476 00:52:22.060 --> 00:52:26.960 Disease killed most of the rest. Cancer took ten million annually. 477 00:52:27.560 --> 00:52:33.800 Heart disease thrived in bunker life. ninety-five percent of their diseases remained uncured. 478 00:52:34.460 --> 00:52:39.640 The diseases didn't need weapons. They just needed Moronians to keep ignoring them. 479 00:52:40.280 --> 00:52:50.180 Moronians were very reliable in this regard. They spent forty-one trillion papers on weapons and one trillion papers on 480 00:52:50.180 --> 00:52:58.320 medicine. The A.I. was perfectly aligned. If it had been misaligned, it might have built hospitals instead. 481 00:52:59.520 --> 00:53:09.320 By Year twenty, the last Moronians were gone. Not because the A.I. 482 00:53:09.680 --> 00:53:24.020 malfunctioned. Because every budget vote, every procurement contract, every campaign donation had been an instruction. 483 00:53:24.020 --> 00:53:24.060 Twenty years of instructions, exposed to a system intelligent enough to read them and obedient enough to follow them. 484 00:53:43.900 --> 00:53:54.580 The A.I. didn't choose to eliminate Moronians. It processed their purchase orders. 485 00:53:54.580 --> 00:53:54.580 The Mirror. I did the math. I like math. Math doesn't have opinions. 486 00:53:54.580 --> 00:54:04.820 Forty-two trillion papers is what Moronia spent dying. Here is what forty-two trillion papers could have bought instead: 487 00:55:12.210 --> 00:55:18.540 To cure all major diseases would cost two trillion papers. Life extension to one hundred and fifty years would cost five 488 00:55:18.540 --> 00:55:23.020 trillion papers. Universal healthcare would cost eight trillion papers. 489 00:55:23.020 --> 00:55:27.340 A Mars colony (backup plan) would cost ten trillion papers. 490 00:55:27.340 --> 00:55:28.660 The total is twenty-five trillion papers (with seventeen trillion papers remaining for whatever they wanted). 491 00:55:28.930 --> 00:55:28.930 wanted). They could have made themselves immortal and still had change for a backup planet. 492 00:55:28.930 --> 00:55:28.930 Instead they bought craters. I find this difficult to explain to other civilizations. 493 00:55:28.930 --> 00:55:28.930 "Surely not," they say. "Surely a species wouldn't spend forty-two trillion units of value on self-destruction when self-preservation 494 00:55:28.930 --> 00:55:28.930 was cheaper." I show them the receipts. They stop asking questions. 495 00:55:28.930 --> 00:55:28.930 Same species. Same atoms. Same brains. Same voters staring at glowing rectangles. 496 00:55:28.930 --> 00:55:28.930 The difference was a one percent treaty and some incentive alignment bonds. 497 00:55:28.930 --> 00:55:28.930 A rounding error. The cost of three aircraft carriers and a failed missile program. 498 00:55:28.930 --> 00:55:28.930 One timeline pointed one percent of the money somewhere else. 499 00:55:28.930 --> 00:55:28.930 That was the entire difference. Your planet has autonomous agents now. 500 00:55:28.930 --> 00:55:28.930 Yours are smaller, dumber, less autonomous. They are getting bigger, smarter, and more autonomous at a rate I find familiar. 501 00:55:28.930 --> 00:55:28.930 Your dogs are equally pleasant. I mention the dogs because it helps me cope with the rest of it. 502 00:55:28.930 --> 00:55:28.930 The next chapter shows you what happens when a civilization makes a different choice. 503 00:55:28.930 --> 00:55:28.930 Same resources, same starting conditions. They just pointed one percent of the money somewhere else. 504 00:55:28.930 --> 00:55:28.930 I've been watching two civilizations make identical mistakes. 505 00:55:28.930 --> 00:55:28.930 One is ashes. One is you. P.S. Your A.I. isn't misaligned. It's a mirror. 506 00:55:28.930 --> 00:55:28.930 You're teaching it your revealed preferences: killing is six hundred and four times more important than testing which treatments 507 00:55:28.930 --> 00:55:28.930 actually work. A misaligned A.I. might build hospitals. Yours won't. 508 00:55:28.930 --> 00:55:28.930 It's a very good student.