Posted on 2021-02-03
Imagine that advanced life like us is terribly rare in the universe. So damn rare that if we had not shown up, then our region of the universe would almost surely have forever remained dead, for eons and eons. In this case, we should still be able to predict when we humans showed up, which happens to be now at 13.8 billion years after the universe began. Because we showed up on a planet near a star, and we know the rate at which our universe has and will make stars, how long those stars will last, and which stars where lived far enough away from frequent sterilizing explosions to have at least a chance at birthing advanced life.
However, this chart (taken from our new paper) calculates the percentile rank of our current date within this larger distribution. And it finds that we are surprisingly early, unless you assume both that there are very few hard steps in the evolution of advanced life (the “power n”), and also that the cutoff in lifetime above which planets simply cannot birth advanced life is very low. While most stars have much longer lives, none of those have any chance whatsoever to birth advanced life. (The x-axis shown extends from Earth’s lifetime up to the max known star lifetime.)
In the paper (in figures 2,17), we also show how this percentile varies with three other parameters: the timescale on which star formation decays, the peak date for habitable star formation, and a “mass favoring power” which says bu how much more are larger mass stars favored in habitability. We find that these parameters mostly make only modest differences; the key puzzle of humans earliness remains.
Yes, whether a planet gives rise to advanced life might depend on a great many other parameters not included in our calculations. But as we are only trying to estimate the date of arrival, not many other details, we only need to include factors that correlate greatly with arrival date.
Why have others not reported the puzzle previously? Because they neglected to include the key hard-steps power law effect in how chances vary with time. This effect is not at all controversial, though it often seems counter-intuitive to those who have not worked through its derivation (and who are unwilling to accept a well-established literature they have not worked out for themselves).
This key fact that humans look early is one that seems best explained by a grabby aliens model. If grabby aliens come and take all the volume, that sets a deadline for when we could arrive, if we were to have a chance of becoming grabby. We are not early relative to that deadline.
Posted on 2022-10-24
Grabby aliens are advanced civs who change the stuff they touch in big visible ways, and who keep expanding fast until they meet each other. Our recent analysis suggests that they appear at random stars roughly once per million galaxies, and then expand at roughly half the speed of light. Right now, they have filled roughly half of the universe, and if we join them we’ll meet them in roughly a billion years. There may be far more quiet than grabby alien civs out there, but those don’t usually do much or last long, and even the ruins of the nearest are quite far away.
While I’ve so far avoided thinking about much war within this scenario, I’ve decided to go there now. So here we go.
First, consider quiet alien wars. Such quiet civs may have internal wars, but different civs rarely get close enough to each other for physical fights. Maybe more advanced ones would sometimes conquer less advanced ones via malicious messages, but I’m skeptical that such events are common. The rare civs who expanded long and quietly mainly to preserve a natural universe and prevent grabby origins within their sphere of control should share goals and thus have little reason to war when they meet. Furthermore, when grabby civs meet quiet ones, abilities would be terribly unequal, and so not much of an occasion for war.
What about grabby civs? After a few million years they’d probably reach near max possible tech abilities. Which I guess makes them pretty immune to malicious messages. But such civs and their parts might vary in how well they had used a shared origin to promote internal cooperation. And a lack of perfect cooperation would likely result in some internal wars. The higher the rate at which they spend a fraction of their fast-access resources to fight or prevent fights, the faster they’d use up such resources. As a result, such fast spending civs might only get resources for a long time if some of their resource sources, like black holes, only allowed slow extraction.
Long-distance ballistic directed energy weapons, which couldn’t be redirected along the way, would only be of use on targets whose locations could be predicted long enough in advance. As a result, grabby cis would usually ensure that the locations of important resources vulnerable to such attacks could not be so predicted. Similarly, they’d end or stay away from objects like stars that might be induced to explode by outside prods. Thus militarily-useful resources would likely need to maintain unpredictable locations and would need to be located quite close to where they’d be used. So conflicts would tend to be won locally by those with more military resources locally available near the point of conflict.
If grabby civs are not more able to or inclined to cooperate internally than with other civs, then each small part of such a civ should be similarly wary of neighboring advanced life, regardless of its civ of origin. In which case, the boundary at which different grabby civs meet might not have that much significance. Who wins each local conflict would mainly depend on their relative size, resources, level of internal cooperation, and local geography, but not civ of origin. On 100Mlyr and larger scales, this should add up to a pretty uniform picture.
However, what if at least some parts of some grabby cigs could use their shared origin to cooperate more strongly internally than they could with other grabby civs? In this case, they’d expect more conflict at the border where grabby civs meet, compared to at other locations. As a result, the cooperating units on both sides might then try to send resources to that border, in anticipation of such conflicts. And then a key question arises: just how fast is it feasible to move militarily useful resources?
Grabby civs expanding at half the speed might seem surprisingly fast, but this does seem roughly feasible given that they can afford to spend huge resources on speeding tiny seeds that can then use local resources to quickly grow exponentially into huge civs. Alas, no similar exponential strategy seems available to move resources from one place to another. If the resources required to accelerate resources to near the speed of light can be efficiently recaptured at a designation location, then perhaps resources could in fact be efficiently sent very far very fast. But otherwise, sending resources far fast (e.g., >2% of c) may only be possible at crazy high costs.
At the border between two grabby civs, imagine that one of the civs had better managed to tax internal regions to send more resources to that border from within that civ, and at a very rapid speed. In this case, then after a while the resources accumulated on one side of that border might be far larger than that on the other side. Then if the natural advantage of defense over offense were not too large, the stronger side might be able to initiate a war and take territory from the other side. And in fact this outcome might become so obvious that the losing side would be very sure to lose, and not even want to fight.
If merely threatening to attack with overwhelming force was usually sufficient to quickly rout the weaker side and win new territory, via induced surrender or flight, or if actual fights did not take too long or destroy too much of an attacker’s resources, then an attacker might continue to move forward into the other side’s territory at a rapid pace. And if that pace were on the order of 2% of the speed of light, that might be sufficient to completely take over all the territory of a neighboring grabby civ within the roughly hundred billion years remaining before the time when, it is now estimated, dark energy makes galaxy clusters disconnected, never more able to see or reach each other. Such attack threats might then be seen as existential risks to such a civ.
Putting this all together seems to me to create a nightmare scenario, one which might greatly worry many young grabby civs who take very long term views. And, importantly, they’d have to decide how scared to be of this scenario long before they had much info on each particular neighboring civ, or even on any other civs besides themselves. Thus fear of the unknown might push many such civs into paying huge costs to maintain strong governance able to heavily tax internal activity to fund the movement of large amounts of resources out to be ready for unknown future border conflicts. Resources which might be mostly wasted if two such well-prepared civs were to meet.
Thus the possibilities of (A) long term civ-level views, (B) cheap fast movement of military resources which were hard to convert back to civilian use, (C) a sufficiently low advantage of defense over offense, (D) within-civ governance strong enough to tax and transfer resources to the border, and (E) weak enough governance unable to prevent your side from fleeing or surrendering given overwhelming attackers, all of this together might induce the waste of much, or perhaps even the vast majority of, available resources. Resources that could instead be used to compute far more meaningful peaceful lives near where the required resources sat originally.
Also note that at the line-shaped borders where three grabby civs meet, all three might have equal resources. Even so, two of them allying against the third would gain an advantage. And if this were sufficient, they might together advanced into the third region, sharing the gains. After which, each of them might have a geometric advantage, partially encircling the other side where their border bends. The possibility of this ally advantage should induce grabby civs to try to seem more similar to each other, to induce others to ally with them.
Posted on 2021-03-08
Our descendants will have far more effects on the universe if they become grabby, and most of their expected effects come in that scenario. Even so, as I discussed in my last post, most see only a small chance for that scenario. So what if we remain a non-grabby civilization? What will be our long-term legacies then? In roughly a billion years, grabby aliens should pass by here, and then soon change this whole area more to their liking. At that point, those grabby aliens will probably have never met any other grabby aliens, and will be very interested in estimating what they might be like, and especially what they might do when the two meet. And one of their main sources of concrete data will be the limited number of non-grabby alien civilizations that they have come across.
Which is all to say that these grabby aliens will be very interested in learning about us, and should be willing to pay substantial costs to do so. So in the unlikely event that our civilization could last the roughly billion years until they get here, those aliens would probably pay substantial costs to protect and preserve us, if that were the cost of learning about us. Of course if they had more advanced tech, they might have other less-fun-for-us ways to achieve that goal.
In the more likely case where we do not last that long, the grabby aliens who arrive here will be looking for any fossils or remnants that they could study. Stuff left here on the surface of the Earth probably won’t survive that long, but stuff left on the surface of geologically dead places like the moon or Mars might well. As could stuff left orbiting between the planets or stars.
Anticipating this outcome, some of us might try to leave data stores about us for them to find. Like we did on the Voyager spacecraft. As our long term legacy. And some of those folks might try to tie their personal revival to such stores. I’m not sure how it could be done, but if you could mix up the info they want with the info that specifies you as an em, maybe you could make it so that the easiest way for them to get the info they want is to revive you.
Of course if a great many people tried this trick, they might bid the “price” down very low. “They want you to revive them for a week to get your info; I only ask one day.” So elites might regulate who is allowed to leave legacy data stores, to keep this privilege to themselves.
Long before grabby aliens got here, they would pass through spacetime events where we’d be active on their past light cone. In fact, sending out a signal from here in most any direction should eventually hit some grabby aliens expanding in our direction. So if we could coordinate with them to send signals out just when they’d be looking at us (such as by sending signals following those from a cosmic explosion), we could tell them about us, and influence them, via such signals.
Some of us might want to try the trick of mixing up their em code with the info aliens want, to force their revival at the receiver end, but the bandwidth to send signals to be received in a 100Myr is rather small. However, as I’ve discussed before, one key function for such signals is that they can prove that they were sent on the date claimed. Later data stores found here are less trustworthy, as they could have been modified in the interim. So perhaps we could send out hash codes to verify datastores saved here now.
We could of course also tell them about any other non-grabby aliens we have discovered. But they’d probably already know about them, assuming they have vastly greater capabilities and tech at least as good as ours.
So is this an exciting legacy to you? A few stories about us that might help some other ambitious civilization calibrate how yet other ambitious civilizations will react upon meeting? No, well then maybe we should work on figuring out how to become grabby ourselves.
Added 4Nov: I missed a big potential legacy: Non-grabby aliens could help to mediate between and coordinate grabby aliens. Before two grabby aliens civs meet, they may have both seen and received messages from dozens of the same non-grabby civilizations. Messages sent by those mediators might set expectations and reference points that help the grabby aliens to coordinate. They might even distribute entangled qubits.
Such messages would be more credible if they embodied costly signals. So what could a non-grabby alien civ do, and show they did, to convince grabby aliens re their expectations of what will happen when grabby aliens meet?
Posted on 2021-01-08
In two posts, I recently explained how a simple 3 parameter model of grabby aliens can explain our apparent early arrival in the universe, via a selection effect: we might give rise to a grabby civ, but that had to happen before other grabby civs took over all the volume. With some collaborators, I’ve been exploring computer sims of this model, and found one striking statistic: at the origin time of a grabby civ, on average ~40% of universe volume is controlled by grabby aliens. A stat which seems obviously contradicted by what we see, namely nothing. In the volumes we see, they can’t be controlling much, at least if control would make it look much different. What gives?
In this post I want to show how this apparent emptiness can be explained by a parameter choice and a selection effect. First, let’s get oriented. Here is a spacetime diagram showing us now, and all the events that we can see from here, as our red backward light-cone.
Next, consider the fact that if we extend a yellow cone back in time from where we are at the grabby civ expansion speed, no grabby civ could have had their origin in that excluded volume, because if so then they would have prevented us, to prevent us from becoming grabby.
Because that’s the definition of grabby: they expand and prevent the origin of other grabby civs within the volumes they control. We could only see grabby civs who have their origin in the green volume, as their expansion would not have reached us yet.
Now if the expansion speed were small, that green area would encompass most of the volume in our past light-cone, and we’d still have a puzzle: why don’t we see them? But as their expansion speed approaches the speed of light, the green volume gets small, making for a low chance of seeing any grabby aliens. (The chance of not seeing one goes as roughly the fraction of their expansion speed to the speed of light.)
Now let’s look at one of those grabby civs we could see:
Since its origin is in the green volume, its forward expanding cone of control (in orange) intersects our backward light-cone. At the closest intersection point, the spatial extent of that civ is given by the horizontal purple line, which is large compared to its distance away. (Imagine space were 2D, fixing one end of the purple line at the origin axis, and rotating the other end out of the diagram.) So it would be absolutely huge in the sky. This diagram also shows our forward expansion cone intersecting its forward cone relatively soon in the future; we meet them soon.
Now look at the vertical purple line in this next diagram. Holding constant the spatial location of this alien origin, consider the other possible times at which this civ could have originated at that location and still be visible to us.
The higher is that origin point in the diagram, and the closer is that origin to our red backward light cone, then the smaller is that vertical purple line. And since geometrically the two purple lines must move in proportion, the smaller of an appearance that civ would make in the sky.
As civ origin times should be roughly uniformly distributed over that vertical range, there is thus only a tiny chance of seeing aliens that take up a tiny fraction of our sky. Either we see them huge, or not at all. So there’s little point in building bigger SETI telescopes or deeper surveys to try to see very tiny grabby aliens very far away.
Thus our grabby aliens model can use selection effects to explain not only why we have appeared so early in the history of the universe, but also why we don’t see them even though they should on average take up (and modify) ~40% of universe volume at the moment. At least if we postulate that their expansion speed is a substantial fraction of the speed of light. Which we already had reason to believe, just based on the idea that “grabby” civs try to grab as fast as they can.
Added 7Mar: Here is the likelihood ratio for seeing our data of no big alien volumes in the sky, as a function of power n and speed s/c:
Posted on 2015-04-19
Charles Stross recently on possible future great filters:
So IO9 ran a piece by George Dvorsky on ways we could wreck the solar system. And then Anders Sandberg responded in depth on the subject of existential risks, asking what conceivable threats have big enough spatial reach to threaten an interplanetary or star-faring civilization. … The implication of an [future great filter] is that it doesn’t specifically work against life, it works against interplanetary colonization. … much as Kessler syndrome could effectively block all access to low Earth orbit as a side-effect of carelessly launching too much space junk. Here are some example scenarios: …
Simplistic warfare: … Today’s boringly old-hat chemical rockets, even in the absence of nuclear warheads, are formidably destructive weapons. … War, or other resource conflicts, within a polity capable of rapid interplanetary or even slow interstellar flight, is a horrible prospect.
Irreducible complexity: I take issue with one of Anders’ assumptions, which is that a multi-planet civilization is … not just … distributed, but it will almost by necessity have fairly self-sufficient habitats that could act as seeds for a new civilization if they survive. … I doubt that we could make a self-sufficient habitat that was capable of maintaining its infrastructure and perpetuating and refreshing its human culture with a population any smaller than high-single-digit millions. … Building robust self-sufficient off-world habitats … is vastly more expensive than building an off-world outpost and shipping rations there, as we do with Antarctica. …
Griefers: … All it takes is one civilization of alien ass-hat griefers who send out just one Von Neumann Probe programmed to replicate, build N-D lasers, and zap any planet showing signs of technological civilization, and the result is a galaxy sterile of interplanetary civilizations until the end of the stelliferous era. (more)
These are indeed scenarios of concern. But I find it hard to see how, by themselves, they could add up to a big future filter.
On griefers (aka “berserkers”), a griefer equilibrium seems to me unstable to their trying sometimes to switch to rapid growth within a sufficiently large volume that they seem to control. Sometimes that will fail, but once it succeeds enough then competing griefers have little chance to stop them. Yes there’s a chance the first civilization to make them didn’t think to encode that strategy, but that seems a pretty small filter factor. On simple war, I find it hard to see how war has a substantial chance of killing everyone unless the minimum viable civilization size is large. And I agree that this min size gets bigger for humans in space, who are more fragile there. But it should get smaller for smart robots in space, or on Earth, especially if production becomes more local via nano-factories. The chance that the last big bomb used in a war happens to kill off the last viable group of survivors seems to me relatively small.
Of course none of these chances are low enough to justify complacency. We should explore such scenarios, and work to prevent them. But we should work even harder to find more worrisome scenarios.
So let me explain my nightmare scenario: general non-diminishing prey. Consider the classic post-apocalyptic scenario, such as described in The Road. Desperate starving people ignore the need to save and build for the future, and grab any food they can find, including each other. First all the non-human food is gone, then all the people. Such situations have been modeled formally via “predator-prey dynamics”:
These are differential equations giving the rates at which counts of predators and prey grow or decline as a function of each other. The standard formulation has a key term whereby prey count falls in proportional to the product of the predator count and the prey count. This formulation embodies an important feature of diminishing returns: the fewer prey are left, the harder it is for predators to find and eat them. Without enough such diminishing returns, any excess of predators quickly leads to the extinction of prey, followed quickly by the extinction of predators. For example, when starving humans are given easy access to granaries, such granaries are emptied quickly. Not made low; emptied. Which is why granaries in famines are usually either well-protected, or empty.
In nature, there are usually many kinds of predators, and even more kinds of prey. So the real predator-prey dynamic is high-dimensional. The pairwise relations between most predators and preys do in fact usually involve strongly diminishing returns, both because predators must usually search for prey, and because some prey hiding places are much better than others.
If the relation between any one pair of predator and prey types happens to have no diminishing returns, then that particular type of prey will go extinct whenever there is a big enough excess of that particular type of predator. Since this selects against such prey, the prey we see in nature almost all have diminishing returns for all their practical predators.
Humans are general predators, able to eat a great many kinds of prey. And within human societies humans are also relatively general kinds of prey, since we mostly all use the same kinds of resources. So when humans prey on humans, the human prey can more easily go extinct.
For foragers, a key limit on human predation was simple distance. Foragers lived far apart, and were unpredictably located. Also, foragers had little physical property to grab, wives were not treated as property, and land was too plentiful to be worth grabbing. These limits mattered less for farmers, who did predate often via war.
The usual source of diminishing returns in farmer war predation has been the wide range of protection in places to hide; humans have often run to the mountains, jungle, or sea to escape human predators. Even so, humans and proto-humans have quite often driven one another to local relative extinction.
While the extinction of some kinds of humans relative to others has been common, the extinction of all humans in an area has been much less common. This is in part because, when there has been a local excess of humans, most have focused on non-human prey. Such prey are diverse, and most have strongly diminishing returns to human predation.
Even if humans expand into the solar system, and even if they create robot descendants, we expect our descendants to remain relatively general predators, at least for a long while. We also expect the physical resources that they collect to constitute relatively general prey, useful to a wide range of our descendants. Furthermore, we expect nature that isn’t domesticated or descended from humans to hold a decreasing quantity of useful resources.
Thus the future predator-prey dynamic should become lower dimensional than it has been in the past. To a perhaps useful approximation, there’d be only a few kinds of predators and prey. Which raises the key question: how strong are the diminishing returns to predation in that new world? That is, when some of our descendants hunt down others to grab resources, how fast does that task get harder as fewer prey remain?
One source of diminishing returns in predation is a diversity of approaches and interfaces. The more different are the methods that prey use to create and store value, the smaller the fraction of that value a predator can obtain via a simple hostile takeover. This increases the ratio of how hard prey and prey fight. As many have noted, in nature prey fight for their lives, while predators fight only for a meal. Even so, nature still has plenty of predation. Even if predators gain only part of the value contained in prey, they still predate if that costs them even less than this value.
As I said above, the main source of diminishing returns in predation among foragers was travel cost, and among farmers it was the diversity of physical places to run and hide. Such effects might still protect our descendants from predator-prey-dynamic extinction, even if they have only one kind of predator and prey. Alas, we have good reasons to fear that these factors may less protect our descendants.
The basic problem here is our improving techs for travel, communication, and surveillance. We are steadily able to move bits and people more cheaply, and to more cheaply and accurately watch spaces for activity. Yes moving out into the solar system would put more distance between things, and make them harder to see. But that one-time effect will be quickly overwhelmed by improving tech.
A colonized solar system is plausibly a place where predators can see most any civilized activities of any substantial magnitude, and get to them easily if not quickly. So if we ever reach a point where predators fight to grab civilized resources with little concern to save some for the future, they might be able to find and grab pretty much everything in the solar system. Much as easy-access granaries are quickly emptied in a famine.
Whether extinction results from such a scenario depends how small are minimum viable civilization seeds, how obscure and well protected are the nooks and crannies in which they might hide, and how many of them exist and try to hide. Yes, hidden viable seeds drifting at near light-speed to other stars could prevent extinction, but such a prey-collapse scenario could play out well before such seeds are feasible.
So, bottom line, the future great filter scenario that most concerns me is one where our solar-system-bound descendants have killed most of nature, can’t yet colonize other stars, are general predators and prey of each other, and have fallen into a short-term-predatory-focus equilibrium where predators can easily see and travel to most all prey. Yes there are about a hundred billion comets way out there circling the sun, but even that seems a small enough number for predators to careful map and track all of them. Worry about prey-extinction scenarios like this is a reason I’ve focused on hidden refuges as protection from existential risk. Nick Beckstead has argued against refuges saying:
The most likely ways in which improved refuges could help humanity recover from a global catastrophe are scenarios in which well-stocked refuges with appropriately trained people help civilization to recover after a catastrophe that leaves a substantial portion of humanity alive but disrupts industrial and agricultural infrastructure, and scenarios in which only people in constantly-staffed refuges survive a pandemic purposely engineered to cause human extinction. I would guess that, in the former case, resources and people stocked in refuges would play a relatively small role in helping humanity to recover because they would represent a small share of relevant people and resources. The latter case strikes me as relatively far-fetched and I would guess it would be very challenging to do much better than the largely uncontacted peoples in terms of ensuring the survival of the species. (more)
Nick does at one point seem to point to the scenario that concerns me:
If a refuge is sufficiently isolated and/or secret, it would be easier to ensure that everyone in the refuge had an adequate food supply, even if that meant an inegalitarian food distribution.
But he doesn’t appear to think this relevant for his conclusions. In contrast, I fear that a predatory-collapse scenario is the most likely future great filter, where unequal survival key to preventing extinction.
Added 10a: Of course the concern isn’t just that some parties would have short term orientations, but that most would pursue short-term predation so vigorously that they force most everyone to put in similar effort levels, even if they take have long-term view. When enemies mass on the border, one might have to turn farmers into soldiers to resist them, even if it is harvest time.
Posted on 2018-08-21
One way to predict the future is to find patterns in the past, and extend them into the future. And across the very long term history of everything, the one most robust pattern I see is: growth. Biology, and then humanity, has consistently grown in ability, capacity, and influence. Yes, there have been rare periods of widespread decline, but overall in the long run there has been far more growth than decline.
We have good reasons to expect growth. Most growth is due to innovation, and once learned many innovations are hard to unlearn. Yes there have been some big widespread declines in history, such as the medieval Black Death and the decline of the Roman and Chinese empires at about the same time. But the historians who study the biggest such declines see them as surprisingly large, not surprisingly small. Knowing the details of those events, they would have been quite surprised to see such declines be ten times larger than as seen. Yes it is possible in principle that we’ve been lucky and most planets or species that start out like ours went totally extinct. But if smaller declines are more common than bigger ones, the lack of big but not total declines in our history suggests that the chances of extinction level declines was low.
Yes, we should worry about the possibility of a big future decline soon. Perhaps due to global warming, resource exhaustion, falling fertility, or institutional rot. But this is mainly because the consequences would be so dire, not because such declines are likely. Even declines comparable in magnitude to the largest seen in history do not seem to me remotely sufficient to prevent the revival of long term growth afterward, as they do not prevent continued innovation. Thus while long-term growth is far from inevitable, it seems the most likely scenario to consider. If growth is our most robust expectation for the future, what does that growth suggest or imply? The rest of this post summarizes many such plausible implications. There far more of them than many realize.
Before I list the implications, consider an analogy. Imagine that you lived in a small mountain village, but that a huge city lie down in the valley below. While it might be hard to see or travel to that city, the existence of that city might still change your mountain village life in many important ways. A big future can be like that big city to the village that is our current world. Now for those implications:
The Great Filter – If our descendants continue to grow, some of them should eventually occupy and rearrange much larger volumes of space. It is unlikely that the best way to rearrange that space will look from a distance just like the dead matter that was there before our descendants arrived. So eventually the large volumes we change should look visibly different from far away. Yet when we look out now into the universe seeking aliens who have visibly changed the universe near them, everything we see looks dead.
This suggests that a great filter lies along the evolutionary paths between simple dead matter and an expanding visible civilization, and raises the key question: how far along this filter are we? Evidence of alien life having reached further along such a path would be bad news, suggesting that our past filter is easier than we thought, and thus our future filter is harder. We have other clues to where we are in this filter.
Cryonics – Today when people’s bodies fail them and current medical science fails them as well, we usually let their bodies decay into nothing. For example, we burn them, or let worms eat them. But it is possible to instead freeze those bodies in liquid nitrogen. The freezing process does some damage, as do antifreeze chemicals often used to limit freezing damage. But once frozen in liquid nitrogen, bodies should stay almost exactly the same for many centuries.
And so if our descendants grow in technical capability, eventually they may be able to repair both the freezing damage and whatever went wrong with those bodies before freezing. And if brain emulations are possible, at an earlier date it should be able to create brain emulations from only mildly damaged frozen brains. Thus people today who “die” might be revived in the future, if their brains can be frozen and stored for long enough. You might want to try to be one of those people. Simulation Argument – We today are often interested in the past. Some of us study it formally, as historians, while others explore history via fiction and games. Some of these ways we explore history can be seen of as “simulations”, intended to mimic some details of a historical period via playing it out step by step. This sort of thing has been going on for thousands of years, and we expect that our descendants to continue this tradition.
Our much more capable descendants should be able to create much more detailed simulations. Such simulations could include individual people simulated to such a detail that they are real living people who don’t know that they are in a simulation; they believe that they are actually living in the historical period being simulated. For a historical event of great interest to our descendants, there might be a great many simulations, containing more simulated people than existed during that actual historical event. Which raises the provocative question: what is the chance that each of us is actually living in such a future simulation?
Biology Replaced – Today the capabilities of our industrial economy are powerful and important, but are overshadowed in many ways by the capacities of biology. Biology has designed and can produce vast complex systems than we today only crudely understand. But our industrial abilities are increasing far faster than those of biology, so eventually industry should displace or assimilate biology. Future industry may incorporate particular biological chemicals, reactions, structures, and even systems, but within a more industry-style system. That is, things will be more designed by engineers, tested in labs, made in factories, and then controlled as centrally as is found useful. They’d be made out of materials from organized mines and trash recyclers, fed energy from large-scale energy systems, move around via large-scale transport systems, and communicate via large-scale communication systems.
If so, the familiar self-reproducing self-feeding self-moving autonomous biological organisms that have dominated Earth for billions of years have a limited future. Some of this might be saved in nature/history preserves/museums, but the distant future will be filled with stuff more integrated into an advanced industrial economy. This includes the descendants of humans, also made more in structured factories than in autonomous wombs. In this sense our descendants, and any advanced aliens we meet, should be “robots.”
Who To Influence – You may not be very interested in influencing empty space or even empty deserts, because you mostly care to influence and help people. If so, then note that you should expect to find the vast majority of people, or at least people-like creatures, in the distant future. So as long as the future contains at least moderate variety, then according to a wide range of criteria of which people you’d most like to influence, most of those people will be found in the distant future. (Helping is a kind of influence.)
Yes, if your interest in people declines with cultural distance, and if future people tend to be more culturally distant, then on average future people are less interesting to you. But you don’t have to influence average people, you can select who to influence, and the future will have a lot more people. So most of those that you most want to influence may still be in the distant future. Even if future folks are richer on average (which I doubt), there are likely to be plenty of suffering folks there as well to help.
In addition, for thousands of years the average rates of return on investments have been consistently higher than economic growth rates. Given this, if you save and invest resources to be spent to influence people later, your influence should rise as a fraction of the world economy! Yes investments are often stolen, and you suffer losses when trying to control future agents who are supposed to execute your instructions about who to help. Even so, after these losses it may still be cheaper to influence future folks than folks today. The future will be big and by you might even be able to influence a larger fraction of that future world than today’s world. This makes the future an obvious place to consider when looking to influence others.
Where to Migrate – If you would consider migrating from where you are now to some new place, a big future means that most interesting places to move may be out there in that future. Yes, you might be picky about the kind of place you want to go, and yes all else equal future place might be more cultural distant and hence less attractive. But being bigger the distant future may still contain most of the places that best fit any given migration criteria.
To get to the future you either need to live a long time, to use some sort of suspended animation such may be found in cryonics, or to have change accelerate so that the future gets bigger faster during your lifetime. All of these seem like viable options, and they can be combined. In addition, using rates of return on investment you can bring more resources with you into the future than you left with from our time. And for a big, rich and capable future, many things you want will be cheaper to buy in that future than they are here today. Furthermore, one of those things is the ability t0 migrate to still even more distant futures.
Who To Impress – We like to pretend that we don’t care what other people think, that we just do things to please ourselves. But this isn’t remotely true, and we can’t just will it to be true. But we do have some flexibility in who exactly we want to impress. And when choosing which people to try to impress, most of our available candidates will be in the distant future. Thus most of the potential audiences who meet any particular audience criteria will also be in the distant future.
In addition, not only will future people be wiser and more knowledgable in many ways, and thus better judges to impress, it should be cheaper to get future folks to pay attention to evaluating us. High rates of return on investments means that a small sum today might be enough to pay several future people to spend a lifetime carefully considering our accomplishments and contributions. Yes, one needs to make sure to save the right sort of data to allow future folks to make such judgements. But it seems possible to save a lot of data cheaply, and many scale and scope economies seem available to groups seeking to share related data together.
Incentives For Honesty – Today we often want to rely on the honesty and forthrightness of experts and advisors, but fear that those experts and advisors often have incentives to mislead us. One simple way to encourage honesty is to have experts make bets supporting their claims. Or, similarly, have them post bonds that they lose if proven wrong. Such approaches, however, require that we can sometimes settle such bets. There must at least be a substantial chance that eventually enough people will know the answer. In this context, we can be encouraged by the fact that the distant future should be wise and knowledgable on many topics. So we can try to have the distant future settle bets.
For a fanciful example, imagine that we save lots of data about today’s academics, including not just their publications, but also their tweets, emails, and much more. We commit to paying distant future folks to use that data to do very detailed analyses of a random one percent of today’s academics, carefully judging their overall intellectual contribution to things that turned out to matter. We then set up assets today that pay in proportion to those distant evaluations, and use the current prices of these assets to rank people for key decisions today like jobs and grants. This sort of system could cut incentives for citation grubbing, mutual-admiration societies, and other games people play today to create the illusion of a consensus on who is good. All by making stronger use of the fact that the future can know more, and can learn more cheaply, including about people today.
Here’s another fanciful example. Today if I sue you today for $10,000, we soon have a court decide if you have to pay me or not. You and I each pay big amounts for lawyers and preparation, and the court also pays to make a decision. A decision that could be wrong. Imagine instead that we saved lots of data about this lawsuit and put off the decision, committing to have distant future folks decide, when the world is wiser and analysis is cheaper. In the meantime you pay $10,000 immediately and in return get the asset “$10,000 if found innocent”, while I get the asset “$10,000 if found guilty.” We can trade these assets to others if we like, and anyone who has both kinds of these assets can merge them back into simple money. We might treat these assets as if they were closely tied to our true innocence or guilt.
In sum, we expect continued growth to create a big distant future, a future that can influence life today just as a big city in the valley might influence life in nearby mountain village. A big future suggests that we worry about being smashed by a great filter. Big future tech abilities suggest that we could survive current death via cryonics, that we might be today actually be part of a future simulation of history, and that biology as we know it won’t last. A big future is an attractive place to migrate, to influence, and to impress. And a wise knowledgeable future offers many ways to improve current incentives for honesty. Like a big city in the valley below, a big future can matter to us now.
Posted on 2021-06-24
(This post is more of an essay, intended to be especially widely accessible.)
Long ago, my physics teachers taught me to arrogantly dismiss the “paranormal”, like ghosts or telepathy. Or UFOs. Yes, we once saw meteorites as paranormal, but not today. Yes, we now accept ball lightning, even though evidence for it is weaker than for UFOs, but we have plausible theories there.
However, when the topic of distant “aliens” came up recently in my research on the origin of life and the future of the universe, I browsed UFO evidence and found it to be much stronger than stuff on ghosts or telepathy. And now a U.S. military report says that intelligently controlled UFOs with amazing abilities seem real to them, even if they don’t know their cause.
Hence my and perhaps your titular reaction, “What the Hell?” How can this make any sense?
Turns out, my prior research prepared me to address this very question, once I gave it some thought. Not on the evidence for UFOs, where others are more expert than I. But on how to fit this idea of strange objects with amazing abilities under intelligent control into your scientific world view. (Note that I’m not claiming this as fact; I’m saying it isn’t crazy.)
While there are many possibilities here, it suffices for me to show just one. And yes, it involves aliens.
We can easily believe that aliens are very advanced, and thus have amazing abilities. But two questions remain:
First, note that our standard best scientific theories predict aliens. That is, they predict that life sometimes arises from simple dead matter, and can eventually evolve to make intelligent creatures like us. And this could happen most anywhere.
Yes, the universe looks completely dead; we see no signs of life outside Earth, even though over millions of years advanced aliens could have made some big visible changes. Some possible explanations:
Of these, only the last two can put aliens here now, and #3 seems too much a conspiracy (i.e., coordinate to hide) theory for my tastes. But scenario #4 works, and could plausibly result from “panspermia.” That is, simple life might have arisen on a planet Eden long ago, via a very rare event. (My research suggests this happens only once per million galaxies.) After life evolved at Eden for billions of years, a rock hit Eden, kicking up another rock that drifted for millions of years carrying life to seed our Sun’s stellar nursery. A nursery that held thousands of new stars packed close with many rocks flying around, allowing life to spread quickly to them all.
Our sun’s siblings then drifted apart, while life evolved on each planet for billions of years. The first sibling planet to develop civilization did so millions of years ago, and it wasn’t Earth. These aliens then sought out their sibling stars and traveled to them to watch civilization maybe evolve there.
Now, to explain the fact that these aliens have not visibly changed our shared galaxy, even though they can travel to here, we must postulate that they enforce a rule against making big visible changes, probably enforced by a strong central government. A rule against mass aggressive expansion, colonization, and disassembling of planets, stars, etc. Maybe due to environmentalist values, maybe to enable regulation, or maybe just to protect central control and status. Yes, this is something of a conspiracy theory, but being smaller, it seems easier to swallow. Okay, that is a not-crazy answer to the first question, on why aliens are here now in a dead universe. What about the second question, on why UFOs act so weird and coy?
To answer this, I postulate two features of sibling alien preferences: 1) they want us get us to comply with their rule against making big visible changes to the universe, and 2) they are reluctant to just kill, crush, enslave, or dominate us to get this outcome (or they’d have already done one of these). Aliens somehow value something about us independent of their influence, and thus prefer us to organically and voluntary comply with their rule.
To induce our voluntary min-change compliance, their plan is put themselves gently at the top of our status ladder. After all, social animals consistently have status ladders, with low status animals tending to emulate the higher. So if these aliens hang out close to us for a long time, show us their very impressive abilities, but don’t act overtly hostile, then we may well come to see them as very high status members of our tribe. (Not powerful hostile outsiders.)
If we are smart enough to figure out that they have a rule against big visible changes, enforced via a “world” government, we may naturally emulate those policies. We may even come to treat UFO aliens as ancient humans treated their gods and top leaders, with respect, deference, and obedience. After all, most ancient people knew little about their gods and leaders beyond their impressive wealth and abilities, but that was usually enough. But why not just land on the White House lawn, meet with our leaders, and explain their agenda? Because once they start talking to us, we will have a lot of questions. Such as on their nature, practices, history, and future plans. And many of us would surely hate some of their answers. They are complete aliens after all, and we are often offended by humans from slightly different subcultures. They reasonably guess that we are just not as open-minded as we like to think.
Sure, maybe if they understood us really well they could just say “no comment” when a discussion got near something likely to offend us. But we’d then reasonably infer that they were hiding bad news near there, make a guess at what it is, and get somewhat offended at that. Far simpler and more robust to not talk at all, except in dire emergencies.
That’s a plan that could be approved long in advance by a far away central power wary of allowing much improvisation and discretion by their local representatives. Especially if their very old and stable centralized civilization has atrophied and lost much of its prior generality and flexibility. Or if they worry that such representatives may “go native” and be persuaded by us to go grabby. This strategy works best if they carefully limit what they show us. Just give us brief simple impressive glimpses that don’t let us figure out their tech, or even the locations of their local bases. The package of simple geometric shapes, crazy accelerations, no sounds or other local side effects, clear intelligent intent, and avoiding harms to us seems to do the trick.
And that’s my story. How UFOs as aliens can make sense. Not an inspiring story, but a plausible one. They could be our panspermia siblings, here to get us to voluntarily obey their rule against aggressive expansion, hopefully via our emulating them because they sit at the top of our status ladder. Not to say that this is obviously true, just to say that it isn’t crazy. Yes, there are other possible scenarios. But UFOs as aliens, that’s no crazier than ball lighting.
Added 4July: The many comments on this post are mostly the result of this mention at Instapundit.
Posted on 2023-05-06
Sam Quirk was ten years old. While on a field trip, his bus had paused at a rest stop, and Sam was sitting in a bathroom stall. From the next stall over, he heard clearly but quietly, “Sam Quirk, ask your parents about ‘royal propriety’”. By the time he could check the next stall, it was empty.
Back at home, Sam asked his parents. They looked grave, and told him that something similar had happened to them roughly every ten years for their entire lives. Except that after the first time, all they heard was just the phrase “royal propriety”. Now they told Sam what their parents had told them: “We are royalty.”
Sam’s dad was descended from a rich and powerful line of royalty that was famous for its being very secretive. Hardly anything was known about them, including where any of them lived. The lawyer who represented them issued rare press releases, which said little.
Long ago, a representative of this royal org had contacted Sam’s great granddad, and explained to him how very important it was that no one from their line ever appear in the public eye. To demonstrate their determination, they promised to give these personal once-a-decade “royal propriety” reminders. The message: they are still around, and still care.
They also hinted at dire consequences for any violations of this rule. So far their family had always complied. And as far as anyone in the family knows, this royal org has never helped them in any way, nor suggested that it ever would.
Should Sam feel lucky to be part of such a rich illustrious royal family? Or unlucky to have such a powerful and hostile family stalker?
I offer this story as an allegory of my best guess of humanity’s situation if some UFOs are in fact aliens. Remember that I don’t claim that they are. Only that I find it hard to see honest mistakes as explaining our strongest most dramatic UFO reports, that due to my grabby aliens work I am something of an expert on the prior for this some-UFOs-are-aliens hypothesis, and also on its social implications, and I thus feel obligated to resist the usual taboos to give my best estimate on both these topics.
Many suggest that UFOs-as-aliens would put us in a position of radical uncertainty, wherein we’d have almost no idea who are these aliens, how many others are out there, or what any of them want. In contrast, I think we can actually say quite a lot. Alas, it is not a pretty picture.
The most likely scenario that I can find consistent with some UFOs being aliens starts with life appearing ~9Gya on a one-in-a-million-galaxies-rare planet Eden somewhere in our galaxy. Then ~5Gya life was transferred via panspermia to many of the ~1000 newborn stars in our Sun’s stellar nursery. Life continued to evolve on those stars, until >~0.1Gya one of them gave rise to an advanced alien civilization.
Civilizations that allow interstellar colonization probably cannot maintain civ-wide governance to regulate, prevent war, and prevent their descendants from evolving into strangeness. For this or other reasons, this particular alien civilization chose to prevent any part of itself from leaving its home system to colonize the universe. However, it made rare exceptions for expeditions to stellar siblings that could be seen in telescopes as hosting life, and thus at risk of birthing another advanced civilization. The main motive was to prevent such a “panspermia sibling” civilization from violating their rule against expansion. But they’d rather achieve this via persuasion, rather than extermination.
As each expedition risked violating their rule by going rogue, home authorities wanted a simple robust strategy; they didn’t trust expeditions to exercise much discretion. While alien visitors to Earth could have remained completely invisible, or become very obvious, they instead chose a third way. Their strategy was this: hang out on Earth at the periphery of our vision, act peacefully, show very impressive abilities, but reveal little else about themselves.
Why? Social animals consistently have status hierarchies, and we humans have consistently domesticated other animals (and ourselves) by putting ourselves at the top of their status hierarchies. So the aliens hoped to do this with us. However, they also guessed that if they revealed too many details about themselves, we’d likely find something to hate, spoiling this status effect.
Thus the plan: if we didn’t come to hate them, then once we became convinced that they really exist, we could figure out their agenda by ourselves without their saying a word. And then we had a decent chance of going along with it; after all, most today who believe that some UFOs are aliens also seem to trust aliens more than their local authorities. If we didn’t go along, they would at some point have to intervene.
So this is our fate if some UFOs are aliens. We will either go along with their no-expansion rule, or become more directly controlled, or exterminated. In the coming centuries they will probably tell us no more about themselves, nor help us in any other way, and we will not figure out much on our own. Not even the location of their local base. Maybe we will identify their home world via telescope. And maybe they have hidden messages in the details of their frequent displays, but probably not.
We should be quite impressed by the fact that these aliens not only managed to keep their civilization going for over 100My, but they’ve also managed to enforce their no-expansion rule for that whole time. After all, it would have taken just one tiny successful rebel slipping away unseen to end it. But we should also wonder how much their abilities may have decayed and rotted over that long period. With enough rot, we might just have a long-shot chance of developing ems or AIs and then slipping them off to colonize the universe unnoticed. But probably not.
Which seems to me much like Sam’s situation. So, do you feel lucky punk?
Posted on 2021-06-08
A careful analysis of UFOs should consider lots of data, and consider it in much detail. I oft hear skeptics seek shortcuts, such as by declaring all testimony invalid, or insisting that only some long conjunction of encounter features could be sufficient. But consider a legal accusation of attempted murder. Even though the prior odds that a random X attempts to kill Y during hour Z is terribly low (<10-12), we are still willing to entertain such claims, and we accept personal testimony as an important part of supporting evidence.
Yes, advocates of things like UFOs seem willing to put more time into such details, and it may seem unfair to expect skeptics to put in as much work. But jurors and lawyers must put in a lot of effort in legal trials. This is the great problem of how to divide intellectual labor; as with most topics, we do best if we task a few with going into great detail on each topic, so the rest of us can defer to their analysis. If you aren’t willing to go into sufficient detail, then admit this isn’t one of your topics, and defer to others on it.
In that spirit, instead of expressing opinions on many UFO topics, let me instead focus on the area where I have the most relative expertise: the priors to associate with the some-UFOs-are-aliens hypothesis. As far as I can tell, the main reason that most give for skepticism that aliens visit Earth in the UFO style is that this theory seems a priori crazy unlikely. But that estimate seems wrong to me. Let me explain.
A full Bayesian analysis of the four main UFO theory categories (error, hoax, secret Earth orgs, aliens) needs eight numbers: one prior and likelihood for each theory. In this post I try only to estimate one of these eight numbers: the prior for the aliens theory. Here goes.
Life exists here on Earth, and our standard best theories say that this was not a miracle, nor was Earth the only place such things could happen. Furthermore, our universe also seems very large (perhaps infinite). Thus our standard best theories predict that advanced life has appeared and will appear many times out there.
These standard best theories also predict a wide range of dates when this could happen. As a result, two independent alien origins are likely to be millions to trillions of years apart in time. Which gives aliens a lot of time to travel to visit other aliens.
So we can break down doubts on prior expectations about UFO as aliens into three parts:
First, how close might aliens be? As my co-authors and I discuss here, humans seem to have arrived quite early in history, at least if one assumes that the universe would remain empty and wait indefinitely for advanced life like us to appear. This is the main reason we offer for postulating a grabby aliens deadline, to explain human earliness. And our grabby aliens model implies that aliens do appear often enough for maybe some of them to have come here by now.
Now grabby aliens arriving by here now would also be quite visible to us now. But our basic model is quite consistent with variations wherein there are many, perhaps thousands or millions, of non-grabby alien civs per grabby civ, all born at the same sort of places and times. These non-grabby civs do not remake their local universe. So either they die fast, life long but do not expand, or they expand long but do little to remake their universe.
In my view, the most likely scenario that puts long-expanding-but-not-remaking aliens here now is panspermia siblings. Life arose long ago on some very rare Eden, which then seeded our Sun’s stellar nursery, with life quickly spreading to most stars in that nursery. At least two of these stars eventually developed advanced life, but Earth was not the first. Aliens at the first star looked for their panspermia siblings, noticed simple life on Earth here long ago, and then long ago traveled to near here to await the arrival of advanced life. Where they now do their weird UFO encounter things. So to explain UFOs as aliens, we must postulate that these first star sibling aliens had preferences and coordination abilities sufficient to do the following:
I estimate a chance of at least 10% for each of the following events, given the prior events:
Note that #1 requires a high enough rate of rock transfer between star systems, #2 requires that most of the great filter happened on Eden, #3 is more likely when civs adopt strong “world” governments, and #4 is relatively likely because we shouldn’t really expect to be able to predict detailed behaviors of strange alien civilizations.
Four factors of 10% gives a minimum prior chance of 10-4, but as most of the probability weight should above these minimums, I estimate the total chance to be at least 10-3. As I’ve said before, combining all the relevant priors and posteriors I judge the hoax and aliens theories to be most likely for the hardest-to-explain UFO cases. But I don’t claim as much expertise on all the other numbers required to judge that, as I do for the one number I estimate here:
The prior chance of the aliens theory of the hardest-to-explain UFO cases is at least 10-3, relative to the other three theory categories of error, hoax, and secret Earth orgs.
This prior is actually pretty high compared to the usual priors in most legal cases. So the types and amounts of evidence on particular cases that is sufficient to convict in legal cases seems sufficient to judge UFOs-are-aliens as more or less likely than not. But again, I have no special expertise to offer you for judging the details of UFO encounters. I can just say that you need to look at such details; you can’t just dismiss UFOs-as-aliens theory with a wave of your philosophical hands.
Added 10June: Many take issue with my estimating 1/10 for the chance that aliens waiting here would be somewhat evasive, but not completely hide nor announce themselves. They don’t see this as a good plan for any goals they can think of.
But we are talking about an entire alien civilization here! Human societies often do things, like fight wars or stop having kids, that seem counter-productive from the point of view of that society as a whole. In addition, individual humans often do things that seem counter-productive until you consider their signaling incentives. I wrote a whole book on this.
If we often have trouble explaining the behaviors of human societies and individuals, I don’t think we should feel very confident in predicting detailed behaviors of a completely alien civilization. After all, many have reasonably doubted if we could even communicate with aliens, or recognize them when we saw them. Having outlined some possible signaling motives for alien UFO behavior, I can see that there are many possible explanations for aliens-as-UFO behavior. Thus a 1/10 prior seems reasonable to me. Added 13Jun: I did 6 Twitter polls to elicit relative priors and likelihoods for the four main theory categories:
Medians of lognormal fits to my recent polls gives these Bayesian estimates re 4 different UFO theories. These numbers seem crazy wrong to me. https://t.co/0jY93DJy2m pic.twitter.com/B7itXtHDyd
— Robin Hanson (@robinhanson) June 10, 2021
Added 14Jun: Thinking through the consequences of the show-but-don’t-talk strategy suggests that it will work out pretty well for the aliens.
Posted on 2021-05-05
Sometimes I pride myself on my taking an intellectual strategy of tackling neglected important questions. However, one indicator of a topic being neglected is that it seems low status; people who discuss it are ridiculed, and their intellectual ability doubted. Thus my strategy risks lowering my status.
To protect against this risk, I can set a policy of only tackling topics that seem to have a substantial synergy with my skills and prior topics. Which seems a valid policy, even if not entirely honest. For a long time this protected me against UFOs as aliens, one of the most ridiculed topics ever. But then I started to study loud very distant aliens, and the topic of alien UFOs became more relevant.
To limit the damage, I once tried to talk only on what UFOs would imply if they really were aliens, but not crossing the line to discuss if they actually are. But on reflection I can see that this topic is in fact neglected, important, and has synergies with my skills and other topics. So now I am shamed into trying to live up to my intellectual ideals, which if truth be told aren’t as strongly rooted in me as I’d like to pretend. Sigh. So here goes, let’s talk about explaining UFO encounters. I see four major categories of explanation:
Now it seems pretty obvious that if we are rather inclusive in our definition of “UFO encounter” then (A) is the best explanation for most of them. The interesting question is how best to explain the few hardest to explain encounters. Here is a related Twitter poll I just did:
How best explain top 10 hardest-to-explain UFO cases?
(A) Delusions, mistakes, & misunderstandings,
(B) Secret advanced our govt. tech, honestly seen,
(C) Lies & hoaxes, both decentralized & govt. funded/ coordinated,
(D) Adv. tech by foreign powers, secret societies, or aliens.
— Robin Hanson (@robinhanson) May 5, 2021
Notice that I made the mistake here of lumping foreign governments into option (D), instead of into option (B) as I do above. If I had done the poll right, my guess is that we’d see: (A) 57%, (B) ~23%, (C) 10%, (D) ~10%.
Over the last few months I’ve been doing a lot of reading and watching and thinking on this topic, and I do think I have a judgement to report, a judgement that should represent news to those inclined to copy my judgment. First, (A) or (B) seems to me much less likely than (C) or (D). Second, between (Ca) spontaneous decentralized hoaxes and lies, and (Cb) hoaxes and lies coordinated by a big central organization, (Cb) seems much more likely. And third, among (Da) aliens, (Db) secret societies, (Dc) time-travelers, and (Dd) something even weirder, (Da) seems more likely.
Thus I see the main choice as between (Cb) and (Da), which would together be supported by only ~10% of poll respondents, and between which I can’t decide. Thus I am making a relatively strong claim here, at least relative to poll opinions. Let me outline some of my reasons.
First, if you look at the details of the usual hardest cases, to ones to which UFO fans most often point, you will see that there are often a lot of pretty sober looking people who all say they saw the same pretty clear and dramatic things under pretty good observing conditions. And often what they say they saw is solid-looking objects with remarkable combinations of location, speed, and acceleration, with no attendant thrust or control surfaces of the sort we’d use if we were trying to achieve those combinations.
I know enough physics and tech to know that these claimed abilities are just far beyond anything Earth governments will have access to for a long time, at least if the past is any guide. Or anything that natural weather could make. And similar abilities have been seen for over a half century, so if governments were hiding these abilities they’d be hiding them for far longer than they usually hide techs.
I also know enough human nature to know that these are not close to the sort of things that honest sober sane people would claim to see, if they just somewhat misinterpreted something that they saw or heard. And most of the people reporting in these strongest cases do seem pretty sober and sane. Thus in these strongest cases, the story that all these people are merely mistaken or deluded just doesn’t work, at least for the sorts of things they say they saw in these hardest cases. Nor does the story work that this is advanced government tech that they will release to show everyone in at most a few decades. So I must reject cases (A) and (B), which leaves me only with cases (C) and (D).
[Added 6May: Note that I am making judgements here about particular cases that I’ve considered in some detail. I am not saying I always believe what anyone says they saw. For a comparison, I find the usual evidence presented re ghosts and fairies to be much less persuasive. ]
Yes, humans like to play practical jokes on one another, and sometimes they take those jokes to some pretty far extremes. Sometimes they even try to make the jokes last for years. And often they are inspired to copy the jokes of others. But to explain most of these hardest cases mainly in terms of practical jokes seems just a bridge too far. Really, thousands of disconnected people all around the world playing the same big scary jokes for decades, and then almost never breaking down and laughing and crowing about their jokes even decades later? In contrast, governments, especially their spy parts, have run some pretty big, well-funded, and long-lasting disinformation campaigns. So I have to favor (Cb) over (Ca) by a big margin.
Regarding (D), time-travel seems impossible without crazy extreme physics, and known secret societies on Earth have never reached within orders of magnitude of the scale and degree of secrecy that this would require. Yet spirits or creatures from other dimensions seems even more crazy. Aliens, in contrast, are predicted to exist by our best theories. It is just a matter of finding a plausible scenario wherein they’d be here now doing what we see them doing, and not doing other stuff we don’t see them doing. I’ve tried to work out such a scenario, and find one that is a bit tortured, but far more believable than secret societies or travel across time or between dimensions. Note that both (Cb) and (Da) are hypotheses that I would have found priori implausible. So the entire existence of the familiar pattern of UFO encounters was a priori implausible, and so now that I see it I struggle to explain it. And as both of the most likely explanations are low status topics, i.e., aliens and a record-breaking-huge government conspiracy, you can see why most people would rather just avoid the topic.
This post is already too long, so I will stop here once I make one last point: (Cb) is a theory of remarkable government competence. Some governments, or a consortium of them, have managed to get thousands of people to either lie and say they saw stuff they didn’t, or paid for expensive enough tech to fool them. And yet this conspiracy has remained hidden for a great many decades, even from the top levels of their own governments.
In contrast, (Da) seems to require a scenario of remarkable incompetence, among the aliens themselves, among our governments, and even among the UFO activists. So which is more likely: surprisingly high government competence, or incompetence? Added 7June: This poll agrees with me.
Posted on 2023-04-29
There have been over 100K UFO sightings reported worldwide since 1940. Roughly 5% or so are “strong” events, which seem rather hard to explain due to either many witnesses, especially reliable witnesses, physical evidence, or other factors. Many of these events are also “dramatic”, wherein the UFOs seem to display amazing abilities, ones well beyond those held by any known Earth orgs. I’d guess there are at least a thousand such strong dramatic reported events.
Due to my work on grabby aliens, I felt a bit of responsibility to consider the UFOs as aliens thesis, as I seem to be a world-level expert on the prior for such a hypothesis. And while I’m not so much an expert on UFO hypothesis likelihoods, I have been tempted to learn about many of these strong dramatic events. (Eg) Which I now somewhat regret, as I now find myself the awkward position of having made an awkward inference.
There are four main ways to explain UFOs:
My awkward inference starts here: it seems clear to me that #1 can only plausibly explain a modest fraction of strong dramatic events. Most errors would have to be much closer to gross incompetence than to “oops”. (If you’ve also looked but can’t see this, I just don’t know what to say. Pay more attention?)
Why is that awkward? Because for the last seventy years, elites, especially STEM elites, have had a very strong social consensus against explanations #3,4. Those who say otherwise are ridiculed and excluded.
But that still leaves explanation #2, right? Well, yes, elites are okay with explaining many UFOs this way. But in order to explain most strong dramatic events this way, I just don’t think it works to postulate scattered amateur liars and hoaxers. Instead I think one needs a big conspiracy, wherein a coalition of orgs has secretly and professionally coordinated to spend big budgets over many decades to have many lie, and to fool others via what are essentially magic tricks.
The US military seems the most plausible anchor for this coalition, as they could have tried to fool the USSR into hesitating before a nuclear first strike. That could be worth a big budget. (Though note, if they’d lie about UFOs, they’d probably also lie about the moon landings.) Yes, the scale of this lie is only modestly bigger than that of some other known US lies, now revealed. But even so, elites also look down pretty hard on those who endorse such big conspiracy theories.
Alas, my situation gets even worse. This is due to my noticing that the USSR also had many UFO reports, including many strong dramatic ones. And I just can’t believe that the US could get so many people inside the USSR to lie or arrange sufficiently elaborate magic tricks. Nor can I believe that both the US and USSR both created huge long-lasting hoaxes trying to fool each other. Nor that the whole Cold War was faked.
Thus I’m stuck with putting substantial weight on explanations #3,4 for at least some of the thousand-plus strong dramatic UFO events. Which feels pretty awkward.
Yes, I might now become more accepted by UFO folks, even if I become less accepted by the usual elites. But while there are exceptions, overall these aren’t exactly our intellectual creme de la creme. Yes, it seems that in this case they happen to be more right, and on something pretty big, but that is probably more due to their contrarian instincts than their brilliant analysts.
Note that if the evidence now supports my inference, it probably has done so for decades, and so can plausibly continue to do so for many more decades, or perhaps even centuries, without the relevant elite consensus changing much. So I’m not yet seeing bet-table predictions to make here.
Added 1p: I should mention that I’ve also looked at the best evidence offered for angels, ghosts, and fairies, and those just look much worse to me. The best evidence for ball lightning in the field is also pretty bad, even though the usual consensus believes in that due to it being created in the lab.
Posted on 2021-03-28
Even though many or even most UFO sightings are best explained as delusions, hoaxes, and ordinary stuff misunderstood, there appears to be a large remnant (>1000) that are much harder to explain, and which show consistent patterns. Such as ~30-1000 second episodes peaking near ~9pm (tied to local sideral time), at random spatial locations, of quiet lights or objects in the sky with intelligent purposes and amazing speeds and accelerations. Sometimes confirmed by many people and recorded by many instruments.
If they aren’t delusions, hoaxes, or misunderstandings, the main remaining explanations are a) some sort of secret society or agency that arose on and is tied to Earth, or b) some sort of aliens. I’m not saying its aliens, but in this post, it’s aliens. That is, here I want to “go there”, and think about how best to explain UFOs, if they are in fact aliens.
Many have worked on trying to explain UFOs in terms of their immediate physical effects. I kinda like “laser pointers for cats” style theories wherein aliens in orbit send beams to paint a local disturbance, while using telescopes to watch local reactions. But these details aren’t that important for whether we believe that UFOs are aliens, as aliens would almost surely be a lot more advanced than us, and so plausibly capable of a wide range of such approaches.
No, it seems obvious to me that the main reason that most resist believing that UFOs are aliens (or secret societies for that matter) is the apparent implausibility of the social thesis. We find it hard to integrate this hypothesis with the rest of our social world views. That is, with our views on what agents can exist, how they are socially organized, and the sorts of behaviors that we expect of social agents within particular kinds of organizations. If aliens are around, why haven’t they made more direct contact, or built more obvious stuff, or traded with us, or conquered or killed us?
If the main block to believing in UFOs as aliens is a lack of a plausible enough social theory of aliens, then it seems a shame that almost no one who studies UFOs is a social science theorist. As I’m such a person, why don’t I step in and try to help? If we can find a more plausible social theory, we could become more willing to believe that UFOs are aliens. And if we can’t, we can at least confirm more expertly that the usual reluctance is justified; the social theories you’d have to invoke are so crazy unlikely that yeah, we gotta attribute UFOs to delusions, hoaxes, and misunderstandings, no matter what our eyes and instruments seem to say.
In social science, we often prepare for theorizing about a topic by first summarizing its “stylized facts”. These are key data patterns in need of explanation, phrased in language that is closer to theory. In this post, I will attempt this “stylized fact” exercise for UFOs-as-aliens. In my next post I’ll take my shot at explaining them. Here are three key stylized facts:
1. LIMITATION – The very idea that UFOs are aliens, rather than a secret society on Earth, implies either a completely independent origin from us, or that any common ancestor was long ago. (~100Myr+.) So unless aliens civilizations are very short-lived, then any modest randomness in the timing along either evolutionary path implies that one of us reached our current level of civilization millions of years before the other. And since we just got here, it must be they who reached our level millions of years ago.
(Note that having a civilization last for many millions of years is itself quite an achievement. Which raises obvious questions: what sort of genetic, cultural, organizational, etc. changes were required to achieve that, and at what cost came such longevity?)
If UFOs on Earth are aliens from elsewhere, then there are in fact aliens out there, who can and do travel between the stars. Because here they are, aliens who have actually traveled between the stars. So right off the bat we must reject theories that say that such travel is impossible or crazy impractical. Or that some motivational convergence ensures that advanced life almost never does actually travel.
Now put these two facts together: they’ve been around for many millions of years, and they can and do travel between the stars. With so many millions of years and this same tech they used to get here, they could have gone everywhere. The big dramatic implication: they could have remade the universe, or at least a big chunk including our galaxy, but have not done so. Somehow they have self-limited their expansion.
(Note that in addition to limiting their expansion, aliens behind UFOS also seem to have limited their tech; UFO tech seems advanced, but not 100Myr+ level advanced.)
Now one possibility that I want to note, and set aside, is that the universe is in fact chock full of aliens who have in fact remade it, but that we are fooled to see otherwise by crazy advanced tech wielded by a vast tightly-coordinated alien conspiracy based on arbitrary inscrutable motives. Like theories of powerful intrusive gods and simulation managers with arbitrary inscrutable motives, it is not that such theories are impossible, but that they offer little room for structured analysis. I see little to gain from discussing them.
Stylized fact #1: UFO aliens are very old, and could have remade universe, but some self-limit stops them.
2. CORRELATION – This failure to remake the universe gets more puzzling the more common are aliens in space and time. If UFOs-as-aliens are as thick on all planets at all times as they are here and now, then there must be a crazy huge number of well-hidden alien facilities out there where UFO equipment is made, repaired, refueled, staffed, etc. All strongly limited to ensure that it never remakes its local universe.
Worse, there have been literally an astronomical number of opportunities for any one deviant alien to start to remake its local universe. If a deviation could last long enough, to acquire enough local resources and power, other aliens would have a hard time shutting it down without also acquiring similar levels of local resources, and thus also remaking their local universe. Even if some sort of local conformity pressure tends to stop most deviations, that pressure has to be crazy extreme reliable to work everywhere always in a vast densely populated universe.
The simplest way to resolve this puzzle is to posit that aliens are in fact pretty rare, and that they coordinate to preserve that rarity. After all, the fewer are the possible alien travel events, the higher of a deviant event chance that we can tolerate in our theory of their behavior.
(If aliens are very short-lived, then there have to be even huger numbers of them for one to be here now, requiring an even more crazily-low chance of any of them allowing any deviations.)
Besides perhaps interstellar travel being impractical, advanced life arising very extremely rarely is the simple story most of us most start out with to explain our empty universe. And even if one must postulate that aliens are only extremely rare, not very extremely rare, to explain humanity’s early arrival in the universe, that still means aliens are so rare that we won’t meet them for roughly a billion years.
But for aliens that rare we have a different problem: why are they right here right now, but almost nowhere else? Something has caused a huge correlation between them and us, so that even though aliens are rare enough for their facilities to stay hidden, and even though they have created local pressures to ensure that they only rarely travel or have opportunities to try to remake the universe, they’ve made an exception for traveling to be with us here now.
The rarer are such aliens, the more time they’d need to get here from where they started. So either they’ve been around for a very long time, and decided to come here based on what Earth looked like a very long time ago, or they happened to start very close to us, a remarkable spatial coincidence in need of explanation.
Stylized fact #2: UFO aliens are rare and self-limited, and yet are here now.
3. INDIRECTION – We can think of a number of plausible practical motives for rare self-limited aliens to make an exception to visit us. First, they may fear us as rivals, and so want to track us and stand ready to defend against us. Second, if their limitation policies are explicit and intentional, then they’d anticipate our possibly violating them, and so want to stand ready nearby to enforce their limitation policies on us.
In either of these two cases, aliens might want to show us their power, and even make explicit threats, to deter us from causing problems. And note the big the question of why they don’t just destroy us, instead of waiting around. A third possible motive that can explain this is that the origins of independent aliens like us are a rare valuable datapoint to them on far-more-capable aliens who they may fear eventually meeting. In this case they’d probably want to stay hidden longer, and then maybe destroy us later. Note that all of these motive theories suggest a substantial ability of these aliens to organize and plan actions on the basis of such abstract, collective, and long-term considerations. A very decentralized alien society might not be capable of it, nor perhaps of maintaining whatever pressures prevent their own travel and remaking the universe.
The most striking fact about UFO encounter events is how little they seem to accomplish, not for any of these goals, nor for any other easily identifiable practical goals. Advanced aliens could surely monitor us sufficiently from a distance unseen, and to control us via commands or threats would require much more direct contact. These UFO events don’t seem to much help them collect useful info or resources, nor do they much limit or expand our info, powers, or resources. Yes, they show some of us that the universe can look weird, but surely they know that we know that fact regardless.
Now we humans are widely known to often act on indirect motives, not tied very closely to simple direct practical outcomes. Many animals “play.” Human ancestors who did things for “symbolic” reasons are often seen as especially “advanced”. People today often have “obsessions” that make them spend far more on some things than simple practical ends can explain. Lazy secure organizations are at times quite “wasteful”, doing things that pretend to achieve practical ends, but in fact achieve them at best quite ineffectively. And I’ve recently coauthored a book on how common are hidden motives in humans today; many things we do just don’t much accomplish the goals to which we give lip service, like learning at schools, and healing at hospitals.
So it isn’t crazy to think that aliens might have indirect obsessive lazy motives for UFO encounters, motives hidden perhaps even from themselves. But this case, of overcoming the usual coordinated limits to take eons to fly to a distant star just to glow-buzz their treetops, seems spectacularly extravagant even by the standards of dreamtime humans today. For this to happen, aliens need a sufficient level of “slack” resources available to spend on such symbolic activities. And even with hidden motives and lazy organizations, we humans usually at least make up vague stories about practical ends served by our actions, even when such stories don’t stand up to close scrutiny. So a decent theory of aliens should explain their level of slack, and suggest some ideas for what stories aliens are telling themselves about the ends they accomplish via UFO encounters. And why they haven’t just destroyed us.
Stylized fact #3: Alien-driven UFO encounters accomplish little, yet must somehow be justified to them.
And those are the key stylized facts that a social theory of aliens must explain. Again, it is the lack of seeing a sufficiently plausible explanation of such facts that is why most are reluctant to believe in UFOs-as-aliens. (Yes, many are not so reluctant, but mostly because they don’t understand enough to be puzzled.)
Added 31Mar: My explanation attempt is here.
Posted on 2021-03-30
In my last post I summarized some key stylized social facts that a theory of UFOs-as-aliens would need to explain:
Any aliens behind UFOs would be amazingly long-lived creatures who have somehow coordinated to limit any small part of themselves from expanding and remaking the universe. This gets easier to believe the smaller and rarer they are. (They also seem to have limited their tech.) Yet they’ve overcome their self-limits to travel to be here now, so they must be close enough to come quickly once they saw signs of advances, or they saw signs of interest very early and traveled very far. We can see practical reasons for them to come here, at least if they can coordinate to achieve such plans. But most such motives seem better served by destroying us than by the usual reported UFO encounters, which seem to accomplish little. Yes, humans today do many things for indirect “symbolic” motives, and lazy organizations often pretend to achieve more than they do. But these require slack, and fig-leaf stories to justify them. So from whence comes alien slack, and what could be their justifications?
My tentative explanation for all this has four main supporting elements: panspermia siblings, world government, moral ideology, and complexity rot.
1. Panspermia Siblings – Imagine that life started on some previous planet Eden, where it went through some very hard steps. Life then spread from Eden to Earth, as well as to some other planets. Even if it were hard for life to spread between planets, the Eden-to-Earth scenario could still be statistically favored compared to the Earth-only scenario, as Eden can start much earlier and can be in more places. This is because Earth had to have a much calmer environment than Eden to host the last half-billion years of fragile multi-cellular life. These few seeded planets might be the only ones with life in the million nearest galaxies. Once seeded, Earth and its sibling planets would then compete to complete the remaining hard steps required to reach advanced life. If one of those planets succeeded before Earth, then it would host close but rare aliens, who share a lot of biology detail (e.g., DNA) with Earth. Those aliens could have then searched out their sibling stars (which have a clear signature that we can see even now), found Earth around one, and then waited perhaps millions of years for civilization to appear here. These aliens had several good and practical possible reasons for coming here in time to see us now up close.
Positing that the aliens behind UFOs come from just one nearby sibling planet, with the nearest other aliens many galaxies away, makes it easier to believe that these aliens have successfully imposed sufficiently-strong self-limits on expansion and on tech advance, leaving the empty universe we see.
2. World Government – Over the last few centuries, one of the most consistent world trends has been an increase in human organization size and complexity, with more functions and decisions drifting up to higher levels. We have developed both better networks and better hierarchical organizations. Plausibly this trend is behind most others; it seems to be the main driver of faster innovation, which is the main cause of more wealth, which drives most other trends.
A straightforward long-term prediction from this trend is “world” (really “civilization-wide”) government. After all, a few have come close to creating this via force, we now have a United Nations by consent, and regulators worldwide share an elite culture that creates a de facto world government on many issues. Stronger, more formal versions seem likely within centuries. Within a star system, talk delays are modest, and it is easy to see and shoot at most anything, making a world government quite feasible there. However, world government is quite hard to start (and if started harder to maintain) once independent self-sufficient colonies at other stars can grow as fast as at the home star system. Thus the existence of such colonies becomes a deadline for the creation of a world government. As near Earth this deadline seems likely to be met, that may also have happened for sibling star aliens.
The advantages of a world government will seem clear and compelling: a civilization that can better coordinate on global problems like war, pollution, and innovation. And that can better enforce widely-liked regulations. Also, global majorities will be eager to impose their will on global minorities, and to lock down their temporary advantages via a permanent world government.
By its very nature, a world government reduces innovation and adaptation in, but also promotes the stability of, the largest scale civilization structures. An advanced star-system-wide civilization probably has a large enough base of knowledge and resources, and a stable enough environment, for this tradeoff to allow for stability over many millions of years. Thus the fact that aliens have lasted for millions of years weakly suggests that they have a world government.
3. Moral Ideology – While pre-human primate groups were held together mainly by kin and alliances, human groups could be larger due to social norms, which were enabled by human weapons and language. Social norms have also aided our other more recent methods of social organization. As norms matter more in collective politics than in private life, a world government would gain legitimacy and stability by more strongly supporting widely-held moral norms.
Thus a world with a world government is likely to impose more stronger regulations in support of widely-held moral intuitions. And in an era of rapidly changing technology often in tension with moral intuitions that evolved in prior eras, that may result in substantial limitations on tech. Sibling star alien world governments might ban advanced artificial intelligence, brain emulations, or nuclear-powered space ships. They might also insist on preserving their biological bodies.
By using strong surveillance, embedded political officers, and using the threat of destruction from a distance, a world government might hope to keep control over a rapidly expanding sphere of interstellar colonies. But surely such control is far easier if substantial interstellar colonies are simply banned. Independent colonies would threaten not only the relative status of the current world government leaders and polity, but they’d also threaten to allow evasion of morally-treasured regulations.
Thus aliens with a world government might limit expansion, and also tech, not just to support environmental and anti-colonialism type ideologies, but also to preserve the relative status of locals and their ability to impose civilization-wide regulations. We have often seen similar behavior in human history, when secure isolated local regimes have discouraged contact with outsiders. The fact that aliens have not yet destroyed us also suggests that they have moral ideologies.
In a very long-lived civilization with a stable world government, the high-level organization of government and its key principles and regulations might become so stable that other structures of that civilization evolve to match them more often than the government evolves to match other structures. The government becomes like a mountain, where life adapts to behave differently at the mountain’s foot versus near its peak. So over millions of years the intuitions and practices of individuals and local groups may well evolve differently to match different parts of the stable world government with which they most strongly interact.
And if their world is more stable than our ancestors’ worlds have been, their minds might become less general, being adapted more to a particular range of situations.
4. Complex System Rot – Since the origin of life, competition has been the main driver of adaptation and innovation. Yes, cooperation has been important, but it is competition that has designed and promoted cooperation. While genetic forms of competition once dominated, cultural competition now matters more. Individuals and their practices compete within organizations, while organizations and their practices also compete for members, customers, investors, and more.
Across this long history, individual organisms, species, human organizations, and even empires have consistently tended to “rot”. That is, their long-lasting materials and structures slowly decay, becoming less flexible or general and more fragile, until they simply die or are eaten or displaced by rivals. This continues to happen today even with software and legal systems, as they try to adapt to new circumstances, and it happens even when their materials do not decay. It is competition that has corrected for this tendency to rot, by ensuring that simpler more general robust structures are available to replace failing fragile versions.
A civilization lasting for millions of years with a stable ideological world government preventing most expansion and tech innovation seems to me a recipe for high level system rot. New agencies, rules, and regulations would slowly accumulate on top of old ones, instead of being sufficiently culled, refactored, or reorganized. Agency growth and changes would have been often made to suit local ambitions instead of external needs, often using newly invented moral imperatives.
In our limited Earth history, we have often seen spectacular waste by stable secure empires, religious authorities, and secure monopolistic firms. Each example has found ways to spin stories justifying its waste, stories accepted by many observers. Many observers have also often believed decaying organizations who claimed that they had not yet lost any flexibility or generality, claims only clearly disproved when they were displaced by rival competitors.
Over millions of years, an ancient alien world government would accumulate far worse wasteful habits, and yet always offer semi-plausible if tortured justifications, stories not yet clearly disproved by competitors, who are not allowed to exist. Such governments would proudly tell themselves that they are still flexible and general, and up to most any challenge. But they’d be lying to themselves.
5. Putting it all together – So here is my best scenario to explain UFOs as aliens. I’m not saying it is good enough to let us believe that some UFOs are more likely than not aliens. I’m just saying that it is the best I’ve been able to come up with. You judge how good.
Life started long ago on Eden, which then seeded both Earth and our siblings’ home planet. Their home is somewhere in our galaxy, and yet they are the nearest advanced aliens for a million galaxies. For many millions of years, they’ve had a stable world government enforcing ideologically-justified regulations limiting expansion, tech innovation, and perhaps much more. Local intuitions and practices have long since adapted to this stable mountain; it feels to them very legitimate.
This world government made an exception to its expansion bans to allow trips to sibling planets hosting life, and allowed the development of whatever tech that required. This was done in support of key ideologies, which is probably why they haven’t destroyed us, and yet they plan to make sure we obey their regulations on expansion and tech innovation. And data on us may help prepare them to meet other aliens. (They may or may not believe they will eventually meet future grabby aliens.)
This long trip, and their management of Earth, is a task calling for great generality and flexibility, which their government mostly lacks, though it claims otherwise. Worse, their fear of allowing an expansion escape led them to tightly control this expedition. So most key choices have been made ahead of time, and aliens here at Earth are kept on a tight leash, dependent on resources and equipment shipped from home, and on tight rules of engagement.
These aliens long ago made their plans for how to monitor Earth civilization, and how to control it if that became necessary, and they built and shipped equipment and resources here based on that plan. Local alien administrators here have little discretion, are watched by local political officers, and have very limited abilities to make equipment or to collect resources beyond their pre-anticipated needs.
In drug regulation on Earth today, we have an ideology wherein conclusions drawn from observations are declared insufficient; one must also have proper government-managed “experiments.” If these aliens have a similar epistemological ideology, they would plan to observe Earth hidden safely from a distance, but they’d also need to periodically “poke” the locals and watch reactions. Alternatively, they might have an ideology of “touch”, wherein they couldn’t in good conscious control us unless they had before touched us “directly” somehow.
So, maybe, this was the safest most robust plan they could come up with to touch/poke us, when planning long ago back on their home planet: They poke us via making local disturbances in air or water usually by sending dark beams from safely hidden orbital projectors.
At a controlled distance, these beams can cause glowing balls of air, or smooth surfaces. (These can cause radar reflections, burn marks on the ground, and even sounds.) Their orbiting projectors would be safe from retaliation by Earthlings who would at first not even notice the beams, and who later would find it hard to trace those beams back to their orbital origins. And even finding those projectors probably doesn’t find the weapons by which they stand ready to destroy us if we get out of control.
So long ago these aliens sent to Earth equipment for installing telescopes, beam projectors, and weapons in orbit around Earth. All supplied with energy, covered to remain unseen, and with supports to keep them running for eons. And the main thing that aliens have done since their arrival is to maintain these facilities, and process the info collected.
These local administrators send regular and positive TPS reports back home regardless of how well things are actually going. The local aliens are likely bad at interpreting all the info they collect, their home world is bad at judging the quality of their efforts, and also at incentivizing such efforts. Thus maybe they not have learned much so far, and may not even be able to understand our electronic traffic that they can hear from space. They may not have detectors on the ground tapping into communications here. Perhaps they don’t even have language among themselves, and so aren’t capable of understanding our talk.
Maybe the fastest that their economy grew back on their home planet, before it slowed down due to regulations, was much slower than the Earth economy is growing now. So they never really planned much for how to react to the rapid change that we are undergoing here now. Local administrators keep sending TPS reports back home, doing the scheduled UFO projector runs, and keeping their fingers nervously on their weapons buttons. But like most government administrators, they are terrified of having to take the initiative to make a big decision, and so would rather wait until the choice becomes completely obvious.
If this all sounds implausibly incompetent to you, consider that if many UFOs are in fact aliens, the U.S. military and many militaries around the world have in fact been spectacularly incompetent at considering UFO reports and studying the threats that they imply. Yet these militaries existed in an era of competition and a burst of UFO reports started during a major war (WII) during which militaries had rapidly evolved to become unusually competent. Imagine how worse would be a military with a secure budget but no actual war for a million years. (Note that UFO activists have also been spectacularly incompetent in many ways.)
You might think that all this alien incompetence would give humans a fighting chance to defy these aliens and break out of their control. Possibly, but probably not. They probably do have their finger on the big kill-all-humans button, and that button probably does actually work. We might have a chance to sneak off and start a very distant stealthy interstellar colony, but that also seems damn hard.
But if the aliens behind UFOs are incompetent at understanding us and communicating with us, that sounds like bad news for our ability to learn and abide by their rules. It would be nice if they had some effective plan for integrating us into their world, beyond just pushing the button on us when we cross some line. But I wouldn’t count on that.
Note that not all of the elements of the story I’ve just told are strictly necessary to explain the stylized facts I’ve outlined. Those extra story elements are indicated by words like “maybe”, and are added to help you see how this story might be realized. If you don’t like my story, what story would you tell to explain all these stylized facts, not just one or two?
Note also that I’ve told this story thrice before, though this version is more elaborate.
Posted on 2020-05-09
[US] Department of Defense formally released three Navy videos that contain ‘unidentified aerial phenomena.’ … When the videos were published in 2017 and 2018 by The New York Times …, they gave new hope to those looking for signs of extraterrestrial life. … ‘it’s time that we make progress to understand the extraordinary technology being observed during these events.’ (More) Still, when you run all the arguments through your mind, is it not possible to come away with an estimate of at least a one-in-a-thousand chance that alien visitations are a real thing? Even such a small chance would be worthy of more discussion. (More) Alexander Wendt, a professor of international relations at Ohio State University. Wendt is a giant in his field of IR theory, but in the past 15 years or so, he’s become an amateur ufologist. … ‘It’s possible they’ve been here all along. … They could just be intergalactic tourists. Maybe they’re looking for certain minerals. It could just be scientific curiosity. It could be that they’re extracting our DNA. I mean, who knows? I have no idea. All I know is that if they are here, they seem to be peaceful.’ (More)
In the above, two social scientists, economist Tyler Cowen and political scientist Alexander Wendt, say to take UFOs-as-aliens more seriously. But in a quick search I can’t find any serious social analysis of this hypothesis. I see studies of why humans might want to believe in aliens, or why they might have a taboo against considering aliens. But not an analysis of alien social behavior, to help us evaluate the UFOs-as-aliens hypothesis. What I find is mostly like the above, “who knows?” So let me try.
To do Bayesian inference well, we need a set of not-crazy scenarios describing what might really be going on, we need a prior describing our beliefs about which of these scenarios seems how likely using our background knowledge, we need some more specific data to consider, and we need likelihood functions that say how likely each piece of specific data would be given each scenario.
Note: to study priors and likliehoods, I’ll need to make some assumptions, and see where they lead. That doesn’t mean I actually believe them.
While many UFO reports can be easily dismissed, a remnant of reports seems harder to dismiss, apparently showing artificial physical objects in the sky with amazing velocities and accelerations, but without the usual physical effects on nearby things.
Regarding these puzzling UFOs, I see three key explanation categories:
These three alternatives don’t obviously exhaust all options, but then again I can’t really think of much else.
Assuming that third scenario, hidden groups whose history and features are not much integrated with ours, we can confidently conclude that they most likely arose long ago and far away. Otherwise their space-time correlation with us would be an unlikely coincidence. Perhaps we and they arose from stars in the same stellar nursery, or Earth life was seeded by them, but that still leaves huge relevant durations and distances. And these pretty strongly support their having spectacular technologies and capacities. They have progressed and innovated for many millions and perhaps billions of years more than we. So they can travel very long distances, and survive very long durations.
Now it isn’t at all crazy to expect that many alien powers might arise over the scope and history of the universe. Our prior there has to start out high. But it is a bit more surprising that over billions of years this hasn’t resulted in visible changes to the universe we see. Somehow, all these advanced aliens have not widely rearranged galaxies, deconstructed stars, and so on. Once we condition on the “great filter” fact that we don’t see aliens out there, it become much less clear how likely we should consider aliens to be, especially aliens capable of and inclined to come near us. But that scenario also isn’t obviously impossible, so let us continue.
To consider UFOs-as-aliens, we must consider ancient aliens who were once very far away long ago, had spectacular tech and capacities, did not visibly change the universe, eventually traveled to here now, and are doing stuff around here now. The most likely scenarios consistent with that description tend to have those aliens be clearly visible around here. They’d be living near, building things, using local resources, dumping trash, fighting with each other, etc. But they are not clearly visible. So we must downgrade our prior again, perhaps a lot, to consider scenarios where active local aliens are clearly visible neither on cosmic scales nor on local scales.
For example, perhaps these aliens have found other attractive resources somewhere else hard-to-see nearby, perhaps dark matter or another dimension, resources so much more attractive than ours that they see no point in using the stuff we see. (But then why do their UFOs come here and interact with our matter?) Or perhaps they’ve coordinated to make our region into a nature preserve, not to be used much. Or perhaps they want to observe Earth and human evolution untouched and uninfluenced. Not crazy scenarios, but also not obviously the most likely ones consistent with our prior knowledge.
We have so far had to cut down our aliens prior to account for the lack of clear alien visibility at both cosmic and local scales. But we still have at least one more puzzling data point to integrate into our analysis: these aliens are sometimes somewhat visible as UFOs. Surely such advanced aliens are well aware of our existence, and can figure out roughly what we can see and infer about them. So either they are purposely allowing us to see glimpses of them in this way, or they are failing to prevent such glimpses.
So far, everything I’ve said I’ve heard before from others. Now come my original points, which I haven’t heard from elsewhere, though I wouldn’t be surprised if others have said them. (Far more is written on this than I have time to survey, as I lack good quality filters in this area.) Under either of these scenarios, purposeful or accidental revelation, it isn’t obvious that UFOs would be the only or main channel of such revelations to us about aliens.
If UFOs are shown to us on purpose, to influence our society in some way via a weak suspicion of local aliens, surely such capable aliens would also have a great many other way to influence us. And it is hard to imagine a purpose, or ability package, which would limit their influence to letting us see UFOs. They could edit our DNA, start pandemics or earthquakes, whisper hints to key leaders or innovators, kill off opponents, etc. And while they might be able to do all these things in ways that remain quite hidden, they could also work less hard to hide, and let some of us sometimes get glimpses of their influence.
Perhaps the fact that we see strange UFO behavior is due to accidental failures to sufficiently monitor or incentivize local alien actors who would otherwise want to influence us. Their abilities to prevent such failures would be quite good, but not quite perfect. But if so, similar failures could also allow local aliens to influence us in other ways. On our end, perhaps editing DNA, whispering hits, etc. On their end, they must at some points collect materials and energy sources, stay at home locations, and discard trash.
So under both types of scenarios, if UFOs are due to aliens we should also expect to sometimes see rare but striking alien influences in many other domains. Thus we should be able to get data to confirm or refute this UFOs-as-aliens theory by looking at many other areas of life, not just at strange objects in the sky. (Or in sea, caves, forests, and other sparse places.)
Sure, it is logically possible that aliens intend for us to see them only via strange sky objects. But our prior doesn’t at all favor that, even after modification to condition on low visibility at cosmic and local scales. So a lack of apparent alien influence in many other areas of life must count as evidence against the UFOs-as-aliens scenarios, both the purposeful-but-weak and the barely-accidental versions. Conversely, UFOs-as-aliens would be confirmed by a consistent pattern of rare but striking hard-to-explain influences in other areas of life, influences that aliens might plausibly want to cause.
I am somewhat of a polymath, pursuing a wider range of areas and topics than do most intellectuals or social scientists. So I consider myself to be more qualified than most to consider the possibility of strange influences on human behavior. And while I have in fact seen many strange things, for almost none does alien influence seem especially helpful in explaining what we see.
Now you might argue that aliens want to limit their purposeful revelations to one main most-effective area, or that due to varying costs of coordination and enforcement, one main area will end up being the hardest to control, and thus the area where the most accidental revelations occur. So why couldn’t strange stuff in the sky be that main area in either case?
Yes, that’s not crazy. But assume then that aliens are trying hard to just barely weakly reveal themselves in only one area, or that they are trying hard to prevent us from seeing them but just failing a bit in one worse area. Neither of these scenarios offer much encouragement for more careful analysis of this UFOs-as-aliens theory. In both cases, aliens are controlling how much we see, and so can plausibly quickly adjust their efforts to hide better if we surprise them with being more perceptive than expected. And if we are less perceptive than expected, they can relax their efforts a bit, to make it easier for us to see.
This is somewhat like the problem of inferring that we live in a sim via errors in the sim. If we lived in a sim, and the people running it could see us noticing errors, then they cold stop the sim at that point, back it up, and restart after putting more effort into cutting errors. So we’d only remember errors if they wanted us to remember them. In this scenario, if we just barely sometimes notice errors that we are not very sure are errors, our putting more effort into studying possible sim errors would only be rewarded by stronger efforts on their parts to hide their errors. That is, if there really are gods around who don’t want us to easily see them, but who sometimes reveal themselves to some of us, we can’t gain much by trying to together better analyze our shared data, to see if they exist. They can control what we see, and control us more directly, in enough ways that we will only know and see what they want us to know and see. Yes, okay, maybe they intend to reward us by revealing themselves to us, but only after we do a good enough collective analysis of our data. But really, given all the other plausible motives and priorities that they might have, our prior has to count that as a quite unlikely scenario. Most likely, when they want us to see them, we’ll see them, but not before.
Yes, aliens might just happen to be at the edge of detectability to us, but not due to efforts on their part to prevent or encourage detection. Yet if that edge region in detectability space is narrow, then it seems that our relative prior on that scenario should be low. The fact that UFOs have remained near our edge of detectability for 80 years of improving sensor tech and increasing sensor density also weakly suggests that more than coincidence is at work here. However, an increasing taboo against UFO-as-aliens may be an adequate explanation for this, and the edge region of detectability space may not in fact be narrow.
Of course even more likely, perhaps, no nearby aliens cause UFOs. But if they do, the best hypothesis, for its combo of likelihood and productivity, seems to be aliens who can travel very far in space and time, who sometimes travel near us, but who care little about us or the types of resources that we can see or use. They do visit here sometimes, where we sometimes meet them accidentally. The rest of us only hear of such meetings when our taboo against reporting them happens to be especially weak. Weirder than I expected, but then the universe has been weirder than we’ve expected before.
Added 5a: A creative scenario is humans finding & using ancient alien tech. Alas, the prior chances seem quite low that alien tech would be abandoned near us, found, still functional after this long, functional outside supporting resources of their civilization, useable by us, and still kept hidden.
Posted on 2021-06-04
Status seems pretty central to the UFO phenomena.
For example, reports have been filed on well over 100K encounters worldwide so far, but most of the books & movies on the topic focus on the same few cases. These cases are chosen in part for having more witnesses, detail, and physical evidence. But they seem especially chosen for having prestigious witnesses and locations. Seen by police or military workers, especially pilots. At military bases, especially housing nukes. These same books and movies are most eager to interview sympathetic people who are very high status, such as heads of state.
Similarly, many ancient legal systems had formal rules relating status to whose legal testimony to believe. And the social status of witnesses matters greatly today in court, even when there are no explicit rules requiring this.
Apparently most who witness UFOs as part of their job don’t report them, fearing reputation consequences. Because UFO fans are widely seen as very low status, at least among cultural elites. Similarly, organizations like police, militaries, airlines, and airports don’t want to be associated with such events, and so discourage reports by members. Unless some outside monitoring system discourages it, such orgs probably simply destroy such reports when they can.
When high officials have been asked privately why they would be reluctant to publicly admit to UFOs, they consistently say that the public rewards them for projecting ability and knowledge re their topic areas. UFOs require them instead to admit that they don’t know, and that there may be other parties around far more able than they. This effect is larger for police and militaries, compared to other agencies. And it is largest for the United States, at least during the period when it has been nearly the world’s dominant military power.
This all fits with several other militaries around the world releasing their UFO reports, long before the US has considered doing so. And it predicts that coming US release will be minimal, at least compared to the data the US could have and may have been collecting.
As a mildly elite academic, I can directly feel the status hit. If UFOs have an exotic intelligent cause, then we as a species have a lot less freedom than we thought to direct our destiny. And our governments, elites, and academics can do less to protect or inform us.
Yes, we might fund more UFO research, but I honestly don’t see the evidence situation changing that much for the indefinite future. Given how much data we already have, I don’t see more funding changing the overall data situation that much. These aren’t events you can seek out; you have to wait for them. And if there are intelligent exotic UFO causers here, they are clearly not eager to clearly show themselves.
And as long as the data situation remains ambiguous, I expect academic elites to remain adamant in dismissing exotic explanations. They’ll probably divert most funding they get on this topic to other topics they respect more. It will be hard to make much intellectual progress here, and those who do will be consistently slighted by academic elites. Even if society comes to accept UFOs more as a legitimate topic of investigation, elites will make very sure that the people who have so far championed this cause will not get more respect. Instead funding and respect will go to existing elites who deign to touch on the topic, at least from acceptable angles.
Status effects may even help explain some key features of UFO behavior. For example, among humans today, the response to an aggressive physical attack usually depends on how strong is the attacker relative to the defender. (The strengths of both sides’ allies are usually included in this calculation.) When the attacker is much stronger, the usual response is submission. And if they have similar strength, then the defender is likely to react vigorously.
However, if the attacker is far weaker, like a toddler attacking an adult, the usual response is to signal one’s strength by easily deflecting the attack, with little harm to either side. And in the reports I’ve read, this seems to be the usual reaction of UFOs to human attack: easy deflection. Which seems to signal their awareness, intelligence, abilities, and status stance. Maybe they sometimes let us seem them just so they can dis us in this way.
Status effects might even explain their lack of communication. (If they exist, of course.) Often small nations are eager to “enter into talks” with big nations just for the status bump this gives; “they take us seriously, and include us among those who must be consulted”. Conversely, the ultimate status dunk is to refuse to talk to or about someone; you act as if they are as worthy of this as a gnat. Might this explain the otherwise-puzzling lack of direct communication from intelligent exotic causes of UFOs?
A perhaps related and more ominous possible reason for their lack of communication is that they expect this to lead to us asking them some awkward questions. About our history, their expectations about us, their previous behavior toward us, their future plans regarding us, etc. Often the simplest way to avoid having to answer awkward questions is simply refusing to talk. Maybe how they plan to treat us reflects their view of our relative status, and we might not react well to hearing this.
Posted on 2019-07-09
In some technology competitions, winners dominate strongly. For example, while gravel may cover a lot of roads if we count by surface area, if we weigh by vehicle miles traveled then asphalt strongly dominates as a road material. Also, while some buildings are cooled via fans and very thick walls, the vast majority of buildings in rich and hot places use air-conditioning. In addition, current versions of software systems also tend to dominate over old older versions. (E.g., Windows 10 over Windows 8.)
However, in many other technology competitions, older technologies remain widely used over long periods. Cities were invented ten thousand years ago, yet today only about half of the population lives in them. Cars, trains, boats, and planes have taken over much transportation, yet we still do plenty of walking. Steel has replaced wood in many structures, yet wood is still widely used. Fur, wool, and cotton aren’t used as often as they once were, but they are still quite common as clothing materials. E-books are now quite popular, but paper books sales are still growing.
Whether or not an old tech still retains wide areas of substantial use depends on the average advantage of the new tech, relative to the variation of that advantage across the environments where these techs are used, and the variation within each tech category. All else equal, the wider the range of environments, and the more diverse is each tech category, the longer that old tech should remain in wide use.
For example, compare the set of techs that start with the letter A (like asphalt) to the set that start with the letter G (like gravel). As these are relatively arbitrary sets that do not “cut nature at its joints”, there is wide diversity within each category, and each set is all applied to a wide range of environments. This makes it quite unlikely that one of these sets will strongly dominate the other.
Note that techs that tend to dominate strongly, like asphalt, air-conditioning, and new software versions, more often appear as a lumpy change, e.g., all at once, rather than via a slow accumulation of many changes. That is, they more often result from one or a few key innovations, and have some simple essential commonality. In contrast, techs that have more internal variety and structure tend more to result from the accumulation of more smaller innovations.
Now consider the competition between humans and computers for mental work. Today human brains earn more than half of world income, far more than the costs of computer hardware and software. But over time, artificial hardware and software have been improving, and slowly commanding larger fractions. Eventually this could become a majority. And a key question is then: how quickly might computers come to dominate overwhelmingly, doing virtually all mental work?
On the one hand, the ranges here are truly enormous. We are talking about all mental work, which covers a very wide of environments. And not only do humans vary widely in abilities and inclinations, but computer systems seem to encompass an even wider range of designs and approaches. And many of these are quite complex systems. These facts together suggest that the older tech of human brains could last quite a long time (relative of course to relevant timescales) after computers came to do the majority of tasks (weighted by income), and that the change over that period could be relatively gradual.
For an analogy, consider the space of all possible non-mental work. While machines have surely been displacing humans for a long time in this area, we still do many important tasks “by hand”, and overall change has been pretty steady for a long time period. This change looked nothing like a single “general” machine taking over all the non-mental tasks all at once.
On the other hand, human minds are today stuck in old bio hardware that isn’t improving much, while artificial computer hardware has long been improving rapidly. Both these states, of hardware being stuck and improving fast, have been relatively uniform within each category and across environments. As a result, this hardware advantage might plausibly overwhelm software variety to make humans quickly lose most everywhere.
However, eventually brain emulations (i.e. “ems”) should be possible, after which artificial software would no longer have a hardware advantage over brain software; they would both have access to the same hardware. (As ems are an all-or-nothing tech that quite closely substitutes for humans and yet can have a huge hardware advantage, ems should displace most all humans over a short period.) At that point, the broad variety of mental task environments, and of approaches to both artificial and em software, suggests that ems many well stay competitive on many job tasks, and that this status might last a long time, with change being gradual.
Note also that as ems should soon become much cheaper than humans, the introduction of ems should initially cause a big reversion, wherein ems take back many of the mental job tasks that humans had recently lost to computers.
In January I posted a theoretical account that adds to this expectation. It explains why we should expect brain software to be a marvel of integration and abstraction, relative to the stronger reliance on modularity that we see in artificial software, a reliance that allows those systems to be smaller and faster built, but also causes them to rot faster. This account suggests that for a long time it would take unrealistically large investments for artificial software to learn to be as good as brain software on the tasks where brains excel. A contrary view often expressed is that at some point someone will “invent” AGI (= Artificial General Intelligence). Not that society will eventually have broadly capable and thus general systems as a result of the world economy slowly collecting many specific tools and abilities over a long time. But that instead a particular research team somewhere will discover one or a few key insights that allow that team to quickly create a system that can do most all mental tasks much better than all the other systems, both human and artificial, in the world at that moment. This insight might quickly spread to other teams, or it might be hoarded to give this team great relative power.
Yes, under this sort of scenario it becomes more plausible that artificial software will either quickly displace humans on most all jobs, or do the same to ems if they exist at the time. But it is this scenario that I have repeatedly argued is pretty crazy. (Not impossible, but crazy enough that only a small minority should assume or explore it.) While the lumpiness of innovation that we’ve seen so far in computer science has been modest and not out of line with most other research fields, this crazy view postulates an enormously lumpy innovation, far out of line with anything we’ve seen in a long while. We have no good reason to believe that such a thing is at all likely. If we presume that no one team will ever invent AGI, it becomes far more plausible that there will still be plenty of jobs tasks for ems to do, whenever ems show up. Even if working ems only collect 10% of world income soon after ems appear, the scenario I laid out in my book Age of Em is still pretty relevant. That scenario is actually pretty robust to such variations. As a result of thinking about these considerations, I’m now much more confident that the Age of Em will happen.
In Age of Em, I said:
Conditional on my key assumptions, I expect at least 30 percent of future situations to be usefully informed by my analysis. Unconditionally, I expect at least 5 percent.
I now estimate an unconditional 80% chance of it being a useful guide, and so will happily take bets based on a 50-50 chance estimate. My claim is something like:
Within the first D econ doublings after ems are as cheap as the median human worker, there will be a period where >X% of world income is paid for em work. And during that period Age of Em will be a useful guide to that world.
Note that this analysis suggests that while the arrival of ems might cause a relatively sudden and disruptive transition, the improvement of other artificial software would likely be more gradual. While overall rates of growth and change should increase as a larger fraction of the means of production comes to be made in factories, the risk is low of a sudden AI advance relative to that overall rate of change. Those concerned about risks caused by AI changes can more reasonably wait until we see clearer signs of problems.
Posted on 2020-01-22
You don’t want to die. If you heard that an asteroid would soon destroy a vast area around your home, you’d pay great costs to help you and your loved ones try to move. Even if you’d probably fail, even of most of your loved ones might not make it, and even if success meant adapting to a strange world far from home. If that’s not you, then this post isn’t for you.
Okay, you think you don’t want to die. But what exactly does that mean?
“You” are the time sequence of mental states that results from a certain large signal processing system: your “brain.” Each small part in this system takes signals in from other parts, changes its local state in response, and then sends signals out to other parts. At the border of this system, signals come in from “sensors”, e.g., eyes, and are sent out to “actuators”, e.g., hands.
You have differing mental states when these signals are different, and you live only as long as these signals keep moving. As best we can tell, from all the evidence we’ve ever seen, when these signals stop, you stop. When they stop for good, you die. As your brain is made out of completely ordinary materials undergoing quite well understood physical processes, all that’s left to be you is the pattern of your brain signals. That’s you; when that stops, you stop. (So yes, patterns feel.)
This signal processing system that produces you happens to be an organ of a particular complex multi-cellular animal: a “human”. This animal keeps working as long as all its organs keep providing their functions to each other. Your brain provides the function of monitoring and controlling high level decisions in this animal. Other organs provide functions like structure (bones), force (muscles), resource and waste distribution (blood), gas intake and disposal (lungs), light to signal conversion (eyes), and so on.
Given peace and prosperity, thankfully the norm for us today, the death of a multi-cellular animal like you typically results from the worn-down failure of just as one key part in one key organ, usually not the brain. At that moment of death, all the other organs, and all other parts of that failing organ, are still working acceptably. Unlike the artificial systems that we make today, natural animal systems usually can’t deal with broken parts by swapping in identical replacements from large stockpiles.
If you lived like most animals, you’d have to rely almost completely on the natural abilities of your organs to repair failing parts and to use redundant parts to replace failing ones. However, you, lucky you, are part of an advanced technical civilization, which offers many additional ways to help prevent and fix organ failures. For example, we can sometimes artificially clean out parasites and random garbage that gets in the way.
And at other times, we can substitute artificial parts and systems to replace failing natural organs. For example, we can ingest chemicals made in factories when our organs fail to make those chemicals. We have made artificial bones to give structure, artificial lungs to exchange gas, artificial hearts to pump blood, and so on. We’ve even made artificial extensions of some signal systems, such as artificial eyes and ears and artificial limb controllers.
Of course there are many organs for which we have not yet made artificial substitutes. But progress is rapid, and we have great reason for hope in the long run. Our civilization has come very far in a relatively short time, and it is quite reasonable to hope that within a few thousand years at most our descendants will have vastly greater capabilities. There’s no good reason to think we can’t eventually make artificial substitutes for all human organ systems. Yes, our civilization might destroy itself, or halt its growth, but there’s a good chance we won’t.
Now you might wonder: how is it even possible for us today to create artificial substitutes for natural organs? After all, our bodies are incredibly complex and detailed on atomic scales, yet we stand meters tall. If our artificial substitutes had to have a complexity that matched our existing complexity in fine detail over the entire scale of our bodies, then to design artificial organs we’d need to understand and design systems of a complexity vastly beyond anything we have ever managed.
What saves us is modularity. Evolution also had a limited ability to handle design complexity. So it used a modular system design. When there are only a few key organs systems, and when each organ system provides only a few main functions, then design variations within each organ can focus on how they effect the key functions of that organ, and mostly ignore other functions. And that means that our artificial design efforts can also focus mainly on how our replacements effect those key functions.
In addition, there’s the key fact that we are multi-cellular animals. Long ago, evolution spent a very long time honing single-celled creatures to survive and thrive on their own in the wild. Those designs became very subtle and efficient, but after a while because also hard to greatly change. When ecological niches opened up later for much larger organisms, evolution couldn’t be bothered to redo single cell type designs for much larger scales. That would require re-thinking many issues through nearly from scratch. Evolution instead just pasted many small cells together to make large organisms. So now each of our organs is mostly made up of many small cells pasted together.
In our organs, each cell is modified only a bit to allow it to serve its key function of allowing its organ system to provide its key functions. The vast majority of the complexity of each cell is mostly left alone. This is enormously wasteful in the sense that most of that cell complexity isn’t actually very useful in helping that cell to do what it needs to as part of that organ. But evolution found it easier to mostly just leave that cell package alone, rather than to try to greatly redesign cells for their roles in differing body systems.
The net result is that our bodies, and our organs, are actually much simpler than all that atom-scale complexity suggests. (Not simple of course, but simpler.) Which is why we can often design artificial replacements for key organs, to let us live longer before the failure of some part of an organ keeps our whole body from working, and thus killing the brain which produces the sequence of mental statues which is you.
Consider now the possibility of an artificial replacement for the brain organ. If we could make a work-alike for the brain, we might put it into a natural body, to prevent brain failures from killing that whole body. But if the main thing you really care about is your brain, where you live, then a much more dramatic anti-dying solution presents itself: replacing all the other body organs with artificial versions. Once you can make an artificial brain, that’s actually much easier than replacing one organ at a time. With artificial eyes, ears, muscles, energy sources, etc. we might make artificial bodies that can last as long as our artificial machines do today: indefinitely.
Today we can keep large complex systems like cars and buildings working indefinitely, by replacing parts as they break. Of course we don’t always do that, as we often find it cheaper to toss old systems and make new systems from scratch. We can even keep very complex computer systems working indefinitely, by making archive copies of files, and moving files to new computers as old ones die or become obsolete. With artificial bodies, we might similarly keep your mind going via replacing other body parts and swapping your mind to new brain hardware as needed. Not forever, but plausibly a lot longer than the current limit of roughly a subjective century.
So what are the prospects for making artificial brains? Unless brains are quite strange compared to other organs, we expect that most of the complexity in each brain cell is not very relevant for the function that cell provides the rest of the brain. Evolution searched for and found modest modifications of existing cell designs, modifications which let cells provide their key brain functions: sending signals to each other, and storing internal states that change in response to those signals.
When humans have designed signal processing systems, we have usually tried hard to insulate (in signal terms) the degrees of freedom in such systems that represent signals and states from all the other physical degrees of freedom in those systems. For example, in electronics we have wires and devices separated by insulators. It makes sense for evolution to also try this, for similar design reasons. So when we find some signals and states in the brain, we should be able follow them to other strongly connected signals and states, and mostly ignore the much larger and more complex relatively isolated nearby parts of the system. This is like following the wires in a piece of electronics, allowing one to ignore all the parts of the system not collected by something like wires to the wired parts. This strategy usually works great for electronics, and when applied to our bodies has already enabled us to create artificial eyes and ears.
So far, we have started in on this wire-tracing task regarding brains, but are far from having finished. In fact, completion may take a century or more. We can clearly see some signals and states, and have traced them to some others, but have not obviously found them all, nor worked out all the local transformation mappings that map incoming signals to changed states and outgoing signals.
Thus it seems reasonable to expect that we will eventually be able to “follow the wires”, and figure out where human brains store their signals and states relevant for brain functions, and learn (to a sufficient accuracy) the signal-state-signal transformation mappings for all the different parts of the brain. Knowing this could enable us to scan a particular functioning human brain, seeing which kinds of cells are where connected to what, and create a work-alike of that particular brain in artificial hardware. If that was your brain, with your signal patterns, then then new artificial brain would be you as well. (And perhaps the only you remaining if the scanning process destroyed your original brain.) Once you were in an artificial brain, how long til you died would become an economic question, not a biological one. You could live for as long as you could afford new replacement parts, and remained part of a civilization available to sell you such parts, and to protect you from predation. To someone like you, who doesn’t want to die, that should seem a very intriguing prospect.
Furthermore, the facts that you are a multi-cellular animal with a body design based on modular organs suggests that this isn’t some distant future in billions of years after inconceivable growth; it could happen in the next few centuries. Brains are probably more complex than the other organs for which we have already created artificial substitutes, but not fantastically more complex. So the fact that we have created many artificial organs in the last century or so suggests we might we make artificial brains within a few centuries. Of course the obvious problem for you today is that we don’t know how to make artificial brains today, and your existing body will probably fail well before the centuries it may take to figure out how to make artificial brains. So how does any of this help you not die soon?
Well, consider the fact that we’ve seen cold people where all the usual apparent brain activity has stopped, and yet they revived just fine when warmed up soon afterward. This tells us that the brain scans that will be needed to create artificial brains can mostly ignore contributions to temporary brain activity. Scans just need to see the more permanent structures that channel temporary activity.
And that raises this key question: does the relevant info that a brain scan will need to see you in your brain, to allow the creation of an artificial brain holding you, does this key info survive the process of freezing your brain down to liquid nitrogen temperatures? At those temperatures, your brain is solid, and basically doesn’t change, for many centuries at least. So if the key info that specifies you survived that freezing process, why then we might freeze you now, keep you frozen for a long time, and then later when we know how to make artificial brains we could scan your frozen brain, grab that key info, and then make a new you in an artificial brain! That’s the relation between all this and your not dying.
Yes, it would cost some money to freeze you soon, to store you for centuries, and then to revive you later. Someone will have to pay for all that. And many things can go wrong over such timescales. Whatever contracts and organizations you support to enable this plan may fail. And even with success you’d come back to a strange different world, one that may lack many people you love. But the costs and risks would come way down if many people wanted this, and you said you are a person who wants to live, even if the chances are low and you’d have to live in a strange world.
We’ll talk more about these social issues in a bit, but before that we should engage this key info question: does the info that specifies you survive freezing your brain? We are pretty sure that the info that specifies you is there in your brain right now, encoded in the spatial patterns of densities of certain chemicals. (That is, we are pretty sure the needed info is not in orientations, spins, electric currents, magnetic fields, etc.) Perhaps many chemicals, though a tiny fraction of all the chemicals in your brain. So if we could suddenly and instantly freeze your brain, keeping all the chemicals in the same place, so that nothing moved afterward, then we’d be pretty sure to have saved the key info.
Unfortunately, changes do happen during the available freezing processes. Some unusual chemical reactions occur, and some things move around. However, we have many reasons to believe that enough info can be saved for it to be extractable from a frozen brain at a reasonable cost. Yes, in principle all info is saved in the universe, and could be regained by a powerful enough entity that could inspect the entire universe in great detail. But we don’t want to pin our hopes on such an extreme and unlikely-looking solution.
When we use our best current freezing tech, and then look via powerful microscopes at post-freezing tissues, most everything seems to be nearly the same, in nearly the same place. Also, we expect a lot of redundancy in brain design and encoding. We know that our organs are designed to function with great randomness in their detailed construction process. We seem to see a lot of redundancy in brain encoding, and know that the brain still functions under many big modifications, such as being whacked in the head or flooded with new chemicals. And we know that a modest rate of errors that require best-guess interpolation to fix seems tolerable; many things now change who we are over the decades, yet we still see ourselves as the same person enough to be worth preserving.
Okay, so it seems there’s a good chance that artificial brains will be possible within a few centuries, and so if we use our best current tech to freeze you today, and store that frozen brain for centuries, then future folks could make an artificial brain of you, and thus something close enough to you could be revived and live in that future world. Given that you don’t want to die, this plan should seem attractive to you.
Not just you, many millions like you. Sure, people vary in how much they don’t want to die, and in their beliefs on future tech progress and brain info freezing damage. But given what people say about how much they value life, a great many should find this attractive. At a typical value of life of five million dollars, then if this plan cost $50 thousand then only a 1% chance of success should be enough to make it seem worthwhile. Or if a strange future life was only worth a fifth of an ordinary life today, then a 5% chance of success should be enough.
Amazingly, a way to freeze your body when medicine gives up on you, and to preserve this frozen body for centuries with at least this chance of success, has been available at near this price for over six decades. Yet even though this option has regularly received free international publicity during this period, only about 3000 people have ever signed up for it, and only about 300 people have ever been frozen. Why so little interest?
One possible explanation is that this “cryonics” option has usually been framed around about a more ambitious plan: the full repair of your ordinary biological body, including all the damage done by freezing, and including the growing of a replacement body if they froze only your brain. While such full repair will probably be feasible eventually, this task seems much harder than creating an artificial brain and body. This really does require handling all the complexity of meter-scale and atomic-scale-detailed bodies, at least for the purpose of identifying failures and repairing them.
I’ve thought a lot about what life would look like given artificial brains, and in fact have written a whole book on it: The Age of Em. Since the artificial brain option seems to me better, and should be available much earlier, that’s the option I’d prefer. Even so, I don’t think we can explain very much of the lack of interest in cryonics in terms of this option being less highlighted so far. The full repair option isn’t crazy, and should still be attractive to people who don’t want to die.
In my other book, The Elephant in the Brain, I explain how we humans are often wrong about our motives. In particular, regarding medicine we think we care about health, yet we actually mainly care about showing that we care about others, and letting them show they care about us. As a result, far more of us actually die faster than we would under a system that was actually driven mainly by a concern for not dying. And yes that probably applies to you as well; you will probably die sooner than you would if your attitude toward medicine were driven mainly by your not wanting to die.
Perhaps your conscious mind is now screaming “No, that’s not me, I really don’t want to die.” But those thoughts aren’t very honest. They might feel sincere, but they aren’t integrated into a whole mental system that actually works that way. When thoughts of death come to your mind, your mind freezes and goes crazy just like everyone else’s, and can’t think death threats through very carefully. You also tell yourself that on something this important you don’t care much what others think, and you’d do whatever it takes even in the face of criticism and conformity pressures. But if your plan to not die faces strong opposition from loved ones, you’ll usually cave.
Regarding the option to not die that I’ve been discussing here, freezing to await artificial brains, your mind has either latched on to some lame excuse why this plan is infeasible or uninteresting, or you’ve told yourself yeah you do want to do this, and you’ll make a vague plan to do so, but you won’t actually do it. Perhaps your loved ones will object, and you’ll cave. Remember: a significant fraction of folks who hear about cryonics plan to sign up, but almost none ever do. (Amazingly, a similar number of people pay a similar amount to have their remains burned and their ashes thrown into space, an option produces far less hostility from loved ones.) Perhaps your mind is still screaming “No, that’s not me, I control my plans, and I don’t want to die!” Well then, go ahead, prove me wrong, and join the 3000. Make me proud of you. (That 3000 is only 10% of my Twitter followers; if I could double that number to 6000, I’d consider my life well spent.)
Is your doubt of the form, “But isn’t this selfish of me, to spend money on my not dying?” Note that we all supposedly spend lots on ourselves via medicine to not die. And as this cryonics product is dominated by fixed costs, the more who sign up to be frozen, the cheaper and more reliable this product can become. This product happens to be unpopular now, but no law of nature requires this. If we could switch to a new social equilibrium where people used cryonics to show that they care, then at a quite modest cost most everyone who is alive today need not die! Isn’t that a noble altruistic cause worth fighting for?
Look, large surveys show that the main reason people actually reject artificial brains is because they seem “unnatural”, the same reason people rejected IVF before that was possible, and yet now widely accept it:
People who value purity norms and have higher sexual disgust sensitivity are more inclined to condemn mind upload. Furthermore, people who are anxious about death and condemn suicidal acts were more accepting of mind upload. Finally, higher science fiction literacy and/or hobbyism strongly predicted approval of mind upload.
Whatever reasons you think you have in your head are mainly excuses for these factors. Will you really let yourself die because a purity norm now makes a thing as yet unseen seem unnatural to you?
Posted on 2019-01-21
The Question
We humans have been writing “code” for many decades now, and as “software eats the world” we will write a lot more. In addition, we can also think of the structures within each human brain as “code”, code that will also shape the future.
Today the code in our heads (and bodies) is stuck there, but eventually we will find ways to move this code to artificial hardware. At which point we can create the world of brain emulations that is the subject of my first book, Age of Em. From that point on, these two categories of code, and their descendant variations, will have near equal access to artificial hardware, and so will compete on relatively equal terms to take on many code roles. System designers will have to choose which kind of code to use to control each particular system. When designers choose between different types of code, they must ask themselves: which kinds of code are more cost-effective in which kinds of applications? In a competitive future world, the answer to this question may be the main factor that decides the fraction of resources devoted to running human-like minds. So to help us envision such a competitive future, we should also ask: where will different kinds of code work better? (Yes, non-competitive futures may be possible, but harder to arrange than many imagine.) To think about which kinds of code win where, we need a basic theory that explains their key fundamental differences. You might have thought that much has been written on this, but alas I can’t find much. I do sometimes come across people who think it obvious that human brain code can’t possibly compete well anywhere, though they rarely explain their reasoning much. As this claim isn’t obvious to me, I’ve been trying to think about this key question of which kinds of code wins where. In the following, I’ll outline what I’ve come up with. But I still hope someone will point me to useful analyses that I’ve missed.
In the following, I will first summarize a few simple differences between human brain code and other code, then offer a deeper account of these differences, then suggest an empirical test of this account, and finally consider what these differences suggest for which kinds of code will be more cost-effective where.
Differences
The code in our heads is the product of learning over our lifetimes, inside a biological brain system that has evolved for eons. Though brain code was designed mainly for old problems and environments, it represents an enormous investment into a search for useful code. (Even if some parts seem simple, the whole system is not.) In contrast, the artificial code that we’ve been writing started from almost nothing a few decades ago. Our brain code seems to come in a big tangled package that cannot easily be broken into smaller units that can usefully function independently. While it has identifiable parts, connections are dense everywhere; brains seem less modular than artificial code. Relatedly, brains seem much more robust to local damage, perhaps in part via having more redundancy. Brains seem designed for the case where communication is relatively fast and cheap within a brain, but between brains it is far more expensive, slow, and unreliable.
The code in our head does not take much advantage of many distinctions that we often use in artificial code. In our artificial systems, we gain many advantages by separating hardware from software, learning from doing, memory from processing, and memory addresses from content. But it seems that evolution just couldn’t find a way to represent and act on such distinctions.
Artificial code seems to “rot” more quickly. That is, as we adapt it to changing conditions, it becomes more fragile and harder to usefully change. As a result, most of the code we now use is not an edited version of the first code that accomplished each task. Instead, we repeatedly re-write similar systems over from scratch. In contrast, while the aspects of brain code that we learn over a lifetime do seem to slowly rot, in that we become less mentally flexible with age, we see little evidence of rot in the older evolved brain systems that we use to learn. Our brain code is designed for hardware with many parallel but slow computing units, while our artificial code has so far mostly been designed for fewer fast and more sequential units. That is, brains calculate many things all at once slowly, while most artificial code calculates one thing at a time.
Our brains do some pre-determined tasks quickly in parallel. Such as simultaneously recognizing both visual and auditory signs of a predator. However, when we humans work on the tasks that most display our versatile generality, the sort of tasks that are most likely to matter in the future, our brains mostly function both slowly and sequentially. That is, we accomplish such tasks by proceeding step by step, and at each step our whole brain works in parallel by adding up many small contributions to that step.
Even so, the power and generality that often results from this process is truly stunning, being far beyond anything we know how to achieve with artificial code, no matter how much hardware we use and how many man-years we devote to writing it. This generality is why humans brains still earn the vast majority of “wages” in our economy. Artificial code is quite useful but gets paid much less in the aggregate, and in this sense is still far less useful than brain code. (“AI” software, i.e., artificial software intended to more effectively mimic brain abilities, earns a much smaller fraction of aggregate wages.)
While the code in our heads resulted largely from simple variation and selection of whole organisms, human brains use more directed processes to generate the artificial code. For example, one common procedure is to repeatedly have a brain imagine the results of particular small sections of code being executed in particular contexts, and search for edits to code such that imagined executions produce desired outcomes. This is commonly interleaved with less frequent actual execution of trial code on test cases. This process works better with more modular code expressed in terms of logical concepts that have sharp boundaries and implications, as it is easier for our brains to predict what happens in such contexts. A few parts of artificial code are generated via statistical analysis of large datasets.
When we have invested lately in having a kind of code actually do a task, such investments tend to give an advantage to that kind of code in continuing to do that task. Also, when each type of code can more easily connect to, or coordinate with, other code of its type, each type of code gains an advantage at a task when more of the tasks that it must coordinate with are also done by the same type of code. Thus each type of code has momentum, in continuing on where it was, and it naturally clumps together, especially in the most highly clumped sections of the network of tasks. Easy Implications
So what do these various considerations imply about which kinds of code win where? We can identify some weak “all else equal” tendencies.
Brain code has at least a small temporary advantage on tasks that have been done by brain code lately, and that must coordinate with many other tasks done by brain code. The fact that that brain code requires an entire brain when being general suggests that artificial code is more cost-effective on small simple problems where brains do not have special abilities. At least when hardware costs to run code are important relative to costs to write code. Artificial code also seems much cheaper to run when a relatively simple sequential algorithm is sufficient, as brain code uses a whole brain to execute simple sequential computations. Artificial code also has advantages when lots of fast precise communication is desired across scopes larger than a brain.
The fact that brain code was designed for old problems suggests that it has advantages on old and long-lasting problems, relative to new problems. On the one hand, brain code being old and old systems being less flexible suggests using artificial code when great adaptability is required beyond the range for which brains were designed. On the other hand, the fact that artificial code rots more quickly suggests that artificial code has advantages when problems or contexts change quickly, which would force new code soon in any case, but disadvantages for stable long-lasting tasks.
In addition to these relatively simple and shallow implications, I have found a somewhat deeper perspective that seems useful. Let me explain.
Code Principles
Two key principles for managing code are: abstraction and modularity. With abstraction, you cut redundancy, via doing the same tasks but replacing many similar systems with a smaller number of more abstracted systems. Abstracted systems are smaller, in a code-length sense, though they may cost more in hardware to execute their code. By avoiding redundancy, abstracted systems make more efficient use of efforts to modify them.
With modularity, you try to limit the dependencies between subsystems, so that changes to one subsystem force fewer changes to other subsystems. Better ways to integrate and organize systems, which better “carve nature at its joints”, allow more effective modularity, and thus fewer design change cascades, wherein a change in one place forces many changes elsewhere.
It can take a lot of work to search for better design architectures that better facilitate modularity. Given the usual rate at which artificial code rots, only a limited amount of work here is justified. Sometimes systems that have partially rotted are “refactored”, by changing their high level architecture. Though expensive, such refactoring can cut the level of system rot, and is often cost-effective in terms of delaying a complete system rewrite. Better abstractions tend to promote better organization, which induces more effective more-slowly-rotting modularity.
A Deep Difference
Today, humans wanting code to do a task will first search for close-enough pre-existing code that might be lightly edited to do the job. Absent that, they will think for a bit, open a blank screen, and start typing. They will connect their new code to existing code as convenient, but will be wary of creating too many dependencies that reduce modularity. They will think and search for good ways to organize this new system, including good abstractions, but will put only limited effort into this. That is, to manage code complexity, humans tend to make new code that is highly modular, but not very well organized.
In contrast, as evolution designed and redesigned brains, it faced strong limits on the amount of brain hardware available. Brains take precious volume and are energetically expensive. And because evolution never managed to separate hardware and software, this hardware limit created strong limits on the amount of software possible. Limits far more restrictive than the memory limits imposed on humans who write code today. To add new software to brains, evolution could only a) add more hardware, b) delete old software, or c) seek more efficient representations and abstractions in order to save space.
Thus evolution just could not take the usual human approach of just opening a blank screen and writing new highly-modular but sloppily-organized code. Evolution instead had to keep searching hard for better ways to organize and integrate existing code, including better abstractions. This was a much more expensive process, but as it played out over eons it resulted in code that was much better organized and integrated, though less modular.
This perspective helps us to understand why brain code seems less modular than artificial code, why brain code doesn’t rot as fast and is more robust to damage, why it is harder to usefully break brain code into small units to do small tasks, why brain code is better at being more general, and why artificial code is more sequential. It also helps us understand why the usual focused processes for having brains make artificial code work reasonably well: brains know enough to predict how small chunks of code will behave, but only when that code is relatively modular.
This perspective also helps us to understand why abstraction is one of the brain’s key organizing principles. As I’ve said, human brains
collect things at similar levels of abstraction. The rear parts of our brains tend to focus more on small near concrete details while the front parts of our brain tend to focus on big far abstractions. In between, the degree of abstraction tends to change gradually.
Another important if less well understood organizing principle of brains is to separate a left and a right brain, perhaps to separate systems of credit assignment that don’t mix well. That is, to separate bottom-up processing that searches for fewer big things to explain many details, from top-down processing that searches for details to best achieve abstract goals. In sum, a deeper perspective can help us to understand how brain and artificial code differ, and thus which kinds of code can win where: Brain code is better integrated and abstracted, but less modular, than artificial code.
Code-Cubed
Let me suggest a way to test this perspective, via data that should already be available, but which I haven’t yet found. In addition to brain code written by evolution, and artificial code written by brains, we can also consider “code-cubed”, i.e., code that is written by artificial code. This is “cubed’ because it is written by artificial code, which we can think of as “code-squared”, which is written by brain code, which we can think of as plain code, which is written by a non-code evolutionary process. Such code-cubed can obviously be written much more quickly than can ordinary code, at least at low levels of quality. But how else does it differ?
A large well-integrated brain that focuses its whole effort on thinking about particular chunks of code should produce in that artificial code a substantial degree of coherence and integration, at least on the scale of those chucks. However, when more modular and less well integrated artificial code writes code, that resulting code-cubed should be less well integrated. And as artificial code can write code much faster than can humans, and has plenty of empty memory available, artificial code will be tempted all the more to rely on new code and modularity to help manage complexity in the code it writes.
Thus this perspective suggests that code-cubed is even more modular, less well organized, and rots more quickly, than ordinary artificial code written by humans. For example, when we change the source code or compiler for a system, we then typically re-execute that complier on the source code, in effect re-writing from scratch. We do this instead of trying to edit the old compiled code in order to match the new source or new compiler.
Thus when we want code that can be usefully modified over longer periods of change, we should prefer ordinary artificial code to code-cubed. We should also prefer this when it is important that the code writer had a wider more general understanding of the task to be performed, and the other systems with which it needs to integrate.
What Wins Where
So in a future world where all types of code have access to the same cheap artificial hardware, and where competition pushes each application to use the type of code that is most locally cost-effective there, where should we expect to find brain code, where artificial code, and where code-cubed?
Brain code represents an enormous investment into a large tangled but well integrated and abstracted package that is hard to understand and modify, but has so far shown unparalleled power when applied to stable general broad tasks. This [suggests](humans-cells-in-multicellular-future-minds) that brain code may have a long future in applications that play to its strengths. (Long at least [in terms](reply-to-christiano-on-ai-risk) of my usual favorite parameter: number of economic doubling times.) Yes, eventually fully artificial code may become comparably well-integrated, but if ems are possible before then, the descendants of brain code may then have become even better organized and designed.
Most human organizations today are hierarchical, with low level activity focused more on relatively narrow contexts that matter less, need faster responses, and evolve more rapidly. In contrast, higher level activity allows slower responses, has more stable considerations, and must consider broader scopes and implications. Most artificial code today is also hierarchical, with low level code that tends to have a more narrow focus and contexts that change more rapidly, compared to high level code that must consider a wider range of more stable contexts, inputs, and other systems. In both types of systems, lower level tasks are more naturally modular, depending on fewer other tasks.
As smaller more focused more modular tasks that change more rapidly are better suited to artificial code, while less modular tasks that must consider wider context are better suited to brain code, we should expect artificial code to be more common at low organization levels, and brain code to be more common at high organization levels. That is, brains will manage big pictures, while artificial code manages details.
Among the tasks that humans do today, we can also distinguish more vs. less [tangled](a-tangled-task-future) [tasks](connected-tasks-cities-win). Tangled tasks are closer to the more tangled center of a network of which tasks must coordinate with which other tasks. While tasks at higher organization levels do tend to be more tangled, some lower level tasks are also highly tangled. Brains also have advantages in these more tangled tasks, and once they are entrenched in many connected tasks are harder to displace from them.
## [Progeny Probabilities: Souls, Ems, Quantum](#table-of-contents)
_Posted on 2019-06-02_
Consider three kinds of ancestry trees: 1) souls of some odd human mothers, 2) ems and their copies, and 3) splitting quantum worlds. In each kind of tree, agents can ask themselves, “Which future version of me will I become?”
SOULS First, let’s start with some odd human mothers. A single uber-mother can give rise to a large tree of descendants via the mother relation. Each branch in the tree is a single person. The leaves of this tree are branches that lead to no more branches. In this case, leaves are either men, or they are women who never had children. When a mother looks back on her history, she sees a single chain of branches from the uber-mother root of the tree to her. All of those branches are mothers who had at least one child.
Now here is the odd part: imagine that some mothers see their personal historical chain as describing a singular soul being passed down through the generations. They believe that souls can be transferred but not created, and so that when a mother has more than one child, at most one of those children gets a soul.
Yes, this is an odd perspective to have regarding souls, but bear with me. Such an odd mother might wonder which one of her children will inherit her soul. Her beliefs about the answer to this question, and about other facts about this child, might be expressed in a subjective probability distribution. I will call such a distribution a “progeny prob”.
EMS Second, let’s consider ems, the subject of my book The Age of Em: Work, Love, and Life when Robots Rule the Earth. Ems don’t yet exist, but they might in the future. Each em is an emulation of a particular human brain, and it acts just like that human would in the same subjective situation, even though it actually runs on an artificial computer. Each em is part of an ancestry tree that starts with a root that resulted from scanning a particular human brain.
This em tree branches when copies are made of individual ems, and the leaves of this tree are copies that are erased. Ems vary in many ways, such as in how much wealth they own, how fast their minds run relative to humans, and how long they live before they end or next split into copies. Split events also differ, such as re how many copies are made, what social role each copy is planned to fill, and which copies get what part of the original’s wealth or friends.
An em who looks toward its next future split, and foresees a resulting set of copies, may ask themselves “Which one of those copies will I be?” Of course they will actually become all of those copies. But as human minds never evolved to anticipate splitting, ems may find it hard to think that way. The fact that ems remember only one chain of branches in the past can lead them to think in terms of continuing on in only one future branch. Em “progeny prob” beliefs about who they will become can also include predictions about life details of that copy, such as wealth or speed. These beliefs can also be conditional on particular plans made for this split, such as which copies plan to take which jobs.
QUANTUM Third, let’s consider quantum states, as seen from the many worlds perspective. We start with a large system of interest, a system that can include observers like humans and ems. This system begins in some “root” quantum state, and afterward experiences many “decoherence events”, with each such event aligned to a particular key parameter, like the spatial location of a particular atom. Soon after each such decoherence event, the total system state typically becomes closely approximated by a weighted sum of component states. Each component state is associated with a different value of the key parameter. Each subsystem of such a component state, including subsystems that describe the mental states of observers, have states that match this key parameter value. For example, if these observers “measured” the location of an atom, then each observer would have a mental state corresponding to their having observed the same particular location.
These different components of a quantum state sum can thus be seen as different “worlds”, wherein observers have different and diverging mental states. Decoherence events can thus be seen as events at which each quantum world “splits” into many child worlds. The total history starting from a root quantum state can be seen as a tree of states, with each state containing observers. And so a quantum history is in part a tree of observers. Each observer in this tree can look backward and see a chain of branches back to the root, with each branch holding a version of themselves. More versions of themselves live in other branches of this tree.
After a split, different quantum worlds have almost no interaction with each other. Which is why we never notice this quantum splitting process in the world around us. So observers typically never see any concrete evidence of that there exist other versions of themselves, other than their past versions in the chain from them now back in time to the root state. That is, we never see other quantum worlds. As observers see only a sequence of past versions of themselves, they can naturally expect to see that sequence continue into the future.
That is, observers typically ask “In the future, what will be the state of the one world, including the one version of my mind?” Even though in fact there will be many worlds, holding many versions of their minds. (Quantum frameworks other than many worlds struggle to find ways, usually awkward, to make this one future version claim actually true.) Beliefs about this “who will I be?” question are thus “progeny probs”, analogous to the beliefs that an em might have about which future copy they will become, or that an odd human mother might have on which future child inherits her soul.
The standard Born rule in quantum mechanics is usually expressed as such a progeny prob. It says that if the current state splits into a weighted sum of future states, one should expect to find oneself in each component of that sum with a chance proportional to the square of that state’s weight in the sum. This is a remarkably simple and context-independent rule. Technically, quantum states are vectors, and the Born rule uses the L2 norm for relative vector size. And a key question about many worlds quantum theory, perhaps the key question, is: from where comes this rule?
IN GENERAL These three cases, of human souls, em copies, and quantum worlds, all have a similar structure. While the real situation is a branching tree of agents, an agent who looks back to see a sequence of ancestors can be tempted to project that sequence forward, predict that they will become only one next descendant, and wonder what that descendant will be like. This temptation is especially strong in the quantum case, where agents never see any other part of the tree than their ancestor sequence, and so can fail to realize that a larger tree even exists.
An agent’s beliefs about which next descendant will “really” be them can be described by a probability distribution, which I’ve called a “progeny prob”. This gives the chance this agent will “really” become a particular descendant, conditional on the details of a situation. For ems, this chance may be conditional on each copy’s wealth, or speed, or job role. For quantum systems, this chance is often conditional on the value of the key parameter associated with a decoherence event.
In the rest of this (long) post, I make three points about progeny probs.
IS FICTION The first big thing to notice is that, for an agent who is a branch in some tree of agents, there is actually no truth of the matter regarding which future branch that agent will “really” be! They will become all descendant branches in that tree. So one of the most fundamental elements of quantum theory, the Born probability rule, is typically expressed in terms of an incoherent concept. Also incoherent is a big question that ems will often ask, “Who will I be next?”
However, even though progeny probs are in this sense fictional, we usually connect them to some very real data: the past sequence of ancestors we see up until today. Agents who believe that their past history was generated by the same sort of progeny prob that applies to their future should expect this history to be typical of sequences generated by such a progeny prob. This test has in fact been applied to the quantum progeny prob, which passes with high accuracy.
If one has has a detailed enough model of how a certain kind of ancestry tree of observers is generated, then one can use this tree model to predict a probability distribution over possible trees. Each such generated tree comes with a set of ancestor sequences, one for each branch in the tree. So given a distribution over trees, one can generate a distribution over ancestor sequences in these trees.
IS RELATIVE However, in order to take a tree model and generate a distribution over ancestor sequences, one needs to pick some relative branch weights, weights which say how much each branch counts relative to others in that tree. And the progeny prob that best fits this total distribution of ancestry sequences will depend on these relative branch weights.
For example, consider all the ems that descend from some particular human, and consider a late time when there are many such descendants. There are several different ways that one could sample from these late ems to create a distribution of ems. For example, one could repeatedly sample 1) a random memory unit able to store part of the mental state of an em, 2) a random processor able to run part of an em mind, or 3) a random dollar of wealth and pick the em who owns it.
The random processor approach tends to fit better with progeny probs which say that you are more likely to be a descendant who runs faster (and who has descendants who run faster). The random memory approach tends to fit better with progeny probs that count descendants more equally, regardless of speed. And the random dollar of wealth approach tends to fit better with progeny probs that say you are more likely to become the descendants who inherit more wealth from you. Which of their descendants an em should expect to become depends on which of these methods this em thinks makes more sense for weighting future ems.
Each progeny prob predicts the existence a tiny fraction of very weird ancestors sequences, ones quite unlikely to be generated by that progeny prob. But such sequences are only actually rare if this progeny prob fits with the correct distribution. For example, few ems chosen by looking at random memories should have ancestor histories that are weird according to a memory-based progeny prob. But most of them might have ancestors histories that are weird according to processor- or wealth-based progeny probs.
For the quantum case, the standard Born rule progeny prob seems to fit well with distributions that sample from later quantum worlds in proportion to the same L2 norm that the Born rule uses. However, we lack a good widely accepted derivation of this distribution from the basic standard core of quantum mechanics. That is, we can’t explain why we should focus on later quantum worlds in proportion to this L2 norm, and mostly ignore the far larger numbers of quantum worlds that have much smaller values of this norm.
Yes, some try to derive this norm from other axioms, but none of these derivations seems a compelling explanation to me. The L2 norm is so simple that it must be implied by a great many sets of axioms. I’ve proposed a “mangled worlds” approach, and show that the Born rule can result from counting discrete worlds equally, if we ignore worlds below a size threshold that are mangled by larger worlds, and so are not hospitable to observers. But my proposal is so far [mostly](quantum-quotes) ignored.
IS COMPLEX Finally, it is worth noting that the implicit assumptions of a progeny prob model are typically violated by the reality of even simple tree models. As a result, the best fit progeny prob for a simple tree model can be quite complex.
The progeny prob framework assumes that one and only one “me” travels along some path in the tree. Conditional on being in one branch, the chances that I become each of the child branches must sum to one. And it may seem natural to have those chances be independent of what would have happened had I instead gone to other branches at earlier times.
But even in simple tree models, there is not a fixed total quantity of branches. So when there is more growth in other branches of the tree, this part of the tree shrinks as a fraction of the whole. And so the sum of the weights for the children of a branch do not usually sum to the weight of that branch. And such weights do typically depend on what happens in distant branches that split off long ago.
It thus seems all the more remarkable that the mysterious Born rule progeny prob for quantum mechanics is so simple and context independent.
## [Em Redistribution](#table-of-contents)
_Posted on 2015-07-17_
I’m in the last few weeks of finishing [my book](oxford-to-publish-the-age-of-em) The Age of Em: Work, Love, and Life When Robots Rule The Earth, about social outcomes in a world dominated by brain emulations. As a teaser, let me share some hopefully non-obvious results about redistribution in the em world.
There are [many kinds](unequal_inequal) of inequality. Inequality exists between different species, between generations born at different times, and between nations of the world at a time. Within a nation at a time, there is inequality both between families and within families. There is also inequality across the moments of the life of each person. In all of these cases, there is not only financial inequality, but also inequality in status, prestige, pleasure, lifespan, happiness, and more. There is also inequality between the size of families, firms, cities, or nations, even when individuals within those groupings are equal.
Today, we have relatively little intentional redistribution between generations or between nations. Redistribution within the moments of a person’s life happens, but that is mostly left to that person to choose and to fund. Similarly, redistribution between siblings is mostly achieved via differential treatment by parents. Instead, most concern today about inequality, and most debate about redistribution to address inequality, focuses on one very particular “standard” kind of inequality.
This standard inequality looks at differences in average individual financial incomes between the families of a nation, all at a given time. This type of inequality is actually one of the smallest. For example, in the U.S. today financial inequality between families is only one third the size of that inequality between siblings within families, and even that is much less than the inequality between individuals from different nations. We may focus our redistribution feelings on this standard inequality because it seems to us the most analogous to the inequality that [forager](foragers) sharing norms addressed. Alternatively, perhaps it is the most profitable type of redistribution for [opportunistic](inequality-is-about-grabbing) rent-seekers.
This history suggests that the em world will have little redistribution between em generations or city states, and also that each clan is mostly in charge of deciding how to address inequality within that clan. After all, em clan members are more similar and closer to each other than are human siblings, even if they may sometimes be more distant from each other than are typical human life moments. Also, clan members have rather complex relations with each other, making it hard pick a natural sub-clan unit to be the standard basis for counting inequality. So that leaves ems with comparing inequality between clans.
A set of em clans can be unequal in two different ways. One way focuses on individual incomes, or perhaps individual happiness or respect, and says that a clan is better off if its individuals are on average better off. The other way focuses on the overall size and success of a clan. Here a clan is better off if it has more members, resources, or respect. Historically, most redistribution efforts have focused on average individual outcomes. For example, we have seen very little efforts to redistribute between human family clans based on family size. That is, we almost never take from families with many descendants in order to give to families that have few descendants. Nor do we take much from big nations, cities, or firms to give to smaller ones.
Because most em wages are near subsistence levels, unregulated wages have less inequality than do wages today. So em clans naturally have less inequality of the standard sort that is the focus of today’s redistribution. In contrast, em clans have enormous inequality in clan size, resources, and respect. However, history gives little reason to expect much redistribution to address this inequality. It is not very analogous to forager sharing, nor does it lend itself to profitable rent-seeking.
Thus the main kind of redistribution that we have reason to expect in the em era is between the clans of a city, based on differences of average within-clan individual income. But we expect less inequality of this sort in the em world, and so expect less redistribution on this basis.
Income taxes are today one of our main mechanisms for reducing the standard inequality that compares individual incomes between families within a nation. Over the last two centuries, big increases in the top marginal tax rates have mostly followed wars where over two percent of the population served in the military. For example, in the U.S. the top marginal tax rate jumped from 15% to 67% in 1917, during World War I. Controlling for this effect, top tax increases have not been correlated with wealth, democracy, or the political ideology of the party running the government. This weakly suggests that the local degree of individual income redistribution between the clans of an em city may depend on the local frequency of large expensive em wars.
If ordinary humans are included straightforwardly in the redistribution systems of the em world, then the simple result to expect is transfers, not only away from richer humans, but also from humans to ems overall. After all, in purely financial terms typical ems are poorer than the poorest humans. Redistribution systems may perhaps correct for the fact that em subsistence levels are much lower than are human subsistence levels. But if so such systems may also encourage or even require recipients of aid to switch from being a human to being an em, in order to lower costs.
During the em era, humans typically have industrial era incomes, which are much higher than subsistence level incomes. While many and perhaps most humans may pay to create a few ems, they tend to endow them with much higher than subsistence incomes. In contrast, a small number of successful humans manage to give rise to large em clans, and within these clans most members have near subsistence incomes. Thus transfers based on individual income inequality take from the descendants of less successful humans and give to descendants of more successful humans.
## [A.I. Old-Timers](#table-of-contents)
_Posted on 2008-05-31_
Artificial Intelligence pioneer Roger Schank at the Edge:
> When reporters interviewed me in the 70’s and 80’s about the possibilities for Artificial Intelligence I would always say that we would have machines that are as smart as we are within my lifetime. It seemed a safe answer since no one could ever tell me I was wrong. But I no longer believe that will happen. One reason is that I am a lot older and we are barely closer to creating smart machines.
> I have not soured on AI. I still believe that we can create very intelligent machines. But I no longer believe that those machines will be like us….
> What AI can and should build are intelligent special purpose entities. (We can call them Specialized Intelligences or SI’s.) Smart computers will indeed be created. But they will arrive in the form of SI’s, ones that make lousy companions but know every shipping accident that ever happened and why (the shipping industry’s SI) or as an expert on sales (a business world SI.) … So AI in the traditional sense, will not happen in my lifetime nor in my grandson’s lifetime. Perhaps a new kind of machine intelligence will one day evolve and be smarter than us, but we are a really long way from that.
This was close to my view after nine years of A.I. research, at least regarding the non-upload A.I. path Schank has in mind. I recently met Rodney Brooks and Peter Norvig at Google Foo Camp, and Rodney told me the two of them tried without much success to politely explain this standard "old-timers" view at a recent Singularity summit. Greg Egan recently expressed himself more harshly:
> The overwhelming majority [of Transhumanists] might as well belong to a religious cargo cult based on the notion that self-modifying AI will have magical powers.
The June IEEE Spectrum is a special issue on singularity, largely skeptical.
My co-blogger Eliezer and I agree on many things, but here [we seem to disagree](biting-evolutio). Eliezer focuses on AIs possibly changing their architecture more finely and easily than humans. We humans can change our group organizations, can train new broad thought patterns, and could in principle take a knife to our brain cells. But yes an AI with a well-chosen modular structure might do better.
Nevertheless, the idea that someone will soon write software allowing a single computer to use architecture-changing ease to improve itself so fast that within a few months the fate of humanity depends on it feeling friendly enough … well that seems on its face rather unlikely. So many other huge barriers to such growth loom. Yes it is possible and yes someone should think some about it, and sure why not Eliezer. But I fear way too many consider this the default future scenario.
Added: To clarify, the standard A.I. old-timer view is roughly that A.I. mostly requires lots and lots of little innovations, and that we have a rough sense of how fast we can accumulate those innovations and of how many we need to get near human level general performance. People who look for big innovations mostly just find all the same old ideas, which don’t add that much compared to lots of little innovations.
More added: I seem to be a lot more interested in the meta issues here than most (as usual). Eliezer [seems to think](an-ai-new-timer) that when the when young disagree with the old, the young tend to be right, [because](an-ai-new-timer) "most of the Elders here are formidable old warriors with hopelessly obsolete arms and armor." I’ll bet he doesn’t apply this to people younger than him; adding in other consideration he sees his current age as near best. And I’ll bet in twenty years his estimate of the optimal age will be twenty years higher.
## [How Lumpy AI Services?](#table-of-contents)
_Posted on 2019-02-14_
Long ago people like Marx and Engels predicted that the familiar capitalist economy would naturally lead to the immiseration of workers, huge wealth inequality, and a strong concentration of firms. Each industry would be dominated by a main monopolist, and these monsters would merge into a few big firms that basically run, and ruin, everything. (This is somewhat analogous to common expectations that military conflicts naturally result in one empire ruling the world.)
Many intellectuals and ordinary people found such views quite plausible then, and still do; these are the concerns most often voiced to justify redistribution and regulation. Wealth inequality is said to be bad for social and political health, and big firms are said to be bad for the economy, workers, and consumers, especially if they are not loyal to our nation, or if they coordinate behind the scenes.
Note that many people seem much less concerned about an economy full of small firms populated by people of nearly equal wealth. Actions seem more visible in such a world, and better constrained by competition. With a few big privately-coordinating firms, in contrast, who knows that they could get up to, and they seem to have so many possible ways to screw us. Many people either want these big firms broken up, or heavily constrained by presumed-friendly regulators.
In the area of AI risk, many express great concern that the world may be taken over by a few big powerful AGI (artificial general intelligence) agents with opaque beliefs and values, who might arise suddenly via a fast local “foom” self-improvement process centered on one initially small system. I’ve [argued](how-deviant-recent-ai-progress-lumpiness) in the past that such sudden local foom seems unlikely because innovation is rarely that lumpy.
In a new book-length technical report, Reframing Superintelligence: Comprehensive AI Services as General Intelligence, Eric Drexler makes a somewhat similar anti-lumpiness argument. But he talks about task lumpiness, not innovation lumpiness. Powerful AI is safer if it is broken into many specific services, often supplied by separate firms. The task that each service achieves has a narrow enough scope that there’s little risk of it taking over the world and killing everyone in order to achieve that task. In particular, the service of being competent at a task is separate from the service of learning how to become competent at that task. In Drexler’s words:
Comprehensive AI services (CAIS) provides a model of flexible, general intelligence in which agents are a class of service-providing products, rather than a natural or necessary engine of progress in themselves. … Strongly self-modifying agents lose their instrumental value even as their implementation becomes more accessible, while the likely context for the emergence of such agents becomes a world already in possession of general superintelligent-level capabilities. …
AI deployment today is dominated by AI services such as language translation, image recognition, speech recognition, internet search, and a host of services buried within other services. … Even applications of AI within autonomous systems (e.g., self-driving vehicles) can be regarded as providing services (planning, perception, guidance) to other system components. … [Service] tasks for advanced AI include:
• Modeling human concerns • Interpreting human requests • Suggesting implementations • Requesting clarifications • Developing and testing systems • Monitoring deployed systems • Assessing feedback from users • Upgrading and testing systems
CAIS functionality, which includes the service of developing stable, task- oriented AI agents, subsumes the instrumental functionality of proposed self-transforming AGI agents, and can present that functionality in a form that better fits the established conceptual frameworks of business innovation and software engineering.
Describing AI systems in terms of functional behaviors (“services”) aligns with concepts that have proved critical in software systems development. These include separation of concerns, functional abstraction, data abstraction, encapsulation, and modularity, including the use of client/server architectures—a set of mechanisms and design patterns that support effective program design, analysis, composition, reuse, and overall robustness.
This vision seems built on the 1988 “Agoric computing” vision of Drexler and Mark Miller, which Miller has also built on in his computer security work. That vision is of computing systems with a fine-grain breakdown into service-providing modules with separate resources and property rights. As Peter McCluskey notes, this vision is also related to Drexler’s later nanotech visions:
Drexler’s CAIS proposal removes the “self-” from recursive self-improvement, in much the same way that nanofactories removed the “self-” from nanobot self-replication, replacing it with a more decentralized process that involves preserving more features of existing factories / AI implementations.
McCluskey is only mildly persuaded:
[By] analogies to people … I’m tempted … to conclude that an unified agent AI will be more visionary and eager to improve. … The novelty of the situation hints we should distrust Drexler’s extrapolation from standard software practices (without placing much confidence in any alternative). … He wants humans to decompose [curing Cancer] into narrower goals (with substantial AI assistance), such that humans could verify that the goals are compatible with human welfare (or reject those that are too hard too evaluate). This seems likely to delay cancer cures compared to what an agent AGI would do, maybe by hours, maybe by months, as the humans check the subtasks. … I haven’t thought of a realistic example where I expect the delay would generate a strong incentive for using an agent AGI, but the cancer example is close enough to be unsettling. … Modularity normally makes software development easier … [but] modularity seems less important for ML.
I’ve found two other critics of this new report. Rohin Shah:
> Typically, we’re worried about the setting where the RL [reinforcement learning] agent is learning or planning at test time, which can happen in learn-to-learn and online learning settings, or even with vanilla RL if the learned policy has access to external memory and can implement a planning process separately from the training procedure. … The lesson of deep learning is that if you can do something end-to-end, that will work better than a structured approach. This has happened with computer vision, natural language processing, and seems to be in the process of happening with robotics. So I don’t buy [Drexler’s vision] — while it seems true that we will get CAIS before AGI since structured approaches tend to be available sooner and to work with less compute, I expect that a monolithic AGI agent would outperform CAIS at most tasks once we can make one.
Richard Ngo:
> The more powerful each service is, the harder it is to ensure it’s individually safe; the less powerful each service is, the harder it is to combine them in a way that’s competitive with unified agents … Our only example of general intelligence so far is ourselves – a species composed of agent-like individuals who pursue open-ended goals. … Many complex tasks don’t easily decompose into separable subtasks. … Requiring the roles of each module and the ways they interface with each other to be … human-comprehensible will be very uncompetitive … If many AI services end up requiring similar faculties to each other, it would likely be more efficient to unify them into a single entity. … Task of combining [modules] to perform well in new tasks seems like a difficult one which will require a broad understanding of the world. … Agents will eventually overtake CAIS superintelligences because they’ll have more efficient internal structure and will be optimising harder for self-improvement. … The … fuzzy notion of “service” which makes sense in our current context, but may not in the context of much more powerful AI technology.
All these critics seem to agree with Drexler that it is harder to see and control the insides of services, relative to interfaces between them. Where they disagree is in seeing productive efficiency considerations as perhaps creating large natural service “lumps.” A big lumpy service does a large set of tasks with a wide enough scope, where it would be much less efficient to break that up into many services, and where we should be scared of what this lump might do if driven by the wrong values.
Note the strong parallels with the usual concern about large firms in capitalism. The popular prediction that unregulated capitalism would make a few huge firms is based on more than productive efficiencies; people also fear market power, collusion, and corruption of governance. But big size induced by productive efficiencies of scale is definitely one of the standard concerns.
Economics and business have large literatures not only on the many factors that induce large versus small firms, but also on the particular driver of production efficiencies. This often goes under the label “make versus buy”; making something within a firm rather than buying it from other firms tends to make a firm larger. It tends to be better to make things that need to be tightly coordinated with core firm choices, and where it is harder to make useful arm-length contracts. Without such reasons to be big, smaller tends to be more efficient. Because of these effects, most scholars today don’t think unregulated firms would become huge, contrary to Marx, Engels, and popular opinion.
If the worry is that it is dangerous to allow the firms that provide AI services to get very large, then it should be a priority to reduce the many other factors that today encourage large firms in the tech area. These include incentives to create patent pools, fixed costs of complying with regulation, taxing and regulating exchanges between but not within firms, and a lack of common carrier approaches in new network industries.
Alas, as seen in the above criticisms, it seems far too common in the AI risk world to presume that past patterns of software and business are largely irrelevant, as AI will be a glorious new shiny unified thing without much internal structure or relation to previous things. (As predicted by [far views](near-far-summary).) The history of vastly overestimating the ease of making huge firms in capitalism, and the similar typical nubbie error of overestimating the ease of making large unstructured software systems, are seen as largely irrelevant.
McCluskey does briefly considers the possibility of this sort of bias:
Maybe there’s a useful analogy to markets – maybe people underestimate CAIS because very decentralized systems are harder for people to model. People often imagine that decentralized markets are less efficient that centralized command and control, and only seem to tolerate markets after seeing lots of evidence (e.g. the collapse of communism). On the other hand, Eliezer and Bostrom don’t seem especially prone to underestimate markets, so I have low confidence that this guess explains much.
It seems crazy cultish to me to, when guessing if this bias might be a problem, to put much weight on estimating the personal bias-resisting abilities of two particular people. It’s a big world, and they too are human.
Oh, many people are very impressed that current machine learning (ML) systems seem to have less visible or understandable structure than the prior systems that the’ve replaced. But it seems to me a vast exaggeration to conclude from this that future systems of vastly larger ability and scope will have little internal structure. Even today’s best ML systems have a lot of structure, and systems of much larger scope will need a lot more. Across history we’ve seen many changes in the degree of integration of particular kinds of systems, without such changes saying much about any huge global future integration trend.
## [A History Of Foom](#table-of-contents)
_Posted on 2013-01-20_
I had occasion recently to review again the causes of the few known historical cases of sudden permanent increases in capacity growth rates in broadly capable systems: humans, farmers, and industry. For each of these transitions, a large number of changes appeared at roughly the same time. The problem is to distinguish the key change that enabled all the other changes.
For humans, it seems that the most proximate cause of faster human than non-human growth was culture – a strong ability to reliably copy the behavior of others allowed useful behaviors to accumulate via a non-genetic path. A strong ritual ability was clearly key. It also helped to have language, to live in large bands friendly with neighboring bands, to cook and travel widely, etc., but these may not have been essential. Chimps are pretty good at culture compared to most animals, just not good enough to support sustained cultural growth.
For farming, it seems to me that the key was the creation of long range trade routes along which domesticated seeds and animals could move. It was the accumulation of domestication innovations that most fundamentally caused the growth in farmers, and it was these long range trade routes that allowed innovations to accumulate so much faster than they had for foragers.
How did farming enable long range trade? Since farmers stay in one place, they are easier to find, and can make more use of heavy physical capital. Higher density living requires less travel distance for trade. But perhaps most important, transferable domesticated seeds and animals embodied innovations directly, without requiring detailed copying of behavior. They were also useful in a rather wide range of environments.
On industry, the first burst of productivity at the start of the industrial revolution was actually in the farming sector, and had little to do with machines. It appears to have come from “amateur scientist” farmers doing lots of little local trials about what worked best, and then communicating them to farmers elsewhere who grew similar crops in similar environments, via “scientific society” like journals and meetings. These specialist networks could spread innovations much faster than could trade in seeds and animals.
Applied to machines, specialist networks could spread innovation even faster, because machine functioning depended even less on local context, and because innovations could be embodied directly in machines without the people who used those machines needing to learn them.
So far, it seems that the main causes of growth rate increases were better ways to share innovations. This suggests that when looking for what might cause future increases in growth rates, we also seek better ways to share innovations.
Whole brain emulations might be seen as allowing mental innovations to be moved more easily, by copying entire minds instead of having one mind train or teach another. Prediction and decision markets might also be seen as better ways to share info about which innovations are likely to be useful where. In what other ways might we dramatically increase our ability to share innovations?
## [I Still Don’t Get Foom](#table-of-contents)
_Posted on 2014-07-24_
Back in 2008 my ex-co-blogger Eliezer Yudkowsky and I discussed his “AI foom” concept, a discussion that we recently spun off into a [book](debate-is-now-book). I’ve heard for a while that Nick Bostrom was working on a book elaborating related ideas, and this week his Superintelligence was finally available to me to read, via Kindle. I’ve read it now, along with a few dozen reviews I’ve found online. Alas, only the two reviews on GoodReads even mention the big problem I have with one of his main premises, the same problem I’ve had with Yudkowsky’s views. Bostrom hardly mentions the issue in his 300 pages (he’s focused on control issues).
All of which makes it look like I’m the one with the problem; everyone else gets it. Even so, I’m gonna try to explain my problem again, in the hope that someone can explain where I’m going wrong. Here goes.
“Intelligence” just means an ability to do mental/calculation tasks, averaged over many tasks. I’ve always found it plausible that machines will continue to do more kinds of mental tasks better, and eventually be better at pretty much all of them. But what I’ve found it hard to accept is a “local explosion.” This is where a single machine, built by a single project using only a tiny fraction of world resources, goes in a short time (e.g., weeks) from being so weak that it is usually beat by a single human with the usual tools, to so powerful that it easily takes over the entire world. Yes, smarter machines may greatly increase overall economic growth rates, and yes such growth may be uneven. But this degree of unevenness seems implausibly extreme. Let me explain.
If we count by economic value, humans now do most of the mental tasks worth doing. Evolution has given us a brain chock-full of useful well-honed modules. And the fact that most mental tasks require the use of many modules is enough to explain why some of us are smarter than others. (There’d be a common “g” factor in task performance even with independent module variation.) Our modules aren’t that different from those of other primates, but because ours are different enough to allow lots of cultural transmission of innovation, we’ve out-competed other primates handily.
We’ve had computers for over seventy years, and have slowly build up libraries of software modules for them. Like brains, computers do mental tasks by combining modules. An important mental task is software innovation: improving these modules, adding new ones, and finding new ways to combine them. Ideas for new modules are sometimes inspired by the modules we see in our brains. When an innovation team finds an improvement, they usually sell access to it, which gives them resources for new projects, and lets others take advantage of their innovation.
Since software is often fragile and context dependent, much innovation consists of making new modules that are rather similar to old ones, except that they work in somewhat different contexts. We try to avoid this fragility via abstraction, but this is usually hard. Today humans also produce most of the value in innovations tasks, though software sometimes helps. We even try to innovate new ways to innovate, but that is also very hard.
Overall we so far just aren’t very good at writing software to compete with the rich well-honed modules in human brains. And we are bad at making software to make more software. But computer hardware gets cheaper, software libraries grow, and we learn more tricks for making better software. Over time, software will get better. And in [centuries](ai-progress-estimate), it may rival human abilities.
In this context, Bostrom imagines that a single “machine intelligence project” builds a “system” or “machine” that follows the following trajectory:
[](BostromFoom)
“Human baseline” represents the effective intellectual capabilities of a representative human adult with access to the information sources and technological aids currently available in developed countries. … “The crossover”, a point beyond which the system’s further improvement is mainly driven by the system’s own actions rather than by work performed upon it by others. … Parity with the combined intellectual capability of all of humanity (again anchored to the present) … [is] “civilization baseline”.
These usual “technological aids” include all of the other software available for sale in the world. So apparently the reason the “baseline” and “civilization” marks are flat is that this project is not sharing its innovations with the rest of the world, and available tools aren’t improving much during the period shown. Bostrom distinguishes takeoff durations that are fast (minutes, hours, or days), moderate (months or years), or slow (decades or centuries) and says “a fast or medium takeoff looks more likely.” As it now takes the world economy fifteen years to double, Bostrom sees one project becoming a “singleton” that rules all:
The nature of the intelligence explosion does encourage a winner-take-all dynamic. In this case, if there is no extensive collaboration before the takeoff, a singleton is likely to emerge – a single project would undergo the transition alone, at some point obtaining a decisive strategic advantage.
Bostrom accepts there there might be more than one such project, but suggests that likely only the first one would matter, because the time delays between projects would be like the years and decades we’ve seen between when different nations could build various kinds of nuclear weapons or rockets. Presumably these examples set the rough expectations we should have in mind for the complexity, budget, and secrecy of the machine intelligence projects Bostrom has in mind.
In Bostrom’s graph above the line for an initially small project and system has a much higher slope, which means that it becomes in a short time vastly better at software innovation. Better than the entire rest of the world put together. And my key question is: how could it plausibly do that? Since the rest of the world is already trying the best it can to usefully innovate, and to abstract to promote such innovation, what exactly gives one small project such a huge advantage to let it innovate so much faster?
After all, if a project can’t innovate faster than the world, it can’t grow faster to take over the world. Yes there may be feedback effects, where better software makes it easier to make more software, speeds up hardware gains to encourage better software, etc. But if these feedback effects apply nearly as strongly to software inside and outside the project, it won’t give much advantage to the project relative to the world. Yes by isolating itself the project may prevent others from building on its gains. But this also keeps the project from gaining revenue to help it to grow.
A system that can perform well across a wide range of tasks probably needs thousands of good modules. Same for a system that innovates well across that scope. And so a system that is a much better innovator across such a wide scope needs much better versions of that many modules. But this seems like far more innovation than is possible to produce within projects of the size that made nukes or rockets.
In fact, most software innovation [seems](why-does-hardware-grow-like-algorithms) to be driven by hardware advances, instead of innovator creativity. Apparently, good ideas are available but must usually wait until hardware is cheap enough to support them.
Yes, sometimes architectural choices have wider impacts. But I was an artificial intelligence researcher for nine years, ending twenty years ago, and I never saw an architecture choice make a huge difference, relative to other reasonable architecture choices. For most big systems, overall architecture matters a lot less than getting lots of detail right. Researchers have long wandered the space of architectures, mostly rediscovering variations on what others found before.
Some hope that a small project could be much better at innovation because it specializes in that topic, and much better understands new theoretical insights into the basic nature of innovation or intelligence. But I don’t think those are actually topics where one can usefully specialize much, or where we’ll find much useful new theory. To be much better at learning, the project would instead have to be much better at hundreds of specific kinds of learning. Which is very hard to do in a small project.
What does Bostrom say? Alas, not much. He distinguishes several advantages of digital over human minds, but all software shares those advantages. Bostrom also distinguishes five paths: better software, brain emulation (i.e., ems), biological enhancement of humans, brain-computer interfaces, and better human organizations. He doesn’t think interfaces would work, and sees organizations and better biology as only playing supporting roles.
That leaves software and ems. Between the two Bostrom thinks it “fairly likely” software will be first, and he thinks that even if an em transition doesn’t create a singleton, a later software-based explosion will. I can at least see a plausible sudden gain story for ems, as almost-working ems aren’t very useful. But in this post I’ll focus on software explosions.
Imagine in the year 1000 you didn’t understand “industry,” but knew it was coming, would be powerful, and involved iron and coal. You might then have pictured a blacksmith inventing and then forging himself an industry, and standing in a city square waiving it about, commanding all to bow down before his terrible weapon. Today you can see this is silly — industry sits in thousands of places, must be wielded by thousands of people, and needed thousands of inventions to make it work.
Similarly, while you might imagine someday standing in awe in front of a super intelligence that embodies all the power of a new age, superintelligence just isn’t the sort of thing that one project could invent. As “intelligence” is just the name we give to being better at many mental tasks by using many good mental modules, there’s no one place to improve it. So I can’t see a plausible way one project could increase its intelligence vastly faster than could the rest of the world.
(One might perhaps move a lot of intelligence at once from humans to machines, instead of creating it. But that is the em scenario, which I’ve set aside here.)
So, bottom line, much of Nick Bostrom’s book Superintelligence is based on the premise that a single software project, which starts out with a tiny fraction of world resources, could within a few weeks grow so strong to take over the world. But this seems to require that this project be vastly better than the rest of the world at improving software. I don’t see how it could plausibly do that. What I am I missing?
Added 2Sep: See also related posts after: [Irreducible Detail](limits-on-generality), [Regulating Infinity](regulating-infinity).
Added 5Nov: Let me be clear: Bostrom’s book has much thoughtful analysis of AI foom consequences and policy responses. But aside from mentioning a few factors that might increase or decrease foom chances, Bostrom simply doesn’t given an argument that we should expect foom. Instead, Bostrom just assumes that the reader thinks foom likely enough to be worth his detailed analysis.
## [Foom Justifies AI Risk Efforts Now](#table-of-contents)
_Posted on 2017-08-03_
Years ago I was honored to share this blog with Eliezer Yudkowsky. One of his main topics then was AI Risk; he was one of the few people talking about it back then. We debated this topic here, and while we disagreed I felt we made progress in understanding each other and exploring the issues. I assigned a much lower probability than he to his key “foom” scenario.
Recently AI risk has become something of an industry, with far more going on than I can keep track of. Many call working on it one of the most effectively altruistic things one can possibly do. But I’ve searched a bit and as far as I can tell that foom scenario is still the main reason for society to be concerned about AI risk now. Yet there is almost no recent discussion evaluating its likelihood, and certainly nothing that goes into as much depth as did Eliezer and I. Even Bostrom’s [book length](30855) treatment basically just assumes the scenario. Many seem to think it obvious that if one group lets one AI get out of control, the whole world is at risk. It’s not (obvious).
As I just revisited the topic while revising Age of Em for paperback, let me try to summarize part of my position again here.
For at least a century, every decade or two we’ve seen a burst of activity and concern about automation. The last few years have seen another such burst, with increasing activity in AI research and commerce, and also increasing concern expressed that future smart machines might get out of control and destroy humanity. Some argue that these concerns justify great efforts today to figure out how to keep future AI under control, and to more closely watch and constrain AI research efforts. Approaches considered include kill switches, requiring prior approval for AI actions, and designing AI motivational system to make AIs want to help, and not destroy, humanity.
Consider, however, an analogy with organizations. Today, the individuals and groups who create organizations and their complex technical systems are often well-advised to pay close attention to how to maintain control of such organizations and systems. A loss of control can lead not only to a loss of the resources invested in creating and maintaining such systems, but also to liability and retaliation from the rest of the world.
But exactly because individuals usually have incentives to manage their organizations and systems reasonably well, the rest of us needn’t pay much attention to the internal management of others’ organizations. In our world, most firms, cities, nations, and other organizations are much more powerful and yes smarter than are most individuals, and yet they remain largely under control in most important ways. For example, none have so far destroyed the world. Smaller than average organizations can typically exist and even thrive without being forcefully absorbed into larger ones. And outsiders can often influence and gain from organization activities via control institutions like elections, board of directors, and voting stock.
Mostly, this is all achieved neither via outside action approval nor via detailed knowledge and control of motivations. We instead rely on law, competition, social norms, and politics. If a rogue organization seems to harm others, it can be accused of legal violations, as can its official owners and managers. Those who feel hurt can choose to interact with it less. Others who are not hurt may choose to punish the rogue informally for violating informal norms, and get rewarded by associates for such efforts. And rogues may be excluded from political coalitions, who hurt it via the policies of governments and other large organizations.
AI and other advanced technologies may eventually give future organizations new options for internal structures, and those introducing such innovations should indeed consider their risks for increased chances of losing control. But it isn’t at all clear why the rest of us should be much concerned about this, especially many decades or centuries before such innovations may appear. Why can’t our usual mechanisms for keeping organizations under control, outlined above, keep working? Yes, innovations might perhaps create new external consequences, ones with which those outside of the innovating organization would need to deal. But given how little we now understand about the issues, architectures, and motivations of future AI systems, why not mostly wait and deal with any such problems later?
Yes, our usual methods do fail at times; we’ve had wars, revolutions, theft, and lies. In particular, each generation has had to accept slowly losing control of the world to succeeding generations. While prior generations can typically accumulate and then spend savings to ensure a comfortable retirement, they no longer rule the world. Wills, contracts, and other organizational commitments have not been enough to prevent this. [Some](chalmers-reply-2) find this unacceptable, and seek ways to enable a current generation, e.g., humans today, to maintain strong control over all future generations, be they biological, robotic or something else, even after such future generations have become far more capable than the current generation. To me this problem seems both very hard, and not obviously worth solving.
Returning to the basic problem of rogue systems, some forsee a rapid local “intelligence explosion”, sometimes called “foom”, wherein one initially small system quickly becomes vastly more powerful than the entire rest of the world put together. And, yes, if such a local explosion might happen soon, then it could make more sense for the rest of us today, not just those most directly involved, to worry about how to keep control of future rogue AI.
In a prototypical “foom,” or local intelligence explosion, a single AI system starts with a small supporting team. Both the team and its AI have resources and abilities that are tiny on a global scale. This team finds and then applies a big innovation in system architecture to its AI system, which as a result greatly improves in performance. (An “architectural” change is just a discrete change with big consequences.) Performance becomes so much better that this team plus AI combination can now quickly find several more related innovations, which further improve system performance. (Alternatively, instead of finding architectural innovations the system might enter a capability regime which contains a large natural threshold effect or scale economy, allowing a larger system to have capabilities well out of proportion to its relative size.)
During this short period of improvement, other parts of the world, including other AI teams and systems, improve much less. Once all of this team’s innovations are integrated into its AI system, that system is now more effective than the entire rest of the world put together, at least at one key task. That key task might be theft, i.e., stealing resources from the rest of the world. Or that key task might be innovation, i.e., improving its own abilities across a wide range of useful tasks.
That is, even though an entire world economy outside of this team, including other AIs, works to innovate, steal, and protect itself from theft, this one small AI team becomes vastly better at some combination of (1) stealing resources from others while preventing others from stealing from it, and (2) innovating to make this AI “smarter,” in the sense of being better able to do a wide range of mental tasks given fixed resources. As a result of being better at these things, this AI quickly grows the resources under its control and becomes in effect more powerful than the entire rest of the world economy put together. So, in effect it takes over the world. All of this happens within a space of hours to months.
(The hypothesized power advantage here is perhaps analogous that of the first team to make an atomic bomb, if that team had had enough other supporting resources to enable it to use the bomb to take over the world.)
Note that to believe in such a local explosion scenario, it is not enough to believe that eventually machines will be very smart, even much smarter than are humans today. Or that this will happen soon. It is also not enough to believe that a world of smart machines can overall grow and innovate much faster than we do today. One must in addition believe that an AI team that is initially small on a global scale could quickly become vastly better than the rest of the world put together, including other similar teams, at improving its internal abilities.
If a foom-like explosion can quickly make a once-small system more powerful than the rest of the world put together, the rest of the world might not be able to use law, competition, social norms, or politics to keep it in check. Safety can then depend more on making sure that such exploding systems start from safe initial designs.
In another post I may review arguments for and against the likelihood of foom. But in this one I’m content to just point out that the main reason for society, as opposed to particular projects, to be concerned about AI risk is either foom, or an ambition to place all future generations under the tight control of a current generation. So a low estimate of the probability of foom can imply a much lower social value from working on AI risk now.
Added Aug 4: I made a twitter poll on motives for AI risk concern:
If AI Risk is priority now, why? Foom: 1 AI takes over world, Value Drift: default future has bad values, or Collapse: property rights fail
> — robin hanson (@robinhanson) August 4, 2017
## [Is The City-ularity Near?](#table-of-contents)
_Posted on 2010-02-09_
The land around New York City is worth a lot. A 2008 analysis estimated prices for land, not counting buildings etc., for most (~80%?) of the nearby area (2750 square miles, = a 52 mile square). The total New York area land value (total land times ave price) was 5.5T$ (trillion) in 2002 and 28T$ in 2006.
The Economist said that in 2002 all developed nation real estate was worth 62T$. Since raw land value is on average about a third of total real estate value, that puts New York area real estate at over 30% of all developed nation real estate in 2002! Whatever the exact number, clearly this agglomeration contains vast value.
New York land is valuable mainly because of how it is organized. People want to be there because they want to interact with other people they expect to be there, and they expect those interactions to be quite mutually beneficial. If you could take any other 50 mile square (of which Earth has 72,000), and create that same expectation of mutual value from interactions, you could get people to come there, make buildings, etc., and sell that land for many trillions of dollars of profit.
Yet the organization of New York was mostly set long ago based on old tech (e.g., horses, cars, typewriters). Worse, no one really understands at a deep level how it is organized or why that works so well. Different people understand different parts, in mostly crude empirical ways.
So what will happen when super-duper smarties wrinkle their brows so hard that out pops a deep math theory of cities, explaining clearly how city value is produced? What if they apply their theory to designing a city structure that takes best advantage of our most advanced techs, of 7gen phones, twitter-pedias, flying Segways, solar panels, gene-mod pigeons, and super-fluffy cupcakes? Making each city aspect more efficient makes the city more attractive, increasing the gains from making other aspects more efficient, in a grand spiral of bigger gains.
Once they convince the world of the vast value in their super-stupendous city design, won’t everyone flock there and pay mucho trillions for the privilege? Couldn’t they leverage this lead into better theories enabling better designs giving far more trillions, and then spend all that on a super-designed war machine based on those same super insights, and turn us all into down dour super-slaves? So isn’t the very mostest importantest cause ever to make sure that we, the friendly freedom fighters, find this super deep city theory first?
Well, no, it isn’t. We don’t believe in a city-ularity because we don’t believe in a super-city theory found in a big brain flash of insight. What makes cities work well is mostly getting lots of details right. Sure new-tech-based cities designs can work better, but gradual tech gains mean no city is suddenly vastly better than others. Each change has costs to be weighed against hoped-for gains. Sure costs of change might be lower when making a whole new city from scratch, but for that to work you have to be damn sure you know which changes are actually good ideas.
For similar reasons, I’m skeptical of a blank-slate AI mind-design singularity. Sure if there were a super mind theory that allowed vast mental efficiency gains all at once, but there isn’t. Minds are vast complex structures full of parts that depend intricately on each other, much like the citizens of a city. Minds, like cities, best improve gradually, because you just never know enough to manage a vast redesign of something with such complex inter-dependent adaptations.
## [The Betterness Explosion](#table-of-contents)
_Posted on 2011-06-21_
We all want the things around us to be better. Yet today billions struggle year after year to make just a few things a bit better. But what if our meagre success was because we just didn’t have the right grand unified theory of betterness? What if someone someday discovered the basics of such a theory? Well then this person might use his basic betterness theory to make himself better in health, wealth, sexiness, organization, work ethic, etc. More important, that might help him make his betterness theory even better.
After several iterations this better person might have a much better betterness theory. Then he might quickly make everything around him much better. Not just better looking hair, better jokes, or better sleep. He might start a better business, and get better at getting investors to invest, customers to buy, and employees to work. Or he might focus on making better investments. Or he might run for office and get better at getting elected, and then make his city or nation run better. Or he might create a better weapon, revolution, or army, to conquer any who oppose him.
Via such a “betterness explosion,” one way or another this better person might, if so inclined, soon own, rule, or conquer the world. Which seems to make it very important that the first person who discovers the first good theory of betterness be a very nice generous person who will treat the rest of us well. Right?
OK, this might sound silly. After all, we seem to have little reason to expect there is a useful grand unified theory of betterness to discover, beyond what we already know. “Betterness” seems mostly a concept about us and what we want – why should it correspond to something out there about which we can make powerful discoveries?
But a bunch of smart well-meaning folks actually do worry about a scenario that seems pretty close to this one. Except they talk about “intelligence” instead of “betterness.” They imagine an “intelligence explosion,” by which they don’t just mean that eventually the future world and many of its creatures will be more mentally capable than us in many ways, or even that the rate at which the world makes itself more mentally capable will speed up, similar to how growth rates have sped up over the long sweep of history. No, these smart well-meaning folks instead imagine that once someone has a powerful theory of “intelligence,” that person could create a particular “intelligent” creature which is good at making itself more “intelligent,” which then lets that creature get more “intelligent” about making itself “intelligent.” Within a few days or weeks, the story goes, this one creature could get so “intelligent” that it could do pretty much anything, including taking over the world.
I put the word “intelligence” in quotes to emphasize that the way these folks use this concept, it pretty much just means “betterness.” (Well, mental betterness, but most of the betterness we care about is mental.) And this fits well with common usage of the term “intelligence.” When we talk about machines or people or companies or even nations being “intelligent,” we mainly mean that such things are broadly mentally or computationally capable, in ways that are important for their tasks and goals. That is, an “intelligent” thing has a great many useful capabilities, not some particular specific capability called “intelligence.” To make something broadly smarter, you have to improve a wide range of its capabilities. And there is generally no easy or fast way to do that.
Now if you artificially hobble something so as to simultaneously reduce many of its capacities, then when you take away that limitation you may simultaneously improve a great many of its capabilities. For example, if you drug a person so that they can hardly think, then getting rid of that drug can suddenly improve a great many of their mental abilities. But beyond removing artificial restrictions, it is very hard to simultaneously improve many diverse capacities. Theories that help you improve capabilities are usually focused on a relatively narrow range of abilities – very general and useful theories are quite rare.
All of which is to say that fearing that a new grand unified theory of intelligence will let one machine suddenly take over the world isn’t that different from fearing that a grand unified theory of betterness will let one better person suddenly take over the world. This isn’t to say that such an thing is impossible, but rather that we’d sure want some clearer indications that such a theory even exists before taking such a fear especially seriously.
## [An Outside View of AI Control](#table-of-contents)
_Posted on 2017-10-01_
I’ve written much on my skepticism of local AI foom (= intelligence explosion). Recently I [said](foom-justifies-ai-risk-efforts-now) that foom offers the main justification I understand for AI risk efforts now, as well as being the main choice of my Twitter followers in a survey. It was the main argument offered by [Eliezer Yudkowsky](debate-is-now-book) in our debates here at this blog, by [Nick Bostrom](30855) in his book Superintelligence, and by [Max Tegmark](tegmarks-book-of-foom) in his recent book Life 3.0 (though he denied so in his [reply](tegmarks-book-of-foom) here).
However, some privately complained to me that I haven’t addressed those with non-foom-based AI concerns. So in this post I’ll consider AI control in the context of a prototypical non-em non-foom mostly-peaceful outside-view AI scenario. In a future post, I’ll try to connect this to specific posts by others on AI risk.
An AI scenario is where software does most all jobs; humans may work for fun, but they add little value. In a non-em scenario, ems are never feasible. As foom scenarios are driven by AI innovations that are very lumpy in time and organization, in non-foom scenarios innovation lumpiness is distributed more like it is in our world. In a mostly-peaceful scenario, peaceful technologies of production matter much more than do technologies of war and theft. And as an outside view guesses that future events are like similar past events, I’ll relate future AI control problems to similar past problems.
Conway’s law of software says that the structure of software tends to reflect the structure of the social organizations that make it. This suggests that the needs of software to have particular modularity structures for particular problems is usually weaker than the needs of organizations to maintain familiar structures of communication and authority. So a world where software does most job tasks could retain a recognizable clumping of tasks into jobs, divisions, firms, professions, industries, cities, and nations.
Today, most lumpiness in firms, industries, cities, etc. is not due to lumpiness in innovation, but instead due to various scale and scope economies, and network effects. Innovation may be modestly lumpy at the scale of particular industries, but not at the level of entire economic sectors. Most innovation comes from many small contributions; big lumps contain only a small fraction of total value.
While innovation is often faster in some areas than in others, most social rates of change tend to be near the doubling time of the economy. An AI world allows for faster growth, as it isn’t held back by slow growing humans; I’ve estimated that it might double monthly. But our first guess should be that social rates of change speed up together; we’ll need specific reasons to expect specific changes, relative to today, in relative rates of change.
Today humans tend to get job specific training in the firms where they work, and general training elsewhere. Similarly, in an AI world specific software may be written in organizations near where it is used, while more general tools are written in more distant organizations.
Today most tasks are done by human brains, which come in a standard-sized unit capable of both doing specific tasks and also related meta-tasks, such as figuring out how to do tasks better, deciding which specific tasks to do when and how, and regrouping how tasks are clumped within organizations. So we tend to first automate tasks that can be done by much smaller units. And while managers and others often specialize in meta-tasks, line workers also do many meta tasks themselves. In contrast, in an AI world meta-tasks tend to be separated more from other tasks, and so done by software that is more different.
In such a division of tasks, most tasks have a relatively narrow scope. For narrow tasks, the main risks are of doing tasks badly, and of hostile agents taking control of key resources. So most control efforts focus on such problems. For narrow tasks there is little extra risk from such tasks being done very well, even if one doesn’t understand how that happens. (War tech is of course an exception; victims can suffer more when war is done well.) The control risks on which AI risk folks focus, of very effective efforts misdirected due to unclear goals, are mainly concentrated in tasks with very wide scopes, such as in investment, management, law, and governance. These are mostly very-meta-tasks.
The future problem of keeping control of advanced software is similar to the past problems of keeping control both of physical devices, and of social organizations. As the tasks we assign to physical devices tend to be narrow, we mostly focus there on specific control failure scenarios. The main risks there are losing control to hostile agents, and doing tasks badly, rather than doing them very well. The main people to get hurt when control is lost are those who rely on such devices, or who are closely connected to such people.
Humans started out long ago organized into small informal bands, and later had to learn to deal with the new organizations and institutions of rulers, command heirarchies, markets, family clans, large scale cultures, networks of talking experts, legal systems, firms, guilds, religions, clubs, and government agencies. Such organizations are often given relatively broad tasks. And even if not taken over by hostile agents, they can drift out of control. For example, organizations may on the surface seem responsive and useful, while increasingly functioning mainly to entrench and perpetuate themselves.
When social organizations get out of control in this way, the people who initiated and then participated in them are the main folks to get hurt. So such initiators and participants thus have incentives to figure out how to avoid such control losses, and this has long been a big focus of organization innovation efforts.
Innovation in control mechanisms has long been an important part of innovation in devices and organizations. People sometimes try to develop better control mechanisms in the abstract, before they’ve seen real systems. They also sometimes experiment in the context of small test versions. But most control innovations come in response to seeing real behaviors associated with typical real versions. The main reason that it becomes harder to implement innovations later is that design features often become entrenched. But if control is important enough, it can be worth paying large costs of change to implement better control features.
Humans one hundred thousand years ago might have tried to think carefully about how to control rulers with simple command hierarchies, and people one thousand years ago might have tried to think carefully about how to control complex firms and government agencies. But the value of such early efforts would have been quite limited, and it wasn’t at all too late to work on such problems after such systems appeared. In peacetime, control failures mainly threatened those who initiated and participated in such organizations, not the world as a whole.
In the AI scenario of this post, the vast majority of advanced future software does tasks of such narrow scopes that their control risks are more like those for physical devices, relative to new social organizations. So people deploying new advanced software will know to focus extra control efforts on software doing wider scope meta-tasks. Furthermore, the main people harmed by failures to control AI assigned to meta-tasks will be those associated with the organizations that do such meta-tasks.
For example, customers who let an AI tell them whom to date may suffer from bad dates. Investors in firms that let AI manage key firm decisions might lose their investments. And citizens who let AI tell their police who to put in jail may suffer in jail, or from undiscouraged crime. But such risks are mostly local, not global risks.
Of course for a long time now, coordination scales have been slowly increasingly worldwide. So over time “local” effects become increasingly larger scale effects. This is a modest reason for everyone to slowly get more concerned about “local” problems elsewhere.
Today is a very early date to be working on AI risk; I’ve [estimated](ai-progress-estimate) that without ems it is several centuries away. We are now pretty ignorant about most aspects of how advanced software will be used and arranged. So it is hard to learn much useful today about how to control future advanced software. We can learn to better control the software we have now, and later on we should expect innovations in software control to speed up roughly as fast as do innovations in making software more effective. Even if control innovations by humans don’t speed up as fast, advanced software will itself be made of many parts, and some parts will want to keep control over other parts.
The mechanisms by which humans today maintain control over organizations include law, property rights, constitutions, command hierarchies, and even democracy. Such broad mechanisms are effective, entrenched and robust enough that future advanced software systems and organizations will almost surely continue to use variations within these broad categories keep control over each other. So humans can reasonably hope to be at least modestly protected in the short run if they can share the use of such systems with advanced software. For example, if law protects software from stealing from software, it may also protect humans from such theft.
Of course humans have long suffered from events like wars and revolutions, events that create risks of harm and loss of control. And the rate of such events can scale with the main rates of change in the economy, which go inversely as the economic doubling time. So a much faster changing future AI economy can have faster rates of such risky events. It seems a robust phenomenon that when the world speeds up, those who do not speed up with it face larger subjective risks if they do not ally with sped-up protectors.
Having humans create and become ems is one reasonable approach to creating sped-up allies for humans. Humans will no doubt also try to place advanced software in such an ally role. Once software is powerful, then attempts by humans to control such software are probably mostly based on copying the general lessons and approaches that advanced software discovers for how to maintain control over advanced software. Humans may also learn extra lessons that are specific to the human control problem, and some of those lessons may come from our era, long before any of this plays out.
But in the sort of AI scenario I’ve described in this post, I find it very hard to see such early efforts as the do-or-die priority that some seem to suggest. Outside of a foom scenario, control failures threaten to cause local, not global losses (though on increasingly larger scales).
From this view, those tempted to spend resources now on studying AI control should consider two reasonable alternatives. The first alternative is to just save more now to grow resources to be used later, when we understand more. The second alternative is to work to innovate with our general control institutions, to make them more robust, and thus better able to handle larger coordination scales, and whatever other problems the future may hold. (E.g., futarchy.)
Okay, this is how I see the AI control problem in a non-em non-foom mostly-peaceful outside-view AI scenario. But clearly others disagree with me; what am I missing?
Added 4 Oct:
In the context of foom, the usual AI concern is a total loss of control of the one super AI, whose goals quickly drift to a random point in the space of possible goals. Humans are then robustly exterminated. As the AI is so smart and inscrutable, any small loss of control is said to open the door to such extreme failure. I have presumed that those who tell me to look at non-foom AI risk are focused on similar failure scenarios.
Today most social systems suffer from agency costs, and larger costs (in % terms) for larger systems. But these mostly take the form of modestly increasing costs. It isn’t that you can’t reliably use these systems to do the things that you want. You just have to pay more. That extra cost mostly isn’t a transfer accumulating in someone else’s account. Instead there is just waste that goes to no one, and there are more cushy jobs and roles where people can comfortably sit as parasites. Over time, even though agency costs take a bigger cut, total costs get lower and humans get more of what they want.
When I say that in my prototypical non-foom AI scenario, AI will still pay agency costs but the AI control problem seems mostly manageable, I mean that very competent future social and software systems will suffer from waste and parasites as do current systems, but that humans can still reliably use such systems to get what they want. Not only are humans not exterminated, they get more than before of what they want.
## [AI Risk, Again](#table-of-contents)
_Posted on 2023-03-03_
Large language models like ChatGPT have recently spooked a great many, and my Twitter feed is full of worriers saying how irresponsible orgs have been to make and release such models. Because, they say, such a system might have killed us all. And, as some researchers say that they are working on how to better control such things, worriers say we must regulate to slow/stop AI progress until such researchers achieve their goals. While I’ve written [on](https://www.lesswrong.com/posts/D3NspiH2nhKA6B2PE/what-evidence-is-alphago-zero-re-agi-complexity) [this](https://www.overcomingbias.com/p/why-not-wait-on-ai-riskhtml) [many](https://www.overcomingbias.com/p/foom-updatehtml) [times](https://www.overcomingbias.com/p/how-lumpy-ai-serviceshtml) [before](https://www.overcomingbias.com/p/agency-failure-ai-apocalypsehtml), it seems time to restate my position.
First, if past trends continue, then sometimes in the next few centuries the world economy is likely to enter a transition that lasts roughly a decade, after which it may double every few months or faster, in contrast to our current fifteen year doubling time. (Doubling times have been relatively steady as innovations are typically tiny compared to the world economy.)
The most likely cause for such a transition seems to be a transition to an economy dominated by artificial intelligence (AI). (Perhaps in the form of brain emulations, but perhaps also in more alien forms.) Especially as the doubling time of a fully-automated factory today is a few months, and computer algorithm gains have been [close](https://www.overcomingbias.com/p/why-does-hardware-grow-like-algorithmshtml) to hardware gains. And within a year or two from then, another transition to an even faster mode might plausibly occur.
Second, coordination and control are hard. Today, org leaders often gain rents from their positions, rents which come at the expense of org owners, suppliers, and customers. This happens more-so at non-profits and publicly-held for-profits, compared to privately held for-profits. Political and military leaders also gain rents, and sometimes take over control of nations via coups. While leader rents are not the only control problem, the level of such rents is a rough indication of the magnitude of our control problems. Those who are culturally more distant from leaders, such as the poor and third world residents, typically pay higher rents.
Today such rents are non-trivial, but even so competition between orgs keeps them tolerable. That is, we mostly keep our orgs under control. Even though, compared to individual humans, large orgs are in effect “super-intelligences”.
Third, there may be extra obstacles to slow bio humans controlling future org ventures. Bio humans would be more culturally distant, slower, and less competent than em AIs. (Though the principle-agent lit doesn’t yet show smarts differences to be an issue.) And non-em AIs could be even more culturally distant. However, even an increase of a factor of two or four in control rents for AIs seems tolerable, offering such bio humans a rich and growing future. Yes, periodically some ventures would suffer the equivalent of a coup. But if, like today, each venture were only a small part of this future world, bio humans as a whole would do fine. Ems, if they exist, could do even better.
Of course the owners of such future ventures, be they bio humans, ems, or other, are well advised to consider how best to control such ventures, to cut leader rents and other related costs of imperfect control. But such efforts seem most effective when based on actual experience with concrete fielded systems. For example, there was little folks could do in the year 1500 to figure out how to control 20th century orgs, weapons, or other tech. Thus as we now know very little about the details of future AI-based ventures, leaders, or systems, we should today mostly either save resources to devote to future efforts, or focus our innovation efforts on improving control of existing ventures. Such as via decision markets.
Most of the worriers mentioned above, however, reject the above analysis, based as it is on expecting a continuation of historical patterns, wherein ventures and innovations have been consistently small compared to the world economy. They instead say that it is possible that a single small AI venture might stumble across a single extremely potent innovation, which enables it to suddenly “foom”, i.e., explode in power from tiny compared to the world economy, to more powerful than the entire rest of the world put together. (Including all other AIs.)
This scenario requires that this venture prevent other ventures from using its key innovation during this explosive period. It also requires that this new more powerful system not only be far smarter in most all important areas, but also be extremely capable at managing its now-enormous internal coordination problems. And it requires that this system be not a mere tool, but a full “agent” with its own plans, goals, and actions.
Furthermore it is possible that even though this system was, before this explosion, and like most all computer systems today, very well tested to assure that its behavior was aligned well with its owners’ goals across its domains of usage, its behavior after the explosion would be nearly maximally non-aligned. (That is, orthogonal in a high dim space.) Perhaps resulting in human extinction. The usual testing and monitoring processes would be prevented from either noticing this problem or calling a halt when it so noticed, either due to this explosion happening too fast, or due to this system creating and hiding divergent intentions from its owners prior to the explosion.
While I agree that this is a logically possible scenario, not excluded by what we know, I am disappointed to see so many giving it such a high credence, given how crazy far it seems from our prior experience. Yes, there is a sense in which the human, farming, and industry revolutions were each likely the result of a single underlying innovation. But those were the three biggest innovations in all of human history. And large parts of the relevant prior world exploded together in those cases, not one tiny part suddenly exterminating all the rest.
In addition, the roughly decade duration predicted from prior trends for the length of the next transition period seems _plenty_ of time for today’s standard big computer system testing practices to notice alignment issues. And note that the impressive recent AI chatbots are especially unlike the systems of concern here: self-improving very-broadly-able full-agents with hidden intentions. Making this an especially odd time to complain that new AI systems might have killed us all.
You might think that folks would take a lesson from our history of prior bursts of anxiety and concern about automation, bursts which have appeared roughly every three decades since at least the 1930s. Each time, new impressive demos revealed unprecedented capabilities, inducing a burst of activity and discussion, with many then expressing fears that a rapid explosion might soon commence, automating all human labor. They were, of course, very wrong.
Worriers often invoke a Pascal’s wager sort of calculus, wherein any tiny risk of this nightmare scenario could justify large cuts in AI progress. But that seems to assume that it is relatively easy to assure the same total future progress, just spread out over a longer time period. I instead fear that overall economic growth and technical progress is more fragile that this assumes. Consider how regulations inspired by nuclear power nightmare scenarios have for seventy years prevented most of its potential from being realized. I have also seen progress on many other promising techs mostly stopped, not merely slowed, via regulation inspired by vague fears. In fact, progress seems to me to be slowing down worldwide due to excess fear-induced regulation.
Over the last few centuries the world did relatively little to envision problems with future techs, and to prepare for those problems far in advance of seeing concrete versions. And I just do not believe that the world would have been better off if we had instead greatly slowed tech progress in order to attempt such preparations. Especially considering the degree of centralized controls that might have been required to implement such a slowdown policy.
As I discussed above, it just looks way too early to learn much about how to control future AI systems, about which we know so few details. Thus when facing the risk of our fear essentially halting progress here, I’d rather continue down our current path, and work harder on controls when we better see concrete serious control problems to manage.
**Added March 7**: Some say that, given enough data and hardware, predict-the-next-token models like ChatGPT will have human or better performance. Using action tokens, that would include many kinds of behavior. But this isn’t sufficient for such a system to rapidly “foom”. To even try, it needs high competence in design and testing alternative system architectures, and there’s no guarantee even with that.
## [Foom Update](#table-of-contents)
_Posted on 2022-05-06_
To extend our reach, we humans have built tools, machines, firms, and nations. And as these are powerful, we try to maintain control of them. But as efforts to control them usually depend on their details, we have usually waited to think about how to control them until we had concrete examples in front of us. In the year 1000, for example, there wasn’t much we could do to usefully think about how to control most things that have only appeared in the last two centuries, such as cars or international courts.
Someday we will have far more powerful computer tools, including “advanced artificial general intelligence” (AAGI), i.e., with capabilities even higher and broader than those of individual human brains today. And some people today spend substantial efforts today worrying about how we will control these future tools. Their most common argument for this unusual strategy is “foom”.
That is, they postulate a single future computer system, initially quite weak and fully controlled by its human sponsors, but capable of action in the world and with general values to drive such action. Then over a short time (days to weeks) this system dramatically improves (i.e., “fooms”) to become an AAGI far more capable even than the sum total of all then-current humans and computer systems. This happens via a process of self-reflection and self-modification, and this self-modification also produces large and unpredictable changes to its effective values. They seek to delay this event until they can find a way to prevent such dangerous “value drift”, and to persuade those who might initiate such an event to use that method.
I’ve argued at length ([1](https://www.overcomingbias.com/2017/08/foom-justifies-ai-risk-efforts-now.html) [2](https://www.overcomingbias.com/2014/07/30855.html) [3](https://www.overcomingbias.com/2013/02/foom-debate-again.html) [4](https://www.overcomingbias.com/2011/06/the-betterness-explosion.html) [5](https://www.overcomingbias.com/2010/02/is-the-city-ularity-near.html) [6](https://www.overcomingbias.com/2008/11/fund-ubertool.html) [7](https://www.overcomingbias.com/2008/11/setting-the-sta.html)) against the plausibility of this scenario. Its not that its impossible, or that no one should work on it, but that far too many take it as a default future scenario. But I haven’t written on it for many years now, so perhaps it is time for an update. Recently we have seen noteworthy [progress](https://www.overcomingbias.com/2022/04/ailanguageprogess.html) in AI system demos (if not yet commercial application), and some have urged me to update my views as a result.
The recent systems have used relative simple architectures and basic algorithms to produce models with enormous numbers of parameters from very large datasets. Compared to prior systems, these systems have produced impressive performance on an impressively wide range of tasks. Even though they are still quite far from displacing humans in any substantial fraction of their current tasks.
For the purpose of reconsidering foom, however, the key things to notice are: (1) these systems have kept their values quite simple and very separate from the rest of the system, and (2) they have done basically zero self-reflection or self-improvement. As I see AAGI as still a long way off, the features of these recent systems can only offer weak evidence regarding the features of AAGI.
Even so, recent developments offer little support for the hypothesis that AAGI will be created soon via the process of self-reflection and self-improvement, or for the hypothesis that such a process risks large “value drifts”. These current ways that we are now moving toward AAGI just don’t look much like the foom scenario. And I don’t see them as saying much about whether ems or AAGI will appear first.
Again, I’m not saying foom is impossible, just that it looks unlikely, and that recent events haven’t made it seem moreso.
These new systems do suggest a substantial influence of architecture on system performance, though not obviously at a level out of line with that in most prior AI systems. And note that the abilities of the very best systems here are not that much better than that of the 2nd and 3rd best systems, arguing weakly against AAGI scenarios where the best system is vastly better.