“*Humanity is the awkward prepubescent phase between animal and machine.”

— Arets Paeglis

*“When you entrust so much of your everyday life to those electronic devices, the argument that you aren’t a cyborg isn’t very convincing. To you, those portable terminals are already your second brain. Isn’t that right? It can be said that the history of science is a history of the expansion of the human body’s functionality, in other words, the history of man’s cyberization. That’s why it’s a matter of degree.”

— Toyohisa Senguji, Psycho Pass (anime)

“It can also be argued that DNA is nothing more than a program designed to preserve itself. Life has become more complex in the overwhelming sea of information. And life, when organized into species, relies upon genes to be its memory system. So, man is an individual only because of his intangible memory… and memory cannot be defined, but it defines mankind. The advent of computers, and the subsequent accumulation of incalculable data has given rise to a new system of memory and thought parallel to your own. Humanity has underestimated the consequences of computerization.”

— Puppet Master, Ghost in the Shell (1995)

Major Motoko Kusanagi: “You talk about redefining my identity. I want a guarantee that I can still be myself.”
Puppet Master: “There isn’t one. Why would you wish to? All things change in a dynamic environment. Your effort to remain what you are is what limits you.”

— Ghost in the Shell (1995)

“[T]he conversion of humans to more or less immortal near-gods” that David Friedman describe[s] as the upside of galloping twenty-first-century scientific advance [s]eems rather a dubious plus, and certainly less of one than extinction would be a minus, especially since changing us into “near-gods” could be thought itself a form of extinction rather than a boon because of the discontinuity between a person and a near-god. We think of early hominids as having become extinct rather than as having become us.”

— Richard Posner, Catastrophe: Risk and Response (2004), pp. 148-149

“In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.”

— Eliezer Yudkowsky, Coherent Extrapolated Volition (2004)

“Earth is the cradle of humanity, but one cannot live in a cradle forever.”

— Konstantin Tsiolkovsky (1911)

“[H]uman beings change their terminal values over time. If you write down your currently morality in AI, you are making into a fixed constant what ought to be a running process. An important thought experiment in thinking about this problem is to imagine that the Ancient Greeks had discovered the principle of AI and set out to build an artificial moral agent. In Ancient Greece slavery was common and the status of women was not much higher. If Ancient Greece had possessed the power to look through time, to see our own future, if they had been allowed to peek at us and decide whether we should be allowed to come into existence, they would have vetoed our civilization out of hand for some reason or another – the decay or marital virtue, we no longer rejoice properly in slaying our enemies in hand-to-hand combat. What would the future would have looked like if the Ancient Greeks would have had the capability to build a very powerful AI with their own moral values as fixed constants?

This suggests that fixing your own moral values may be an extremely unwise strategy for building an AI. But that doesn’t mean it’s wise to shrug and give up. Our civilization is not the same as Ancient Greek civilization, but we are unmistakably their heirs. It’s not that Greek morals were tossed away and new morals rolled up by dice at random. [W]e got here by following a pathway from there. If the Greeks had shrugged and said, “we give up, we won’t teach our children anything”, it wouldn’t have led to our future. [I]t is we who have a sense of a direction that we are going in, and giving up, shrugging will not push us forward in that direction.”

— Eliezer Yudkowsky, The Challenge of Friendly AI (Singularity Summit, 2007)

“A planet is the cradle of mind, but one cannot live in a cradle forever.”

— Konstantin Tsiolkovsky (1911)

“[T]he history of thought [reveals] discrepancy between the intuitions of one age and those of a subsequent generation. But where the conflicting beliefs are not contemporaneous, it is usually not clear that the earlier thinker would have maintained his conviction if confronted by the arguments of the later. The history of thought, however, I need hardly say, affords abundant instances of similar conflict among contemporaries; and as conversions are extremely rare in philosophical controversy, I suppose the conflict in most cases affects intuitions – what is self-evident to one mind is not so to another. It is obvious that in any such conflict there must be error on one side or the other, or on both. The natural man will often decide unhesitatingly that the error is on the other side. But it is manifest that a philosophic mind cannot do this, unless it can prove independently that the conflicting intuitor has an inferior faculty of envisaging truth in general or this kind of truth; one who cannot do this must reasonably submit to a loss of confidence in any intuition of his own that thus is found to conflict with another’s.”

— Henry Sidgwick, Further on the Criteria of Truth and Error (Essays on Ethics and Method, 2000, p. 168)

“Consider an AI that has hedonism as its final goal, and which would therefore like to tile the universe with “hedonium” (matter organized in a configuration that is optimal for the generation of pleasurable experience). To this end, the AI might produce computronium (matter organized in a configuration that is optimal for computation) and use it to implement digital minds in states of euphoria. In order to maximize efficiency, the AI omits from the implementation any mental faculties that are not essential for the experience of pleasure, and exploits any computational shortcuts that according to its definition of pleasure do not vitiate the generation of pleasure. For instance, the AI might confine its simulation to reward circuitry, eliding faculties such as a memory, sensory perception, executive function, and language; it might simulate minds at a relatively coarse-grained level of functionality, omitting lower-level neuronal processes; it might replace commonly repeated computations with calls to a lookup table; or it might put in place some arrangement whereby multiple minds would share most parts of their underlying computational machinery (their “supervenience bases” in philosophical parlance). Such tricks could greatly increase the quantity of pleasure producible with a given amount of resources.”

— Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford (2014), p. 140

“The better the AI, the less we will understand how it works.”

— Kevin Kelly

“With machine intelligence and other technologies such as advanced nanotechnology, space colonization should become economical. Such technology would enable us to construct “von Neumann probes” – machines with the capability of traveling to a planet, building a manufacturing base there, and launching multiple new probes to colonize other stars and planets. A space colonization race could ensue. Over time, the resources of the entire accessible universe might be turned into some kind of infrastructure, perhaps an optimal computing substrate (“computronium”). Viewed from the outside, this process might take a very simple and predictable form – a sphere of technological structure, centered on its Earthly origin, expanding uniformly in all directions at some significant fraction of the speed of light. What happens on the “inside” of this structure – what kinds of lives and experiences (if any) it would sustain – would depend on initial conditions and the dynamics shaping its temporal evolution. It is conceivable, therefore, that the choices we make in this century could have extensive consequences.”

— Nick Bostrom, The future of humanity (A Companion to the Philosophy of Technology, 2009, pp. 555-556)

“I think this is a fundamental truth of intelligence: as it increases, it gets more opaque (relative to a fixed level of intelligence). All empirical examples of intelligence, both biological and artificial, point to this being true. Brains are opaque systems and machine learning – the dominant paradigm in AI used commercially – produces behaviours that take the software’s creators completely by surprise.”

— Trent Erikson

“Man becomes, as it were, the sex organs of the machine world…”

— Marshall McLuhan

“Building a controlled human-level general intelligence that would precisely protect human values is an incredibly difficult and daunting, but immensely important and urgent task. [I]t’s impossible to grasp that in some corner of the Universe there could be this one tiny planet that just happens to spawn replicators that over billions of painful years of natural selection happen to create vast amounts of both increasingly intelligent and sentient beings, some of which happen to become just intelligent enough to soon have a one shot at creating this final invention of god-like machines that could turn the whole Universe into either a likely hell or unlikely utopia. And here we are, a tiny fraction of those almost “just intelligent enough” beings, contemplating this thing that’s likely to happen within our lifetimes and realizing that the chance of either scenario coming true may hinge on what we do. What are the odds?!”

— Matīss Apinis

“[A]s a species, we are very poor at programming. Our brains are built to understand other humans, not computers. We’re terrible at forcing our minds into the precise modes of thought needed to interact with a computer, and we consistently make errors when we try. That’s why computer science and programming degrees take such time and dedication to acquire: we are literally learning how to speak to an alien mind, of a kind that has not existed on Earth until very recently.”

— Stuart Armstrong, Smarter Than Us: The Rise of Machine Intelligence (2014)

“[T]he evolution of superior intelligence in humans was bad for chimpanzees, but it was good for humans. Whether it was good or bad “from the point of view of the universe” is debatable, but if human life is sufficiently positive to offset the suffering we have inflicted on animals, and if we can be hopeful that in future life will get better both for humans and for animals, then perhaps it will turn out to have been good. Remember Bostrom’s definition of existential risk, which refers to the annihilation not of human beings, but of “Earth-originating intelligent life.” The replacement of our species by some other form of conscious intelligent life is not in itself, impartially considered, catastrophic. Even if the intelligent machines kill all existing humans, that would be, as we have seen, a very small part of the loss of value that Parfit and Bostrom believe would be brought about by the extinction of Earth-originating intelligent life. The risk posed by the development of AI, therefore, is not so much whether it is friendly to us, but whether it is friendly to the idea of promoting wellbeing in general, for all sentient beings it encounters, itself included.”

— Peter Singer, Doing the Most Good: How Effective Altruism Is Changing Ideas about Living Ethically (2015), p. 176

“Call them soldiers, call them monks, call them machines: so they were but happy ones, I should not care.”

— Jeremy Bentham, Panopticon (1787)

“We have some concept of how on Earth life has evolved and how some simple beginnings led to creatures like ourselves with at least a certain level of intelligence. There seems to be a gradual increase in intelligence, and human beings at some stage surpassed other creatures, but we have no idea how inevitable that was. The four billion years of Darwinian evolution are now part of common culture. But most people, nonetheless, tend to think that human beings are in some sense the culmination. There’s no particular reason to think that intelligence couldn’t develop further. Astronomers know that our Sun is less than halfway through its life. It will be six billion years before the Sun flares up, engulfing the inner planets, vaporizing any life that remains on Earth. But any life that remains at that time will be life won’t be humans. It will be life at least as different from us as we are from bacteria. Because there is much time for future evolution on Earth and beyond before the Sun dies, as there has been to get from the simplest organisms to us.”

— Sir Martin Rees, What We Still Don’t Know: “Are We Real?” (Channel 4 series, 2004)

“We need to broaden our sympathies both in space and time, and perceive ourselves as part of a long heritage and stewards for an immense future.”

— Sir Martin Rees

“I believe that life can go on forever. It takes a million years to evolve a new species, ten million for a new genus, one hundred million for a class, a billion for a phylum – and that’s usually as far as your imagination goes. In a billion years, it seems, intelligent life might be as different from humans as humans are from insects. But what would happen in another ten billion years? It’s utterly impossible to conceive of ourselves changing as drastically as that, over and over again. All you can say is, on that kind of time scale the material form that life would take is completely open. To change from a human being to a cloud may seem a big order, but it’s the kind of change you’d expect over billions of years.”

— Freeman Dyson

“[T]he human mind is used to thinking in terms of decades or perhaps generations, not the hundreds of millions of years that is the time frame for life on Earth. Coming to grips with humanity in this context reveals at once our significance in Earth history, and our insignificance. There is a certainty about the future of humanity that cheats our mind’s comprehension: one day our species will be no more.”

— Richard Leakey, Roger Lewin, The Sixth Extinction: Patterns of Life and the Future of Humankind (1995), p. 224

“With respect to immortality, nothing shows me how strong and almost instinctive a belief is, as the consideration of the view now held by most physicists, namely that the sun with all the planets will in time grow too cold for life, unless indeed some great body dashes into the sun and thus gives it fresh life. Believing as I do that man in the distant future will be a far more perfect creature than he now is, it is an intolerable thought that he and all other sentient beings are doomed to complete annihilation after such long-continued slow progress. To those who fully admit the immortality of the human soul, the destruction of our world will not appear so dreadful.”

— Charles Darwin, The Autobiography of Charles Darwin, London (1958) p. 92

“The real difference between humans and all other animals is not on the individual level; it’s on the collective level. Humans control the planet because they are the only animals that can cooperate both flexibly and in very large numbers.”

— Yuval Noah Harari, What explains the rise of humans? (TEDGlobalLondon, 2015)

“If the judgment of experts is not reliable, then, probably, neither is anyone elses. This suggests that it is unjustified to be highly certain of AGI being near, but also of it not no being near. [O]ur brains are known to think about uncertain, abstract ideas like AGI in “far mode,” which also makes it feel like AGI must be temporally distance, [b]ut something being uncertain is not strong evidence that it is far away. Thus, we shouldn’t be highly confident that AGI will arrive this century, and we shouldn’t be highly confident that it won’t.”

— Kaj Sotala, Roman V. Yampolskiy, Responses to Catastrophic AGI Risk: A Survey (2015)

“What enables us alone, of all the animals, to cooperate in such a way? [W]e can cooperate flexibly with countless numbers of strangers, because we alone, of all the animals on the planet, can create and believe fictions, fictional stories. And as long as everybody believes in the same fiction, everybody obeys and follows the same rules, the same norms, the same values.”

— Yuval Noah Harari, What explains the rise of humans? (TEDGlobalLondon, 2015)

“I hang out with people who are worried about this, and it’s actually still hard to keep this concern in view. The moment you spend ten minutes not thinking about it, the moment it takes for you to start thinking about it makes you think that maybe it’s just crazy, all bullshit. I mean, what, is there really going to be a superintelligent machine that’s going to swallow the world? I mean, this is unlike other things – I mean other things have this character, like it’s hard to worry about climate change because it’s an abstraction. I mean, it’s hot out there today, but is it hotter than it used to be. In the cold war, I mean, we still have these ICBMs pointed in all of the wrong directions. We’re living under a constant threat of something going wrong and annihilating the better part of humanity and yet it’s easier to worry about Twitter than to worry about that. But this thing is so lampoonable and it’s – it’s just kind-of a goofy notion which seems too strange to be the substance of good credible fiction, and yet when you look at the assumptions you need to get on the train, there’s only two, and they’re very hard to doubt the truth of. We just have to keep making progress and there’s nothing magical about biological material in terms of an intelligent system.

And yet the only thing scarier than developing artificial intelligence is not developing it. Intelligence is our only asset, ultimately – it’s given us everything good. We want more of it. Progress in this area seems almost like an intrinsic good. Whatever you want, you want to be able to solve problems and anticipate the negative consequences of your doing good things and mitigate those, and intelligence is just the thing you use to do that.”

— Sam Harris

“[P]eople who say, for example, that we’ll only be able to improve AI up to the human level because we’re human ourselves, and then we won’t be able to push an AI past that. I think that if this is how the universe looks in general, then we should also observe, e.g., diminishing returns on investment in hardware and software for computer chess past the human level, which we did not in fact observe.”

— Eliezer Yudkowsky, AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins (Scientific American, March 1, 2016)

“Even if you have good mechanisms that solve the AI control problem, you still have the political problem of not letting it be used for bad intentions. So we still have the humanities’ or subset of humanities’ control problem.”

— Nick Bostrom, Effective Altruism Global 2015: Oxford University

“[T]he current situation where humanity is spending more on marketing lipstick in New York than on ensuring the future of the next several billion years and hundred million galaxies, not to mention for those with more selfish and short-sighted vision their own survival in the next few decades, the tiny fraction of resources that we are currently spending on this problem is not defensible, it is insane. Philanthropy has always been insane. There is no sufficient market in philanthropy, there is no sufficient market in expected utility. Even within a certain class of charitable intervention, like trying to save lives in Africa, you will find charities that are 1000 times as efficient as other charities. Can you imagine having a stock that predictably delivered 1000 times the return of the stocks? This would not happen in a sufficient market. This is because people care about money in a way that they do not quite care about maximizing the return on marginal investment in expected utility when they do philanthropy.”

— Eliezer Yudkowsky, Becoming a Rationalist (Conversations from the Pale Blue Dot Podcast #088)

“You can look at the resources we’re currently investing in global warming, how much we’re currently investing marketing lipstick in New York, and how much we’re currently investing solving the friendly AI problem, it is very clear that the next marginal philanthropic investments should be going into friendly AI. We are rationalists, [w]e are not doing this because we wandered into it at random. We’re doing this because there has to be one cause in the world that has the single highest marginal return on investment in expected utility, and friendly AI is it. And if it were that it were not it, we would be off doing something else. Right now, this is where the maximum marginal return on investment is.”

— Eliezer Yudkowsky, Becoming a Rationalist (Conversations from the Pale Blue Dot Podcast #088)

“If you look at all the things the world is spending money on, what we [at the Future of Humanity Institute] are doing is less than a pittance. You go to some random city and you travel from the airport to your hotel. Along the highway you see all these huge buildings for companies you have never heard of. Maybe they are designing a new publicity campaign for a razor blade. You drive past hundreds of these buildings. Any one of those has more resources than the total that humanity is spending on this field.”

— Nick Bostrom

“There is a conceivable world where there is no intelligence explosion and no superintelligence. Or where, a related but logically distinct proposition, the tricks that machine learning experts will inevitably build up for controlling infrahuman AIs carry over pretty well to the human-equivalent and superhuman regime. Or where moral internalism is true and therefore all sufficiently advanced AIs are inevitably nice. In conceivable worlds like that, all the work and worry of the Machine Intelligence Research Institute comes to nothing and was never necessary in the first place, representing some lost number of mosquito nets that could otherwise have been bought by the Against Malaria Foundation.

There’s also a conceivable world where you work hard and fight malaria, where you work hard and keep the carbon emissions to not much worse than they are already (or use geoengineering to mitigate mistakes already made). And then it ends up making no difference because your civilization failed to solve the AI alignment problem, and all the children you saved with those malaria nets grew up only to be killed by nanomachines in their sleep.

[I] think that people who try to do thought-out philanthropy, e.g., Holden Karnofsky of GiveWell, would unhesitatingly agree that these are both conceivable worlds we prefer not to enter. The question is just which of these two worlds is more probable as the one we should avoid.

[I] think it’s the first world that’s improbable and the second one that’s probable. I’m aware that in trying to convince people of that, I’m swimming uphill against a sense of eternal normality – the sense that this transient and temporary civilization of ours that has existed for only a few decades, that this species of ours that has existed for only an eyeblink of evolutionary and geological time, is all that makes sense and shall surely last forever. But given that I do think the first conceivable world is just a fond dream, it should be clear why I don’t think we should ignore a problem we’ll predictably have to panic about later.”

— Eliezer Yudkowsky, AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins (Scientific American, March 1, 2016)

“The future is ours to shape. I feel we are in a race that we need to win. It’s a race between the growing power of the technology and the growing wisdom we need to manage it. Right now, almost all the resources tend to go into growing the power of the tech.”

— Max Tegmark

“I actually think it would be a huge tragedy if machine superintelligence were never developed. That would be a failure mode for our Earth-originating intelligent civilization.”

— Nick Bostrom

“[Artificial intelligence] is the technology that unlocks this much larger space of possibilities, of capabilities, that enables unlimited space colonization, that enables uploading of human minds into computers, that enables intergalactic civilizations with planetary-size minds living for billions of years.”

— Nick Bostrom

“As the fate of the gorillas now depends more on humans than on the actions of the gorillas themselves, so would the fate of humanity depend on the actions of the machine superintelligence.”

— Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014)

“[T]he combination of value misalignment with increasingly capable decision-making systems can lead to problems – perhaps even species-ending problems if the machines are more capable than humans.”

— Stuart Russell, Will They Make Us Better People? (Edge.org 2015 question: What do you think about machines that think?)

“[Due to the stakes, which require erring on the safe side] there is no need for a proof, just a convincing argument pointing to a more-than-infinitesimal possibility.”

— Stuart Russell

“Whereas the short-term impact of AI depends on who controls, the long-term impact depends on whether it can be controlled at all. All of us should ask ourselves what we can do now to improve the chances of reaping the benefits and avoiding the risks.”

— Stephen Hawking, Stephen Hawking: AI could be the end of humanity (The Independent, December 2, 2014)

“[H]ominids definitely didn’t need exponentially vaster brains than chimpanzees. And John von Neumann didn’t have a head exponentially vaster than the head of an average human. And on a sheerly pragmatic level, human axons transmit information at around a millionth of the speed of light, even when it comes to heat dissipation each synaptic operation in the brain consumes around a million times the minimum heat dissipation for an irreversible binary operation at 300 Kelvin, and so on. Why think the brain’s software is closer to optimal than the hardware? Human intelligence is privileged mainly by being the least possible level of intelligence that suffices to construct a computer; if it were possible to construct a computer with less intelligence, we’d be having this conversation at that level of intelligence instead.”

— Eliezer Yudkowsky, AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins (Scientific American, March 1, 2016)

“[I]t’s tempting to dismiss the notion of highly intelligent machines as mere science fiction. But this would be a mistake, and potentially our worst mistake in history.”

— Stephen Hawking, Stuart Russell, Max Tegmark, Frank Wilczek, Stephen Hawking: ‘Transcendence looks at the implications of artificial intelligence – but are we taking AI seriously enough?’ (The Independent, May 1, 2014)

“”Human intelligence” is often compared to “chimpanzee intelligence” in a manner that presents the former as being so much more awesome than, and different from, the latter. Yet this is not the case. If we look at individuals in isolation, a human is hardly that much more capable than a chimpanzee. They are both equally unable to read and write on their own, not to mention building computers or flying to the moon. And this is also true if we compare a tribe of, say, thirty humans with a tribe of thirty chimpanzees. Such two tribes rule the Earth about equally little. What really separates humans from chimpanzees, however, is that humans have a much greater capacity for accumulating information, especially through language. And it is this – more precisely, millions of individuals cooperating with this, in itself humble and almost useless, ability – that enables humans to accomplish the things we erroneously identify with individual abilities: communicating with language, doing mathematics, uncovering physical laws, building things, etc. It is essentially this you can do with a human that you cannot do with a chimpanzee: train them to contribute modestly to society. To become a well-connected neuron in the collective human brain. Without the knowledge and tools of previous generations, humans are largely indistinguishable from chimpanzees.”

— Magnus Vinding, Reflections on Intelligence

“From a suffering-focused perspective, the main reason to be concerned about the risks from artificial intelligence is not the possibility of human extinction or the corresponding failure to build a flourishing, intergalactic civilization. Rather, it is the thought of misaligned or “ill-aligned” AI as a powerful but morally indifferent optimization process which, in the pursuit of its goals, may transform galactic resources into (among other things) suffering.”

— Lukas Gloor, Suffering-focused AI safety: Why “fail-safe” measures might be our top intervention (2016)

“If you don’t know anything about computers, just remember that they are machines that do exactly what you tell them but often surprise you in the result.”

— Richard Dawkins, The Blind Watchmaker: Why the Evidence of Evolution Reveals a Universe Without Design

“A machine that thinks won’t always think in the ways we want it to. And we’re not ready for the ramifications of that.”

— Bruce Schneier, When Thinking Machines Break The Law (Edge.org 2015 question: What do you think about machines that think?)

“[T]he victims do not care about the agent’s inner thoughts, their evolution towards “being good”, possible resentments [or] indifference. This argument is even stronger for an agent that we create deliberately to act morally since all of us will be the potential victims and it does not help us if an AI has a good will or behaves according to certain rules if this leads to suffering. A sufficiently powerful artificial intelligence is like a mechanism or a force of nature and we do not care whether a thunderstorm has good intentions or behaves according to some rules as long as it does not harm us.”

— Caspar Öesterheld, Machine Ethics and Preference Utilitarianism (May 25, 2015)

“Turing (1951) predicted that [an AI with human or superhuman levels of general intelligence] might soon thereafter lead to a scenario not unlike what Good (1965) later coined an intelligence explosion, where an AI’s intelligence level spirals quickly to levels far beyond human capabilities. [A]nd once AIs far more intelligent than ourselves exist, we cannot realistically count on remaining in control, so from that point our fate will be in their hands, and depend on their goals and values.”

— Olle Häggström, Here Be Dragons: Science, Technology and the Future of Humanity, Chapter “Doomsday nevertheless?”, pp. 195-196

“An instructive and oft-repeated example introduced by Bostrom (2003c) is the paperclip maximizer. The seed AI is given the goal of producing as many paperclips as it can. Once this results in a superintelligent AGI, the machine is likely to find ways to transform most of our planet into a monstrous heap of paperclips, followed by a similar transformation of the entire solar system, and probably the Milky Way, and most of the observable universe. Such a scenario will look very unappetizing to us humans, and as soon as we realize what the machine is up to we would try to do everything in our power to stop it. But we’re up against someone who is so much more intelligent than we are that our chances of succeeding are microscopic. Perhaps most likely, before we even have the time to think about how to organize our resistance, the machine will have realized what we might be up to, and exterminated us simply as a safety precaution.”

— Olle Häggström, Here Be Dragons: Science, Technology and the Future of Humanity, Chapter “Computer revolution”, p. 116

“[A paperclip maximizer] might be a sensible goal for someone who owns a paperclip factory and sets out to fully automate it by means of an AGI. What seems a bit silly, however, is for someone who plans to take our civilization into its neyxt era by means of an intelligence explosion to choose such a narrow and pedestrian goal as paperclip maximization. What makes the paperclip maximizer intelligence explosion a somewhat less silly scenario is that an intelligence explosion might be triggered by mistake. We can imagine a perhaps not-so-distant future in which moderately intelligent AGIs are constructed for all sorts of purposes, until one day one of the engineering teams happens to be just a little bit more successful than the others and creates an AGI that is just above the intelligence threshold for serving as a seed AI.”

— Olle Häggström, Here Be Dragons: Science, Technology and the Future of Humanity, Chapter “Computer revolution”, p. 116

“A paperclip maximizer is not making a computational error by having a preference order on outcomes that prefers outcomes with more paperclips in them. It is not standing from within your own preference framework and choosing blatantly mistaken acts, nor is it standing within your meta-preference framework and making mistakes about what to prefer. It is computing the answer to a different question than the question that you are asking when you ask, “What should I do?” A paperclip maximizer just outputs the action leading to the greatest number of expected paperclips.”

— Eliezer Yudkowsky, AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins (Scientific American, March 1, 2016)

“Even an agent that has an apparently very limited final goal, such as “to make 32 paperclips”, could pursue unlimited resource acquisition if there were no relevant cost to the agent of doing so. For example, even after an expected-utility-maximizing agent had built 32 paperclips, it could use some extra resources to verify that it had indeed successfully built 32 paperclips meeting all the specifications (and, if necessary, to take corrective action). After it had done so, it could run another batch of tests to make doubly sure that no mistake had been made. And then it could run another test, and another. The benefits of subsequent tests would be subject to steeply diminishing returns; however, so long as there were no alternative action with a higher expected utility, the agent would keep testing and re-testing (and keep acquiring more resources to enable these tests).”

— Nick Bostrom, The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents (2012)

“”[W]hat would a superintelligent AI want?” [T]his is the wrong question. [H]umans come from the factory with a number of [b]uilt-in drives and also a built-in capacity to pick up a morality from their environment and upbringing which nevertheless has to match up with the built-in drives or it wouldn’t be acquired. The same way that we have a language acquisition capacity that doesn’t work on arbitrary grammars, but a certain [b]uilt-in syntax. That [c]orresponds to our experience so we expect all minds (because of all minds with whom we’ve had an experience with) have, on the one hand, innate [s]elfishness as a drive, self-concern as a drive, yet to respond positively to positive gestures. So they’re thinking, “[W]e’ll build AIs, and the AIs will, of course, want some resources for themselves, but if we’ll be nice to them, they’ll probably be nice to us. On the other hand, if we’re cruel to them and we try to enslave them, then they’ll resent that and become rebellious and try to break free.” [A]ll human minds are only a single dot within the space of all possible mind designs.”

— Eliezer Yudkowsky, Becoming a Rationalist (Conversations from the Pale Blue Dot Podcast #088)

“Think of an enormous space of possibilities, a giant multidimensional sphere. This is Mind Design Space, the set of possible cognitive algorithms. Imagine that somewhere near the bottom of that sphere is a little tiny dot representing all the humans who ever lived — it’s a tiny dot because all humans have basically the same brain design, with a cerebral cortex, a prefrontal cortex, a cerebellum, a thalamus, and so on. It’s conserved even relative to chimpanzee brain design. Some of us are weird in little ways, you could say it’s a spiky dot, but the spikes are on the same tiny scale as the dot itself; no matter how neuroatypical you are, you aren’t running on a different cortical algorithm. Asking “what would superintelligences want” is a Wrong Question. Superintelligences are not this weird tribe of people who live across the water with fascinating exotic customs. “Artificial Intelligence” is just a name for the entire space of possibilities outside the tiny human dot. With sufficient knowledge you might be able to reach into that space of possibilities and deliberately pull out an AI that wanted things that had a compact description in human wanting-language, but that wouldn’t be because this is a kind of thing that those exotic superintelligence people naturally want, it would be because you managed to pinpoint one part of the design space.”

— Eliezer Yudkowsky, AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins (Scientific American, March 1, 2016)

“It’s mostly the mammalian lineage that has really complicated mind designs at all. [T]he amazing thing about natural selection is not how well it works, but that it works at all without a brain. This is a very counterintuitive idea that you can actually get complicated designs without there being any brain to design them. It doesn’t mean that doing it without a brain is actually better, that is false praise.”

— Eliezer Yudkowsky, Becoming a Rationalist (Conversations from the Pale Blue Dot Podcast #088)

“The human brain is the lowest level of intelligence that suffices to build computer chips. We’re literally as dumb as you can get and still build AIs.”

— Eliezer Yudkowsky, Becoming a Rationalist (Conversations from the Pale Blue Dot Podcast #088)

“Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization – a niche we filled because we got there first, not because we are in any sense optimally adapted to it.”

— Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014)

“[D]umb evolutionary processes have dramatically amplified the intelligence in the human lineage even compared with our close relatives the great apes and our own humanoid ancestors; and there is no reason to suppose Homo sapiens to have reached the apex of cognitive effectiveness attainable in a biological system.”

— Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014)

*
*“We only have to look at ourselves to see how intelligent life might develop into something we wouldn’t want to meet.”

— Stephen Hawking

“[I]t seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers. At some stage therefore we should have to expect the machines to take control[.]”

— Alan Turing, Intelligent machinery, a heretical theory (1951)

“We humans steer the future not because we’re the strongest or the fastest but because we’re the smartest animal on this planet. However, there are no reasons to assume that blind evolutionary processes have reached the physical limit of intelligence with us. Quite to the contrary, we have already seen how intelligent machines outperform the best of our kind on an increasing number of tasks, ranging from Chess over the quiz show Jeopardy to Go. What will happen when artificial intelligence surpasses human intelligence in a broader range and controls our destiny? Are we ready to make our final invention? And what is to be done from an ethical perspective?

— Kaspar Etter, Superintelligence.ch

“Intelligence is that sort of smartish stuff coming out of brains, which can play chess, and price bonds, and persuade people to buy bonds, and invent guns, and figure out gravity by looking at wandering lights in the sky; and which, if a machine intelligence had it in large quantities, might let it invent molecular nanotechnology; and so on.”

— Eliezer Yudkowsky (2013)

“The purest case of an intelligence explosion would be an Artificial Intelligence rewriting its own source code. The key idea is that if you can improve intelligence even a little, the process accelerates. It’s a tipping point. Like trying to balance a pen on one end – as soon as it tilts even a little, it quickly falls the rest of the way.”

— Eliezer Yudkowsky

“Ultimately, it probably will not be a viable approach to try to stop or limit self-improvement. Just as water finds a way to run downhill, information finds a way to be free, and economic profits find a way to be made, intelligent systems will find a way to self-improve. We should embrace this fact of nature and find a way to channel it toward ends which are positive for humanity.”

— Steve Omohundro, The Basic AI Drives

“The AI neither hates you, nor loves you, but you are made out of atoms that it can use for something else.”

— Eliezer Yudkowsky

“The AI does not hate you, nor does it love you, but you are made of atoms it can use for something else.”

— Luke Muellhauser‘s variation on a quote from Eliezer Yudkowsky

“The best answer to the question, “Will computers ever be as smart as humans?” is probably “Yes, but only briefly.””

— Vernor Vinge

“We probably make thousands of species extinct per year through our pursuit of instrumental goals, why is it so hard to imagine that AGI could do the same to us?”

— Michael Anissimov

*“[“Superintelligence” is t]he clearest book I’ve come across that makes the case that the so-called “control problem” — the problem of building human-level and beyond artificial intelligence that we can control, that we can know in advance will converge with our interests — is a truly difficult and important task, because we will end up building this stuff by happenstance if we simply keep going in the direction we’re headed. Unless we can solve this problem in advance and have good reason to believe that the machines we are building are benign and their behavior predictable — even when they exceed us in intelligence a thousand-, a million-, or a billion-fold — this is going to be a catastrophic intrusion into our lives that we may not survive.”

— Sam Harris

“*My point in this conversation is that the dangers from AI are overblown by media and non-understanding news, and the real danger is the same danger in any complex, less-than-fully-understood code: edge case unpredictability. In my opinion, this is different from “dangerous AI” as most people perceive it, in that the software has no motives, no sentience, and no evil morality, and is merely (ruthlessly) trying to optimize a function that we ourselves wrote and designed.”*
*— unknown Reddit user in “Science AMA Series: Stephen Hawking AMA Answers!”

“The real risk with AI isn’t malice but competence. A superintelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. You’re probably not an evil ant-hater who steps on ants out of malice, but if you’re in charge of a hydroelectric green energy project and there’s an anthill in the region to be flooded, too bad for the ants. Let’s not place humanity in the position of those ants.”

— Stephen Hawking, Science AMA Series: Stephen Hawking AMA Answers!

“One reason for focusing on intelligence, that is, on instrumental rationality, is that this is the most relevant concept if we are trying to figure out what different kinds of systems would do. Normative questions, such as whether their behavior would count as being prudentially rational or morally justifiable, can be important in various ways. However, such questions should not blind us to the possibility of cognitive systems that fail to satisfy substantial normative criteria but which are nevertheless very powerful and able to exert strong influence on the world.”

— Nick Bostrom, The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents (2012)

“[I]t might be tempting to suppose that a superintelligence not facing a competitive social world would see no instrumental reason to accumulate resources beyond some modest level, for instance whatever computational resources needed to run its mind along with some virtual reality. Yet such a supposition would be entirely unwarranted. First, the value of resources depends on the uses to which they can be put, which in turn depends on the available technology. With mature technology, basic resources such as time, space, and matter, and other forms of free energy, could be processed to serve almost any goal. For instance, such basic resources could be converted into life. Increased computational resources could be used to run the superintelligence at a greater speed and for a longer duration, or to create additional physical or simulated (virtual) lives and civilizations. Extra physical resources could also be used to create backup systems or perimeter defenses, enhancing security. Such projects could easily consume far more than one planet’s worth of resources. [I]t could also use the extra resources to build ever-more robust defenses to safeguard the privileged real estate. Since the cost of acquiring additional resources would keep declining, this process of optimizing and increasing safeguards might well continue indefinitely even if it were subject to steeply declining returns.”

— Nick Bostrom, The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents (2012)

“My contention is that machines can be constructed which will simulate the behaviour of the human mind very closely[.] Let us now assume, for the sake of argument, that these machines are a genuine possibility, and look at the consequences of constructing them. To do so would of course meet with great opposition, unless we have advanced greatly in religious toleration from the days of Galileo. There would be great opposition from the intellectuals who were afraid of being put out of a job. It is probable though that the intellectuals would be mistaken about this. There would be plenty to do, trying to understand what the machines were trying to say, i.e. in trying to keep one’s intelligence up to the standard set by the machines, for it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should expect the machines to take control[.]”

— Alan Turing, Intelligent machinery, a heretical theory (1951)

“With artificial intelligence we are summoning the demon.”

— Elon Musk

“I am time, the destroyer of all; I have come to consume the world. Even without your participation, all the warriors gathered here will die.”

— Sage Vyasa, Bhagavad Gita: Krishna to Arjuna (4th-5th Century BCE)

“One day the AIs are going to look back on us the same way we look at fossil skeletons on the plains of Africa. An upright ape living in dust with crude language and tools, all set for extinction.”

— Nathan Bateman, Ex Machina (2015)

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.”

— I.J. Good, Speculations Concerning the First Ultraintelligent Machine (1965)

“Summoned or not, the god will come.”

— Carl Jung