“Think of an enormous space of possibilities, a giant multidimensional sphere. This is Mind Design Space, the set of possible cognitive algorithms. Imagine that somewhere near the bottom of that sphere is a little tiny dot representing all the humans who ever lived — it’s a tiny dot because all humans have basically the same brain design, with a cerebral cortex, a prefrontal cortex, a cerebellum, a thalamus, and so on. It’s conserved even relative to chimpanzee brain design. Some of us are weird in little ways, you could say it’s a spiky dot, but the spikes are on the same tiny scale as the dot itself; no matter how neuroatypical you are, you aren’t running on a different cortical algorithm. Asking “what would superintelligences want” is a Wrong Question. Superintelligences are not this weird tribe of people who live across the water with fascinating exotic customs. “Artificial Intelligence” is just a name for the entire space of possibilities outside the tiny human dot. With sufficient knowledge you might be able to reach into that space of possibilities and deliberately pull out an AI that wanted things that had a compact description in human wanting-language, but that wouldn’t be because this is a kind of thing that those exotic superintelligence people naturally want, it would be because you managed to pinpoint one part of the design space.”

— Eliezer Yudkowsky, AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins (Scientific American, March 1, 2016)