“[H]uman beings change their terminal values over time. If you write down your currently morality in AI, you are making into a fixed constant what ought to be a running process. An important thought experiment in thinking about this problem is to imagine that the Ancient Greeks had discovered the principle of AI and set out to build an artificial moral agent. In Ancient Greece slavery was common and the status of women was not much higher. If Ancient Greece had possessed the power to look through time, to see our own future, if they had been allowed to peek at us and decide whether we should be allowed to come into existence, they would have vetoed our civilization out of hand for some reason or another – the decay or marital virtue, we no longer rejoice properly in slaying our enemies in hand-to-hand combat. What would the future would have looked like if the Ancient Greeks would have had the capability to build a very powerful AI with their own moral values as fixed constants?

This suggests that fixing your own moral values may be an extremely unwise strategy for building an AI. But that doesn’t mean it’s wise to shrug and give up. Our civilization is not the same as Ancient Greek civilization, but we are unmistakably their heirs. It’s not that Greek morals were tossed away and new morals rolled up by dice at random. [W]e got here by following a pathway from there. If the Greeks had shrugged and said, “we give up, we won’t teach our children anything”, it wouldn’t have led to our future. [I]t is we who have a sense of a direction that we are going in, and giving up, shrugging will not push us forward in that direction.”

— Eliezer Yudkowsky, The Challenge of Friendly AI (Singularity Summit, 2007)