“I think this is a fundamental truth of intelligence: as it increases, it gets more opaque (relative to a fixed level of intelligence). All empirical examples of intelligence, both biological and artificial, point to this being true. Brains are opaque systems and machine learning – the dominant paradigm in AI used commercially – produces behaviours that take the software’s creators completely by surprise.”

— Trent Erikson