From Danny Hillis: The Pattern on the Stone, which I am currently reading hunting for simple explanations of technological things:
Because computers can do some things that seem very much like human thinking, people often worry that they are threatening our unique position as rational beings, and there are some who seek reassurance in mathematical proofs of the limits of computers. There have been analogous controversies in human history. It was once considered important that the Earth be at the center of the universe, and our imagined position at the center was emblematic of our worth. The discovery that we occupied no central position – that our planet was just one of a number of planets in orbit around the Sun – was deeply disturbing to many people at the time, and the philosophical implications of astronomy became a topic of heated debate. A similar controversy arose over evolutionary theory, which also appeared as a threat to humankind’s uniqueness. At the root of these earlier philosophical rises was a misplaced judgment of the source of human worth. I am convinced that most of the current philosophical discussions about the limits of computers are based on a similar misjudgment.
And that, I think, is one way to think about the future and intelligence, natural and artificial. Works for me, for now. No idea, of course, whether this still is Danny’s position, but I rather think it is.
See also https://chappyessays.wordpress.com/2017/04/05/the-singularity-will-be-televised/