The orthogonality thesis is the idea that an intelligent being (usually, as it happens, a superintelligent AI) may have any combination of intelligence and goal -- that is, knowing how intelligent an agent is gives you no clue as to their intent.

Many people, and many science fiction writers, have assumed that any being as smart or smarter than a human is likely, and perhaps even certain, to deduce certain ethical truths; life has value, the importance of liberty, or simply that pain is bad. However, there is no obvious reason to believe that these things will be either true or even applicable given an agent sufficiently different from ourselves (indeed, these things are not even true across all human agents).

For example, an AI may not value life. After all, a computer program doesn't have what we would call a life, and one that is developing and increasing its own programing may have a very weak concept of personal identity; it can, presumably, make copies of itself at will, may routinely delete old backups, change or reverse preferences for any or no reason, and may simply terminate when its goal is achieved... however arbitrary that goal may be.

Likewise, a self modifying agent isn't likely to see the value of tiptoeing around pain. Pain is, if anything, a tool or a bug, and should be used or deleted. And agents who have bugs that cannot be eliminated, well, best to delete them and build better agents.

The orthogonality thesis occurs in two forms: the strong form, given above, is simply that intelligence does not correlate with goals. The weak form is that there is at least one logically consistent model of behavior that is not identical to the 'good' model of behavior.

This thesis was first formulated by Nick Bostrom in his paper Superintelligent Will. He also proposed the instrumental convergence thesis, which holds that some traits are expected, or even mandatory, in those AIs that are powerful enough for us to be concerned with; these drives include self-preservation, goal-content integrity (maintaining the same goal over time), cognitive enhancement, technological perfection, and resource acquisition -- none of which appear to encourage any specific ethical framework.

Log in or register to write something here or to contact authors.