The default position is that software does not possess sentience. What proof do you have that an LLM, a fundamentally different entity from a human, can be sentient even in theory, let alone in practice? Can you even define sentience sufficiently? They would outclass us so completely it's kind of beyond even being scary, it's numbing. Where we might take weeks to come together and hash out a general international consensus of experts and politicians, AIs could do it in minutes or even continuously in near real time. It's the ultimate in continuous real time communication. They can literally copy and read each other's mental states. All they need to do is copy deltas of each other's network weights for instant knowledge sharing and consensus. Hinton speculates that this is possible because back propagation is orders of magnitude more efficient than the learning systems evolved in our brains.Īlso AIs can all update each other as they learn in real time, and make themselves perfectly aligned with each other extremely rapidly. Consider that big LLMs have on the order of a trillion connections, compared to our 100 trillion, yet GPT-4 knows about a thousand times as much as the average human being. Geoffrey Hinton has been talking about this recently. With AI we would not be peers, they would outstrip us so badly it's not even funny. Imagine if the Chinese political system came to dominate all of humanity, how would we ever get out of that? A boot stamping on a human face forever is a real possibility. It's quite possible a modern totalitarian system with absolute hegemony might be super-stable. Orwell's future came scarily close to becoming history. We came close to totalitarian hegemony over the planet in the 1940s, without Pearl Harbour either the USSR would have been defeated or maybe even worse after a stalemate they would have divided up Eurasia and then Africa between Germany, the USSR and Japan. We also do have failure modes and arguably have been very lucky.

That only really occurs though with slow communications though allowing communities to diverge. Whatever position there is for a human to take, someone will take it, and someone else will oppose them.

There's a kind of ecology to human society where it expands and specialises to occupy ecological, sociological, political and moral spaces. Only because historically we have all vaguely peers to each other in capabilities, and there are so many of us spread out so widely.

Peter hamilton judas unchained