Discussion about this post

User's avatar
The Next Evolution's avatar

The offloading-versus-surrender distinction is the right split, but System 3 isn't as new as the framing suggests. Every era has had a mechanism that generated judgement on someone's behalf, not just executed it. A storyteller round a fire decided what was worth remembering. A printed Bible asserted what was true before most of the room could read it themselves to check. An encyclopaedia editor decided what counted as knowledge and what got left out. None of those are offloading by Shaw and Nave's own definition — they're System 3, a long time before anyone had the term for it.

What's changed isn't that a system now generates judgement instead of supporting it. It's that the older versions of System 3 were slow enough, and visibly authored enough, to interrogate. You could ask who wrote the entry, and when, and check it against something else. A confident wrong answer from an authority nobody built time to question isn't new either — a satnav has been sending drivers into rivers for years, with total conviction and zero judgement about whether the route made sense.

Confidence inflation is the right diagnosis for what's happening now. But the fluency was a design choice, not a discovered property of the model. The Tri-System framework tells us where cognitive surrender happens. It doesn't yet ask why the interface was built to sound equally certain whether it's right or wrong, rather than uncertain by default until it's earned otherwise.

Wendy Willard's avatar

As someone designing enterprise software, I find this to be quite a thought-provoking question: How might we encourage users to think critically about what AI suggests, without discounting those very same suggestions?

1 more comment...

No posts

Ready for more?