Why Human Oversight of AI Fails: Amazon Blames Inattention

Amazon’s top security engineer is challenging a cornerstone of modern AI governance. Eric Brandwine, VP and distinguished engineer at Amazon Security, told The Register that relying on human-in-the-loop oversight as a safety net for AI systems is fundamentally flawed. “Humans are not terribly consistent,” Brandwine stated bluntly. “Human-in-the-loop isn’t necessarily the gold standard.”

His critique strikes at the heart of a widely held assumption: that human review can reliably catch AI errors or prevent harmful outcomes. Brandwine argues that human attention is inherently unreliable, especially when monitoring repetitive or automated tasks. Over time, people become desensitized to anomalies, a phenomenon known as normalization of deviance. This psychological drift means that what once seemed like an alarming error gradually becomes accepted as routine, undermining the very purpose of human oversight.

The Amazon executive’s comments highlight a growing tension in the tech industry. While regulators and ethics boards push for more human involvement in AI decision-making, engineers on the front lines are warning that this approach can create a false sense of security. If the human monitor is expected to catch every mistake but is prone to boredom, fatigue, and cognitive bias, the system may actually be less safe than a well-designed automated one.

Brandwine did not propose abandoning human oversight entirely. Instead, he suggested that companies need to rethink the role of the human in AI governance. Rather than placing a person in the loop as a last line of defense, organizations should design systems that assume human attention will fail and build in additional safeguards accordingly. This might include more robust automated monitoring, clear escalation protocols, and regular retraining to combat desensitization.

The debate is not just academic. As AI systems become more autonomous and handle increasingly sensitive tasks, from content moderation to hiring decisions, the question of who watches the watchers becomes critical. Amazon’s stance serves as a reminder that good intentions in AI governance are not enough. Without acknowledging the very real limitations of human attention, even the most well-meaning oversight frameworks can become a dangerous illusion.