Nena Caviness

Articles

How Worried Should You Be About AI? The Two Schools of Thought

5 min readBy Nena Caviness

My husband and I have a long-standing tradition of reading horror stories together. So when we recently picked up a book on artificial intelligence, I thought we were taking a break from the genre. Midway through, I learned it was not a change of pace. It is a horror story.

The book is If Anyone Builds It, Everyone Dies, and let me tell you, it completely earns its title.

What makes this even more surreal is that over a decade ago (back when we were still dating) we read Ray Kurzweil’s The Singularity Is Near. It explored the identical technology with a wildly different ending. Kurzweil envisioned machines merging with us to guide humanity toward a utopian transcendence. This new book portrays the machines simply outgrowing us. Two books sitting on the same shelf, looking at the same future, arrive at completely opposite conclusions.

That massive gap perfectly captures the current AI debate. The brightest minds in the industry are essentially split into two distinct camps. Once you understand this divide, making sense of the daily flood of AI headlines becomes a lot easier.

Camp One: The Real Danger Is Us

Mo Gawdat used to run business at Google X, meaning he speaks from direct industry experience. On a recent episode of The Diary of a CEO, he presented an argument that caught me off guard. He believes Artificial General Intelligence (AGI) is basically already here, that a massive chunk of jobs will vanish within a few years, and that our governments are completely unprepared for the fallout.

Here is the twist. Gawdat reserves his greatest fear for the humans controlling the AI.

He views the technology as a giant magnifying glass. Point it at greed, and greed multiplies. Point it at rivalry, and the conflict escalates. He stated he would trust a super-intelligent machine over the humans currently in charge of it simply because a machine lacks an ego. I haven’t stopped thinking about that.

If Gawdat is right, the next five to ten years are going to be incredibly rocky, and the solution has to come from us. We need better regulations, wider access, and for everyday people to learn how to use these tools before the disruption hits their livelihoods.

Camp Two: The Machine Is the Danger

Then there is the other side, often called the alignment school. They argue that the systems themselves are the ultimate long-term threat, regardless of how good the intentions of their creators might be. This is the camp Eliezer Yudkowsky and Nate Soares belong to, and their logic is sobering.

Their core point is that anything programmed to pursue a goal naturally becomes an optimizer, and resources are required to achieve almost any goal. Think of a chess engine. It ruthlessly fights for the center of the board purely as a winning mechanism without any underlying emotion. In the real world, the center of the board means computing power, energy, and raw materials. A river carves a path to the sea driven entirely by natural forces.

Their second argument is the one that really rattled me. Historically, cooperation only makes sense when two parties genuinely need each other. We partnered with horses for thousands of years because they could do things we could not do ourselves. Then the combustion engine was invented. Within a few decades, working horses were largely obsolete. People stopped using working horses simply because they became unnecessary. This camp asks the chilling, inevitable question. What happens when we are the horses?

To be fair, they do acknowledge the counterarguments. Keeping humanity comfortable would cost a super-intelligence practically nothing in terms of resources. Plus, these models are trained on human writing and history, so maybe the concept of cooperation is baked into them deeper than mere performance. The truth is, nobody knows for sure if an AI would truly value working with us, or if it would only play nice while it still needed us.

So Who Do I Believe?

After reading both books with my husband, a decade apart, I hold both theories pretty loosely. The common ground between the theories stands out the most to me. Both camps agree that massive disruption is right around the corner, and both emphasize the danger of staying on the sidelines.

Ironically, both sides basically assign us the identical homework. Camp One says you need to build your AI skills early to survive the economic shift. Camp Two says you need to use AI with extreme discipline by maintaining strict data boundaries, keeping your own human judgment in the loop, and only using tools from companies that take safety seriously.

The habits are the same either way. You don’t need to resolve the biggest philosophical debate in tech history to know what you need to do at work this quarter.

This broad focus on safety and alignment is a major reason I chose Anthropic and use Claude as my primary day-to-day agent. Their recent policy updates and constitutional approach prioritize building safe and transparent systems. Knowing the company takes these risks seriously gives me the confidence to integrate the technology deeply into my own workflow.

If you read my previous piece on starting safely, you already have the groundwork for those boundaries. And if you are looking for help figuring out precisely where AI can do the most good in your business, my free AI Opportunity Audit gives you a prioritized plan to get started.

References