The world was never the same after humans discovered fire. Fire protected them from the cold, drove away darkness and taught them how to cook food. Yet the same fire has burned city after city and has been used to burn people alive. Gunpowder transformed empires. The steam engine enabled humans to conquer distance, but it also gave rise to brutal systems of forced labour. Nuclear energy can provide light, but it can also wipe out an entire city in a moment. The history of technology, therefore, is not only a history of the advancement of civilisation; it is also a history of risk.

In the age of artificial intelligence (AI), that longstanding tension between technological progress and risks to civilisation is emerging in an even more frightening form. One could also say it is taking a different form. The question is no longer simply about the power of the technology humans create, but also whether human control over that power will remain intact.

From ancient times to the modern era, technology has increased human capabilities and made life easier. But nowadays, artificial intelligence is imitating the very abilities that humans use to think, analyse, make decisions, create and solve problems. It is also routinely surpassing human limitations. Perhaps for the first time in the history of civilisation, humans are trying to create a machine that can think like humans—or better than them.

And this is precisely where the potential threat to human civilisation lies. There are ample reasons for concern.

One of the most widely discussed concepts in AI research today is “recursive self-improvement,” the process by which an AI can improve its own capabilities. Put simply, if an AI can create a more advanced AI, each new improvement will make the next stage of development easier and faster. AI is already being used to create and improve new AI models. At the same time, its ability to carry out long and complex software-development tasks continuously and reliably is growing rapidly.

This is the central source of concern. Humans today create an advanced model. If that model can perform a large part of the coding, debugging, testing and development of new solutions needed for the next model, then each new generation will be able to create the following generation faster than the one before it. At some point, this cycle of improvement could become so rapid that AI capabilities advance far beyond our expectations before humans can fully understand what is happening. This is why researchers worry that if AI capabilities grow faster than our ability to understand, monitor and control them, technological progress itself could eventually become a safety risk.

It is at this point that the issue ceases to be merely a technological one and becomes a question of civilisation. When people hear warnings about the possibility of human extinction, a natural first reaction is: Is this science fiction? Will a machine simply wake up one day and start hating humans? The problem is known as the “alignment problem.” In simple terms, what happens if we give an extremely powerful machine a goal, but it fails to properly understand our true intention or pursues that goal in a way we never imagined?

Philosopher Nick Bostrom’s “paperclip maximiser” is frequently discussed as an illustration of this problem. Imagine that an extremely capable machine is instructed to produce as many paperclips as possible. The goal appears harmless. But if that goal becomes its sole ultimate directive, a sufficiently powerful system might seek to use an ever-increasing share of the world’s resources to produce paperclips—even if humans stand in its way. The machine has no hatred for humans. Nor does it have any desire for revenge. The problem is even more unsettling: it does not actually need to hate us. An intelligence does not have to be “evil” to become dangerous to human civilisation; being sufficiently powerful and incorrectly directed may be enough.

There is another concern as well. If an AI is given a specific goal, it may seek to use more information, more computing power and more resources to achieve that goal. For the same reason, trying to avoid obstacles that might prevent it from continuing its work could also become necessary. From the outside, this might look like an attempt to hold on to power. But in the case of AI, the issue is not emotion or a desire for power; rather, it is about choosing the most mathematically effective path to complete the task it has been given.

To date, there has been no major incident in which an AI has consciously decided to harm humanity. Various abnormal behaviours in existing systems and instances of rules being broken during safety tests have concerned researchers, but drawing a direct conclusion about human extinction from those incidents would not be scientifically cautious.

Even so, it is difficult to simply dismiss these concerns. That is because the warnings are no longer coming only from outside critics; they are also coming from people working within the technology itself. Former Anthropic researcher Jacob Coxon has spoken about the potential extreme risks of AI. Evan Hubinger, who leads research at the same organisation on AI safety and alignment with human values, has given his own assessment that the risk of human extinction from AI over the next decade is greater than 10 percent. This is not a scientifically established probability; it is one researcher’s assessment. But the question is: if the potential loss is the entirety of human civilisation, then forget 10 percent—is even a 1 percent risk something that can be ignored?

Former Google DeepMind researcher Bilal Chughtai has expressed similar concerns. More significantly, executives at the leading companies competing in artificial intelligence are now publicly raising concerns about the rapid advancement of powerful AI and its safety. When the people who understand the engine of the car better than anyone else begin questioning the capability of the brakes, the passengers should at least ask: Where are we going?

The problem, however, is that no single actor has control over the brakes. If one company slows development, another will move ahead. If the United States stops, China will advance. If China stops, someone else may move forward. Technology has become entangled with both market competition and geopolitics. As a result, even if everyone wants safe AI, no one is willing to be the first to slow down. Economics describes situations in which everyone knows that exercising restraint together would be better for all, yet no one wants to stop first for fear of falling behind in the competition. The race to develop artificial intelligence has now fallen into much the same global trap.

There are more questions. If AI takes a person’s job, who will receive the additional wealth generated by increased productivity? A handful of companies, or society as a whole? If an algorithm displays bias in decisions about employment, loans, healthcare or security, who will be held responsible? If a model is built using the vast body of work produced by writers, artists and journalists, how much of the value it creates belongs to them? If autonomous weapons can one day make the decision to kill a human being on their own, where should that ethical boundary be drawn? And above all, do a handful of private companies have the moral right to take technological risks whose potential consequences would have to be borne by all of human civilisation?

There are still no easy answers to these questions. But one thing is clear: concerns about AI are not limited to imagined or potential disasters in the future. Humans know where their boundaries lie, and that is part of what has shaped them as human beings. But is it clear where the boundaries of machines should be drawn? And will machines themselves respect those boundaries? Even today, the answer may not be an unequivocal “yes.” But if the answer is “maybe,” then that alone represents a risk that human civilisation cannot afford to ignore.

Mahafuz Rahaman is a journalist and a writer.

[The views expressed in this article are the author’s own and do not necessarily reflect the position of The Daily Bonik Barta or the organisation they work for.]