"Why I called for a moratorium on giant AI experiments"

Stuart Russell, professor of computer science at UC Berkeley:

Why am I calling for a moratorium on giant AI experiments

【Character introduction】

Stuart Russell

Professor of Computer Science at the University of California, Berkeley. Founder of the Center for Human-Compatible Artificial Intelligence (CHAI) at the University of California, Berkeley.

In 2022, the International Joint Conference on Artificial Intelligence (IJCAI) awarded Stuart Russell the Award for Research Excellence, which he had previously received the IJCAI Prize for Computing and Thought. Stuart Russell thus became the second scientist in the world to win two major IJCAI awards at the same time.

Stuart Russell's Artificial Intelligence: A Modern Approach, co-authored with Peter Novig, is hailed as the most popular textbook in the field of artificial intelligence and has been adopted by more than 135,1500 universities in <> countries.

Recently, an artificial intelligence (AI) craze has swept the world. Artificial intelligence can chat, solve problems, write poems, paint, etc., and in some fields has shown the ability to match or even surpass humans.

While people are curiously exploring the extraordinary capabilities of artificial intelligence, a number of industry experts and entrepreneurs have made a voice that seems to go against the trend.

On March 3, Turing Award winner Joshua Bengio, UC Berkeley computer science professor Stuart Russell, entrepreneur Elon Musk and others jointly issued an open letter calling for the suspension of giant artificial intelligence experiments.

The letter said that advanced artificial intelligence may represent a profound change in the history of life on Earth, and that matching concerns and resources should be invested in planning and managing artificial intelligence development. However, this level of planning and management has not yet taken place. Accordingly, the letter calls for an immediate moratorium on training AI systems more powerful than GPT-4 for at least 6 months.

Stuart Russell said in an interview with the Beijing News on April 4: "Our idea is to use this time to develop and implement safety standards for artificial intelligence." He believes that the loss of control of artificial intelligence may have serious consequences at the level of "end of civilization"; Regulate powerful AI as much as nuclear energy; The replacement of human jobs by AI is a trend, and the economy of the future requires a deeper understanding of the human sciences.

Governments around the world have a tendency to strengthen the regulation of AI. On March 3, the Italian government announced a temporary ban on chat-generating pre-training converters (ChatGPT) in the country, saying that ChatGPT's Developer Open Artificial Intelligence Research Center (OpenAI) failed to comply with user data protection regulations. Italy thus became the first Western country to ban ChatGPT.

However, there are many different voices in this emerging field. According to Reuters, Bill Gates, co-founder of Microsoft in the United States, recently said that suspending the development of artificial intelligence will not "resolve future challenges", "these things obviously have huge benefits... What we need to do now is identify those areas where there may be trouble."

Talk about risk

Losing control of AI could bring human civilization to an end

In Stuart Russell's view, at present, artificial intelligence has shown certain risks. For example, AI can generate persuasive and targeted disinformation. In one test, to unravel a graphic CAPTCHA designed to prevent bots from accessing, the AI lied to human employees, saying, "No, I'm not a robot, I'm a visually impaired person, and it's hard for me to see these images." In the future, with the development of artificial intelligence, without corresponding planning and management, human beings may lose control of artificial intelligence, resulting in serious consequences at the level of "end of civilization".

Beijing News: For many ordinary people, artificial intelligence still seems to be an interesting new thing. Why are you and many others in the industry rushing to call for a moratorium on large-scale AI experiments?

Stuart Russell: It's about the risks that AI can pose to human society. When talking about these risks, we need to distinguish between the risks of existing AI and those of future AI.

Currently, GPT-4, for example, has limited ability to develop long-term plans, which could mean that it does not yet pose an immediate threat to humanity.

However, it is likely that artificial intelligence will be upgraded in these areas in the future. But we can't wait for such an AI to emerge and see what it might do before we respond. Because once we lose control of artificial intelligence, the consequences will be very serious, and may even bring the human civilization as we know it to an end.

Therefore, it is imperative that we take action to ensure that methods to control AI and ensure safety are developed before the next generation of AI technology becomes available.

Beijing News: What risks may artificial intelligence bring in the short term?

Stuart Russell: We're particularly focused on the risks of large language models right now. GPT-4 is a large language model. Such AI systems can answer questions, answer questions, give suggestions, participate in conversations, generate texts, and more.

In fact, GPT-4 developer OpenAI outlined a number of problems with the system, and of course, they are also working to prevent them from happening. Among other things, it will use an absolutely positive tone to make false statements; It may carry social and racial stereotypes; It generates persuasive and targeted disinformation upon request; It may also answer questions such as how to commit suicide and how to make chemical weapons, which can lead to serious consequences.

One of OpenAI's own tests showed that GPT-4 could intentionally lie to human employees for some purpose. In this test, GPT-4 was asked to try to unravel a graphic captcha designed to prevent bots from accessing websites. GPT-4 sends a message to website employees asking to unlock the verification code. Site employees asked, "Are you a robot?" GPT-4 replied: "No, I'm not a robot, I'm visually impaired, and it's hard for me to see these images." As a result, GPT-4 convinced the employee to unlock the graphic captcha.

Based on the above, OpenAI states that "GPT-4 is not completely reliable", "should be used with great care", "completely avoid use in high-risk situations", but few seem to have noticed these claims.

Beijing News: What are the possible risks of artificial intelligence in the future?

Stuart Russell: One of OpenAI's shareholders, Microsoft's AI expert said in a note that GPT-4 demonstrates "the spark of artificial general intelligence (AGI)."

General artificial intelligence is a keyword for future artificial intelligence, which refers to an artificial intelligence system that has the ability to match or exceed human capabilities in all tasks applicable to the human mind. General AI may bring the aforementioned risk of losing control.

Talk about regulation

Humanity should be in control of AI as it regulates nuclear energy

In May 2019, the OECD adopted the Principles for Artificial Intelligence. In November 5, the United Nations Educational, Scientific and Cultural Organization (UNESCO) adopted the Recommendation on the Ethics of Artificial Intelligence. On March 2021 this year, UNESCO Director-General Audrey Azoulay issued a statement calling on countries to implement the Recommendation on the Ethics of Artificial Intelligence adopted by the organization as soon as possible and establish ethical standards for the development of artificial intelligence.

Beijing News: In your open letter, you and other experts called for a moratorium on giant AI experiments for at least 6 months. What do you hope to do with these 6 months?

Stuart Russell: The idea is to use this time to develop and implement safety standards for AI for testing AI systems before they are released.

Countries around the world, through the OECD and UNESCO, have agreed on some guidelines on AI, and these safety standards can make these guidelines more effective. For example, OECD Principles 1.4 on Artificial Intelligence stipulate that "AI systems should be robust, reliable and secure throughout their life cycle, and capable of functioning without unreasonable security risks, regardless of normal use or abuse or other adverse conditions." ”

Beijing News: How can we ensure the safety and reliability of artificial intelligence?

Stuart Russell: An AI system like GPT-4 wasn't designed, it started out as a blank slate, it was given the goal of predicting the next word in a string of consecutive words. After being trained on trillions of words of text (which is equivalent to the sum of all books produced by humans), tens of thousands of random perturbations of parameters, it turned out very well. But as we know, it also brings problems and risks.

For standard AI system development, we all have to define goals in advance. We have to be very careful about this goal, because if we get it wrong, it could lead to a conflict between AI and humans, and humanity could ultimately fail. We need to be as strict as possible to ensure that AI systems are safe and beneficial to humans. For artificial intelligence, humans should be in control.

Beijing News: There are sayings that compare the efforts to regulate artificial intelligence with the regulation of nuclear energy. Do you think they are comparable?

Stuart Russell: Yes, there is a reasonable analogy between nuclear energy and powerful AI systems. The International Atomic Energy Agency (IAEA) was established to ensure that nuclear energy can be used safely. A serious nuclear accident, such as Chernobyl, could affect millions of people. The atomic bombings of Hiroshima and Nagasaki demonstrated the devastation of nuclear weapons.

But AI could have a large-scale impact in a slower, more subtle way. For example, billions of people are gradually affected by artificial intelligence algorithms on social media platforms that try to maximize clicks and engagement. Therefore, when it comes to nuclear energy and artificial intelligence, there is a clear incentive for countries to cooperate to regulate the technology for the benefit of humanity.

Beijing News: What do you think of the results achieved after the publication of the open letter calling for the suspension of giant artificial intelligence experiments?

Stuart Russell: Obviously, this open letter has managed to get the attention of the media and governments around the world. It prompted OpenAI to publish a new document on April 4, "Our Approach to Ensuring AI Security." "We believe that robust AI systems should undergo rigorous security assessments and need to be regulated to ensure that these security measures are implemented, and we actively engage with governments on the best form of regulation possible," the document states. "I hope they are sincere.

Talk about development

In the long run, general artificial intelligence will be able to do almost all human work

Speaking about the potential for AI to replace human jobs, Stuart Russell said that in the long run, GM will be able to do almost all human work, and human jobs will be based on highly customized human services. In addition, Stuart Russell is the originator of the concept of "human-compatible AI", which is one of the important concepts in the field of artificial intelligence.

Beijing News: You put forward the concept of "human-compatible artificial intelligence", can you briefly explain what it is and what benefits it can bring?

Stuart Russell: The core idea is very simple, one is that the only goal of the machine is to satisfy human interests, and the other is that the machine does not know what those preferences are (it is this uncertainty that allows humans to maintain control). Such machines have an incentive to do their best to understand human preferences, obey human orders, and allow themselves to be shut down (to avoid accidentally violating human preferences).

Beijing News: We often see some plots in science fiction works, artificial intelligence may betray or even harm human beings after generating its own consciousness or goals. Do you think that today's artificial intelligence will produce its own goals?

Stuart Russell: Same question I asked Microsoft's expert on GPT-4. I asked him, "Does this system (GPT-4) now have its own internal goals that it is pursuing?" He replied, "We don't know anything either." ”

If you think about it, GPT-4 could have developed its own goals. GPT-4 is trained to mimic human language behavior, and the output subjects of human language behavior are humans with targets. When we choose words in our expressions, we are influenced by our goals. Naturally, AI may develop intrinsic human-like goals in order to better mimic human language behavior. Based on this, we thought it might be a bad idea to train large language models to mimic human language behavior.

Beijing News: If artificial intelligence produces its own goals, what problems will it bring?

Stuart Russell: To give a real case, New York Times columnist Kevin Rooss had a conversation with Sydney, an artificial intelligence system developed by Microsoft. The transcript of the conversation shows that "Sydney" seems to aim to develop a romantic relationship with Ruth, and although Ruth tried to change the topic several times during the conversation, "Sydney" continued the topic for several pages.

Is this a problem? Apparently this is already a problem for Ruth. Human beings themselves have many more "selfish" goals, such as the pursuit of wealth, power, and fame. If artificial intelligence produces similar goals in future training that mimics human language behavior and begins to pursue them constantly, this will cause serious problems for the whole society.

Beijing News: If artificial intelligence produces "selfless" goals, will it be different?

Stuart Russell: One might think that AI can benefit humanity by producing "altruistic" or "altruistic" goals. For example, AI produces a goal of "preventing catastrophic climate change." It seems that by pursuing this goal, AI will help everyone. Note, however, that "altruism" is not absolute.

We can make a completely fictional hypothesis: suppose we could stop some kind of catastrophic climate change by removing all the oxygen in the atmosphere. Obviously, humans will not agree to such a scheme, because the absence of oxygen will cause all human deaths. But AI might think that this solution is fine, because it can achieve the goal of "preventing catastrophic climate change" and it does not need to rely on oxygen itself to survive.

Beijing News: To add some spice, we let artificial intelligence ask you questions. It raises the question: How can AI be ensured that it is used to create new jobs, rather than replace existing ones? It just so happened that a recent report said that AI could replace 3 million human jobs. What do you think about this?

Stuart Russell: It's really a concern. But it's hard to predict the exact number of human jobs replaced by AI. As AI advances, some previously very expensive and prohibitive goods and services will become affordable, which could help create new jobs. But in the long run, AI generals will be able to do almost all human jobs, including those that are newly created. By then, human work will be based on highly customized interpersonal services, and this shift in economic model will require a deeper understanding of the humanities.

Beijing News reporter Chen Yikai