China News Service, Beijing, February 22 (Reporter Yuan Xiuyue) "Try reading "The Romance of the Three Kingdoms" to it" ""The Three-Body Problem" can be directly converted from a novel to a movie"... Recently, OpenAI released the first Vincentian video model Sora. The Internet quickly became popular, and many netizens were eager to try it, looking forward to using AI to restore scenes in novels in the future.
Some people predict that in the future everyone will be able to make movies and TV series that they are satisfied with.
Will this be possible in the future?
In other words, how far is it from realizing this?
Screenshot from video generated by Sora
Enter the novel into AI, and then what?
Before talking about Sora, I have to mention that many people in the film, television and game industry have already applied AIGC (artificial intelligence generated content) to content production.
Mr. Feng ("AI Madhouse"), a blogger from Sichuan, has worked in the CG field for 15 years and has been engaged in film and television special effects post-production, game development, etc.
He said that the film and television industry is now widely accepting the entry of AI, which is currently mainly used in conceptual design and stage design in the early stages of film and television. Some animation teams and game development companies are also laying out or introducing AI production lines.
Mr. Feng recently released a number of "Journey to the West" series of AI concept animations on the short video platform, which have received nearly one million views.
He told reporters that the tools he used were the AI painting tool Midjourney and the AI video generation tool Runway.
"Each shot must be conceived first, and then drawn through Midjourney. A shot may require thousands of drawings, and finally one is selected. For the episode "The Monkey King Comes Out", I drew three to four thousand, and finally selected one hundred. The left and right shots are then fed to AI tools to generate animations, and then edited. The lines are designed first, and AI dubbing is used to dub them. For some special ones, I will dub them myself, and then use a voice changer to adjust the effects."
Mr. Feng said it would take about a week to produce such a video using AI, but it could take several months if done manually.
He said that generally doing animation requires conceptual design, original painting, 2D frame-by-frame storyboarding, 3D scene construction and other processes. But with AI assistance, you only need the original painting, and AI will help you draw the picture and generate animation effects. The process will save a lot of time.
The advent of Sora may save more of this process.
In Mr. Feng’s view, Sora can generate multiple shots in one picture, which is something he couldn’t do with the tools he used in the past. This means that he can create smoother and more complete works in the future.
Screenshot from video generated by Sora
So, if you input the novel into it, can you get a satisfactory video?
Mr. Feng believes that it still needs a process. If it is quick, it may take three or four years.
"Actually, it's not a technical problem. The difficulty is that humans can understand literary works from different countries and understand their backgrounds and different cultural elements, but AI doesn't understand these things well yet."
Mr. Feng mentioned that in the process of using AI tools to create, he could clearly feel that it did not have a thorough understanding of different cultures. For example, Chinese culture seems to have similar styles, but if you look closely, the patterns on clothes and the shapes of armor are not the same. Doesn't really exist.
However, he also said that AI learning is growing exponentially, and it only took more than a year to iterate from the abstraction of pictures to the ability to understand the richness and style of pictures.
Sora does have flaws, but this is just an iterative process of AI and will not be a big deal in the future.
Screenshot from video generated by Sora
Sora comes out and everyone becomes a director?
Although Sora is not currently open to the public, many people believe that its significance is no less than the release of ChatGPT a year ago.
From a technical perspective, where is Sora strong?
Zhang Jinbao, associate professor at the Department of Education of Beijing Normal University, said that Sora uses a combination of diffusion model and Transformer architecture (Diffusion Transformers, DiTs).
Diffusion models corrupt the image by gradually adding noise and then learn the inverse process to restore the image.
The Transformer architecture can capture global dependencies in images.
Sora's generation process can be divided into the following steps: convert the text description into a series of semantic vectors; input the semantic vectors into the model to generate a potential representation of the video; decode the potential representation into pixels to generate the final video.
Zhang Jinbao said that what we have seen so far is that Sora uses a large number of video and image data sets for training, including movies, TV series, documentaries, game scenes, etc.
These data sets cover a variety of scenes, characters, and actions, providing the model with rich learning material.
In his view, the birth of Sora not only marks a major advancement in video generation technology, but also brings unprecedented opportunities and challenges to content creation, media, entertainment and other industries.
For example: The emergence of Sora indicates that competition in the field of video generation will become more intense, promoting the rapid development of multi-modal AI and various AI application scenarios, especially in industries that require real-world modeling.
Compared with traditional film and television production, the production process using Sora is simpler, the investment cost is lower, and the creative results are faster.
Zhang Jinbao believes that Sora allows creators to use AI tools more freely to express their ideas, reduces the constraints of industrial processes, and provides new perspectives and tools for content creation.
Although Sora is not yet able to completely replace traditional film and television production, its powerful capabilities indicate that the direction of relying on AI to assist human creativity is becoming increasingly clear, which may redefine the way film and television content is created and consumed.
Screenshot from video generated by Sora
"This is one of the reasons why AI has caused so much discussion and will bring pressure to everyone. In fact, the practitioners around me are basically not affected. If you are a mature and experienced art or special effects person, it will not be affected too much. , it will become an auxiliary tool for you." Mr. Feng believes that even with AI, it is unrealistic for everyone to become a director, and in the end it will still be in the hands of a few professionals.
Ma Heliang, executive secretary of the Science Fiction Film Working Committee of the China Film Association, also said in an interview with the media that in the short term, positions related to concept design and video preview will be affected by Sora to a certain extent, but the effect it can actually achieve is not It is not at the level of a theatrical movie, and the movie has a subjective stance, perspective, and expression in its creation, and there is emotional communication and emotional projection when watching it. It is far from being replaced by simply generating a video. Therefore, AI as a technical assistance method has indeed changed. Film production methods and processes can optimize and improve film creation, but it is still too early to say that it can "subvert the entire film industry."
Screenshot from video generated by Sora
What other possibilities are there for AI?
"Since Pandora's box has been opened, it is unrealistic to expect it to be closed." In Mr. Feng's view, in addition to painting, making videos, copywriting, etc., AI still has greater room for development.
"The capabilities demonstrated by the Sora model can allow people to further imagine more application possibilities and bring new changes and innovations to various fields." Zhang Jinbao gave an example. For example, in the field of education, it can be used to produce personalized teaching videos. Simulate experiments and scenarios to create virtual classrooms and provide more convenient educational resources.
In the medical field, it can be used to produce medical animations to help doctors and patients better understand the condition; simulate surgical procedures to help doctors conduct preoperative planning and training; and conduct telemedicine to provide convenient medical services to patients in remote areas.
In the field of scientific research, it can be used to simulate scientific experiments, generate scientific data visualization, and build virtual worlds for scientific research and exploration.
In the commercial field, it can be used to produce product promotion videos, create virtual showrooms, for market research and analysis, to help companies better understand customer needs, and so on.
Screenshot from video generated by Sora
At the same time, the risks that Sora may bring have also attracted much attention, such as it may be used for fraud or deception, the content generated by Sora may be biased or wrong, etc.
Legal professionals believe that with the development of AI technology, legal supervision also needs to keep up.
Service providers or providers also need to take effective measures to prevent various types of discrimination in the process of algorithm design, training data selection, model generation and optimization, and service provision, while avoiding the production, copying, publishing, and dissemination of false information.
In Zhang Jinbao's view, intelligence will fully penetrate human society, and people will learn to understand technology, capture the potential of emerging technologies, and promote the orderly and dynamic development of society in the process of widely using various intelligent technologies to enhance competitiveness and complete tasks.
“Capturing new trends in technological development and making correct decisions within controllable limits not only tests the adaptability of one person, but also the entire society.” (End)