[Report file] 'Secrets of communication with avatars' learned from the founder of Metaverse

Today, we would like to reveal the secret of the implementation of Neil Stevenson's 'Avatar' in the <SBS D Forum (abbreviated SDF)>, which made a technologically meaningful attempt with Neil Stevenson, the creator of 'Metaverse'.

This is the story of

technology.

SDF is SBS's representative social contribution knowledge sharing platform, which has been planned since 2004 by the Future Team of SBS Press Headquarters since 2004 to observe and find solutions to essential problems in our society that are difficult to look into in depth in general news.

In particular, what kind of 'storytelling' as well as 'agenda setting' befitting the public responsibility of terrestrial broadcasting can get more people's attention and sympathy, and what new experiences technically can be shared through the SDF platform. SDF has always been thinking about whether or not there will be.

This year, in particular, science fiction writer Neil Stevenson, who first coined the terms 'metaverse' and 'avatar' in the 1990s, was the speaker for the SDF.

Since it was SDF's first appearance on the domestic stage, he had to find a meaningful metaverse experience. 

In a pre-meeting with writer Neil Stevenson before the forum, we discussed how to implement the 'metaverse', and suggested how to appear on stage as a 3D avatar.

Then, the company that the artist directly recommended was 'Reallusion', a company that specializes in character and animation production solutions headquartered in Silicon Valley. 

On the 14th, I asked John Martin, vice president of 'Reallusion', how it was working with SBS D Forum.

We were very excited too.

The famous writer Neil Stevenson.

After that, I also got to know about SBS and D-Forum, and I was so surprised to see the confirmed speakers and the topics covered.

It was so good that we were able to do it together.

I found out about Reallusion because Neil Stevenson personally recommended it.

Have you worked with Neil Stevenson in the past?

no.

But I am a fan of him.

He is the father of 'Metaverse' and 'Avatar'.

I was always keeping a close eye on all of his work.

As digital characters and animations began to appear everywhere, it could be said that he was actually honoring his work.

His work always inspires us a lot.

We're basically working on creating digital characters.

I really like the definition of avatar by Neil Stevenson at this SBS D Forum.

Neil Stevenson defined the 

.

With that definition, the possibilities of digital characters that can be implemented are endless.

 <Various versions of 'Neil Stevenson' avatars produced by Reallusion>

What was the most important part of implementing Neil Stevenson's character?

We were mainly in charge of creating characters and making them move.

We wanted the costumes to be changed during the lecture, so we had to make Neils in different outfits.

Neil Stevenson sent a full-body video of the story he wanted to tell, so I was able to apply the audio of that video to the character.

I think we were more than 6000 miles apart, but together we were able to achieve what we wanted.

It became a wonderful collaboration with Epic Games as well.

<Various versions of 'Neil Stevenson' avatars produced by Reallusion>

There are many excellent companies in graphic design in Korea too.

Even so, when we said we wanted to do 3D motion capture, there weren’t many places that we did.

For such a large project, several parts will be involved.

There is also character related.

It also has to do with how similar it can be implemented.

It also has to do with how many different outfits can be implemented.

There is a part about voice and lip sync.

There is also the motion capture part that implements the motion.

And the skill to put them all together using a tool is required.

As a result, vitality is realized on stage.

That's the heart of our technology.

And another thing that was possible was that our organization was relatively small, so we were able to respond to various situations more agilely.

It is not easy for large organizations to respond quickly and agilely.

About 5-6 people from our side participated in this project.

Also,

is becoming more and more important for

these technologies 

.

Many large studios are now gradually changing their perception of real-time as well as tools that respond to real-time, but there are still limitations in going beyond their past practices.

We are very excited to be working together this time and to present a good example of real-time technology.

How did you feel about seeing the final version of Prism and Epic Games working together?

It was fun.

In particular, I thought it was really cool to have the avatar appear in the form of AR in the lecture hall of the SBS D forum.

I've been watching the Korean presidential candidates' sessions in front of the forum.

It was so impressive.

I've always been very interested in American politics, and I've always been interested in the ups and downs of politicians.

It was so good that the presidential candidates' lectures were included.

Neil Stevenson was there.

There was also the director of <Squid Game>.

There were so many wonderful speakers from the point of view of someone watching in Kentucky, like me.

I don't know how many hours I've watched.

I even introduced it to my friends, saying, “Take a look at this, it’s so cool.”

Real-time 3D motion capture technology is expected to be used more and more.

This is the first case where SBS courageously tried such a technology, but I think that such real-time cases will increase.

Recently, examples of using these avatars are increasingly appearing in the US media as well.

Currently, a contest for avatar singers is being held on US TV broadcasts.

<A contest program for avatar singers called Alter Ego on US FOX Broadcasting Company>

Last year, on Facebook and Facebook Watch, the hip-hop star Cardi B and his wife entered the game and became a game character. Besides, I worked with singer Twenty One Pilots.

The media is particularly interested in virtual performance in the music industry.

<Experience of famous American rapper Cardi B and his wife becoming a game character- Facebook Watch program production scene @Reallusion> From

I believe that after the host formally introduced the speaker, Neil Stevenson's avatar appeared in the real stage AR of SBS D-Forum, and in the transition from the virtual world to the real world, Neil Stevenson's avatar disappeared with a book on his chest. In fact, I thought that the part where Neil Stevenson appeared in San Francisco with a book on his chest and continued the story was well connected.

<The process of transitioning from Neil Stevenson avatar to real Neil Stevenson (via 'Snow Crash' book)>

Neil Stevenson's Avatar gave a lecture, and I was amazed at how people focused and immersed in the Avatar's words. It felt like a unique experience. What's the trick?

I think I have an answer to the question you just asked. In my opinion, the secret is 

. What the avatar says and does is what matters. No matter how many avatars there are around me in the virtual world, and no matter how realistic the avatars look, if they aren't doing something special, they won't make sense.

Whether the avatar next to you is a dragon, a character from a game, or someone you really know, if you just have to look at it passively, you'll quickly lose interest. 

. It should feel like we're doing something together. So, in the end, what kind of content is the key. Just like SBS D Forum did. What Neil Stevenson's Avatar had to say was interesting, and people wanted to hear him, so I was able to focus even though I knew it was an avatar. 

The part we were thinking about was that we wanted to reflect the current Neil Stevenson, but we also wanted to reflect the character of his book as a sign of respect. So, I made a figure in which the protagonist of <Snow Crash> is equipped with a sword and clothes that the protagonist would wear in the metaverse. Also, as speaker Neil Stevenson, I made a variety of things, from wearing a shirt and pants that would suit him, to a more sci-fi look.

If he says his beard is too long, that kind of feedback is acceptable. I'll get you a haircut. I am glad that the author liked it, and it was wonderful to see the character come to life in the audio and video he delivered.

I also want to make a character who is exactly what I am. Because it can be a challenge because it is much harder. However, the person I am now is what I already have in the world, so I want to be a different person in another world. It would be nice to be able to fly. I really like hummingbirds, and I think a hummingbird-like character would be good too. Or I think it would be great for a large use. Avatars are super fun and can change at any time. If I answered the question again, it seems to change every time. It may change on a daily basis, or it may change often. It will be different from time to time. Maybe in another world you have to wear a name tag. Because every time I meet, I become a different character, so people may not recognize me.

Yes, I may finally understand fully.

As AI and machine learning advance, 

[1] and 

[2] Concerns have already been raised about the same issues.

When we give our users tools, we actually give them the freedom to implement things.

Nevertheless, I think everyone should take responsibility for how to use the tool.

<Jodeun Phil movies美Obama image obtained by synthesizing the president of the state to the voice of Directors>

Currently, if you look at YouTube, you can easily see Tom Cruise deepfake videos or former US President Obama deepfake videos.

As far as we know that it is a deepfake, we think it's funny or funny, but I think there are definitely areas where we need to look at these issues with a more keen eye. 

                                <Collaboration with the movie 'Replica' starring Keanu Reeves in 'Reallusion'> @Reallusion

Reallusion is a world-class character development company that participated in James Gunn's film Suicide Squad, released earlier this year, and Replica, starring Keanu Reeves, released in 2018.

The reason that SBS D Forum was able to collaborate with Reallusion this year was purely thanks to the direct recommendation and connection of writer Neil Stevenson, who was the speaker, but Vice President John Martin paradoxically said that it was also an opportunity brought by the pandemic. 

As the old methods no longer work properly, shiny moments that we would not have thought of before began to appear.

More globally connected, more opportunities open, and more opportunities to work together than ever before.

He also said that the collaboration with SBS D Forum was such a moment.

The Neil Stevenson session's metaverse experience was possible thanks to collaboration with Epic Games in addition to Reallusion, which created the character.

An additional interview with 'Epic Games' will be delivered in January.

Please also pay a lot of attention to the 2nd Epic Games!