Filmmakers Discuss Bringing Virtual Worlds to Life: The Mandalorian Season Two

The Mandalorian filmmakers discuss ILM’s innovative StageCraft virtual production technology, which has been advanced even further for the second season of Lucasfilm’s Emmy Award-winning series. Leveraging Helios, ILM’s real-time cinema render engine, the series breaks new ground in high fidelity, real-time visual effects. The discussion includes commentary from filmmakers Kathleen Kennedy, Janet Lewin, Rob Bredow, Jon Favreau, Dave Filoni, Carrie Beck, Richard Bluff, Bryce Dallas Howard, Taika Waititi, Rick Famuyiwa, Doug Chiang, Greig Fraser, Robert Rodriguez, and Deborah Chow.
StageCraft is a purpose-built, production-hardened platform that allows filmmakers to explore new ideas, communicate concepts, and execute shots in a collaborative and flexible production environment.

[@AverytheCubanAmerican]

From working on Shanghai's Pirates of the Caribbean to Mandalorian, Industrial Light Magic truly is a talented group. The unsung heroes of the franchise

[@adambomber3384]

I hope they try out more animated films one of these days, on a tangentially related note. I love RANGO but have yet to watch Strange Magic to the end

[@ElMiguelFigueroa]

Ok, very convincing. I'll buy it.

[@BonoTungTube]

So do we 💯

[@bloun003]

This technology is absolutely amazing. I can't wait to see how The KENOBI Show will benefit further by using it. When you can bring Tatooine and other planets to you versus seeking them out at various locations around the world...that's gotta be a game changer.

[@arthurballs9632]

It's a poor script that's the major concern for Kenobi. We already know that Lucasfilm approved the first script for production but someone at the parent company overrode the decision.

[@spenzur]

@@mehface Well, you were wrong.

[@trashpandaqc]

yeah, so boring to see internet trolls try to push this mysogynistic narrative onto Star Wars. With all respect to the work of ILM and the crews involved - the problem, if anything, is that Disney is going after the cash with nostalgia, and you can't write an inspired story if your goal is to constantly remind audiences of a familiar one. "Diversity" is never the reason for a bad script, that makes no sense... and especially not in a story that was a critique of American imperialism during Vietnam and had strong female characters from the get-go.

[@Cyberskip]

I had the privilege of seeing this in person, it's really amazing. It's going to change how productions are shot.

[@cyserro14]

Lucasfilm, ILM, StageCraft, how I love all this innovations that Lucasfilm always brings since 77.

[@Drebe73]

If these incredible behind the scenes features were never shared, I would think that everything was shot in real locations. This makes me love the show even more. Please keep these coming and thank you for making my favorite Star Wars story.

[@tediousmaximus1067]

This LED technology reminds me of the front-projection screens used by Stanley Kubrick in 2001: A Space Odyssey. This is much more advanced technology but the result is the same, you can see the final result in the camera immediately on stage.

[@UE5n00b]

The idea isn't new but the technology is, not only because of the panels but the Unreal Engine 5 capabilities on bringing reallity to the human eye

[@randomthings7875]

Please make more of this kind of video. Make it an hour longer please. Thanks

[@hartdr8074]

Everyone talking about how ILM feels like this is their own technology but not even mentioning Unreal Engine who's actually doing the heavy lifting in this VFX.

[@moradtabbicca9204]

shh....

[@HangYuriYangFX]

Well, only mandalorian season 1… then it was their own software..

[@hartdr8074]

@@HangYuriYangFX right...

[@HangYuriYangFX]

@FOX CRAFT proprietary software I think

[@Dadziu27]

I am really happy for the producers and directors, that they can work in such media. Amazing! Pleasure for the crew and the viewer!

[@TheTuubster]

The comeback of the set extension as back projection (as it was common since the silent movie era until the early 1980s). The first indication were the dynamic cloud projections for the movie "Oblivion" to surround the sky house set and provide natural light (and reflections). It will be only a matter of time, until these screens are common in standing sets on sound stages to extend corridors, rooms or provide dynamic environments with natural light outside windows.

[@AngeloBarovierSD]

Indeed. OBLIVION certainly inspired many DPs to look into practical-virtual lighting options and backdrops, certainly contributing to the development of this LED-borne technology. I'm glad the film hasn't been forgotten for its technical achievements.

[@Daniel-Strain]

True. And they used a similar screen as Oblivion to have realistic views out the cockpit windows in Rogue One and Solo. But what makes the volume revolutionary and unique from all previous back projection systems, is that the cameras are tied into the perspective shifting of the imagery. So as they move them around, the perspective angles of the image changes instead of being foreshortened like a painting on a wall would be.

[@jurgendenederlandse2449]

The quality of the special FX in the S.W. modern tv series is absolutely parallel to the S.W. films AWESOME. STAR WARS VETERAN FAN SINCE 1981. AUTHOR ON 4 STAR WARS BOOKS IN EUROPE. S.W. ACTION FIGURES COLLECTOR.

[@HOSKIN5]

You don't realise how much work went into this show till you see this video .. I'm amazed by how they done this show with the effects that they have .. the mandalorian was amazing it's what brought starwars back after return of the jedi 😀 can't wait for season 3 and 4 and so on I hope they keep em coming and release on bluray.

[@0heck]

Can't help but smile through the whole video. This tech is really an amazing step forward.

[@tbip2001]

Would love to see someone make this 360 with floor and ceiling for a 'holodeck' experience. Add in some augmented reality glasses so the viewer can interact with characters and you're there!

[@NuclearFridge1]

It's going to get there, maybe even in my lifetime

[@Spidouz]

@@NuclearFridge1 maybe, or maybe by the time we get there, there won’t be any need for camera and actors… what I mean is that 3D virtual set and models improved a lot to the point we pretty much can’t dissociate them from reality. So we won’t need actors to act in front of a camera, you will have animator to move to create movement of a 3D models of a character in the virtual 3D set. Characters, props, etc… everything will be 3D generated, like for video games. You will have models to provide a look, voice actors and animators to provide the expressions and voices that will be part of the overall movie/TV Shows editing. If you want to change a line, adjust vocal for another language, etc… the 3D models will follow the local face animator for lip syncing, etc…. And at some point, with improve of technology, you might even be able to create your own virtual you to upload to the movie platform to have the hero of the movie being played with your own face… 20 or 30 years from now, there won’t be any “actor” anymore, mainly when you can get choice of any model, modify them as you want, keep them young for decades, etc… just like Bart Simpsons is still 10 years old today. Long story short, I don’t think we will have another step with floor and ceiling because the technology is moving very fast and I bet that in less than 10 years we will see the first movies entirely done virtually.

[@lothean2099]

This technology is only in its infancy. I see movie theaters, home theaters. One day, can you imagine playing a video game with a moving floor. Oh to live long enough to see this dream become a reality.

[@SoBoss300]

Never get tired of hearing about the volume. I hope more productions start utilizing it

[@billkendrick1]

Jon Favreau calling them "sprites" confirms he played on Atari and/or Commodore computers back in the 80s. 👍

[@corvetteguy2223]

No, it's more common terminology than you think.

[@ScottBaietti]

I'm shocked to learn the building behind the Asoka fight was digital on the volume screen. Before seeing this I would have sworn they built that building and the volume was presenting the sky and some of the the other scenery behind that set piece.

[@FelipeRRM]

When Obi-Wan appeared: "Hello there!"

[@SpencerRayFitch]

ILM is always pushing the envelope. That's why they are the best!

[@chaosfire321]

So from my understanding, ILM used their experience in Volume to make it into a standardized film making setup called Stagecraft?

[@moritzstrohriegel8724]

fantastic work.

[@PanteraRossa]

Beautfiul. Cuaron and Chivo pushing the limits again before we even knew it! Miranda and Kosinski as well. Don't know how this "democratizes" special fx though, considering these are 8-9 figure productions and the stage rental alone is probably several hundred thousand a day, not to mention Disney owns ILM and could choose to lock any other studios out from using their ecosystem.

[@KillahMate]

ILM aren't the only ones doing this - this version is the most sophisticated implementation ever, but there's a lot of prior art so the tech can't really be patented. Unreal Engine now has built-in support for this type of rendering, and a lot of smaller companies are building out smaller-scale stages with this tech all over the world, so there's a spectrum of options from ILM blockbuster-size volumes to tiny setups just large enough to shoot some good looking driving sequences.

[@trashpandaqc]

yeah, I don't think this specific stage setup democratizes anything, but I think software like Blender and Unreal makes similar processes much easier on a DIY level. You can definitely at least motion track footage at home, or use a virtual camera via a phone and a laptop, to have some of the same flexibility.

[@georgelewis8831]

Revolutionary!!!

[@gfxfilms2012]

Incredible. Can't wait to be on set...

[@jessetaran7116]

Are you gonna be on set? If so I hope you have a great time!

[@dannymorales787]

What department? Extra?

[@egoalter1276]

Congratulations. You made a goddamn holodeck.

[@gridcoregilry666]

God I love ILM!!!

[@ThomasMusings]

Daaaang. That is awesome.

[@ekojar3047]

This is the best new film invention since face and body motion capture . This is going to make CGI work so much better with real objects and actors, the light from the cgi world being cast directly onto the set is a game changer. Most of the problems you have with cgi not looking good is it looking out of place because the lighting of the two things you're trying to combine does not match. And it usually ends up with you having to replace the whole set and actors with cgi, and hand painting lighting onto the actor that gives you those edges where you can tell they aren't really there.

[@devonmarr9872]

Dynamic real time high quality Matt paintings

[@-jaymes]

This would be an insanely awesome place to work. So creative and innovative!

[@Tymonello]

This is so incredible.

[@ibunkatraining]

Oh, it's an ad! Well done.

[@orthovasky]

It is amazing the power of real-time rendering, it is incredible!

[@auntvesuvi3872]

Thanks, Kathleen! Thanks, Janet! Thanks, Rob! Thanks, Jon! Thanks, Dave! Thanks, Carrie! Thanks, Richard! Thanks, Bryce! Thanks, Taika! Thanks, Rick! Thanks, Doug! Thanks, Greig! Thanks, Robert! Thanks, Deborah! Thanks to *everyone* at ILM! 🎬

[@WW_Studios]

So... when can I get one of these?

[@bode260]

I'm gonna need this.. They got em at Walmart?

[@gregkilby7043]

Somewhere, Georges Méliès is smiling :)

[@deep.space.12]

Why's the display area outside the camera's field of view different and not tracking? Say, 2:22 and 2:27, you clearly have the model outside the area. Perhaps to match perspectives without confusing the actors?

[@KillahMate]

From what I understand from various interviews, it's both because tracking the entire volume would disorient the actors, yes, but also because if the entire background image moved then the diffuse lighting from that image would shift slightly with every camera move instead of the light holding 'naturally' steady as the camera moves.

[@jessepitcherband]

Also it’s easier on the rendering hardware when you have less pixels to recalculate

[@duranbodasing]

JOHN FAVREAU IS IRONMAN - for realz dude

[@sschulak07]

fix it in post! no we fixin this right now hold my la criox

[@danielsails50]

Sold. I’ll take two please.

[@UnrealTutorialsPro]

Definitely unreal engine is the future of filmmaking

[@talaris-uk]

The rendering engine is no longer Unreal, they now use the in-house Helios engine.

[@yelistener]

Does every major VFX house now have this kind of virtual production facility? I mean, this clearly has been a game changer, and I imagine other big companies would definitely follow the steps.

[@dragonteracorps5059]

As of now, only ILM has this thing which is called 'The Volume'

[@yelistener]

@@dragonteracorps5059 I'm surprised other big VFX houses haven't created their own Volume yet. This came out 2 years ago and it doesn't look like a super complicated technology that other companies need years before finishing making their own. I guess the industry has not yet warmed up to this way of content making.

[@dragonteracorps5059]

yelistener there are plans if I'm not mistaken

[@dragonteracorps5059]

yelistener Disney is expanding its usage btw. Some marvel movies will be using the volume

[@kaijudiorama]

Yes. Star Trek is using it for their TV series.

[@Stigcrafter]

The first time I've seen the Aussie LED volume, that's so cool!!

[@leokimvideo]

It's game over for the old film dinosaurs who do it for real. It's scary how real it all looks.

[@deep.space.12]

3:01 I was wondering where's Jar Jar Binks.

[@OliverHollingdale]

Hello ILM! As an Indie Filmmaker about to embark on a Fantasy Project very minimal Budget! Do you do packages that can allow us to utilise this technology? Thank you

[@flaviopresutti]

Great advance for filmmaking!

[@Nrde]

No mention of Unreal engine?

[@AznMetalHead]

Oh snaps, that's such a quick tease of Obi-Wan

[@mgers75]

The prequels would have been 200% better if they had this technology in 1998

[@lightningbeast4556]

How far will virtual production will evole and advance and innovate

[@cruelus17500]

Beautiful and amazing. What a best name than Light and Magic?

[@StudioArtFX]

Same day release as the new Disney Galleries episode about The Mandalorian season 2? What else am I missing here?

[@Ollision]

Fun fact: Without George Lucas, this video wouldn't exist.

[@thedoctor4327]

Fun fact: Without George Walton Lucas Sr. and Dorothy Ellinore Lucas, George Lucas wouldn’t exist and this video wouldn’t exist 😉

[@deep.space.12]

Wouldn't the reflections in the water reveal the "joint" of the roof LED panel?

[@deep.space.12]

Around 4:17 you can see the roof.

[@adarnia]

would be erased in post

[@isamuominae]

How is possible some dumb dislike a video like this?

[@mattk.3645]

Disney does own a minority position in Epic Games, the maker of the Unreal Engine. So, it's all in the family.

[@JohnSmith-wg3pl]

StageCraft doesn't use Unreal Engine to render on LED anymore.

[@websnarf]

@@JohnSmith-wg3pl Then what do they use? For real-time graphics rendering, I was unaware that there was even laughable competition with Unreal Engine.

[@JohnSmith-wg3pl]

@@websnarf They have their own engine, Helios.

[@MartinFutter70]

Sublime stuff

[@Meta8_Eth]

The Holo-deck from Star Trek

[@vooyas.mp4]

KENOBIIIIIIIIIIIIIIIIIIII

[@NigelMoss]

Maul?? 🙂

[@ErelH]

Amazing.

[@mattk.3645]

This method has so many advantages over using green and blue screens.

[@bijaytamrakar9482]

woda wowow woda wowwow woda wow

[@jmalmsten]

I am curious. Stagecraft and the Volume stages has been around for a few years now. But I only see the show about the Half Man Half DeLorean mentioned as using it. I mean. Ok. ObiWan and a few other disney shows start to use it. But it's still kind of limited to Star Wars. If this is so democratizing... where are the Demos? Should we not be seeing tons of shows from all kinds of studios? I mean... there is a lot of potential here. I want this to be available to everyone as well. I just find it curious that it's only used on Star Wars shows.

[@hiimchrisj]

Yeah, only for around a few years now. In terms of television and film production, that's not a long time. Mind you it was essentially designed and tailored around The Mandalorian and by extension Disney productions. And even then the planning for these productions happens years ahead of time. Things aren't going to shift just because this stuff has become available, especially prior to it having been proven with The Mandalorian. If more movies and and shows start using this technology we're not going to be seeing the fruits of all that for at least a few years from now.

[@AngeloBarovierSD]

ILM's proprietary system isn't freely available but there are LED volume studios (powered by Unreal, etc.) which popping up all over. From massive studio space to boutique studios. The methodology is open to all who are able to assemble the components necessary.

[@jmalmsten]

@@AngeloBarovierSD Yeah, the tech itself is not that complex. It's essentially taking the idea of Douglas Trumbulls Magicam system where they had a motion control camera in a miniature set match, in real time, the movement of the camera recording the actors in front of blue-screens and combining both video feeds using a live color keyer. And combining that with really old school rear projection methodologies. The big difference being that they now can do the background image digitally with modern game engines and expand the screen so big that it not only envelopes the set but also can even provide live lighting on the real life subjects. In hindsight the development is inevitable. And it has been on the march for a long time. But still. As we are now seeing productions shot during the pandemic being released. And the first season of Mandalorian was released a year before the lockdown if I recall correctly. I am still a bit surprised that we haven't seen more shows using derivations of it even if we factor in a year long production cycle.

[@AngeloBarovierSD]

@@jmalmsten Okay, so I've dabbled in it recently and there are hurdles for people who want to dive in. Most of them are obvious and they're not technology-related. They're methodology related. In essence, as is mentioned in the video, it's a return of sorts to prep and not post. That's where many filmmakers of today will initially find difficulty because of the current workflow du jour. And, quite frankly, I'm all for it! In VFX-heavy environments, too many filmmakers took "We'll fix it in post" as an excuse to be lazy in prep. They figured a blue-/greenscreen meant flexibility and that, short of cursory storyboards, they'd "Figure it out on the day." This is how we end up with face-wraps on digital doubles looking like Gumby figures in a scene. The LED volume tech (or "virtual production") might look, on the surface, to be the same. However, it is actually best served by planning and coordination. Y'know, as it should be. This means everything from virtual production design, integration of practical and virtual set dec, pre-viz, blocking, framing, etc. Again, as it should be. Unfortunately, some filmmakers will struggle with this because, with the democratization of affordable gear, they learned to wing it and relied on post production VFX to fix any mistakes. And while they can still bumble their way through the LED stage, the gaps in their methodology will be exposed and highlighted in the virtual sandbox. Certain toggles won't get built in (ergo more time as the VAD adjusts), certain set dec won't get built (limiting framing), certain shots which existed only in their heads will become bones of contention, just all the little sloppy things will accrue lost minutes, then hours, then days. But the steady and professional filmmakers will quickly learn the limitations and benefits, they'll adapt, and I'm sure we'll start to see some great stuff.

[@tsukuneboy]

The Batman uses it

[@Upmacher]

I will still call it "the volume"

[@incrediblesarath]

Why 3dsmax?

[@dannymorales787]

Wondering how this tech will affect Animators/Riggers/Modelers/VFX team etc. with job opportunities, will it limit or expand jobs.

[@JasonCobble01]

For Animators/Riggers/Modelers it could increase work because this opens up opportunities for the show to include many models and characters. For compositors it could decrease work. If you have no need for green screen or compositing in characters, you'll need less artists to handle all that work.

[@dannymorales787]

@@JasonCobble01 Agree! Thanks for the response!

[@DavidConant]

I feel like we’ll still be needed, just in a different way. VFX is always about problem solving, it’s just the puzzles that are changing.

[@JasonCobble01]

@@DavidConant Totally agree. And many places cannot afford the LED screens. They are extremely expensive; for now. And even the Mandalorian had many fully CG shots that didn't use the screens. But the demand will certainly decrease for outsourcing initially. You might see less work going to India to do roto for example.

[@firatcicek9885]

@@JasonCobble01 Do you know Steve Wright? He is a master nuke compositor and he wrote an article about this.

[@chrissoba5104]

I didn't see MANDO's face here either.

[@Hhhh-o1p7l]

이건 ㄹㅇ 영화 혁명이다...

[@jackle7896]

I was just watching your last video from 4 months ago last night

[@erikals]

Buy 3 - Get 1 Free !

[@BigChiken44]

Yeah, those money from credit cards of parents of kids playing Forntine were spent good) Epic Games are Epic

[@VIPINFX]

ILM and Weta

[@lemondemerveilleuxdechrist6515]

Awesome

[@toqkaizogou1636]

0:58 Jesus christ... they basically invented the fucking holodeck here.

[@1ksweatyrikers455]

this tech is amazing, but it can't emulate single-point direct sunlight well

[@J-Gx]

That's why they can take out any led panel they want and put a hard source through it.

[@RobbGardner]

That's where the band-aid of high powered HDMI wattage lights come in as a helper.

[@J-Gx]

@@RobbGardner make that HMI and I'll agree. There's not a lot of light in HDMI cables.. :D

[@RobbGardner]

@@J-Gx Typo, meant HMI :)

[@arthurballs9632]

3:00 - Grief Karga meeting Boba Fett! That didn't make it into the show.

[@MrLeeziebee]

#firekathleenkennedy

[@blabla1177-r7u]

Mandalorian is for me the best show since Battlestar Galactica. So sad that sequel trilogy got such a mess of a script.

[@DaManDaMythDaLegnd]

this shit is insane.

[@DigitalContentNetwork]

YOUR4REAL

[@incogneato6395]

Unreal Engine is literally an unreal engine.

[@chasemorello60]

I hear you're really frugal

[@maxsaltin1001]

I don’t know why kathleen is here, with all due respect she should not be anywhere near Star Wars.

[@cineentreamigos7758]

Is the president of Lucasfilm and Executive producer of the Mandalorian. She has something to say.

[@juliocamacho8354]

Fire KK!

[@snakeeater6738]

Was having a good time until Kathleen started talking

[@SupremeLeaderKimJong-un]

Technology has come a long way thanks to critical thinking on my behalf

[@jackwilson1818]

FIRE Kathleen Kennedy (aka Darth Menopause).

[@shekiba.646]

Nice but Work too much.

[@tsarbomba1]

Shame you had to start the video out with the one woman who's almost destroyed the entire Star Wars universe... Kennedy.

[@MartinNewton1981]

Interesting video!... *BUT KK IS IN IT.* Just because of that: im going to *Dyslike* it. SHE NEEDS TO GO. "Without respect we reject".

[@EldiabloCompadre]

OMG is that? Zip...

[@peterhaller1217]

And who needs Darth Kennedy in this post? Kennedy's only contribution to the Mandalorian is to have Gina Carano fired.

[@abhijitleihaorambam3763]

First

[@ForceMaximus84]

Can we get Lucas’ VII-IX, please?

[@MCCreativeLegends]

Now fix the terrible writing..

[@adambomber3384]

they're not gonna see your wet blanket comment

[@furynotes]

I actually agree about the writing. If my dad lost interest. That's a problem. That was in season 1. My dad is in his 60s.

[@richardashton5015]

Kennedy needs to just leave.