Case study
Storytelling Fellowship
This was the short test I was able to do before I jointed the Fellowship
In August 2021 I was accepted into the Unreal Fellowship. I thought I’d do a little write up about it to document my experience and perceptions about it.
I don’t remember exactly when I first heard about the Fellowship. Maybe it was from this FX Guide article, or maybe through social media postings. I do know that as soon as I heard about it I applied. It sounded too good to be a real thing, you get paid to learn Unreal, something that I had wanted to do anyway. Learning Unreal had been on my ‘to-do’ list for years, but The Mandelorian was the breakthrough that made it more urgent. Anyone could see that this was going to be a major thing in VFX in the years to come.
What I didn’t know was that so many people would have the same idea as me. Even though you had to have years of experience, 6000 people applied to the first intake of only 100 spots. I would have to wait until I was lucky enough to get selected. In the meantime I started to try to learn it on my own. It was slow going, my schedule of teaching and working at the studio made it really hard to find the time. Still, July 2021 I had made this short piece.
Even though I had made this, I knew that it wasn’t right. My project file was a mess, I had done most of the setup in Houdini, and I really didn’t have a good understanding of Unreal or it’s strengths. Everything was such a mess with this project I knew that I was missing critical pieces of the puzzle.
Earlier in 2021, I had received an email from Epic that they were considering allowing educators to take the Fellowship. Since I had signed up for an online ‘educators day’ where they showed Unreal videos and talks directed to educators, I was on a mailing list and I was given the opportunity to apply for the Summer 2021 Fellowship.
I spent two weeks preparing my application, revamping my website, getting my wife to help me re-write my bio (she’s a much better writer than me) so that I would stand a chance of getting accepted. It would mean a lot to me to get accepted.
The August timeline was the perfect time of the year when I would have a gap between semesters. During other times of the year it would be impossible to take the time off from the school. Even during this window I would have to move my classes to Sundays for two weeks since we were finishing up the semester when the Fellowship would be starting, but luckily my students didn’t mind.
I received my acceptance letter on July 13th. I remember when I got it, I was so happy and excited. It had been a year of waiting but I was finally going to get my shot. Five weeks to absorb as much as I can. I responded with my acceptance right away. But then days and weeks went by and I hadn’t heard anything back from Epic. Did they receive my acceptance? Was everything OK?
As the days got closer to the start of the Fellowship I was worried that my acceptance had been lost in email transit or something. I reached out to a few people I knew where associated with the Fellowship on Discord and they got me in contact with the admin and everything was fine. But one of the people I contacted was a TA in the Fellowship, and he gave me some advice:
“I would say this: Make sure to get your storyboards and animatics done and nailed down before hand. They are pretty crucial”
— Jet Olano
I knew that as part of the Fellowship I would get to make a short film. I had listened to Brian Pohl’s interview on the Allen McKay podcast, and I had watched his talk on the Unreal YouTube channel, so I had an idea of what I was getting into. I started to roll around some ideas in my mind and I started watching videos from past Fellowships to see what previous classes would produce.
I could see that a lot of previous classes used the Echo character from the Unreal Engine 5 demo and Metahumans. The Metahumans faces look amazing. If you light them right, you can start to get close to photoreal expressions. The downside to them is that they don’t have a wide selection of clothing. But as I was going through the Fellowship videos, I noticed that there were a couple that had Metahuman heads that had different clothes. There was one in particular that stood out, Curio by David Williams. When I saw that one, I realized that it would be possible to mix and match Metahuman heads with other bodies, which would open up a lot of storytelling possibilities. In fact, David made a tutorial explaining exactly how to do what I had in mind. I also knew that there was going to probably be some dialogue in my film, so I started to look at how to import facial capture from my phone into characters in Unreal. I saw this tutorial on YouTube that explained how to do this using Faceware. Faceware is a facial capture software that is free for Unreal users until the end of the year. After watching those two tutorials, I tried to do a test before the Fellowship started.
An early test of me using Faceware and applying it to a Metahuman.
Tuesday July 27th was our orientation day. I could tell right away that this was well organized. Brian Pohl was the head lecturer, and welcomed us to our first Zoom session. He explained that the Fellowship was made up of several different components:
-Formal classes taught by an instructor. The classes would start with basic concepts like Static Mesh Ingestion, Introduction to Materials, Introduction to Sequencer and eventually move to Animation Blueprints and Take Recorder, Sequencer: Performance Capture with Live Link Face and Intro to Optimization. There were project files that were distributed to all of us well before each class that we could work through as the class was going on.
One part of the classes that I was particularly impressed with was how they dealt with questions. Instead of stopping the class for one person to ask a question, we were encouraged to write our questions into a dedicated Slack channel where an army of TAs would be takes screen shots, recording short videos and answering questions. This way the instructor could go through all the content while not getting bogged down by one person’s question. It worked really well. After the class there was always an hour scheduled where we could ask the instructor questions, get them to demonstrate different techniques again, etc… The organization and execution was very impressive.
-Live Labs were run by the TAs. It was a large block of time where you could ask questions about anything. They had breakout rooms on Zoom where they had specific topics and TAs that specialized in specific areas of Unreal.
-We were broken up into 4 different teams. During Scrums is where our team would get together to share our progress on the short film we would have to make during the Fellowship. To complete the Fellowship you had to have 45 seconds of animation produced in Unreal. Scrums were a great time to get to know people in a smaller setting, see what other people were working on, get critiqued on your story and to report on your progress. Each team was headed up by a Mentor. As luck would have it, I was on Team Williams. My Mentor was the same guy who made the tutorial about putting a Metahuman head on a Paragon body that I had watched on YouTube a few weeks before.
-We had several guest speakers come in for talks, including Dr. Mike Seymour talking about realtime digital humans, several former Fellows talking about the process they went through to make their films and other industry experts like Mahmoud Rhanama from Pixomondo and Sam Nicholson from Stargate Studios talking about Virtual Production.
-Something else that really helped us get to know one another was Coffee Hour and Happy Hour that often bookended the day. They were both informal Zoom sessions where we could just hang out and talk.
-Ending each week was Weeklies, which were similar to Scrums except with the entire Fellowship. A few people from each team would present their work to everyone so we could see their progress. It was always exciting to see what other Fellows on the other teams were doing, since we weren’t seeing their work all the time in Scrums.
All of this was to work towards our final project, a short film. Every Fellowship they have a theme that they use as a guide to help structure our films. The theme for ours was ‘The Butterfly Effect’, where a small action can cascade into having larger effects. Some people followed this very literally while others disregarded it entirely.
Something that was really cool about the Fellowship was the very diverse group of people that were in it. Some people were long time CG industry veterans while some people had zero CG experience at all. There were Concept Artists, CG Supervisors, Cinematographers, Producers, Motion Graphics Artists, Matte Painters, Generalists and for this Fellowship they invited several instructors. I’m pretty sure that’s how I made it in.
I really liked the way that they broke us out into separate smaller teams. It gave you a chance to get to know people better. We were all going through this intense experience together and I learnt a lot from our Scrums when I could see all the work everyone was doing. It was really inspirational to see everyone’s project come together as the weeks went on.
It’s important to point out that the team of TAs, Mentors and Instructors that were assembled for this was really incredible. They had a page on the Online Learning Portal that had everyone’s bio on it and I couldn’t believe how impressive they all were. It seemed like everyone had a ton of experience and their resumes were really impressive. It made you feel like you were in good hands.
So I began the process of making my film. With such a short timeframe we were encouraged to look through the Unreal Marketplace and and leverage existing assets as much as possible. I started looking through the Marketplace, knowing that the Paragon assets were probably some of the higher quality ones that were free. I liked the look of the character Twinblast, he had metal arms that reminded me of the Winter Soldier from the Marvel films and I thought he’d be a good Super-Hero type.
I was looking for other characters to use and I saw Boris the Bear from Paragon. A thought flashed in my mind, ‘imagine that was a real bear, it’s just maim the other heroes like in the Revenant. That was the spark for my idea, if the Bear was so violent that it shocked the other hero. I started to rough out my script and flesh out the story.
This was my test to see if I could make the Blendshapes for TwinBlast and use my iPhone to capture my expressions. I thought it was ok, but not great. What helped me immensely was a Blender plugin called FaceIt. It shortened the process of making the blendshapes from days or weeks to hours. Keep in mind that I know NOTHING about Blendshapes, rigging or Blender, so this was a bit of an adventure. It says a lot for that plugin that I was able to get results in a day or two.
With that looking ok, I started testing out the Blendshapes for Bear.
In this test I was able to get his face moving and changing expression with the same plugin. The next thing was getting his groom working. I imported Bear’s head geo into Houdini and used it’s hair tools to set the groom up and rendered out a test in Unreal.
The last member of my cast would be a bad guy that would feel the wrath of Bear. Steel looked pretty bad ass, so I picked him. Once I had my characters picked out, I got to work on figuring out how to make them talk. I wanted to use a Metahuman head for TwinBlast and Steel while keeping the Paragon body, but Bear I would have to make the blendshapes for. I also wanted to see if I could make a groom for him since in Unreal 4.26 they added a bunch of new hair and fur tools. The issue I ran into was that for the outfit I wanted TwinBlast to have, he’s wearing a T-Shirt with a deep collar. The metahuman head and neck extends down to the torso, but not that far. So I thought I’d make the blendshapes for TwinBlast as well and skip the Metahuman.
By August 4th I was able to produce this test. My three main characters with some facial animation data on them. Now I had to get going on the story and get that in better shape.
Things started to fall into place. I started to work on the ‘head rip’ effect. I wanted it to be over the top, ridiculously gory and bloody. Hopefully so crazy that it’d be funny. I started working on a test for that. I was able to get it close, but eventually I emailed a former student of mine for some help. Jeffy Matthew Phillip worked at a few VFX studios before landing a job at SideFX, so I emailed him and he gave me some really useful ideas for the setup. I think this was the version I sent him asking for help, I liked the stretchy effect I was getting on the left, but I was having issues controlling the effect. He totally came through with a nice setup.
While all these tests were going on, I was starting to work on the animatic. I remembered Jet’s advice from a few weeks ago, to try to get the animatic and script nailed down early. This was the animatic I showed at our second Scrum. No animation, no voices or sound effects, but trying to get a feel for the pacing.
I presented it at the Scrum to mostly positive feedback, but it was clear that a lot of work needed to be done to tighten up the story. While I was working on it I showed the script to a co-worker of mine at the school, Jim Zub. Jim works in the animation department of the school, but he’s also a very successful writer at Marvel Comics. He offered some suggestions, but mostly I wanted to make sure that the story wasn’t crazy.
When I wasn’t working on the script/animatic, I kept moving forward with my tests. I did another test of TwinBlast and Bear.
This test revealed some problems with TwinBlast's mouth, I'd need to go back and fix up his Blendshapes.
But it was time to move forward with Steel. His skin is blue so I tried to export the texture out of Unreal. For some reason you can't export Paragon textures (under Asset Actions the only option is to Migrate to another project) so I just took a screen cap of the texture and warped it to fit the Metahuman geometry. I was able to produce this test:
This test was a revelation in just how good the Metahuman faces were. A million times more expressive and realistic then what I was making in Blender.
As soon as I saw the results of my Steel test I knew that I'd have to use Metahumans for TwinBlast. It was a million times better than my earlier blend shape tests. I started looking through his different skins to see if there was a version of his costume that could hide his neck. In a day I had a new version of my hero.
While I was testing my characters to see if I could get the Metahuman/Paragon combination to work, I was trying to cast some actors for the voices. Luckily at the college I teach at there's an Acting for Camera and Voice program. I reached out to the coordinator and I started to get emails of voice reels trickling in over a few weeks.
I was working on the next version of my Animatic. I tried to put my characters into the environment with the voice actors. It was at this stage that I realized the environment I picked was a bit heavy, so just moving my characters around got slow and laggy. I submitted this Animatic on August 8th, about a week into the Fellowship.
This was when I started to panic a little. I could tell that this wasn't really that funny. Very luckily for me, my Mentor David Williams really zeroed in on some of the problems and gave me some very specific and helpful feedback. I’m printing it all here since it really was the turning point for my project.
Hey Joe, looking good. Seeing it like this really helps identify what is working and what's not. I think it could be beneficial to think about the short in 3 parts, the leadup where we establish the team/stakes, the confrontation/fight and then the conclusion. To me the concept of the short that's funny is how it plays on the genre tropes and the audiences expectations. We don't expect the bad guy to suddenly get ripped apart or for the mechanical bear to comment on his own ridiculous character. The Twinblast character also clearly expects things to play out more according to those super hero tropes which becomes its own source of comedy. For these jokes to land then it's important to play along and reference these genre tropes.
In my mind there's 2 things playing against it in the first scene. The first is bear using the "i'm a bear" punchline too early. And the 2nd is becoming too meta so quickly that it actually undermines some of the other jokes like the head getting ripped off. You want to almost trick the audience into thinking this is a straight-laced superhero show and then flip it on them later with bears commentary in a "does this really make sense to you?" kind of way. If you can catch the audience off guard with this it hits much harder. "One Punch Man" does a fantastic job of this as reference, where even though you know the premise of the show (a dude beating everyone in one punch) you get sucked in over and over again.
The second scene is where your main story beat happens but it's all over a bit too quickly. I think you want a bit more back and forth, maybe make it all from Twinblast’s perspective since he is the one we empathize with. If you have him pinned down, playing up the action in a dramatic way and clearly getting super into it then it makes it all the more funny when Bear casually ends things. Also don't be afraid to have a beat or two for us and the characters to absorb things. The argument scene at the end is a great time to get some funny jokes in but the key, I feel, is to be short and concise. At the moment the dialogue is overbearing and repeating things we already know and that the characters already know. The emotion could be conveyed in a more visual way or with events that complicate things more for the character. Less commentary on past actions and more raising of the stakes could make it more interesting and funny.
A quick example could be having bears response simply be a (yeah bro) shrug and "I'm a bear" while HQ/police radio in to take the bad guy into custody and Twinblast is freaking out; trying to put this guys head back together, flashing lights and sirens getting closer and bear just standing around watching. Even the same original dialogue while this is happening would be great. Having Twinblast come to the realization of oh yeah, this guy is a bear what was I thinking while paused scooping together bits of head would add a lot.
Overall I'd say check out references for parody but also for straight laced action and see how much you can squeeze into your film, including camera angles.
"One Punch Man" is great for superhero scenes that play with genre and are over in funny anti-climaxes.
"The Boys" similarly has some moments that play on expectations and end scenes in horrifying ways
"Scott Pilgrim vs the World" has some great cheesy hero/villain scenes and monologues + lots of visual camera-based comedy
Most of the older DC and Marvel stuff has unironic serious villain talking while a hero tries to overcome them scenes, I think there's even some warehouse ones in one of the Captain America movies. This is all optional advice so feel free to ignore anything that doesn't support the direction you want to take. There's also the opportunity for feedback from others during our Scrum if nothing here clicks. You're in a great place ahead of where you need to be so I recommend spending as much time as possible getting the key moments right before moving on.
Honestly, re-reading this critique, this was the major turning point of my project. While I had been concentrating a lot on the look of the characters, I wasn’t focusing as much on the story and especially the pacing. This feedback was so helpful and vital in making me reassess things. So I took a step back and decided I was focusing too much on things that didn’t really matter right now. Why was I trying to nail the lighting when the story wasn’t working? I went back to the drawing board and thought “I need to work light and fast”. I put aside the nice renders and just worked with the UE Mannequin with no environment. I also recast my lead actor, 4 days later I had this version.
A few more revisions later and most of the story beats are in there. By version 6 I had them in the environment, added more action in the beginning.
After this it was a lot of massaging camera angles and trying to tighten up the edit.
I showed this version at our Weeklies where we would watch the work in progress from about 4 or 5 artists from the different teams. This was shown on August 16th, about 50% of the way through the Fellowship.
Later that week I got the retargeting process down, taking the animation I had on the Mannequin and could start to transfer it over to my characters.
One of the cooler days of the Fellowship was when we could direct a motion capture session with the MoCap team from Epic. They guys there did a great job, and later in the week I was able to see how it looked being applied to Bear. I asked if they could try to do the 'facerip' action and it turned out really well. The main issue was that Bear's proportions are so long that his arms were down to the floor. I modified the animation so that his lower half was still going to jump on his back, but I used the upper animation from the MoCap session, along with some Control Rig modifications.
For the beheading, I worked on a FLIP simulation in Houdini and brought it in as an Alembic cache into Unreal.
At this point I was just tweaking things until I ran out of time. I still had to capture my facial performace saying the actor’s lines into an iPhone and try to get every camera cut just right.
I was able to render my project with Raytracing, but looking back I wish I had spent more time on lighting it. I think lighting and camera blocking are things I don’t have a lot of practice with and it showed. Ultimately, version 20 was my final one and the one that’s at the top of this page.
By the time the Fellowship ended on September 3rd, I think everyone was really sad that it was over. I know I was. It’s a lot like how I imagine summer camp would be, where you’re smashed together with a group of strangers for a month, you spend a lot of time with one another, and then just when you start getting comfortable with everyone, it’s over. It’s an amazing program, I’m so happy to have been able to take part in it.
There are a lot of people to thank. There are executives at Epic that I’ll never meet, but they funded the program and see the value in it. Brian Pohl oversees it but so many people contribute. Rebecca with tons of stuff behind the scenes, Eddie who kept the online learning portal running and updated, the team that kept the Virtual Machines up and running, so many people! That’s not counting the truly incredible instructors and TAs! Not only were they subject experts but they were so excited to teach us. You could see the joy they had working in Unreal and it was like they could barely wait to show us the next new thing to try. As someone who’s been teaching for 20 years, you can tell when people are going through the motions, and they were the opposite of that. So thank you, everyone, I’ll never forget it!