Military Insider's Guide to Extended Reality (VR/AR/MR)

Episode Introduction

Tom Constable: Hello and welcome to episode six of the Warfighter podcast. I'm Tom Constable, and this is Colin Hillier.

Colin Hillier: Hello. How are you doing?

Tom Constable: I'm very well, this is our second instalment of our Education series and it's gonna be another corker. 

Colin Hillier: Well, I hope so. 

Tom Constable: Well, you found the guests and I thoroughly enjoyed the chat as it was talking about all things extended reality its unsurprising to our regular listeners , but before we go into that, people who have been following our education series will know that today is the day that I set down the gauntlet.  These education series episodes have been sponsored by Conductrr, which is a crisis management software provider and training provider and I have been working with them over the last month or so to create a scenario and a challenge for you, Colin.

Are you ready to hear what I have in store for you?

Colin Hillier: Yes, bring it on.

Tom Constable: So over the next month I would like you to deal with a crisis, the crisis that I'm gonna or have set for you is all about being hacked, dealing with ransomware with an organisation you are trying to send relief , but you are at C level within a large Internet of things organisation. And you are responsible , your organisation has been doing some perceptively, maybe unethical dealings with another company, and there's a hacking organisation now targeting you guys and your organisation in order to just disrupt what you do and shine a light on these potentially unethical dealings.So you now have to deal with the fallout of that, manage your company's reputation while trying to make the right choices to actually deal with the attack itself. You have to also go through your pre designated crisis checklist and ensure that every step of the way you're minimising the cost to the organisation, the disruption, media coverage and damage to your brand and corporate image.

Do you think that you are prepared and willing to deal with this crisis, Colin? 

Colin Hillier: You know, I realise. No one's really prepared for a crisis that's the definition of a crisis

Tom Constable: All right, smart arse.

Colin Hillier: It wouldn't be a crisis if I were ready for it 

Tom Constable: What I'd like to do and I'm not promising this to the listeners, but what I'd like to do with this is screen capture when you go through the process, I'll be bird in the sky watching you go through the crisis giving you some helpful support as you can imagine, and potentially we'll put it up on to the YouTube channel so that people can see how you deal and performing a crisis. What do you think? 

Colin Hillier: Yeah, sounds fun, can't wait. 

Tom Constable: OK, so without further ado let's move on to the education section and Colin, could you please introduce the guests and the topic for today? 

Colin Hillier: Yes, well we're very lucky to be joined by Ian Ferguson, who's by day a Warrant Officer in the Rural School of Artillery working in the SIM cell. He's been dealing with simulation, training matters and a lot of the education that goes on there education across the army in all things training simulation, we've asked him to come on because he's got really good background in XR, mixed reality, augmented reality, virtual reality. 

These are words that are sort of bandied about a lot these days, and probably I'd admit they're confusing even to me, which isn't hard, we've asked him to sort of give us a bit more clarity on the technologies that are out today. He's tried a lot of them. That's why I think it's useful. He doesn't particularly back any horse, you know. I know there's people out there that have their favourites, so I think it'll give us a good balanced view of where the technologies are and how best to employ them. 

Tom Constable: Yeah, and what I particularly liked about the chat with Ian was that his perspective, being kind of from an inside out, You know from an inside organisation looking out to industry was very very similar to industry but just nuance towards his perspective, which is a privilege to be able to hear to be honest and to allow us to educate ourselves. Not being on the inside. So let's get into the interview shall we? 

Start of Interview with Ian Ferguson of the Royal School of Artillery - 00:03:42 

Colin Hillier: Ian, How are you? 

Ian Ferguson: Not too bad Colin. Nice sort of Christmas break at the moment. 

Tom Constable: I love this Ian, so you've taken time This is your Christmas break and you know military personnel as we know, as we all know, as used to be is, we'd like to enjoy having our downtime over Christmas, but you've volunteered your time to come on as a guest to educate the listeners. I think that is utterly commendable. 

Ian Ferguson: Well, It's a good subject to be honest, and it's not boring and quite engaging, so it's something that I like teaching or delivering presentations on so hopefully your listener base will enjoy it. 

Colin Hillier: For those who have not met you or don't know anything about you, Ian, just give us a bit about your background, where you come from and what you do. 

Ian Ferguson: So I work at the Royal School of Artillery in Larkhill. I’m a qualified gunnery instructor, a QGI. What I normally deal with in my day-to-day business is simulation. I moved into that role from a unmanned air systems background, more commonly called drones. Some of the sidelines that we deal with in my office is implementation of new or emerging technologies and the impact that has within the training environment. XR comes firmly underneath that. Because it is such a niche area in military terms, not in civilian terms it's quite well known,  it's a part of my role to educate the chain of command and other soldiers on the impact of XR within the learning environment and how it could be implemented. We have done a lot of experimentation mainly with VR and MR, which we’ll go into the terminology in a moment, but I haven't really touched on augmented reality, AR, at all. But I can see some uses for that. It's just not been relevant for the training space that I'm currently in. 

When we move forward with the podcast, I'll describe some of the headsets that I've used and that's really the depth of utility and exploration that I've done within this sort of technological environment. 

Tom Constable : Great, well thank you so much for giving the time and it's nice to get in these education pieces someone from the military to be able to be that voice of experience so thank you. 

So let's move on to the first section. I think it'll be remiss not to do a kind of introduction to XR. Explain what XR is ,so extended reality, and what that term is a catch all for and how it's used so over to you. 

Ian Ferguson: So I think the first thing I'll state ,before we begin, is that any thoughts or opinions that I express on this podcast are purely my own. They're not that of my employer, but what is XR? As you've already heard. XR is a phrase that's getting thrown around a lot, and we'll come to that at the end. But first of all, we'll discuss the three different realities, and there are only three. There's AR augmented reality, MR mixed reality and VR virtual reality and we'll go through these in a more logical order. So if you look at your hands and your left hand is the real world and your right hand is a completely virtualized world. In your left hand will sit augmented reality, so that's as close to the real world as you can get, and it's effectively overlaying digital information over the real world. So in a smaller pair of Google glasses, you can display information directly to the user, but it's not interactive, so that user cannot interact with anything that they see through those glasses, and any interaction that appears is completely false. 

So a good example of this was the Pokémon Go up for mobile phones. It was an augmented reality game and it turned lots of people into zombies as they were running around on their phones trying to find the little dinosaurs and capture them. But when they went through that whole capture process, that was completely fake and done off of GPS. So there was no real sort of interaction there. We flip all the way over to your right hand. This is now a completely virtualized world as far away from the real world as we can get, and that's where VR sits. So you can make your user appear as anything, be anywhere and do anything, so there's no real limits on that. And then that leaves us with mixed reality. What was found is a bridge needed to be made between augmented reality and virtual reality and that's where mixed reality comes in. So it takes elements of augmented and virtual reality and blends them together so it's a more interactive experience for the user and you can get the more complex data down than augmented reality can offer. 

That leads us to the 4th acronym, XR, and all XR is is extended reality as an umbrella term that covers AR MR and VR. And it's what I call a danger term. If anyone says to you, I'm gonna sell you an XR headset. You have to ask what are you gonna get? Because that could be AR, MR or VR you could be expecting a top of the range VR headset and end up with a pair of Google Glass glasses. They haven't done anything wrong and they've sold you a XR headset, but it's really important that I think we start to understand this terminology. As we go through the podcast there's probably more terms that are gonna come out like HMD, which I've recognised I've used a couple of times. 

So what does that stand for? HMD is the head mounted display and that's literally all it is. It's whatever device you're using that's mounted on your head to see any of the realities, and that's how you're gonna sort of interact with that. There are certain areas around that we need to be aware of. The quality should reflect the requirement and we'll come back to requirements and training objectives as we move forward. The price will definitely reflect the quality of whatever you get. We can downscale HMD's, but we can't upscale and what I mean by that is an augmented reality headset can only ever be an augmented reality headset. It can't be anything else. A mixed reality headset can be a mixed reality headset and by definition is also an augmented reality headset, but it can never be a virtual reality headset. A virtual reality headset could be mixed reality or augmented reality, so this is the way that I start to look at these different HMD's and we can then try and work out which one we need for whatever training task we're doing, which brings us neatly into a why. Why do we need to know this stuff being green people in the military? Well we need to understand this on how it's gonna impact our lessons and what impact on lesson delivery is gonna have. 

We need to understand the terminology and the technology to make informed decisions, because how else are we gonna make an informed decision if we don't understand what we're talking about and that's the same as when we start to define our requirements? If they're going to be fit for purpose, we have to understand the technologies we're speaking about, and that really covers the amount of terminology we'll discuss during the podcast. Is there any points on that you would like me to clarify? 

Tom Constable: Just a quick one. Now, you mentioned that a specifically designed mixed reality headset couldn't be a virtual reality headset. Now, I've used the Varjo headsets. They are mixed reality headsets. One of the one of the best in the industry at this moment in time. But that also can do virtual reality. So where does that sit in your continuum? 

 Ian Ferguson: So the Varjo XR3 is actually based on the Varjo VR3, so it's a virtual reality headset that has the pass through cameras and light our sensors and all of the rest of it on that enables that headset to move or to accomplish more things than just virtual reality. So that's how it then achieves mixed reality or augmented reality. So it's based off of that virtual reality headset, not a mixed reality headset. 

Colin Hillier: Ian, we're discussing this before because i kind of like history. But before we go into sort of the individual areas, a lot of these concepts and techniques aren't new, are they? And how long has VR for example, been around? 

Ian Ferguson: Yeah, we can look at a bit of history, so I’ve sort of done a bit of digging around to find when the first instance of VR, sort of, came around and I managed to date back to 1935. A short story called Pig Mellion spectacles written by a sci-fi author, Stanley Weinbaum, actually first described VR. So you can start seeing that people are starting to become interested in this sort of technology. You know,I mean obviously back then it was looking forward into the future, but it was only 22 years after that, In 1957, Morton Heilig invented something called the sensor armour, which was a cabinet that stimulated the user with the viewing screen for sight,  oscillating fans for touch, devices that emitted smells and audio speakers for sound the interesting point about this is that the new VR experience that are starting to be implemented today include fans and motion platforms to further immerse the user in the virtual environment. So something or technologies that were implemented in 1957 are now enhancing modern day VR systems  in the gaming environment. Which I found quite interesting. 

Colin Hillier: I think it is interesting because a lot of the technologies that we see used ,for example you mentioned, I think you mentioned green screen and or Chroma Key, has been used for years in the TV industry. You know the weather map is a classic example where that's used. What's interesting is it's drawing a lot of sort of established technologies and pulling them all together.

Ian Ferguson: Yeah, once we move beyond 1957, in 1968, Ivan Southfield invented the 1st true VR/AR head mounted display called the Sword of Damocles. It was a crazy large device that had to be suspended from the ceiling and placed over people's heads. It is huge. It's quite a funny thing to go and Google and have a look at. 

Tom Constable: We'll definitely put a link to that in the show notes. 

Ian Ferguson: Yeah, and then after that we moved into the 90s and the 2000s and it was a bit up and down for VR through those periods, a bit hit and miss, however, I think the driving factor there was that it moved into the gaming space. And I think gaming is what's really pushed VR over the last 10-15 years. Moving into the current day and we have headsets where I think we're getting to the point that it's going to be very difficult for the everyday user or a normal user to start to distinguish between what is real and what is virtual. So I think it's come a hell of a long way since the sword of Damocles was invented, which is quite good. 

But that's really focusing on the VR  side of life, because that's been more prevalent when we look at the history of augmented reality. Even though that could be linked all the way back to the sensor armour or the sword of Damocles because these could display digital information over the real world, the term AR was actually coined by Thomas Cordell in 1992 during a boeing venture to teach its manufacturing staff in a virtualized way. So it's a lot newer in comparison and then mixed reality as well came along even later, and that's came up in some research by Paul Mingram and Fumio Kishino in 1994, their paper, a taxonomy of mixed reality visual displays and that starts to foresee the need for a middle ground reality that bridges the gap between AR and VR, and it's effectively what's come around today. There's a lot less development and history with AR and MR compared to VR, I think the whole VR piece took precedence. 

Tom Constable: And you can definitely see that in the maturity of the products related to each extended reality as well. 

Ian Ferguson: Yeah, yeah I absolutely agree and I think the mixed reality piece, once that gets implemented more in a gaming environment, I think will be very interesting to see how that goes. 

Tom Constable: Thanks for that potted history of extended reality.  And now I think it's a good opportunity to go into each reality  in a bit more detail and talk through from your from perspective, the considerations for each one, pros and cons,  in use cases may be just again, it doesn't have to be in massive detail, but you know you've been seeing it implemented from a defence perspective, so I think I'd be great to hear your perspective. 

Ian Ferguson: Yeah, absolutely. If we kick off with virtual reality, some of the head mounted displays I've used are the HTC Vive Pro, HTC Index PICO, Oculus Quest headsets and the Varjo VR3. I've also used the Varjo XR 3, some Sony ones, but only when I've been out at demos and shows, but these are the ones that we actually have in my office that I use quite regularly. We look at primarily trying to integrate that with defence virtual simulation, and our latest little bit of development is going to be with a motion platform and to look at how we can create driving simulators with full motion inside of a VR space environment. However, I think the most important part that you need to discuss implementation of any of these technologies are the considerations that we must put in place or at least think about before any form of implementation. 

I have a few that are probably the top of my hit list for a VR implemented solution and the first one is quite different. It's simulation versus gamification. We automatically focus on simulation all of the time, but is it the accuracy of that simulation that is needed? For whatever you're trying to teach, because if it's not, can we not just use a game that's already been created that's perfect for a virtualized environment, to achieve what our training goal is. So the first thing I always look at is there something already there that will meet the training objectives? Does it have to be simulated or can it be more gamified? So I think that's a good step to take at the start, and then you're going to have to look at are you going to use an untethered or a tethered option for your headset? Now, most people automatically go ‘I want it to be untethered so I can go walking around’ not understanding the massive reduction in processing power that brings. With an untethered headset you're not gonna get crystal clear graphics. You're not gonna get very small text that you can sort of read, in the same way that you can off of a tethered headset like the Varjo VR3. 

Tom Constable: So tethered meaning, for those that are super new to XR, is be just cable connecting the headset to a PC or a laptop, which then gives the headset the graphical and computing power of that larger machine, which therefore allows the headset to have better resolution, frame rate, et cetera. 

Ian Ferguson: Absolutely, for the next point that we move on to, it's a really important part of the technology to understand is how frame rates are moving away from the use of resolution inside VR headsets because I think it's pretty much proven now that it's not the way we need to go with them. When you look at all of these 8K headsets they're not delivering as clear a user experience as some of the lower resolution headsets. What we need to leverage is that computing power from a high end, potentially gaming PC to run these devices. You know until we minimise all of the computing technology that can fit into a headset. That's the way it's always going to be, unfortunately, because when you look at an untethered headset like an Oculus or a Pico headset. All of the processing power, the battery and everything's contained within the headset itself, so it's quite restrictive. 

Tom Constable: Technology is definitely moving towards our untethered, but being able to use the graphical computing power of a PC and know that the Quest 2 has the ability to project over Wi-Fi, I think it's in beta stage now so is in development but I have used it and it's pretty scary to be using a untethered headset, but with the computing grunt power of a laptop it is quite nice and that seems to be way industry is moving. 

Ian Ferguson: Well, that's interesting because the only time I've seen that was with HTC's addition that you could put onto the headset, but it didn't really work that well. So it had quite a few issues, so it will be interesting to see if that's been solved. 

Tom Constable: Yeah, it functions and it will I imagine get better. 

Ian Ferguson: Yeah, that's really interesting. And these qualities of HMD that we're talking about, and the resolutions. All this brings us round to probably the biggest hurdle for the training environment for pretty much any environment, but specifically for the training environment is what I'm going to speak about and that's motion sickness. There's no real way around this, and I've read a few papers on it, and the most recent states, effectively, that 100% of users will feel motion sickness from using a VR headset. Now that's heavily caveated because obviously people use VR headsets and they feel absolutely fine and it's more about how that motion sickness manifests in the individual.

 Now when I first started using VR, I felt really bad motion sickness and I don't get that from anything else, so it was quite strange. I sort of wore a headset for about 40 seconds and then that was being wiped out for about 3 hours and then there's no way anyone was putting that back on my face. It just wasn't going to happen. If you're delivering a lesson that's been put into a virtualized environment using these headsets, and even if you only get one student out of your whole class that feels that way. How are you going to teach them? So what we have to look at is what are your secondary or reversionary teaching methods going to be if someone cannot use that technology? Can you stream this out to a TV? Can they get the same or some kind of learning experience from observing others? All of this you have to look at, because if there's an integral training objective that they need to complete with the headset on and the student can't wear the headset. They cannot pass your lesson. So that's probably the biggest barrier. 

Tom Constable: I'm really glad you bring that up because I think that is the key thing that anyone in defence who's looking to procure these products needs to understand. It is a real consideration. There are lots of things that people don't do very well when it comes to mitigating motion sickness. 

People tend to ,for when we first use virtual reality headsets, they're probably at a trade show or an event and they're being watched by their peers and they're being thrown into the middle of a scenario with little training and then being expected to do everything really quickly under pressure. And that tends to be people's first exposure to virtual reality, and understandably they get motion sickness because they're trying to do everything really quickly under pressure in a not controlled environment. And there's lots of ways in which actually, when you train and take people through a process and exposure process, slowly put the headset on for 10 seconds, just look around, take it off, OK, relax, normalise, put it back on again for a minute, look at your hands, pick up a block, take your headset off and over 1/2 an hour exposure process, acclimatise better and have a real positive first experience. I've seen really good outcomes from it, but it doesn't mitigate the reality that someone, one of those fifty people you're trying to teach, just cannot experience virtual reality and what do you do.

 So great thing to highlight. I'm not sure we can provide an answer here, but it is important to identify that. 

Ian Ferguson: It is that motion sickness issue is being addressed and that's by the higher frame rates and the different way that they're now working the display systems inside the HMD's and the one that I'll point out is the Varjo VR3 that we use. So when I hold demonstrations and things like that, we put lots of people through, effectively, an older HTC Vive Pro, and the newer Varjo VR3. And that's just to show the difference in the quality that these headsets have developed over the last sort of few years and with the Varjo VR3, I've noticed that less people get motion sick.

However, other issues start to manifest, so I've had people that they put it on and it's almost like their brains saying to them ‘this is too real take it off’ And they'll take it off again and then it will take two or three attempts, and then they relax and they're into it, and they're fine. They're not feeling any form of motion sickness, they just get that instant ‘oh this isn't right’ and they take the headset off, but you can overcome that one quite quickly. I haven't seen this yet, but some people that were testing the headsets online, you can get the videos on YouTube, were feeling physical touches because it's hard to describe how visually clear that Varjo VR3 is. 

Colin Hillier: Phantom sensations? That's interesting. 

Ian Ferguson: Yeah, so I think with the higher quality headsets that are coming in now. I think other issues are going to start presenting themselves. 

Tom Constable: This at danger of being the world's longest chat which I love. Colin's like ‘I knew this was gonna go longer than it was because Tom was gonna talk as well’ but I'm gonna say one last thing on VR and maybe we could move on to XR, but this is a generic catch on I really wanna highlight this as well and you did say it at the start, but I want to again hammer it home, it's about understanding what are we trying to achieve with the headset, it's not about technology for the sake of technology. What's the training outcomes we want to achieve? And people start getting excited when they start putting any headset on and they start suddenly going ‘Oh my it's gonna be ready player one and we're gonna be running around these massive halls and it could be any war zone and how great is this gonna be and how realistic is gonna be? But actually, every single extended reality concept has its own drawbacks, and it's about understanding them at the minute level and going. ‘OK, what training outcome do we want and which headset should we use in order to achieve that at the lowest cost and highest value’ An example of that is when people talk virtual reality, they instantly jump to, at least when I speak with customers, the big ,whole hall concept where you're moving around, free roaming and fighting in this completely kind of synthetic environment. 

What I talk them through at that point is that look, think about the cost and actually think about the constraint, although that is a big hall when we talk about military training, a hall isn't very big when we're talking about trying to train a even a fire team to do a decent fire team attack, or even doing any kind of urban built up training. The hall is actually very very restrictive in terms of size. So then when we go ‘ actually. Well, let's not do free roam. let's tether up that headset, and now I can build you a virtual reality environment that can go on for 10, 20, 30 kilometres and that means your trainees can then do free roam throughout those 30 kilometres and and make realistic decisions not be constrained by the environment. Then maybe that's a better and more low cost solution to achieve the training outcomes you require’ and it's just about going ‘What are the training outcomes and how do we tailor the requirement and the headset to be, again, least expensive. Most bang for your buck again essentially.’ Does that make sense and does that kind of echo your experience?

Ian Ferguson: Yeah, absolutely. And it's one of the things I hear quite a lot is that, oh, we've got to do VR. No, you don't. If it fits the training that you're trying to do, then great we can explore that, but you have to understand that some of the considerations that we've spoken about before, you know, the motion sickness and tethered untethered like you mentioned, and getting that requirement right is definitely key. I've seen so many projects and there's one I'm keeping an eye on now I won't name and shame it's a military project that's going to be very interesting. And all I'll say is there's three different types of motion going on, and the user is not on a motion platform, but we'll be experiencing 3 different types of motion within that environment, so motion sickness, I think, is gonna play a very big part in that sole project. 

Tom Constable: What I did previously, one of the things I was desperate to do trials with was take our product onto a ship, see how much the kind of the increased negative effect that would have, because it was interesting. So of course it's going to have, I imagine. But how much of an effect is and then is there still value there? Anyway, moving on because otherwise we're gonna overrun. So, brilliant so, thank you so much and that's been a great insight, especially from a kind of end users perspective. What reality would you like to speak about next? 

Ian Ferguson: So we'll cover mixed reality so the head mounted displays that I've used for this primarily is the Microsoft HoloLens 2. But I have sort of experimented with some normal VR headsets using their pass through camera systems. Generally it's the HoloLens 2 that we utilise for this for the mixed reality piece. We've got to look at how instructors are going to develop lessons. How easy is it for the instructor to create or manage a lesson using this technology? Up until recently, it just wasn't possible, so you'd need programmers and 3D modellers, and all of this to make some content for your mixed reality headset, and then go through the whole process.

 Microsoft created a Dynamics 365 set of packages. One called Dynamics 365 guides and that is a step by step process that anyone can use to create a mixed reality lesson. Really, really simple and not surprisingly just like setting up a PowerPoint presentation, except you're restricted to 8 lines of text and a video or some pictures. All of this is presented to the user, and then there's holographic imagery that you then place around your training environment to guide the students through whatever you're doing. 

I've used these to great effect, and I created a lesson about building a PC because that's what I found people were scared of doing. So what better way than create a lesson for that? And I'd literally just have to tell the people how they use and interact with the software on the headset, step away and they would go through the whole process of building a PC without any more interaction from me. And this worked out really well, and so that might lead you to ask the question of why isn't this then across the whole of the MOD. Because when we come back to our considerations, just like with VR, we have to look at whether we want tethered versus untethered for exactly the same reasons. Processing power primarily, but there's also other issues with mixed reality that a lot of people won't really talk about . 

Lighting for one, and I don't mean lighting inside. as soon as you take them outside natural sunlight washes away holograms. The only way we've found to get around this is to get some sunglasses and put them over the front of the hololens, and that works quite well. No doubt something needs to be developed along those lines. 

Tom Constable: It sounds like it's a maturity question at this stage for this kind of technology from your perspective. 

Ian Ferguson: Absolutely. It needs to grow a bit more I think, and the difference between, and i’ll specifically talk about the Hollow lens 1 and 2, it is night and day. The Hololens 2 is massively forward compared to the first one, but it's still restrictive in what it's going to deliver. A lot of people think that you can take one of these headsets outside and watch a mixed reality battle happen on the side of a hill or a mountain in front of you when the reality is, if you walk probably 10 feet away from a hologram, it will start to distort and discolour. 

Tom Constable: That's interesting, and also the field of view is a big constraining factor. 

Ian Ferguson: Yeah, that sort of fits into what we've been saying over the whole VR system. What is your requirement? What are you trying to achieve? Some of the things that we are doing at the moment we're developing an application in partnership with DASA and an industry expert to look at a virtualized bird table to teach battle space management on . Now the restricted sort of viewing space of the HoloLens 2. Even though yes, it could be better, it does what we need it to do on that headset, because it's not a fully enclosed VR style headset we're not losing that interaction with the student. We can still talk to them. We've got no barriers there. We can flip up the visor so we can see their eyes and see their facial and the micro expressions. So it's a lot better inside that training environment.

 My sort of biggest issue, I think that I have, with mixed reality at the moment, and specifically probably the hololens, is what they call Azure offsite servicing and the Microsoft tenant system, and what these are is in effect how it works. The Azure offsite services can range from anything from remote rendering, so you can effectively render full 3D images and it can also run sort of CAD programmes and things like that. So high level detail engineering projects within it that's far too complex for the headset Itself to run, it uses an offsite service to do that. Really, really expensive. But there's a whole suite of Azure offsite services that your headset interacts with, and it creates this whole ecosystem that's caught up in the Microsoft tenant system, and it's probably the most convoluted way I've seen for software management to happen. 

Now anyone that's gonna implement these headsets, if they don't work with an industry partner to create its own application, will have to understand how the Microsoft tenant system works, and if they don't, their headsets will stop working because as soon as something updates within the tenant ecosystem, that has to be pushed out to all of the headsets. If it's not, it just all stops. Now when we look at the skill sets that are generally within the military, I think this is too much of a step outside of that, so we will either have to be, again, paid for for an industry partner to deliver, or ,what I would prefer to do, is probably develop a separate application outside of the Microsoft tenant system and then understand that we won't have some of that utility. 

Tom Constable: Talking like generalities with mixed reality, I think conceptually it's really exciting, isn't it? The idea that you could do a FIBUA training session with a headset on and have virtual enemy throughout the buildings. But, there's so many constraints that now, with the current technology and the software available, and like you say, the additional constraints people don't understand until you start scratching the surface. 

Colin Hillier: Probably the health warning we discussed was quite a gulf between the marketing done both on VR as well, you know, and versus what the reality is. And that's not to say they don't have applications. But yeah, certainly if you go to the website to look at all . you know there's one, there's a classic one, there's a whale in a gymnasium, you know, and everyone's looking at it, and they go well ‘That that's not what this does at all in any sense’. 

Tom Constable: It might do it. If you don't see the tail and the face because of the field of view constraints and then it was a bit faded because the lighting was off and I mean you know. 

Ian Ferguson: Yeah, because it's interesting what you're saying about FIBUA there. It's one of the uses I looked at. How could we use these headsets for something like that? Doing sort of instant response for something that's happening in your building and you could literally go over the same with that, have virtualized targets and things like that. The issue that I found pretty straight away was how the holograms themselves are anchored and you can only have at the moment 1 anchor per scenario, and that doesn't necessarily keep everything exactly where it was the next time you come back. So you'd have to go in and modify it before each use so, but that's all stuff that people are working on, and I'm sure over the coming years we'll get better, but I think that's the point over the coming years it will get better. That's the thing, it's not ready now, so people are looking at these headsets for solutions that it just cannot deliver, but when we go back to bird tables and virtualized stuff and that way it absolutely can. 

Tom Constable: And one area that I have seen mixed reality delivering well on in my opinion is using the kind of pass through sensors. So basically, having a synthetic environment and then having everything within a set amount of distance from you so that maybe a metre or 1.5 metres being the real world. I've seen a joint fires training solution that's using unreal engine that puts them in front of emulated equipment, so the physical equipment that they can touch and see in the real world interact with, but then as they look beyond their 1.5 metre bubble they can see the synthetic environment. I've seen that done well with high quality headsets and I think with a good training outcome, so I think there's specific use cases within mixed reality that are becoming more mature and productized, but is still a way to go.

Ian Ferguson: Yeah, I think that's when we look at solutions like that. I think what we're actually doing is complex VR solutions where we're utilising those pass through camera systems to then create for what we used to do with green screen, so you'd have green screen elements around, showing a synthetic world. Everything outside of that green screen you can see the normal world and I see a lot of use for that, especially in our driver training systems and it's one of the things I'm sort of going for at the moment, as well as when we look at motion platforms, how much extra are we gonna get out of that inside a virtualized environment? Is it going to be worth doing or is it going to be, I don't know whether you're familiar with the first motion, stuff like kicker plates, and effectively all that was was a plate that you put underneath your chair that if you went over a bump, it it basically hit you in the backside and and gave you that experience of motion, but it was quickly found that after about 10 or 15 minutes your brain just ignored it. Now we've got these more up-to-date motion platforms. Is that going to be the same? I don't know. 

Colin Hillier: It's interesting, the brain is easily trickable. I once did my officer on watch training in a bridge simulator and it was probably ,these days, pretty rudimentary, but I walked out of it and I said to the guys. So what motion systems that they use and they looked at me oddly and go ‘it's anchored to an 18th century building you moron there's no motion systems’ like I could swear I was moving so yeah. So I was like how was that possible 

Ian Ferguson: I think that's why I'd like this space so much because it is so cutting edge in some areas, but the rate that it moves forward is really interesting and what's gonna happen over the next few years, I think will be really good. 

Tom Constable: Great, well it's been really useful to have an insider's view of extended reality within defence. Thanks for bringing your research and informed view. Now I'm asking you to put your finger in the air and let me know,kind of, which way you think the wind is going to blow in terms of the future use of XR within defence. No pressure

Ian Ferguson: I think that will be driven more by how the technology evolves, so that's probably what I'll focus on because that will open up more use cases for the MOD. I think the visual quality and processing power of untethered headsets, that will increase as we start to either find Wi-Fi solutions like you spoke about earlier, or as the whole of the computing power process sort of shrinks that will be put into those headsets and then the size of HMD's in entirety. So not just from VR or MR just all of them. How long is it going to be before we're wearing sunglasses or contact lenses that's delivering all of this? That's going to be a truly game changing element. Even if it was contact lenses delivering augmented reality data, the impact that that could have on your life would be quite astonishing. And then if we look at how things like the XR 3 is really starting to blur the barriers between VR, MR and AR, how long is it going to be before those terminologies just disappear, we just have a different reality, and it's all combined into one. I think that's something as the HMD's develop that will also become more prevalent and what needs to really happen, especially for the training space is we need increased social interactions. 

When teaching ,it's a really important part that the instructor can see and talk to that student and inside VR environments you lose the non verbal communication that you have in the day-to-day world. That's starting to be addressed with things like the inward facing cameras and stuff, but it needs to be more accurate. That way, then, the instructors can really have that one-on-one time with students, even though it's inside a virtualized environment and it'll be really interesting to see how that develops over time. And my last point will probably be more data collection, more ways that we can gather data from these devices and then exploit that within the training environment. Even though we didn't really touch on it as we went through the data collection, that is probably the most important point of using any of this technology, so we want to see the stresses that the students under as they're going through any sort of scenarios that we place them through and that data is giving a whole new level of understanding for the instructor towards the student, and I think that's quite important. 

Tom Constable: Perfect. Thank you very much, Ian. I think, for me, that is all. It's been great. It's tough, isn't it? When it's a topic as broad as we've given you trying to get as in a succinct fashion to convey all the nuances of all the different bits of content. It's a challenge and I think you've definitely risen to the challenge. Colin anything from you?

Colin Hillier: Thank you again Ian, for all your insights and explanations. 

Ian Ferguson: No worries, it's been a pleasure and hopefully people will find it useful. 

End of Interview- 00:41:51

Colin Hillier: Well, that was a great overview from Ian. I thought as ever with these episodes it illustrates the depth of the subject that we've chosen, so it's really hard to squeeze it in even into 40 minutes. And there were plenty of areas we could have gone into more depth that we deliberately didn't. 

Tom Constable: Yeah, I mean the classic one is the augmented reality. We didn't get enough time to service that area because it's probably less mature and less relevant to defence at this moment in time. Not saying it won't change in the future and it definitely is being used it's just probably not quite as big an area topic to cover, but, like I said at the top, really useful to hear that perspective, and I personally have taken away a number of nuggets that I think is valuable for me and help will help me going forward. The key one I'll highlight is the importance for people who are looking to acquire virtual reality systems to think about the individual. The one in 10 in 50 who can't physically or will not go into virtual reality. What do you do with that person, do you just say ‘suck it up’, it's just like the CS Gas chamber. It's just like, you know, seasickness, if you wanna be a sailor. It's just part of the course.

Colin Hillier: If you want to be a cybernaut, you're gonna have to be ill. 

Tom Constable: If you want to be an FPV, VR headset, fighting the robot wars of the future, then you're gonna have to suck up some motion sickness. Or is it about being able to provide alternatives for that and actually thinking about that during the recruitment process? That was the biggest golden nugget, I think, for me.

Colin Hillier: Yeah, I mean for me the the big takeaway is, even if something doesn't work for that use case, it's useful to go to the process of trying something or doing some research into it because I think unless you do that, you won't know what's gonna work for you and probably just to put a point on it not to write off one element that's not suitable and then sort of tire out the whole brush, because it definitely is about finding what's appropriate for each training case. 

Tom Constable: Yeah, absolutely continuing to stay open minded and using the data point of failure to go ‘OK. That thing didn't work in that scenario. However, I'm still open minded to the same technology working in other scenarios, not just saying this is all rubbish.’ Which I agree completely. 

End off  Podcast - 00:44:06