Flying People

Neuroscience meets aviation: measuring pilot workload to improve training

• Jonathan • Episode 1

Share episode

Ever wondered how the human brain adapts to the high-pressure world of aviation? Join us in conversation with Evy Van Weelden, a PhD candidate at the University of Tilburg, discussing her pioneering research in the fields of neuroscience, cognitive science, and aviation. Evy's unique journey from neuropsychology to exploring brain-computer interfaces and virtual reality in flight training offers fascinating insights into how these technologies are reshaping the way pilots are trained.

Evy provides a glimpse into how advanced EEG tech promises real-time insights into workload management. She explains how this technological evolution could transform the training environment and improve pilot performance by offering a deeper understanding of mental fatigue and recovery.

We also discuss what new research might reveal about communication channels and their effect on workload, and delve into the intricacies of language barriers and traditional training methods that can lead to cognitive overload.

Join us on an enlightening journey through the convergence of aviation and cognitive science.

Jonathan Duke:

Welcome to the Flying People podcast, where we discuss the hard science behind the soft skills that make great aviators. My name's Jon Duke. I was an air traffic controller in the Royal Air Force and a helicopter pilot and flying instructor with the Royal Navy over a 20-year career in aviation. My co-host is Matt Harding, a former Navy flying instructor who now flies for a UK-based airline.

Jonathan Duke:

On this episode, we're going to discuss the concept and measurement of pilot workload and how techniques to measure mental workload could be combined with technologies like virtual reality to optimise the flight training experience for student pilots. Providing the expertise is Evy van Weelden. Evy is a fourth-year PhD candidate at the University of Tilburg Department of Cognitive Science and Artificial Intelligence. She has a degree in neuropsychology, a Master's in neuroscience and cognition, and has held internships and studies that cover experimental psychopathology, neurocognitive development and biomedical engineering. She has a particular interest in brain computer interfacing and pilot and instructor workload during training, how it's measured and what technologies like virtual reality and brain-computer interfacing can offer both the instructor and the student pilot. Evy, thank you very much for joining us.

Evy van Weelden:

Thank you so much for inviting me.

Jonathan Duke:

So, Evy, I wonder if we can start by kind of exploring your background a little bit, if you could tell us how you firstly, how you came to be involved in this really interesting area of science and then how you sort of how that developed into the position you're at now where you sit at this really sort of fascinating crossroads between the science, neuroscience in particular, and aviation and aviation Sure.

Evy van Weelden:

So I started studying psychology, neuropsychology, but found out that I did not want to become a clinical psychologist. But I found research very, and especially neuroscience, and so I moved on, applied to a neuroscience master's did that, gained some experience with different types of research, but mainly research into brain-computer interfaces, and brain-computer interfaces. And brain-computer interfaces are literally combining data that stems from the brain, which could be electrical signals, with computers in lots and lots of ways, so you could measure something from the brain and try to interpret it using algorithms. There are lots of possibilities there and you could choose to just monitor someone's brain or even give feedback. So that's something that I wanted to continue on. And then I found this vacancy for this PhD function about brain computer interfaces in virtual reality flight training. So that's very specific, but I love the idea. I already had some familiarity with virtual reality from like a bachelor internship, so it just seemed very fitting at the time to apply to it that's cool, it's uh, it's, it's, it's really uh.

Jonathan Duke:

It's really cool that you sort of casually mentioned that you you completed your master's in uh, in neuroscience. You know no problem. And then brain computer interfaces yeah, no problem. It seems like that's pretty much at the cutting edge of. Is that quite close to the cutting edge of where we are in terms of human interaction with machines, right?

Evy van Weelden:

uh, yeah, so a neural link is like the example that everyone thinks of when you mention brain computer interfaces, but those types of brain computer interfaces, or BCIs, really focus on people that have trouble with moving their bodies, either being completely unable to move or partially. There's a very medical application of BCIs and we in this project are focusing on BCIs and the application of operators functioning in real-world working situations. So, way more applicable, it's also called the field of neuroergonomics, so it's like ergonomics for your brain that's a really interesting concept.

Jonathan Duke:

So so that you started off um in the medical field and then transitioned into aviation or yes, I did.

Evy van Weelden:

Yeah. So I did an internship at the university of melbourne in austral and I worked with Synchron and they are working on stents that are placed in a vessel in the brain and that stent has some electrodes attached to it and I did an internship there and it was super interesting. Then the pandemic happened and I needed to go home, so I was left with some of the data that I collected and had to write a thesis about that. Everything worked out in the end.

Jonathan Duke:

Gosh, you've covered an awful lot of ground in your studies, you know medicine and neuroscience through to aviation. Do you hold a pilot's license as well? No, I don't.

Evy van Weelden:

That can make it difficult sometimes to research aviators as a whole. So I did do my first flight lesson.

Evy van Weelden:

That's the video that you saw on LinkedIn. I thought it was a good opportunity to share that with my network, like, yes, I am finally trying it out myself. So that's where that video came from and it was very fun. And it was a good experience for me as a researcher as well, because, as I speak with a lot of pilots, lots of novice pilots, some instructors, and most of them tell me that VR is, or virtual reality is, is cool and it's very useful. It's a good tool to learn some, some of the skills that they need, but, um, it does not feel exactly like how it would feel in the real world. Sure, yeah, so it's good that, uh, I finally experienced it myself and, uh, I will keep it in mind when I'm doing my research.

Jonathan Duke:

Yeah, yeah, I imagine taking on flying qualification while you're doing all this research would be quite a lot. So can you tell us a little bit more about the current work that you're doing and the sort of projects that you're working on at the moment?

Evy van Weelden:

Sure. So I work at Tilburg University at the Department of Cognitive Science and Artificial Intelligence. This project that I am doing is part of MindLabs, which is something in between an educational setting and a business setting. So my project is one of five PhD projects under the name of Masterminds, and all of these five have collaborators from either governmental institutes or businesses either governmental institutes or businesses and in my case, the partners are the Royal Netherlands Air Force at MultiSim BV and so my project is funded by the Masterminds Project, which is part of the Region Deal, mid and West Brabant, and it's co-funded by the Ministry of Economic Affairs and Municipality of Tilburg.

Jonathan Duke:

Brilliant, thank you. So I'd like to explore this idea of pilot workload in a little bit more depth, and you've already talked about being able to measure brain activity to the extent where it's actually possible to interact with a machine and some of the things that you mentioned about being able to record activity quite deep in the brain. I'd love to explore a little bit about how you define the term pilot workload and how we understand that sort of intuitively, but also how we understand it from the point of view of data from the point of view of data.

Evy van Weelden:

Yeah, so I always use the definition of workloads to be as the level of arousal that is moderated by one's perception of difficulty of a task. So there's multiple areas to explore in this definition. No one is maybe too extreme, but it is hard to find multiple studies that use the same definition. So it is related to the amount of mental capacity that's necessary to conduct a task well and to perform well. It is intertwined with, like I said, task awareness, task difficulty, situational awareness, as well as task engagement and mental fatigue, and I could go on and on, and there's just lots of factors in there and in research. We can manipulate workload if we want to study what the effect of variations in workload is, by either changing the task difficulty in workload is by either changing the task difficulty, changing from one task to multitasking yeah, another possibility is just to monitor one's subjective workloads along one task. So there's lots of different possibilities to study and manipulate workload sure?

Jonathan Duke:

so quite a um, quite a broad sort of range of definitions for the actual thing that you're.

Evy van Weelden:

You're studying, potentially yeah, in my point of view, workload is a bit of an umbrella term.

Jonathan Duke:

Okay, interesting, and so if it's kind of difficult to pin down as a term, how do you go about measuring it?

Evy van Weelden:

And then how do you go about controlling it?

Evy van Weelden:

So we have measured workload in some of the ways that I previously mentioned. So we measure it either using subjective questionnaires, some of which have multiple skills, such as temporal demand of the task or situational stress, while others don't have any skills and it's just a skill, from one to five, from least to most workloads experienced during the task. But we have also used we have also used variations in the task difficulty. So previously we did a study in which novice pilots had to repeat the same flight task over and over and over and we added an additional task to some of those repetitions and then afterwards we could compare those two situations that they were in and then we say, okay, the additional task is the higher workload condition, but if that is really true, for the experience during the task, that is something different. That's really something that you need to ask, but I have to say, not every pilot is aware of their workload, so that makes things difficult sometimes so how are you actually what kind of tasks are you getting the pilots to do?

Jonathan Duke:

to to kind of um moderate workload?

Evy van Weelden:

so we mostly use very simple tasks, uh to make it easier to um vary in the different conditions that we research, and those are usually like a medium turn on a 30 degree bank or a speed change from 180 knots to 110 knots or vice versa. But I've also done a study in which we did four consecutive flight traffic patterns just simple ones in a simulator, but also in real flights, which is interesting. So I also have videos and photos of those. So maybe you've seen those uh, but that was really cool. That was uh with some of our collaborators in toulouse from um I have to say this correctly isa a super 8o.

Jonathan Duke:

Okay, so it sounds like um you're able to monitor um novice pilots in both a simulated environment and in a real-life environment, and I think most pilots would agree that there's a. It doesn't really matter how high fidelity the simulation is, and maybe we'll get onto this a little bit later when we talk about VR, high fidelity, the simulation is, and maybe we'll get onto this a little bit later.

Jonathan Duke:

When we talk about vr, there's a. There's a subjective difference in the experience of flying in a, in a real aircraft, where there's where there's real jeopardy and where you have those other physiological factors um, playing out, um, um, can you talk a little bit about how you're actually measuring workload? So my experience and Matt probably has a similar experience to me, in that we were never really taught, although we were taught to teach people to fly and it was always stressed how important it is not to tip the student over into over-arousal and stress the knowledge of how to measure their workload is very subjective and kind of the kind of a bit of a mythology around it and a bit of sort of um received wisdom, I suppose. Um, so I have my idea of what what a student looks like under high workload. Matt will have his. You, I suspect, have a much more scientific way of observing it and I suspect also that my um instinctive understanding of of what high workload looks like probably is not correct. Would that be fair to say?

Evy van Weelden:

uh, I wouldn't know if that's the case. So what we do is we use the subjective questionnaires and we use the pilot's physiology. So in the past I've used ECG and EET measures. So that is cardiography of the heart and EET is electroencephalography, which measures electrical brain activity, and we can use those types of physiological signals to say something about a user's workload. We can use that data for lots and lots of things. It's possible to and lots of things. It's possible to derive some information about other mental states than just workloads as well, but we focus on workload mostly.

Matthew Harding:

Do you ever try and correlate those sort of scientific measures of workload against any outward indicators? So the kind of things that I would look for from an instructional point of view. You look at body language. You can kind of you know it's like when you're in a room with someone, you can kind of sense and that's not really very scientific at all. But you can start to see, you know white knuckles on the control column coming in. You start to notice that that relaxed demeanor and the way that they're talking in the cockpit it's not relaxed anymore and then the talking stops happening and then they stop absorbing things and yeah, as that workload ratchets up and there are loads of little telltales like that. So have you ever tried correlating the scientific measures against some of the less scientific sort of physiological type things?

Evy van Weelden:

So we we can correlate the subjective measures to the physiological measures. So what I can tell about my own research is that we see, if we compare a conventional flight simulator to a real flight condition, that heart rate increases a lot and that makes sense for a lot of people. So that's something that maybe you could recognize. And when well it's, it's difficult to talk about the, the results from the brain, because it is so multi-dimensional. So we use for our fundamental research, because we want to gain lots and lots of new information.

Evy van Weelden:

We use 32 electrodes to measure signals from the brain when we do research. So that is at least 32 snippets of information from each person or each experiment or each condition that you use, each person or each experiment or each condition that you use. And then there is, within the information, the EEG signals. There are different frequencies that we might be interested in. So maybe you recognize pictures or images that you've seen on the internet or on TV, maybe that it's literal brain waves, and those waves go in different frequencies and we are interested in some of those frequencies, how they relate to each other, how these waves that we're interested in are associated or not with flight performance, and then you can measure flight performance objectively as well by using deviations from flight parameters that you've instructed the people to control.

Evy van Weelden:

So it's so much information that it's hard to describe what is exactly happening. But there are certain areas and certain brain waves that do tell us something about workload of a user. For example, tita waves have been related to workloads in lots of studies. So as workload goes up, the theta waves of the brain also go up, but there are also some other waves that will be less powerful. It's just a lot of information. So in some cases we use machine learning to select the important features for us instead, because we can make an estimated guess if we're interested in a certain topic. But if we really have this much information, then machine learning helps us choosing what we're going to use for either predictive models or statistical tests.

Jonathan Duke:

And do those waves? Do they correspond to a particular area of the brain? Do they correspond to a particular sort of mode of thinking?

Evy van Weelden:

Mm-hmm. Yeah, so these waves are more or less pronounced in different areas of the brain. So the brain is separated into different lobes. So you have the frontal lobe, the temporal lobe, occipital lobe, parietal lobe, etc. So, for example, the occipital lobe has been proved to be associated with visual processing, has been proved to be associated with visual processing. The frontal lobe of the brain is really important in workloads. It is the hub of planning executive functioning. It's really important. At least the EEG information from this area gives us a lot of information about someone's workload.

Jonathan Duke:

I'd love to dive into that in a bit more depth. And you talked about the frontal lobe being responsible for executive function. Can you explain that a little bit? Is that involved in decision making or is that sort of? When you talk about executive function, what, what kind of things in a flying environment are we talking about?

Evy van Weelden:

so the frontal lobe, um, is seen as being responsible for planning, structuring, decision making, um, and I think all of those can fall under the umbrella of executive functioning. So the brain has some localizations of function to a certain degree. So some of your cognitive functions are very localized in specific locations, while others are more distributed, such as your memory or your skills can form a network in your brain as opposed to being localized in one place. But we can generally say that the frontal lobe is responsible for executive functioning, generally say that the frontal lobe is responsible for executive functioning, that your occipital lobe is responsible for visual processing, that your parietal lobe is responsible for spatial processing and some motor functions. It's hard to explain the brain is complex. Maybe I should just say that uh and uh, you definitely um need your whole brain to to function well and do your task and be able to fly yeah, yeah, for sure, yeah.

Jonathan Duke:

And and do you see a difference there in um in high workload, where a person is experienced in the task, versus high workload, where they're learning a new task? Is that um?

Evy van Weelden:

yes, that's definitely a thing. So there are neural differences between novice pilots and experienced pilots. That's just something that I would say is a fact.

Jonathan Duke:

Okay, that's interesting. Okay, certainly there are behavioral differences that Matt and I have experienced, but what's going on in the brain is somewhat difficult to infer in real time. For us you kind of alluded to this earlier it's kind of the cutting edge of understanding about workload. I think I read in one of your papers that the previous understanding or previous methods of measuring workload using sort of observed physical behaviours were considered, are no longer considered to be too accurate or accurate enough to make an inference about the pilot's workload. Is that fair to say? I guess the question I'm asking is this a really big advance in terms of understanding pilot workflow?

Evy van Weelden:

well. It is a field that is growing exponentially right now, so it is kind of new, but the idea is not. However, measuring performance or subjective measures to infer about someone's mental state is still good enough, because our technology still needs some improvements. Our predictive models still need some improvements, the hardware still needs some improvements. So there's still a lot to work on, and it's great to see in real time that this field is growing, that these new technologies emerge.

Evy van Weelden:

For example, there are now VR headsets that have some EEG electrodes on it, and I think that's just amazing that there are people and companies working on this, and I think, as these technologies change, our research will change, but also the application of it will change, because now EEG is not really popular, so to speak, because it's not very practical. So I am actually hoping that in these next few years, there is going to be a product that is very practical to use, not only in a lab setting, as we do now, but also in the real world. Not only in a lab setting, as we do now, but also in the real world, and I do see lots of technological advances and new products coming up, but I have yet to see something that is really going to be very useful and very mainstream. Yeah, that's the word that I'm looking for.

Jonathan Duke:

I have yet to see something that's going to be mainstream EEG recording during real world tasks or training alluded to having the machine learning algorithms and processes in place so that people like me can understand the data that I'm actually getting out of a device like this and the quality of that data as well.

Jonathan Duke:

I mean it would be. It would be fascinating to to be able to have some sort of scientific measure of your own workload or students workload, or even a colleague's workload, and this is something that matt and I were talking about, um in sort of in doing the preparation. The reading for the podcast was this idea of um, the difficulty of operating at a high workload is a part of the difficulty is understanding in the moment that you're at a high workload and then and then broadcasting that so that your, your co-pilot or your colleagues, the rest of the crew, can can moderate or change their, their behavior, or or maybe introduce some sort of technique to to try to reduce and manage that um that situation?

Jonathan Duke:

um, one of the questions that I that I did want to ask you and this this is sort of related to the, the question about novices and experience um aviators as well um, I wonder how, like you you mentioned you can see some of this or you can make some deductions about workload in real time. Do you do you see? Um, do you see, are you able to say whether the brain is recovering immediately from that, from that workload, um that high workload situation? Or are you able to to discern sort of a recovery period or um, like a, a follow-on period of, of um sort of gradual reduction in workload? And the reason I ask the question is to me, it seems, looking back at my career and I've had plenty of opportunities to be at very high workload because, um, I'm not a particularly talented pilot, but the the times that I've had I've been really high workload. I haven't immediately recovered it's, but it's taken some time to, kind of you call it, getting back in the cockpit um, are you able to discern that kind of activity?

Evy van Weelden:

um so we have not studied that. So in our research, mostly our conditions are very controlled. And there are, the tasks are timed, they last for a certain duration, then it stops. Then we can ask the participant something or have a little break, then do the next task. So that's very different from real life.

Evy van Weelden:

But I think what you might have been experiencing at that moment would be mental fatigue that is a result of having a high workload state for a prolonged duration of a task, and that is also something that can be measured from EEG signals Very interesting too. But that could be the result of a high workload state and I unfortunately do not know how long it could take for that to recover. But it is very important to keep track of someone's workload and mental fatigue because both of these high states high workload or high mental fatigue can impact one's performance and vigilance, and that's not something that you want if you're flying. Something that I found very interesting is so we have tried to assess workload of novice pilots in virtual reality training. They had to repeat certain amount of tasks and we try to increase workload by adding an additional cognitive task and we try to predict whether there was low or high workload based on the normal flight task or flight task plus the additional cognitive task, using their EEG signals, as well as a behavioral measure that we derived from data from the joystick.

Evy van Weelden:

Data from the joystick and we expected, as a lot of the literature backs up, is that behavioral data is very predictive of workload, performance can be very predictive of workload, so we expected that that would be very important in our predictive models too. However, that was not the case. So our EEG sorry, I should rephrase, our EEG features were more important than our behavioral feature from the joystick, and to me that was surprising and it intrigues me, and I just really want to dive deeper into this behavioral measure, see what we can do with it, how it would perform next to less EEG data. So now we had lots and lots of EEG data in this model, and I want to know if we decrease it, if then, the behavioral measure really adds something to the predictive value of the model or not.

Jonathan Duke:

Interesting.

Evy van Weelden:

I hope it's a bit clear what I mean, because this really goes into machine learning and features and feature elimination.

Jonathan Duke:

Yeah, I'm sort of I'm hanging in there, I think I'm. I hope I'm understanding correctly. Is that your expectation was that the behavioral predictors would be more correlated with student workload or performance?

Evy van Weelden:

I should probably state it different. So we did find differences using this behavioral measure only. So if we just run simple comparison tests, yes, there is a difference in these two different conditions using this behavioral measure. But when we put it next to our EEG data, the EEG data outperforms the behavioral measure, and that was surprising.

Jonathan Duke:

Okay, so the EEG data on its own is better data effectively.

Evy van Weelden:

Yes, so it was more important. So when we in our machine learning how do I say this? So in our machine learning model we get rid of lots of features that are redundant or not important by using this algorithm. And this algorithm just did not select the behavioral feature, it only selected EEG features.

Jonathan Duke:

Yeah, that is interesting.

Matthew Harding:

I guess there's almost an element of intuitive sense, though, and I mean, actually I don't know if you've had one of these experiences. I've had training flights where I have felt like I am hanging onto the tail of the aeroplane like miles behind, really struggling, and you get to the end and your instructor will say, yeah, that went quite well. You think really, and so maybe sometimes you are able to keep together a performance and it may feel like you're scraping it together and hanging onto it it, and that feels like it's a very subjective way of saying well, actually, maybe my internal workload and turmoil frankly, just didn't translate through my behavior and through the way that I I achieved things, which doesn't necessarily mean that I was doing a great job. It just meant that it wasn't obvious how hard I was working. And I mean, this is very much of me being very subjective.

Matthew Harding:

But as you were describing that measure, I thought, well, yeah, if my instructor could have peered into my brain which I guess is what you're doing exactly with those EEGs then I've had a very different opinion of how I was getting on on that particular day. But as it happened, they only got to see the fact that I didn't commit any egregious errors which made it look like things were going okay. So I don't know, I don't know if that's the same thing, but it sort of struck a chord with me when you were describing that. I thought, yeah, I've had experiences that would match that.

Evy van Weelden:

This example that you gave really highlights why it's important to track someone's physiology next to their behavior, because, indeed, performance can look comparable, but someone's mental state can be very different, and that really affects how someone is learning. So that's really why we think it's important to measure EEG in the first place.

Jonathan Duke:

So I'd kind of like to link through. We've talked a lot about workload and measuring workload and how a better understanding of workload might come about and what benefits that might have. I mean, there are plenty more benefits that we haven't talked about, such as understanding fatigue and and being able to more accurately, I suppose, predict when, when people might be cognitively impaired, um, rather than just tired, um. But I'd like to link through to something that you we've talked a little bit about already but but haven't explored in depth, and that's the, that's virtual reality in the training realm, and then talk a little bit about how that might fuse with the, the studies that you've already talked about in terms of measuring workload.

Jonathan Duke:

I have a particular interest in in VR, and we also have seen the various responses of people with different attitudes towards that training. Well, I suppose we really ought to start first of all by by talking a little bit about definitions. Again, my understanding is that there are important differences between the concepts of presence, immersion and realism, and they're probably important to the discussion. Would you give us an expert's eye on the important differences between those terms?

Evy van Weelden:

Sure. So the difference between presence and what did you call it?

Jonathan Duke:

immersion yes, immersion okay it's probably the wrong term, so please no, no, it's fine.

Evy van Weelden:

So their presence and immersion can be used interchangeably, but it's important to know that immersiveness is a characteristic of the technology that's used, while presence is a feeling of a user. So there's a difference there. So I could say that virtual reality is more immersive than a desktop monitor simulation, just because virtual reality allows for, like, a 360 degree view and the desktop monitor does not. And then presence, like I said, it's a feeling, it's the feeling of being present in the virtual environment that's being displayed. So that's just very briefly what the difference is of those. And then realism, uh, I have to say it's not a word that I use a lot, but, um, you could define a virtual simulation by its realism, as in how much it's, um, uh, how much it seems like a copy of the real world or not. So realism is, in my opinion, something that's not even related to immersiveness and presence, because you can feel very present in a simulation that is like a fantasy world, but that's not realistic.

Jonathan Duke:

But then again you can feel, uh, depending on on the visuals, like if there are trees and those look very real, then there's like different degradations in there sure, yeah, I think perhaps a better word um that I could have used might have been fidelity when talking about the sort of um, the level to which the simulator replicates the real um, the real machine, in terms of function and and um and feeling and sort of physical yeah, and that sort of thing.

Evy van Weelden:

Yeah, so I do use the word fidelity a lot. Okay, um, and then uh, in our I use fidelity to again say something about the characteristics of the simulation and there are very low fidelity simulators. So there are basic multitasking tasks that you can do on a laptop screen. That is very low fidelity. But if you have a full motion flight simulator then that's very high fidelity. But then again a real aircraft is like, yeah, it's, it's the real world, so it's, it's like the highest, yeah, yeah sure, um.

Jonathan Duke:

So I'd like to dive into that um in a little bit more detail, but but before we do I to dive into that in a little bit more detail, but before we do, I wonder if you could explain a little bit about the sort of the subject of the study that you've done or the work that you've done around virtual reality, to learn whether that was related to gauging any of those things that we've just talked about, or whether it was around the utility of virtual reality in delivering effective training or just the general sort of nature of the research.

Evy van Weelden:

Yeah, so there's two directions. So I have tried to compare a desktop monitor simulation to a virtual reality simulation and I have also compared conventional simulator to real flight. I have not combined all of those. I have not combined all of those, but then I also look into these different variations in workload within the VR environment. So those are the two, I'd say, different directions that I have taken and if you want, I can tell something about the results.

Jonathan Duke:

Yeah, yeah.

Evy van Weelden:

Okay, good. So what we see is that in virtual reality there is higher subjective engagement and higher presence than compared to a desktop monitor simulation, and that's something very expected. It's not really weird to see those results. But it is interesting to note that in our case it does not have an effect on subjective workload. So the fidelity of the simulation that we used for tasks did not impact the user's workload. That is interesting. So that makes workload very defined by the task itself, as opposed to the training environments that's being used. That's being used.

Evy van Weelden:

And then, like I said before, when we compare conventional simulator to real flight, we see these differences in heart rate, also some other measures derived from these ECG signals. But we also see that in this case subjective workload does differ a lot. So the workload is higher in real flights in comparison to the conventional simulator. So that's interesting that we do not see this difference when we compare a desktop simulation to VR simulation, a desktop simulation to VR simulation. But there is a difference in subjective workload when we compare a conventional simulator to real flight. But I would say that any simulator would show this difference in comparison to real flight. That's at least what I think.

Jonathan Duke:

Yeah, that's really interesting that the workload is. The difference in presentation of the environment in terms of VR versus monitor doesn't result in a difference in workload and I assume that's for the same task.

Evy van Weelden:

Yes, yep.

Jonathan Duke:

So, yeah, that's super interesting. So potentially and this is something that I happen to be looking into at the moment in my current job is this idea that how do you help pilots control their workload? I mean, in some areas it means reducing it, but in most areas I mean, there is a certain amount of work that needs to be done right, and a lot of it bunches around particular parts of the sortie, and what I'm looking at is how do you enable pilots to manage that workload by moving particular things that they might currently be forced to do at a high workload portion of the sortie and enable them to do those packages of work where the workload is lower, so that they they don't have to be very, very busy. They can just be very busy, and there's a lot of sort of um chin scratching about. Well, how do we, how would we measure that? How would we, how would we replicate that situation? And, if I'm, if I'm sort of understanding correctly, actually the way that you would simulate that is not as important as you might think.

Jonathan Duke:

As in it might not be necessary to go into a full flight simulator and do these tasks. If it's just purely workload that you're trying to measure, would that be fair to say, or have I missed the point?

Evy van Weelden:

It's a difficult one, missed the point. It's a difficult one, um. So, uh, from a scientific perspective, just because we have not found it does not mean that it doesn't exist, um, but uh, I have to say that, um, we do find these differences in um subjective engagement, subjective presence, and those can impact one's motivation for doing tasks, and I think that's also very important to take into account. Yeah, so I am very biased towards choosing virtual reality as opposed to something on a flat screen, because it is immersive. You can view the virtual environment more naturally. So on a desktop screen you would maybe use a mouse or your keyboard to move the visuals, but in virtual realities it's way more natural and I think that's important. It depends on the task, really, but I think it's important.

Matthew Harding:

Yeah, yeah, that makes sense you were saying about the workload doesn't particularly change to the same tasks across different medium. That leads me to two, two questions there is, first of all, do you notice if you get a or maybe you've not done the experiment but if you get a subject to repeat a task several times, does the workload step down, assuming it's the same task, as they get used to the medium, which maybe gives you an idea of you know they're getting accustomed to using the desktop, and they get used to the medium, which maybe gives you an idea of you know they're getting accustomed to using the desktop and they get used to the interface, or the same for VR?

Matthew Harding:

And if so, is there a difference between how long it takes a subject to get accustomed to one medium or other? You know, maybe VR they get used to it more quickly or less quickly, or, you know, because there might be an advantage to that.

Evy van Weelden:

Yeah, so I think what you're talking about is what we call the novelty effect, where there is this um, maybe a bit of a surprise, or um, maybe stress or anxiety, different things that could be happening to a person when they're first using this new technology that they might not be familiar with.

Evy van Weelden:

So in our research we always try to include a habituation phase in which the users can practice a task, get used to the environment, get used to what it is like to look around in the environment and touch all the equipment that we have around, just to get rid of that novelty effect, in hopes that after this habituation phase and practicing that, we can measure activity that is closer to what it would be in reality, is closer to what it would be in reality.

Evy van Weelden:

And we do see that if subjects repeat the task over and over and over, depending on what the subject group is because we've also used complete novices, just students from Tilburg University but as they repeat their interactions with the virtual reality simulation, they get better. But if we talk about pilots, they also might get bored and they might experience an underload, which is also which can also negatively impact their performance. So there is a bit of a trade-off. And yeah, you just have to make the best of it, because if you want to have as much data as possible, you need to have your subjects do these amount of repetitions. So it's something that we try to account for. So when we model our data, then we check if there is an effect of the piece of data and its place in the order of the task that they repeated, just as a check.

Jonathan Duke:

And is that something that you're able to see in real time? Is the sort of the brainwaves of a board pilot versus the brainwaves of a board pilot versus the brainwaves of a of a of a highly strong, high workload um pilot, or or is that something that you discover later on?

Evy van Weelden:

that's something that we discover later on. But if we keep track of the subjective measures, then yes, we, at least over these past months in which I did some experiments with lots of repetitions either the workloads remain stable and sometimes the mental fatigue can increase a little bit. But I have to say that measuring these subjective feelings of workload, mental fatigue or other constructs is difficult when you use a simulation, because pilots are used to the real-world environments and they are way more stressful, way more complex to work with. So if I ask a pilot's your workload uh, how was your workload during this task that you just did? Then it's almost always on the lower end of the skill. And I think it has to do because, because unconsciously, they are biased towards this, these real flight conditions.

Jonathan Duke:

That's really interesting. Yeah, so their benchmark is different to that of somebody who hasn't experienced that?

Jonathan Duke:

Yeah, that's a really interesting dynamic, and do you see the same? When we talked earlier on about the level of I shan't use the word realism, I would describe it as an abstract form of workload, or at least abstract from a purely you know high fidelity cockpit environment, like I guess. The question I'm asking is there like different types of workload? Is it? Does it there's a pilot who is, or anybody else for that matter, who is engaged in trying to operate a simulated aircraft? Does their workload, or does the data um, reflect a different kind of workload to somebody who's just doing, for example, a lot of mental arithmetic, which which can also increase um? You know your subjective workload at least. Do you are you able to discern differences like that, or is it not that granular?

Evy van Weelden:

Yes, so theoretically you could divide mental workload in these different dimensions, such as visual, spatial, verbal. There are these different dimensions that you can use and you could in experiments, but maybe even during training. You can try and manipulate these different channels and see how they affect a person. This is called the multiple resource theory for mental workload and it basically says that there is a capacity like a mental capacity. It has a border and you can, like, divide your mental efforts into these different dimensions, these different channels, and then you can still perform fine.

Evy van Weelden:

But you get into trouble when the workload is high for a shared task, for a shared channel. So you could, for example, perform very well for a task that asks for your input in your verbal channel. So let's say you have to remember words or you have to communicate, while also trying to navigate and really be into your spatial channel. Then you could perform fine. But if you have these different tasks asking your effort into, for example, your verbal channel, you have these different tasks asking your effort into, for example, your verbal channel. You have, for example, communication and remembering words and I don't know something else verbal then you get into trouble because you cannot manage them all at the same time, your capacity in the channels filled up doesn't work anymore that's really interesting.

Jonathan Duke:

So so that, um, yeah, that's not something that I've thought about before, but it I'm, I'm sort of thinking back to my experience learning and teaching flying as well, although I have more experience learning than I do teaching um, and this idea that um aviation comes with its own language and and certainly for for people who are involved in learning to fly, particularly in the military, there is, there's a separate language that exists around military aviation as well, the particular words and it's important that you use the correct word and I'm, I'm just kind of fascinated by this idea that I'd never considered before, which is that student pilots are engaged in learning a language. At the same time, they're trying to use that language in order to function in the environment and they don't really get a choice about. It's not like learning a um, a language, in the sense that you can kind of control the environment. If you've you've not learned, if you've not done the, the french module on how to go to the bakery, you just don't go to the bakery and then you won't have to sort of use that, that set of language. You go to the train station and buy a ticket instead. They don't have any control over, or they have a very limited control over the environment.

Jonathan Duke:

In that sense and I'm just sort of thinking back to the times when I've really seen students struggle, um, and where they're where that struggle and that that workload that tip over into, into over arousal and and and sort of capacity limitation, has been caused by by language, and it's almost always been where an air traffic controller, typically um, has said something that either they haven't understood or they haven't quite caught it, and then they're thinking about their response but they're also interpreting what it is that they've been told to do and they're also in trying to interpret the words that were actually used and it's it's. It's clearly a huge, a huge problem, but on the other side of the cockpit seems like a very simple thing, because there's a lot more sort of contextual understanding. Um, when you've been doing it for 20 years and you're just much more familiar with the language, you're not having to decode it in real time.

Matthew Harding:

Well, I'll tell what, what Evy was saying to me makes real intuitive sense and it's, you know, and actually I this is something which I've kind of I've thought on those ones for a while like I can't speak and listen, like if I'm talking I'm not, I'm not, if I'm listening, I can't form a coherent sentence, like there's only so much that can go in and out on that channel. And it's the same with various other other bits and pieces. You know, patting your head and rubbing your tummy and all that sort of stuff. And what struck me as I was listening to what you said just now actually, is how many times I don't know if you've had this experience. I'm looking more at you, john, from the flying side of things, but maybe there's something happening in the real world and either you're the student or you're in an aircraft and there's a student and they can't get the words out.

Matthew Harding:

So what do they do? They try and use some other way. So there's another aircraft, so they'll point, and what's the first thing? The instructor says don't point. You stop that, use your words. And you think, in that moment of desperation, that student doesn't maybe understand what's going on. But they know that they can't use this bit. They can't use the mouth, they've got to use the hands or something else. And the sort of received wisdom is to shut that down straight away. And maybe we're actually we're robbing them of the opportunity to unburden themselves and maybe get over that hurdle and get on. And that'd be fascinating to see. Like if they were allowed to release that, would they maybe drop a peg or two down the workload and be able to carry on? Yeah, are we hemming them in?

Evy van Weelden:

I really like this idea, how this couples to at least this information from practice and how it couples to the theory behind it. I like that Interesting.

Jonathan Duke:

I have very vivid memories when I was learning to fly helicopters as part of a crew. You get taught a particular way to describe something and, as Matt said, it's like you must not. Pointing is absolutely unacceptable, and I can't even remember what the mnemonic is now that they teach you. It's like distance, direction and description.

Matthew Harding:

A, b, C, d, e, E, f. Oh is it, oh right.

Jonathan Duke:

Action because clock code, distance elevation, further information, teach you it's like distance direction and a, b, c, d, e, e, f.

Matthew Harding:

Oh right, okay, well, we had action, because clock code distance elevation.

Jonathan Duke:

Further information, yeah, there you go it wasn't, it wasn't what I was thinking of, but thank you. So there's, not only is there like a, there's a particular set of words you have to use words, you have to use particular words in a particular order. I just couldn't. I couldn't do it and I would revert to pointing and then I'd get um slapped. But how do I? Yeah, it's, yeah, it's a really interesting. I've it's the first time I've ever um heard somebody explain the kind of the science behind why that might be um, and and it kind of makes me feel a little bit better about not being to be completely honest.

Jonathan Duke:

I'm glad there are other people that that struggle yeah.

Matthew Harding:

It's nice to think that there's a reason behind something which sort of feels like it feels like it's the way it is. It's not just you and me.

Evy van Weelden:

I have to add that there there is this research group. I think it's from the US. What is it? Maybe I can look it up later, but there is a research group that does this research about these different channels in workload and then tries to use EEG signals to back that up that theory. So that is very interesting. So they try to manipulate these different channels and see how it affects the user and their brain.

Jonathan Duke:

There's some really interesting things to pick up here around the, I suppose things to to to pick up here and around the, I suppose the um, how some of this science might be applied in in the realm of um, of both monitoring pilots for the purposes of understanding, uh, what a safe environment looks like you know what an ease that what an environment looks like for somebody to cope with easily versus what a difficult environment looks like, and where that presents risks as providing data that is going to inform an application of some sort of technology in the cockpit, or is it data that's going to inform the application of sort of a more subjective measure? So, are you trying to correlate the data with things that are already detectable, or are you looking to develop tools to detect new signals that would help pilots understand their level of workload, or is it purely laboratory work to understand the neurophysiology of workload itself?

Evy van Weelden:

It's a bit of both. So we want to know more about what signals are important generally. Which signals do tell us something about a pilot's workload? Do tell us something about pilots' workload. But the end goal is creating this system in which pilots could practice certain flight tasks or different environmental conditions while they are passively being monitored by using their physiological signals. In that way, they do not have to actively think or reflect on how they're doing or how they're feeling. It would be automatically without their input. So that's why we call it passive brain-computer interfaces, because the user doesn't really have to do anything extra. So we have been working on that, creating a system in which the virtual reality would change according to the user's neurological signals.

Jonathan Duke:

Fascinating stuff. So the other point that I wanted to ask you about is kind of unfortunately it's unrelated to that, so I shall have to find a way in the edit to make this seem like I'm not jumping around all over the place. But you mentioned earlier on using ECG as well as EEG. But you mentioned earlier on using ecg as well as eeg is easy is your heart rate or other. I'll rephrase that is how there are there aspects measurable by ecg that are reliable indicators of workload yes.

Evy van Weelden:

So one of those would be heart rate variability. So, briefly, explains the variety in the spaces in between each heartbeat. Tell us something about someone's workload. Quite all right and I'm saying that carefully because I am biased but I would say, for example, eeg can give so much more information than heart rate does, because it's just one channel and there's only heartbeat and you can derive multiple measures from these intervals in your heartbeats, but it's essentially just one signal that's coming in. Maybe I'm making some other researchers mad, perhaps, but I think it's a bit too one-dimensional, because stress affects your heart rate, physical activity affects your heart rate. Physical activity affects your heart rate, lots of emotions do, and I think it's hard to separate all those different things. Yeah, and say something about just one mental state.

Jonathan Duke:

Yeah, that's fair enough.

Matthew Harding:

I could imagine that the difference in my heart rate um while I'm subjected to student aerobatics is certainly very different to my heart rate um in the instrument pattern but but that said, you know I have one of those sports watches that has a heart rate variability monitor and it's, you know, yells at me apparently if I'm getting stressed out.

Matthew Harding:

And you know, I've spotted, I've spotted some trends in myself, but I suppose that's a watch on my wrist now.

Matthew Harding:

If heart rate variability is potentially a limited metric anyway and I imagine that the over-the-counter technology is is perhaps of a little bit less fidelity and giving me less information than you'd be able to glean in the lab, but nevertheless, might it be possible that something like this watch I've got and there are some brands of watches will do ecgs, won't they as well? So is it something that maybe a wearable technology like a watch might be able to give me enough information in a flight or professional? Why? Why? Why limited to pilots, right In a professional context, to say you might be working quite hard now and that's just enough to give me that sort of not just me looking at myself waiting to feel like I'm busy, but something giving me a poke to be alive, to that and then maybe make a change or at least consider my actions, ask for help. Consider my actions, ask for help, something like that. Do you think that that technology is there and that application is valid yet, or or we're not there.

Evy van Weelden:

I know that the technology is there, um, but the question always remains on how valid it can be, and that's something that's just um has to be researched continuously. Um, I'm not sure if there will ever be a conclusion to that research, but I have to say why I am biased in saying that I think ECG does not give me as much information as EEG would. It is still very reliable in the sense that because it's one channel, it's easy to track changes. Maybe I should state it that way and for EEG that's way harder. Ecg has been researched so much that it's quite clear on what the changes mean, and for EEG, some changes that we find are still a question mark for us.

Jonathan Duke:

That's interesting, so there might be. It sounds like there's likely much more to discover in terms of the EEG research. Yeah, that's really interesting, and I mean, how long is it going to be before we're putting on flying helmets that are in some way measuring our brain activity, right?

Evy van Weelden:

Is that possible? It should be possible. I think Again, I am biased, but I think it should not even have to take that long anymore, since nowadays there are headphones and glasses with EEG electrodes in there. We can always try and see what happens if you put one in a helmet. I don't think we're far off. It needs a person or a company or an institute that's proactively going to create it and research it.

Jonathan Duke:

Yeah, matt and I were talking about where this tech would take us in aviation and whether it would end up with um, like warning lights and sirens if you reach a certain level of workload, you suddenly you get a warning light and a buzzer going off to really take you, you know.

Evy van Weelden:

Yeah I'm not sure how helpful that. Yeah, that's a bit difficult because it doesn't have to be used for workload only. Maybe that's something to consider. But we can keep in mind that EEG does not only have to work for alarms or anything. It can perhaps just switch to more autopiloting of certain tasks that are too much to manage at a certain moment when an overload is detected, and I hope that's perhaps one of the directions that this BCI work can go into. So for now we're just focusing on training, but it can be useful in these real situations when somebody is experiencing an overload. Perhaps it doesn't have to give a warning to the user as they are already in an overload state, but it can give a warning to maybe someone on the ground or I don't know, like a co-pilot or indeed being used to change some autopilot functions, something like that. I think that could be useful.

Matthew Harding:

It's a bit dark, but I suppose it could even be flight data recording and that kind of stuff, flight data recording and that kind of stuff, so you get a sense if you're ever having to pull data from the aircraft or flight data monitoring, like what state, mental state were the air crew in when this happened?

Matthew Harding:

And you know, I suppose the other tech side of it's always the the exciting bit and we always want to imagine a high-tech future. But could it not potentially equally lead to, you know, rules of thumb? And guidance is where you say you know, in training we notice that when we do these things, these things happen and you know. So. Therefore, rather than actually having to go down the route of just embracing tech in the aircraft, actually you start to learn. Actually, here are the patterns. So maybe we can change, change the way that we organize flight decks, write procedures, deliver training, structure a syllabus, organize a flight, all that kind of stuff, to try and keep people in a more optimal workload and to try and avoid, you know, bunching up and then periods of boredom.

Evy van Weelden:

Mm-hmm. Yeah, so that is really the core purpose of our research to personalize flight training, in our case, specifically virtual reality flight training, and we can find some way or method to use information from EEG to personalize and improve these individual training tracks.

Jonathan Duke:

That's a really interesting point. It's not something that I picked up from what you mentioned earlier when you were talking about your partnership with the RNLAF and Multisim BV. I wonder if you can sort of describe what the you know? What does the ideal future look like, given that partnership, given that it's you know, it's the RNLAF and they have a particular type of pilot that they want to produce.

Evy van Weelden:

they want to produce people who can operate in combat and they have a particular timescale over which they're've derived already and what insights you're hoping to derive to deliver that future scenario. So our final goal of this project was always to create this BCI, use it in real time during virtual reality flight training. We have managed to set something up that is very close. So it measures EEG in real time, interprets EEG data, classifies it into lower high workload states and then couples back to the simulation in a certain way. So in that sort of sense, we did reach our end goal of this project. But there could be so many things that could be better, I have to say, and I have certain ideas about that, but I'm not really planning to elaborate too much about it because I've been working on some proposals. So I want to keep it a little bit secret.

Jonathan Duke:

That's fair enough. I used to talk to my students actually about the role of the instructor really being one to provide a safety net and the other to control their workload. But certainly, once they've learned the fundamentals, really um, in flight, you're really there just to act as a safety net and control the workload and, I think, having more information and, more importantly, accurate information about what's really what workload they they're actually experiencing and being able to close that loop and, as you say, feedback in real time and moderate that so that they get in the absolute optimum learning environment, I think is a hugely interesting area of experimentation and, hopefully, development into technologies that can be applied in the near future.

Jonathan Duke:

They're being extremely exciting to see av. Thank you very much for for being our inaugural guest. I don't know whether this will be the first podcast episode to go out, but it is the first that we've recorded um. You've been tremendously patient with matt and I, and, and thank you for for taking a big risk. We haven't even got your properly branded website yet and yet here you are providing your expertise and we really appreciate it. It's really kick-started this project. It's something that Matt and I have been wanting to do for a while, and it's something that's only been possible because you've agreed to be part of it. So thank you very much indeed, matt. Thank you for bearing with me and for coming along on the ride.

Jonathan Duke:

Um, uh. For those of you who are listening, um, hopefully there are some other podcast episodes for you to come to, to listen to, uh, to follow on from this one. Um, uh. Thank you very much, uh. Please like and and leave a comment and, and. If you really would like to hear more of this kind of thing, then subscribe and we will see you on the next episode, thank you.