Nilanjan Sarkar, PhD, a Professor of Mechanical Engineering Electrical Engineering and Computer Science discusses a virtual reality system that can help new drivers with Autism.
Interview conducted by Ivanhoe Broadcast News in June 2017.
What made you come up with the idea for this virtual program? Tell us why you felt it was necessary to do this and then we’ll get in to how it works.
Sarkar: Autism spectrum disorder as you know affects a lot of children in the United States. Not all of them need lifelong care; some of them might be able to live independently if they can hold a job. One of the primary criteria for holding a job is to be able to go to the job and in United States that means you should be able to drive. Many of these individuals who can live independently in their adult life will be benefited if they are able to learn how to drive a car. The problem here is that the regular driving program that are designed for everybody does not differentiate between who needs special attention and who don’t need special attention. For example, individuals with autism look at the scene differently and they have problem anticipating other people’s intention and things like that. The current driving program is not designed to help them in those aspects. What we thought that if we can create a simulator that can address these special needs for individual with autism they can learn basic driving skills in the safety of a room, they don’t have to go on road where if they make mistakes those mistakes may be very costly. Instead if they can do this in a virtual environment where they can make mistakes but there is no actual accidents. They can be told what mistakes that have made, and then that can be corrected time and again with proper encouragement and feedback.
It would seem like up until now has there been anything to help children on the autism spectrum drive? You were saying they can be taught to drive but they have a fear of doing it or their parents were afraid to let them drive. So this is really a way to get them to really have that experience.
Sarkar: You are exactly right that we have heard from parents that they are really worried that their children will go on the road. They don’t think that learning the driving skill, the basic skills, aren’t that difficult but the problem is trying to do many things at the same time. Trying to drive within the speed limit, trying to follow all the rules, trying to understand what other drivers are doing, trying to understand whether the pedestrians are crossing or standing in front of you. All those myriad of things they have to process and that causes the problem. On top of that, many of these individuals feel very anxious and they also have a different gaze pattern. Their looking pattern may be far from your typical individual. The question is, can we design a simulator that not only teaches them the basic driving skills but also understands that uniqueness of their disability, of their disorder, and address those issues one at a time.
You have created this simulator what is it, do you guys have a name for it?
Sarkar: It changes every time, I don’t remember. Maybe my students can tell you.
Is this artificial intelligence and how does it work?
Sarkar: This simulator allows you to practice your driving skills. This is a virtual world created like a city driving in addition to highway driving and local roads, schools, near hospitals and everything there in a virtual world. You can drive your car through that and it will tell you whether you are driving properly, meaning that if you are at the thirty mile speed limit if you are following the speed limit. If it is a school zone you are driving slowly, if you are on a highway you are driving according to the speed limit, things like that. Also it monitors where this individual is looking while driving. There are important landmarks within this driving simulator where you are supposed to look at. For example, if there is a stop sign somewhere you are supposed to look at the stop sign. If there is an ambulance or fire truck coming you are supposed to look at it and then stop. So this simulator has the capability of tracking your gaze, where you are looking at every instant of time. Based on whether you are looking at the appropriate places of importance it can give you great feedback that, well you made this mistake because you did not look at the speed sign and that’s why you did not know at what speed you should drive. Or you take a left, you have taken a left turn but you did not check oncoming traffic. Maybe you may have been lucky this time but you should be able to judge whether other traffic are crossing the road before you make your turn – that is a safe driving habit. This simulator has this intelligence to understand what mistakes you are making and how to rectify those mistakes not only based on how you are actually driving but also where you are looking at that instant. It also has this ability to measure how stressed you are. Are you getting anxious, are you getting frustrated, are you paying attention, all those things can be measured. We do it using biological sensors, like we monitor their heart rate. We monitor their skin sweating and how they are breathing, those type of signals can be used to infer whether a person is getting stressed out or if the person is comfortable. That the simulator not only measures your actual driving performance, but also knows that whether you are looking at appropriate places and whether you are getting stressed out. Combining all this information it makes an individualized driving plan for each individual. Because as you know autism is a spectrum disorder not everybody needs exactly the same types of learning environment.
So is the simulator giving the student immediate feedback as they’re driving or is it a report that comes at the end of the session, or is it a combination of both?
Sarkar: No it gives the participant immediate feedback. We can generate a report at the end of the whole experiment session but the most useful feedback that we found out is if when they make the mistakes they know what mistakes they made and the simulator allows them to practice that scenario again and again until they rectify these mistakes.
How long have you been doing this?
Sarkar: This project, t from the initial conception to today, has taken about three years.
Has it been successful?
Sarkar: It has been successful to the extent that we have run more than twenty participants, with this system nobody dropped out, everybody like it, we were able to get their gaze data in real time and provided feedback. We also saw within the limited lab setting their driving performance improved. We must make it clear that this improvement was within the lab setting and within the virtual environment. We haven’t made any experiments where they learn driving with this simulator and then go on to the on road driving and assess whether they have improved, that’s our next step.
That still is yet to come?
Sarkar: That is yet to come.
There is no other program like this that exists?
Sarkar: That we are aware of, yes.
The entire goal as far as the driver is concerned, the teenager with autism is to get them to be able to live an independent life?
Sarkar: That’s the ultimate goal, that if they can drive safely then they will be able to go to their work and they will be able to live more independently than otherwise.
Are you hoping, and I know that parents with children with autism are very interested in this, is this something that you hope will one day be available as part of a software program for families to use?
Sarkar: Yes, when we reported our first finding on this simulator we got lots of inquires from parents all over the countries and they were really interested and asked us whether this system will be available for their children to try out. But unfortunately this system is now a research system and is not yet ready for a commercial implementation. What we have decided is that we want to make it commercially available to all parents at a reasonable cost and that’s our next goal is that we would like to build this simulator that parents can buy and take it home for their children.
You said the students seem to have been positive—they seem to like it.
Sarkar: They seem to like it. The primary benefit would be, one, is it’s a safe environment. No matter how many mistakes you make you are not going to hurt yourself or others, so that’s the primary benefit. Research literature shows that an individual with autism makes more errors while on road driving than typical individuals. There is a big benefit in our opinion that if we can teach them how to develop safe driving habit. The second thing is this simulator pinpoints what their particular deficits are. For example, many individuals with autism have a different looking pattern. This simulator tracks their looking pattern and if they are missing out important information regarding driving in terms of signs, in terms of rules, in terms of other traffic it will automatically and quickly in real time let them know their errors and will advise them in an encouraging way how to mitigate those mistakes. That is the second thing. The third thing is it has the ability to adapt to your comfort level for driving. We are measuring their stress so we can know that if a particular person is able to drive but getting stressed out the simulator will automatically know using some artificial intelligence technique that well, this individual needs to be a little more calm and they will reduce the driving difficulty of the task so that he or she feels, more comfortable. Then when they are less stressed out then the system will automatically but very incrementally change the driving task so that they can lift their driving skills slowly without really making them stressed out.
Does it really form to the individual and their responses and that way they can progress?
Sarkar: They can progress, each person at his own comfort level.
Do we know how many sessions, does the teenager go in for one time or they will probably come back every so often and try again, right?
Sarkar: what we are currently looking at how to design a driving curriculum where this individual with autism will start from basic skills and slowly develop their driving skills with a number of sessions. We are currently thinking about three hour total driving session, three to six hours is our goal divided in five, six sessions. We want to then take them on the road and try to see what improvement they need to actually be useful to the real world.
That would be the next step?
Sarkar: That would be the next step.
And you’re going to take them driving?
Sarkar: Yes it’s very interesting that when the story came out someone from Vanderbilt Medical School contacted us and told that they had a drivers program and they have car and there is a rehabilitation specialist who specializes in autism. She contacted us and with her and us are designing a program that she will assess those individuals at the very beginning and then they will go through the simulator training and then she will assess them again at the end. She will take another group who don’t go through the simulator training and try to see whether the simulator training makes them a safer driver, whether their skills improve and whether they require less on road time then they can practice in their home. Because at home it’s much cheaper if you have to pay the on road driver assistant then it would be much expensive.
That will just further this research and you’ll know more.
Sarkar: We’ll know more actually, we’ll be able to objectively assess the value of the simulator with regard to a transfer of skill in real life. That’s the next step.
END OF INTERVIEW
This information is intended for additional research purposes only. It is not to be used as a prescription or advice from Ivanhoe Broadcast News, Inc. or any medical professional interviewed. Ivanhoe Broadcast News, Inc. assumes no responsibility for the depth or accuracy of physician statements. Procedures or medicines apply to different people and medical factors; always consult your physician on medical matters.
If you would like more information, please contact:
Nilanjan Sarkar
615-343-7219
nilanjan.sarkar@vanderbilt.edu
Sign up for a free weekly e-mail on Medical Breakthroughs called First to Know by clicking here.
