Ghost Autonomy CEO John Hayes was onstage at the Financial Times Future of the Car in London this past week discussing the latest wave of GPTs and artificial intelligence, the implications for autonomous driving and the software-defined car, and how the automotive industry can turn this generational platform shift into a better business model for the long term.
Here are some of the highlights from the discussion with Peter Campbell, Global Motor Industry Correspondent from the Financial Times, with the full transcript available below.
Highlights
New general purpose AI models (GPTs) have real applications for autonomous driving, with new ways to understand complex scenes and new ways to interface with the car controls.
Recent developments in general purpose AI have largely focused on new applications for text and image generation, creating complex artifacts from simple prompts. The next frontier for this technology is going in reverse, taking complex scenes and images and turning them into simple heuristics (e.g. “am I in a parking lot right now?”). Complex scene understanding is at the heart of the autonomous driving challenge, and powerful new AI offers a new tool set to transform existing solutions in perception, planning and mapping.
“A lot of the problem of driving is dealing with lots of complex information. So you're trying to intersect the facts that you see in the world with a large set of laws, with a large set of preferences, and that's a very, very difficult problem to program.
What's emerging is the possibility of using text itself, where you describe in English what your preferences are, what you would like to do in a bunch of scenarios, and then have the GPT generate a thought process and then a conclusion from that thought process.
I think it's going to change the way that driving systems are developed from trying to make very, very complex algorithms and debugging those algorithms, which is where every company is mired and substitute them for these much fuzzier AI systems that will consistently make more reliable decisions.”
Transformational AI capabilities are being made with camera images and videos, not lidar.
Cameras and AI have a broad base of attention and R&D investment across the consumer space for lots of new AI applications, especially relative to specialized sensors like lidar. This broad R&D effort drives big cost savings, performance gains and transformational new capabilities – cameras and AI are simply improving a lot faster than any specialized sensor. This is especially true with the advent of GPTs, unlocking powerful new generalized image-to-text capabilities not possible with other sensors.
“You have a massive consumer industry that's pushing the development of cameras to the tune of billions of dollars of R&D a year. Plus you also have a massive AI industry and that AI industry is built on the mobile platform. So you have software that's created for cameras and you really just don't have the equivalent with LiDAR. Cameras are getting better faster than LiDAR.”
Powerful new AI capabilities can now be made safe for automotive applications.
AI is no longer be a black box that is impossible to understand or reason about for safety critical applications. It is now possible to audit neural networks and determine which nodes and pathways determine outcomes, and determine which types of inputs the network has trained on (and those that might be new). Ghost is building safeguards both at a system level and at a network level to bring powerful new AI applications to automotive.
“You train a system based on lots and lots of examples. You can come up with a probabilistic score. There are new techniques that started last year where you can actually look at how the network is activated. And you can see it's like, am I in my samples? Am I following a path that I've never seen before? And so there are better feedback and mechanisms to figure out how to tune it, but ultimately to make it better, it's an iterative process where you just have to show it a very, very wide variety of data. And then you'll get a program that a human won't be able to interpret, but you can interpret statistically.”
AI can be trained to “fit in” a human environment, with behavior indistinguishable from the best human driver. Driving like a (good) human driver is critical to finding a safe equilibrium on today’s roads.
“I think ultimately the way that humans would drive in a scenario becomes a stable equilibrium for how autonomous vehicles should behave. Because the first autonomous vehicles will be launched into environments where it's mostly human drivers. And furthermore, to not disturb those human drivers, you're going to want to imitate them. And so I think we're going to end up with a long-term equilibrium where without any communication, all of the autonomous vehicles will come to this human-like driving equilibrium.”
The first application for consumer autonomy is on the highway – highways are simpler, relatively uniform across the world, and represent the majority of miles driven in many countries.
Ghost is partnering with OEMs to deliver the first L4 “attention-free” autonomous driving experience for consumer cars, starting on the highway. The highway represents the fastest and most scalable path to safely deploy autonomous driving at scale across the world.
“The nice thing about starting on the freeways is that they're very uniform worldwide. It's when you get into the much slower streets that becomes more and more local. Any of the car companies we talk to do sell vehicles worldwide, so highways are very interesting to them.”
“In the US most people live in suburbia. You don't take taxis, that's not a thing. You store stuff in your car. If you have children, you have car seats, you change them every two years. And so the idea that that [robotaxis] would take off in the suburbs, I thought that that was a major gap in the market. And so what we focused on is something we can complete.”
Complete Transcript
Peter Campbell:
So we hear a lot about AI, don't we? It's almost impossible to spend any time on the internet or reading things without hearing about it. And we hear lots about its potential for the car industry. I'm delighted now that we're joined by John Hayes who is the founder and CEO of Ghost Autonomy, who's going to talk about their approach and talk about AI as well. John, thank you so much for being with us.
John Hayes:
Glad to be here this afternoon.
Peter Campbell:
I know that you've flown in especially for this. Not everybody watching this will be immediately familiar with Ghost. So tell us a little bit about the business, how you founded it and what it does.
John Hayes:
So we focus on autonomy for consumer cars, and that's using ordinary cameras, ordinary CPUs, and taking advantage of AI to eliminate a lot of the special-purpose hardware sensors that are usually associated with level four autonomy.
Peter Campbell:
And do you do that on previous vehicles? Do you retrofit that? How does it work?
John Hayes:
We get manufactured in. What you see is the car platforms are evolving towards having high-powered CPUs, having cameras in all directions, and what we create is software that runs on that stack.
Peter Campbell:
And how does that work in terms of the sensor suite because there's a lot of debate in the industry over use of cameras, use of radar, LiDAR, other things? We had Musk last year who was defending the use of cameras because that's what he uses. How do you think about that whole debate?
John Hayes:
So I think about it in two ways. One of them is what is practical? So right now, cameras are very inexpensive. You can put lots of them on a car and create redundancy. But then the other thing, which is more subtle, is what is getting better faster? And with cameras, you have a massive consumer industry that's pushing the development of cameras to the tune of billions of dollars of R&D a year. Plus you also have a massive AI industry and that AI industry is built on the mobile platform. And so you have software that's created for cameras and you really just don't have the equivalent with LiDAR. And so now you have a lot of pressure in terms of research on making camera plus AI. And so I believe that even at a given point in time if LiDAR was better, cameras are getting better faster than LiDAR.
Peter Campbell:
And what is it that the AI does, right? Because there's lots of different types of AI. This is not the kind of AI that's going to write us an impromptu love poem. What is the AI specifically that we're talking about here, and how does it bridge the gap between a camera and actually being something that is mission-critical to autonomy?
John Hayes:
Well, it's amazing that the AI that wrote love poems got more attention than all the previous AI research that has ever occurred. And all the way up to government saying, "Oh my God, we have to regulate this. We'll have unrestricted love poems. People will fall in love. It'll be a disaster." But what that is, it's part of a track. And that's a change in how AI is developed. So if you go back to the dark ages, the dark ages being 2021, what you would do to train a model is you would probably hire thousands of people to write labels and you'd write a special-purpose model that was designed to recognize things in the environment. And the big change that happened, it started late last year, late 2022, was the rise of general-purpose models. And what general purpose models are is instead of trying to train it for a very particular application, I want one to recognize particular types of signs, particular types of sentences.
Instead, you train a model on the totality of human knowledge and then you do fine tunings based on that. And what happened in March with OpenAI releasing GPT4, this is a model that I think they were quoted costing between 30 and 40 million dollars of CPU time to actually train. And built on top of that has been thousands of special purpose applications that have been built on this general purpose model. And so how we approach AI is, we want to understand the scene and the first layer is we want to understand the scene as actual geometry, which we can take from the cameras. And you train the cameras to recognize geometry and depth, a lot of how LiDAR would do it in the same way. But then you want to understand how to interpret the scene. And that's where the GPT models are coming in because we're looking at the first layer, we're looking at the poetry layer, which is, "Okay, can I take text and can I turn that into other texts or can I turn that into APIs that tell the car what to do?"
The next layer that's coming out is the visual layer, which is can I take a scene and then have that conversation with that scene? And that's been announced by OpenAI. You see preview versions out of Meta, it's been announced by Google, but you'll be able to ask, take pictures and ask it. And what that does is that breaks up all of the general purpose training that you used to have to do to make an autonomous vehicle. The third layer, and this is just in the research phase, is called the embodiment layer, where you can describe and text what you want to do. And what it does is it predicts what motion is required. So now we're talking about the automotive stuff, which is fine motion is now entirely predictable in AI. And so you have this explosion that's been created by GPT models and what you're going to see is a lot of applications from very, very simple applications, like "am I at home" to "am I in a parking context" to, "please move my car from here to here," spoken in gestures or very general terms. And I think it's going to change people's relationships with their cars.
Peter Campbell:
And this just, there's a lot to dig into there. One issue around AI is the whole issue of certification. Right now, we all know that there are lots of companies developing autonomy, but only two so far who have released it in the real world, in the US at scale. And if your system is reliant on AI to take it from cameras to being functional autonomy, how do you certify that given that, for instance, they were trying to test one of the AI systems and a group of people took it and asked it how to safely land a plane and every single answer it gave would involve in the plane crashing and everyone on board dying. So how do you know that the answer the AI has given you is going to evolve into a safe car?
John Hayes:
No one knows the answer to that. I'm serious. And it's because it's a new technology and part of the reason you employ AI is to answer questions that are hard to quantify. So if your answer was easy to make, meaning that, "Oh, if I knew certain facts, I know what to calculate." You hear this out of companies. But the problem is that the real world is complicated and noisy and the solutions for that, where we've attempted to write heuristics that could be certified and could be fully understood, the problem is that those programs don't work and they tend to fail very frequently in real world scenarios.
And so what you do is you train a system based on lots and lots of examples. You can come up with a probabilistic score. There's new techniques that started last year where you can actually look at how the network is activated. And you can see it's like, am I in my samples? Am I following a path that I've never seen before? And so there are better feedback and mechanisms to figure out how to tune it, but ultimately to make it better, it's an iterative process where you just have to show it a very, very wide variety of data. And then you'll get a program that a human won't be able to interpret, but you can interpret statistically.
Peter Campbell:
And what happens in terms of AI at the moment, some of the AI programs are open, anyone can use them. And what we saw suddenly early on with autonomy development was the network effect where Waymo, Google had so much more virtual tests than everybody else, they got much faster, much better than everyone else. How do you guys, A: catch up with that if that's what you want to do? And also are there others who will be behind you who can't catch up or can catch up or can overtake you? Because everyone can use the AI system. How does the competitive landscape work if you're using AI?
John Hayes:
I think that what Google and crew sought to do is make an end to end driver that work in all scenarios. And that's an interesting product to move people from place to place or move goods from place to place. As a startup, that is not the product philosophy you want to pursue. You don't want to pursue a product philosophy where you have to do everything in version one. And if you think about what they have to implement, one of the ends of that trip is going to be a very busy place because people go, they don't go from somewhere sparse to somewhere else sparse. They go from somewhere sparse to somewhere busy or vice versa, meaning that in their version one they had to solve the hardest problems. Now when I think about this as a startup, it's like I want a problem that's actually containable, that can be solved at all.
And for us that's freeway driving, still plenty complex and it hasn't been solved to an L4 quality, but it's something that can be quantified in terms of what it wants to do and in terms of capabilities. And so that's the different path that we're pursuing in that we have pursued the consumer market. In the US most people live in suburbia. You don't take taxis, that's not a thing. You store stuff in your car. If you have children, you have car seats, you change them every two years. And so the idea that that would take off in the suburbs, I thought that that was a major gap in the market. And so what we focused on is something we can complete. And yes, there'll always be competition and all we can do is say that, are we following the most efficient development path? And if we are, it will be hard for someone to follow.
Peter Campbell:
So how does it work physically on the vehicles? It's a retrofit program? Do you have to put other stuff on? How does that happen?
John Hayes:
No, you manufacture in the sensors and the computer and then it's software. They'll be marketed by the OEMs.
Peter Campbell:
So you take another car from an OEM and you manufacture it into that?
John Hayes:
Yeah, it would be manufactured in their ordinary manufacturing process.
Peter Campbell:
And how many of the car makers, you don't have to, you can name them if you want to. Are you talking to about this and where are those discussions?
John Hayes:
That's not something, we're a white label company, so we don't talk about that.
Peter Campbell:
It was worth a try.
John Hayes:
It was worth a try. Exactly. But what you're seeing is a lot of the OEMs are moving to central compute systems, central sensory systems. And so what we're aiming for is the platform architecture that the OEMs are building anyways.
Peter Campbell:
And what are you thinking about in terms of deployment and time scales and which markets? Is it going to be West Coast? Is it going to be somewhere else? Arizona, Europe? I mean you're here in London, right?
John Hayes:
Yes.
Peter Campbell:
So where's it going to be?
John Hayes:
Well, the nice thing about starting on the freeways is that they're very uniform worldwide and it's when you get into the much slower streets that becomes more and more local. And so any of the car companies we talk to do sell vehicles worldwide. And so that's something that's very interesting to them to begin with.
Peter Campbell:
And so how do you deal with that in terms of getting it regulated? Because I remember years ago when Audi bought out the new A8, they said it's got level three on and it's only just now that that system has been allowed to be deployed. So how do you, given the natural caution regulators have over allowing the stuff in the real world, how do you get around that issue? Or is that a problem for the OEMs?
John Hayes:
That's clearly a problem for the OEMs because they're marketing the car. And so it comes down to what's their relationship with the regulators. I think you need someone who has some industrial footprint in order to have any leverage with the government. And often there's a question about how they'd want to market it anyways. And so you can put a technology in, you can begin by marketing it as assistance, collect data, because you just had a bunch of conversation about data connectivity. And then when you see the end-to-end results, then you can change how you market it.
Peter Campbell:
And so you think that being able to deploy this on highways makes it potentially far more scalable because it's able to get highways as you say, go kind of anywhere. What's the big challenge then when you're deploying, when you're thinking about scaled deployment? Is it regulation, is it driver behavior, which can be very different in different places? Is it something else?
John Hayes:
I would say that the challenges are mainly engineering challenges. I think that regulation at its best follows evidence. And so in the development of the product, you create evidence and then that's what you want to talk to the regulators about. So it often becomes how are you dealing with very unusual scenarios? How are you dealing with construction where there are no lines? How are you dealing with emergency vehicles? And so coming up with a scalable system that communicates clearly with the driver when the system is competent to drive and when it's not competent to drive.
Peter Campbell:
And how are you thinking about the issue of, because making a car that can obey the rules of the road and drive on a highway is one thing. Making a car that can do that and can correct for the moron two cars in front, right, is a totally different issue. How are you thinking about training the cars on human behavior?
John Hayes:
So we encountered this in California and so for example, the way, we don't actually set speed or necessarily follow the speed limit, what we do is we actually profile the surrounding traffic and we found that that was important to make something that fit in. And so we combine that and we look at natural human behavior. How do people drive? There's a lot of subtlety even on the highway, if someone's a little too close on the side, if you have a commuter lane where one flow of traffic is going much faster than another flow of traffic, there's a lot of different adaptation that you do. And so a lot of what we're doing is encoding that into the system. And so when you talk about an erratic driver, you get a lot of secondary cues in the environment where other people slow down. And so you can take those cues and incorporate something that would seem like a comfort signal and actually turns out to be a safety signal. And so you can take, that now becomes a much easier system to implement.
Peter Campbell:
And when you're doing that, I don't want to name any names here, but do you factor in brands as well that if a person uses a certain type of car...
John Hayes:
That's about as silly as when I'm in California, people ask, do you factor in turn signals? It's like not in California. It's like people signal randomly. But no, not yet.
Peter Campbell:
Not yet.
John Hayes:
Not yet.
Peter Campbell:
Okay. I didn't name any brands. That was fine. Anyone got a question from the room? Oh, we got lots of questions, right? Okay, we'll get a mic. We'll come to you first. We'll get a mic to you. There we go.
Audience Question:
How does your system differentiate between car, bicycle, jogger, somebody with a pram or just a pedestrian?
John Hayes:
So we begin, we have two layers to how we recognize. So mainly we use geometry. So in other words, we don't train it to recognize particular things. We say roads are flat, you should stay on flat things. And so what that means is all sorts of obstacles be they unusual vehicles, be they construction vehicles, be they cones, the system will not drive through it. And I think that that's the fundamental basis of safety is even if you don't know what it is, you probably shouldn't drive through it. So because you don't want to differentiate that, if you can avoid it.
Audience Question:
So in your system, where does the responsibility for the outcome lie? Will it be with the people who train the AI? Will it be with the OEM that installs the AI? Will it be with the user who activates the AI? The classic exact thought experiment is you're on the freeway load falls off a truck in front of you. If you don't break, you'll die. If you swerve to the left, you'll hit the motorcyclist. If you swerve to the right, you'll hit the Volvo with the baby on board sticker. Humans will make the decision. What decision will AI make and who has the responsibility?
John Hayes:
I was waiting the trolley problem. Okay. Trolley problem. But very, very few accidents follow that scenario. The vast majority of accidents are people just not looking at all. And the vast majority of deaths are people being unconscious. I think that when you're in a no win scenario and you have a harm reduction problem, yeah, you'd probably prefer inanimate objects rather than objects to the side. And swerving itself is pretty controversial because if you do something, if you swerve and someone has access to the steering wheel, there's a very good chance that they'll try and grab it because they won't understand what's going on. In terms of responsibility, I can only talk about how US insurance is set up.
This is what our insurance company tells us is that, look, there's a very good insurance mechanism where right now it's the individual owns the car and they have primary liability. And there's a reason why auto companies are not subject to lawsuits when there's a collision. And that's because that insurance pays out the people who are injured. There's a secondary process called abrogation where if they believe there's mis-engineering or mal-engineering or mal-marketing, then the insurance companies have a reconciliation process where the sort of driver's insurance company reconciles with the auto company's insurance company.
Audience Question:
But in your scenario then you wouldn't break in time, your system would kill the people in the car rather than the people on either side?
John Hayes:
I can't look into people's souls, so that's not an answerable question.
Peter Campbell:
But one day AI can, yes. Right. We'll have a chap here in the front.
Audience Question:
Okay. Have you ever considered off-highway vehicles? So now, I don't know, tractors? You may have less liabilities, less constraints.
John Hayes:
Yeah. That that's certainly possible. Our business is for the consumer and trying to improve driving for the consumer. I think there's lots of other businesses that are pursuing that sort of closed-circuit automation.
Peter Campbell:
Great. Hands up. I'm taking an executive decision. We're going to keep going because we're having fun. So those we'll get through you three. Those three.
Audience Question:
So Tesla's been trying to build a very specific AI for driving and earlier you spoke about GPT sort of models, large language models coming in helping to solve this problem. Can you talk a little bit more about how large language models essentially help to solve such a specific problem?
John Hayes:
Let me think about the not too long answer. So a lot of the problem of driving is dealing with lots of complex information. So you're trying to intersect the facts that you see in the world with a large set of laws, with a large set of preferences, and that's a very, very difficult problem to program. What's emerging is the possibility of using text itself where you describe in English what your preferences are, what you would like to do in a bunch of scenarios, and then have the GPT generate a thought process and then a conclusion from that thought process. Tesla has been incorporating reinforcement learning to figure out multiple paths. They've covered that in their AI day, which is sort of a precursor to GPTs, but I think it's going to change the way that driving systems are developed from trying to make very, very complex algorithms and debugging those algorithms, which is where every company is mired and substitute them for these much fuzzier systems that will probably consistently make more reliable decisions.
Peter Campbell:
Probably, consistently. It's amazing in this entire discussion, it might make it easier for this and this and this, but it's kind of the safety and threshold of getting it over the line of something that's acceptable to other people. Something that I just think is a kind of fascinating conundrum, and I don't know if that given we're using AI, if that's ever solvable.
John Hayes:
Well, I think the problem we have is that we can't encode our preferences specifically enough for a conventional program. I think we've been trying that for a long time and I don't think anyone knows what a human specific preferences would be, but what we can do is we can say, "Hey, is this human enough that it's acceptable to the people in the car, the people around the car, and then behaves predictably and reasonably to the environment?"
Peter Campbell:
That's always the statistic there, that humans are essentially responsible for a million road deaths a year. It's very difficult to see a situation where society happily accepts robots killing half a million people a year on the roads. It's just totally different threshold.
John Hayes:
It's a totally different threshold, but the reason that that happens with humans is often just not paying attention, not collecting relevant information for the decision. Humans actually make very good decisions consistently when they're looking out the window and they have the information they need to make that decision.
Peter Campbell:
Exactly. I'm conscious we have two more questions in the room. We're running into a break after this. Okay, so two, take the two in the room. Chap at the back.
Audience Question:
Oh, for example, one car in California's AI system learned a condition that AI shouldn't be behaving like that. Will that condition be shared with the other vehicles? And when the other vehicles take that data, it happened in California in a highway, but I'm in, for example, Italy. My roads are narrow. I cannot apply this. Will there be something, that fail-safe, or each individual machine learning will be for that vehicle?
John Hayes:
I think it's an open question, how general you make a self-driving system. One of the changes is that if you can have a large general model, then you can localize and train for local scenarios. Often models are classification engines and if you can simplify the question that you're asking, you tend to get more reliable results. I think that it would depend very much what the problem was.
Peter Campbell:
Thank you. Final question.
Audience Question:
My question is about multi-agent AI models. So right now I think your model is kind of trained to go on a flat surface. If it's not flat, don't go. And I think your model's probably optimized for that, but then there might be other vehicles whose models are optimized based on different LiDAR sensor or et cetera, et cetera. My question is, what do you see the role of multi-agent AI models in the autonomous area?
John Hayes:
I think ultimately the way that humans would drive in a scenario becomes a stable equilibrium for how autonomous vehicles should behave. Because the first autonomous vehicles will be launched into environments where it's mostly human drivers. And furthermore, to not disturb those human drivers, you're going to want to imitate them. And so I think we're going to end up with a long-term equilibrium where without any communication, all of the autonomous vehicles will come to this human-like driving equilibrium.
Peter Campbell:
Perfect. Thank you. Thank you so much. John Hayes from Ghost Autonomy. That was terrific. Thank you very much.
John Hayes:
Thank you.
