Sonatus Advanced Vehicle Data Collection with AWS

John Heinlein’s presentation

JOHN: Hi, I’m John Heinlein. I’m the Chief Marketing Officer from Sonatus. Sonatus is an automotive software supplier based in Sunnyvale, California. We have offices around the world in all the major automotive development centers in the U.S., in Europe, and Asia as well. We’ve been winning awards, including–- we’re really proud— last night, alongside AWS, we won two Motortrend SDV Innovator Awards last night at the show so we’re very proud of that. We work closely with automotive OEMs as well as Tier 1 suppliers integrating our software into their vehicles and into their subsystems. And our motto at our company is “Accelerating Vehicle Software Innovation.” Our goal is not to replace OEM teams, replace Tier 1 teams, but rather provide them a lot of foundational software that they can build upon and get to market faster. We’re in production today with Hyundai, Kia, and Genesis, and we just announced we’ve passed 1 million production vehicles, around a dozen models. And by the end of next year, it’ll be many dozens and many millions of vehicles. So we’re really proud of that. We sell the Sonatus vehicle platform. This is all software. Vehicle software. And it comprises five interlocking products and some solutions based on top of that that help accelerate the shift to software-defined vehicles. Every OEM across the entire show is moving toward software-defined vehicles. 100%. But every one of them is in a different place in their journey. And we’re trying to help provide important foundations that can help them accelerate their shift. We begin with Sonatus Foundation. Sonatus Foundations is in production today. It’s underlying SDV technologies that provide things like networking management, including dynamic network configurability. Vehicle networks are getting more complicated all the time, and it’s important to be able to configure them and update them dynamically as needs change. Things like data management, application services, and cybersecurity. And on top of that, we have a number of valuable platforms that link to that. For example, Collector, I’ll talk more about it in a moment. Sonatus Automotive provides flexible “If this, then that” type of capabilities to do feature-on-demand development, prototyping, vehicle testing and so on. In our software, there is both in the vehicle and in the cloud. And that’s why we have such a strong connection to AWS, which we’ll talk about extensively today. Our Sonatus Guard Cloud platform provides cybersecurity, especially focusing on network firewalls and intrusion detection. And just last week, we announced our brand new Sonatus Updater solution for world-class automotive OTA . So we think this is going to really address the unique needs of automotive software updates. But the rest of the time we’re going to spend focusing on Sonatus Collector. And Sonatus Collector is our groundbreaking, highly configurable, dynamically deployable data platform in production today, rolling down the road that allows OEMs to dynamically deploy specifically, surgically targeted data collection policies to solve important needs. And we’re going to show you a scenario today that’s a great example of what you can do with that and how we’re working closely with AWS. When you look at the value proposition of Sonatus Collector, it allows OEMs to innovate throughout the lifecycle. Typical approaches to data collection are quite brute force. Typical telematic systems collect a specific set of data every 30 seconds, every 5 minutes, whether you need it or not. But if you have specific problems and we’ll show you an example of a specific problem, we can dynamically deploy a data collection policy to the vehicle while it’s driving down the road. You don’t have to wait for a reboot, you don’t have to wait for an OTA overnight. It’s not a seven gigabyte download, it’s maybe a kilobyte download to the vehicle to have it begin collecting data you need, and then action that data to do important things. It allows you to do a range of use cases. It can be for future vehicle planning. Things like understanding how the vehicle is being used. It can be recall avoidance. Looking at a specific scenario that says we have a potential issue in this very specific scenario, let’s do a very targeted policy that helps us identify that potential problem and work to mitigate it, which is exactly the scenario we’re going to show you in a minute. It also can improve customer experience. Our partner is using this today to do a better job of understanding vehicle service needs. So when the customer brings their vehicle in for service, they have a better experience. Perhaps they can plan ahead parts and logistics, and so on. All of these things can be provided by Sonatus Collector. The other really important thing is it’s very efficient. By targeting the data collection for the specific needs you have, we’re not uploading data you don’t need all the time, but rather we’re uploading only specific targeted data. This can optimize not only the LTE upload cost, which can be significant but also improve and optimize your cloud storage and processing needs with AWS. It’s very important. So this is our product, but then by working with Amazon, we’re able to do much more on top of this. We’re able to take this data, which is valuable targeted data from the vehicle, and then upload it to Amazon cloud and enrich it with valuable services from them. And this can happen in near real-time. So if you’re trying to diagnose issues— in this scenario, we’ll show you is diagnosing an issue—we can in real-time enrich the data with other things. That could be weather data, it can be other manufacturing data, things like whether a particular version of the hardware is a problem or other things like machine learning and A/B testing, which we’ll talk about. This allows you to do very fine-grained decision-making quickly, addressing problems rapidly, and it allows us to access the broad ecosystem of partners working with Amazon that can then connect to the vehicle data. So it’s a very powerful way for OEMs, Tier 1s, and their suppliers to have a rich platform for innovation. And this kind of analytics, come to our Sonatus booth, which is just literally right behind us about 50 feet that way, and we’ll show you a number of examples of how we’re doing this. This is really compelling. Now, the scenario we’re going to talk about for the rest of our presentation today is imagine you’re hearing reports from your customers that the ADAS system seems to be failing to detect pedestrians at a specific intersection. Now, by law, most cars now have to have automatic emergency braking to protect you from hitting pedestrians, okay. But let’s say you’re getting reports that for some reason at this intersection, you’re having an indication that braking is not happening automatically and drivers have to brake to avoid collisions. Well, that’s a big problem. You could kill someone. We want to fix that and we need to fix it today. How can we do that? Well, we’re going to define a specific targeted policy to identify events of harsh braking at that particular geofence location. We’re going to detect detailed vehicle information such as what was the ADAS system doing? What was the braking, what was the dynamics? As well as multimedia data like vehicle camera data and other sensor data. And we’re going to upload that to AWS Cloud. This allows us to do a number of things, including A/B testing of different models, for example. We may have different ADAS models in development. We may realize one of those models is having a weakness. It may allow us to do other kinds of analysis that says actually version seven of the camera has a problem and the cars affected are all version seven and it’s a hardware fix. Or maybe it’s a driver problem in the camera. It allows us to detect that quickly. This enrichment with Amazon allows us to then take that… Instead of taking terabytes of data, bulk data, that doesn’t help us collect it. We can surgically choose data we need, enrich it, and then hand on a silver platter to the ADAS engineers that these are exactly the scenarios that are a problem to help you identify and fix the problem quickly. So there’s a lot of technical details that make this possible. And so to talk through that, I’m going to hand it over to my partner, Michael Garcia from Amazon to talk through how we’re doing this enrichment based on top of the data from Sonatus.

Michael Garcia’s presentation

MICHAEL: Thank you, John. So, hi, everybody. My name is Michael. I’m in Principal Product Solutions at AWS. So now for the second part of the talk, we’re going to go a little bit more into the technical details and then you’ll also see a screenshot of the solution. So let’s dive into the technical architecture. We’re going to start on the left with a fleet vehicle that is first using Route 53 to get connected to the proper endpoint. Today, Sonatus is using a mix of NGINX, Mosquito and elastic load balancing to do the MQTT and HTTP. And they’re currently considering migrating to AWS IoT Core to do that in a managed way in the most capable way. So here we’re using MQTT for near real-time needs and also anything that is, you know, vehicle signals that is quite small in size. Everything that is more heavy in terms of data, anything that is media, like, for example, images coming from cameras, all of that is going directly to Amazon S3 and uploaded with HTTP. Once the data is ingested, it goes into a data pipeline system. So all the data is funneled through a Kafka cluster. You can see here the use of Amazon-managed Kafka. And then for all the computing that has to be done on that data… So there’s many different needs for example, doing transformation on the data. That can be unit conversion or more complicated transformation. Correlation of the data, timestamp on the vehicle, and correlating which vehicle signal goes with which frames on the system. So for all of that, Sonatus is using containers. So it’s a very elegant solution to keep adding more over time and having flexibility. And all of that is deployed through EKS. In terms of data storage that is being used, PostgreSQL is powering everything that has to do with the management and configuration of the whole solution. And then for data store, RDS is used. And there again, in terms of migration, the Sonatus team is looking at using RDS and ElastiCache in the future. So now we’re going to zoom in to specifically one piece of the data processing that is the enrichment that John just talked about. And so how are we achieving that? So for this, we’re using a lambda function that is going to call, you know, many different services. In the example of today that you can see on the Sonatus booth, we’re first calling Amazon recognition to do computer vision on the camera frames. So Amazon recognition will give us back bounding boxes with confidence on detecting pedestrians, traffic lights, cars, anything that has to do with the automotive context. And for that, it’s using, you know, the previously uploaded camera frame from S3. Here in the example, we’re using Amazon recognition. But as John mentioned, you can use this to do A/B testing. So the real interest there is that you would actually use your own ADAS algorithm being able to line many of them. So if you want to improve and test a new version, you can use that and then compare the results. If you want to help with the selection of what should go on to the vehicle in an embedded fashion, you know, there’s a tradeoff of the model has to be lighter, it has to be very fast, but then it becomes less accurate. So that’s a great way here to compare the different options and make a decision on, okay, this model has great performance. We’re good with the accuracy. We’ll get it to the car. Maybe this one is bigger, even more accurate. We’ll keep it on the cloud and maybe there’s an external call from the car to the cloud to bring the information back to the vehicle. And then the last piece in our example is actually getting information from the hardware and software version of the ADAS system that can be pulling information from manufacturing systems or external partners for road and weather conditions, for example. All of that data enriched is put back and then exposed through many different interfaces. So GraphQL interface, REST API, access to the Kafka cluster. And here again, we see a selection of AWS services for… currently under consideration for migration. All of that is programmatic access. You can build your own UI if you want to integrate or you can just use the out of the box UI provided by Sonatus. That’s the Sonatus Command Center. And now in the next couple of slides I’m going to show you actually screenshots of that and of the demonstration. So here what you see on the screen is what John mentioned. First, all the data are being collected at that intersection where we had harsh braking. Every line is an event and we can see there’s more than a thousand events there. If you open one up, you can see the camera come in from the camera frame, sorry, coming from the vehicle, some vehicle signals. But if you have to go through a thousand of them to try to understand or pipe that into, you know, later in the pipeline, it’s ETL’s jobs. It’s going to be more complicated. It’s going to take more time. And so that’s where the enrichment comes in. Now that the enrichment is done in near real-time, what you can see is another column which is here with many different labels, one of them is pedestrian and that’s what came back from Amazon Rekognition. And so now all the data you’ve seen can be filtered down to 8 events. And so that’s going to accelerate a lot the work of the analysts that are trying to understand and troubleshoot. Further along now you can see on the data the actual bounding boxes that came from the computer vision detection and also other information. In this example, we have the hardware version of the vehicle. We have the software version of the ADAS deployed. We have the calibration of the camera. So all of that will have an impact on the machine learning algorithm and their inference. And so you have it all in one place here in the same UI. So as John was mentioning, now you have everything that you need to do your job in one place. And again, you can share that then downstream for further analysis even later on. So everything that you’ve seen today with the ADAS use case, it’s just an example. It can be applied to any other type of use case where a product engineering team at an OEM is trying to understand better how a part of the vehicle is behaving. And so we can adapt the data collected and the integration to make it, you know, tailored to your use case.