Let AI Drive Your Car: A Look Under The Hood of Comma.AI

Welcome to this week’s Deep-fried Dive with Fry Guy! In these long-form articles, Fry Guy conducts an in-depth analysis of a cutting-edge AI development or developer. Today, our dive is about a company called Comma.ai, which is leveraging the ability of AI and machine learning to give people a driving assistance solution unlike any other. We hope you enjoy!

*Notice: We do not gain any monetary compensation from the people and projects we feature in the Sunday Deep-fried Dives with Fry Guy. We explore these projects and developers solely for the purpose of revealing to you interesting and cutting-edge AI projects, developers, and uses.*

AI can drive your car for you. Yes, your car, right now. This is the vision behind Comma.ai and their latest innovation, the Comma 3X.

HOW CAN AI DRIVE MY CAR?

Comma.ai primary aims to build an end-to-end machine learning driving system, which involves learning to drive by watching humans drive. In this way, there is no hand coding involved, which makes it quite different from other driving assistance systems.

Comma produces a variety of products which they sell directly to consumers. Their latest product is the Comma 3X, a self-installable device which runs OpenPilot on over 250+ types of cars which would not otherwise have this sort of feature. This level two device controls gas, brakes, and steering primarily on the highway.

The reliability of this device is in the numbers. Out of Comma X’s users, 56% of miles driven are driven by OpenPilot. Although humans have to pay attention and take over in complicated situations, the fact that users are relying on this system for 56% of driven miles portrays its ability to safely and reliably operate a variety of vehicles, especially for those who do a lot of highway driving.

HOW DID THIS PROJECT ORIGINATE?

Harald Schäfer, originally from Belgium, pursued a graduate education in engineering in the United States and has always enjoyed tinkering with the electrical side of a variety of products. Early on, he was more focused on electrical engineering projects than AI, though he always saw the potential for its application to a variety of fields. In 2017, he got his first internship at Comma.ai working as a GPS engineer. He has worked his way up the company and now serves as the chief technology officer (CTO) of the company, running all of the sophisticated machine learning components of the systems.

Upon joining Comma.ai, Schäfer and the company realized that the reason self-driving cars have not been actualized is not a hardware issue. It is not because cars do not have the proper actuators or hardware or computers to function in this way, but rather self-driving is primarily a software issue. For this reason, the focus of the company has been on using AI to train software to operate the car.

At first, Comma.ai was re-purposing Android phones and running OpenPilot on them with the help of new operating systems. Since this early inception, Comma.ai has turned to something that is purpose built, completely in-house. Comma 3X functions similarly to a mobile phone in that it has a central computer, connectivity, it runs a version of Linux, and it has cameras, microphones, and speakers. It operates with the car through an interface called “Panda,” which allows the system to read messages from the car’s canvas and send messages back to the car’s canvas. The way it works on most cars is that there is a camera mounted on the windshield that will send messages to the car, such as “left, right, brake, and gas.” The system can then read those messages, reverse engineer them through OpenPilot and send its own messages back to the car’s canvas, essentially possessing full electronic control of the car while also allowing for the preservation of the car’s application programming interfaces (APIs). In this way, all the safety systems that are active in the car are still present when the Comma 3X system takes over. As Schäfer says, “There are ways you can electronically control a car that you definitely wouldn’t want to drive on the road, and we don’t do that. [We use the car’s] APIs in the way they are intended to be used.”

Installing the Comma 3X device is easy and takes around 15-20 minutes. It is basically installed like a dash cam with a simple cord plug in, making it simple for the user to install. Any updates to the software are then done automatically through WiFi, so there is no need to purchase or install a new device amidst further upgrades to the system.

HOW DOES THE SYSTEM WORK?

There are a few guidelines that Comma follows to ensure safety on the road, which are enforced by the system to ensure safety and overriding human control of the vehicle:

1. If the human takes control over the vehicle, control will be given back to the human without restraint. This transition is done easily by the simple grabbing of the wheel by the human driver.

2. The vehicle should never react so quickly that the human does not have time to safely react.

3. A human must be paying attention at all times. This is enforced by the Comma 3X system through a camera which monitors the driver’s activity, head position, eyes being open, etc.

The Comma 3X system works similar to cruise control in that the user can set upper limits for speed, and the system will automatically slow down for lead cars. It will not switch lanes unprompted, but if the driver does desire to switch lanes, all they have to do is put on the blinker and touch their steering wheel and the system will execute the lane change slowly and safely.

There is also an end-to-end mode where the system will mimic human behavior by slowing down on turns, drive at what it deems appropriate speeds, stop at red lights and stop signs, and more. This mode is currently experimental and is only used by about 25% of Comma 3X drivers, but much improvement continues to be done on this mode through machine learning.

AI DRIVING LIKE HUMANS?

The focus of Comma is to build an end-to-end system that looks like human driving—a system which learns to drive like humans. They attempt to do this by creating a simulator and having the system train in that simulator. This means that the quality of the final system is limited by the quality of the simulator. One alternative is to learn via an artificial driving simulator, but this is not realistic. In these situations, the roads are artificial, and so is the environment. The Comma.ai approach is to instead build their simulator off real data. As Schäfer says, “We want to build a simulator that is based off real data so that if we continue to take in more data, our system will become capable of driving in that kind of data.”

The way the Comma simulator works is by analyzing human driving. Some assisted driving systems are classical or rule-based. This means they have a set of optimizations which the system adheres to, such as staying between lane lines, staying behind stop lines, not hitting pedestrians, etc. But Comma doesn’t believe this approach scales as well as their human-like approach, where they train the systems to mimic human driving. This makes them the only company that ships directly to consumers which mimics human behavior as opposed to a rule-based approach. They aim to continue to develop this simulator in ways that utilize machine learning to predict driving patterns. Once this piece is further developed, their end-to-end system ability is expected to skyrocket.

Comma.ai believes their human-like driving approach scales much better than a rules-based approach for several reasons. One of these is that it can learn how to drive based on the demeanor of certain people around the road or in the crosswalk, the speed of certain objects and people near the road, how to turn on odd roads, and more. These sort of instances make encoding rule sets difficult and allows the human-based system to more naturally and safely operate according to such unpredictable and nuanced circumstances. Also, in this way, the system learns how to stop at stop lights and stay in lanes without programming what a stop light means for instance.

Some people might raise concerns that mimicking human behavior could lead to some safety concerns, as humans don’t always follow the safety rules of the road. However, as Schäfer points out, “How can I guarantee that a human is going to drive well? I can’t. We perform tests on humans with expectations, and by how they perform on those tests and evaluations, we make assumptions about how they will perform most of the time. With some statistical tests, we can be pretty confident about how the system will perform. We can simulate it in all sorts of driving conditions and see how well it will perform and be confident that it will be reliable.”

The Comma 3X is a project with immense potential to transform how we think about assisted driving cars. The AI revolution is here to stay, and AI entities seem to be making their way into just about everything. With that being said, why not let them drive our cars for us, too?