On October 20th of last year Tesla Motors published an official blog post announcing an important development:
“as of today, all Tesla vehicles produced in our factory – including Model 3 – will have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.”
Tesla backed up this bold claim with a slick video, set to The Rolling Stones’ “Paint It Black,” which depicted one of the company’s Model X SUVs driving itself from a home in the Bay Area to the company’s headquarters near the Stanford University campus, apparently with no driver input. In a tweet linking to the video, Tesla’s CEO Elon Musk described this demonstration in no uncertain terms:
“Tesla drives itself (no human input at all) thru urban streets to highway to streets, then finds a parking spot”
After months of negative news about Tesla’s Autopilot in the wake of a deadly crash that the system had failed to prevent, the video prompted a return to the fawning, uncritical media coverage that characterized the initial launch of Autopilot. And by advertising a new sensor suite that made all existing Teslas obsolete, the company was able to bolster demand for its cars even as it discontinued the discounts that had driven sales in the third quarter. Like so many of Tesla’s publicity stunts, the video was a masterpiece of viral marketing that drove the company’s image to new heights… but like so many of Tesla’s publicity stunts it also turns out to have been extremely misleading.
Companies like Tesla who test drive autonomous vehicles on public roads in California are required to submit an annual report the the state Department of Motor Vehicles detailing every instance in which the autonomous drive system “disengaged” while on public roads and a human driver took over. This week the California DMV released the “disengagement reports” each company submitted for 2016, providing the public with its first insight into the various autonomous drive technology testing efforts underway in the state. Thanks to Tesla’s report [PDF] we now know that the company’s self-driving demonstration video did not in fact depict the unassisted autonomous trip the company initially claimed.
In 2016 Tesla recorded just 550 miles of autonomous drive testing on California roads, fewer than the majority of its competitors which include established automakers, Tier One suppliers and Silicon Valley technology firms. All but 20 of those miles were driven in the month of October, and the vast majority of those miles were driven between the 14th and the 17th of the month. A source who lives in the Palo Alto area and witnessed production of the demonstration video tells Daily Kanban that Tesla was filming over those four days, and independently confirmed that the demonstration followed the route depicted here. This eyewitness also tells Daily Kanban that Tesla shot sections of the video in multiple takes and that the company stopped filming on Highway 280 when rush hour traffic picked up, apparently in order to make the demonstration less challenging.
According to the same source, it rained in Palo Alto during the first few days of filming and Tesla’s disengagement report confirms that its autonomous vehicle experienced wet road conditions on the 14th, 15th and 16th, while road conditions on the 17th were dry. Tesla’s video shows its Model X navigating dry roads, suggesting that most or all of the footage that made the final cut was obtained on or after the 17th (a subsequently-released video does show a self-driving Tesla in more humid conditions). Musk had initially tweeted that the unveiling of the new Autopilot hardware suite would take place on the 17th, but on the 16th (after several days of filming marred by rain) he announced that the announcement would take place two days later, saying “Moving the Tesla announcement to Wednesday. Needs a few more days of refinement.” On the evening of the 19th Musk tweeted “Will post video of a Tesla navigating a complex urban environment shortly. That was what took the extra couple of days,” and just after midnight he followed up with a tweet stating “Still working on the video. Fully autonomous drive demo only completed several hours ago.” None of Musk’s tweets or Tesla’s official communications mention weather as a cause for the delay, which might have raised questions about the new Autopilot system’s ability to operate in adverse weather conditions.
More troublingly, Tesla’s California DMV disengagement report undercuts the fundamental claim Tesla and Musk made about its demonstration video. The video begins with the following notice:
“The person in the driver’s seat is only there for legal reasons. He is not doing anything. The car is driving itself.”
Yet Tesla’s disengagement report shows that a human driver had to take over control of the vehicle numerous times during filming, for a variety of reasons. The majority of these disengagments are categorized with vague descriptions like “Follower Output Invalid” and “Planner Output Invalid” but they include more specific categories as well including “ACC Cancel,” “Cruise Fault,” and the ominous-sounding “Health Monitor.” 177 of the 182 disengagements that Tesla reported during 2016 took place between October 14 and October 17th, indicating that a human driver had to take over control of Tesla’s ostensibly self-driving car many times during the filming of its demonstration video.
Based on these data alone it is impossible to say definitively that a self-driving Tesla did not drive the demonstration route without a single disengagement or any human input. But combined with eyewitness reports of multiple days worth of filming, it’s clear that at a minimum Tesla’s system had many attempts to practice the route before recording a truly autonomous run. This alone would have given the system an edge that takes away from the video’s impact as a demonstration of autonomous drive technology. So to do the reports that Tesla only filmed during the hours between the morning and evening rush hours, which made the demonstration far less challenging. The fact that a Tesla representative insisted that the demonstration drive started in Fremont, CA (resulting in a correction to this post of mine at Bloomberg View) when both our eyewitness source, a Redditor and SeekingAlpha’s Paulo Santos all independently reported that the route started in Palo Alto suggests a pattern of systematic misrepresentation. This pattern in turn builds off of a much larger pattern of misleading communications by Tesla about its Autopilot system and its putative capabilities.
This pattern has been noticed by Tesla fans who are already aggravated by the slow rollout of Autopilot capabilities to vehicles with the new hardware suite. One commenter at the Tesla Motors Club forum notes
Today’s Electrek article documents why Tesla’s autonomous driving project isn’t progressing quite as quickly as some of us would like. Seems they drove about 500 miles in October 2016, and most of that apparently was focused on producing Elon Musk’s self-driving car video. Once that was out the door, the testing stopped almost completely. Stats like these fairly accurately document the vaporware that most AP2 buyers are experiencing.
Another adds
Holy crap that report is depressing.
If you look at all the self driving, it happened between the 14th and 17th of October.
They were originally supposed to announce AP2 on Oct 17th, but they didn’t until the 19th for “more refinement.” Now we know why.
It appears it took them 550 miles and 168 disconnect events. That’s a disconnect every 3.3 miles. Given they wanted a video without disconnects, it quite possibly took them 169 tries to get a video they could use.
It’s never good when a company’s customers and fans feel like they’ve been sold a bill of goods, but in this case there is even more at stake. The run of bad news about Autopilot that Tesla’s video managed to wash away was based not on a technical failure of the system itself, but on drivers overestimating the capabilities of the system. When drivers believe that a semi-autonomous system is more capable than it really is, they pay less attention and run the risk of over-relying on a system that is not actually capable of keeping them safe. Tesla has made some progress on this issue, for example making Autopilot’s 8.0 software more insistent about reminding drivers to maintain control at all times, but by selling the new Autopilot hardware as being capable of becoming fully self-driving and then bolstering that claim with a video that understated the lack of driver input actually needed, Tesla took another big step in the wrong direction. At best it misled and frustrated Tesla’s customers and fans; at worst, it might have planted a perception that could cause a Tesla driver to put a fatal amount of trust in the new Autopilot system.
