Thread by @greentheonly

Recently the Tesla "shadow mode" has been in the news prominently again with raging debates if it exists or not and if it does - to what extent.

Let me explain to you what I learned on this topic in my reasearch #TSLA #TSLAQ 1/

We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose 2/

What Tesla actually has is automatic data collection of "events". This data collection has two facets: fixed and "campaigns" as I am going to refer to them. The difference is the "fixed" events are hardcoded in the code while "campaigns" are uploaded to (some) cars often 3/

So what fixed campaigns are there? Obviously if you had an airbag deploy event or approach something real close at high speed - this kinds of events generate some camera footage and some metadata (more on it later) and sends it to Tesla. 4/

here's a crash footage sample: 5/

Here's a "sidesweep" sample: 6/

They also collect crashdumps if software crashed (happens very often), but this is boring so we will not explore it here.

Now the other fixture of fixed events is "autopilot trip logs" - these are ALWAYS collected. Yes, even if you opt out of data collection 7/

What's in there? Entire route you took from placing car into D to placing it into P. The route is split into "buckets" describing how fast you drove, was autopilot available or not, how well was the road recognized by AP and such. Like this:
8/

different colors you see in that picture represent availability of AP, since we needed to select only some stuff from what was being recorded, but there's much more in there of course.

Again, this data is sent to Tesla at all times and also over cell connection. 9/

I think this is how they publish those "X miles on autopilot" and such stats.
Another one of this kind is a "disengagement report". It sounds exciting, but is actually rather boring. The data there is coordinates, how did you disengage, speed, heading and time 10/

no pictures in those fixed disengagement reports so they are really small (under 1kb).

Now a more interesting topic is "campaigns" as I am going to call them, they are usually updated daily, but I have seen more and less frequent occurences We are on campaign 260 today: 11/

Tesla uploads what they call "triggers" to some cars by some criteria (mostly related to how often you drive it seems, also if you are on wifi often). These criteria outline what sort of conditions they are looking for and what data to collect. The results are only via wifi 12/

If the car does not get on wifi in some time the results expire and are deleted. The triggers themselves also have an expiration data (typically 1 week) and also a limit on number of snapshots they can generate (varies). 13/

Additionally when a trigger is hit car checks with mothership with some general information about it and mothership might instruct the car to discard the results without sending (criteria unknown) 14/

Triggers actually have a huge list of things they can match from general vehicle position/state to individual image "labels" (made by NNs) from any camera, to some internal autopilot state. 15/

Often they also have a blanket "starfish" trigger that would fire ~30 seconds after any other trigger and would collect internal autopilot object recognition status. If we were to draw this status on a blank canvas, it looks like this: 16/

it's actually a bit more detailed. This file is relatively small (~10MB) compared to what is conveyed but it only useful in conjunction with some actual pictures to see how good are the detections. data conveys about 2 minutes of activity. 17/

the "image NN labels" are actually not text labels, but raw NN output ~4K in size that allows them to fine-tune their detections to produce less false positives and in general fill their library of particular objects based on NN raw output. 18/

From what I can tell, almost all cars get these "campaigns", but not every car gets all of them. You see these uploads as big spikes on your car wifi stats if you track that stuff.

Now look at a sample campaign from last week to see what's in there: 19/

Every trigger has a name and I will use these and describe what they do:
img-parking-aisle: if path planning believes you are in a parking isle it takes latest frame from all cameras (~2M per camera)
20/

img-sustained-steer-gt8: if you move under 80mps and and moving faster than 8 mps and steering wheel is at 55.75 angle it captures last 6 sec video and latest still frames from fisheye, main and pillar cams - only two samples requested probability 1% 21/

img-sustained-steer-dc: same as above only you are in lat 37.381414..37.417791 lon -122.171607..-122.133067 rectangle

img-sustained-steering: like above but lat 32.734052..41.91059 lon -125.083973..-114.038702, speed between 4 and 12 mps steering above 80deg. 22/

img-mu-estimate: some IMU estimation, 1 sample, collects 9 frames from main camera at 1 fps when car is moving and vehicle_mu_confidence is above 80 with probability 1%.
23/

starfish-ped-bike-motorbike-depth: collects metadata and 3 forward looking cams at 1fps when pedestrians/bikes/... are detected around. 1 data request only.
img-fc-vv-0, img-fc-vv-10 - 1 frame from every camera + metadata, 1 sample each. 24/

that fc-vv stuff has somethingto do with field calibration
depth-sample-lr, depth-sample-rr - only collect some calibration data in case of some depth-percieved objects on any of the cameras. 1 sample each. 25/

occupant-cls-qf: - if you are in gear 4 and the occupant is classified a certain way - collect gps track - one sample

img-lb-weather-rp: if rp camera frame matches a supplied label - collect last right pillar image and coordinates 26/

img-lb-curb: if backup camera maches label collect backup camera latestframe

img-lb-curb-close-rp: if rp image matches label - collect last rp frame

img-lb-over-curb: if backup camera image matches label - colllect lat backup frame

27/

img-lb-over-curb-lr: as above only for left repeater

img-s-rad-hwy-cipv: if following a car and collision warning activated and not on autopilot and speed between 16 and 80 m/s and on controlled access road- collect fisheys, main, and pillar cams 6s video buffer + 1fps 10s buff.

- 3 requests

img-s-rad-local-cipv: as above and we are NOT on controlled acces road - 3 requests

img-vid-don-rand-intervention: 30fps and 10fps video of all cameras if using DoM and disengaged by any method when NOT changing lanes automatically. probability 0.1% - 2 requests

img-vid-don-rand-abort: similar to above - 2 requests (these are VERY large requests at 200+M each)

img-vid-don-ulc-l-diseng: as above only early in ULC left maneuver. 1 request

img-vid-don-ulc-r-diseng: as above only right 1 request 30/

img-vid-don-gore-diseng: DoN aborted and gore zone detected. 2 requests at 0.1% prob.

img-vid-don-steer-diseng: jerking steering wheel above some speed dependent value. 1 request at 0.1% prob. 31/

img-vid-don-brake-diseng: as above only disengage by braking and frontal collision threat asseesnebt if long. acceleration less than 3 m/s^2 1 request at 0.1%

that's the end of the triggers
32/

As you can see - they do collect some elaborate data, but a lot less than many would lead you to believe. And apparently for this reason this knowledge is universally hated by "Bulls" and "bears" alike. 33/

tagging people that touched this topic recently (even though some of them seems to blocked me):

@bobby_tips @FredericLambert @TeslaCharts @brisance @mjd1735 @Paul_M_Huettner @ValueDissenter @quirkyllama @forstall_ @Andreas_Hopf @ContrarianShort @markbspiegel 34/

And no, I have no idea if anybody at Tesla looks at the results, but new triggers do generated frequently and have some variety. No idea how useful the results are either. They are somewhat cautionous with upload sized since storage is not free it looks like.

the data is uploaded to amazon S3 buckets and that bandwidth is not free either I imagine.

Anyway, next time somebody brings the shadow mode or data collection topic - show them this.

Also this is my independent research, I am not am employee of Tesla. Samples available for interested parties.

/FIN