@aelluswamy's talk at CPVR has a lot of very impressive improvements to Tesla's 3D voxel models. There's some subtle but very important things in the slides that I'm excited to incorporate into my own models. ⬇️

https://twitter.com/aelluswamy/status/1561151207858573312

1) Image positional encoding: This adds in an x/y position encoding to each of the image space features. This should make it easier for the transformer to go from image space to 3D

It seems like a hybrid between a traditional CNN and ViT

Curious what I've been up to in the past 6 months? 😅

I've been working on a novel approach to depth and occupancy understanding for my FSD models!

It's much simpler than existing techniques and directly learns the 3D representation ⬇️ I posted the full write up on my about a month ago and I've had a number of PhD students, companies and labs ask to collaborate on papers/projects so I think it's state of the art 🙂

I haven't seen any papers on this

Full write up: fn.lc/post/voxel-sfm/

Is the Tesla repeater light bleed a problem? I grabbed some captures from a 2020 Model 3 to find out

Here's some of the raw 10-bit captures and my analysis ⬇️

Thanks to @greentheonly for suggesting this! When looking at this data there's two main things to consider: the static world around the vehicle and the dynamic objects in the scene such as cars or people

For static objects information from the forward facing cameras can compensate for lack of info on the repeaters

I spent some time over my 2 week holiday creating my own self driving models from the ground up in PyTorch 🙂

Open source self driving anyone?

Check out the full write up at: fn.lc/post/diy-self-…

I'll be summarizing it below ⬇️ 1/n These were trained from the raw footage without any Tesla NNs or outputs. Makes it more fun this way and a lot more possible to iterate

I built everything here using just using 5 of the 8 cameras and the vehicle speed, steering wheel position and IMU readings

Curious what Tesla means by upreving their static obstacle neural nets?

Lets see how the Tesla FSD Beta 10.5 3D Voxel nets compare to the nets from two months ago.

The new captures are from the same area as the old ones so we can directly compare the outputs

1/N This first example is a small pedestrian crosswalk sign in the middle of the road. It's about 1 foot wide so it should show up as 1 pixel in the nets.

Under the old nets it shows up as a large blob with an incorrect depth. Under the new nets it's much better.

Tesla has added new voxel 3D birdseye view outputs and it's pretty amazing!

Nice of them to start merging some bits of FSD into the normal firmware in 2021.36 so we can play with the perception side 🙂

Thanks to @greentheonly for the help! Most of the critical FSD bits are missing in the normal firmware. These outputs aren't normally running but with some tricks we can enable it.

This seems to be the general solution to handling unpredictable scenarios such as the Seattle monorail pillars or overhanging shrubbery.

We recently got some insight into how Tesla is going to replace radar in the recent firmware updates + some nifty ML model techniques

⬇️ Thread From the binaries we can see that they've added velocity and acceleration outputs. These predictions in addition to the existing xyz outputs give much of the same information that radar traditionally provides
(distance + velocity + acceleration).

Got a sample of the Tesla Insurance telemetry data. The insurance records are on a per drive basis. Here's the fields:

* Unique Drive ID
* Record Version
* Car Firmware Version
* Driver Profile Name
* Start / End Time
* Drive Duration
* Start / End Odometer

(1/2) * # of Autopilot Strikeouts
* # of Forward Collision Warnings
* # of Lane Departure Warnings
* # of ABS activations (All & User)
* Time spent within 1s of car in front
* Time spent within 3s of car in front
* Acceleration Variance
* Service Mode
* Delivered

(2/2)