Thread by @braddwyer

In SF at the @TechCrunch #TCDisrupt hackathon with @josephofiowa this week.

I’ll be live tweeting what we build for the next ~48 hours. Smash that subscribe button and buckle in!

We settled in at table 51 in the hackathon area if you’re around and want to stop by!

We’re planning to build a computer vision powered Chess utility that uses your phone’s camera to understand the state of the board and add software features to real-world Chess!

First step: collecting training data for our machine learning model.

Opening ceremonies currently underway. Hearing about the sponsor prizes... wonder if there's a way we can frame Chess as fintech or healthcare related 🧐

We have our architecture planned!

The iPhone will stay on the tripod running a ML model to determine the game state & relay the info to the cloud (we hope to let it be handheld in a future version).

A server will pass state to @stockfishchess.

And an iPad will display the UI.

@stockfishchess Testing with one of Apple's sample projects to see if we can get any functionality out of the built in Vision ML models in iOS13.

Hoping possibly to use the built-in rectangle detection to isolate the board and crop out the rest of the photo.

developer.apple.com/documentation/…

@stockfishchess BINGO! The built-in rectangle detection model is passable for isolating the board. That means we can focus on training a model for detecting which pieces are where.

@stockfishchess ✅ Repo created, firebase project created.

Joseph is going to keep working on collecting and labeling chess board data and then training the ML model.

I'm working on spinning up a server and interfacing with the StockFish chess engine.

@stockfishchess Oh nice, there's already a JS port of StockFish!

Going to see if using stockfish.js is viable instead of communicating with the mainline engine over stdin/out. github.com/nmrugg/stockfi…

I pity @josephofiowa for having to label all this training data he just collected 😂

Wonder how long before he gives up and farms it out to Mechanical Turk.

@josephofiowa Everything seems to be going swimmingly so far... (which from past experience means everything's probably about to blow up)

Stockfish.js is working great (once I read the UCI protocol's spec). After I wire it up to Firebase the server part will be done! wbec-ridderkerk.nl/html/UCIProtoc…

@josephofiowa Boom! The server is streaming Stockfish's board analysis results to Firebase.

Moving on to consuming these updates with a web UI. Going to look and see if there are any prebuilt tools for visualizing a chess board (via FEN notation) in HTML.

@josephofiowa Aaaaaand @josephofiowa is still labeling images 😂

@josephofiowa Live view of Joseph's screen from now until the end of time.

@josephofiowa After digging into a bunch of options, I think I'm going to go with chessboard.js for the visualization. With a little bit of hacking I think it should work well for what we need!

Have a basic frontend UI responding to updates streaming in from Stockfish.

Next I want to add an arrow between the squares to make it more clear which move is being recommended.

Well that was easier than I thought it was going to be. I just used LeaderLine... but I need to figure how to tell it to take the straightest path instead of trying to get fancy. anseki.github.io/leader-line/

Got that fixed now. Joseph is going to send me some labeled images and I'm going to run them through CreateML so I have a (rudimentary) mlmodel with which to start building the iPhone app. developer.apple.com/machine-learni…

It’s training! 🤞

The results of our first quick and dirty model training run are... not great. But also kind of promising.

I'm collecting more data... @josephofiowa is looking into Mechanical Turk.

Things are just not going @josephofiowa’s way today. After over an hour of work setting up a Mechanical Turk task we find out that Amazon SageMaker Ground Truth can only support a maximum of 10 labels (???) but there are 12 types of chess pieces.

After training the model with more data it did over 2x better but still wasn’t giving good answers.

Our working theory is that because the model’s input size is so small (417x417) and the frame is so zoomed out that after resizing the pieces are too small to differentiate.

I wrote a script to crop the images so that we waste less of the model’s capacity processing irrelevant background pixels.

I’m leaving my laptop to train on the cropped images overnight but after 10 minutes of training it’s already beating our best uncropped model!

After an hour of training it’s performing 2x better than the model I mentioned a little while ago (that was 2x better than our first attempt).

I’m letting this one train for quite a bit longer to see where it lands.

Hopefully it will be good enough to start building the app around it while @josephofiowa refines his custom @TensorFlow model.

@josephofiowa @TensorFlow Checking on a model you left training overnight feels a bit like being a kid waking up on Christmas morning. And I must have been a good boy this year!

Got another 50% improvement. It's definitely not perfect but it'll do! The new model looking at an image it hasn't seen before:

@josephofiowa @TensorFlow WOW, @rectlabel can use our existing model to automatically annotate new images we add! Then we just have to correct the mistakes rather than starting completely from scratch.

This is a game changer. It will make labeling additional training data so much less time consuming!

@josephofiowa @TensorFlow @rectlabel The biggest problem the model is having right now is with pieces being occluded behind others. Still trying to figure out the best way to address this. Any ideas?

It’s working! Real-time output of our machine learning model trained at the #DisruptSF hackathon.

Next we have to correlate these bounding boxes with their position on the board. Then we can pass the bord state to the chess engine and UI that I built yesterday!

Finally got visualization of the predicted pieces working... really starting to see the inadequacies of this model.

Hopefully @josephofiowa comes to the rescue with his custom model soon!

@josephofiowa

https://twitter.com/josephofiowa/status/1179906244246568960?s=20

The mishmash of coordinate systems in this project is making my brain explode.

Fixed a major issue where things wouldn't work unless the board was directly centered in the frame though!

Now you can even move the camera around and it will keep tracking and processing:

I got the calculations working to determine where a piece is on the A-H axis! (Check out the representation on the iPad as I move the rook around)

Once I get the formula right for the 1-8 axis it’ll be working end to end! Then just need to plug in the more accurate ML model.

Moscone kicks everyone out at 9pm... calling it quits for the night so soon isn't going to work for us. Too much left to do!

Luckily @josephofiowa is tight with @GA so we're camping out at their SF campus for a bit. Thanks @jakeschwartz!

@josephofiowa @GA @jakeschwartz Aaaaand submitted. (With only 5 minutes to spare 😱)

It's not perfect but I'm pretty proud of what we were able to do in such a short amount of time at the #DisruptSF hackathon! devpost.com/software/chess…

@josephofiowa @GA @jakeschwartz Ready to demo for the judges! If you're at @Techcrunch #DisruptSF, we're at table 20. Come on by.

@josephofiowa @GA @jakeschwartz @TechCrunch No big deal, just @josephofiowa and I demoing our smart chess board to Snapchat CEO @evanspiegel.

Well... we didn’t make the finals. So that’s it for us. Had fun, learned a lot. Thanks for tuning in!

If you’re interested in the future of enhancing board games with computer vision sign up for the beta of our new app, BoardBoss, launching soon on iOS!

boardboss.com