Thread by @cyrildiagne

4/10 - Cut & paste your surroundings to Photoshop

Code: github.com/cyrildiagne/ar…

Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓

#ML #AR #AI #AIUX #Adobe #Photoshop

The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal.

The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service / docker image: github.com/cyrildiagne/ba…

And again, the OpenCV SIFT trick to find where the phone is pointing at the screen.

I also packaged it as a small python library: github.com/cyrildiagne/sc…

Send a camera image + a screenshot and you get accurate x, y screen coordinates!

I wrote the mobile app using @expo as I wanted to try the @reactnative platform. A few rough edges with Android support but the dev workflow is impressively smooth.

Could become an ideal ML interaction design research platform if it could run #TFLite models without ejecting.

Right now, latency is about ~2.5s for cut and ~4s for paste.
There are tons of ways to speed up the whole flow but that weekend just went too fast 🤷🏿‍♂️

Anyway that’s it for now. Let me know if you have any question. Otherwise see you next week for another AI+UX prototype!