Could you tell which room was virtual? 👀

Traditional AR can’t tell you how a couch feels. Our team @Shopify explored "Reverse AR" to wrap a virtual room around a real thing. Sit on a couch in-store while seeing it in your space—best of both worlds!

👇 How we built it… 🧵1/12
At home, shoppers scan their space using @Apple’s #RoomPlan or a photogrammetry app like @RealityComposer_ or @Polycam3D. It only takes a few seconds. 🧵2/12
@Apple @Polycam3D In store, shoppers can instantly visualize their room around an object. Computer vision systems identify and segment furniture and people from the video stream and mask them into the virtual room. 🧵3/12
@Apple @Polycam3D Multiplayer allows two connected shoppers to share the same preview, making it easier to collab. Hand and gesture tracking can be used to adjust the orientation and position of the room, creating some mind-bending moments. 🧵4/12
@Apple @Polycam3D First we needed some way to identify and segment furniture from a live camera feed. We played with CV frameworks like YOLOv3 and DeepLabv3, but the segmentation quality didn’t pass the bar for us. 🧵5/12
@Apple @Polycam3D In parallel, we explored RealityKit’s implementation, trying to figure out how @Apple got their segmentation so clean. It looked to us like they were leveraging meshes constructed from LiDAR depth data. 🧵6/12
@Apple @Polycam3D We built a quick demo that generates a mesh when it detects a chair object. Mesh-based segmentation was passable for some furniture pieces, but we weren’t stoked by the effect on many common styles. 🧵7/12
@Apple @Polycam3D We discovered that simple depth-based segmentation provided a much cleaner edge but it didn’t work in many scenarios. To generalize it we’d need to find a way to segment the depth data. 🧵8/12
@Apple @Polycam3D That’s when we realized that we could use depth-buffer segmentation to more cleanly cut objects out of the scene if we had a bounding box to constrain the depth buffer to world space. 🧵9/12
@Apple @Polycam3D After quickly setting up the bounding box, the mask shape can be refreshed every frame from the depth data and produces a fast, clean segmentation. 🧵10/12
@Apple @Polycam3D Now we can apply shader effects for a magical reveal. To position the furniture simply drag the room model around with your finger or use two-fingers to rotate it. 🧵11/12
@Apple @Polycam3D Reverse AR unlocks fun in-store opportunities. In addition to shopper’s homes, merchants could transport shoppers and products anywhere: a Paris flat, a serene mountain cabin in Japan, or next to an infinity pool looking over a desert oasis. 🧵12/12 ✌️
Huge thanks to and, two fantastic @Shopify merchants for providing such beautiful shops for our shoots 🙏🙌

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Russ Maschmeyer

Russ Maschmeyer Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @StrangeNative

Jul 13
Lots of interest in this “reset button” for your room we built @Shopify that uses @apple’s new #RoomPlan API. Here’s how we built it… 👇 🧵1/12
@Shopify @Apple AR makes it easy to try new couches, but the couch already in your space can get in the way. What if every room had a reset button? Space Eraser models spaces in high-fidelity—capturing room-defining objects, their size, position, orientation… [2/12]
@Shopify @Apple Live pixel data combines with LiDAR depth data to enable a lifelike digital twin of your room that can be overlaid onto your real space using AR and then edited digitally… [3/12]
Read 12 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!


0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy


3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!