Lots of interest in this “reset button” for your room we built @Shopify that uses @apple’s new #RoomPlan API. Here’s how we built it… 👇 🧵1/12
@Shopify @Apple AR makes it easy to try new couches, but the couch already in your space can get in the way. What if every room had a reset button? Space Eraser models spaces in high-fidelity—capturing room-defining objects, their size, position, orientation… [2/12]
@Shopify @Apple Live pixel data combines with LiDAR depth data to enable a lifelike digital twin of your room that can be overlaid onto your real space using AR and then edited digitally… [3/12]
@Shopify @Apple With an empty room, shoppers can now swipe through room sets which leverage the orientation of existing furniture to arrange themselves automatically… [4/12]
@Shopify @Apple With a digital twin of your room and your furniture you could simply swipe your original sofa away to explore options that better match the rest of the room… [5/12]
@Shopify @Apple To determine if this would be possible today we explored two questions: how do we align the model produced by #RoomPlan with your room and how do we texture the model? [6/12]
@Shopify @Apple #RoomPlan outputs an untextured USDZ model composed of unit cubes; one for each door, window, wall, and room-defining object—but, surprise! Models don’t contain floor or ceiling elements. We align the model with the real world using ARWorldMap. [7/12]
@Shopify @Apple Since RoomPlan exports untextured models, we explored ways we might texture it automatically. The best way would be to localize and map the texture when capturing, but we make do with an aligned model and a real-time camera feed. [8/12]
@Shopify @Apple We reversed the graphics pipeline: instead of sampling from a texture and writing to the screen, we sample from the camera and write to a texture. RealityKit made this hard, so we used SceneKit instead. [9/12]
@Shopify @Apple This approach creates issues when furniture obscures a surface. In video above 👆, our couch obscured the wall. #RoomPlan provides bounding box references for room-defining objects. With this we can determine what parts of the texture to redact. [10/12]
@Shopify @Apple We can now use inpainting techniques to fill in the redacted texture. We were pleasantly surprised by the results produced by off-the-shelf tools like Photoshop and theinpaint.com. [11/12]
@Shopify @Apple AR and #RoomPlan are exciting tools for commerce. The ability to select and remove real objects from a scene brings us one step closer to a fully editable reality. Speaking of… time to bring back the dog ✌️[12/12]

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Russ Maschmeyer

Russ Maschmeyer Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!


0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy


3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!