Some simulation platforms that caught my eye this year 🧵 RoboThor from @allen_ai

ai2thor.allenai.org/robothor/chall…

Transformers for point clouds arxiv.org/abs/2012.09164

They outperform all previous methods on semantic segmentation, shape classification and object part segmentation. unlike transformers used in language and image based tasks, the positional encoding is also learned.

finally got around to reading this eccv20 best paper award winner work on optical flow that has interesting ideas: multi-scale 4D correlation volumes, learned-upsampling (using convex weights of lower res pixels), and iterative refinement of flow.

arxiv.org/abs/2003.12039 correlation volumes

starting a thread of interesting python features / modules / libraries that I found over time 🧵

https://twitter.com/ankurhandos/status/1309376964960989184

first up, joblib's memory. If you are reading a huge file Memory class helps you cache that on your disk during the first call. Successive calls load the data much faster. Assuming you didn't change both the function as well as the contents of the file.

This is one of the finest lecture notes in computer vision by Svetlana Lazebnik. Highly recommended to everyone in CV. It mentions the origins, various historic perspectives and anecdotes in CV. Also talks about ethical and societal impacts of CV.

slazebni.cs.illinois.edu/spring20/ How computer vision evolved over the decades.

Some recent interesting work on hand tracking and pose estimation that I liked. Creating a thread 🧵 MEgATrack: Monochrome Egocentric Articulated Hand Tracking for Virtual Reality

research.fb.com/wp-content/upl…

A slower speed of light is a fun 3D game where speed of light slows down progressively as you collect points. While playing, you come across many relativistic effects: Doppler, the searchlight, time dilation, Lorentz transformation and the runtime effect. gamelab.mit.edu/games/a-slower… came across this game while watching this video "What happens as you approach the speed of light?"

Nice use of Q-Networks and MCTS to do scene arrangement. Given an initial layout it learns to find a sequence of moves (actions) that bring it close to the target layout all with a collision free path.

github.com/HanqingWangAI/…

full video: Q-Network used in this work

Impressive and inspiring demos of brain implants in humans by BrainGate some time ago



braingate.org paralyzed woman uses thoughts to sip coffee.

Interesting SIGGRAPH courses that I liked

1. Physics-Based Differentiable Rendering - A Comprehensive Introduction

shuangz.com/courses/pbdr-c… Understanding AR inside and out

dl.acm.org/doi/pdf/10.114…

Impressive low-cost robot that can be built with stuff that costs no more than $50 and uses the smartphone as the main computing platform.

openbot.org

Some of the recent efforts in low-cost robotics in the past few years also include

pyrobot github.com/facebookresear…

SOLO: Segmenting Objects by Locations

arxiv.org/abs/1912.04488

Instead of doing "detect-then-segment" as in Mask R-CNN, they turn the problem into position aware classification via two parallel heads: one that predicts the category and the other, the instance mask. The grid S x S could be thought of as a bounding box that you slide across the image.

Good survey chapter from Vincent Lepetit.

arxiv.org/abs/2006.05927 and conclusions

GRAB: A Dataset of Whole-Body Human Grasping of Objects ecva.net/papers/eccv_20…

An interesting dataset where they record human and object data with MoCap and fit models of body shape and pose, and the articulated face and hands, and the 3D object pose.

grab.is.tue.mpg.de some examples

"Can a biologist fix a radio?" is a classic and entertaining read.

cell.com/cancer-cell/pd… written in similar spirit is this paper "Could a Neuroscientist Understand a Microprocessor?"

ncbi.nlm.nih.gov/pmc/articles/P…

Ishikawa lab from toyko has amazing high speed tracking and control systems. They focus more on optimising hardware so that simple algorithms can run at high speeds. I just visited their website few days and found this list of cool videos

running
card throwing and shooting by high speed vision system and multi-fingered hand.

ContactPose is a dataset of registered hand poses with images from 3 viewpoints with the interesting part of recordings from thermal cameras as well to get the contact traces of human hand on the objects. (from @samarth_robo)

arxiv.org/abs/2007.09545

contactpose.cc.gatech.edu The studio for capturing hand pose and the thermal camera used in this work.

I liked this work on model-based tracking (for hand) for HoloLens2 using lifted optimisation(where search for correspondence and model paramters is done jointly) and a phong surface model which allows smooth surface normal and vertex interpolation.

arxiv.org/pdf/2007.04940… Traditional ICP based methods solve for correspondence first and then model parameters. Here they do it jointly on the surface triangle mesh.

If you want to regress to rotations with neural networks, they suggest to regress to 10 params of 4x4 symmetric matrix and use a differentiable quadratic program layer to recover quaternions rather than directly regressing.

roboticsproceedings.org/rss16/p007.pdf The link to quadratic program comes from the well known Wahba problem.

dyna-kinematics is the simplest and the most basic physics engine with examples on various components involved in implementing a physics engine.

github.com/diegomacario/D… This summarises how it is implemented and the assumptions used.

This is a great tutorial by Adam Bargteil and Tamar Shinar at siggraph last year that covers various elements of physics engines from paratermisations, integration schemes and collisions and constraints.

dl.acm.org/doi/pdf/10.114… this is a non-acm pdf link cs.ucr.edu/~shinar/papers…

as I realised that acm link is open only until the end of this month.