Video-XL (CVPR25) is a really cool paper, which allows to do video understanding (with a VLM) on hour-long videos
The idea is to extract visual tokens (individually from N frames of a video with a visual encoder), and then instead of passing all these tokens ... [1/4]
The idea is to extract visual tokens (individually from N frames of a video with a visual encoder), and then instead of passing all these tokens ... [1/4]
to the LLM (which would blow up the memory if the sequence is too long) they sequentially compress them (with a factor of M) into smaller representations, in the form of KV-cache [2/4]
This means that when you need to create the last compressed visual token, the query needs to attend to N/M KV instead of N
In practice, you sacrifice some parallelization to reduce memory usage
It is probably easier to understand how it works by looking at the figure [3/4]
In practice, you sacrifice some parallelization to reduce memory usage
It is probably easier to understand how it works by looking at the figure [3/4]
And because I like to draw comparisons between vision and NLP, Video-XL is somewhat similar to DeepMind's LLM paper called "Long Context In-Context Compression by Getting to the Gist of Gisting" [4/4]
Video-XL: arxiv.org/abs/2409.14485…
Gist of Gisting: arxiv.org/abs/2504.08934
Video-XL: arxiv.org/abs/2409.14485…
Gist of Gisting: arxiv.org/abs/2504.08934
• • •
Missing some Tweet in this thread? You can try to
force a refresh
