Jie Lei Profile picture
21 Jul, 6 tweets, 3 min read
Presenting "QVHighlights": 10K YouTube videos dataset annotated w. human written queries, clip-wise relevance, highlightness/saliency scores & Moment-DETR model for joint moment localization + highlight/saliency prediction


tlberg @mohitban47 @uncnlp

Detecting customized moments & highlight saliencies from videos given NL user queries is an important but under-studied topic. One of the challenges in pursuing this direction is lack of annotated data. Thus, we present Query-based Video Highlights (QVHighlights) dataset!

10K YouTube videos, covering everyday activities+travel in lifestyle vlog videos to social+political activities in news. Each video is annotated w. (1) human free-form NL query (2) relevant moments in video wrt query (3) 5-point scale saliency scores for all query-relevant clips
We also present a strong baseline Moment-DETR: transformer enc-dec model that views moment retrieval as a direct set prediction problem, taking extracted video &query representations (SlowFast, CLIP) as inputs and predicting moment coordinates and saliency scores end-to-end

While our model does not utilize any human prior, we show that it performs competitively wrt well-engineered architectures. With weakly supervised pretraining using ASR captions, Moment-DETR strongly outperforms previous methods. We also present several ablations+visualizations.
Data and code is publicly available at: github.com/jayleicn/momen…

Our evaluation is hosted on CodaLab, come try your latest video+language models on this new important+challenging task!

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Jie Lei

Jie Lei Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!