Discover and read the best of Twitter Threads about #instructblip
Most recents (2)
Introducing š„InstructBLIPš„ - our new Multimodal Foundation Models with instruction tuning on BLIP2, achieving new SOTA results on various VL benchmarks and enjoying various advantages over GPT-4.
Paper: arxiv.org/abs/2305.06500
Code: github.com/salesforce/LAVā¦
(1/n)
Paper: arxiv.org/abs/2305.06500
Code: github.com/salesforce/LAVā¦
(1/n)
InstructBLIP unlocks a range of diverse multimodal capabilities for building next-generation AI agents, including complex visual scene understanding and reasoning, knowledge-grounded image description, multi-turn visual conversation, etc.
(2/n)
(2/n)
Built on the success of #BLIP2, InstructBLIP proposes a general instruction-tuning framework, where Q-Former extracts instruction-aware visual features from output embeddings of frozen image encoder, and feeds the visual features as soft prompt input to the frozen LLM.
(3/n)
(3/n)
A new member in the BLIP family: š„InstructBLIPš„, a vision-language instruction tuning framework. InstructBLIP achieves SoTA zero-shot performance with various advantages over other multimodal models such as GPT-4!
Github: github.com/salesforce/LAVā¦
Paper: arxiv.org/abs/2305.06500
Github: github.com/salesforce/LAVā¦
Paper: arxiv.org/abs/2305.06500
Our paper conducts a systematic study on vision-language instruction tuning. InstructBLIP substantially outperforms both BLIP-2 and the largest Flamingo on zero-shot evaluation. It also has SOTA finetuning performance when used as the model initialization on downstream tasks.
In addition, we introduce instruction-aware visual feature extraction, a new method that enables the model to extract informative features tailored to the given instruction, leading to enhanced generalization performance.
