Thread by @LiJunnan0409

A new member in the BLIP family: 🔥InstructBLIP🔥, a vision-language instruction tuning framework. InstructBLIP achieves SoTA zero-shot performance with various advantages over other multimodal models such as GPT-4!
Github: github.com/salesforce/LAV…
Paper: arxiv.org/abs/2305.06500

Our paper conducts a systematic study on vision-language instruction tuning. InstructBLIP substantially outperforms both BLIP-2 and the largest Flamingo on zero-shot evaluation. It also has SOTA finetuning performance when used as the model initialization on downstream tasks.

In addition, we introduce instruction-aware visual feature extraction, a new method that enables the model to extract informative features tailored to the given instruction, leading to enhanced generalization performance.

We open-source a suite of InstructBLIP models using two family of LLMs: FlanT5 and Vicuna. Using our LAVIS library, you can run these models with two lines of code!
github.com/salesforce/LAV…

Great work from our intern @Wenliang_Dai at @SFResearch and collaborators @DongxuLi_ @AlbertBoyangLi @stevenhoi!

InstructBLIP demonstrates a variety of strong multimodal capabilities including complex visual scene understanding and reasoning, knowledge-grounded image description, multi-turn visual conversation, etc. Checkout this demo video!

InstructBLIP demonstrates strong visual reasoning capability of complex scenes, generalizing beyond its training data to OOD images.

#instructblip with #midjourney images