Repository: BradyFU/Awesome-Multimodal-Large-Language-Models

Stars: 17662

README.md

Awesome-Multimodal-Large-Language-Models

✨ Highlights of NJU-MiG

🔥🔥 Surveys of MLLMs | 💬 WeChat (MLLM微信交流群)

- 🌟 MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
arXiv 2025, Paper, Project

- 🌟 A Survey of Unified Multimodal Understanding and Generation: Advances and Challenges
arXiv 2025, Paper, Project

- A Survey on Multimodal Large Language Models
NSR 2024, Paper, Project

---

🔥🔥 VITA Series Omni MLLMs | 💬 WeChat (VITA微信交流群)

- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
NeurIPS 2025 Highlight, Paper, Project

- VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting
arXiv 2025, Paper, Project

- VITA: Towards Open-Source Interactive Omni Multimodal LLM
arXiv 2024, Paper, Project

- Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
arXiv 2025, Paper, Project

- VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
NeurIPS 2025, Paper, Project

---

🔥🔥 MME Series MLLM Benchmarks

- 🔥 Video-MME-v2: Towards the Next Stage in Video Understanding Evaluation

<div align='center' > [🍎 Project Page] [📖 Paper] [🤗 Dataset] [🏆 Leaderboard] </div>

- 🌟 MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
arXiv 2025, Paper, Project

- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
NeurIPS 2025 DB Highlight, Paper, Dataset, Eval Tool, ✒️ Citation

- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
CVPR 2025, Paper, Project, Dataset

- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
ICLR 2025, Paper, Project, Dataset

---

<center> Table of Contents </center>
- Awesome Papers
- Multimodal Instruction Tuning (& Latest Works)
- Multimodal Hallucination
- Multimodal In-Context Learning
- Multimodal Chain-of-Thought
- LLM-Aided Visual Reasoning
- Foundation Models
- Evaluation
- Multimodal RLHF
- Others
- Awesome Datasets
- Datasets of Pre-Training for Alignment
- Datasets of Multimodal Instruction Tuning
- Datasets of In-Context Learning
- Datasets of Multimodal Chain-of-Thought
- Datasets of Multimodal RLHF
- Benchmarks for Evaluation
- Others
---

Awesome Papers

Multimodal Instruction Tuning (& Latest Works)

Multimodal Hallucination

Multimodal In-Context Learning

| Title | Venue | Date | Code | Demo |
|:--------|:--------:|:--------:|:--------:|:--------:|
| Visual In-Context Learning for Large Vision-Language Models | arXiv | 2024-02-18 | - | - |
| !Star RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model | RSS | 2024-02-16 | Github | - |
| !Star Can MLLMs Perform Text-to-Image In-Context Learning? | arXiv | 2024-02-02 | Github | - |
| !Star Generative Multimodal Models are In-Context Learners | CVPR | 2023-12-20 | Github | Demo |
| Hijacking Context in Large Multi-modal Models | arXiv | 2023-12-07 | - | - |
| Towards More Unified In-context Visual Understanding | arXiv | 2023-12-05 | - | - |
| !Star MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning | arXiv | 2023-09-14 | Github | Demo |
| !Star Link-Context Learning for Multimodal LLMs | arXiv | 2023-08-15 | Github | Demo |
| !Star OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models | arXiv | 2023-08-02 | Github | Demo |
| !Star Med-Flamingo: a Multimodal Medical Few-shot Learner | arXiv | 2023-07-27 | Github | Local Demo |
| !Star Generative Pretraining in Multimodality | ICLR | 2023-07-11 | Github | Demo |
| AVIS: Autonomous Visual Information Seeking with Large Language Models | arXiv | 2023-06-13 | - | - |
| !Star MIMIC-IT: Multi-Modal In-Context Instruction Tuning | arXiv | 2023-06-08 | Github | Demo |
| !Star Exploring Diverse In-Context Configurations for Image Captioning | NeurIPS | 2023-05-24 | Github | - |
| !Star Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models | arXiv | 2023-04-19 | Github | Demo |
| !Star HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace | arXiv | 2023-03-30 | Github | Demo |
| !Star MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action | arXiv | 2023-03-20 | Github | Demo |
| !Star ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction | ICCV | 2023-03-09 | Github | - |
| !Star Prompting Large Language Models with Answer Heuristics for Knowledge-based Visual Question Answering | CVPR | 2023-03-03 | Github | - |
| !Star Visual Programming: Compositional visual reasoning without training | CVPR | 2022-11-18 | Github | Local Demo |
| !Star An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA | AAAI | 2022-06-28 | Github | - |
| !Star Flamingo: a Visual Language Model for Few-Shot Learning | NeurIPS | 2022-04-29 | Github | Demo |
| Multimodal Few-Shot Learning with Frozen Language Models | NeurIPS | 2021-06-25 | - | - |

Multimodal Chain-of-Thought

| Title | Venue | Date | Code | Demo |
|:--------|:--------:|:--------:|:--------:|:--------:|
| !Star Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models | arXiv | 2024-11-21 | Github | - |
| !Star Cantor: Inspiring Multimodal Chain-of-Thought of MLLM | arXiv | 2024-04-24 | Github | Local Demo |
| !Star Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models | arXiv | 2024-03-25 | Github | Local Demo |
| !Star Compositional Chain-of-Thought Prompting for Large Multimodal Models | CVPR | 2023-11-27 | Github | - |
| !Star DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models | NeurIPS | 2023-10-25 | Github | - |
| !Star Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic | arXiv | 2023-06-27 | Github | Demo |
| !Star Explainable Multimodal Emotion Reasoning | arXiv | 2023-06-27 | Github | - |
| !Star EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought | arXiv | 2023-05-24 | Github | - |
| Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction | arXiv | 2023-05-23 | - | - |
| T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering | arXiv | 2023-05-05 | - | - |
| !Star Caption Anything: Interactive Image Description with Diverse Multimodal Controls | arXiv | 2023-05-04 | Github | Demo |
| Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings | arXiv | 2023-05-03 | Coming soon | - |
| !Star Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models | arXiv | 2023-04-19 | Github | Demo |
| Chain of Thought Prompt Tuning in Vision Language Models | arXiv | 2023-04-16 | [Coming soon]() | - |
| !Star MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action | arXiv | 2023-03-20 | Github | Demo |
| !Star Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models | arXiv | 2023-03-08 | Github | Demo |
| !Star Multimodal Chain-of-Thought Reasoning in Language Models | arXiv | 2023-02-02 | Github | - |
| !Star Visual Programming: Compositional visual reasoning without training | CVPR | 2022-11-18 | Github | Local Demo |
| !Star Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering | NeurIPS | 2022-09-20 | Github | - |

LLM-Aided Visual Reasoning

Foundation Models

| Title | Venue | Date | Code | Demo |
|:--------|:--------:|:--------:|:--------:|:--------:|
| Introducing GPT-5 | OpenAI | 2025-08-07 | - | - |
| !Star VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding | arXiv | 2025-01-22 | Github | Demo |
| !Star Emu3: Next-Token Prediction is All You Need | arXiv | 2024-09-27 | Github | Local Demo |
| Llama 3.2: Revolutionizing edge AI and vision with open, customizable models | Meta | 2024-09-25 | - | Demo |
| Pixtral-12B | Mistral | 2024-09-17 | - | - |
| !Star xGen-MM (BLIP-3): A Family of Open Large Multimodal Models | arXiv | 2024-08-16 | Github | - |
| The Llama 3 Herd of Models | arXiv | 2024-07-31 | - | - |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | arXiv | 2024-05-16 | - | - |
| Hello GPT-4o | OpenAI | 2024-05-13 | - | - |
| The Claude 3 Model Family: Opus, Sonnet, Haiku | Anthropic | 2024-03-04 | - | - |
| Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context | Google | 2024-02-15 | - | - |
| Gemini: A Family of Highly Capable Multimodal Models | Google | 2023-12-06 | - | - |
| Fuyu-8B: A Multimodal Architecture for AI Agents | Blog | 2023-10-17 | Huggingface | Demo
| !Star Unified Model for Image, Video, Audio and Language Tasks | arXiv | 2023-07-30 | Github | Demo |
| PaLI-3 Vision Language Models: Smaller, Faster, Stronger | arXiv | 2023-10-13 | - | - |
| GPT-4V(ision) System Card | OpenAI | 2023-09-25 | - | - |
| !Star Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization | arXiv | 2023-09-09 | Github | - |
| Multimodal Foundation Models: From Specialists to General-Purpose Assistants | arXiv | 2023-09-18 | - | - |
| !Star Bootstrapping Vision-Language Learning with Decoupled Language Pre-training | NeurIPS | 2023-07-13 | Github | - |
| !Star Generative Pretraining in Multimodality | arXiv | 2023-07-11 | Github | Demo |
| !Star Kosmos-2: Grounding Multimodal Large Language Models to the World | arXiv | 2023-06-26 | Github | Demo |
| !Star Transfer Visual Prompt Generator across LLMs | arXiv | 2023-05-02 | Github | Demo |
| GPT-4 Technical Report | arXiv | 2023-03-15 | - | - |
| PaLM-E: An Embodied Multimodal Language Model | arXiv | 2023-03-06 | - | Demo |
| !Star Prismer: A Vision-Language Model with An Ensemble of Experts | arXiv | 2023-03-04 | Github | Demo |
| !Star Language Is Not All You Need: Aligning Perception with Language Models | arXiv | 2023-02-27 | Github | - |
| !Star BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models | arXiv | 2023-01-30 | Github | Demo |
| !Star VIMA: General Robot Manipulation with Multimodal Prompts | ICML | 2022-10-06 | Github | Local Demo |
| !Star MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge | NeurIPS | 2022-06-17 | Github | - |
| !Star Write and Paint: Generative Vision-Language Models are Unified Modal Learners | ICLR | 2022-06-15 | Github | - |
| !Star Language Models are General-Purpose Interfaces | arXiv | 2022-06-13 | Github | - |

Evaluation

Multimodal RLHF

| Title | Venue | Date | Code | Demo |
|:--------|:--------:|:--------:|:--------:|:--------:|
| !Star R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning | arXiv | 2025-05-09 | Github | - |
| !Star Aligning Multimodal LLM with Human Preference: A Survey | arXiv | 2025-03-23 | Github | - |
| !Star MM-RLHF: The Next Step Forward in Multimodal LLM Alignment | arXiv | 2025-02-14 | Github | - |
| Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization | arXiv | 2024-10-09 | - | - |
| !Star Silkie: Preference Distillation for Large Visual Language Models | arXiv | 2023-12-17 | Github | - |
| !Star RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback | arXiv | 2023-12-01 | Github | Demo |
| !Star Aligning Large Multimodal Models with Factually Augmented RLHF | arXiv | 2023-09-25 | Github | Demo |
| !Star RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data | arXiv | 2024-08-22 | Github | - |

Others

| Title | Venue | Date | Code | Demo |
|:--------|:--------:|:--------:|:--------:|:--------:|
| !Star TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models | arXiv | 2024-11-17 | Github | - |
| !Star Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models | arXiv | 2024-02-03 | Github | - |
| !Star VCoder: Versatile Vision Encoders for Multimodal Large Language Models | arXiv | 2023-12-21 | Github | Local Demo |
| !Star Prompt Highlighter: Interactive Control for Multi-Modal LLMs | arXiv | 2023-12-07 | Github | - |
| !Star Planting a SEED of Vision in Large Language Model | arXiv | 2023-07-16 | Github |
| !Star Can Large Pre-trained Models Help Vision Models on Perception Tasks? | arXiv | 2023-06-01 | Github | - |
| !Star Contextual Object Detection with Multimodal Large Language Models | arXiv | 2023-05-29 | Github | Demo |
| !Star Generating Images with Multimodal Language Models | arXiv | 2023-05-26 | Github | - |
| !Star On Evaluating Adversarial Robustness of Large Vision-Language Models | arXiv | 2023-05-26 | Github | - |
| !Star Grounding Language Models to Images for Multimodal Inputs and Outputs | ICML | 2023-01-31 | Github | Demo |

---

Awesome Datasets

Datasets of Pre-Training for Alignment

Datasets of Multimodal Instruction Tuning

| Name | Paper | Link | Notes |
|:-----|:-----:|:----:|:-----:|
| Inst-IT Dataset | Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning | Link | An instruction-tuning dataset which contains fine-grained multi-level annotations for 21k videos and 51k images |
| E.T. Instruct 164K | E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding | Link | An instruction-tuning dataset for time-sensitive video understanding |
| MSQA | Multi-modal Situated Reasoning in 3D Scenes | Link | A large scale dataset for multi-modal situated reasoning in 3D scenes |
| MM-Evol | MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct | Link | An instruction dataset with rich diversity |
| UNK-VQA | UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models | Link | A dataset designed to teach models to refrain from answering unanswerable questions |
| VEGA | VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models | Link | A dataset for enhancing model capabilities in comprehension of interleaved information |
| ALLaVA-4V | ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model | Link | Vision and language caption and instruction dataset generated by GPT4V |
| IDK | Visually Dehallucinative Instruction Generation: Know What You Don't Know | Link | Dehallucinative visual instruction for "I Know" hallucination |
| CAP2QA | Visually Dehallucinative Instruction Generation | Link | Image-aligned visual instruction dataset |
| M3DBench | M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts | Link | A large-scale 3D instruction tuning dataset |
| ViP-LLaVA-Instruct | Making Large Multimodal Models Understand Arbitrary Visual Prompts | Link | A mixture of LLaVA-1.5 instruction data and the region-level visual prompting data |
| LVIS-Instruct4V | To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning | Link | A visual instruction dataset via self-instruction from GPT-4V |
| ComVint | What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning | Link | A synthetic instruction dataset for complex visual reasoning |
| SparklesDialogue | ✨Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models | Link | A machine-generated dialogue dataset tailored for word-level interleaved multi-image and text interactions to augment the conversational competence of instruction-following LLMs across multiple images and dialogue turns. |
| StableLLaVA | StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data | Link | A cheap and effective approach to collect visual instruction tuning data |
| M-HalDetect | Detecting and Preventing Hallucinations in Large Vision Language Models | [Coming soon]() | A dataset used to train and benchmark models for hallucination detection and prevention |
| MGVLID | ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning | - | A high-quality instruction-tuning dataset including image-text and region-text pairs |
| BuboGPT | BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs | Link | A high-quality instruction-tuning dataset including audio-text audio caption data and audio-image-text localization data |
| SVIT | SVIT: Scaling up Visual Instruction Tuning | Link | A large-scale dataset with 4.2M informative visual instruction tuning data, including conversations, detailed descriptions, complex reasoning and referring QAs |
| mPLUG-DocOwl | mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding | Link | An instruction tuning dataset featuring a wide range of visual-text understanding tasks including OCR-free document understanding |
| PF-1M | Visual Instruction Tuning with Polite Flamingo | Link | A collection of 37 vision-language datasets with responses rewritten by Polite Flamingo. |
| ChartLlama | ChartLlama: A Multimodal LLM for Chart Understanding and Generation | Link | A multi-modal instruction-tuning dataset for chart understanding and generation |
| LLaVAR | LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding | Link | A visual instruction-tuning dataset for Text-rich Image Understanding |
| MotionGPT | MotionGPT: Human Motion as a Foreign Language | Link | A instruction-tuning dataset including multiple human motion-related tasks |
| LRV-Instruction | Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning | Link | Visual instruction tuning dataset for addressing hallucination issue |
| Macaw-LLM | Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration | Link | A large-scale multi-modal instruction dataset in terms of multi-turn dialogue |
| LAMM-Dataset | LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark | Link | A comprehensive multi-modal instruction tuning dataset |
| Video-ChatGPT | Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models | Link | 100K high-quality video instruction dataset |
| MIMIC-IT | MIMIC-IT: Multi-Modal In-Context Instruction Tuning | Link | Multimodal in-context instruction tuning |
| M3IT | M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning | Link | Large-scale, broad-coverage multimodal instruction tuning dataset |
| LLaVA-Med | LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day | Coming soon | A large-scale, broad-coverage biomedical instruction-following dataset |
| GPT4Tools | GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction | Link | Tool-related instruction datasets |
| MULTIS | ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst | Coming soon | Multimodal instruction tuning dataset covering 16 multimodal tasks |
| DetGPT | DetGPT: Detect What You Need via Reasoning | Link | Instruction-tuning dataset with 5000 images and around 30000 query-answer pairs|
| PMC-VQA | PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering | Coming soon | Large-scale medical visual question-answering dataset |
| VideoChat | VideoChat: Chat-Centric Video Understanding | Link | Video-centric multimodal instruction dataset |
| X-LLM | X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages | Link | Chinese multimodal instruction dataset |
| LMEye | LMEye: An Interactive Perception Network for Large Language Models | Link | A multi-modal instruction-tuning dataset |
| cc-sbu-align | MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models | Link | Multimodal aligned dataset for improving model's usability and generation's fluency |
| LLaVA-Instruct-150K | Visual Instruction Tuning | Link | Multimodal instruction-following data generated by GPT|
| MultiInstruct | MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning | Link | The first multimodal instruction tuning benchmark dataset |

Datasets of In-Context Learning

| Name | Paper | Link | Notes |
|:-----|:-----:|:----:|:-----:|
| MIC | MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning | Link | A manually constructed instruction tuning dataset including interleaved text-image inputs, inter-related multiple image inputs, and multimodal in-context learning inputs. |
| MIMIC-IT | MIMIC-IT: Multi-Modal In-Context Instruction Tuning | Link | Multimodal in-context instruction dataset|

Datasets of Multimodal Chain-of-Thought

| Name | Paper | Link | Notes |
|:-----|:-----:|:----:|:-----:|
| EMER | Explainable Multimodal Emotion Reasoning | Coming soon | A benchmark dataset for explainable emotion reasoning task |
| EgoCOT | EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought | Coming soon | Large-scale embodied planning dataset |
| VIP | Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction | [Coming soon]() | An inference-time dataset that can be used to evaluate VideoCOT |
| ScienceQA | Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering | Link | Large-scale multi-choice dataset, featuring multimodal science questions and diverse domains |

Datasets of Multimodal RLHF

| Name | Paper | Link | Notes |
|:-----|:-----:|:----:|:-----:|
| VLFeedback | Silkie: Preference Distillation for Large Visual Language Models | Link | A vision-language feedback dataset annotated by AI |

Benchmarks for Evaluation

| Name | Paper | Link | Notes |
|:-----|:-----:|:----:|:-----:|
| Inst-IT Bench | Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning | Link | A benchmark to evaluate fine-grained instance-level understanding in images and videos |
| M3CoT | M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought | Link | A multi-domain, multi-step benchmark for multimodal CoT |
| MMGenBench | MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective | Link | A benchmark that gauges the performance of constructing image-generation prompt given an image |
| MiCEval | MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps | Link | A multimodal CoT benchmark to evaluate MLLMs' reasoning capabilities |
| LiveXiv | LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content | Link | A live benchmark based on arXiv papers |
| TemporalBench | TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models | Link | A benchmark for evaluation of fine-grained temporal understanding |
| OmniBench | OmniBench: Towards The Future of Universal Omni-Language Models | Link | A benchmark that evaluates models' capabilities of processing visual, acoustic, and textual inputs simultaneously |
| MME-RealWorld | MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? | Link | A challenging benchmark that involves real-life scenarios |
| VELOCITI | VELOCITI: Can Video-Language Models Bind Semantic Concepts through Time? | Link | A video benhcmark that evaluates on perception and binding capabilities |
| MMR | Seeing Clearly, Answering Incorrectly: A Multimodal Robustness Benchmark for Evaluating MLLMs on Leading Questions | Link | A benchmark for measuring MLLMs' understanding capability and robustness to leading questions |
| CharXiv | CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs | Link | Chart understanding benchmark curated by human experts |
| Video-MME | Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis | Link | A comprehensive evaluation benchmark of Multi-modal LLMs in video analysis |
| VL-ICL Bench | VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning | Link | A benchmark for M-ICL evaluation, covering a wide spectrum of tasks |
| TempCompass | TempCompass: Do Video LLMs Really Understand Videos? | Link | A benchmark to evaluate the temporal perception ability of Video LLMs |
| GVLQA | GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning | Link | A benchmark for evaluation of graph reasoning capabilities |
| CoBSAT | Can MLLMs Perform Text-to-Image In-Context Learning? | Link | A benchmark for text-to-image ICL |
| VQAv2-IDK | Visually Dehallucinative Instruction Generation: Know What You Don't Know | Link | A benchmark for assessing "I Know" visual hallucination |
| Math-Vision | Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset | Link | A diverse mathematical reasoning benchmark |
| SciMMIR | SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval | Link | | A benchmark for multi-modal information retrieval in the science domain |
| CMMMU | CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark | Link | A Chinese benchmark involving reasoning and knowledge across multiple disciplines |
| MMCBench | Benchmarking Large Multimodal Models against Common Corruptions | Link | A benchmark for examining self-consistency under common corruptions |
| MMVP | Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs | Link | A benchmark for assessing visual capabilities |
| TimeIT | TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding | Link | A video instruction-tuning dataset with timestamp annotations, covering diverse time-sensitive video-understanding tasks. |
| ViP-Bench | Making Large Multimodal Models Understand Arbitrary Visual Prompts | Link | A benchmark for visual prompts |
| M3DBench | M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts | Link | A 3D-centric benchmark |
| Video-Bench | Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models | Link | A benchmark for video-MLLM evaluation |
| Charting-New-Territories | Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs | Link | A benchmark for evaluating geographic and geospatial capabilities |
| MLLM-Bench | MLLM-Bench, Evaluating Multi-modal LLMs using GPT-4V | Link | GPT-4V evaluation with per-sample criteria |
| BenchLMM | BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models | Link | A benchmark for assessment of the robustness against different image styles |
| MMC-Benchmark | MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning | Link | A comprehensive human-annotated benchmark with distinct tasks evaluating reasoning capabilities over charts |
| MVBench | MVBench: A Comprehensive Multi-modal Video Understanding Benchmark | Link | A comprehensive multimodal benchmark for video understanding |
| Bingo | Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges | Link | A benchmark for hallucination evaluation that focuses on two common types |
| MagnifierBench | OtterHD: A High-Resolution Multi-modality Model | Link | A benchmark designed to probe models' ability of fine-grained perception |
| HallusionBench | HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models | Link |An image-context reasoning benchmark for evaluation of hallucination |
| PCA-EVAL | Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond | Link | A benchmark for evaluating multi-domain embodied decision-making. |
| MMHal-Bench | Aligning Large Multimodal Models with Factually Augmented RLHF | Link | A benchmark for hallucination evaluation |
| MathVista | MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models | Link | A benchmark that challenges both visual and math reasoning capabilities |
| SparklesEval | ✨Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models | Link | A GPT-assisted benchmark for quantitatively assessing a model's conversational competence across multiple images and dialogue turns based on three distinct criteria. |
| ISEKAI | Link-Context Learning for Multimodal LLMs | Link | A benchmark comprising exclusively of unseen generated image-label pairs designed for link-context learning |
| M-HalDetect | Detecting and Preventing Hallucinations in Large Vision Language Models | [Coming soon]() | A dataset used to train and benchmark models for hallucination detection and prevention |
| I4 | Empowering Vision-Language Models to Follow Interleaved Vision-Language Instructions | Link | A benchmark to comprehensively evaluate the instruction following ability on complicated interleaved vision-language instructions |
| SciGraphQA | SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs | Link | A large-scale chart-visual question-answering dataset |
| MM-Vet| MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities | Link | An evaluation benchmark that examines large multimodal models on complicated multimodal tasks |
| SEED-Bench | SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension | Link | A benchmark for evaluation of generative comprehension in MLLMs |
| MMBench | MMBench: Is Your Multi-modal Model an All-around Player? | Link | A systematically-designed objective benchmark for robustly evaluating the various abilities of vision-language models|
| Lynx | What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? | Link | A comprehensive evaluation benchmark including both image and video tasks |
| GAVIE | Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning | Link | A benchmark to evaluate the hallucination and instruction following ability |
| MME | MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models | Link | A comprehensive MLLM Evaluation benchmark |
| LVLM-eHub | LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models | Link | An evaluation platform for MLLMs |
| LAMM-Benchmark | LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark | Link | A benchmark for evaluating the quantitative performance of MLLMs on various2D/3D vision tasks |
| M3Exam | M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models | Link | A multilingual, multimodal, multilevel benchmark for evaluating MLLM |
| OwlEval | mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality | Link | Dataset for evaluation on multiple capabilities |

Others

| Name | Paper | Link | Notes |
|:-----|:-----:|:----:|:-----:|
| IMAD | IMAD: IMage-Augmented multi-modal Dialogue | Link | Multimodal dialogue dataset|
| Video-ChatGPT | Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models | Link | A quantitative evaluation framework for video-based dialogue models |
| CLEVR-ATVC | Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation | Link | A synthetic multimodal fine-tuning dataset for learning to reject instructions |
| Fruit-ATVC | Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation | Link | A manually pictured multimodal fine-tuning dataset for learning to reject instructions |
| InfoSeek | Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? | Link | A VQA dataset that focuses on asking information-seeking questions |
| OVEN | Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities | Link | A dataset that focuses on recognizing the Visual Entity on the Wikipedia, from images in the wild |

Repository: BradyFU/Awesome-Multimodal-Large-Language-Models

Stars: 17662

README.md Copy Chapter

Awesome-Multimodal-Large-Language-Models

✨ Highlights of NJU-MiG Copy Chapter

Awesome Papers

Multimodal Instruction Tuning (& Latest Works) Copy Chapter

Multimodal Hallucination Copy Chapter

Multimodal In-Context Learning Copy Chapter

Multimodal Chain-of-Thought Copy Chapter

LLM-Aided Visual Reasoning Copy Chapter

Foundation Models Copy Chapter

Evaluation Copy Chapter

Multimodal RLHF Copy Chapter

Others Copy Chapter

Awesome Datasets

Datasets of Pre-Training for Alignment Copy Chapter

Datasets of Multimodal Instruction Tuning Copy Chapter

Datasets of In-Context Learning Copy Chapter

Datasets of Multimodal Chain-of-Thought Copy Chapter

Datasets of Multimodal RLHF Copy Chapter

Benchmarks for Evaluation Copy Chapter

Others Copy Chapter

README.md

✨ Highlights of NJU-MiG

Multimodal Instruction Tuning (& Latest Works)

Multimodal Hallucination

Multimodal In-Context Learning

Multimodal Chain-of-Thought

LLM-Aided Visual Reasoning

Foundation Models

Evaluation

Multimodal RLHF

Others

Datasets of Pre-Training for Alignment

Datasets of Multimodal Instruction Tuning

Datasets of In-Context Learning

Datasets of Multimodal Chain-of-Thought

Datasets of Multimodal RLHF

Benchmarks for Evaluation

Others