Row 7482

Row ID: 7482 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 7482 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Hey everyone!

I'm excited to share our latest work on **MileBench**, a new benchmark designed to evaluate the performance of Multimodal Large Language Models (MLLMs) in long-context tasks involving multiple images and lengthy texts.

**Title:** MileBench: Benchmarking MLLMs in Long Context

**Homepage:** [https://milebench.github.io/](https://milebench.github.io/)

**Paper:** [https://arxiv.org/abs/2404.18532](https://arxiv.org/abs/2404.18532)

**Code:** [https://github.com/MileBench/MileBench](https://github.com/MileBench/MileBench)

**Data:** [https://huggingface.co/datasets/FreedomIntelligence/MileBench](https://huggingface.co/datasets/FreedomIntelligence/MileBench)

# Why MileBench?

Existing benchmarks often overlook the complexity of tasks that involve multimodal long contexts. MileBench is the first to focus on these challenging scenarios, offering a more realistic assessment of MLLMs.

# Evaluation Types:

* Diagnostic Evaluation: Tests recall in long contexts with needle-in-a-haystack and image retrieval tasks. * Realistic Evaluation: Simulates real-world scenarios with time-sequence and semantically related image tasks.

We collected **6,440** multimodal long-context samples from **21** existing or self-constructed datasets, with an average of **15.2** images and **422.3** words per sample. The table and figure show the detailed statistics of our datasets.

# Key Findings:

* Closed-source **GPT-4o** excelled in both diagnostic and realistic evaluations, but still short of perfect 100%. * Most open-source MLLMs struggled with long-context tasks. Only Mantis and Qwen-VL-7B managed notable scores.

These results underscore that there are **"miles to go"** towards fully-realized long-context MLLMs.

# In-depth Analyses:

**Analysis 1: How do MLLMs perform with different context lengths?**

* Most models' performance drops as the number of images increases. * Some models, like GPT-4o, perform better with a medium number of images.

**Analysis 2: Is there a "Lost in the Middle" Phenomenon in Long Contexts?**

* Strong long-text processing capabilities are crucial * Qwen-VL-Chat showed some "Lost in the Middle" effects.

**Analysis 3: Combining Images Helps?**

To address input limitations, we combined multiple images into a single large image.

* Closed-source models performance dropped when combining images, except for Gemini 1.0 * High-resolution capabilities are crucial * Some open-source models showed improved performance with combined images

# Want to Dive Deeper?

Our paper includes detailed experimental analyses, covering data contamination issues and task diversity. Check it out here: [https://arxiv.org/abs/2404.18532](https://arxiv.org/abs/2404.18532)

# Looking Ahead:

* Expanding MileBench to larger contexts and other modalities. * Developing MLLMs that can efficiently handle complex, multimodal long-context tasks.

For more details, visit our project page: [https://milebench.github.io](https://milebench.github.io)

Let’s discuss and explore how we can push the boundaries of MLLMs together!

FieldValue
text Hey everyone! I'm excited to share our latest work on **MileBench**, a new benchmark designed to evaluate the performance of Multimodal Large Language Models (MLLMs) in long-context tasks involving multiple images and lengthy texts. **Title:** MileBench: Benchmarking MLLMs in Long Context **Homepage:** [https://milebench.github.io/](https://milebench.github.io/) **Paper:** [https://arxiv.org/abs/2404.18532](https://arxiv.org/abs/2404.18532) **Code:** [https://github.com/MileBench/MileBench]…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-16
username_encoded Z0FBQUFBQm5LakwzcTYzeUtlYzUyQTNaUUdKSEZNUXJGdGxqVUxPMC1RR2FHLXA1ZVFCY21nSllaU1M4Tm1yM2sza0JSM3hmaGp4TFF4Zllyc1V6Yy15OEI4eWxRaUdqM0E9PQ==
url_encoded Z0FBQUFBQm5Lak9ISkFpajRGOTZEMVhRRTZXREdNckZZYlNwMF9TcXltU0tKSWM5UTRncDBKYnBUdFp0NjE1MUh2WmJEZWRLUjBTbHNaOS1qVlFYMmp1Yi1jM2pXTUZLUS1yMkxJMnkwYjIyaVVCT3VpSDVFZnJOUUo5V1RFUVN4Nm5fU3lpVWYxTWZhNlMxb1dCbjdPZDhWLXljUDJWZWtNWlozd1pTNE1ZLU9xM1RFZlZ6bG1jYTdyM0ZLak0yM0NGZVNBcUV3dlcwTUktVnRpUU9LSnZPNk9rZjduQ1FvQT09

Raw Record

{
  "text": "Hey everyone!\n\nI'm excited to share our latest work on **MileBench**, a new benchmark designed to evaluate the performance of Multimodal Large Language Models (MLLMs) in long-context tasks involving multiple images and lengthy texts.\n\n**Title:** MileBench: Benchmarking MLLMs in Long Context\n\n**Homepage:** [https://milebench.github.io/](https://milebench.github.io/)\n\n**Paper:** [https://arxiv.org/abs/2404.18532](https://arxiv.org/abs/2404.18532)\n\n**Code:** [https://github.com/MileBench/MileBench](https://github.com/MileBench/MileBench)\n\n**Data:** [https://huggingface.co/datasets/FreedomIntelligence/MileBench](https://huggingface.co/datasets/FreedomIntelligence/MileBench)\n\n# Why MileBench?\n\nExisting benchmarks often overlook the complexity of tasks that involve multimodal long contexts. MileBench is the first to focus on these challenging scenarios, offering a more realistic assessment of MLLMs.\n\n# Evaluation Types:\n\n* Diagnostic Evaluation: Tests recall in long contexts with needle-in-a-haystack and image retrieval tasks.\n* Realistic Evaluation: Simulates real-world scenarios with time-sequence and semantically related image tasks.\n\nWe collected **6,440** multimodal long-context samples from **21** existing or self-constructed datasets, with an average of **15.2** images and **422.3** words per sample. The table and figure show the detailed statistics of our datasets.\n\n# Key Findings:\n\n* Closed-source **GPT-4o** excelled in both diagnostic and realistic evaluations, but still short of perfect 100%.\n* Most open-source MLLMs struggled with long-context tasks. Only Mantis and Qwen-VL-7B managed notable scores.\n\nThese results underscore that there are **\"miles to go\"** towards fully-realized long-context MLLMs.\n\n# In-depth Analyses:\n\n**Analysis 1: How do MLLMs perform with different context lengths?**\n\n* Most models' performance drops as the number of images increases.\n* Some models, like GPT-4o, perform better with a medium number of images.\n\n**Analysis 2: Is there a \"Lost in the Middle\" Phenomenon in Long Contexts?**\n\n* Strong long-text processing capabilities are crucial\n* Qwen-VL-Chat showed some \"Lost in the Middle\" effects.\n\n**Analysis 3: Combining Images Helps?**\n\nTo address input limitations, we combined multiple images into a single large image.\n\n* Closed-source models performance dropped when combining images, except for Gemini 1.0\n* High-resolution capabilities are crucial\n* Some open-source models showed improved performance with combined images\n\n# Want to Dive Deeper?\n\nOur paper includes detailed experimental analyses, covering data contamination issues and task diversity. Check it out here: [https://arxiv.org/abs/2404.18532](https://arxiv.org/abs/2404.18532)\n\n# Looking Ahead:\n\n* Expanding MileBench to larger contexts and other modalities.\n* Developing MLLMs that can efficiently handle complex, multimodal long-context tasks.\n\nFor more details, visit our project page: [https://milebench.github.io](https://milebench.github.io)\n\nLet’s discuss and explore how we can push the boundaries of MLLMs together!",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-16",
  "username_encoded": "Z0FBQUFBQm5LakwzcTYzeUtlYzUyQTNaUUdKSEZNUXJGdGxqVUxPMC1RR2FHLXA1ZVFCY21nSllaU1M4Tm1yM2sza0JSM3hmaGp4TFF4Zllyc1V6Yy15OEI4eWxRaUdqM0E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9ISkFpajRGOTZEMVhRRTZXREdNckZZYlNwMF9TcXltU0tKSWM5UTRncDBKYnBUdFp0NjE1MUh2WmJEZWRLUjBTbHNaOS1qVlFYMmp1Yi1jM2pXTUZLUS1yMkxJMnkwYjIyaVVCT3VpSDVFZnJOUUo5V1RFUVN4Nm5fU3lpVWYxTWZhNlMxb1dCbjdPZDhWLXljUDJWZWtNWlozd1pTNE1ZLU9xM1RFZlZ6bG1jYTdyM0ZLak0yM0NGZVNBcUV3dlcwTUktVnRpUU9LSnZPNk9rZjduQ1FvQT09"
}
Explore Dataset Explore Row

Entry Information