Row 5896

Row ID: 5896 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 5896 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

**Paper**: [https://arxiv.org/abs/2405.00332](https://arxiv.org/abs/2405.00332)

**Abstract**:

>Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission ***Grade School Math 1000*** (**GSM1k**). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with several families of models (e.g., Phi and Mistral) showing evidence of systematic overfitting across almost all model sizes. At the same time, many models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show minimal signs of overfitting. Further analysis suggests a positive relationship (Spearman's *r*^(2)=0.32) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that many models may have partially memorized GSM8k.

FieldValue
text **Paper**: [https://arxiv.org/abs/2405.00332](https://arxiv.org/abs/2405.00332) **Abstract**: >Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission ***Grade School Math 1…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-05
username_encoded Z0FBQUFBQm5LakwyOFJ5UF9RT1lMN1ZYXzViZVI2YUdOcXhJbFB2ZGhXS1g2ZUxoOUlIMU5KWXozcWpYekNIN3RmZDlxcmFwY2FqbVlnak1XWHRDdkhZa18xemplZnYtVGc9PQ==
url_encoded Z0FBQUFBQm5Lak9HNjJTdVRDYlpvV3NJcElJWEJjU2s0QVFTVExqbWJoSHBydk5DaUtMSEc2aUxfRHhtM3U3WVdSa3djWDBBY3pIQjFLQUU0THZGMlNVY1dUYV9xNnJYX1pTc05ZQmhLRUxEQ1E1aHdNSTJic2o1NTVWbkV5T3dJYnRNY25lMFZIZFZGQzRUZ1JRSXVfWEEtbXZaSlhjQjhVYm1XUVpMMmtIeHM1bmFidGJsYTM2M1N3djBkSEhoU2w2bXZaMm1uMVBFVXROMjNUV2taU1JEWmc2YjZsQkRiZz09

Raw Record

{
  "text": "**Paper**: [https://arxiv.org/abs/2405.00332](https://arxiv.org/abs/2405.00332)\n\n**Abstract**:\n\n>Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission ***Grade School Math 1000*** (**GSM1k**). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with several families of models (e.g., Phi and Mistral) showing evidence of systematic overfitting across almost all model sizes. At the same time, many models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show minimal signs of overfitting. Further analysis suggests a positive relationship (Spearman's *r*^(2)=0.32) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that many models may have partially memorized GSM8k.",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-05",
  "username_encoded": "Z0FBQUFBQm5LakwyOFJ5UF9RT1lMN1ZYXzViZVI2YUdOcXhJbFB2ZGhXS1g2ZUxoOUlIMU5KWXozcWpYekNIN3RmZDlxcmFwY2FqbVlnak1XWHRDdkhZa18xemplZnYtVGc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HNjJTdVRDYlpvV3NJcElJWEJjU2s0QVFTVExqbWJoSHBydk5DaUtMSEc2aUxfRHhtM3U3WVdSa3djWDBBY3pIQjFLQUU0THZGMlNVY1dUYV9xNnJYX1pTc05ZQmhLRUxEQ1E1aHdNSTJic2o1NTVWbkV5T3dJYnRNY25lMFZIZFZGQzRUZ1JRSXVfWEEtbXZaSlhjQjhVYm1XUVpMMmtIeHM1bmFidGJsYTM2M1N3djBkSEhoU2w2bXZaMm1uMVBFVXROMjNUV2taU1JEWmc2YjZsQkRiZz09"
}
Explore Dataset Explore Row

Entry Information