Row 7598

Content Data

This page contains data entry 7598 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Raw Record

{
  "text": "(Yes, [the creators already did this](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro), but I wanted to try reproducing their results.)\n\nFirst of all, there's definitely still issues with the dataset.\n\n1. Questions mostly have 10 possible answers, but some have fewer \"to remove unreasonable choices\". This is supposed to\n   be done by actually having <10 answers (I assume), but in some cases questions are padded with N/A at the end\n   multiple times. Removing `N/A` from the end of the options list until only one existed resulted in removing 3314\n   options from the test. AFAIK `N/A` is never the correct answer to any question (I checked).\n2. Some questions are duplicates. I kept only the first instance encountered. This removed 86 questions from the test.\n   I didn't check if their answer options were also duplicates.\n3. Some questions have duplicate answer options (i.e. A and C or whichever being the same answer). As this is\n   obviously incorrect I removed these questions from the test. This removed ~400 questions. It should be pretty easy\n   to fix these but I didn't want to spend more time.\n4. Some questions are impossible to answer as they're missing info.\n     * For example, question ID 2668 is \"Which of the following scans can image brain function?\" but it does not\n       give further details, and the answer options are things like \"I, II and III only\".\n     * I only noticed one of these but I'm pretty sure there are more.\n5. Some questions have very bad formatting, like 11797:\n     * \\textquotedblleftNew business firms can under price older firms, because thenewer firms can buy the latest equipment and are not bur-denedwith the older, less economical equipment which olderfirms must retire before they can economically adopt thenew. This is why continued technological progress contributesto maintaining a competitive economic system.\" Explain the er-rors in both sentences.\n     * Smarter models will probably do better on things like this, but I don't think the formatting is intentional.\n\nFor my first run, I used temperature 0.2 (they used 0.1) and this system prompt:\n\n>You are a knowledgeable expert. When given a multiple choice question, think it through then give your single final answer as `The answer is ...`.\n\nHere is their system prompt:\n\n>You are an knowledge expert, you are supposed to answer the multi-choice question to derive your final answer as `The answer is ...`.\n\nI'm not a fan of the phrasing, but more importantly in my preliminary testing I noticed that GPT-4o interpreted it to mean that it could give multiple answers. My \"single final answer\" was intended to prevent that. This mostly worked, but when answering questions that involved a lot of LaTeX in the CoT it would sometimes format the answer using LaTeX, making regex matching unreasonably hard. (I didn't fix those, scoring them as failures.)\n\nFor my second run, I used temperature 0.5 (which OpenAI uses in their `simple-evals` repo) and this system prompt:\n\n>Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: ($LETTER)' (without quotes) where LETTER is a single one of the options. Think step by step before answering.\n\nThis is based on [this prompt from OpenAI's evals](https://github.com/openai/simple-evals/blob/294cb1fb18f7aed4e21dc567350b0761a9e6f699/common.py#L12-L13). OpenAI's says to pick one of \"ABCD\" but I was too lazy to generate that dynamically so I changed it a bit. It seems to work well. I used the regex `Answer\\s*:\\s*\\(\\$?([A-Ja-j])\\)` to capture the answer.\n\nIf the model's output didn't seem to give an answer at all I manually checked it. This was:\n\n* Mostly GPT-4o saying that none of the answers are correct\n* Sometimes the model entering an infinite loop until it ran out of output space\n* A few times the model giving a numeric answer, often one that had no correspondence to possible answers. Scored as a failure.\n* A few times very slightly bad formatting that my regex didn't match. If the answer it gave was correct I manually fixed it, on the rationale that a slightly better regex could have caught it, otherwise I left it alone (since it wouldn't affect the score either way).\n\nHere are the results I got, plus the original results from TIGER Lab @ UWaterloo.\n\n| Models                 | Prompting | Overall | Biology | Business | Chemistry | ComputerScience | Economics | Engineering | Health | History | Law    | Math   | Philosophy | Physics | Psychology | Other  |\n|:-----------------------|:----------|:--------|:--------|:---------|:----------|:----------------|:----------|-------------|:-------|:--------|:-------|:-------|:-----------|:--------|:-----------|:-------|\n| GPT-4o (TIGER Lab)     | CoT       | 0.7149  | 0.8504  | 0.7852   | 0.7428    | 0.7512          | 0.784     | 0.5401      | 0.7152 | 0.6718  | 0.5393 | 0.762  | 0.6947     | 0.7355  | 0.7726     | 0.7091 |\n| GPT-4o (my first run)  | CoT       | 0.7100  | 0.8441  | 0.7802   | 0.7502    | 0.7404          | 0.7763    | 0.5740      | 0.7061 | 0.6761  | 0.5753 | 0.7901 | 0.6760     | 0.7349  | 0.7611     | 0.7117 |\n| GPT-4o (my second run) | CoT       | 0.7113  | 0.8441  | 0.7725   | 0.7628    | 0.7506          | 0.7662    | 0.5584      | 0.6964 | 0.6641  | 0.5597 | 0.7772 | 0.6474     | 0.7378  | 0.7445     | 0.7041 |\n\nSo I'd say the results themselves are basically \"yep, that sure is how it scores\". I don't know if my overall scores are *actually* slightly worse or if it's the effect of removing almost 500 questions from a 12.3k question dataset.\n\nTotal cost: maybe $160 based on token counts, messed up cost tracking at first.\n\nThe second run had approx 2.17M input tokens and 5.04M output tokens. Naively scaling this to other models (without actually using their tokenizers, and assuming they'll give identical output lengths) gives these approximate prices:\n\n* Claude 3 Opus: $410\n* GPT-4-0314: $367\n* GPT-4-32k-0314: $608\n\nI don't intend to run those.\n\nI'll post the inputs/outputs of both runs if anyone knows a good place to upload password-protected zips (to avoid accidental scraping, since everything posted to reddit is trained on).",
  "label": "r/openai",
  "dataType": "post",
  "communityName": "r/OpenAI",
  "datetime": "2024-05-16",
  "username_encoded": "Z0FBQUFBQm5LakwzdVZKMnAwZ1l5SUd1b2hXYmhFc0hyR1ZvZjB0Vk1Ub0Fvc3JnWW83X3Vvb3BlQkplUU1UclVKLUxLZkVkN0VXb2VfaVBtN0NoWmd4dzFOWmhGN2E5REE9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9IbU5PNlFmcXdTTm9GQzZ0RnllNGpSblc5QjhVcGY3YWVKVHVacnhtMmZWSE1Ea0VkV2hha05SRWx5c1FZTWk5dkFmeUtQZnNhQy1SeXlTa25uODYwQUpQaEE4dkJvelQ3aDZ5TWRWQ213VVBwM2FuMWVoSVEwMjFPdVg1WFR1bENyaU5SSVBIODFqQkZkVnB4Vll0OVZKaGo5RWRFMjFzWWNXTXl3OHNvMmtTalI2X3d0NzFoS1ZJWW9OVWtza3NhNnBMaVkyVnhhTWNYRERmTWplV3FDQT09"
}

Entry Information

• Entry ID: 7598
• Repository: Axioma AXP
• Dataset: arrmlet/reddit_dataset_36
• Total Entries: 100,000