Row 96359

Row ID: 96359 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 96359 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

An interesting part of the paper is an independent replication attempt of another sampling method, [In-Context Pretraining](https://arxiv.org/abs/2310.10638), which relies on semantic similarity of the documents. Back when it was released, it advertised handy gains over random sampling baseline. But the results from the replication are nowhere near that impressive: marginal boost for common NL tasks and somewhat larger gain in long-context-oriented tasks (but the boost still isn't consistent across benchmarks).

Possible explanations, in my view, may boil down to insufficient scale of the replication. The original work trained models up to 7B parameters on 300B tokens while the replication was, if I understood correctly, at 1B-parameters, 100B tokens scale.

A chart in the original work with some downstream performance vs. token count shows that consistent gain over random sampling emerges only after 150B tokens. And in NQ benchmark, In-Context Pretraining even underperforms random baseline up to 100B tokens. Which suggests that models of smaller scale won't benefit from this method (albeit a single benchmark doesn't make this hypothesis super robust).

On the other hand, pplx metric in the original work is consistently better that random baseline regardless of model's scale and dataset size.

Which brings another point: the authors of the Dataset Decomposition paper do not report loss/pplx comparison at all! With the sole exception of training stability experiment where the hyperparameters were artificially tuned to induce instability.

This is isn't the only recently published paper in language modelling that ditches loss-based metrics altogether. In fact, it's become a worrying trend. So a few points to consider.

1. If your method aims at broad enough application, perhaps even universal use, downstream NL benchmarks still do not capture the full spectrum of potential NL tasks and use cases. No one says that loss is an ideal metric from this point of view. But it's still the broadest possible evaluation of model's capacity.

If you aim at specialist models, like code- or math-specialized, then it's totally fine to use only downstream benchmarks. Even then, in code case, I personally would find eval pplx on The Stack more illustrative than some borderline ridiculous pass@1024+maj vote.

2. If you research a novel sampling method, it becomes \_very\_ easy to optimize for downstream benchmarks. The same, to a lesser extent, can be said about novel loss objectives.

3. In most cases the difference in benchmark results vs. baselines/ablation choices is miniscule. In the paper presented, it sometimes drops below half a percent. Throw in the fact that benchmarking is \_extremely\_ sensitive to the choice of examples in few-shot settings (esp. with smaller models), and you are very quickly starting playing games with random noise. Charted pplx over training course is way more honest in this aspect.

FieldValue
text An interesting part of the paper is an independent replication attempt of another sampling method, [In-Context Pretraining](https://arxiv.org/abs/2310.10638), which relies on semantic similarity of the documents. Back when it was released, it advertised handy gains over random sampling baseline. But the results from the replication are nowhere near that impressive: marginal boost for common NL tasks and somewhat larger gain in long-context-oriented tasks (but the boost still isn't consistent ac…
label r/machinelearning
dataType comment
communityName r/MachineLearning
datetime 2024-05-25
username_encoded Z0FBQUFBQm5Lak12dE9QbnZMb0xHVUVLMW5NUXdjeVl1Z0JTWUREYm05ek5YMWZNMzlEeVdRRF9hazFrLW02TGdINXZ6Wm1NU1hvRGNjbXhOY1NZclZERlF5LW9fRTMxQ1h0Z3E1aVE5cGZQSGNnckNLZEEza3M9
url_encoded Z0FBQUFBQm5LalBBV3JiNkhxUmJRYV9xenZpYnhscXEyNHlYcjB3QlcyV2lCM1ZqeGR3VjkxakczbG5DTU1iUGJXNzlBR1Q3WldRQzNGQndkVGdoMm04YUVZOUNyeWo2bEpXU0dBT1hod081emFpQ0dwbkpCNHFKa3RzS1ZRaFhsQU9UclZJZ0lmUHB1Z0hua1VFLVlTNk1uQUNHMmVCV0lTdTV0TUJfU1d0YmpsWEotaWllQ2pFTGtvX1VfTmpzd0lhX2RzSjVQUGc1UFpuaGhRYXlaaE1LdS1nS1ljRFNNUFZjdEhKbEFHLWZ0WlFhUGh2TGgwMD0=

Raw Record

{
  "text": "An interesting part of the paper is an independent replication attempt of another sampling method, [In-Context Pretraining](https://arxiv.org/abs/2310.10638), which relies on semantic similarity of the documents. Back when it was released, it advertised  handy gains over random sampling baseline. But the results from the replication are nowhere near that impressive: marginal boost for common NL tasks and somewhat larger gain in long-context-oriented tasks (but the boost still isn't consistent across benchmarks).\n\nPossible explanations, in my view, may boil down to insufficient scale of the replication. The original work trained models up to 7B parameters on 300B tokens while the replication was, if I understood correctly, at 1B-parameters, 100B tokens scale.\n\nA chart in the original work with some downstream performance vs. token count shows that consistent gain over random sampling emerges only after 150B tokens. And in NQ benchmark, In-Context Pretraining even underperforms random baseline up to 100B tokens. Which suggests that models of smaller scale won't benefit from this method (albeit a single benchmark doesn't make this hypothesis super robust).\n\nOn the other hand, pplx metric in the original work is consistently better that random baseline regardless of model's scale and dataset size.\n\nWhich brings another point: the authors of the Dataset Decomposition paper do not report loss/pplx comparison at all! With the sole exception of training stability experiment where the hyperparameters were artificially tuned to induce instability.\n\nThis is isn't the only recently published paper in language modelling that ditches loss-based metrics altogether. In fact, it's become a worrying trend. So a few points to consider.\n\n1. If your method aims at broad enough application, perhaps even universal use, downstream NL benchmarks still do not capture the full spectrum of potential NL tasks and use cases. No one says that loss is an ideal metric from this point of view. But it's still the broadest possible evaluation of model's capacity.\n\nIf you aim at specialist models, like code- or math-specialized, then it's totally fine to use only downstream benchmarks. Even then, in code case, I personally would find eval pplx on The Stack more illustrative than some borderline ridiculous pass@1024+maj vote.\n\n2. If you research a novel sampling method, it becomes \\_very\\_ easy to optimize for downstream benchmarks. The same, to a lesser extent, can be said about novel loss objectives.\n\n3. In most cases the difference in benchmark results vs. baselines/ablation choices is miniscule. In the paper presented, it sometimes drops below half a percent. Throw in the fact that benchmarking is \\_extremely\\_ sensitive to the choice of examples in few-shot settings (esp. with smaller models), and you are very quickly starting playing games with random noise. Charted pplx over training course is way more honest in this aspect.",
  "label": "r/machinelearning",
  "dataType": "comment",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-25",
  "username_encoded": "Z0FBQUFBQm5Lak12dE9QbnZMb0xHVUVLMW5NUXdjeVl1Z0JTWUREYm05ek5YMWZNMzlEeVdRRF9hazFrLW02TGdINXZ6Wm1NU1hvRGNjbXhOY1NZclZERlF5LW9fRTMxQ1h0Z3E1aVE5cGZQSGNnckNLZEEza3M9",
  "url_encoded": "Z0FBQUFBQm5LalBBV3JiNkhxUmJRYV9xenZpYnhscXEyNHlYcjB3QlcyV2lCM1ZqeGR3VjkxakczbG5DTU1iUGJXNzlBR1Q3WldRQzNGQndkVGdoMm04YUVZOUNyeWo2bEpXU0dBT1hod081emFpQ0dwbkpCNHFKa3RzS1ZRaFhsQU9UclZJZ0lmUHB1Z0hua1VFLVlTNk1uQUNHMmVCV0lTdTV0TUJfU1d0YmpsWEotaWllQ2pFTGtvX1VfTmpzd0lhX2RzSjVQUGc1UFpuaGhRYXlaaE1LdS1nS1ljRFNNUFZjdEhKbEFHLWZ0WlFhUGh2TGgwMD0="
}
Explore Dataset Explore Row

Entry Information