Row 7868

Row ID: 7868 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 7868 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I am following along with the [TinyStories](https://arxiv.org/abs/2305.07759) dataset paper and it claims “We use the GPT-Neo tokenizer but only keep the top 10K most common tokens”. I am trying to create my own tokenizer that does this, but realized I don’t have a merges file- removing the top tokens doesn’t handle the missing byte-pair encodings without retraining. My understanding on this isn’t great, and I asked GPT and it suggested it wasn’t a great idea to ignore it and use the old file without the top tokens removed.

When looking on the huggingface for it, their merges and vocab use the full 50k tokens, so I’m a bit confused on how to accomplish this. Thanks!

FieldValue
text I am following along with the [TinyStories](https://arxiv.org/abs/2305.07759) dataset paper and it claims “We use the GPT-Neo tokenizer but only keep the top 10K most common tokens”. I am trying to create my own tokenizer that does this, but realized I don’t have a merges file- removing the top tokens doesn’t handle the missing byte-pair encodings without retraining. My understanding on this isn’t great, and I asked GPT and it suggested it wasn’t a great idea to ignore it and use the old file wi…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-18
username_encoded Z0FBQUFBQm5LakwzOWcxLU9MUlY0WHJXcUI4cDl1anF1bDkyV3QxSlRzYWRTMGVseWZCZ1gySG5ZZUU1TEdvU2d5cUN1Z2w2eWVyQ3VrYndPbWNkLUI2ZDVrSFAwRTdfQ1E9PQ==
url_encoded Z0FBQUFBQm5Lak9ISy1oYm1HMWsxZFBZdWVpV3E2UFM0VGtIcFprZjFnNm05Wm1XZGN1YTFhWWtsR2E1b2lreFZBS0hac1VIb2w0SW43b05Eb3JNSUNDbERQWXVoOGZ5dHhsYWY4cC1pOWtfTkFVeWI4bmtkRTlJNWJENVpQVG1jd1NUSTdkNU1ldjFzMzB0bzlvR2pTanNKdFBJQTVFRUo4SURpbTZLRmJKVjE4bDc1ZFYzdWViUDYwdF9qbWVSbTNGcEVtR3g5X09OLWlpMXVDOXFnRXI5SGhSWUIxX1JmQT09

Raw Record

{
  "text": "I am following along with the [TinyStories](https://arxiv.org/abs/2305.07759) dataset paper and it claims “We use the GPT-Neo tokenizer but only keep the top 10K most common tokens”. I am trying to create my own tokenizer that does this, but realized I don’t have a merges file- removing the top tokens doesn’t handle the missing byte-pair encodings without retraining. My understanding on this isn’t great, and I asked GPT and it suggested it wasn’t a great idea to ignore it and use the old file without the top tokens removed. \n\nWhen looking on the huggingface for it, their merges and vocab use the full 50k tokens, so I’m a bit confused on how to accomplish this. Thanks!\n",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-18",
  "username_encoded": "Z0FBQUFBQm5LakwzOWcxLU9MUlY0WHJXcUI4cDl1anF1bDkyV3QxSlRzYWRTMGVseWZCZ1gySG5ZZUU1TEdvU2d5cUN1Z2w2eWVyQ3VrYndPbWNkLUI2ZDVrSFAwRTdfQ1E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9ISy1oYm1HMWsxZFBZdWVpV3E2UFM0VGtIcFprZjFnNm05Wm1XZGN1YTFhWWtsR2E1b2lreFZBS0hac1VIb2w0SW43b05Eb3JNSUNDbERQWXVoOGZ5dHhsYWY4cC1pOWtfTkFVeWI4bmtkRTlJNWJENVpQVG1jd1NUSTdkNU1ldjFzMzB0bzlvR2pTanNKdFBJQTVFRUo4SURpbTZLRmJKVjE4bDc1ZFYzdWViUDYwdF9qbWVSbTNGcEVtR3g5X09OLWlpMXVDOXFnRXI5SGhSWUIxX1JmQT09"
}
Explore Dataset Explore Row

Entry Information