Row 7228

Row ID: 7228 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 7228 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

The recent GPT-4O model got me thinking whether they actually tokenized the audio and trained their GPT on text + audio tokens. Are there any successful audio tokenizers that seem to work well with auto regressive models? People have used VQ-VAE\[1\] for learning discrete representation of audio samples but the encoder and decoder of such VQ-VAE uses covnets applied over Mel-Spectrogram which I think in practice cannot enable audio streaming (As it applied 1d and 2d covnets over the entire audio signal and also doing this makes the representations non casual)

\[1\] - [https://arxiv.org/pdf/1711.00937](https://arxiv.org/pdf/1711.00937)

Edit:

A more general question I have is that is this method of tokenizing audio even feasible(will it even work?) or it's better to incrementally sample from the audio and proj each sample to an embedding and then pre train the GPT on those embeddings instead of the embeddings learned from tokens?

FieldValue
text The recent GPT-4O model got me thinking whether they actually tokenized the audio and trained their GPT on text + audio tokens. Are there any successful audio tokenizers that seem to work well with auto regressive models? People have used VQ-VAE\[1\] for learning discrete representation of audio samples but the encoder and decoder of such VQ-VAE uses covnets applied over Mel-Spectrogram which I think in practice cannot enable audio streaming (As it applied 1d and 2d covnets over the entire audio…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-15
username_encoded Z0FBQUFBQm5LakwzMmY0ZXdteG9TS3J0QlhyQUN5bFNFRUpSMVREbXZpQlg4N3JmVUduaDdwZmxNRVhEd2VMOHNuNEZRUzZLazBvTlRmcTVhSVJKTGR0cGhFbHM5Y3VtcG80Q1I4WFRvZGNidG44TmhBTzNPcUk9
url_encoded Z0FBQUFBQm5Lak9ITGlIWVlkSkpnb1BBc0tiQ251UTc2TUpLMDNLQV9GZVlaMXdDVlJOb3lPS2dDRjVaVFdKaDEzV1FoV2l6NUFocnM4d0hOYjNJQTV0b1Y3bzhrTHIwRkpZYXlrRDhVd3l5RWJ4bmpHVHV4dkEyRHVjQnE3czdMZWQ3ZVJ3YjI0dTdDa19ob2pvTlpDbGpyTS1xM1ljRktOS2o5N2s4bTd1Wjd4RTdSM0Q5TlZvPQ==

Raw Record

{
  "text": "The recent GPT-4O model got me thinking whether they actually tokenized the audio and trained their GPT on text + audio tokens. Are there any successful audio tokenizers that seem to work well with auto regressive models? People have used VQ-VAE\\[1\\] for learning discrete representation of audio samples but the encoder and decoder of such VQ-VAE uses covnets applied over Mel-Spectrogram which I think in practice cannot enable audio streaming (As it applied 1d and 2d covnets over the entire audio signal and also doing this makes the representations non casual)\n\n\\[1\\] - [https://arxiv.org/pdf/1711.00937](https://arxiv.org/pdf/1711.00937)\n\n  \nEdit:\n\nA more general question I have is that is this method of tokenizing audio even feasible(will it even work?) or it's better to incrementally sample from the audio and proj each sample to an embedding and then pre train the GPT on those embeddings instead of the embeddings learned from tokens?",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-15",
  "username_encoded": "Z0FBQUFBQm5LakwzMmY0ZXdteG9TS3J0QlhyQUN5bFNFRUpSMVREbXZpQlg4N3JmVUduaDdwZmxNRVhEd2VMOHNuNEZRUzZLazBvTlRmcTVhSVJKTGR0cGhFbHM5Y3VtcG80Q1I4WFRvZGNidG44TmhBTzNPcUk9",
  "url_encoded": "Z0FBQUFBQm5Lak9ITGlIWVlkSkpnb1BBc0tiQ251UTc2TUpLMDNLQV9GZVlaMXdDVlJOb3lPS2dDRjVaVFdKaDEzV1FoV2l6NUFocnM4d0hOYjNJQTV0b1Y3bzhrTHIwRkpZYXlrRDhVd3l5RWJ4bmpHVHV4dkEyRHVjQnE3czdMZWQ3ZVJ3YjI0dTdDa19ob2pvTlpDbGpyTS1xM1ljRktOS2o5N2s4bTd1Wjd4RTdSM0Q5TlZvPQ=="
}
Explore Dataset Explore Row

Entry Information