Row 42485

Row ID: 42485 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 42485 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Eh, I'd probably frame it as pedagogical more so than misleading. The story about interpolation is technically true, and it follows in an intuitive way from binary encodings, which are themselves intuitive and easy to understand.

Relating tokens by ensuring that the inner products of their vector representations have certain desirable properties is, by contrast, a very abstract way of understanding the issue, and it's difficult for people without a strong math background to follow it. I actually quite like the presentation in the article, I think it strikes a good balance between pedagogy and technical accuracy.

And, really, neither of these things was the true "motivator" for sinusoidal embeddings; all this stuff about interpolation or inner products was been developed in hindsight by followup research. The *real* story is that the people who first developed sinusoidal embeddings probably tried a whole bunch of different things and, out of all the things they thought to try, sinusoidal embeddings worked best. The ad-hoc nature of sinusoidal embeddings is suggested by their original formulation, which involved some weirdly arbitrary frequency coefficients, and also by later developments like rotary embeddings that are more principled.

FieldValue
text Eh, I'd probably frame it as pedagogical more so than misleading. The story about interpolation is technically true, and it follows in an intuitive way from binary encodings, which are themselves intuitive and easy to understand. Relating tokens by ensuring that the inner products of their vector representations have certain desirable properties is, by contrast, a very abstract way of understanding the issue, and it's difficult for people without a strong math background to follow it. I actual…
label r/machinelearning
dataType comment
communityName r/MachineLearning
datetime 2024-05-22
username_encoded Z0FBQUFBQm5Lak1ONTFaM1JPaTBpN29uNDJfSFNFSVI2WFZaWlF5eHl6MHhzVTNuZVdtbEpoRTE4cTJRY21kU1IwS09KTERDdGVvdS1LVUd0ZmhmdlRDTTVUR2tmZDUzb3c9PQ==
url_encoded Z0FBQUFBQm5Lak9kN3l1dmFuV2U4WmxKN2xfTEtMRDdDNWZEN3ZxTDREUHFTNklsZklSSl9qbnVrTC1rcjAxYkNrRHRTbVpvSDNiVVRDQjhCNVlQSURaaHZ4eGlkQ0NnQzNkTHpFTkZWclBmT0xGdUFyMkM0NkpSM2p5LUFvWjR0eERhZjFCTW5obzRBMndsQ0VrMDRFSVZrQ3FtVlNoWjZ3ZzMxN3RGRjhiSnhST3pGbmxqeVpmYV91RUZ0aEFYOUd6bWYtM1BYNzVr

Raw Record

{
  "text": "Eh, I'd probably frame it as pedagogical more so than misleading. The story about interpolation is technically true, and it follows in an intuitive way from binary encodings, which are themselves intuitive and easy to understand. \n\nRelating tokens by ensuring that the inner products of their vector representations have certain desirable properties is, by contrast, a very abstract way of understanding the issue, and it's difficult for people without a strong math background to follow it. I actually quite like the presentation in the article, I think it strikes a good balance between pedagogy and technical accuracy.\n\nAnd, really, neither of these things was the true \"motivator\" for sinusoidal embeddings; all this stuff about interpolation or inner products was been developed in hindsight by followup research. The *real* story is that the people who first developed sinusoidal embeddings probably tried a whole bunch of different things and, out of all the things they thought to try, sinusoidal embeddings worked best. The ad-hoc nature of sinusoidal embeddings is suggested by their original formulation, which involved some weirdly arbitrary frequency coefficients, and also by later developments like rotary embeddings that are more principled.",
  "label": "r/machinelearning",
  "dataType": "comment",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-22",
  "username_encoded": "Z0FBQUFBQm5Lak1ONTFaM1JPaTBpN29uNDJfSFNFSVI2WFZaWlF5eHl6MHhzVTNuZVdtbEpoRTE4cTJRY21kU1IwS09KTERDdGVvdS1LVUd0ZmhmdlRDTTVUR2tmZDUzb3c9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9kN3l1dmFuV2U4WmxKN2xfTEtMRDdDNWZEN3ZxTDREUHFTNklsZklSSl9qbnVrTC1rcjAxYkNrRHRTbVpvSDNiVVRDQjhCNVlQSURaaHZ4eGlkQ0NnQzNkTHpFTkZWclBmT0xGdUFyMkM0NkpSM2p5LUFvWjR0eERhZjFCTW5obzRBMndsQ0VrMDRFSVZrQ3FtVlNoWjZ3ZzMxN3RGRjhiSnhST3pGbmxqeVpmYV91RUZ0aEFYOUd6bWYtM1BYNzVr"
}
Explore Dataset Explore Row

Entry Information