Row 19142

Row ID: 19142 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 19142 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Yes, I think that LeCun is empirically wrong about his "error" expansion theory as applied to GPTs. I think that GPTs do NOT necessarily perform "auto regressive error amplification" Rather, the GPTs apparently tend to drift back towards the familiar/correct despite any typo, word-omission, misspelling, or any false statement, within the original prompt or within the subsequently generated next-token sequence. GPTs can detect and ignore nonsense text in their prompts, or token sequences. Even very tiny GPT models I have seen can immediately recover to coherent text even after deletion of prior words in the prompt or previous next-token sequence (e.g., immediately ignoring the omission of previously included tokens or words). I think that the method of token-sampling "logits" step at the output-head of GPT LLMs creates a error-correcting band gap that filters out token-sequence errors. There is a range of error-tolerance in many or most embeddings dimensions (among the logits) and the "correct next token" will still be selected despite errors.

Mar Terr BSEE scl JC mcl

P.S. LeCun also nonsensically compares large GPTs to being less "intelligent" than "cats"? I cant even figure out where he would obtain an objective metric that could support that assertion. I do not know of any Cats that can replace Call-Center Workers or Poets.

FieldValue
text Yes, I think that LeCun is empirically wrong about his "error" expansion theory as applied to GPTs. I think that GPTs do NOT necessarily perform "auto regressive error amplification" Rather, the GPTs apparently tend to drift back towards the familiar/correct despite any typo, word-omission, misspelling, or any false statement, within the original prompt or within the subsequently generated next-token sequence. GPTs can detect and ignore nonsense text in their prompts, or token sequences. Even…
label r/machinelearning
dataType comment
communityName r/MachineLearning
datetime 2024-05-21
username_encoded Z0FBQUFBQm5LakxfYlBwdTYtUGxsN3FlTThGeEEydGJaSjVndzhtLVUzYkpzRl9SdzB3TDRlS0w3dkpUSlpXY0RvYXVCLUJJaHAtU0p0STJtLUgzcjJOVWhrNzBqWTNNcVE9PQ==
url_encoded Z0FBQUFBQm5Lak9PR0wzTkg3am91ODZvTUozM3lZazhSdTlFQk1OUEVHTE5PQ19OZW9zVERjVTNRemRoNmU3Ni1yM1BEVzV1akZVWmNmLVNscE1Ed2ctZ01HTEplNXl5a3FaVUljOWp5d3ZCbGpxUl9wQldmRXBBaWo2T0hjWGcxTXV5MkxKbEVOYUMyNzBnbXltLUM5Y3Ezdlg4Zm5IT2lhYW82cVdDSUlXS09Wb2ppZ1Z2N25KQXdvQlBJU3lvbEtwYmNiSzVYSnQycl83VXhvT2tEbV9lX3FIUHVsVlJsbmRqZnhvaXp6VVRfZTE3QjQ3ZVFWUT0=

Raw Record

{
  "text": "Yes, I think that LeCun is empirically wrong about his \"error\" expansion theory as applied to GPTs.  I think that GPTs do NOT necessarily perform \"auto regressive error amplification\" Rather, the GPTs apparently tend to drift back towards the familiar/correct despite any typo, word-omission, misspelling, or any false statement, within the original prompt or within the subsequently generated next-token sequence.  GPTs can detect and ignore nonsense text in their prompts, or token sequences.  Even very tiny GPT models I have seen can immediately recover to coherent text even after deletion of prior words in the prompt or previous next-token sequence (e.g., immediately ignoring the omission of previously included tokens or words).  I think that the method of token-sampling \"logits\" step at the output-head of GPT LLMs creates a error-correcting band gap that filters out token-sequence errors.  There is a range of error-tolerance in many or most embeddings dimensions (among the logits) and the \"correct next token\" will still be selected despite errors.\n\nMar Terr BSEE scl JC mcl\n\n  \nP.S.  LeCun also nonsensically compares large GPTs to being less \"intelligent\" than \"cats\"?  I cant even figure out where he would obtain an objective metric that could support that assertion.  I do not know of any Cats that can replace Call-Center Workers or Poets.",
  "label": "r/machinelearning",
  "dataType": "comment",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-21",
  "username_encoded": "Z0FBQUFBQm5LakxfYlBwdTYtUGxsN3FlTThGeEEydGJaSjVndzhtLVUzYkpzRl9SdzB3TDRlS0w3dkpUSlpXY0RvYXVCLUJJaHAtU0p0STJtLUgzcjJOVWhrNzBqWTNNcVE9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9PR0wzTkg3am91ODZvTUozM3lZazhSdTlFQk1OUEVHTE5PQ19OZW9zVERjVTNRemRoNmU3Ni1yM1BEVzV1akZVWmNmLVNscE1Ed2ctZ01HTEplNXl5a3FaVUljOWp5d3ZCbGpxUl9wQldmRXBBaWo2T0hjWGcxTXV5MkxKbEVOYUMyNzBnbXltLUM5Y3Ezdlg4Zm5IT2lhYW82cVdDSUlXS09Wb2ppZ1Z2N25KQXdvQlBJU3lvbEtwYmNiSzVYSnQycl83VXhvT2tEbV9lX3FIUHVsVlJsbmRqZnhvaXp6VVRfZTE3QjQ3ZVFWUT0="
}
Explore Dataset Explore Row

Entry Information