Row 5313

Row ID: 5313 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 5313 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

In 2021, Deepmind published [Improving language models by retrieving from trillions of tokens](https://arxiv.org/abs/2112.04426) and introduced a Retrieval-Enhanced Transformer (RETRO). Whereas RAG clasically involves supplementing input tokens at inference time by injecting relevant documents into context, RETRO can access related embeddings from an external database during *both* training and inference. The goal was to decouple reasoning and knowledge: by allowing as-needed lookup, the model can be freed from having to memorize all facts within its weights and instead reallocate energy toward more impactful computations. The results were pretty spectacular: RETRO achieved GPT-3-comparable performance with 25x fewer parameters, and is theoretically without knowledge cutoffs (just add new information to the retrieval DB!).

And yet: today, AFAICT, most major models don't incorporate RETRO. LLaMA and Mistral certainly don't, and I don't get the sense that GPT or Claude do either (the only possible exception is Gemini, based on the fact that much of the RETRO team is now part of the Gemini team and that it is both faster and more real-timey in my experience). Moreover, despite that RAG has been hot and that one might argue MoE enables it, explicitly decoupling reasoning and knowledge has been relatively quiet as a research vector.

Does anyone have a confident explanation of why this is so? I feel like RETRO's this great efficient frontier advancement sitting in plain sight just waiting for widespread adoption, but maybe I'm missing something obvious.

FieldValue
text In 2021, Deepmind published [Improving language models by retrieving from trillions of tokens](https://arxiv.org/abs/2112.04426) and introduced a Retrieval-Enhanced Transformer (RETRO). Whereas RAG clasically involves supplementing input tokens at inference time by injecting relevant documents into context, RETRO can access related embeddings from an external database during *both* training and inference. The goal was to decouple reasoning and knowledge: by allowing as-needed lookup, the model c…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-04-28
username_encoded Z0FBQUFBQm5LakwydGFGUDFEakR5QXQ1ZVVlSlE2aExaejlydEFlWU56QnVlWEpxTFBWb0N1eUs1X2ZtT0ZVRXhYWlJYZXhweVRIOUhTYV9uWHRJMjl1WXpzSHJXQ0tENnc9PQ==
url_encoded Z0FBQUFBQm5Lak9GNHZSSUEtU1RYT2kyMVZ6OUZGMW53V2U5Wkx6SHNDVHhFaDd6SlozZVZZLVU3bGtPOF83clZvbmRZTWFTNEs4aHAtU25zbXJjRWgwUG53SGRpaHRDQ3ZsUk1kZmFVSUU1SS1mMUhib0JfN00xaG9vY004Sk1seFNmOVBxMXFpUmJZZ2RlS0VUam5IaXZ3eEJJeWpoY2wzZVY1VVRmbHBpMTZJYW5nQndxOWx0LVBsWVdrbl90a2FSQ0t1Y2lzZXQ1TUhZbjJoS3Y0cEVxVlRXM0NlZUF6UT09

Raw Record

{
  "text": "In 2021, Deepmind published [Improving language models by retrieving from trillions of tokens](https://arxiv.org/abs/2112.04426) and introduced a Retrieval-Enhanced Transformer (RETRO). Whereas RAG clasically involves supplementing input tokens at inference time by injecting relevant documents into context, RETRO can access related embeddings from an external database during *both* training and inference. The goal was to decouple reasoning and knowledge: by allowing as-needed lookup, the model can be freed from having to memorize all facts within its weights and instead reallocate energy toward more impactful computations. The results were pretty spectacular: RETRO achieved GPT-3-comparable performance with 25x fewer parameters, and is theoretically without knowledge cutoffs (just add new information to the retrieval DB!).\n\n  \nAnd yet: today, AFAICT, most major models don't incorporate RETRO. LLaMA and Mistral certainly don't, and I don't get the sense that GPT or Claude do either (the only possible exception is Gemini, based on the fact that much of the RETRO team is now part of the Gemini team and that it is both faster and more real-timey in my experience). Moreover, despite that RAG has been hot and that one might argue MoE enables it, explicitly decoupling reasoning and knowledge has been relatively quiet as a research vector.\n\n  \nDoes anyone have a confident explanation of why this is so? I feel like RETRO's this great efficient frontier advancement sitting in plain sight just waiting for widespread adoption, but maybe I'm missing something obvious.",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-04-28",
  "username_encoded": "Z0FBQUFBQm5LakwydGFGUDFEakR5QXQ1ZVVlSlE2aExaejlydEFlWU56QnVlWEpxTFBWb0N1eUs1X2ZtT0ZVRXhYWlJYZXhweVRIOUhTYV9uWHRJMjl1WXpzSHJXQ0tENnc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9GNHZSSUEtU1RYT2kyMVZ6OUZGMW53V2U5Wkx6SHNDVHhFaDd6SlozZVZZLVU3bGtPOF83clZvbmRZTWFTNEs4aHAtU25zbXJjRWgwUG53SGRpaHRDQ3ZsUk1kZmFVSUU1SS1mMUhib0JfN00xaG9vY004Sk1seFNmOVBxMXFpUmJZZ2RlS0VUam5IaXZ3eEJJeWpoY2wzZVY1VVRmbHBpMTZJYW5nQndxOWx0LVBsWVdrbl90a2FSQ0t1Y2lzZXQ1TUhZbjJoS3Y0cEVxVlRXM0NlZUF6UT09"
}
Explore Dataset Explore Row

Entry Information