Row 8387

Row ID: 8387 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 8387 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I have been playing with the concept of attending to more than two tokens in transformer models. Instead of having one query and one key for example, having two keys and one query, and for every query sum over every pair of previous tokens.

It makes the algorithm even slower ( O(n\*\*3) instead of O(n\*\*2)), but I think it is a fun concept. Some results where surprising to me, like how good it is at finding the longest increasing subsequence.

I want it to share it: [https://github.com/Gusanidas/n-way-attention/tree/main](https://github.com/Gusanidas/n-way-attention/tree/main)

And to ask if anyone knows of papers that treat the concept, or mention it.

FieldValue
text I have been playing with the concept of attending to more than two tokens in transformer models. Instead of having one query and one key for example, having two keys and one query, and for every query sum over every pair of previous tokens. It makes the algorithm even slower ( O(n\*\*3) instead of O(n\*\*2)), but I think it is a fun concept. Some results where surprising to me, like how good it is at finding the longest increasing subsequence. I want it to share it: [https://github.com/Gusan…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-19
username_encoded Z0FBQUFBQm5Lakw0dmIwUlhKVXFITUE3Y1NhOGcxd2FiVFQ0NkNfS0lSa0tVMTJlMGpYNWthRWNzVXBXamxRUFdfUlJNOHJERV9ZcVRvVmNDa25TZzFoWnozT0xpb3RQMnc9PQ==
url_encoded Z0FBQUFBQm5Lak9IeUg4ek0zOWN5NjRILUJJcUhkYTlmTFBvY3JLODV4cHhQU3FXUVNoalBaem5kbkhHNUpnQlZFazJBSGRoRVoycHdKX2xaSE9jTVJZUVg0QVN1LVFfWFluRVRHX0tTN21XYm9HMkJVZElUaWt2SV9wMGx5MkVKZ0swTzhhc2ZlYUNQQl9DQVdPSTkxMDJjUTJFdlFOQm4tN0ZJNG8yM2lIb2RtTGgxWUdKMkRBPQ==

Raw Record

{
  "text": "I have been playing with the concept of attending to more than two tokens in transformer models. Instead of having one query and one key for example, having two keys and one query, and for every query sum over every pair of previous tokens.\n\nIt makes the algorithm even slower ( O(n\\*\\*3) instead of O(n\\*\\*2)), but I think it is a fun concept. Some results where surprising to me, like how good it is at finding the longest increasing subsequence.\n\nI want it to share it:  \n[https://github.com/Gusanidas/n-way-attention/tree/main](https://github.com/Gusanidas/n-way-attention/tree/main)\n\nAnd to ask if anyone knows of papers that treat the concept, or mention it.",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-19",
  "username_encoded": "Z0FBQUFBQm5Lakw0dmIwUlhKVXFITUE3Y1NhOGcxd2FiVFQ0NkNfS0lSa0tVMTJlMGpYNWthRWNzVXBXamxRUFdfUlJNOHJERV9ZcVRvVmNDa25TZzFoWnozT0xpb3RQMnc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9IeUg4ek0zOWN5NjRILUJJcUhkYTlmTFBvY3JLODV4cHhQU3FXUVNoalBaem5kbkhHNUpnQlZFazJBSGRoRVoycHdKX2xaSE9jTVJZUVg0QVN1LVFfWFluRVRHX0tTN21XYm9HMkJVZElUaWt2SV9wMGx5MkVKZ0swTzhhc2ZlYUNQQl9DQVdPSTkxMDJjUTJFdlFOQm4tN0ZJNG8yM2lIb2RtTGgxWUdKMkRBPQ=="
}
Explore Dataset Explore Row

Entry Information