Row 28889

Row ID: 28889 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 28889 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I actually had that same question and spent a lot of time exploring the multimodal landscape and eventually found this paper: [Transformer with Untied Positional Encoding](https://arxiv.org/pdf/2006.15595). It turns out, afaik, that you can differentiate content embeddings from positional embeddings by projecting them separately into two different spaces, as they encode different info about the words (take for example the word “bank”). (remember that if you add the content embedding with the positional embedding you’ll get: ((wi + pi) W^Q,1 ) ((wj + pj) W^K,1 )^T , which tells us that the two are correlated, we don’t want that, just as the word bank means different things in different contexts). to untie the correlations between positions and words you can rewrite the similarity score in the self attention. initially we get an expanded dot product with four terms or correlations: word-to-word, word-to-position, position-to-word, position-to-position. cleverly, we can remove the two in the middle, as they are spurious, by using different learnable projection matrices U^Q and U^K . now we have something like this: (xi W^Q,l ) (xj W^K,l )^T + (pi U^Q ) (pj U^K ) ^T . notice that there similarity is computed twice now, for words and positions in a separate manner. however, the catch is that this mechanism can be used for any modality just as the transformer can be used for different modalities (see the OFA model by Wang et. al. 2022: [OFA](https://arxiv.org/pdf/2202.03052)). and as i see that people pointed out, it supports relative positional encoding as well, so really useful for the particular situation where we treat text and image tokens as part of the same latent space, as done in OFA and other unified architectures.

FieldValue
text I actually had that same question and spent a lot of time exploring the multimodal landscape and eventually found this paper: [Transformer with Untied Positional Encoding](https://arxiv.org/pdf/2006.15595). It turns out, afaik, that you can differentiate content embeddings from positional embeddings by projecting them separately into two different spaces, as they encode different info about the words (take for example the word “bank”). (remember that if you add the content embedding with the po…
label r/machinelearning
dataType comment
communityName r/MachineLearning
datetime 2024-05-21
username_encoded Z0FBQUFBQm5Lak1FWFRNT2hKRGZ6T3hQZFRMX0hvcHMtT3JDNzhuaWJvZGtSUGVETHBWLVYzNk02SjN3X09lNlFET1Bsc0g2X1VlTG02dU9xRk4wTkZraWUtSUZ4dnNiT1E9PQ==
url_encoded Z0FBQUFBQm5Lak9VSDVMd0gyZ3ZpS1ZFbmdqWmxLMXpzOHU5UG0xLWp3QkJMNHYwcGVZZlBlUXBXN0plU01NS25aRE5tR3c0aDlfeTlhU0pCSUhvR1FxYWRfdVNFa0ZRZGFwRFRscG8taktHd25KdExtcmEwV19MYVpaS2s1TnF3dnpzN1gybW5yZjBhWE5WSTlMMGZHSXFvQWlmUUJqNll3dzNoamh2WUtSWHpNbGdRb01vSTY1bUx0WmI1R2V5c3RLdmYwSDdKVzRzaDdUWTdPWG1ESXdUcjE4eGhsVzdSRGJ1SW1yUjl1RkpIbnBaajRkaFJBWT0=

Raw Record

{
  "text": "I actually had that same question and spent a lot of time exploring the multimodal landscape and eventually found this paper: [Transformer with Untied Positional Encoding](https://arxiv.org/pdf/2006.15595). It turns out, afaik, that you can differentiate content embeddings from positional embeddings by projecting them separately into two different spaces, as they encode different info about the words (take for example the word  “bank”). (remember that if you add the content embedding with the positional embedding you’ll get: ((wi + pi) W^Q,1 ) ((wj + pj) W^K,1 )^T , which tells us that the two are correlated, we don’t want that, just as the word bank means different things in different contexts). to untie the correlations between positions and words you can rewrite the similarity score in the self attention. initially we get an expanded dot product with four terms or correlations: word-to-word, word-to-position, position-to-word, position-to-position. cleverly, we can remove the two in the middle, as they are spurious, by using different learnable projection matrices U^Q and U^K . now we have something like this: (xi W^Q,l ) (xj W^K,l )^T + (pi U^Q ) (pj U^K ) \n^T . notice that there similarity is computed twice now, for words and positions in a separate manner. however, the catch is that this mechanism can be used for any modality just as the transformer can be used for different modalities (see the OFA model by Wang et. al. 2022: [OFA](https://arxiv.org/pdf/2202.03052)). and as i see that people pointed out, it supports relative positional encoding as well, so really useful for the particular situation where we treat text and image tokens as part of the same latent space, as done in OFA and other unified architectures.",
  "label": "r/machinelearning",
  "dataType": "comment",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-21",
  "username_encoded": "Z0FBQUFBQm5Lak1FWFRNT2hKRGZ6T3hQZFRMX0hvcHMtT3JDNzhuaWJvZGtSUGVETHBWLVYzNk02SjN3X09lNlFET1Bsc0g2X1VlTG02dU9xRk4wTkZraWUtSUZ4dnNiT1E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9VSDVMd0gyZ3ZpS1ZFbmdqWmxLMXpzOHU5UG0xLWp3QkJMNHYwcGVZZlBlUXBXN0plU01NS25aRE5tR3c0aDlfeTlhU0pCSUhvR1FxYWRfdVNFa0ZRZGFwRFRscG8taktHd25KdExtcmEwV19MYVpaS2s1TnF3dnpzN1gybW5yZjBhWE5WSTlMMGZHSXFvQWlmUUJqNll3dzNoamh2WUtSWHpNbGdRb01vSTY1bUx0WmI1R2V5c3RLdmYwSDdKVzRzaDdUWTdPWG1ESXdUcjE4eGhsVzdSRGJ1SW1yUjl1RkpIbnBaajRkaFJBWT0="
}
Explore Dataset Explore Row

Entry Information