Row 5139

Row ID: 5139 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 5139 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I am just trying to get the decoder of my model (which uses cross-attention) to overfit to one sequence of 20 tokens as debugging/sanity check. When the tokens are the all the same, the model learns quickly and cross-entropy loss drops to 0 with a few epochs. This is true even if sequence length is 300+. But for even a small sequence of 5, if there is more than one token type e.g. \[6,6,7,7,7\] it struggles and plateaus around a loss of .8 or so.

Any idea why this happens?

Model details:

* using just the decoder from [https://huggingface.co/microsoft/speecht5\_asr](https://huggingface.co/microsoft/speecht5_asr) * main parts of code

>config = SpeechT5Config( vocab\_size=81, d\_model=768, max\_length=450 ) model = SpeechT5ForSpeechToText(config) model.speecht5.encoder = None # I am using my own pretrained encoder > ><other code> > >encoder\_outputs = (last\_hidden\_states,) # shape is (1, seq\_len, hidden\_dim) outputs = model(encoder\_outputs=encoder\_outputs, labels=labels)

full code (mostly):

>config = SpeechT5Config( vocab\_size=81, d\_model=768, max\_length=450 ) model = SpeechT5ForSpeechToText(config) model.speecht5.encoder = None # using my own encoder so don't need model = model.to('cuda') my\_encoder = model\_enc.to('cuda') optimizer = optim.Adam(model.parameters(), lr=cfg.LR) rand\_labels = torch.zeros(2, dtype=torch.int64) rand\_labels\[0\] = 6 rand\_labels\[1\] = 7 for epoch in range(cfg.EPOCHS): total\_loss = 0 \# dataloader has batchsize of 1 and the data for each batch is identical progress\_bar = tqdm(dataloader, desc=f'Epoch {epoch + 1}/{cfg.EPOCHS}') for batch in progress\_bar: frames = batch\['frames'\]\[0\] # always the same frames labels = rand\_labels.unsqueeze(0) # (1, 2) frames = frames.to('cuda') labels = labels.to('cuda') with torch.no\_grad(): outputs = my\_encoder(frames) last\_hidden\_states = outputs.last\_hidden\_state # (\[1, 3137, 768\]) batchsize, seq len, hidden\_dim \# Model forward pass encoder\_outputs = (last\_hidden\_states,) outputs = model(encoder\_outputs=encoder\_outputs, labels=labels) \# Loss computation loss = outputs.loss total\_loss += loss.item() \# Backpropagation optimizer.zero\_grad() loss.backward() optimizer.step() progress\_bar.set\_postfix({'loss': loss.item()})

&#x200B;

FieldValue
text I am just trying to get the decoder of my model (which uses cross-attention) to overfit to one sequence of 20 tokens as debugging/sanity check. When the tokens are the all the same, the model learns quickly and cross-entropy loss drops to 0 with a few epochs. This is true even if sequence length is 300+. But for even a small sequence of 5, if there is more than one token type e.g. \[6,6,7,7,7\] it struggles and plateaus around a loss of .8 or so. Any idea why this happens? Model details: * us…
label r/deeplearning
dataType post
communityName r/deeplearning
datetime 2024-04-26
username_encoded Z0FBQUFBQm5LakwyaDlKNWE4NWNXYzRhY3d2WU9VdTVpZkZxRGxpa09ucFZVdm1oYmEybVZZY25zS0NiTVhYd3VGS2w2bG5lUTZoTXE3TEprdElJYU9mNkxXZUVRa0pkY0E9PQ==
url_encoded Z0FBQUFBQm5Lak9GdjhmX1VCQlhlWGRzQzBpU1BuRkowdXNrMGZtVExzN2VoTGQxWW5SY3IwX2NHS0dCQU9ZNXZOMXRDX08zdHU1VFU5SWVhZ3ZXdF94TU4yWFpvTG5mOVVGcFM4OGdEYy02Vng4N0pSUDNSS3d0Zk0ycU11MFh5Zl9aSEx5OFdBZzh1N3BvejIwVnVURXVxUldUSVVjNzVLSnJ5QkVjcFFZM1JMYnlUZWlWZ2hFRW9jbzZyVlNDc1dZT3VBVkZQOUhMWGQ0LVZmWVJ3Y3Y5c3hTb2FCbWdoUT09

Raw Record

{
  "text": "I am just trying to get the decoder of my model (which uses cross-attention) to overfit to one sequence of 20 tokens as debugging/sanity check. When the tokens are the all the same, the model learns quickly and cross-entropy loss drops to 0 with a few epochs. This is true even if sequence length is 300+. But for even a small sequence of 5, if there is more than one token type e.g. \\[6,6,7,7,7\\] it struggles and plateaus around a loss of .8 or so.\n\nAny idea why this happens?\n\nModel details:\n\n* using just the decoder from [https://huggingface.co/microsoft/speecht5\\_asr](https://huggingface.co/microsoft/speecht5_asr)\n* main parts of code\n\n>config = SpeechT5Config(  \nvocab\\_size=81,  \nd\\_model=768,  \nmax\\_length=450  \n)  \nmodel = SpeechT5ForSpeechToText(config)  \nmodel.speecht5.encoder = None # I am using my own pretrained encoder  \n>  \n><other code>  \n>  \n>encoder\\_outputs = (last\\_hidden\\_states,) # shape is (1, seq\\_len, hidden\\_dim)  \noutputs = model(encoder\\_outputs=encoder\\_outputs, labels=labels)\n\nfull code (mostly):  \n\n\n>config = SpeechT5Config(  \nvocab\\_size=81,  \nd\\_model=768,  \nmax\\_length=450  \n)  \nmodel = SpeechT5ForSpeechToText(config)  \nmodel.speecht5.encoder = None # using my own encoder so don't need  \nmodel = model.to('cuda')  \nmy\\_encoder = model\\_enc.to('cuda')  \noptimizer = optim.Adam(model.parameters(), lr=cfg.LR)  \nrand\\_labels = torch.zeros(2, dtype=torch.int64)  \nrand\\_labels\\[0\\] = 6  \nrand\\_labels\\[1\\] = 7  \nfor epoch in range(cfg.EPOCHS):  \ntotal\\_loss = 0  \n \\# dataloader has batchsize of 1 and the data for each batch is identical  \nprogress\\_bar = tqdm(dataloader, desc=f'Epoch {epoch + 1}/{cfg.EPOCHS}')  \n for batch in progress\\_bar:  \nframes = batch\\['frames'\\]\\[0\\] # always the same frames  \nlabels = rand\\_labels.unsqueeze(0) # (1, 2)  \nframes = frames.to('cuda')  \nlabels = labels.to('cuda')  \n with torch.no\\_grad():  \noutputs = my\\_encoder(frames)  \nlast\\_hidden\\_states = outputs.last\\_hidden\\_state # (\\[1, 3137, 768\\]) batchsize, seq len, hidden\\_dim  \n \\# Model forward pass  \nencoder\\_outputs = (last\\_hidden\\_states,)  \noutputs = model(encoder\\_outputs=encoder\\_outputs, labels=labels)  \n \\# Loss computation  \nloss = outputs.loss  \ntotal\\_loss += loss.item()  \n \\# Backpropagation  \noptimizer.zero\\_grad()  \nloss.backward()  \noptimizer.step()  \n   \nprogress\\_bar.set\\_postfix({'loss': loss.item()})\n\n&#x200B;",
  "label": "r/deeplearning",
  "dataType": "post",
  "communityName": "r/deeplearning",
  "datetime": "2024-04-26",
  "username_encoded": "Z0FBQUFBQm5LakwyaDlKNWE4NWNXYzRhY3d2WU9VdTVpZkZxRGxpa09ucFZVdm1oYmEybVZZY25zS0NiTVhYd3VGS2w2bG5lUTZoTXE3TEprdElJYU9mNkxXZUVRa0pkY0E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9GdjhmX1VCQlhlWGRzQzBpU1BuRkowdXNrMGZtVExzN2VoTGQxWW5SY3IwX2NHS0dCQU9ZNXZOMXRDX08zdHU1VFU5SWVhZ3ZXdF94TU4yWFpvTG5mOVVGcFM4OGdEYy02Vng4N0pSUDNSS3d0Zk0ycU11MFh5Zl9aSEx5OFdBZzh1N3BvejIwVnVURXVxUldUSVVjNzVLSnJ5QkVjcFFZM1JMYnlUZWlWZ2hFRW9jbzZyVlNDc1dZT3VBVkZQOUhMWGQ0LVZmWVJ3Y3Y5c3hTb2FCbWdoUT09"
}
Explore Dataset Explore Row

Entry Information