Row 5052

Row ID: 5052 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 5052 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

It seems to me that thanks to the residual path the gradient that flows to each layer is the same regardless of the transformer layer/block. Example:

ProjectionAndCost(X + L1(X) + L2(X + L1(X)) + L3(X + L1(X) + L2(X + L1(X))) ...)

Since the input to ProjectionAndCost is just sum of outputs from all layers and initial embeddings then the gradient that comes to the layer L1 is the same as the gradient that comes to L2 or L3.

So we could:

* first train only L1: ProjectionAndCost(X + L1(X)) * freeze L1, include L2 and train: ProjectionAndCost(X + L1(X) + L2(X + L1(X))) * freeze L1 and L2, include L3 and train: ProjectionAndCost(X + L1(X) + L2(X + L1(X)) + L3(X + L1(X) + L2(X + L1(X)))) * .. and so on

We can't train first L2 then L1, because the input to L2 depends on L1, but we could train lower layers first then gradually add and train deeper layers. Is there any problem with that approach?

FieldValue
text It seems to me that thanks to the residual path the gradient that flows to each layer is the same regardless of the transformer layer/block. Example: ProjectionAndCost(X + L1(X) + L2(X + L1(X)) + L3(X + L1(X) + L2(X + L1(X))) ...) Since the input to ProjectionAndCost is just sum of outputs from all layers and initial embeddings then the gradient that comes to the layer L1 is the same as the gradient that comes to L2 or L3. So we could: * first train only L1: ProjectionAndCost(X + L1(X)) * fr…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-04-25
username_encoded Z0FBQUFBQm5LakwydmhIS0M4SktKT1ZNcFg1aWxYWkM0a0NMQVMxX2s4QmZDTjllSFhqWlJYQkNubHpzdEFteWxHS2lZcjFtb1pGV0tCSUt1ZnoyVlY4Q0xfdkRjRHZzZGc9PQ==
url_encoded Z0FBQUFBQm5Lak9GclpBcEtWNkZ6VE1hWXF3U0tYcmNndzlHdTItRTFvWk1LZmtLQWtfaE84T1IxM1l6ODZnODBTX2hISm5xOWd0Vi0xek5YNmtZakdnYVZJckZTYVc5UkhzM3ZJS0ZsVnVhc0hIN0JhR0g1dTIwSXF1aGpOOWJTc1lqWWhvaEszb0dZaVlkeHRvYjhQb2gzbERIdTNRa19wR0tsUlpsd282blJURVpwVDNDMEYwc1NHdzdIYXdCTm1JN1o1cl9HLWtod3d5NXJSSWVpNURpSnNxVXR5VWJxUT09

Raw Record

{
  "text": "It seems to me that thanks to the residual path the gradient that flows to each layer is the same regardless of the transformer layer/block. Example:\n\nProjectionAndCost(X + L1(X) + L2(X + L1(X)) + L3(X + L1(X) + L2(X + L1(X))) ...)\n\nSince the input to ProjectionAndCost is just sum of outputs from all layers and initial embeddings then the gradient that comes to the layer L1 is the same as the gradient that comes to L2 or L3.\n\nSo we could:\n\n* first train only L1: ProjectionAndCost(X + L1(X))\n* freeze L1, include L2 and train: ProjectionAndCost(X + L1(X) + L2(X + L1(X)))\n* freeze L1 and L2, include L3 and train: ProjectionAndCost(X + L1(X) + L2(X + L1(X)) + L3(X + L1(X) + L2(X + L1(X))))\n* .. and so on\n\nWe can't train first L2 then L1, because the input to L2 depends on L1, but we could train lower layers first then gradually add and train deeper layers. Is there any problem with that approach?",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-04-25",
  "username_encoded": "Z0FBQUFBQm5LakwydmhIS0M4SktKT1ZNcFg1aWxYWkM0a0NMQVMxX2s4QmZDTjllSFhqWlJYQkNubHpzdEFteWxHS2lZcjFtb1pGV0tCSUt1ZnoyVlY4Q0xfdkRjRHZzZGc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9GclpBcEtWNkZ6VE1hWXF3U0tYcmNndzlHdTItRTFvWk1LZmtLQWtfaE84T1IxM1l6ODZnODBTX2hISm5xOWd0Vi0xek5YNmtZakdnYVZJckZTYVc5UkhzM3ZJS0ZsVnVhc0hIN0JhR0g1dTIwSXF1aGpOOWJTc1lqWWhvaEszb0dZaVlkeHRvYjhQb2gzbERIdTNRa19wR0tsUlpsd282blJURVpwVDNDMEYwc1NHdzdIYXdCTm1JN1o1cl9HLWtod3d5NXJSSWVpNURpSnNxVXR5VWJxUT09"
}
Explore Dataset Explore Row

Entry Information