Row 6925

Row ID: 6925 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 6925 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

**Paper**: [https://arxiv.org/abs/2403.03183](https://arxiv.org/abs/2403.03183)

**Code**: [https://anonymous.4open.science/r/transformer\_higher\_order-B80B/](https://anonymous.4open.science/r/transformer_higher_order-B80B/)

**Abstract**:

>Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-order optimization algorithms for in-context learning and even second order ones for the case of linear regression. In this work, we study whether Transformers can perform higher order optimization methods, beyond the case of linear regression. We establish that linear attention Transformers with ReLU layers can approximate second order optimization algorithms for the task of logistic regression and achieve ϵ error with only a logarithmic to the error more layers. As a by-product we demonstrate the ability of even linear attention-only Transformers in implementing a single step of Newton's iteration for matrix inversion with merely two layers. These results suggest the ability of the Transformer architecture to implement complex algorithms, beyond gradient descent.

FieldValue
text **Paper**: [https://arxiv.org/abs/2403.03183](https://arxiv.org/abs/2403.03183) **Code**: [https://anonymous.4open.science/r/transformer\_higher\_order-B80B/](https://anonymous.4open.science/r/transformer_higher_order-B80B/) **Abstract**: >Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-order optimization algorithms for in-con…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-14
username_encoded Z0FBQUFBQm5LakwzNFNqS0ZmRGQyM0dHZXV2eW1IY3B5clc0MEVlR1A0ajlLdndGOXgyUnF1ZG9ycW5JZjVOTXNrRlQxMVNSU093LTk3enRmU0xoc3E0dnRxSXJhTlQ5dEE9PQ==
url_encoded Z0FBQUFBQm5Lak9HSlN5QXNsamxjOVIyS0ZDUDdoRlViQ2gzeFI4Rk5RMmNqSWMxVm40TUJ3MDNzS3l1ODdEaEdOQ00zZ3JpMmtMeXJxSWdHQ1F3a1A1UHMwbW5HZk5xNzJIcXhmWUlGUUJTY0JmQm1QQ0FJdzBSVFRJek8yQ2QzN056amE1Mkt6S1ctWk9wR293Y2pXRUJXTl9wajdnZktPdUlhZjVacTNPWXRCMTl6cW0yTHhpYkp2RlpwdEJ5MUNSZmROWHlZOVZYTWtxTUM3Y3dsb3RxMlp6amhtSldaUT09

Raw Record

{
  "text": "**Paper**: [https://arxiv.org/abs/2403.03183](https://arxiv.org/abs/2403.03183)\n\n**Code**: [https://anonymous.4open.science/r/transformer\\_higher\\_order-B80B/](https://anonymous.4open.science/r/transformer_higher_order-B80B/)\n\n**Abstract**:\n\n>Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-order optimization algorithms for in-context learning and even second order ones for the case of linear regression. In this work, we study whether Transformers can perform higher order optimization methods, beyond the case of linear regression. We establish that linear attention Transformers with ReLU layers can approximate second order optimization algorithms for the task of logistic regression and achieve ϵ error with only a logarithmic to the error more layers. As a by-product we demonstrate the ability of even linear attention-only Transformers in implementing a single step of Newton's iteration for matrix inversion with merely two layers. These results suggest the ability of the Transformer architecture to implement complex algorithms, beyond gradient descent.",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-14",
  "username_encoded": "Z0FBQUFBQm5LakwzNFNqS0ZmRGQyM0dHZXV2eW1IY3B5clc0MEVlR1A0ajlLdndGOXgyUnF1ZG9ycW5JZjVOTXNrRlQxMVNSU093LTk3enRmU0xoc3E0dnRxSXJhTlQ5dEE9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HSlN5QXNsamxjOVIyS0ZDUDdoRlViQ2gzeFI4Rk5RMmNqSWMxVm40TUJ3MDNzS3l1ODdEaEdOQ00zZ3JpMmtMeXJxSWdHQ1F3a1A1UHMwbW5HZk5xNzJIcXhmWUlGUUJTY0JmQm1QQ0FJdzBSVFRJek8yQ2QzN056amE1Mkt6S1ctWk9wR293Y2pXRUJXTl9wajdnZktPdUlhZjVacTNPWXRCMTl6cW0yTHhpYkp2RlpwdEJ5MUNSZmROWHlZOVZYTWtxTUM3Y3dsb3RxMlp6amhtSldaUT09"
}
Explore Dataset Explore Row

Entry Information