Row 6650

Row ID: 6650 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 6650 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

**Paper**: [https://arxiv.org/abs/2402.04553](https://arxiv.org/abs/2402.04553)

**Code** (*toy experiments*): [https://github.com/lixilinx/psgd\_torch](https://github.com/lixilinx/psgd_torch)

**Code** (*large scale experiments*): [https://github.com/opooladz/Preconditioned-Stochastic-Gradient-Descent](https://github.com/opooladz/Preconditioned-Stochastic-Gradient-Descent)

**Abstract**:

>We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature information obtained from Hessian-vector products or finite differences of parameters and gradients, similar to the BFGS algorithm. Our approach involves two preconditioners: a matrix-free preconditioner and a low-rank approximation preconditioner. We update both preconditioners online using a criterion that is robust to stochastic gradient noise and does not require line search or damping. To preserve the corresponding symmetry or invariance, our preconditioners are constrained to certain connected Lie groups. The Lie group's equivariance property simplifies the preconditioner fitting process, while its invariance property eliminates the need for damping, which is commonly required in second-order optimizers. As a result, the learning rate for parameter updating and the step size for preconditioner fitting are naturally normalized, and their default values work well in most scenarios. Our proposed approach offers a promising direction for improving the convergence of SGD with low computational overhead. We demonstrate that **Preconditioned SGD** (**PSGD**) outperforms SoTA on Vision, NLP, and RL tasks across multiple modern deep-learning architectures. We have provided code for reproducing toy and large scale experiments in this paper.

FieldValue
text **Paper**: [https://arxiv.org/abs/2402.04553](https://arxiv.org/abs/2402.04553) **Code** (*toy experiments*): [https://github.com/lixilinx/psgd\_torch](https://github.com/lixilinx/psgd_torch) **Code** (*large scale experiments*): [https://github.com/opooladz/Preconditioned-Stochastic-Gradient-Descent](https://github.com/opooladz/Preconditioned-Stochastic-Gradient-Descent) **Abstract**: >We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature informa…
label r/neuralnetworks
dataType post
communityName r/neuralnetworks
datetime 2024-05-12
username_encoded Z0FBQUFBQm5LakwzeUJ0ZW1GLXdiNU5saDVkeUdTMVQwTmZDNEpXLWFTYlJiZlRpeDd0SFd6NmlNYUlwX3dKRGN6NExFSFdxSVpGaGlBLXd5YWp1b09OMkRmTHFDNWtWc3c9PQ==
url_encoded Z0FBQUFBQm5Lak9HYXNKckNmSEE0R2ZTNXBzOGxJZEhJQXo1NnlvZDl0Y2MxcEhVcm5EeG81WnBBRmFQTC15VGFNLXQ5bDY2WGFHUXRaRzJMR2dEamtjdlVkaU5jMllvMmNGb3pnQjNGYTV1RTJpRDBocVRYdDlldUdGdXVzOU55ZkRYTTE3M3puUGhKZEdmaFdVZFc2VjEtcDQ5a1E4S2RwejJvbm1xOG5SSzkyMWJWcklqVXhITERudE80Sk9PNXlvdk5KakRrck5pc1dSczRUUW92V2JpUmIyOHZQcm05dz09

Raw Record

{
  "text": "**Paper**: [https://arxiv.org/abs/2402.04553](https://arxiv.org/abs/2402.04553)\n\n**Code** (*toy experiments*): [https://github.com/lixilinx/psgd\\_torch](https://github.com/lixilinx/psgd_torch)\n\n**Code** (*large scale experiments*): [https://github.com/opooladz/Preconditioned-Stochastic-Gradient-Descent](https://github.com/opooladz/Preconditioned-Stochastic-Gradient-Descent)\n\n**Abstract**:\n\n>We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature information obtained from Hessian-vector products or finite differences of parameters and gradients, similar to the BFGS algorithm. Our approach involves two preconditioners: a matrix-free preconditioner and a low-rank approximation preconditioner. We update both preconditioners online using a criterion that is robust to stochastic gradient noise and does not require line search or damping. To preserve the corresponding symmetry or invariance, our preconditioners are constrained to certain connected Lie groups. The Lie group's equivariance property simplifies the preconditioner fitting process, while its invariance property eliminates the need for damping, which is commonly required in second-order optimizers. As a result, the learning rate for parameter updating and the step size for preconditioner fitting are naturally normalized, and their default values work well in most scenarios. Our proposed approach offers a promising direction for improving the convergence of SGD with low computational overhead. We demonstrate that **Preconditioned SGD** (**PSGD**) outperforms SoTA on Vision, NLP, and RL tasks across multiple modern deep-learning architectures. We have provided code for reproducing toy and large scale experiments in this paper.",
  "label": "r/neuralnetworks",
  "dataType": "post",
  "communityName": "r/neuralnetworks",
  "datetime": "2024-05-12",
  "username_encoded": "Z0FBQUFBQm5LakwzeUJ0ZW1GLXdiNU5saDVkeUdTMVQwTmZDNEpXLWFTYlJiZlRpeDd0SFd6NmlNYUlwX3dKRGN6NExFSFdxSVpGaGlBLXd5YWp1b09OMkRmTHFDNWtWc3c9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HYXNKckNmSEE0R2ZTNXBzOGxJZEhJQXo1NnlvZDl0Y2MxcEhVcm5EeG81WnBBRmFQTC15VGFNLXQ5bDY2WGFHUXRaRzJMR2dEamtjdlVkaU5jMllvMmNGb3pnQjNGYTV1RTJpRDBocVRYdDlldUdGdXVzOU55ZkRYTTE3M3puUGhKZEdmaFdVZFc2VjEtcDQ5a1E4S2RwejJvbm1xOG5SSzkyMWJWcklqVXhITERudE80Sk9PNXlvdk5KakRrck5pc1dSczRUUW92V2JpUmIyOHZQcm05dz09"
}
Explore Dataset Explore Row

Entry Information