Row 9145

Content Data

This page contains data entry 9145 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Raw Record

{
  "text": "Hi, my intention is to prepare myself for the upcoming academic year on September.\n\nI want to have the ability to write my own PyTorch's Module, Loss, and Optimizer. I intend to start easy with Module (Linear), Loss (Least Squares) and Optimizer (Gradient Descent).\n\nMy goal is to implement:\n1. `LeastSquares` loss which subclass the `torch.nn.modules.loss._Loss` class.\n2. `GradientDescent` optimizer which subclass the `torch.optim.optimizer.Optimizer` class.\n2. `Cubic Polynomial` module which subclass the `torch.nn.Module` class.\n\nWhat I've tried:\n1. Read [Learning PyTorch with Examples](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html)\n\n   The Learning PyTorch with Examples teach how to define autograd functions. This became [a dedicated StackOverflow question](https://stackoverflow.com/questions/78504721/why-torch-autograd-function-backward-pass-a-grad-output)\n\n   The rest of the examples teach how to use existing Loss function (such as `torch.nn.MSELoss`), Optimizer (such as `torch.optim.SGD`) and Module (such as `torch.nn.Linear`).\n\n\nAfter spending a day. I am confused on how `torch.nn.modules.loss._Loss`, `torch.optim.optimizer.Optimizer`, and `torch.nn.Module` works. Is there a tutorial how to create your own PyTorch Loss, Optimizer, and Module?\n\nThe code below is trying to mimic `PyTorch`. But, I don't know how `optimizer.zero_grad()`, `loss.backward()`, and `optimizer.step()` can update the `model.parameters()`. So, I put everything in 1 class.\n\n**I am open to learn the design pattern and so on. Please point me to the right direction.**\n\n```\nimport numpy as np\nimport math\nimport matplotlib.pyplot as plt\n\nclass Module: pass\n\nclass CubicPolynomial(Module):\n  a: np.float64\n  b: np.float64\n  c: np.float64\n  d: np.float64\n\n  learning_rate: np.float64\n\n  def __init__(self, learning_rate=1e-6) -> None:\n    self.a = np.random.randn()\n    self.b = np.random.randn()\n    self.c = np.random.randn()\n    self.d = np.random.randn()\n\n    self.learning_rate = learning_rate\n\n  def __call__(self, x: np.ndarray):\n    return self.a + self.b * x + self.c * x ** 2 + self.d * x ** 3\n  \n  def loss_ln(self, y_pred: np.ndarray, y: np.ndarray) -> np.float64:\n    return ((y_pred - y) ** 2).sum()\n  \n  def zero_grad(self) -> None:\n    # Manually zero the gradients after updating weights\n    self.grad_a = None\n    self.grad_b = None\n    self.grad_c = None\n    self.grad_d = None\n  \n  def backward(self, y_pred: np.ndarray) -> None:\n    # Backprop to compute gradients of a, b, c, d with respect to loss\n    grad_y_pred = 2.0 * (y_pred - y)\n    # d/da (y_pred - y)²\n    #      2 * (y_pred - y)\n    self.grad_a = grad_y_pred.sum()\n    \n    # d/db (a + (b * x) + (c * x²) + (d * x³) - y)²\n    #      2 * (y_pred - y) * x\n    self.grad_b = (grad_y_pred * x).sum()\n\n    # d/dc (a + (b * x) + (c * x²) + (d * x³) - y)²\n    #      2 * (y_pred - y) * x²\n    self.grad_c = (grad_y_pred * x ** 2).sum()\n\n    # d/dd (a + (b * x) + (c * x²) + (d * x³) - y)²\n    #      2 * (y_pred - y) * x³\n    self.grad_d = (grad_y_pred * x ** 3).sum()\n\n  def step(self) -> None:\n    # Update weights using gradient descent\n    self.a -= self.learning_rate * self.grad_a\n    self.b -= self.learning_rate * self.grad_b\n    self.c -= self.learning_rate * self.grad_c\n    self.d -= self.learning_rate * self.grad_d\n\ndtype = np.float64\n\n# Create random input and output data\nx = np.linspace(-math.pi, math.pi, 2000, dtype=dtype)\ny = np.sin(x)\n\nplt.plot(x, y, 'blue')\n\nlearning_rate = 1e-6\nmodel = CubicPolynomial(learning_rate=learning_rate)\n\nfor t in range(2000):\n  # Forward pass: compute predicted y\n  y_pred = model(x)\n\n  # Compute and print loss\n  loss = model.loss_ln(y_pred, y)\n  if t % 100 == 99:\n    print(t, loss)\n\n  # Before the backward pass, use the optimizer object to zero all of the\n  # gradients for the variables it will update (which are the learnable\n  # weights of the model). This is because by default, gradients are\n  # accumulated in buffers( i.e, not overwritten) whenever .backward()\n  # is called. Checkout docs of torch.autograd.backward for more details.\n  model.zero_grad()\n\n  # Backward pass: compute gradient of the loss with respect to model\n  # parameters\n  model.backward(y_pred)\n\n  # Calling the step function on an Optimizer makes an update to its\n  # parameters\n  model.step()\n\n\nprint(f'Result: y = {model.a} + {model.b} x + {model.c} x^2 + {model.d} x^3')\n\nplt.plot(x, y_pred, 'orange')\n\n%reset -f\n```",
  "label": "r/datascience",
  "dataType": "post",
  "communityName": "r/datascience",
  "datetime": "2024-05-20",
  "username_encoded": "Z0FBQUFBQm5Lakw0NzI3RFdEWlNfQXZVaXRxZHpSV3B2b2RlcFdrYUhKaFRQMnctRkQ0T25vZzFwdGhTUHVFN1Z2YXRSMUJIQ2JfRXRlaUVhSjBMVEVON19NRDMxenh0YUE9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9JLTlDSnpGOVdSaFR1WDFRb1VhWkxhXzVUU1I0bGZQdXc2SHNZdGhDd280a29uSlBXcE8yaldnWWdRZE1ESWxFSFRveHpRUmJOTkNCT3JsSklhWW10cGRSQ2RDdWdOWmU2b2JhU2FOcFQzc3c0Y05VQWJtRWJ2d2UyZ3R3NXI2c1ZBdmxvdEY5alZaYkxabmh3MV9OWGVWZWtfeTFjcEdja0tjMFExQVpNcUpiQ19GU3otS1NWa2x3aUQ3N2hKQWI5bENKTUpYN3RCLWNkd1o4bU5XZ05Ydz09"
}

Entry Information

• Entry ID: 9145
• Repository: Axioma AXP
• Dataset: arrmlet/reddit_dataset_36
• Total Entries: 100,000