Row 6134

Row ID: 6134 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 6134 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

So the title speaks for itself

import torch import torchvision import torchvision.transforms as transforms torch.autograd.set_detect_anomaly(True) # Transformations to be applied to the dataset transform = transforms.Compose([     transforms.ToTensor() ]) # Download CIFAR-10 dataset and apply transformations trainset = torchvision.datasets.CIFAR10(root='./data', train=True,                                         download=True, transform=transform) trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,                                           shuffle=True, num_workers=2) testset = torchvision.datasets.CIFAR10(root='./data', train=False,                                        download=True, transform=transform) testloader = torch.utils.data.DataLoader(testset, batch_size=4,                                          shuffle=False, num_workers=2) X_train = trainset.data y_train = trainset.targets X_train = torch.from_numpy(X_train) y_train = torch.tensor(y_train) y_train_encoded =  torch.eye(len(trainset.classes))[y_train] X_train_norm = X_train / 255.0 def loss(batch_labels, labels):     # Ensure shapes are compatible     assert batch_labels.shape == labels.shape         # Add a small epsilon to prevent taking log(0)     epsilon = 1e-10         # Compute log probabilities for all samples in the batch     log_probs = torch.log(batch_labels + epsilon)         # Check for NaN values in log probabilities     if torch.isnan(log_probs).any():         raise ValueError("NaN values encountered in log computation.")         # Compute element-wise product and sum to get the loss     loss = -torch.sum(labels * log_probs)         # Check for NaN values in the loss     if torch.isnan(loss).any():         raise ValueError("NaN values encountered in loss computation.")         return loss def softmax(A):     """     A: shape (n, m) m is batch_size     """     # Subtract the maximum value from each element in A     max_A = torch.max(A, axis=0).values     A_shifted = A - max_A         # Exponentiate the shifted values     exp_A = torch.exp(A_shifted)         # Compute the sum of exponentiated values     sums = torch.sum(exp_A, axis=0)         # Add a small constant to prevent division by zero     epsilon = 1e-10     sums += epsilon         # Compute softmax probabilities     softmax_A = exp_A / sums         if torch.isnan(softmax_A).any():         raise ValueError("NaN values encountered in softmax computation.")         return softmax_A def linear(X, W, b):     return W @ X.T + b batch_size = 64 batches = X_train.shape[0] // batch_size lr = 0.01 W = torch.randn((len(trainset.classes), X_train.shape[1] * X_train.shape[1] * X_train.shape[-1]), requires_grad=True) b = torch.randn(((len(trainset.classes), 1)), requires_grad=True) for batch in range(batches - 1):     start = batch * batch_size     end = (batch + 1) * (batch_size)     mini_batch = X_train_norm[start : end, :].reshape(batch_size, -1)     mini_batch_labels = y_train_encoded[start : end]     A = linear(mini_batch, W, b)     Y_hat = softmax(A)     if torch.isnan(Y_hat).any():         raise ValueError("NaN values encountered in softmax output.")         #print(Y_hat.shape, mini_batch_labels.shape)     loss_ = loss(Y_hat.T, mini_batch_labels)     if torch.isnan(loss_):         raise ValueError("NaN values encountered in loss.")         #print("W_grad is", W.grad)     loss_.retain_grad()     loss_.backward()     print(loss_)     print(W.grad)     W = W - lr * W.grad     b = b - lr * b.grad     print(W.grad)       W.grad.zero_()     b.grad.zero_()     break

And the ouput is the following. The interesting part is that initially it is computed as needed but when I try to update it becomes None

Files already downloaded and verified Files already downloaded and verified tensor(991.7662, grad_fn=<NegBackward0>) tensor([[-0.7668, -0.7793, -0.7611, ..., -0.9380, -0.9324, -0.9519], [-0.6169, -0.5180, -0.5080, ..., -0.2189, -0.1080, -0.4107], [-0.8191, -0.7615, -0.4608, ..., -1.3017, -1.1424, -0.9967], ..., [ 0.2391, -0.1126, -0.2533, ..., -0.1137, -0.3375, -0.3346], [ 1.2962, 1.2075, 0.9185, ..., 1.5164, 1.3121, 1.0945], [-0.7181, -1.0163, -1.3664, ..., 0.2474, 0.2026, 0.2986]]) None <ipython-input-3-d8bbcbd68506>:120: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.) print(W.grad) <ipython-input-3-d8bbcbd68506>:122: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.) W.grad.zero_() --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in <cell line: 96>() 120 print(W.grad) 121 --> 122 W.grad.zero_() 123 b.grad.zero_() 124 break <ipython-input-3-d8bbcbd68506> AttributeError: 'NoneType' object has no attribute 'zero_'

FieldValue
text So the title speaks for itself import torch import torchvision import torchvision.transforms as transforms torch.autograd.set_detect_anomaly(True) # Transformations to be applied to the dataset transform = transforms.Compose([     transforms.ToTensor() ]) # Download CIFAR-10 dataset and apply transformations trainset = torchvision.datasets.CIFAR10(root='./data', train=True,                                         download=True, transfor…
label r/pytorch
dataType post
communityName r/pytorch
datetime 2024-05-07
username_encoded Z0FBQUFBQm5LakwycnF6anJyUnNHWXJENWR3RWJIbUVXN1BPeTRNakp4dWU3djJQQzJtT01yeXFENmoya1IzY1hVSTNZcy1lV003d0NxX01xeUlYN1ROeGNfZXMtX21qNHVxcmtRVlJnWWJGREMyYU0yUXpsUTg9
url_encoded Z0FBQUFBQm5Lak9HTDhfZ1hpQlp3RE14c19FR0dzcHFFYmdKMUxJM3NpMko1VXpLMUFvQ2hXbmNraEdOQlpFQjJ6NzVYNkF5Mk53YUtva1VnMFVJXzdMLUpTWmRzZWN2RGRKTjZER1JpY2p0ZHAzX2tfdEcyNHJ6QVRMaWJLRmVzeWF1WVBQTWtoaFNneXBCNnFXdGU0RnhqdmN0dGZaRWIwM1hUN3N1emJmbHBNbTZXeTZwVU5VeGpjQjRKMWtyenQzNmhIYW16ZENl

Raw Record

{
  "text": "So the title speaks for itself\n\n    import torch\n    import torchvision\n    import torchvision.transforms as transforms\n    \n    torch.autograd.set_detect_anomaly(True)\n    \n    # Transformations to be applied to the dataset\n    transform = transforms.Compose([\n        transforms.ToTensor()\n    ])\n    \n    # Download CIFAR-10 dataset and apply transformations\n    trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n                                            download=True, transform=transform)\n    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,\n                                              shuffle=True, num_workers=2)\n    \n    testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n                                           download=True, transform=transform)\n    testloader = torch.utils.data.DataLoader(testset, batch_size=4,\n                                             shuffle=False, num_workers=2)\n    \n    X_train = trainset.data\n    y_train = trainset.targets\n    \n    X_train = torch.from_numpy(X_train)\n    y_train = torch.tensor(y_train)\n    \n    \n    y_train_encoded =  torch.eye(len(trainset.classes))[y_train]\n    \n    X_train_norm = X_train / 255.0\n    \n    def loss(batch_labels, labels):\n        # Ensure shapes are compatible\n        assert batch_labels.shape == labels.shape\n        \n        # Add a small epsilon to prevent taking log(0)\n        epsilon = 1e-10\n        \n        # Compute log probabilities for all samples in the batch\n        log_probs = torch.log(batch_labels + epsilon)\n        \n        # Check for NaN values in log probabilities\n        if torch.isnan(log_probs).any():\n            raise ValueError(\"NaN values encountered in log computation.\")\n        \n        # Compute element-wise product and sum to get the loss\n        loss = -torch.sum(labels * log_probs)\n        \n        # Check for NaN values in the loss\n        if torch.isnan(loss).any():\n            raise ValueError(\"NaN values encountered in loss computation.\")\n        \n        return loss\n    \n    def softmax(A):\n        \"\"\"\n        A: shape (n, m) m is batch_size\n        \"\"\"\n        # Subtract the maximum value from each element in A\n        max_A = torch.max(A, axis=0).values\n        A_shifted = A - max_A\n        \n        # Exponentiate the shifted values\n        exp_A = torch.exp(A_shifted)\n        \n        # Compute the sum of exponentiated values\n        sums = torch.sum(exp_A, axis=0)\n        \n        # Add a small constant to prevent division by zero\n        epsilon = 1e-10\n        sums += epsilon\n        \n        # Compute softmax probabilities\n        softmax_A = exp_A / sums\n        \n        if torch.isnan(softmax_A).any():\n            raise ValueError(\"NaN values encountered in softmax computation.\")\n        \n        return softmax_A\n    \n    def linear(X, W, b):\n        return W @ X.T + b \n    \n    \n    batch_size = 64\n    batches = X_train.shape[0] // batch_size\n    lr = 0.01\n    \n    \n    W = torch.randn((len(trainset.classes), X_train.shape[1] * X_train.shape[1] * X_train.shape[-1]), requires_grad=True)\n    b = torch.randn(((len(trainset.classes), 1)), requires_grad=True)\n    \n    \n    for batch in range(batches - 1):\n        start = batch * batch_size\n        end = (batch + 1) * (batch_size)\n        mini_batch = X_train_norm[start : end, :].reshape(batch_size, -1)\n        mini_batch_labels = y_train_encoded[start : end]\n    \n        A = linear(mini_batch, W, b)\n        Y_hat = softmax(A)\n        if torch.isnan(Y_hat).any():\n            raise ValueError(\"NaN values encountered in softmax output.\")\n        \n        #print(Y_hat.shape, mini_batch_labels.shape)\n        loss_ = loss(Y_hat.T, mini_batch_labels)\n        if torch.isnan(loss_):\n            raise ValueError(\"NaN values encountered in loss.\")\n        \n        #print(\"W_grad is\", W.grad)\n        loss_.retain_grad()\n        loss_.backward()\n        print(loss_)\n        print(W.grad)\n        W = W - lr * W.grad\n        b = b - lr * b.grad\n    \n        print(W.grad)  \n    \n        W.grad.zero_()\n        b.grad.zero_()\n    \n        break\n\nAnd the ouput is the following. The interesting part is that initially it is computed as needed but when I try to update it becomes None\n\n    Files already downloaded and verified\n    Files already downloaded and verified\n    tensor(991.7662, grad_fn=<NegBackward0>)\n    tensor([[-0.7668, -0.7793, -0.7611,  ..., -0.9380, -0.9324, -0.9519],\n            [-0.6169, -0.5180, -0.5080,  ..., -0.2189, -0.1080, -0.4107],\n            [-0.8191, -0.7615, -0.4608,  ..., -1.3017, -1.1424, -0.9967],\n            ...,\n            [ 0.2391, -0.1126, -0.2533,  ..., -0.1137, -0.3375, -0.3346],\n            [ 1.2962,  1.2075,  0.9185,  ...,  1.5164,  1.3121,  1.0945],\n            [-0.7181, -1.0163, -1.3664,  ...,  0.2474,  0.2026,  0.2986]])\n    None\n    \n    <ipython-input-3-d8bbcbd68506>:120: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)\n      print(W.grad)\n    <ipython-input-3-d8bbcbd68506>:122: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)\n      W.grad.zero_()\n    ---------------------------------------------------------------------------\n    AttributeError                            Traceback (most recent call last)\n    in <cell line: 96>()\n        120     print(W.grad)\n        121 \n    --> 122     W.grad.zero_()\n        123     b.grad.zero_()\n        124     break\n    \n    <ipython-input-3-d8bbcbd68506>\n    AttributeError: 'NoneType' object has no attribute 'zero_'",
  "label": "r/pytorch",
  "dataType": "post",
  "communityName": "r/pytorch",
  "datetime": "2024-05-07",
  "username_encoded": "Z0FBQUFBQm5LakwycnF6anJyUnNHWXJENWR3RWJIbUVXN1BPeTRNakp4dWU3djJQQzJtT01yeXFENmoya1IzY1hVSTNZcy1lV003d0NxX01xeUlYN1ROeGNfZXMtX21qNHVxcmtRVlJnWWJGREMyYU0yUXpsUTg9",
  "url_encoded": "Z0FBQUFBQm5Lak9HTDhfZ1hpQlp3RE14c19FR0dzcHFFYmdKMUxJM3NpMko1VXpLMUFvQ2hXbmNraEdOQlpFQjJ6NzVYNkF5Mk53YUtva1VnMFVJXzdMLUpTWmRzZWN2RGRKTjZER1JpY2p0ZHAzX2tfdEcyNHJ6QVRMaWJLRmVzeWF1WVBQTWtoaFNneXBCNnFXdGU0RnhqdmN0dGZaRWIwM1hUN3N1emJmbHBNbTZXeTZwVU5VeGpjQjRKMWtyenQzNmhIYW16ZENl"
}
Explore Dataset Explore Row

Entry Information