Row 7609

Row ID: 7609 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 7609 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Hello guys The following code came back and worked perfectly, but using data that he downloaded from him. I tried to change it to use local data of my choice and I did not succeed.The change only applies to the first function if possible . Is there any help with it?

Thanks in advance

The code

import torch from torch.utils.data import random_split, DataLoader from torchvision.transforms import ToTensor, Normalize, Compose from torchvision.datasets import MNIST

def get_mnist(data_path: str = "./data"): """Download MNIST and apply minimal transformation."""

tr = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])

trainset = MNIST(data_path, train=True, download=True, transform=tr) testset = MNIST(data_path, train=False, download=True, transform=tr)

return trainset, testset

def prepare_dataset(num_partitions: int, batch_size: int, val_ratio: float = 0.1): """Download MNIST and generate IID partitions."""

# download MNIST in case it's not already in the system trainset, testset = get_mnist()

# split trainset into `num_partitions` trainsets (one per client) # figure out number of training examples per partition num_images = len(trainset) // num_partitions

# a list of partition lenghts (all partitions are of equal size) partition_len = [num_images] * num_partitions

# split randomly. This returns a list of trainsets, each with `num_images` training examples # Note this is the simplest way of splitting this dataset. A more realistic (but more challenging) partitioning # would induce heterogeneity in the partitions in the form of for example: each client getting a different # amount of training examples, each client having a different distribution over the labels (maybe even some # clients not having a single training example for certain classes). If you are curious, you can check online # for Dirichlet (LDA) or pathological dataset partitioning in FL. A place to start is: https://arxiv.org/abs/1909.06335 trainsets = random_split( trainset, partition_len, torch.Generator().manual_seed(2023) )

# create dataloaders with train+val support trainloaders = [] valloaders = [] # for each train set, let's put aside some training examples for validation for trainset_ in trainsets: num_total = len(trainset_) num_val = int(val_ratio * num_total) num_train = num_total - num_val

for_train, for_val = random_split( trainset_, [num_train, num_val], torch.Generator().manual_seed(2023) )

# construct data loaders and append to their respective list. # In this way, the i-th client will get the i-th element in the trainloaders list and the i-th element in the valloaders list trainloaders.append( DataLoader(for_train, batch_size=batch_size, shuffle=True, num_workers=2) ) valloaders.append( DataLoader(for_val, batch_size=batch_size, shuffle=False, num_workers=2) )

# We leave the test set intact (i.e. we don't partition it) # This test set will be left on the server side and we'll be used to evaluate the # performance of the global model after each round. # Please note that a more realistic setting would instead use a validation set on the server for # this purpose and only use the testset after the final round. # Also, in some settings (specially outside simulation) it might not be feasible to construct a validation # set on the server side, therefore evaluating the global model can only be done by the clients. (see the comment # in main.py above the strategy definition for more details on this) testloader = DataLoader(testset, batch_size=128)

return trainloaders, valloaders, testloader

FieldValue
text Hello guys The following code came back and worked perfectly, but using data that he downloaded from him. I tried to change it to use local data of my choice and I did not succeed.The change only applies to the first function if possible . Is there any help with it? Thanks in advance The code import torch from torch.utils.data import random_split, DataLoader from torchvision.transforms import ToTensor, Normalize, Compose from torchvision.datasets import MNIST def get_mnist(data_path: s…
label r/pytorch
dataType post
communityName r/pytorch
datetime 2024-05-16
username_encoded Z0FBQUFBQm5LakwzTTZoaXlEb0RYUkZvV01iZU1iSHR6d2RETXJjUUptVEdTOFVBOW5iWGZYZHIxTl9IeXJHQ3hOeEtobjFZQjhPYkRiVE9paVo0bjBWUkJCVG5oTlZZWUE9PQ==
url_encoded Z0FBQUFBQm5Lak9IZjNxNGVkTnBOSUExbHNsbXlJVmtlc3FuMzBnN2tJekIxV0VodkwzNVFJT0xDVUR4ZW9PUDlPRm16MWIza2VnLXJqbjNnczRUcXNEMW8tY0tmNWY0YjA4T05lOWZrcE5FZVZNZmFFaFVxUUVDRVk2Z01tb21jNjdXN2t6cEx0eGY3UGx2QktOZFNqdEViMlF3X09iT2NVTnZ0eFV0dUhEbjEyaEduY011YTBrPQ==

Raw Record

{
  "text": "Hello guys  The following code came back and worked perfectly, but using data that he downloaded from him. I tried to change it to use local data of my choice and I did not succeed.The change only applies to the first function if possible . Is there any help with it?  \n\nThanks in advance \n\nThe code\n\n\nimport torch\nfrom torch.utils.data import random_split, DataLoader\nfrom torchvision.transforms import ToTensor, Normalize, Compose\nfrom torchvision.datasets import MNIST\n\n\ndef get_mnist(data_path: str = \"./data\"):\n    \"\"\"Download MNIST and apply minimal transformation.\"\"\"\n\n    tr = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])\n\n    trainset = MNIST(data_path, train=True, download=True, transform=tr)\n    testset = MNIST(data_path, train=False, download=True, transform=tr)\n\n    return trainset, testset\n\n\ndef prepare_dataset(num_partitions: int, batch_size: int, val_ratio: float = 0.1):\n    \"\"\"Download MNIST and generate IID partitions.\"\"\"\n\n    # download MNIST in case it's not already in the system\n    trainset, testset = get_mnist()\n\n    # split trainset into `num_partitions` trainsets (one per client)\n    # figure out number of training examples per partition\n    num_images = len(trainset) // num_partitions\n\n    # a list of partition lenghts (all partitions are of equal size)\n    partition_len = [num_images] * num_partitions\n\n    # split randomly. This returns a list of trainsets, each with `num_images` training examples\n    # Note this is the simplest way of splitting this dataset. A more realistic (but more challenging) partitioning\n    # would induce heterogeneity in the partitions in the form of for example: each client getting a different\n    # amount of training examples, each client having a different distribution over the labels (maybe even some\n    # clients not having a single training example for certain classes). If you are curious, you can check online\n    # for Dirichlet (LDA) or pathological dataset partitioning in FL. A place to start is: https://arxiv.org/abs/1909.06335\n    trainsets = random_split(\n        trainset, partition_len, torch.Generator().manual_seed(2023)\n    )\n\n    # create dataloaders with train+val support\n    trainloaders = []\n    valloaders = []\n    # for each train set, let's put aside some training examples for validation\n    for trainset_ in trainsets:\n        num_total = len(trainset_)\n        num_val = int(val_ratio * num_total)\n        num_train = num_total - num_val\n\n        for_train, for_val = random_split(\n            trainset_, [num_train, num_val], torch.Generator().manual_seed(2023)\n        )\n\n        # construct data loaders and append to their respective list.\n        # In this way, the i-th client will get the i-th element in the trainloaders list and the i-th element in the valloaders list\n        trainloaders.append(\n            DataLoader(for_train, batch_size=batch_size, shuffle=True, num_workers=2)\n        )\n        valloaders.append(\n            DataLoader(for_val, batch_size=batch_size, shuffle=False, num_workers=2)\n        )\n\n    # We leave the test set intact (i.e. we don't partition it)\n    # This test set will be left on the server side and we'll be used to evaluate the\n    # performance of the global model after each round.\n    # Please note that a more realistic setting would instead use a validation set on the server for\n    # this purpose and only use the testset after the final round.\n    # Also, in some settings (specially outside simulation) it might not be feasible to construct a validation\n    # set on the server side, therefore evaluating the global model can only be done by the clients. (see the comment\n    # in main.py above the strategy definition for more details on this)\n    testloader = DataLoader(testset, batch_size=128)\n\n    return trainloaders, valloaders, testloader",
  "label": "r/pytorch",
  "dataType": "post",
  "communityName": "r/pytorch",
  "datetime": "2024-05-16",
  "username_encoded": "Z0FBQUFBQm5LakwzTTZoaXlEb0RYUkZvV01iZU1iSHR6d2RETXJjUUptVEdTOFVBOW5iWGZYZHIxTl9IeXJHQ3hOeEtobjFZQjhPYkRiVE9paVo0bjBWUkJCVG5oTlZZWUE9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9IZjNxNGVkTnBOSUExbHNsbXlJVmtlc3FuMzBnN2tJekIxV0VodkwzNVFJT0xDVUR4ZW9PUDlPRm16MWIza2VnLXJqbjNnczRUcXNEMW8tY0tmNWY0YjA4T05lOWZrcE5FZVZNZmFFaFVxUUVDRVk2Z01tb21jNjdXN2t6cEx0eGY3UGx2QktOZFNqdEViMlF3X09iT2NVTnZ0eFV0dUhEbjEyaEduY011YTBrPQ=="
}
Explore Dataset Explore Row

Entry Information