Row 8466

Row ID: 8466 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 8466 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

How large is your dataset (in terms of GB)?

I think the issue might be connected to the data loading. Do you load all data in the Dataset __init__? If so, every data loader (each worker) will copy the data. That works for small DSs but if you have multiple GB and they get copied over multiple workers, thats too much. Try to load only part of the data, one sample or so. For images you can find examples online that do the image reading in the __getitem__ call specifically for this reason. Of course for tabular data you'd need to be a little creative, use a different format (I think parquet allows partial reading and is a table-like format?).

FieldValue
text How large is your dataset (in terms of GB)? I think the issue might be connected to the data loading. Do you load all data in the Dataset __init__? If so, every data loader (each worker) will copy the data. That works for small DSs but if you have multiple GB and they get copied over multiple workers, thats too much. Try to load only part of the data, one sample or so. For images you can find examples online that do the image reading in the __getitem__ call specifically for this reason. Of cour…
label r/deeplearning
dataType comment
communityName r/deeplearning
datetime 2024-05-19
username_encoded Z0FBQUFBQm5Lakw0QkJJdlA4VkdwMWFzUThDa2c4UU9WWnhMR0NrX2tRaEhiT3hZaGc3cW93RFo2Z3V3cmRiRVhkZ1NneTNxeS1ianNoUjdiWUJVcnFSVHc0U3BsMU53OUE9PQ==
url_encoded Z0FBQUFBQm5Lak9IanBKZWJUSE1TMG5FQ2xEcnFNR0RUTFFlX3I4bE1RVktjVHBZSUYtd3prWWxWdEdzMFZieXg5WnFpSHo0bkF6cWIzaWppRzZVbmcxM1cyaVdfU2hkNzRsN3FNZUxuT3dfQjRBZDk0VXZXa2ZvU2ZCVDZqOEtwWmJZdzR3amtGZmZXWUpxMEhNSHJFUWNOTXlSWE1KN0pITk9IckFXbUVxUVpsaWNvYWVfOUYxXy13NVlQT3drRGMyRm0wU21SaEZhU3E0ZjhBbVFQMVM1R2tTMGVyUlMxZz09

Raw Record

{
  "text": "How large is your dataset (in terms of GB)?\n\nI think the issue might be connected to the data loading. Do you load all data in the Dataset __init__? If so, every data loader (each worker) will copy the data. That works for small DSs but if you have multiple GB and they get copied over multiple workers, thats too much. Try to load only part of the data, one sample or so. For images you can find examples online that do the image reading in the __getitem__ call specifically for this reason.\nOf course for tabular data you'd need to be a little creative, use a different format (I think parquet allows partial reading and is a table-like format?).",
  "label": "r/deeplearning",
  "dataType": "comment",
  "communityName": "r/deeplearning",
  "datetime": "2024-05-19",
  "username_encoded": "Z0FBQUFBQm5Lakw0QkJJdlA4VkdwMWFzUThDa2c4UU9WWnhMR0NrX2tRaEhiT3hZaGc3cW93RFo2Z3V3cmRiRVhkZ1NneTNxeS1ianNoUjdiWUJVcnFSVHc0U3BsMU53OUE9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9IanBKZWJUSE1TMG5FQ2xEcnFNR0RUTFFlX3I4bE1RVktjVHBZSUYtd3prWWxWdEdzMFZieXg5WnFpSHo0bkF6cWIzaWppRzZVbmcxM1cyaVdfU2hkNzRsN3FNZUxuT3dfQjRBZDk0VXZXa2ZvU2ZCVDZqOEtwWmJZdzR3amtGZmZXWUpxMEhNSHJFUWNOTXlSWE1KN0pITk9IckFXbUVxUVpsaWNvYWVfOUYxXy13NVlQT3drRGMyRm0wU21SaEZhU3E0ZjhBbVFQMVM1R2tTMGVyUlMxZz09"
}
Explore Dataset Explore Row

Entry Information