Row 6577

Row ID: 6577 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 6577 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Hello everyone. I’m currently have issues with my GPU drivers. I’m using 2 T4 GPUs on GCP and everything works great for a while…

Out of the blue when I try to retrain a model I get the error “UserWarning: can’t initialize NVML” when this happens I can’t even run nvidia-smi to make sure things are working. I’ve attempted to uninstall and reinstall the drivers but it gets followed with an error in dkm files. Anyone have any tips?

One more thing to note is when I cleanup it seems only rank 0 gets clean and it leaves the program running.

Any help would be appreciated

FieldValue
text Hello everyone. I’m currently have issues with my GPU drivers. I’m using 2 T4 GPUs on GCP and everything works great for a while… Out of the blue when I try to retrain a model I get the error “UserWarning: can’t initialize NVML” when this happens I can’t even run nvidia-smi to make sure things are working. I’ve attempted to uninstall and reinstall the drivers but it gets followed with an error in dkm files. Anyone have any tips? One more thing to note is when I cleanup it seems only rank 0 get…
label r/deeplearning
dataType post
communityName r/deeplearning
datetime 2024-05-11
username_encoded Z0FBQUFBQm5LakwzN3NtTG9GQmI4Znp6QUxZdnVQVzJ4UVQwTXhpMHBTcEtURElFQ0dXdWJxYzIzUUhWdWRueVN4S2pkcVJLdWdNcWxCSEFfWjd4dU9obGM0aVR1bHZJWWc9PQ==
url_encoded Z0FBQUFBQm5Lak9HaUw1NVIzbUZBckVhek9LdVNGdkJZTEFfblhUTVBHUzNILV9XSEJYOUhKWkstampCM0xjbTB3ZEVfeFhoMUF4eGJoZ3daeGtqNkJxVS1QOE00c2FzaTZGcW5PZjdwYnZYLUgwemhWdHNTcGhZcFJkNFRqVE9TZlZOZy02Y25jVkFHRHZVenJlb2J6Z0lZbVhZTEo1MlJ1VGFFaC00clBYelRHeU42S0gzVUJzOXFaSUZyYng3aDE2UHV0amI0Q01rTG5vR1dZeVBoWUl2WmR6dkhSZ0lKQT09

Raw Record

{
  "text": "Hello everyone. I’m currently have issues with my GPU drivers. I’m using 2 T4 GPUs on GCP and everything works great for a while…\n\nOut of the blue when I try to retrain a model I get the error “UserWarning: can’t initialize NVML” when this happens I can’t even run nvidia-smi to make sure things are working. I’ve attempted to uninstall and reinstall the drivers but it gets followed with an error in dkm files. Anyone have any tips?\n\nOne more thing to note is when I cleanup it seems only rank 0 gets clean and it leaves the program running. \n\nAny help would be appreciated",
  "label": "r/deeplearning",
  "dataType": "post",
  "communityName": "r/deeplearning",
  "datetime": "2024-05-11",
  "username_encoded": "Z0FBQUFBQm5LakwzN3NtTG9GQmI4Znp6QUxZdnVQVzJ4UVQwTXhpMHBTcEtURElFQ0dXdWJxYzIzUUhWdWRueVN4S2pkcVJLdWdNcWxCSEFfWjd4dU9obGM0aVR1bHZJWWc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HaUw1NVIzbUZBckVhek9LdVNGdkJZTEFfblhUTVBHUzNILV9XSEJYOUhKWkstampCM0xjbTB3ZEVfeFhoMUF4eGJoZ3daeGtqNkJxVS1QOE00c2FzaTZGcW5PZjdwYnZYLUgwemhWdHNTcGhZcFJkNFRqVE9TZlZOZy02Y25jVkFHRHZVenJlb2J6Z0lZbVhZTEo1MlJ1VGFFaC00clBYelRHeU42S0gzVUJzOXFaSUZyYng3aDE2UHV0amI0Q01rTG5vR1dZeVBoWUl2WmR6dkhSZ0lKQT09"
}
Explore Dataset Explore Row

Entry Information