Row 6279

Row ID: 6279 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 6279 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I successfuly have reproduced the example from pytorch that combines Tensor parallelism + fsdp. However the example is using multiple GPUs for a single node.

torchrun --nnodes=1 --nproc_per_node=${2:-4} --rdzv_id=101 --rdzv_endpoint="localhost:5972" ${1:-fsdp_tp_example.py}

How can I do the same example with multiple nodes (4 GPUs for each node)? Shard the model and data across different nodes.

https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py

FieldValue
text I successfuly have reproduced the example from pytorch that combines Tensor parallelism + fsdp. However the example is using multiple GPUs for a single node. torchrun --nnodes=1 --nproc_per_node=${2:-4} --rdzv_id=101 --rdzv_endpoint="localhost:5972" ${1:-fsdp_tp_example.py} How can I do the same example with multiple nodes (4 GPUs for each node)? Shard the model and data across different nodes. https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py
label r/pytorch
dataType post
communityName r/pytorch
datetime 2024-05-09
username_encoded Z0FBQUFBQm5LakwyUk01dmdoVWdQM1dNeWwxY0ZRaVhQZllWUmpqdjl3WnVORVZmLXg3QTJUQi04UnRxcGJSZjVvckplcGlCTTFiOVhTUWE0dDdTVFRBbFpHc3JteFZGOHc9PQ==
url_encoded Z0FBQUFBQm5Lak9HenBOWGNfbkItUnhGWVNORnBlQmswTnZkeDBHRVRRMFk5MWU4QmRrVTJqcEtSOVpKVEJZcVNrbzUyVWNHMVFzdXBHMW9ra09acFNFTzVSYjBKT3JzUS1GVm1Pb1pEX19yenFZTjUwVjlUUzNTai1mVXBPbHc4RzBvRi1FelM4SGthZDJzYmdlSlVPTFJ6SzU5dGFLamNjRi1NYklTVUhUYjdtUWRtcEpDbHBxdXg3YW5RT2gwYk5VdHhzckNYZ2Q1

Raw Record

{
  "text": "I successfuly have reproduced the example from pytorch that combines Tensor parallelism + fsdp. However the example is using multiple GPUs for a single node.\n\ntorchrun --nnodes=1 --nproc_per_node=${2:-4} --rdzv_id=101 --rdzv_endpoint=\"localhost:5972\" ${1:-fsdp_tp_example.py}\n\nHow can I do the same example with multiple nodes (4 GPUs for each node)? Shard the model and data across different nodes.\n\nhttps://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py",
  "label": "r/pytorch",
  "dataType": "post",
  "communityName": "r/pytorch",
  "datetime": "2024-05-09",
  "username_encoded": "Z0FBQUFBQm5LakwyUk01dmdoVWdQM1dNeWwxY0ZRaVhQZllWUmpqdjl3WnVORVZmLXg3QTJUQi04UnRxcGJSZjVvckplcGlCTTFiOVhTUWE0dDdTVFRBbFpHc3JteFZGOHc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HenBOWGNfbkItUnhGWVNORnBlQmswTnZkeDBHRVRRMFk5MWU4QmRrVTJqcEtSOVpKVEJZcVNrbzUyVWNHMVFzdXBHMW9ra09acFNFTzVSYjBKT3JzUS1GVm1Pb1pEX19yenFZTjUwVjlUUzNTai1mVXBPbHc4RzBvRi1FelM4SGthZDJzYmdlSlVPTFJ6SzU5dGFLamNjRi1NYklTVUhUYjdtUWRtcEpDbHBxdXg3YW5RT2gwYk5VdHhzckNYZ2Q1"
}
Explore Dataset Explore Row

Entry Information