Row 41353

Row ID: 41353 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 41353 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Some ideas for you, and you can probably find some references to dive into:

- Synchronous SGD: Some notes \[here\](https://www.cs.ubc.ca/labs/lci/mlrg/slides/MLRG\_Synchronous\_Stochastic\_Gradient.pdf). You might also try searching to data parallelism

- Federated learning has some really neat distributed computing problems in ML

- Model parallelism, iirc AlexNet used this because the model couldn't fit on the GPUs he had. I haven't heard of active research in this area but there could be

- A couple years back when my team was doing contrastive learning, we found that the quality of the model scaled almost linearly with the batch size, which quickly ran us into the GPU memory limit. There might be some clever way to distribute that

There are several different areas on the serving side, such as:

- How to do autoscaling without a big latency hit when scaling up with big models, because they can take a while to load. I once saw a clever paper at NAACL that stored the word embeddings in Dynamo and and the rest of the model was so small, so it loaded really fast and they did queries to Dynamo for the embedding lookups

- Software patterns for models that depend on the outputs from other models: I've heard of this getting very complicated at some companies, like complicated to version, debug, etc. I haven't heard of a term for it though

If you're more into the cloud part but not necessarily the distributed part, there are important topics like detecting drift of the production data vs the training data.

Anyway, just tossing out some ideas - hopefully some of these give you things to search for! And once you find the first paper in an interesting area I recommend Connected Papers to browse the citation graph

FieldValue
text Some ideas for you, and you can probably find some references to dive into: - Synchronous SGD: Some notes \[here\](https://www.cs.ubc.ca/labs/lci/mlrg/slides/MLRG\_Synchronous\_Stochastic\_Gradient.pdf). You might also try searching to data parallelism - Federated learning has some really neat distributed computing problems in ML - Model parallelism, iirc AlexNet used this because the model couldn't fit on the GPUs he had. I haven't heard of active research in this area but there could be - …
label r/machinelearning
dataType comment
communityName r/MachineLearning
datetime 2024-05-22
username_encoded Z0FBQUFBQm5Lak1NRjRieTlrUWhJTUFxMEJpcXFjeThLYWVodGZvdjlmWl8xenpoYzd3S3BORl9ncGhXU0l0RXM2Qk9SVWZEdlB3ODBVMmlfYTAwTHZySXBfMGJsU0swQXc9PQ==
url_encoded Z0FBQUFBQm5Lak9jNnFRUmNxMTZ3RFFtTjM4NFh3MTZua0dtRjJSOW1BSFJYMURRakNwNGZZb1FBWjBiZmktSE1NbG1CbG5yOHB4RThHb1hIVkZaQXFtaVhHNTc0OGxSWkJHZ3JXam1Dc0hia1Bnb19XMW50MXE2MC1lUTItSnpOVTN6SzBOZlNEWUkxUFI3VE9xbkJKdlZ5VU5TV3BCNzlGek81b0tDREM5YVdvMkoxM0FhVnBfOVZ5b3M3TVY4OVYwazJTWDNMa2x3MHpYTVFWdzhOZlkwQ0RqMzBMbElRZXI1c3MxbEFRaU1MaGpIMWZvT0xDND0=

Raw Record

{
  "text": "Some ideas for you, and you can probably find some references to dive into:\n\n- Synchronous SGD: Some notes \\[here\\](https://www.cs.ubc.ca/labs/lci/mlrg/slides/MLRG\\_Synchronous\\_Stochastic\\_Gradient.pdf). You might also try searching to data parallelism\n\n- Federated learning has some really neat distributed computing problems in ML\n\n- Model parallelism, iirc AlexNet used this because the model couldn't fit on the GPUs he had. I haven't heard of active research in this area but there could be\n\n- A couple years back when my team was doing contrastive learning, we found that the quality of the model scaled almost linearly with the batch size, which quickly ran us into the GPU memory limit. There might be some clever way to distribute that\n\nThere are several different areas on the serving side, such as:\n\n- How to do autoscaling without a big latency hit when scaling up with big models, because they can take a while to load. I once saw a clever paper at NAACL that stored the word embeddings in Dynamo and and the rest of the model was so small, so it loaded really fast and they did queries to Dynamo for the embedding lookups\n\n- Software patterns for models that depend on the outputs from other models: I've heard of this getting very complicated at some companies, like complicated to version, debug, etc. I haven't heard of a term for it though\n\nIf you're more into the cloud part but not necessarily the distributed part, there are important topics like detecting drift of the production data vs the training data.\n\n  \nAnyway, just tossing out some ideas - hopefully some of these give you things to search for! And once you find the first paper in an interesting area I recommend Connected Papers to browse the citation graph",
  "label": "r/machinelearning",
  "dataType": "comment",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-22",
  "username_encoded": "Z0FBQUFBQm5Lak1NRjRieTlrUWhJTUFxMEJpcXFjeThLYWVodGZvdjlmWl8xenpoYzd3S3BORl9ncGhXU0l0RXM2Qk9SVWZEdlB3ODBVMmlfYTAwTHZySXBfMGJsU0swQXc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9jNnFRUmNxMTZ3RFFtTjM4NFh3MTZua0dtRjJSOW1BSFJYMURRakNwNGZZb1FBWjBiZmktSE1NbG1CbG5yOHB4RThHb1hIVkZaQXFtaVhHNTc0OGxSWkJHZ3JXam1Dc0hia1Bnb19XMW50MXE2MC1lUTItSnpOVTN6SzBOZlNEWUkxUFI3VE9xbkJKdlZ5VU5TV3BCNzlGek81b0tDREM5YVdvMkoxM0FhVnBfOVZ5b3M3TVY4OVYwazJTWDNMa2x3MHpYTVFWdzhOZlkwQ0RqMzBMbElRZXI1c3MxbEFRaU1MaGpIMWZvT0xDND0="
}
Explore Dataset Explore Row

Entry Information