Row 3864

Row ID: 3864 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 3864 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I am using an LLM to generate text for inference. I have a lot of resources and the model computation is being distributed over multiple GPUs but its using a very small portion of VRAM of what is available.

Imagine the code to be something like:

from transformers import Model, Tokenizer model = Model() tokenizer = Tokenizer() prompt = "What is life?" encoded_prompt = tokenizer.encode(prompt) response = model(encoded_prompt)

I am using an LLM to generate text for inference. I have a lot of resources and the model computation is being distributed over multiple GPUs but it's using a very small portion of VRAM of what is available.

https://preview.redd.it/1ylsrj5qhvqc1.png?width=822&format=png&auto=webp&s=3eb2efcfdd5d0dddc7b7938c261e12aa4d08c325

Is there any way to speed up the inference?

FieldValue
text I am using an LLM to generate text for inference. I have a lot of resources and the model computation is being distributed over multiple GPUs but its using a very small portion of VRAM of what is available. Imagine the code to be something like: from transformers import Model, Tokenizer model = Model() tokenizer = Tokenizer() prompt = "What is life?" encoded_prompt = tokenizer.encode(prompt) response = model(encoded_prompt) I am using an LLM to generate …
label r/pytorch
dataType post
communityName r/pytorch
datetime 2024-03-27
username_encoded Z0FBQUFBQm5LakwxRXlHOEk4Yy12WFZxSEtLMHlJcUNtV0xOanNldkNybmVtbjNlQzhUUWc0WkZHZ3UwTTJNeUR4c3o2UmtoZF9MX195SUNFUlRqTThVcGxiaGU2X2dYU0E9PQ==
url_encoded Z0FBQUFBQm5Lak9FaGNUWHJsV3hSVWhoQXZwbk9vem14YmhVLWw0Qk5sOXBZWEQ0RDRKa2twVFA4UFZhbmhkRDQ3X0NEdkFzckdkMUNPQnZhb1VGQ2dzVkNhcmJNLVp3VUtiU1JFVFRxc0ZTY01tUFg4eGJhWHU1ODBsZ0FCUjl2RHJWaE51RXo4QUV0eUctZlVvVjhuUnhuYnRwZTk2bG1DdENtT1FkQUJTcTVXRFJpM21DVGdJPQ==

Raw Record

{
  "text": "I am using an LLM to generate text for inference. I have a lot of resources and the model computation is being distributed over multiple GPUs but its using a very small portion of VRAM of what is available.\n\nImagine the code to be something like:\n\n    from transformers import Model, Tokenizer\n    \n    model = Model()\n    tokenizer = Tokenizer()\n    \n    prompt = \"What is life?\"\n    encoded_prompt = tokenizer.encode(prompt)\n    \n    response = model(encoded_prompt)\n\nI am using an LLM to generate text for inference. I have a lot of resources and the model computation is being distributed over multiple GPUs but it's using a very small portion of VRAM of what is available.\n\nhttps://preview.redd.it/1ylsrj5qhvqc1.png?width=822&format=png&auto=webp&s=3eb2efcfdd5d0dddc7b7938c261e12aa4d08c325\n\nIs there any way to speed up the inference?",
  "label": "r/pytorch",
  "dataType": "post",
  "communityName": "r/pytorch",
  "datetime": "2024-03-27",
  "username_encoded": "Z0FBQUFBQm5LakwxRXlHOEk4Yy12WFZxSEtLMHlJcUNtV0xOanNldkNybmVtbjNlQzhUUWc0WkZHZ3UwTTJNeUR4c3o2UmtoZF9MX195SUNFUlRqTThVcGxiaGU2X2dYU0E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9FaGNUWHJsV3hSVWhoQXZwbk9vem14YmhVLWw0Qk5sOXBZWEQ0RDRKa2twVFA4UFZhbmhkRDQ3X0NEdkFzckdkMUNPQnZhb1VGQ2dzVkNhcmJNLVp3VUtiU1JFVFRxc0ZTY01tUFg4eGJhWHU1ODBsZ0FCUjl2RHJWaE51RXo4QUV0eUctZlVvVjhuUnhuYnRwZTk2bG1DdENtT1FkQUJTcTVXRFJpM21DVGdJPQ=="
}
Explore Dataset Explore Row

Entry Information