Row 8546

Row ID: 8546 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 8546 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I want to create AI that generate openpose from textual description for example if input "a man running" output would be like the image I provided Is there any model architecture recommend for me?

my data condition is

* canvas\_width: 900px * canvas\_height: 300px * frames: 5 (5 person)

[expected output](https://drive.google.com/file/d/1vNcc5-zxpGodaGG3vat3TSPkbuC6olMK/view?usp=sharing)

I trying to train RNN for this task and I use sentence transformer for embedding text and then pass to RNN and the loss is look like image below

from sentence_transformers import SentenceTransformer sentence_model = SentenceTransformer("all-MiniLM-L6-v2") text = "a man running" text_input = torch.tensor(sentence_model.encode(text), dtype=torch.float)

[loss image with num\_layers=3](https://drive.google.com/file/d/1580_OdM146PefgIeRrK-Cmn-mIU_7AwB/view?usp=sharing)

My RNN setting

embedding_dim = 384 hidden_dim = 512 num_layers = 3 output_dim = 180 num_epochs = 100 learning_rate = 0.001 rnn_model = RNN(embedding_dim, hidden_dim, num_layers, output_dim)

but the problem is whatever I input the output is the same everytime! but when I try changing num\_layers to 1 and keep other setting the same like this

embedding_dim = 384 hidden_dim = 512 num_layers = 1 output_dim = 180 num_epochs = 100 learning_rate = 0.001 rnn_model = RNN(embedding_dim, hidden_dim, num_layers, output_dim)

the loss now look like this [loss image with num\_layers=1](https://drive.google.com/file/d/1c-Q6qOr8yPWn0gYL74tnbPm7zBIPrFOz/view?usp=sharing) and now the problem is gone !!

Also I try to check the cause of the "output is the same everytime" problem I check dataloader and other code but no problem was found only num\_layers=3 that cause the problem num\_layers=1 fixed it

This is my training loop

criterion = nn.MSELoss() optimizer = torch.optim.Adam(rnn_model.parameters(), lr=learning_rate) trainingEpoch_loss = [] validationEpoch_loss = [] for epoch in range(num_epochs):     step_loss = []     rnn_model.train()     for idx, train_inputs in enumerate(train_dataloader):         optimizer.zero_grad()         outputs = rnn_model(torch.unsqueeze(train_inputs['text'], dim=0))         training_loss = criterion(outputs, train_inputs['poses'])         training_loss.backward()         optimizer.step()         step_loss.append(training_loss.item())         if (idx+1) % 1 == 0: print (f'Epoch [{epoch+1}/{num_epochs}], Step [{idx+1}/{len(train_dataloader)}], Loss: {training_loss.item():.4f}')     trainingEpoch_loss.append(np.array(step_loss).mean())     rnn_model.eval()     for idx, val_inputs in enumerate(val_dataloader):       validationStep_loss = []       outputs = rnn_model(torch.unsqueeze(val_inputs['text'], dim=0))       val_loss = criterion(outputs, val_inputs['poses'])       validationStep_loss.append(val_loss.item())     validationEpoch_loss.append(np.array(validationStep_loss).mean())

This is my Inference

text = "a man running" processed_text = torch.tensor(sentence_model.encode(text), dtype=torch.float) output_poses = rnn_model(processed_text.unsqueeze(0)) print(output_poses.shape) #shape=(1, 180) 1 person is 36 (original data for 1 person is 54 but I change to 36 because I want only x and y and not z so cut out the z axis) and there's 5 person so 5*36 = 180

My question is

1. Is there any model architecture recommend for this task other than RNN? 2. Why whatever I input the output is the same everytime when num\_layers=3 I'm very confused because the loss wouldn't go down if the model was giving the same output right? that's mean it give the same output in the Inference phase

Expected Answer

1. Model architecture that suit best for my task any papers or github repo related given would be appreciated 2. Answer why whatever I input the output is the same everytime when num\_layers=3

FieldValue
text I want to create AI that generate openpose from textual description for example if input "a man running" output would be like the image I provided Is there any model architecture recommend for me? my data condition is * canvas\_width: 900px * canvas\_height: 300px * frames: 5 (5 person) [expected output](https://drive.google.com/file/d/1vNcc5-zxpGodaGG3vat3TSPkbuC6olMK/view?usp=sharing) I trying to train RNN for this task and I use sentence transformer for embedding text and then pass to RNN…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-19
username_encoded Z0FBQUFBQm5Lakw0UDhxR1U4WGFiVGE2S3hOMkJVR3JJb0U3ampNMlBwcDVFUGxjYzVSQkNRSnNSVjBIcHNnZGV1a2lMeEx5NFF4NV95UXpiakZ2QTRNTThTY3JrcWdja0E9PQ==
url_encoded Z0FBQUFBQm5Lak9IbVVGQllwTHU5Q0dCSzdZamdud3lIcDU1allNMFZ4N19XZXJ0dEcyNGpjb0NrLWgxRHlIa1ByM3NaNnNweGQ1XzY2dG1FaEZFSkNMQ2FJVDRRNlpXTjEtMWctWm5Ra3l2b1U2elNTNHFzYl9CRlhsRmdlT1M0WGxjT3Q4Rm9zdmNwWnNPT0JybFpCRDRIdjRVbkY2dG1PSXpaYTFqSU02S3ZrNDR0alN3cTZvVEFQZzhfeEhDeDVXZmJmblBESXg0czVQSWdIbkN6ZS1yM2tJeUFxLUxMdz09

Raw Record

{
  "text": "I want to create AI that generate openpose from textual description for example if input \"a man running\" output would be like the image I provided Is there any model architecture recommend for me?\n\nmy data condition is\n\n* canvas\\_width: 900px\n* canvas\\_height: 300px\n* frames: 5 (5 person)\n\n[expected output](https://drive.google.com/file/d/1vNcc5-zxpGodaGG3vat3TSPkbuC6olMK/view?usp=sharing)\n\nI trying to train RNN for this task and I use sentence transformer for embedding text and then pass to RNN and the loss is look like image below\n\n    from sentence_transformers import SentenceTransformer \n    sentence_model = SentenceTransformer(\"all-MiniLM-L6-v2\")\n    text = \"a man running\"\n    text_input = torch.tensor(sentence_model.encode(text), dtype=torch.float)\n\n[loss image with num\\_layers=3](https://drive.google.com/file/d/1580_OdM146PefgIeRrK-Cmn-mIU_7AwB/view?usp=sharing)\n\nMy RNN setting\n\n    embedding_dim = 384\n    hidden_dim = 512\n    num_layers = 3\n    output_dim = 180\n    num_epochs = 100\n    learning_rate = 0.001\n    rnn_model = RNN(embedding_dim, hidden_dim, num_layers, output_dim)\n\nbut the problem is whatever I input the output is the same everytime! but when I try changing num\\_layers to 1 and keep other setting the same like this\n\n    embedding_dim = 384\n    hidden_dim = 512\n    num_layers = 1\n    output_dim = 180\n    num_epochs = 100\n    learning_rate = 0.001\n    rnn_model = RNN(embedding_dim, hidden_dim, num_layers, output_dim)\n\nthe loss now look like this [loss image with num\\_layers=1](https://drive.google.com/file/d/1c-Q6qOr8yPWn0gYL74tnbPm7zBIPrFOz/view?usp=sharing) and now the problem is gone !!\n\nAlso I try to check the cause of the \"output is the same everytime\" problem I check dataloader and other code but no problem was found only num\\_layers=3 that cause the problem num\\_layers=1 fixed it\n\nThis is my training loop\n\n    criterion = nn.MSELoss()\n    optimizer = torch.optim.Adam(rnn_model.parameters(), lr=learning_rate)\n    \n    trainingEpoch_loss = []\n    validationEpoch_loss = []\n    \n    for epoch in range(num_epochs):\n        step_loss = []\n        rnn_model.train()\n        for idx, train_inputs in enumerate(train_dataloader):\n            optimizer.zero_grad()\n            outputs = rnn_model(torch.unsqueeze(train_inputs['text'], dim=0))\n            training_loss = criterion(outputs, train_inputs['poses'])\n            training_loss.backward()\n            optimizer.step()\n            step_loss.append(training_loss.item())\n    \n            if (idx+1) % 1 == 0: print (f'Epoch [{epoch+1}/{num_epochs}], Step [{idx+1}/{len(train_dataloader)}], Loss: {training_loss.item():.4f}')\n        trainingEpoch_loss.append(np.array(step_loss).mean())\n    \n        rnn_model.eval()\n        for idx, val_inputs in enumerate(val_dataloader):\n          validationStep_loss = []\n          outputs = rnn_model(torch.unsqueeze(val_inputs['text'], dim=0))\n          val_loss = criterion(outputs, val_inputs['poses'])\n          validationStep_loss.append(val_loss.item())\n        validationEpoch_loss.append(np.array(validationStep_loss).mean())\n\nThis is my Inference\n\n    text = \"a man running\"\n    processed_text = torch.tensor(sentence_model.encode(text), dtype=torch.float)\n    output_poses = rnn_model(processed_text.unsqueeze(0))\n    print(output_poses.shape) #shape=(1, 180) 1 person is 36 (original data for 1 person is 54 but I change to 36 because I want only x and y and not z so cut out the z axis) and there's 5 person so 5*36 = 180\n\nMy question is\n\n1. Is there any model architecture recommend for this task other than RNN?\n2. Why whatever I input the output is the same everytime when num\\_layers=3 I'm very confused because the loss wouldn't go down if the model was giving the same output right? that's mean it give the same output in the Inference phase\n\nExpected Answer\n\n1. Model architecture that suit best for my task any papers or github repo related given would be appreciated\n2. Answer why whatever I input the output is the same everytime when num\\_layers=3",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-19",
  "username_encoded": "Z0FBQUFBQm5Lakw0UDhxR1U4WGFiVGE2S3hOMkJVR3JJb0U3ampNMlBwcDVFUGxjYzVSQkNRSnNSVjBIcHNnZGV1a2lMeEx5NFF4NV95UXpiakZ2QTRNTThTY3JrcWdja0E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9IbVVGQllwTHU5Q0dCSzdZamdud3lIcDU1allNMFZ4N19XZXJ0dEcyNGpjb0NrLWgxRHlIa1ByM3NaNnNweGQ1XzY2dG1FaEZFSkNMQ2FJVDRRNlpXTjEtMWctWm5Ra3l2b1U2elNTNHFzYl9CRlhsRmdlT1M0WGxjT3Q4Rm9zdmNwWnNPT0JybFpCRDRIdjRVbkY2dG1PSXpaYTFqSU02S3ZrNDR0alN3cTZvVEFQZzhfeEhDeDVXZmJmblBESXg0czVQSWdIbkN6ZS1yM2tJeUFxLUxMdz09"
}
Explore Dataset Explore Row

Entry Information