Row 56307

Row ID: 56307 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 56307 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Hi all -

I am working on a problem which is a multi-variate timeseries regression problem, in particular, developing a surrogate for a (very slow) physics based simulation.

Inputs:

34 features x 35040 timesteps (15 min interval data for one year) + 90 categorical features (ie static for entire year) with 4-56 options for each category.

Outputs:

8 features x 35040 timesteps

My basic approach is this:

1. (b, 90 features) > embedding layer > MLP network (a few layers, intermittent skip connections etc) > (b, 96 timesteps, m features) ie create a unique latent representation of the categorical features for each interval of a day > repeat x 365 (to get to 35040 timesteps) 2. Concatenate latent timeseries representation of categorical features with the 34 timeseries feature. 3. d_model for mamba is m + 34 (currently working with m=94 so d_model is 128) 4. Chunk the timeseries according to a desired context size and send through mamba and a final linear layer to mix down to the final number of timeseries in the output. Using a single chunk means the entire year is processed in a single pass, whereas using 73 chunks corresponds to 5 days.

Now, the interesting question is what to set as the context/chunking size for Mamba. From a physics perspective, the maximum causal horizon (correct term?) is probably about a day - ie the state of the inputs and system over the last 24 hours are relevant to compute the next timestep in a white box solver, but beyond that the past doesn’t affect the system. However, because the timeseries inputs/timesteps are not IID/uniformly random in the dataset (they theoretically could be mathematically but realistic data is used), it’s of course possible (and at least somewhat likely) that there is mutual information shared between timesteps beyond the causal horizon.

I’m getting pretty decent results with a context size of 480 timesteps (5 days) and backpropping after each chunk (and during inference, processing a whole year in chunks of the same size) but I don’t have any good intuition about tradeoffs of using a larger or smaller context size here besides how it affects the VRAM requirements during training.

I’d love to start a discussion here about it!

Or anything else in this architecture. If you have any good papers that implement a similar approach to combining categorical and timeseries data I would love to see it!

FieldValue
text Hi all - I am working on a problem which is a multi-variate timeseries regression problem, in particular, developing a surrogate for a (very slow) physics based simulation. Inputs: 34 features x 35040 timesteps (15 min interval data for one year) + 90 categorical features (ie static for entire year) with 4-56 options for each category. Outputs: 8 features x 35040 timesteps My basic approach is this: 1. (b, 90 features) > embedding layer > MLP network (a few layers, intermittent skip con…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-23
username_encoded Z0FBQUFBQm5Lak1XaVRHU0lnQkZXaHNoTEJEWDRLRGppWXV2UjFCRjVMSVpUX0JsamhYV2ZuOTRWUVcwTWVLSWdJa2tuRnA0eTNLNzZaczgyREQzNk5RZnlJTHlDaE5PUWVJWGNCT3poRWhCYjh0UEw0QVptdzA9
url_encoded Z0FBQUFBQm5Lak9tN3B4TDhzdFF3UXk1T0FTMEpOWG9RbnN0YUVtbzBkb3pvbzJxTWxKdTdYcDVXRTVtOGlYWUhwUjByQ1ZkUVppOVhhNDBqYXFjZFBIOS1XTHFCTHVWMTVUN1MzbFplMGdZNmk2S1NvTnBJREdYNjl6YVBhQjlOZnEybDZaQ1MtMEd0YnlxcUJIbE1XUGwzaUw0VUo1TW1LNFhPcHVJaDlBeDlQVnJXRVRSdXI5VXc4b2Z1bUY0ZXdoeHY1c0dSNk1uZDBhdzRqV3hyS2w0MGdHUmt2dXdFUT09

Raw Record

{
  "text": "Hi all -\n\nI am working on a problem which is a multi-variate timeseries regression problem, in particular, developing a surrogate for a (very slow) physics based simulation.\n\nInputs: \n\n34 features x 35040 timesteps (15 min interval data for one year) + 90 categorical features (ie static for entire year) with 4-56 options for each category. \n\nOutputs:\n\n8 features x 35040 timesteps\n\nMy basic approach is this:\n\n1. (b, 90 features) > embedding layer > MLP network (a few layers, intermittent skip connections etc) > (b, 96 timesteps, m features) ie create a unique latent representation of the categorical features for each interval of a day > repeat x 365 (to get to 35040 timesteps)\n2. Concatenate latent timeseries representation of categorical features with the 34 timeseries feature. \n3. d_model for mamba is m + 34 (currently working with m=94 so d_model is 128)\n4. Chunk the timeseries according to a desired context size and send through mamba and a final linear layer to mix down to the final number of timeseries in the output. Using a single chunk means the entire year is processed in a single pass, whereas using 73 chunks corresponds to 5 days. \n\nNow, the interesting question is what to set as the context/chunking size for Mamba. From a physics perspective, the maximum causal horizon (correct term?) is probably about a day - ie the state of the inputs and system over the last 24 hours are relevant to compute the next timestep in a white box solver, but beyond that the past doesn’t affect the system. However, because the timeseries inputs/timesteps are not IID/uniformly random in the dataset (they theoretically could be mathematically but realistic data is used), it’s of course possible (and at least somewhat likely) that there is mutual information shared between timesteps beyond the causal horizon. \n\nI’m getting pretty decent results with a context size of 480 timesteps (5 days) and backpropping after each chunk (and during inference, processing a whole year in chunks of the same size) but I don’t have any good intuition about tradeoffs of using a larger or smaller context size here besides how it affects the VRAM requirements during training. \n\nI’d love to start a discussion here about it!\n\nOr anything else in this architecture. If you have any good papers that implement a similar approach to combining categorical and timeseries data I would love to see it!",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-23",
  "username_encoded": "Z0FBQUFBQm5Lak1XaVRHU0lnQkZXaHNoTEJEWDRLRGppWXV2UjFCRjVMSVpUX0JsamhYV2ZuOTRWUVcwTWVLSWdJa2tuRnA0eTNLNzZaczgyREQzNk5RZnlJTHlDaE5PUWVJWGNCT3poRWhCYjh0UEw0QVptdzA9",
  "url_encoded": "Z0FBQUFBQm5Lak9tN3B4TDhzdFF3UXk1T0FTMEpOWG9RbnN0YUVtbzBkb3pvbzJxTWxKdTdYcDVXRTVtOGlYWUhwUjByQ1ZkUVppOVhhNDBqYXFjZFBIOS1XTHFCTHVWMTVUN1MzbFplMGdZNmk2S1NvTnBJREdYNjl6YVBhQjlOZnEybDZaQ1MtMEd0YnlxcUJIbE1XUGwzaUw0VUo1TW1LNFhPcHVJaDlBeDlQVnJXRVRSdXI5VXc4b2Z1bUY0ZXdoeHY1c0dSNk1uZDBhdzRqV3hyS2w0MGdHUmt2dXdFUT09"
}
Explore Dataset Explore Row

Entry Information