Row 7874

Row ID: 7874 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 7874 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Hi

I'm trying to figure out how to apply filtering to parquet datasets at read time that include transformations applied to columns. I want to apply a function to a column, filter based on the output of that function, and only load those rows that pass the filtering. Specifically, one of my columns is a date. I want to select only those rows, where the floor(date) is within a specific set of dates.

I know how to filter using simple predicates e.g.

`filters=[('x', 'in', some_list), ('y', '<', some_value)...]`

but I specifically would like to filter based on transformations.

I can do the filtering \*after\* loading the parquet dataset into memory

from datetime import datetime import pyarrow as pa import pyarrow.parquet as pq allowed_dates = ['2001-05-01', '2001-06-01', '1999-07-01'....] def to_pa_datetime(date: str): y,m,d = list(map(int, date.split('-'))) return pa.scalar(datetime(y, m, d)) allowed_dates = pa.array([to_pa_datetime(k) for k in allowed_dates]) table = pq.read_table(file) pa.compute.is_in(pc.floor_temporal(table['date'], unit='month'), value_set=allowed_dates)

however, this entails that I load the entire dataset into memory first.

Any help is appreciated.

FieldValue
text Hi I'm trying to figure out how to apply filtering to parquet datasets at read time that include transformations applied to columns. I want to apply a function to a column, filter based on the output of that function, and only load those rows that pass the filtering. Specifically, one of my columns is a date. I want to select only those rows, where the floor(date) is within a specific set of dates. I know how to filter using simple predicates e.g. `filters=[('x', 'in', some_list), ('y', '<'…
label r/datascience
dataType post
communityName r/datascience
datetime 2024-05-18
username_encoded Z0FBQUFBQm5LakwzS3lkbTJjaE9NdHJJbHNVN3d4T3BnYzlTMDNSejhrSmVQZDZUa2RJN0Qtd1VmRjlmX29DUDlXTThBVkFYZ3NMVG9zdW1RaDBQZ1drTGlVN0ZRcGkzZm5KMGFLdVZEWTBBLWdiN1BPeUdnZzg9
url_encoded Z0FBQUFBQm5Lak9IaHFGNjcteGdMZkN6NXhiUkNhMUM4cS1idWFrd05tZV9lWjJmNWZsazZ2LXBNX2Vra1IwbDFwTTVlSnFiZFBkRDc2UUNhamxZME1DRlZQSDhBZWozT1M4R2hMRGxvSnQ0aUlqOGdCYXJHY1ZGdVBOWnRyYm9sTm53azBjdzNfV3lOT19DV01Da1A0NWlicHkyRGlVa2xyN0dnSllHbnBiNmxhdWFTeVF6SVNvUkZqZTQwY21PcDZUQ0NnRDVHOC1PMzgzaEUzcnpiSFFmOWJfNUhmM2t4Zz09

Raw Record

{
  "text": "Hi\n\nI'm trying to figure out how to apply filtering to parquet datasets at read time that include transformations applied to columns.  I want to apply a function to a column, filter based on the output of that function, and only load those rows that pass the filtering. Specifically, one of my columns is a date.  I want to select only those rows, where the floor(date) is within a specific set of dates.\n\nI know how to filter using simple predicates e.g.\n\n`filters=[('x', 'in', some_list), ('y', '<', some_value)...]`\n\nbut I specifically would like to filter based on transformations.\n\nI can do the filtering \\*after\\* loading the parquet dataset into memory\n\n    from datetime import datetime\n    import pyarrow as pa\n    import pyarrow.parquet as pq\n    \n    allowed_dates = ['2001-05-01', '2001-06-01', '1999-07-01'....]\n    def to_pa_datetime(date: str):\n        y,m,d = list(map(int, date.split('-')))\n        return pa.scalar(datetime(y, m, d))\n    \n    allowed_dates = pa.array([to_pa_datetime(k) for k in allowed_dates])\n    table = pq.read_table(file)\n    pa.compute.is_in(pc.floor_temporal(table['date'], unit='month'), value_set=allowed_dates)\n\nhowever, this entails that I load the entire dataset into memory first.\n\n  \nAny help is appreciated.\n\n",
  "label": "r/datascience",
  "dataType": "post",
  "communityName": "r/datascience",
  "datetime": "2024-05-18",
  "username_encoded": "Z0FBQUFBQm5LakwzS3lkbTJjaE9NdHJJbHNVN3d4T3BnYzlTMDNSejhrSmVQZDZUa2RJN0Qtd1VmRjlmX29DUDlXTThBVkFYZ3NMVG9zdW1RaDBQZ1drTGlVN0ZRcGkzZm5KMGFLdVZEWTBBLWdiN1BPeUdnZzg9",
  "url_encoded": "Z0FBQUFBQm5Lak9IaHFGNjcteGdMZkN6NXhiUkNhMUM4cS1idWFrd05tZV9lWjJmNWZsazZ2LXBNX2Vra1IwbDFwTTVlSnFiZFBkRDc2UUNhamxZME1DRlZQSDhBZWozT1M4R2hMRGxvSnQ0aUlqOGdCYXJHY1ZGdVBOWnRyYm9sTm53azBjdzNfV3lOT19DV01Da1A0NWlicHkyRGlVa2xyN0dnSllHbnBiNmxhdWFTeVF6SVNvUkZqZTQwY21PcDZUQ0NnRDVHOC1PMzgzaEUzcnpiSFFmOWJfNUhmM2t4Zz09"
}
Explore Dataset Explore Row

Entry Information