-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add initial Arrow PyCapsule support #517
Conversation
for i in range(20): | ||
# Break cache by removing one row each iteration | ||
movies_inner = movies.iloc[i:] | ||
movies["Title"] = movies["Title"].astype(str) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wish polars/pyarrow could handle loading mixed str/num columns from pandas
func: Arc::new((*STR_TO_UTC_TIMESTAMP_UDF).clone()), | ||
args: vec![field_col, lit(default_input_tz)], | ||
}) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sure there are a lot more of these that will need to be updated, but just did this one to get tests passing
Supersedes #501 now that everything needed has been updated.
Adopts arro3 / pyo3-arrow / narwhals for managing DataFrames in Python and the passing to Rust in arrow format with zero copy.
Uses
get_column_usage
to determine which columns are needed for each input inline dataset, and then uses narwhals to downproject prior to wrapping in anarro3
table to pass to rust. This approach removes the need for the Python data source framework, so I removed this all together.Performance of this workflow is improved from prior handling of pandas/polars, and is faster than the duckdb code path in my limited testing. We'll want to do more comprehensive benchmarks before release.