[DO NOT MERGE] cudf-polars chunked parquet reader #16789
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-config Use new polars engine config object in cudf-polars callback
## Description <!-- Provide a standalone description of changes in this PR. --> <!-- Reference any issues closed by this PR with "closes rapidsai#1234". --> <!-- Note: The pull request title will be included in the CHANGELOG. --> Adapts to IR changes in polars 1.4 and handles nrows/skiprows a little more correctly. ## Checklist - [ ] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. --------- Co-authored-by: Lawrence Mitchell <[email protected]>
Add support for ``pl.col.str.replace`` and ``pl.col.str.replace_many`` Authors: - Thomas Li (https://github.com/lithomas1) Approvers: None URL: rapidsai#16039
…sts (rapidsai#16441) (rapidsai#16605) Authors: - Thomas Li (https://github.com/lithomas1) - https://github.com/brandon-b-miller Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#16605
…ai#16509) contributes to rapidsai#16478 This implements "cum_min", "cum_max", "cum_prod", "cum_sum" "cum_count" is not implemented for now, since there's no exact libcudf match (I imagine the non-grouped case is also not used that much but haven't checked). I suppose we could implement it by creating a column of 1s and copying the null mask over, and doing a cum_sum on that. Let me know if you want to try that. Authors: - Thomas Li (https://github.com/lithomas1) Approvers: - https://github.com/brandon-b-miller URL: rapidsai#16509
…olumn Use a key column rather than a placeholder for count agg
## Description - Closes rapidsai#16477
…16596) polars.from_arrow renames empty column names (see pola-rs/polars#11632). This causes problems when round-tripping specially crafted dataframes. Avoid the problem by constructing the table with fake names and then renaming.
Add support for additional unaryops through `cudf-polars`. Closes rapidsai#16566 --------- Co-authored-by: Lawrence Mitchell <[email protected]>
Forward-merge 24.08
Add support for string `strip` in `pylibcudf` and `cudf-polars`. --------- Co-authored-by: Lawrence Mitchell <[email protected]>
This PR adds datetime/timestamp parsing from string columns in pylibcudf and cudf-polars. Closes rapidsai#16174
Support `pl.Expr.quantile` in cudf-polars. --------- Co-authored-by: Vyas Ramasubramani <[email protected]>
Since the full-frame `Agg` handler for first and last doesn't construct a request (because we can do it without a `from_scalar` call), we didn't handle these in a groupby context. Fortunately it is easy to add.
We were previously not calling the superclass __post_init__ in custom validations of IR nodes. This meant that we would sometimes fail to raise when the schema contained an EMPTY column. Since we can't really compute with these types, we just fall back.
…sai#16476) We previously didn't support this case correctly, but it's not too bad. This would be much easier if we could do it in libcudf, hence: rapidsai#16475
Correctly handle `pow` and `log` by translating to binary expressions when we observe the node. Upgrade our minimum supported polars version (so that we see all these function names from the rust IR). Also tighten check for which groupby-aggs are supported when the expression contains a unary function.
Add support for unpivoting a DataFrame. We raise for cases where the concatenation of the value columns produces a cast that is not supported by standard fixed-width unary casting.
Reject two more edge cases that we do not support. We could easily support the case where the parquet read just needs to read the metadata, but it is low priority, so have not done so here.
--------- Co-authored-by: brandon-b-miller <[email protected]> Co-authored-by: Bradley Dice <[email protected]> Co-authored-by: Lawrence Mitchell <[email protected]>
…ai#16755) This field renaming was due to a recent refactor in (as-yet-unreleased) polars 1.7.
## Description We implement a small pytest plugin that defaults the polars engine to GPU (by monkeypatching `LazyFrame.collect`, yet another reason to have a global default somehow). As well as this, we collate all the known failures and classify them. ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes.
brandon-b-miller
added
the
5 - DO NOT MERGE
github-actions
bot
added
libcudf
brandon-b-miller
added
feature request
|
/ok to test |
|
/ok to test |
github-actions
bot
added
libcudf
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Test PR to generate some wheels exploring some I/O functionality on top of feature/cudf-polars.
Do not merge.