New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s3: make use of start-after available for list_objects_v2 #132

Open

yarikoptic opened this issue May 8, 2023 · 0 comments

Member

yarikoptic commented May 8, 2023

see https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax

ATM we are doing "seek"ing locally after having fetched all versions for all keys: https://github.com/datalad/datalad-crawler/blob/master/datalad_crawler/nodes/s3.py#L177 . That might be prohibitively expensive. We better take advantage of start-after

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment