Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make process resumable and enhance IP classifier #54

Merged
merged 56 commits into from
Aug 21, 2024
Merged

Conversation

CodyCBakerPhD
Copy link
Member

Various services like GitHub, AWS, VPNs are now labeled as such in the IP mapping step (and hash cache)

Also added some additional early exit cases suggested by Yarik to make the primary parsing faster

@CodyCBakerPhD CodyCBakerPhD self-assigned this Aug 21, 2024
@CodyCBakerPhD CodyCBakerPhD marked this pull request as ready for review August 21, 2024 22:19
@CodyCBakerPhD CodyCBakerPhD merged commit 95d437e into main Aug 21, 2024
3 checks passed
@CodyCBakerPhD CodyCBakerPhD deleted the make_resumable branch August 21, 2024 22:19
Copy link

codecov bot commented Aug 22, 2024

Codecov Report

Attention: Patch coverage is 67.64706% with 88 lines in your changes missing coverage. Please review.

Project coverage is 63.96%. Comparing base (b7d35b3) to head (75579e9).
Report is 1 commits behind head on main.

Files Patch % Lines
src/dandi_s3_log_parser/_ip_utils.py 42.59% 31 Missing ⚠️
src/dandi_s3_log_parser/_s3_log_file_reducer.py 73.73% 26 Missing ⚠️
src/dandi_s3_log_parser/_command_line_interface.py 0.00% 13 Missing ⚠️
...g_parser/_bin_all_reduced_s3_logs_by_object_key.py 82.05% 7 Missing ⚠️
.../dandi_s3_log_parser/_dandi_s3_log_file_reducer.py 81.25% 6 Missing ⚠️
src/dandi_s3_log_parser/_s3_log_line_parser.py 76.19% 5 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #54      +/-   ##
==========================================
- Coverage   65.13%   63.96%   -1.17%     
==========================================
  Files          13       14       +1     
  Lines         499      555      +56     
==========================================
+ Hits          325      355      +30     
- Misses        174      200      +26     
Flag Coverage Δ
unittests 63.96% <67.64%> (-1.17%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/dandi_s3_log_parser/__init__.py 100.00% <100.00%> (ø)
src/dandi_s3_log_parser/_config.py 100.00% <100.00%> (ø)
src/dandi_s3_log_parser/_globals.py 100.00% <100.00%> (ø)
..._s3_log_parser/_map_binned_s3_logs_to_dandisets.py 96.07% <100.00%> (ø)
src/dandi_s3_log_parser/_s3_log_line_parser.py 84.78% <76.19%> (+7.00%) ⬆️
.../dandi_s3_log_parser/_dandi_s3_log_file_reducer.py 80.95% <81.25%> (-7.29%) ⬇️
...g_parser/_bin_all_reduced_s3_logs_by_object_key.py 82.05% <82.05%> (ø)
src/dandi_s3_log_parser/_command_line_interface.py 0.00% <0.00%> (ø)
src/dandi_s3_log_parser/_s3_log_file_reducer.py 76.36% <73.73%> (-23.64%) ⬇️
src/dandi_s3_log_parser/_ip_utils.py 54.05% <42.59%> (-5.53%) ⬇️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant