Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print sccache stats in all builds #111

Open
17 of 18 tasks
vyasr opened this issue Oct 22, 2024 · 1 comment
Open
17 of 18 tasks

Print sccache stats in all builds #111

vyasr opened this issue Oct 22, 2024 · 1 comment
Assignees

Comments

@vyasr
Copy link
Contributor

vyasr commented Oct 22, 2024

Adding an sccache -s call after every build is free and it is very useful to help diagnose issues when builds are suddenly and unexpectedly slow.

Notes

These can be done in any order.

Do the GNN repos last, as some of the cugraph repos are currently being shuffled around. See e.g. rapidsai/cugraph-gnn#58 (comment).

GNN repos

@jameslamb
Copy link
Member

Totally support this, for the reasons mentioned in rapidsai/cuspatial#1473 (comment).

I'm going to take this and make these changes at the same time as the changes for #108 .

@jameslamb jameslamb self-assigned this Oct 22, 2024
rapids-bot bot pushed a commit to rapidsai/cuspatial that referenced this issue Oct 22, 2024
Contributes to rapidsai/build-planning#108

Contributes to rapidsai/build-planning#111

Proposes building `libcuspatial` wheels with `--no-build-isolation`, to improve the rate of `sccache` cache hits and therefore reduce CI times.

Also proposes printing `sccache` stats to CI logs after wheel and conda builds.

## Notes for Reviewers

#

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Mike Sarahan (https://github.com/msarahan)
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Bradley Dice (https://github.com/bdice)

URL: #1473
rapids-bot bot pushed a commit to rapidsai/cucim that referenced this issue Oct 28, 2024
…792)

Contributes to rapidsai/build-planning#108
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* removes the use of build isolation in wheel builds
* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* using the RAPIDS conventions `py_build_{project}` and `py_rapids_build_{project}` in `dependencies.yaml`
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #792
rapids-bot bot pushed a commit to rapidsai/cudf that referenced this issue Oct 28, 2024
Contributes to rapidsai/build-planning#108
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* building `libcudf` wheels with `--no-build-isolation` (for better `sccache` hit rate)
* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

#

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17088
rapids-bot bot pushed a commit to rapidsai/rmm that referenced this issue Oct 29, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* reducing verbosity of `pip wheel` in wheel builds

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)

URL: #1712
rapids-bot bot pushed a commit to rapidsai/wholegraph that referenced this issue Oct 29, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip wheel` verbosity
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `python` in calls to `rapids-upload-wheels-to-s3`

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)

URL: #231
rapids-bot bot pushed a commit to rapidsai/kvikio that referenced this issue Oct 29, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #524
rapids-bot bot pushed a commit to rapidsai/pynvjitlink that referenced this issue Oct 29, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0) and `rapidsai/pre-commit-hooks` (v0.4.0)
* always explicitly specifying `python` in calls to `rapids-upload-wheels-to-s3`

While doing that, I also noticed this project wasn't using `sccache`. This PR proposes using `sccache` to drive compilation, to speed up builds a bit.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #112
rapids-bot bot pushed a commit to rapidsai/cuvs that referenced this issue Oct 29, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`
* modifying `dependencies.yaml` to match RAPIDS-wide naming conventions

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #413
rapids-bot bot pushed a commit to rapidsai/ucx-py that referenced this issue Oct 29, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* removing unnecessary calls to `rapids-configure-sccache` (this project does not use `sccache`)

## Notes for Reviewers

I'd originally started this PR with the goal of printing `sccache` stats in builds here... but realized this project does not use `sccache`.

I chose not to pursue adding `sccache` here. The fact that this project doesn't use CMake means it'd take some effort to figure out how to inject `sccache` into the compilation (`CC` / `CXX` environment variables? something else?). Conda and wheel builds are only spending around 20 seconds actually compiling Cython code, so it doesn't seem worth the effort.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Mike Sarahan (https://github.com/msarahan)

URL: #1089
rapids-bot bot pushed a commit to rapidsai/cugraph-gnn that referenced this issue Oct 30, 2024
Another steps towards completing the work started in #53 

Fixes #15

Contributes to rapidsai/build-planning#111

Proposes changes to get CI running on pull requests for `cugraph-pyg` and `cugraph-dgl`

## Notes for Reviewers

Workflows for nightly builds and publishing nightly packages are intentionally not included here. See #58 (comment)

Notebook tests are intentionally not added here... they'll be added in the next PR.

Pulls in changes from these other upstream PRs that had not been ported over to this repo:

* rapidsai/cugraph#4690
* rapidsai/cugraph#4393

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Alex Barghi (https://github.com/alexbarghi-nv)

Approvers:
  - Alex Barghi (https://github.com/alexbarghi-nv)
  - Bradley Dice (https://github.com/bdice)

URL: #59
rapids-bot bot pushed a commit to rapidsai/cugraph that referenced this issue Oct 30, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #4719
rapids-bot bot pushed a commit to rapidsai/ucxx that referenced this issue Oct 31, 2024
Contributes to rapidsai/build-planning#108
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* building `libucxx` wheels with `--no-build-isolation` (for better `sccache` hit rate)
* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`
* moving more one-wheel-specific logic into `build_wheel_{project}.sh` scripts, mimicking how `cudf` has structured its scripts

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #301
rapids-bot bot pushed a commit to rapidsai/cuml that referenced this issue Oct 31, 2024
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #6111
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants