Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

print sccache stats in builds #2470

Merged
merged 12 commits into from
Oct 31, 2024
+33 −5
Merged

print sccache stats in builds #2470

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8cc1316
build wheels without build isolation
jameslamb Oct 15, 2024
4d356e3
revert unrelated index change
jameslamb Oct 15, 2024
51cdd7d
enable all CI
jameslamb Oct 15, 2024
e1a07a2
Merge branch 'branch-24.12' into wheel-build-no-isolation
jameslamb Oct 23, 2024
b667737
update to new conventions
jameslamb Oct 23, 2024
059bfce
print sccache stats for raft-dask too
jameslamb Oct 23, 2024
8310b10
rapids-upload-wheels-to-s3
jameslamb Oct 23, 2024
11696ce
github actions
jameslamb Oct 23, 2024
5e46826
copy-paste error
jameslamb Oct 23, 2024
72c541d
Merge branch 'branch-24.12' into wheel-build-no-isolation
jameslamb Oct 29, 2024
474d33d
revert no-build-isolation stuff
jameslamb Oct 29, 2024
4063756
move zero-stats inside if statement
jameslamb Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ repos:
cpp/cmake/modules/FindAVX[.]cmake
- id: verify-alpha-spec
- repo: https://github.com/rapidsai/dependency-file-generator
rev: v1.13.11
rev: v1.16.0
hooks:
- id: rapids-dependency-file-generator
args: ["--clean"]
Expand Down
4 changes: 4 additions & 0 deletions ci/build_cpp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ rapids-print-env

rapids-logger "Begin cpp build"

sccache --zero-stats

RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry mambabuild conda/recipes/libraft

sccache --show-adv-stats

rapids-upload-conda-to-s3 cpp
13 changes: 13 additions & 0 deletions ci/build_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,26 +22,37 @@ git_commit=$(git rev-parse HEAD)
export RAPIDS_PACKAGE_VERSION=${version}
echo "${version}" > VERSION

sccache --zero-stats

# TODO: Remove `--no-test` flags once importing on a CPU
# node works correctly
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
conda/recipes/pylibraft

sccache --show-adv-stats
sccache --zero-stats

rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/raft-dask

sccache --show-adv-stats
sccache --zero-stats

# Build ann-bench for each cuda and python version
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/raft-ann-bench

sccache --show-adv-stats
sccache --zero-stats
jameslamb marked this conversation as resolved.
Show resolved Hide resolved

# Build ann-bench-cpu only in CUDA 11 jobs since it only depends on python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(For a follow-up PR:) @cjnolet Do we still need to build raft-ann-bench now that cuVS benchmarks are being packaged?

# version
RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
Expand All @@ -51,6 +62,8 @@ if [[ ${RAPIDS_CUDA_MAJOR} == "11" ]]; then
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/raft-ann-bench-cpu

sccache --show-adv-stats
fi

rapids-upload-conda-to-s3 python
18 changes: 14 additions & 4 deletions ci/build_wheel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ source rapids-date-string

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

rapids-generate-version > VERSION
rapids-generate-version > ./VERSION

cd "${package_dir}"

Expand All @@ -39,10 +39,20 @@ case "${RAPIDS_CUDA_VERSION}" in
;;
esac

# Hardcode the output dir
python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check
sccache --zero-stats

rapids-logger "Building '${package_name}' wheel"

python -m pip wheel \
-w dist \
-v \
--no-deps \
--disable-pip-version-check \
.

sccache --show-adv-stats

mkdir -p final_dist
python -m auditwheel repair -w final_dist "${EXCLUDE_ARGS[@]}" dist/*

RAPIDS_PY_WHEEL_NAME="${underscore_package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 final_dist
RAPIDS_PY_WHEEL_NAME="${underscore_package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 python final_dist
Loading