Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build wheels without build isolation #1473

Merged
merged 31 commits into from
Oct 22, 2024
Merged

build wheels without build isolation #1473

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
b983143
test wheel-build sccache fixes
jameslamb Oct 11, 2024
9e2ceef
fiddle with options
jameslamb Oct 11, 2024
8c747f5
run fewer build jobs
jameslamb Oct 11, 2024
7cf37ee
build with --no-build-isolation
jameslamb Oct 11, 2024
4477b70
paths
jameslamb Oct 11, 2024
d6f05c6
more paths
jameslamb Oct 11, 2024
d18c3f7
Merge branch 'wheel-build-sccache' of github.com:jameslamb/cuspatial …
jameslamb Oct 11, 2024
6c64e73
restore all CI
jameslamb Oct 11, 2024
cbfaf85
only pre-install build backend
jameslamb Oct 14, 2024
9744bcf
Merge branch 'wheel-build-sccache' of github.com:jameslamb/cuspatial …
jameslamb Oct 14, 2024
8256e35
only run wheel builds
jameslamb Oct 14, 2024
28e34eb
GHA fix
jameslamb Oct 14, 2024
8251aaa
go back to installing everything
jameslamb Oct 14, 2024
130e761
re-enable all CI
jameslamb Oct 14, 2024
29b005e
move PIP_CONSTRAINT stuff back, remove 'declare'
jameslamb Oct 14, 2024
fc8d575
Merge branch 'wheel-build-sccache' of github.com:jameslamb/cuspatial …
jameslamb Oct 14, 2024
ec06a4a
Merge branch 'branch-24.12' into wheel-build-sccache
bdice Oct 16, 2024
cb6b07d
only use an un-isolated build for libcuspatial
jameslamb Oct 16, 2024
75eaf12
Merge branch 'wheel-build-sccache' of github.com:jameslamb/cuspatial …
jameslamb Oct 16, 2024
7b87cd6
Merge branch 'branch-24.12' of github.com:rapidsai/cuspatial into whe…
jameslamb Oct 17, 2024
f59a067
test with new rapids-dependency-file-generator
jameslamb Oct 17, 2024
1199e31
enable all CI
jameslamb Oct 17, 2024
5e54225
shell variable
jameslamb Oct 17, 2024
9f979b5
actually turn off build isolation
jameslamb Oct 17, 2024
aa62971
more fiddling with PIP_NO_BUILD_ISOLATION
jameslamb Oct 17, 2024
b4cd177
ugh ok yeah it really is PIP_NO_BUILD_ISOLATION=0
jameslamb Oct 17, 2024
69fa9e8
try more libcuspatial builds
jameslamb Oct 17, 2024
8b841c2
restore all CI
jameslamb Oct 17, 2024
e5d02b9
use published rapids-dependency-file-generator packages
jameslamb Oct 22, 2024
dd7a7a6
run another build, hopefully with better cache hit rate
jameslamb Oct 22, 2024
336bc27
print sccache stats for conda builds
jameslamb Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ repos:
- id: verify-copyright
- id: verify-alpha-spec
- repo: https://github.com/rapidsai/dependency-file-generator
rev: v1.13.11
rev: v1.16.0
vyasr marked this conversation as resolved.
Show resolved Hide resolved
hooks:
- id: rapids-dependency-file-generator
args: ["--clean"]
Expand Down
4 changes: 4 additions & 0 deletions ci/build_cpp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,11 @@ rapids-print-env

rapids-logger "Begin cpp build"

sccache --zero-stats

RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry mambabuild \
conda/recipes/libcuspatial

sccache --show-adv-stats
Copy link
Member Author

@jameslamb jameslamb Oct 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added sccache stats printing for conda builds as well based on rapidsai/build-planning#111 and offline conversation with @vyasr .

example build with these: https://github.com/rapidsai/cuspatial/actions/runs/11465788119/job/31906443973?pr=1473

Why not build.sh?

Here, I'm proposing that these be put in ci/build_{cpp,python}.sh, wrapped around any conda mambabuild invocations that will end up compiling anything (so, for example, I included them in build_python.sh because cuspatial and cuproj both have some Cython extensions).

I think this is preferable to build.sh, because the build_{cpp,python}.sh scripts are unambiguously for CI, and can unconditionally invoke sccache. build.sh is invoked in CI, but it's also intended for local use, and I didn't want to assume the presence of sccache for all local builds are introduce the complexity of having to check for its use.

There are also some projects, like cuvs, that have their own handling of printing sccache stats and creating build summary reports based on them in build.sh, and I didn't want to interfere with that.

example:

https://github.com/rapidsai/cuvs/blob/e7f1085b71c340b9600f5f38f7f0059a5c7aa806/build.sh#L364-L366

https://github.com/rapidsai/cuvs/blob/e7f1085b71c340b9600f5f38f7f0059a5c7aa806/build.sh#L408-L424


rapids-upload-conda-to-s3 cpp
8 changes: 8 additions & 0 deletions ci/build_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,28 @@ CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

rapids-logger "Begin py build cuSpatial"

sccache --zero-stats

# TODO: Remove `--no-test` flag once importing on a CPU
# node works correctly
RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
conda/recipes/cuspatial

sccache --show-adv-stats

rapids-logger "Begin py build cuProj"

sccache --zero-stats

# TODO: Remove `--no-test` flag once importing on a CPU
# node works correctly
RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
conda/recipes/cuproj

sccache --show-adv-stats

rapids-upload-conda-to-s3 python
10 changes: 9 additions & 1 deletion ci/build_wheel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,15 @@ RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

cd "${package_dir}"

python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check
rapids-logger "Building '${package_name}' wheel"
python -m pip wheel \
-w dist \
-v \
--no-deps \
--disable-pip-version-check \
.

sccache --show-adv-stats
vyasr marked this conversation as resolved.
Show resolved Hide resolved

mkdir -p final_dist
python -m auditwheel repair \
Expand Down
24 changes: 23 additions & 1 deletion ci/build_wheel_libcuspatial.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,26 @@

set -euo pipefail

ci/build_wheel.sh libcuspatial python/libcuspatial cpp
package_name="libcuspatial"

rapids-logger "Generating build requirements"
matrix_selectors="cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true"

rapids-dependency-file-generator \
--output requirements \
--file-key "py_build_${package_name}" \
--file-key "py_rapids_build_${package_name}" \
--matrix "${matrix_selectors}" \
| tee /tmp/requirements-build.txt

rapids-logger "Installing build requirements"
python -m pip install \
-v \
--prefer-binary \
-r /tmp/requirements-build.txt

# build with '--no-build-isolation', for better sccache hit rate
# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735)
export PIP_NO_BUILD_ISOLATION=0
vyasr marked this conversation as resolved.
Show resolved Hide resolved

ci/build_wheel.sh "${package_name}" python/libcuspatial cpp
Loading