Update fmt (to 11.0.2) and spdlog (to 1.14.1). #16806

jameslamb · 2024-09-16T13:55:28Z

Description

Replaces #15603

Contributes to:

Now that most of conda-forge has been updated to fmt >=11.0.1,<12 and spdlog>=1.14.1,<1.15 (rapidsai/build-planning#56 (comment)), we're attempting to upgrade RAPIDS to similar versions of those libraries.

This improves the likelihood that RAPIDS will be installable alongside newer versions of its
dependencies and complementary packages on conda-forge.

Notes for Reviewers

This PR is testing changes made in rapidsai/rapids-cmake#689.
It shouldn't be merged until those rapids-cmake changes are merged and any testing-specific details have been removed.

jameslamb · 2024-09-19T04:38:47Z

cpp/cmake/thirdparty/get_spdlog.cmake

-    rapids_export_find_package_root(
-      BUILD spdlog [=[${CMAKE_CURRENT_LIST_DIR}]=] EXPORT_SET cudf-exports
-    )
-  endif()


Context for this change: rapidsai/rapids-cmake#387 (comment)

jameslamb · 2024-09-19T04:42:22Z

cpp/cmake/thirdparty/get_spdlog.cmake

@@ -16,21 +16,8 @@
 function(find_and_configure_spdlog)

  include(${rapids-cmake-dir}/cpm/spdlog.cmake)
-  rapids_cpm_spdlog(FMT_OPTION "EXTERNAL_FMT_HO" INSTALL_EXPORT_SET cudf-exports)
-  rapids_export_package(BUILD spdlog cudf-exports)
+  rapids_cpm_spdlog(FMT_OPTION "EXTERNAL_FMT_HO")


I've paired removing the spdlog export here with adding this over in cpp/CMakeLists.txt:

target_linked_libraries(cudf PRIVATE spdlog::spdlog_header_only)

Without that, the pip devcontainer build here fails like this while building the pylibcudf wheel:

CMake Error at /home/coder/.local/share/venvs/rapids/lib/python3.10/site-packages/cmake/data/share/cmake-3.30/Modules/CMakeFindDependencyMacro.cmake:76 (find_package): By not providing "Findspdlog.cmake" in CMAKE_MODULE_PATH this project has asked CMake to find a package configuration file provided by "spdlog", but CMake did not find one. Could not find a package configuration file provided by "spdlog" with any of the following names: spdlogConfig.cmake spdlog-config.cmake Add the installation prefix of "spdlog" to CMAKE_PREFIX_PATH or set "spdlog_DIR" to a directory containing one of the above files. If "spdlog" provides a separate development package or SDK, be sure it has been installed.

I think because in devcontainers, libcudf.so is built outside of wheel builds, which means that this condition is hit:

cudf/python/libcudf/CMakeLists.txt

Lines 29 to 31 in a0c6fc8

if(cudf_FOUND)

return()

endif()

And then later code that makes spdlog available isn't run.

This seems plausible to me but I'd like a confirmation from a CMake expert. @vyasr or @KyleFromNVIDIA?

If libcudf is only using spdlog privately, and its public headers don't depend on spdlog, then I think this is fine.

Thank you! I'd thought it was only using it privately, but now that I search around again I'm really not sure.

This makes me think libcudf's logger is considered part of its public API:

cudf/cpp/doxygen/developer_guide/DEVELOPER_GUIDE.md

Lines 1084 to 1087 in 2676924

* Global logger object exposed via `cudf::logger()` - sets the minimum logging level at runtime.

For example, calling `cudf::logger().set_level(spdlog::level::err)`, will exclude any messages that

are not errors or critical errors. This API should not be used within libcudf to manipulate logging,

its purpose is to allow upstream users to configure libcudf logging to fit their application.

Here's the one non-test header directly #include-ing spdlog:

cudf/cpp/include/cudf/utilities/logger.hpp

Line 21 in 2676924

#include <spdlog/spdlog.h>

And there I can see that cudf::logger is a spdlog::logger.

cudf/cpp/include/cudf/utilities/logger.hpp

Line 46 in 2676924

spdlog::logger& logger();

So maybe it does need to be exported? But if so then I don't understand how the pylibcudf and cudf builds and tests could be succeeding with it marked private like this.

One other helpful piece of context that might help explain why this is even coming up now... we were previously patching spdlog in rapids-cmake, which took it down the "always download with CPM and patch" code paths there. Now that we're not patching, different rapids-cmake codepaths are being followed. Something related to exporting must be different between those cases.

Since we're using the header-only version of spdlog, if the downstream targets already have include directories that contain spdlog, they don't need to link against the spdlog target to get those headers. So perhaps it is currently just working by accident.

Ok yeah the pip devcontainers job (where I'd originally observed the issue at the top of this thread) is passing with these changes: https://github.com/rapidsai/cudf/actions/runs/10962944541/job/30443351446?pr=16806

I feel much better about this state, INTERFACE seems right to me.

Yes, that sounds right to me. Apologies for not catching this earlier.

Didn't see this earlier. Yes, the spdlog logger is currently part of the public API for cudf via the logger. That is why we also wound up with symbol issues around spdlog before we went through and hid all the necessary symbols.

The current code looks close, but I don't think INTERFACE is right. It should be a PUBLIC dependency because it is required both by consumers of libcudf (because of the spdlog bits in the logger's public interface) and by libcudf internals (to compiled logger.cpp). I suspect that right now INTERFACE is working because we transitively inherit the spdlog INTERFACE include from rmm.

Ah I see, ok that makes sense! Let me test switching it to PUBLIC, shouldn't take too long.

Ok yeah I tested that in devcontainers and it worked great (yes really that fast... yay sccache).

And looks like it's working here in CI too: https://github.com/rapidsai/cudf/actions/runs/10963768468/job/30445951463?pr=16806

bdice

Looks good to me, but maybe a second eye on the CMake changes would be helpful to confirm that we're going the right way.

bdice · 2024-09-20T01:02:43Z

cpp/cmake/thirdparty/get_spdlog.cmake

@@ -16,21 +16,8 @@
 function(find_and_configure_spdlog)

  include(${rapids-cmake-dir}/cpm/spdlog.cmake)
-  rapids_cpm_spdlog(FMT_OPTION "EXTERNAL_FMT_HO" INSTALL_EXPORT_SET cudf-exports)
-  rapids_export_package(BUILD spdlog cudf-exports)
+  rapids_cpm_spdlog(FMT_OPTION "EXTERNAL_FMT_HO")


This seems plausible to me but I'd like a confirmation from a CMake expert. @vyasr or @KyleFromNVIDIA?

jameslamb · 2024-09-20T13:32:20Z

It's expected that the conda devcontainers job will fail here.

  LibMambaUnsatisfiableError: Encountered problems while solving:
    - package librmm-24.10.00a38-cuda12_240910_g800dc874_38 requires fmt >=10.1.1,<11, but none of the providers can be installed
    - package rmm-24.10.00a38-cuda11_py310_240913_g800dc874_38 requires librmm >=24.10.0a.38,<24.11.0a0, but none of the providers can be installed
  
  Could not solve for environment specs
  The following packages are incompatible
  ├─ fmt >=11.0.2,<12  is requested and can be installed;
  ├─ librmm 24.10.*,>=0.0.0a0  is not installable because it requires
  │  ├─ fmt >=10.1.1,<11 , which conflicts with any installable versions previously reported;
  │  └─ spdlog >=1.12.0,<1.13 , which can be installed;
  ├─ rmm 24.10.*,>=0.0.0a0  is not installable because it requires
  │  └─ librmm [>=24.10.0a.38,<24.11.0a0 |>=24.10.0a.39,<24.11.0a0 |>=24.10.0a.40,<24.11.0a0 ], which cannot be installed (as previously explained);
  └─ spdlog >=1.14.1,<1.15  is not installable because it conflicts with any installable versions previously reported.

(build link)

Because it's installing rmm / librmm from the rapidsai-nightly channel. Once rapidsai/rmm#1678 is merged and packages are built from it, that job will pass.

KyleFromNVIDIA · 2024-09-20T14:41:55Z

cpp/cmake/thirdparty/get_spdlog.cmake

@@ -16,21 +16,8 @@
 function(find_and_configure_spdlog)

  include(${rapids-cmake-dir}/cpm/spdlog.cmake)
-  rapids_cpm_spdlog(FMT_OPTION "EXTERNAL_FMT_HO" INSTALL_EXPORT_SET cudf-exports)
-  rapids_export_package(BUILD spdlog cudf-exports)
+  rapids_cpm_spdlog(FMT_OPTION "EXTERNAL_FMT_HO")


If libcudf is only using spdlog privately, and its public headers don't depend on spdlog, then I think this is fine.

…spdlog

Update fmt (to 11.0.2) and spdlog (to 1.14.1).

1a3e4b7

jameslamb added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels Sep 16, 2024

github-actions bot added the CMake CMake build issue label Sep 16, 2024

use RMM CI artifacts

6bc98d2

github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Sep 16, 2024

This was referenced Sep 16, 2024

update fmt (to 11.0.2) and spdlog (to 1.14.1) rapidsai/rapids-cmake#689

Open

update fmt and spdlog across RAPIDS rapidsai/build-planning#56

Open

jameslamb added 9 commits September 17, 2024 15:27

Merge branch 'branch-24.10' into fmt-and-spdlog

d3b87bc

test if spdlog export is still needed

c9980ab

Merge branch 'branch-24.10' into fmt-and-spdlog

b294193

move rapids-cmake overrides [skip ci]

13fd917

stop exporting spdlog

3a3e24a

test some builds in CI

8d1bac6

workflow branch

2235405

merge branch-24.10, only run builds

b5ab6e5

comment out static-configure

0a71521

jameslamb commented Sep 19, 2024

View reviewed changes

jameslamb mentioned this pull request Sep 19, 2024

[FEA] rapids_cpm_spdlog generate correct build config support rapidsai/rapids-cmake#387

Open

jameslamb added the 5 - DO NOT MERGE Hold off on merging; see PR for details label Sep 19, 2024

jameslamb changed the title ~~WIP: [DO NOT MERGE] Update fmt (to 11.0.2) and spdlog (to 1.14.1).~~ Update fmt (to 11.0.2) and spdlog (to 1.14.1). Sep 19, 2024

jameslamb marked this pull request as ready for review September 19, 2024 16:52

jameslamb requested review from a team as code owners September 19, 2024 16:52

jameslamb requested review from raydouglass and bdice and removed request for raydouglass September 19, 2024 16:52

bdice approved these changes Sep 20, 2024

View reviewed changes

jameslamb added 3 commits September 20, 2024 07:56

run all CI

a1fce83

Merge branch 'branch-24.10' into fmt-and-spdlog

f962642

cmakelint

9ba1063

KyleFromNVIDIA approved these changes Sep 20, 2024

View reviewed changes

jameslamb added 3 commits September 20, 2024 09:31

export spdlog and mark it as an INTERFACE dependency of cudf::cudf

058b0d3

Merge branch 'branch-24.10' of github.com:rapidsai/cudf into fmt-and-…

b9cdb5d

…spdlog

make spdlog::spdlog_header_only a PUBLIC dependency of cudf::cudf

ab6d345

vyasr approved these changes Sep 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update fmt (to 11.0.2) and spdlog (to 1.14.1). #16806

Update fmt (to 11.0.2) and spdlog (to 1.14.1). #16806

jameslamb commented Sep 16, 2024 •

edited

Loading

jameslamb Sep 19, 2024

jameslamb Sep 19, 2024

bdice Sep 20, 2024

KyleFromNVIDIA Sep 20, 2024

jameslamb Sep 20, 2024

KyleFromNVIDIA Sep 20, 2024

jameslamb Sep 20, 2024

bdice Sep 20, 2024

vyasr Sep 20, 2024

jameslamb Sep 20, 2024

jameslamb Sep 20, 2024

bdice left a comment

bdice Sep 20, 2024

jameslamb commented Sep 20, 2024

KyleFromNVIDIA Sep 20, 2024

	* Global logger object exposed via `cudf::logger()` - sets the minimum logging level at runtime.
	For example, calling `cudf::logger().set_level(spdlog::level::err)`, will exclude any messages that
	are not errors or critical errors. This API should not be used within libcudf to manipulate logging,
	its purpose is to allow upstream users to configure libcudf logging to fit their application.

Update fmt (to 11.0.2) and spdlog (to 1.14.1). #16806

Are you sure you want to change the base?

Update fmt (to 11.0.2) and spdlog (to 1.14.1). #16806

Conversation

jameslamb commented Sep 16, 2024 • edited Loading

Description

Notes for Reviewers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb commented Sep 20, 2024

Choose a reason for hiding this comment

jameslamb commented Sep 16, 2024 •

edited

Loading