chore: add support for Python 3.13 #712

veenstrajelmer · 2024-10-23T11:32:25Z

Steps:

tried a lot since it was cumbersome to add python 3.13 support before merging chore: add support for numpy v2 #660 and chore: drop python 3.8 support #710
after merging these PR's, it just worked when I added python 3.13 to ci.yml
the python 3.13 wheels for windows for netcdf4 were published 1 day after the rest of the netcdf4 1.7.2 release. There was some inconvenient timing, so the lockfile had to be updated without using cache (so it would search for new netcdf4 wheels again) with poetry lock --no-cache
poetry version was updated, just because
still failed windows with py3.13 tests, but with SystemError: <method 'is_done' of '_thread._ThreadHandle' objects> returned a result with an exception set (details below). A local poetry install showed issues for numpy installation, which is resolved by specifying a different minimal numpy version for py 3.13 and higher. This also resolved this (apparently related) issue.
however, two tests are failing on tkinter errors (see below). Might be related to the fact that we use virtualenvs? win 10 Python-3.13.0 tkinter fails in venv python/cpython#125235. It seems that it can only be fixed with python 3.13.1 (expected in December 2024).

SystemError details (resolved)

Windows fatal exception: access violation
Current thread 0x00000bb0 (most recent call first):
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\numpy\_core\getlimits.py", line 52 in __init__
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\numpy\_core\getlimits.py", line 232 in _register_known_types
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\numpy\__init__.py", line 295 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\meshkernel\py_structures.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\meshkernel\errors.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\meshkernel\__init__.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\net\models.py", line 7 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\net\__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\mdu\models.py", line 24 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\mdu\__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\__init__.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\tests\dflowfm\ini\test_ini.py", line 9 in <module>
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\assertion\rewrite.py", line 184 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "C:\hostedtoolcache\windows\Python\3.13.0\x64\Lib\importlib\__init__.py", line 88 in import_module
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\pathlib.py", line 582 in import_path
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 493 in importtestmodule
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 546 in _getobj
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 284 in obj
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 562 in _register_setup_module_fixture
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 549 in collect
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 389 in collect
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 391 in pytest_make_collect_report
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_manager.py", line [12](https://github.com/Deltares/HYDROLIB-core/actions/runs/11601906361/job/32306114261#step:9:13)0 in _hookexec
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 567 in collect_one_node
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 835 in _collect_one_node
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 970 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 809 in perform_collect
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 347 in pytest_collection
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_hooks.py", line 5[13](https://github.com/Deltares/HYDROLIB-core/actions/runs/11601906361/job/32306114261#step:9:14) in __call__
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 336 in _main
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
Exception ignored on threading shutdown:
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.13.0\x64\Lib\threading.py", line [15](https://github.com/Deltares/HYDROLIB-core/actions/runs/11601906361/job/32306114261#step:9:16)24, in _shutdown
    if _main_thread._handle.is_done() and _is_main_interpreter():
SystemError: <method 'is_done' of '_thread._ThreadHandle' objects> returned a result with an exception set

Tkinter error:

=========================== short test summary info ===========================
FAILED tests/dflowfm/test_net.py::test_create_1d_by_branch - _tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    C:/hostedtoolcache/windows/Python/3.13.0/x64/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/library D:/a/HYDROLIB-core/HYDROLIB-core/library D:/a/HYDROLIB-core/HYDROLIB-core/tcl8.6.14/library D:/a/HYDROLIB-core/tcl8.6.14/library
This probably means that Tcl wasn't installed properly.
FAILED tests/dflowfm/test_net.py::test_create_1d_2d_1d2d - _tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    C:/hostedtoolcache/windows/Python/3.13.0/x64/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/library D:/a/HYDROLIB-core/HYDROLIB-core/library D:/a/HYDROLIB-core/HYDROLIB-core/tcl8.6.14/library D:/a/HYDROLIB-core/tcl8.6.14/library
This probably means that Tcl wasn't installed properly.
=========== 2 failed, 1426 passed, 10 skipped, 1 warning in 41.32s ============

sonarcloud · 2024-10-31T14:50:54Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

veenstrajelmer and others added 7 commits October 9, 2024 09:29

add python 3.13 to ci.yml

29db20e

allow lxml 5

713d71b

deleted lockfile

aaa5b0c

add minimal pandas version for py3.13

045f59c

add minimal pandas version for py3.13

5f83774

updated comments

470edc8

avoid conflicts

159f316

veenstrajelmer linked an issue Oct 23, 2024 that may be closed by this pull request

Investigate Python 3.13 support #707

Open

7 tasks

veenstrajelmer changed the title ~~chore: add support for Python 313~~ chore: add support for Python 3.13 Oct 23, 2024

veenstrajelmer added 3 commits October 23, 2024 14:00

resolved conflicts

3190a32

updated lockfile

51c1ddf

comments

9906b84

veenstrajelmer requested a review from priscavdsluis October 23, 2024 12:07

veenstrajelmer added 4 commits October 23, 2024 14:18

lockfile

8d90ea7

updated poetry lockfile

bd148d2

updated poetry lockfile

bed1c37

updated numpy dependency and lockfile

8bb8627

veenstrajelmer mentioned this pull request Oct 31, 2024

Run Python 3.13 testbank Deltares/dfm_tools#559

Open

20 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add support for Python 3.13 #712

chore: add support for Python 3.13 #712

veenstrajelmer commented Oct 23, 2024 •

edited

Loading

sonarcloud bot commented Oct 31, 2024

chore: add support for Python 3.13 #712

Are you sure you want to change the base?

chore: add support for Python 3.13 #712

Conversation

veenstrajelmer commented Oct 23, 2024 • edited Loading

sonarcloud bot commented Oct 31, 2024

Quality Gate passed

veenstrajelmer commented Oct 23, 2024 •

edited

Loading