Support Py_LIMITED_API in separate package as a dependency to PyCall.jl

[kdheepak]

The Python C ABI/API has stability guarantees for a limited number of structs and functions.

https://www.python.org/dev/peps/pep-0384/

It'd be nice to support this as a separate package and use that package as a dependency in PyCall.jl. This separate package can implement the Py_LIMITED_API structs and functions without invoking the Python interpreter.

A (very early) prototype is located here:

https://github.com/kdheepak/Python.jl/blob/33b98177b0e624588a3f46ccfa96b317d57f1201/src/Python.jl

The motivation for this is so that we can build a shared object from a Julia file using PackageCompiler.jl that is also a Python module. See screenshot below as an example:

No linking to the Python library occurs at compile time and the Python module does not embed a Python interpreter; it is a pure Julia shared library. This allows one to create a Python module without having Python installed and the structs and functions described in the Py_LIMITED_API would be sufficient to make this work.

Taking this a step further, we could allow for something like the following in the future (See https://github.com/yglukhov/nimpy and https://github.com/PyO3/PyO3 for similar projects in Nim and Rust):

Write Julia code

# hello.jl
using Python

@pycallable function hello(name::String)::Cvoid
    println("Hello $name!")
end

Compile

$ julia --project juliac.jl -vasij `pwd`/hello.jl > /dev/null

Run in Python

$ ipython
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import hello

In [2]: hello.hello("John")
Hello John!

In [3]:

My understanding is that this would pose some challenges with the current setup in PyCall.jl since PyCall.jl initializes a interpreter in __init__ and when PackageCompiler runs, the references to that interpreter are embedded in the shared library.

Comments?

[tkf]

since PyCall.jl initializes a interpreter in __init__

PyCall will not initialize the Python interpreter if it is already initialized

PyCall.jl/src/pyinit.jl

Line 152 in 9f43875

already_inited = 0 != ccall((@pysym :Py_IsInitialized), Cint, ())

the references to that interpreter are embedded in the shared library

IIUC, it doesn't have to be the case:

And is the constantness of the lib path really necessarily for efficient ccall?

According to Jameson, yes, it really is. Fortunately, we can work around this because by manually dlopen()'ing everything in __init__() we don't have to pass the absolute path in, we just need to pass in the SONAME of each library.

--- JuliaLang/Pkg.jl#841 (comment)

We'd need to do some runtime check in __init__ to make sure that libpython is ABI compatible with the precompiled cache of PyCall.

However, IIRC, if you compiling PyCall into sysimage this current limitation doesn't apply. PyJulia already implements a way to build a custom system image https://pyjulia.readthedocs.io/en/latest/sysimage.html. It should be possible to include custom packages and also making it relocatable. See also JuliaPy/pyjulia#327.

[kdheepak]

Thanks for the detailed answer! I was not able to get PackageCompiler.jl to work when compiling a script I'd written in Julia using PyCall, which was the impetus for opening this issue. I'll try using pyjulia's sysimage feature and get back to you.

[tkf]

Including packages in PyJulia's sysimage is not documented ATM JuliaPy/pyjulia#311. You probably need to dig into the code (which is not so complicated). If you have some trouble, ask it in PyJulia.

By the way, I'm not against the idea of decoupling the "Python.h" part and the runtime part in PyCall. For example, since you can initialize multiple Python interpreters in the same process, it may be interesting to set up a Python interpreter for each thread in Julia. Having a separated "Python.h" package maybe makes experiment like this easier.

If you want to go this direction, maybe you can first create the package by just copying the relevant code from PyCall? I guess it wouldn't be hard to add it as a dependency of PyCall after it drops Python 2 support. (If @stevengj is not against this approach.)

[kdheepak]

That's what I was planning to do! I wanted to lift pieces from PyCall.jl, and then submit a PR in the future to add it as a dependency, if there's interest to do so. I'll look at pyjulia first, and then reassess the plan.