Fix 3rd Party Imports Cross-Platform

[switchupcb]

What information does an external contributor need to fix this specific issue; such that pre-installed Go modules (by this user; before runtime) can be used by the interpreter at runtime programmatically (without requiring the user to recompile each time)?

We are implementing dynamic template functionality with the use of an interpreter. We are currently using yaegi, which has trouble supporting third party go modules during interpretation without prior reflection. It seems to me that gomacro can use 3rd party libraries programmatically, with a few limitations: recompilation required on platforms with majority-usage share. For more information, read Usage Share of Operating Systems.

Example

Let's say that I want to be able to import an already-installed function in generate.go by using:

import (
	"fmt"

	"github.com/switchupcb/copygen/cli/generator" // third party module; not reflected
	"github.com/switchupcb/copygen/cli/models" //  extracted module
)

func Generate(gen *models.Generator) (string, error) {
	content := string(gen.Keep) + "\n"
	fmt.Println(generator.GenerateFunction) // custom type in third party module (a string alias) // or any other object
        ...
}

When I have already go get github.com/switchupcb/copygen@... prior to running this code.

Further Inquiry

Is using gomacro possible in gomacro?

[cosmos72]

Hello @switchupcb,
gomacro supports importing third-party packages without recompilation only on Linux and Mac OS X, because it compiles them as a shared library then loads the shared library with Go function plugin.Open() which is currently only available on those two operating systems - as far as I know, nobody implemented it yet on Windows.

For your specific case, I tried executing

import (
	"fmt"

	"github.com/switchupcb/copygen/cli/generator" // third party module; not reflected
	"github.com/switchupcb/copygen/cli/models" //  extracted module
)

inside gomacro running on Linux/arm64 (inside UserLand on my Android mobile) and it produced some errors:

// debug: running "go get github.com/switchupcb/copygen/cli/generator" ...
go: downloading github.com/switchupcb/copygen v0.2.4
go: downloading github.com/switchupcb/yaegi v0.10.2
go: downloading golang.org/x/tools v0.1.6
go: downloading golang.org/x/sys v0.0.0-20210927052749-1cf2251ac284
go: added github.com/switchupcb/copygen v0.2.4
go: added github.com/switchupcb/yaegi v0.10.2
go: added golang.org/x/mod v0.5.1
go: added golang.org/x/sys v0.0.0-20210927052749-1cf2251ac284
go: added golang.org/x/tools v0.1.6
go: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
// debug: running "go mod tidy" ...
// debug: compiling "/home/max/go/src/gomacro.imports/github.com/switchupcb/copygen/cli/generator/generator.go" ...
# golang.org/x/tools/internal/typeparams
/home/max/go/pkg/mod/golang.org/x/[email protected]/internal/typeparams/typeparams_go118.go:78:6: sig.SetTypeParams undefined (type *types.Signature has no field or method SetTypeParams)
/home/max/go/pkg/mod/golang.org/x/[email protected]/internal/typeparams/typeparams_go118.go:88:6: sig.SetRecvTypeParams undefined (type *types.Signature has no field or method SetRecvTypeParams)
/home/max/go/pkg/mod/golang.org/x/[email protected]/internal/typeparams/typeparams_go118.go:98:15: iface.IsConstraint undefined (type *types.Interface has no field or method IsConstraint)
/home/max/go/pkg/mod/golang.org/x/[email protected]/internal/typeparams/typeparams_go118.go:144:7: info.Inferred undefined (type *types.Info has no field or method Inferred)
// more errors...

I will have a look if it works better on a PC (i.e. Linux/amd4).

About using gomacro inside gomacro: there are at least three different ways to do that.

1. Using some kind of eval() function to interpret arbitrary source code. This is supported natively: from gomacro prompt (or equivalently, when starting gomacro as a library from compiled code) you can use the Eval() function.

2. You can import "github.com/cosmos72/gomacro/fast" from interpreted code to load and instantiate another interpreter, and interact with it: send code to be executed to it, etc.

3. Interpret the interpreter: if you try to load gomacro source code inside another gomacro interpreter and run them (i.e. interpreting the interpreter) they will not work for several reasons - the main one is that gomacro relies on unsafe.Pointer in someplaces, but does not implement it: interpreted code cannot* currently use unsafe.Pointer.

[cosmos72]

Update: import ( ... ) now compiles and imports multiple packages using a single Go plugin - which helps solving dependency versions conflicts.
We are getting nearer, but the imports above do not work yet. The following happens on Linux/amd64:

gomacro> import (
           "fmt"
       
           "github.com/switchupcb/copygen/cli/generator" // third party module; not reflected
           "github.com/switchupcb/copygen/cli/models" //  extracted module
           )
// debug: running "go get github.com/switchupcb/copygen/cli/generator github.com/switchupcb/copygen/cli/models" ...
go: downloading github.com/switchupcb/copygen v0.3.7
go: downloading golang.org/x/tools v0.1.10
go: downloading github.com/switchupcb/yaegi v0.10.2
go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
go: downloading golang.org/x/sys v0.0.0-20220422013727-9388b58f7150
go: downloading golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f
go: added github.com/switchupcb/copygen v0.3.7
go: added github.com/switchupcb/yaegi v0.10.2
go: added golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
go: added golang.org/x/sys v0.0.0-20220422013727-9388b58f7150
go: added golang.org/x/tools v0.1.10
go: added golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f
// debug: running "go mod tidy" ...
// debug: compiling plugin "/home/max/go/src/gomacro.imports/gomacro_pid_89600/import_1" ...
error loading plugin "/home/max/go/src/gomacro.imports/gomacro_pid_89600/import_1/import_1.so": plugin.Open("/home/max/go/src/gomacro.imports/gomacro_pid_89600/import_1/import_1"): plugin was built with a different version of package golang.org/x/sys/execabs