A little help on why a package may not be working

[mauricioszabo]

This will be a long post, but I decided to explain a little on why a specific package can run under Atom 1.60 and not on Pulsar. I'll organize this discussion sorting by "the most probable problem" to the "less probable problem", with troubleshoot on how to fix either the package or Pulsar itself.

TL; DR; - we bumped Electron and that broke some web APIs, or Node APIs (specially native ones or Javascript that's not valid anymore) that some packages are using - most package authors are not working on they anymore, so these are not being fixed.

Intro:

Pulsar shares almost 100% of Atom's code. The only differences we have are the new tree-sitter implementation, a newer Electron (12, vs 9 on Atom 1.60), and a new way of packaging the binary (which also drops V8 Snapshots - more on this later).

Now these changes can, and in some cases will, break packages:

Electron update

Updating Electron means updating Node.JS, Chromium, and Electron itself. Each of these can make packages don't work anymore, so let's again sort by "most probable" to "less probable":

Chromium Updates

At the time Atom was developed, there was an API for webcomponents V0 that only Chrome/Chromium implemented. This API was so flawed that no "formal documentation" exists now. Atom decided to implement most things in this API, and because of this decision, some package authors followed the same path, just for Webcomponents to be deleted on newer Chromium (and, unfortunately, on Electron 11) - meaning that Pulsar uses a polyfill to try to "simulate" this old API, and it's completely flawed (because the current webcomponents are, on the good tradition of the web and Javascript, completely incompatible with the old version - to the point we actually can't convert from one to the other).

Possible fixes are to migrate the package to the new webcomponents version, or to help on the polyfill. We current have our own copy of this V0 polyfill just to solve, partially, a bug where some methods were not being passed to components because the original repository is archived.

Node.JS updates

On a newer version, Node decided to enforce a strict policy on Javascript's ESM modules. This means that, for some Pulsar packages, they will start to flag some otherwise valid Javascript code as "invalid" or containing "syntax error"s. There might be other breaking changes, but the biggest will be the native API.

Basically, most packages that depend on some native module (like Hydrogen, terminal packages, etc) will either use NAPI or Nan (two ways of writing packages in languages like C or C++). If the package is written with the NAPI bindings, it'll probably not give us too much problems on Pulsar (for now); if it's written in Nan, there may be issues - depending on the Node.JS version, the APIs change a little bit, things get deprecated, etc...

... but there's more to it. Basically, an Electron version binds a specific Node version, but when Native APIs come to play, Electron might remove an old, deprecated API, faster than Node (which happened basically on Electron 15, where it supposedly is running Node.JS version 16, but things that do install on Node 16 will not install on Electron 15).

To solve these issues, there's nothing Pulsar can do - the package's author need to either fix the native APIs, the Javascript code, or both.

Electron native APIs problems

There's an issue that we still don't have, but we'll soon have - that is, native APIs will be harder to handle on newer Electron versions. Starting from Electron 14, modules will need to be "context-aware" - meaning that they need to change, and avoid using global variables or even memcpy and other direct-memory access APIs to generate JS objects.

But it may be worse than that, because starting on Electron 21, there's a new feature called V8 Memory Cage that may actually break even context-aware and perfectly fine native modules (and the reason why we migrated TextMate grammars to WASM instead of using the native module).

Again, there's nothing Pulsar can do on this - if we don't bump Electron versions, we will stay with a slower, fragile, more insecure editor that can't run on modern operating systems. Bumping to Electron 12 allowed us to get Pulsar working on Silicon Apple, and bumping further will add support for Wayland (and solve some slowdowns on Apple too)

Dropping V8 Snapshots

Custom V8 Snapshots are an interesting way to make Electron (and other V8 apps) to load faster, but they come with a huge cost - maintaining them.

Atom made a quite complicated library that allowed one to generate the right command to generate these snapshots, and the way that it's configured is by making a script that defines what can be snapshotted, and what can't. Here we have the old Atom way of configuring the snapshots - almost 300 lines of code...

When we dropped the snapshots, Pulsar got a performance hit to load - which was a price we paid to allow for faster development and to make some bold changes like the TextMate mentioned above, experimental Electron bumps, the new tree-sitter and some other changes. We also removed more than 5,000 lines of code that basically generated the "binary" format of Atom - the .deb, .rpm, and other versions of the editor, in favor of using Electron Builder which is more reliable, works better, supports more formats, and the most important element of all - it's not maintained by us so we can focus only on Pulsar and not on the complicated intricacies of building specific operating system's formats.

But there was a catch - V8 snapshots allowed some instances of the atom global to be used in places that the global wasn't defined - and as someone that solved some of these bugs, I honestly don't know how that was possible. That gave us some bugs on some packages (that we already solved) and for the last few months, we haven't seem any bug happening on a package because of this change - but I decided to explain that it might be a remote possibility anyway.

Conclusion

While Pulsar is the same codebase, with the same API, that Atom had, we bumped some dependencies, some libraries, and changed the way we package things. Unfortunately, the Node.JS world is not stable and together with Electron it's basically an ecosystem that don't care too much for any kind of backwards compatibility. So, while we try really hard to avoid breaking stuff, the same can't be said for everything else we're using - specially when it comes to Electron.

Another thing is that we have to be aware that most packages were written for Atom - and now that Atom is dead, most of these packages are abandoned, and the author might not want to keep developing it or publishing a new version to the Pulsar registry. So, unfortunately, unless somebody forks the package, change stuff, and make it work again, there's few things we, the Pulsar team, can do.

[confused-Techie]

@mauricioszabo This is a fantastic post! I really appreciate you putting this all together, and I bet this can be an invaluable resource for any package authors that want to update their packages to work on Pulsar, or any community members attempting to update other packages.

Really this seems like a good of place as any to continually add on any big breaking changes, to allow them to all be found in one place