Remove wrench, fs-plus in favor of fs-extra (Reviving #836)

[DeeDeeG]

Requirements

• Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
• All new code requires tests to ensure against regressions

Description of the Change

From the original description at #836:

wrench has been deprecated for a few years in favor of fs-extra. However, since both fs-plus and fs-extra happen to extend fs, it would be illogical to use both of them in the same codebase. Therefore, this PR removes both wrench and fs-plus and uses fs-extra instead.

The benefit of this PR over that one, is that this has had master properly merged back into it. There is also a developer available to work on it at the moment (me, @DeeDeeG), and if you have any requests, comments or updates you'd like to see done to this, please do let me know.

Alternate Designs

Possibly reintroducing the fs.coffee wrapper so that try/catch blocks can be consolidated into that file. See this review comment on the original PR: #836 (review)

Benefits

From #836:

• Using a better-maintained fs alternative
• No longer depending on a deprecated package that isn't being updated

Possible Drawbacks

See: #836

As far as I can tell, all the concerns in the original PR were addressed, other than perhaps the review comment above.

Verification Process

CI passes. All command-line commands I have tried work (./bin/apm [command] [args]).

Applicable Issues

None.

[winstliu]

Possibly reintroducing the fs.coffee wrapper so that try/catch blocks can be consolidated into that file.

I'd recommend doing this. I think I was a bit too optimistic when writing the PR that there wouldn't be any exceptions thrown. Just never got around to re-editing all the files 😔.

[DeeDeeG]

As I understand (let me know if this isn't right) the idea is tracking down which methods throw errors vs failing silently, and ensuring those that failed silently before this PR, but now may throw errors, are wrapped in try blocks?

There is native recursive fs.mkdir as of Node 10.12. I can replace all fs-extra.mkdirp with that, since we bundle new enough Node, and older Node versions aren't really supported anymore. Would using a native fs method help alleviate some of the concerns about throwing errors?

For completeness of discussion of native Node options: There is also native recursive fs.rm in Node 12.something, but it's not stable yet, so I suppose we can't really rely on that (it could be removed from Node entirely, or the API could change under us.)

As far as I can tell, there is no native fs way to copy directories and their contents, just single files at a time, but I admit I haven't looked exhaustively at that.

[aminya]

Possibly reintroducing the fs.coffee wrapper so that try/catch blocks can be consolidated into that file.

I'd recommend doing this. I think I was a bit too optimistic when writing the PR that there wouldn't be any exceptions thrown. Just never got around to re-editing all the files 😔.

Happy to hear from you! 🎉 @50Wliu

As I understand (let me know if this isn't right) the idea is tracking down which methods throw errors vs failing silently, and ensuring those that failed silently before this PR, but now may throw errors, are wrapped in try blocks?

@DeeDeeG

Not all of the functions throw an error. You should go through them and check if they throw.

There is native recursive fs.mkdir as of Node 10.12. I can replace all fs-extra.mkdirp with that, since we bundle new enough Node, and older Node versions aren't really supported anymore. Would using a native fs method

fs-extra already exports the native function. Most of them are identical! You should check the documentation of fs-extra.

[aminya]

Personally, I prefer that we add the functions that fs-plus misses (from fs-extra and wrench). We can use a bundler like Rollup to tree-shake and compact the library. We can also make the try-catched version of the functions in fs-plus if needed.

I have a WIP version of fs-plus under development here: https://github.com/atom/fs-plus/pull/48/files

[winstliu]

Would using a native fs method help alleviate some of the concerns about throwing errors?

Not really, unless fs.mkdir can't throw. The issue is that fs-plus suppressed all errors thrown from statSync, which a bunch of the other fs-plus methods used under the hood (existsSync, is*Sync, etc). It's very hard to guarantee that we were using those methods in apm in such a way that statSync would never throw (I certainly didn't look into it too rigorously), and fs-extra doesn't provide the same non-throw guarantees that fs-plus does.

fs-extra already exports the native function. Most of them are identical!

This is great and will make it very easy in the future to switch to the native versions if we so desired :).

Happy to hear from you!

👋 My life's at a point where I can't commit to working on Atom anymore, but I should still be able to pop in once in a while on the issues that I have direct subscriptions to.

[winstliu]

Personally, I prefer that we add the functions that fs-plus misses (from fs-extra and wrench).

I'm not too sure what you mean here; are you saying that we should add the methods from fs-extra/wrench that fs-plus doesn't have to fs-plus?

If so, I would recommend against that. I admit my knowledge of Atom is one year stale at this point, but we realized that fs-plus had served its purpose, and there were better-maintained fs libraries out there.

[aminya]

Personally, I prefer that we add the functions that fs-plus misses (from fs-extra and wrench).

I'm not too sure what you mean here; are you saying that we should add the methods from fs-extra/wrench that fs-plus doesn't have to fs-plus?

If so, I would recommend against that. I admit my knowledge of Atom is one year stale at this point, but we realized that fs-plus had served its purpose, and there were better-maintained fs libraries out there.

Yes, that's what I meant. If there are missing functions, we can add those to fs-plus. Using ES6 capabilities, it is very easy these days to reexport the functions, and we don't need things like a proxy, etc. That means, we can just reexport the functions of fs-extra and wrench!

The API of the some of the functions in fs-plus is different from fs-extra and switching to a new library means that we should change the whole Atom ecosystem. In contrast, by improving fs-plus we just change the code in one place, and all the other packages stay unchanged.

Also, as you mentioned, there are some features that are not available in fs-extra. So, I think there is value in having and improving fs-plus.

However, this is my opinion about fs-plus only. For many of the other libraries, I have a different opinion, and I think there are better libraries out there. e.g nsfw, node-source-map-support, fuzzaldrin, etc.

[DeeDeeG]

For context, it sounds like @aminya you would be willing to have us take on maintaining a fork of fs-plus?

That's doable in the very short (immediate) term, but if we get tired of it, that's one more stale dependency. That's basically a similar situation to why fs-plus is stale today.

If we can use another module that someone else is maintaining, perhaps wrapped in some try blocks when needed, I think that's more sustainable. Most of these modules have stable APIs, so we can keep pulling in updates for the foreseeable future if they stay maintained.

I don't prefer us to start maintaining custom stuff "because we can". The archeologist/anthropologist in me agrees with your assessment that Atom's maintainers did this a lot, then some core folks who used to work heavily on Atom had other life stuff come up, then a lot of Atom's roll-its-own dependencies became unmaintained and/or stale.

So I would lean toward fs-extra, something popular that I know I personally don't have to commit to maintaining. (And inching toward directly using stuff that's part of the main Node API where possible).

[DeeDeeG]

The API of the some of the functions in fs-plus is different from fs-extra and switching to a new library means that we should change the whole Atom ecosystem. In contrast, by improving fs-plus we just change the code in one place, and all the other packages stay unchanged.

I don't think this is exactly true. These are basic operations around moving, stating, copying, deleting files. And that's around installing packages, if that. Not running them or changing how they are compiled...

These are some basic universal file operations. When I delete something, I want to delete it. When I copy something, I want to copy it. The differences between modules, where they exist, are painstakingly documented. It's worth the initial pain of the switch in order to get a widely used module someone else is committed to maintaining.

Or... give up on being current and just use something older. Maintaining new fs-plus features is IMO the worse option than those two. We are reinventing the wheel if we do that. The initial folly of fs-plus, arguably, was to reinvent the wheel. But with hindsight, a lot of Atom code is from a different time, when a lot of popular use-cases were still not covered with popular modules solving them. We can now adopt the modern widespread conventions and popular modules, and benefit from de-duplication of work.

P.S. by substantially changing fs-plus, that would need to be done just as carefully so as to not to break existing use-cases or impact the ecosystem. Just because we wrote it doesn't mean we won't introduce bugs by mistake. (Popular modules have more eyeballs on them to spot mistakes and offer fixes.)

P.P.S. npm does a lot of the heavy lifting in apm, so the notion that the fs module we use heavily changes the way apm works, I mean I just don't think that's the case from a technical standpoint.

[aminya]

I am not against using better libraries, but there should be a benefit in doing so. For example, I am replacing etch with solid. But to make the transition seamless, I am writing a wrapper library to provide the exact same API.

It is not about popularity but about the features and performance we gain by doing this.

• fs-extra just reexports fs functions with a different name. This introduces a layer of useless complexity without adding any benefits.

• fs-extra does not even support ES6. Something I have been planning to do in my PR. This means fs-extra has stayed pre-2015.

jprichardson/node-fs-extra#746

• to rewrite everything using fs-extra, you should rewrite 100s of packages. This is much more tiring than improving an already working library (fs-plus

• Chaing the code in 100s of packages is riskier than improving the code in one place. Here in apm we have rewritten this, but you can't rewrite 100s of other packages.

• If there are any functions or features missing from fs-plus, we can add it by reexporting, and then use tree-shaking to get rid of the additional code.

If there is a native module which offers promisified fs functionality (like Deno), then we can discuss writing a wrapper around it inside fs-plus.

[DeeDeeG]

Sorry for the confrontational tone yesterday. I guess I was grumpy about unrelated life stuff.

We don't have to get rid of fs-plus everywhere, but for the particular needs of just this repo, and keeping the core infrastructure current, I think this is for the best. I'd like it to keep working and have bump-able dependencies even if we suddenly vanish. It's about leaving the core infrastructure in a good state for the next folks who come around.

(Leaving it in a "good" state not meaning the same as "excellent." Don't let the perfect be the enemy of the good. New tech has to be thoroughly proved out, which takes time I don't know that we have. We can and should move forward with something, even if it isn't ready for ES6. Which is stil experimental in Node, by the way, so by convention not suitable for core infrastructure stuff: https://nodejs.org/api/esm.html#esm_ecmascript_modules)

Maintaining a modern fs-plus is a task that requires active Atom maintainers, which are in short supply. It's a "nice to do" for now, while we're active, but demonstrating a clear path to non-custom and widely used dependencies is valuable to the ecosystem's maintainability. Others can copy the approach here if they want to.

Popularity isn't valuable in a vacuum, but it means the package gets a lot of attention, is likely to receive bugfixes, and in general is less likely to become suddenly unmaintained without a clear replacement.

I'm averse to spreading ourselves thin on things like maintaining a working custom fs module ourselves, when that task has been handled by the wider community already.

I wouldn't want to stop you from modernizing fs-plus, but I still think it's good for apm to rely on something maintained by the broader community (deduplication of work, addresses the bus factor, separation of concerns).

[aminya]

If you want to do this only for apm, you should first merge all the open pull requests. This pull request is changing all the files of apm and so a lot of conflicts are going to be introduced because of merging this.

[DeeDeeG]

I like merging larger PRs first, but I am confident in my ability to rebase either way, since my other PRs are tiny. These are supposed to be drop-in replacements, so if we do this PR right, the trail of carnage left behind should be minimal. (And I'll understand these changes, being a partial author to them...)

Anyhow... I have to do the boring or tedious part (and out of my expertise area) of researching which methods throw or don't.

I'll be looking at the main source .coffee file at fs-plus to see how things should work, and trying to replicate the API exactly if that's what we need to do. If some intervention is needed to make this happen, coercing a method to behave slightly differently, I expect to do it in fs.coffee.

Tentative plan, but it's better than no plan! I may not get postable progress on this right away. @aminya you are more the expert among us two for javascript. If you wanted to work on this I'd be glad for it. But we have plenty of things to choose from in terms of what to work on and in the mean-time we can use apm as-is. It's not urgent that it happen quickly.

I'd like to be able to document and/or describe how this is the case (that htese are suitabke drop-in replacements) in a way good enough for a reviewer to (hopefully easily) verify. Or understand it deep enough to answer questions and assure such a reviewer of the details.

But that's getting ahead of myself.

In summary: next step is researching fs-plus to basically reverse-engineer it and help ensure the reimplemented methods are truly drop-in replacements. One by one. Sounds like fun! 😛

[aminya]

Related to this. atom/fs-plus#50