Proposed Keystone Package Architecture

[estilles]

Proposed Keystone Package Architecture

Now that we're approaching the big 3.0 (or should I say 0.3.0), I think it would be a good idea to revive the Plugin Architecture issue once again. This is just an initial first draft of my ideas on the topic. Please feel free to comment and criticize at will. I don't presume that my ideas are the best solution for this issue. However, I have already successfully implemented a subset of these in some of my client's apps and, with their permission, I'm sharing them here.

Before we get to the technical stuff, some folks have suggested to me that I should use terms such as modules or packages, as opposed to plugins. I know its just a matter of semantics, but at some point we should probably agree on what terminology to use. I prefer not to use the term modules to avoid confusion with npm modules. So, for the sake of this discussion, I will refer to them as packages. We can always revisit the semantics issue at a later time.

Package Categories

The way I envision Keystone's package architecture, practically every Keystone feature will be encapsulated in a package. Packages, will be divided into four (4) major categories:

Category	Description	Examples
System	There will be at least 2 system packages (maybe more): the Keystone engine and the Keystone Admin-UI	Keystone, admin-ui
Core	Will include many of Keystone's core functionalities	core fields types, basic authentication
Contributed	User contributed packages providing functionality not provided by Keystone core packages	storage, third-party authentication, custom admin-ui themes
Custom	Application specific functionality	anything else we can think of :-)

Not all of these package categories will be available on the initial version. Check out the next section (the 3 Phased Approach) to see how I propose to phase these into the project.

3 Phased Approach

Even if done correctly, this project will be a massive undertaking, and will require contributions and feedback from the entire Keystone community. Given the size and complexity of the project, not to mention the potentially breaking changes it may introduce, we need to carefully plan its implementation.

The following three (3) phased approach, in my opinion, will help us mitigate these issues.

Phase	Description	Impact
I	In this phase the project will basically be an add-on to Keystone. If detected, Keystone will initialize the package architecture, which will in turn discover and load any packages that are present, within a prescribed directory structure. Only contributed and custom/local packages will be available in this phase.	Low
II	Here is when we convert some of Keystone's functionality into core packages.	Medium
III	Finally, we should convert the Keystone engine and the Admin-UI into system packages.	High

Phase I should be a relatively easy to implement and should not introduce any breaking changes, given that the architecture itself will be an optional feature.

Phase II, should not include any breaking changes either. If designed and implemented correctly, core packages should work seamlessly with existing applications. This phase will, however, force developers to use/understand the package architecture. Hopefully, they would have already done so during Phase I.

Phase III is a whole different ballgame and will definitely break existing apps. Keystone configuration and bootstrapping will likely undergo dramatic changes. We can, of course, mitigate this with proper documentation and dissemination of the breaking changes, but it's something we definitely need to consider.

Proposed Feature Set

Our package architecture should including (at minimum) the following feature set:

• Ability to publish packages
• Ability to install published packages
• Ability to create local/custom packages
• Package generator (initial scaffolding)
• Package auto-discovery
• Dependency injection
• Third-party library management (added per @cameronjroe's suggestion)
• Client-side asset aggregation (for css and javascript)
• LESS/SASS/STYLUS support (added Stylus per @morenoh149's suggestion)
• Custom routes/route controllers
• Custom views
• Custom models
• Package mount path should be configurable (though each package should have a "default")

Not all of these features will be available on the initial version. We need to come up with a road map specifying a target version for each feature.

Can anyone think of any features I may have missed?

Package Publishing/Installation

Publishing and installing packages is one of the most important features of our package architecture. It is imperative that package publication and installation be as easy as humanly possible. There are a number of viable options we can pursue to accomplish this.

1. Use existing tools (npm/yeoman)
2. Use of Git submodules
3. Create a CLI tool for package management

The npm/yeoman alternative seems like the most logical candidate to me right now. It's simple and straight forward, and we're all quite familiar with these tools. That said, as Keystone matures, I would like to see a Keystone CLI tool to handle these and other rapid development functions. A Keystone CLI, though, is way beyond the scope of this document, and is something, the viability of which, we should address at a later time.

Third-Party Library Dependency Management

Thanks to @cameronjroe's for bringing up this issue.

Third party library dependency installation and deconfliction is a reality of any system with community developed packages. Fortunately, we already have excellent tools available to help us with this task. We can use both npm and bower for this purpose.

Just like with the package publishing/installation subject above, we can tackle this by implementing a well-defined workflow that employs these tools (npm/bower) or we can create our own CLI tool which would take care of this workflow for us.

The choice between a manual workflow versus a custom CLI tools warrants further discussion.

Proposed Directory Structure

The following is what I consider should be the basic directory structure for the project. It is self-explanatory and will make package discovery very simple to code.

packages
--- system/
    --- keystone/
    --- admin-ui/
--- core/
    --- field-types/
    --- admin-ui-theme/ (... or maybe just "theme/")
    --- auth/
--- contributed/
    ...
--- custom/
    ...

Each package, in turn, will have the following proposed structure:

sample-package/
--- public/ (optional, will be served statically by Express)
    --- assets/
        --- css/ (can optionally be aggregated and minimized)
        --- js/ (can optionally be aggregated and minimized)
        --- img/
--- server/ (optional)
    --- fields/ (will contain custom React fields)
    --- models/ (will contain Mongoose models that will automatically be loaded)
    --- routes/ (will contain Express routes that will automatically be loaded)
    --- controllers/ (route route controllers and middleware)
--- index.js (required, will contain package registration, initialization code, and dependency requirements)

This file structure, I believe, is flexible enough to create almost any type of package (with both front-end and backend features), from a simple React field type, to an entire authentication subsystem.

Package Generation

Just like for package publication/installation there are a number of viable alternatives for this task. Since we already have a Yeoman generator for keystone apps, I figured the simplest solution would be to extend it with sub generators for different types of packages. keystone:fieldtype and keystone:theme are two that immediately come to mind, but I'm sure we can think of more.

What ever we use, we need to ensure that, in addition to the basic scaffolding for the package, it also include basic tests to help/encourage package developers with unit-testing.

Package Dependency Injection

When I started to tackle the dependency injection issue in my own projects I started developing my own DI library, when I came upon an npm module called dependable. It's an open source DI library developed by the folks at i.TV, in which every package takes the form of a container. Containers can be anything (a string, an object, a function, etc.) and can specify any other containers as its dependencies. I found it simple and elegant, and have been using it for several months now without incident.

Here's an example of how to use dependable.

var dependable = require('dependable');

var containerA = dependable.container();
containerA.register('packageA', function () { // <-- define packageA
    var package = {};

    .... (some initialization code)

    package.doThis = function doThis() {
        ...
        return 'I did this';
    };

    package.doThat = function doThat() {
        ...
        return 'I did that';
    };

    return package;
});

var containerB = dependable.container();
containerB.register('packageB', { 'something': 'Some data ...' }); // <-- define packageB

var containerC = dependable.container();
containerC.register('packageC', function (packageA, packageB) { // <-- inject packageA and packageB
    var package = {};

    package.doSomethingElse = function() {
        console.log(packageA.doThis()); // <-- use injected packageA 
        console.log(packageA.doThat()); 
        console.log(packageB.something); // <-- use injected packageB
    };

    return package;
});

It's that simple. Those of you who have used Angular and other frameworks with DI will find this very familiar.

I believe dependable is an excellent fit for this project. However, if anyone has any other viable alternatives I am open to suggestions.

Client-Side Asset Aggregation

To deal with package asset aggregation on the client side I've been using an npm module called node-assetmanager.

Asset aggregation and minification is definitely a feature we need to have available for this project. I've only used node-assetmanager in a single project, so I can't say for certain it is the ultimate solution for us. While I have yet to encounter any negative issues with node-assetmanager, I have by no means covered the wide number of use cases we will likely face.

So far, this is the best module I've found for this, but alternative suggestions are more than welcome. :-)

Keystone Events

The package architecture will not directly handle anything related to Keystone events. However, the Keystone object will be available as an injectable dependency, so package developers can choose to use it to listening for/hook into Keystone events. So long as the Keystone event system is well documented this should not be an issue.

Conclusion

That's the gist of it! I hope I didn't miss anything. Like I mentioned earlier, my intention is just to restart the discussion from issue #185, so this is by no means a comprehensive outline of the project.

A package architecture for Keystone will open a world of possibilities. The ease with which the community would be able to contribute new functionality makes this project not just attractive, but a logical next step for Keystone.

@JedWatson, is this more or less a direction in which you want to take Keystone?

Once we agree on a general architecture, the next steps will be to come up with a roadmap and a well-defined API, followed by a prototype. I know we're not there yet, so let's discuss!

[morenoh149]

I'd like to see STYLUS listed along sass and less :3
For the proposed directory structure could it be two folders? core & custom. Otherwise sounds exciting.

[estilles]

@morenoh149 I forgot about Stylus, will add to the writeup. Regarding the dir structure, IMHO, I believe we should have at minimum 3 folder (core - obviously, custom - for local/non-published packages, and contributed - for published packages that were installed).

I really appreciate the comments. :-)

[ninjasort]

What about in the case where two packages use the same vendor library in the client js? Would that potentially create duplication across packages? Any thoughts on using bower's api?

[estilles]

@cameronjroe, you're right on the money. Sorry I failed to mention dependency management for the packages. My intention is to use a combination of both npm and bower to the installation/deconfliction of both client and server third party library dependencies. We would either need to establish some sort of clear and well-defined workflow (which would require manually invoking npm/bower) or make it part of a CLI tool for package management (using the APIs for npm/bower).

Thank you so much for bringing it up. I will updated the write-up to reflect your suggestions.

[ninjasort]

Cool! One other question I had in regards to user contributed packages is how one might potentially hook into Keystone's lifecycle at different points. Would a package author be able to utilize the keystone npm/bower modules to hook into Keystone's lifecyce? Essentially utilizing it in their package for integration with keystone?

[estilles]

@cameronjroe, if I understood your question correctly, my answer is yes. Packages, as I envision them in this architecture, will be discovered and initialized by Keystone. During the initialization process, the discovered package will be able hook itself into different aspects of Keystone's lifecycle (create routes, listen to events, create/replace views, etc.) How exactly we will accomplish this is yet to be determined. As you can see, this discussion is still in its infancy. We welcome any suggestions and insights from the community on the subject.

[wmertens]

2 comments:

• With the admin interface being in React, it becomes easy to provide custom components to keystone for the admin UI so you can have custom editors for certain fields etc. Obvious, but I wanted to point it out.
• It would be great if Keystone could also incorporate a best-practices build system (webpack + HMR etc of course) in such a way that projects simply have to provide a minimal configuration and the building happens by Keystone.

Both comments have a common theme: Keystone should be something a project require()s and declaratively configures, with minimal-to-no boilerplate. That way when Keystone upgrades, projects can simply enjoy the new bugfixes/features without being stuck in the tarpit that code generators make.

[LorbusChris]

I'd like to share my thoughts on this, too. It would be awesome if keystone went really modular in a way that would allow for a pluggable approach looking something like this:

• Keystone
  • KS Core (Keystone engine, adminUI, fields types, authentication, server(express,koa))
    • Is it really necessary to split Core + System up?
  • KS Contributed (storage, third-party authentication, custom adminUI themes)
• Index/Router (standard entry point for the browser, integrate React-Router here?)
• Public/Custom
• Style Sheets (maybe via jss?)

In any way, keep up the good work guys, thumbs up :)

[JedWatson]

@wmertens @LorbusChris +1k

This is absolutely the direction we're going in. I actually have some thoughts that, down the track, would allow the Admin UI (or other components) to be used independently of the back-end service, which would then open up all sorts of possibilities for alternate databases, architectures, etc.

The move to React is really opening up a lot of possibilities and right now we're focusing on what's in front of our noses (rebuild existing functionality, port everything to a REST API w/ SPA Admin UI, rework Lists so custom Field types can be used, and port the UI to use Elemental) but once this is complete we're going to really open the flood gates on modularity and customisation 😀

I'll be doing more detailed write-ups as soon as we cross those milestones.

[sachingill]

I like the idea of separating admin-UI and providing rest endpoints. How can I track or check this progress. Are these feature incorporated in keystone.

[morenoh149]

@sachingill you can watch the keystone project to get notifications.

[wmertens]

Right now this is the longest-ago-updated issue. @JohnnyEstilles is there anything you want to change about the issue given 0.4? Is everything still valid?

What would be the first steps (issues) to get there?

[gautamsi]

Keystone 4 is going in maintenance mode. No major changes expected. see #4913 for details.

https://github.com/keystonejs/keystone-5 is built ground up with modular approach.