Multiple language support

[sebmck]

Consolidates #744, #39

[morenoh149]

Keyword i18n i18next

[tomasztunik]

Happy to see it on the list for 0.4.0 - Would be amazing if there was a "tab switch" to change between different languages on create/edit forms if multiple languages were set for given model. Was going to look into that myself as I'm in dire need of that for my current project but currently can't really see an easy way to extend the fields without forking and mutating core code - bit afraid to break compatibility for updates at this early stage of 0.3.0 (still going over everything you've just released, great job with 0.3!)

What I'm looking into is elegant and "keystone" way for localised API responses for my SPA that would play nicely with the admin interface. It handles localised routes on client side but I could imagine that for server side rendered apps and pages also localised routes could be something desired.

Also localised fields would be awesome for non text types like money (item cost for shopping cart for example) or even localised images further down the line in case you wanted to prepare alternative images for different languages as sometimes cultural differences/fashion might make more sense for one language then for another.

Have you guys talked about any ETAs for multilang/i18n support? Do you have something in the works or planned already?

[morenoh149]

I'm looking into this @tomasztunik would you agree that https://github.com/i18next/i18next-node and it's client-side version https://github.com/i18next/i18next are the most mature language libraries in the node ecosystem?
https://github.com/krakenjs/makara looks good but not sure how tightly coupled it is to dust template language

[tomasztunik]

i18next looks good, used it quite some time ago in one project and it was solid back then. I'm pretty sure it only matured by that time. I know Makara only from reading about it as part of kraken but never used it. Looking at the docs it seems to be built more as an integral part of kraken rather than standalone component.

If you don't mind me asking - are you planning to work on translations for both static and dynamic content? And if for static as well how did you envision that?

Was thinking myself a bit about it and not 100% sure what would be the best way I'd approach it - now I'm thinking about adding i18n dict field type to allow to create and edit multiple dictionaries in case for example you were building SPA app and didn't want to load whole dict with the app but rather have it per module/component to be lazy loaded using custom api endpoints. This kind of special field could also maybe make it possible to register a dictionary under given namespace for use in the server side templates... but yeah - just thinking out loud :)

[joaosamouco]

Do you have an ETA of when keystone will fully support i18n?
And how deep are you in the development of this feature?

We're looking into this but we do not wan't to dive into it if someone else is really close to achieve this and this feature could be released any time soon.

[morenoh149]

hasn't been started apart from preliminary discussions. The details outlined here keystonejs/keystone#744 are pretty spot on minus recent changes in the node ecosystem.
If you'd like to start building this out please share a link to your feature branch if you'd like guidance.

Otherwise I could work on this in late April.

[joaosamouco]

We are willing to dive into this but we're still not sure how to handle both static and dynamic content. I've been reading all these issues related to multilanguage support and you mostly discuss lib related stuff.

Some people said something kinda related to how to implement this with keystone but nothing to specific and I'm still not sure if any of you came to any conclusion about how this should be designed and implemented inside keystone.

I want to be sure that we follow the right approach and this fits perfectly with keystone.
Did any of you guys that have looked into this issue thought about how we will handle dynamic content and implement everything overall?

It's clear that static content inside the views will be tackled with the classic approach of wrapping that same content in a translation function (e.g. __). Though, dynamic content is way more tricky. Any thoughs?

@morenoh149 @sebmck @JedWatson

[morenoh149]

@joaosamouco it was fleshed out pretty well by @sebmck There's probably a way to detect the language requested by the end-user http://devdocs.io/express/index#req.acceptsLanguages and/or enable an easy way to specify all routes under a path should be a specific language e.g. foo.com/en/... and foo.com/es/... The latter is trickier.

[danielkhan]

Hello. I hope it's ok if I jump in right here.

Locale on path and detection:
Look at i18n-abide. They do it quite well. I'm using this on http://crowdlynx.com.
The important part is to remove the language url part after registering the language and before passing it down the routing chain.

Translation of statics:
Also i18n-abide - they use jsgettext for string replacement. It is easy to use this with translation tools like phraseapp then.

Content Translation:
Why not add a i18n attribute to the schema declaration?
If it is set there will be the field with subfields for every language.
Like:

headline { de: 'Heute ist ein schöner Tag', en: 'Today is a good day' }

I would also recommend adding a switch to explicitly add a translation to an entity because often not every content is available for every language.

Just my 2 cents.

[magalhas]

@danielkhan the i18n field is a good idea, but what happens afterwards? In some use cases you want to translate in the backoffice, in other use cases you want to provide a translation file to third party service.

[danielkhan]

@magalhas - usually translation offices can deal with CMS systems but anyways - if we take it to this level we are talking about workflows for translation, approval, etc. I won't take it that far.
Usually it's enough to use translation files for statics and and i18n-able database schema. This is how it works in other systems like SilverStripe or Typo3. They all offer some kind of mechanism to create language mutations and then offer a easy view that shows the master language along with some input element to insert the translation.

[woodyrew]

I've not had any experience with i18n but will most likely be doing a bilingual site soon. If it's in a state to test I'll happily help out with bug fixes, let me know a tag to checkout if so.

[tinganho]

Hmm, I'm maybe a little bit late in discussion. I'm the author of an i18n tool called l10ns. I'm also apart of the javascript-globalization group. I also happen to recently be a user of Keystone. I would be happy to help with any design around Keystone's i18n part.

I think it would be good if you choose a solution that's supports the standard ICU messageformat. It's a DSL for localization. And it support more complicated scenarios than i18next. ICU messageformat might also soon be a proposed as a native object to ECMAScript(there exist one strawman draft).

I think Keystone should probably separate localizing labels(label text, button text etc.) and localizing content(bread text, titles, header or everything that exists in the mongodb). I think Keystone's job is to localize content only and localizing labels should be done by an external tool. When we query an object we can pass in a locale to get different localized content.

I think it would be good if you take a look at https://github.com/SlexAxton/messageformat.js. Because it might suit your needs.

[joaosamouco]

@tinganho Nice contribution to this topic.
I took a look into your tool l10ns and it looks really good.
My only issue is: how hard is it for a non-technical translator to work with the tool interface and do his job? Did you ever hired some translator that had to work with your tool? How was it?

[tinganho]

No it is quite easy. Though for more complex translations they might ask a few question about the syntax.

[LorbusChris]

http://formatjs.io/ by Yahoo

[ericelliott]

I'm looking at starting a project which requires i18n. I'm very interested in formatjs. Just need to be sure that our translation service will play nice with the message format. Thanks @LorbusChris!

[ericelliott]

BTW, anybody interested in helping with this, I've been investigating options today. Will probably have a locale negotiation PR landing soon. I'd love it if some of you would like to review PRs or contribute to the effort.

If you don't mind me asking - are you planning to work on translations for both static and dynamic content? And if for static as well how did you envision that?

I plan to use translations for both static and dynamic content, but I'll probably want @JedWatson to be involved in planning dynamic translation support.

I agree we may want separate mechanisms for them. Dynamic content will probably require fewer features re: pluralization, grammar rules, etc, and instead just point to alternate db resources (a different text blob, different image URL, etc...).

Whereas static content, such as displaying UI labels, relative times, pagination numbers, etc... will require more thorough support, such as ICU messaging, etc...

We may also need to load alternate CSS / image resources, etc... depending on locales to cover things like right to left text display formatting, different length labels, and images containing text.

[ericelliott]

@tinganho The PR is ready for your review. If you can jump on it fast, I'd appreciate it. I want to pull this into Keystone ASAP so we can continue progress on this issue. =)

@LorbusChris Thanks for tracking down those resources. I'll check them out! =)

[ericelliott]

@tinganho has done a great job with language negotiation in express-request-language. I've been working with @tinganho to merge in support for query string language setting, which will make it easy to test language options on any page.

Now looking at integrating the library with Keystone. Here's what I've drafted for preferences. Comments/questions are welcome and encouraged:

language options: Object (optional)

The language options is an object containing your language support preferences. Defaults to all default values and en-US as the only supported language.

supported languages: Array

A list of supported languages. Preferred languages come first. Language tags must comply with BCP47 standard:

language[-script][-region]

1. language subtag (en, zh).
2. script subtag (optional, e.g. Hant, Latn).
3. region subtag (optional, US, CN).

Examples: en, en-US and zh-Hant-TW.

language cookie: String (optional)

Cookie name: Name of the language cookie. It will store the current language tag of the user's session and remain until maxAge expires or until it is changed by using the browser URL overrides.

language cookie options: Object (optional)

Options for the language cookie. This is passed through to express. See the Express cookie options for details.

language select url: String (optional)

An optional URL used to change language preferences. It will redirect back to the origin URL if you send a referrer header and default to / if it don't send a referrer header.

Usage:

/languages/{language}

E.g., visiting /languages/en-US will set your language preference to en-US.

language query name: String (default = 'language')

You can optionally set the language on any page without a redirect using a query string. This option allows you to set the name of the query parameter that triggers the language setting. The default value is language.

E.g., if you set language query name to 'locale', adding ?locale=zh-CN to any URL path will set the language to 'zh-CN'. Use the special language value, default to unset the language preference and fall back on the default setting as if you are a first-time visitor.

disable

Disable the language middleware.

[geloescht]

I am working on a project that needs content localization more badly than i18n of the admin interface or strings. I believe apart from locale negotiation, localization of content and strings/admin interface are separate issues. Since locale negotiation has already been tackled, I think we could discuss both now.
I created a wiki page with ideas on how content i18n could be implemented: https://github.com/keystonejs/keystone/wiki/Content-localization---Proposal
Please add your comments, ideas, pros and cons of different solutions etc. Regarding database structure I tend to solution 1.ii, but I am not very familiar with MongoDB.

(I would open a new issue for this purpose but saw that #1450 has been closed. Maybe a member of the keystone team can decide if this is the right place to discuss content localization, #1450 should be reopened or a new issue should be filed)

[tinganho]

I like the language-code keys 1.a.i solution. Though I think this should be fixed upstream. MongoDB should provide a localizable field solution.

[sbaechler]

@geloescht I suggest the solutions for content localization be provided in framework independent libraries. This way a developer can chose which approach fits his needs and the library itself has a larger maintainer base. Each framework would then just need a small adapter for each library.
Maybe the translatable configuration can be in an external file and be added to an existing model. This way it is possible to re-use models that have no translation fields. A good approach of this can be found in Django Modeltranslations for Python.
Your two DB structure proposals need the current active language value at different times within the lifecycle. If all languages are stored in one model, then the current language is needed during getting and setting the values. While if the solution uses child models for translated fields, then the current language is needed for the database query. I prefer that the language is passed to the database request explicitly.

[geloescht]

@sbaechler
I agree that making content localization an independent module would be desirable. However, I also think it is essential that keystone comes with a sensible default implementation (be it an adapter to an existing library or internal) and integration with the admin UI. Relying on application developers to integrate their translation library of choice every time they start a new keystone project would pose too big of a big barrier and lead to poor testing. If someone could point to a working, maintained content translation library for mongoose, please do. If we opt for totally pluggable content localization we would have to define an API for that purpose.
I am not sure what you mean by model reuse. Do you mean that translatable fields should not be declared in the same file as the main list registration? I am not quite sure why they shouldn't, considering that parameters for the admin UI, tracking etc. are also declared there. It is true that changing the translation options for a list later might pose problems with already existing documents, but that is true for all implementations, whether they use secondary files to define translatable fields or not. That should be addressed in the implementation.
I do not intent to supply a language each and every time a value in a document is accessed. Rather, the current language is saved alongside each document when queried and the correct translated fields should be saved automatically. I believe mongoose can provide that amount of flexibility without the need for additional wrapper objects. (If that is what you mean by child model. keystone has list inheritance, but that does something different)

[sbaechler]

@geloescht I agree that Keystone should have a default implementation for content localization. I just think it is advisable that there is a layer of abstraction between it and Keystone. Like there is between the template engine and Keystone. Imagine that in two years Mongo DB supported i18n out of the box. In that case you would just have to update the modeltranslation module when you updated your DBMS and not all of Keystone. It would be more work upfront but it made things easier down the road.

The dependency injection way of adding translations to a model is just a suggestion for an implementation approach. I think it works very well for Django applications. It's easy to add i18n to existing projects or 3rd party modules that don't have it.

[geloescht]

@sbaechler I see. I edited my proposal to allow for a translation service to be specified for each list separately as an alternative to a built-in service. This way we could provide a smooth upgrade path for switching translation services and even mixing of different translation schemata within the same application. Should the built-in translation service ever change in an incompatible way, a legacy service could be kept around.
Please feel free to add alternative approaches to the proposal on the wiki :)
https://github.com/keystonejs/keystone/wiki/Content-localization---Proposal

BTW: keystone already has a directory "schemaPlugins" with various mongoose middleware and functions. I imagine a content translation service to be one of those schema plugins. I even started on an implementation and it looks quite promising to do it that way. However those schema plugins are currently not truly pluggable.

[joernroeder]

I just found this issue during my research for an upcoming project which needs content localization as well. In this project every url should be translatable too (for SEO reasons) and i think that's a regular use-case.
As all fields can therefore differ i propose an additional _localisationGroupId key which is shared across all localisations and defines the relationship (grouping) between the documents. I would also prefer to create a new document for each localisation which contains a simple locale key.

{_id: 'fhdsjfs', _localisationGroupId: 'hgdfgjdskfh', locale: 'en', 'field1': 'foo' }
{_id: 'kdjhfgk', _localisationGroupId: 'hgdfgjdskfh', locale: 'de', 'field1': 'bar' }

// find other translations of the current article
context.list('Article').model.findOne({ _localisationGroupId: { $not: { article.locale }})

Shared fields (fields which are untranslated) can be updated in all localisations via:

// mongo query
collection.update({ _localisationGroupId: groupId }, { $set: {untranslatedField: 'foobar' }}, { multi: true });

I would also propose to group fields in the admin UI by locale (with tabsets for example) or link to other locales similar to relationships.

[geloescht]

@joernroeder Thanks for sharing your ideas. I've added your proposal to the Wiki. I think it might be a little harder to implement than structures where all localization data is stored in one document (something it has in common with other approaches). Do you have an idea on how to make your approach opaque via mongoose middleware?
BTW, Valeri Karpov of mongoose has accepted one of my patches for mongoose 4.5 that is necessary for my implementation to work, so hopefully I will put something out there soon :)

[joernroeder]

@geloescht i'll look into mongoose middleware. maybe i can come up with something helpful.
I know that it's harder to implement than using a single document but this seems to be more flexible to me and should (when you clone untranslated fields into every localisation) be faster to read than "joining" documents together.

[wmertens]

Now, there are several separate issues here:

• i18n of the admin UI, including form labels
• i18n of content. I prefer being able to say per field if it should get i18n or not, instead of using multiple document copies of the same thing in different languages.
• i18n of frontend views

I18n is a great bikeshedding opportunity. I propose that we implement only the minimum needed to move forward. For example, in Polish counting years goes "rok, 2 lata, 3 lata, 4 lata, 5 lat, 6 lat, …". You could implement something complex that handles all of these cases across all languages, or you could do "lat: x". If you don't worry about pluralization, you can already do a lot of good things easily, just by providing a string per language, maybe with interpolation markers.

@joernroeder we have had good success by making documents like {id: xxx, isFoo: true, someNumber: 7, title: {de: "Hallo", fr: "Bonjour", en: "Hi"}, …}. Documents are entities, and i18n is representation of some values inside the entities.

So I propose that we determine exactly what we want to to for each of the issues, create new issues and implement them.

[mdorda]

It's really a shame that Keystone can not display mongoose-intl in the Admin UI. That would be all I need :-)

[JonathanJJudge]

Hi everyone! I'm in a situation where a client really wants suburb to read as city for the location fields when managing through the Admin UI. Perhaps as part and parcel of this discussion we could add a simple configuration for overriding the subfield labels/placeholders on the Location field? I'd be happy to attempt putting together something and make a pull request for the latest release of 3, for more immediate use while 4.0 is in beta, and if that goes well, for 4.0 as well? It might be a sufficient stop-gap measure.

[seidtgeist]

I'm with @wmertens, it would help if there were separate issues for I18N admin, I18N content and I18N for frontend views. Each one is a big topic by itself.

My current requirement is I18N for content for which I'm going to add extra fields with locales in their field name. I found that this offers maximum control and is not that different from having something like document.field['en-US'], etc.

[kelkmere]

I want to ask a question to people who work at Thinkmill. What happens when one of your clients need a multilingual website?

[gggllm]

I am not sure if we are able to choose a language to render in adminUI in 4.0 beta, I am actually not that skilled with English. Can someone tell me something about it?

[jschutijzer]

What is the progress on this feature. I would like to use keystone but without multilingual feature I can't use it for customers.

[mroz22]

+1

[quaeched]

+1

[glimow]

Same here, also I'd be glad to help admin translation to french if this feature comes out !

[gautamsi]

see #4913 Keystone 4 is going in maintenance mode. No major changes expected.

There is discussion for this in v5 see keystonejs/keystone-5#322