Revert nginx config if faulty

[joaosa]

Nginx support hot config reloads. This enables configuration updates to be deployed quickly.

This is particularly relevant in case a bad config is pushed to the network.
To handle this, we could provide a hook for nodes to fetch a previous config in case the current one is faulty (i.e. Nginx fails to start).

While not a full rollback as described here, this could provide invaluable resilience in case something goes amiss. This feature goes around the versioning system. As such, it should only be used as a last resort to favor node stability.

[vorburger]

This is particularly relevant in case a bad config is pushed to the network.

By/through what - a new container image?

provide a hook for nodes to fetch a previous config in case the current one is faulty

Where would you want Nodes to fetch such previous configs from? Would the Orchestrator "push" Nginx config to Nodes? (Although that may be a "slippery slope", no?) If Nodes would do this automatically in update.sh, maybe; that adds complexity, best avoided? For Node operators to have to manually do this, that's probably hard, at scale?

Perhaps an alternative take on this could be to think about if version updates could be made faster? (E.g. with #125?)

Then a Nginx config "hotfix" is "just a new container image". Because why are Nginx configs more special than any other file in the container? Problems WILL happen. That's normal. But they will appear in various places, not just the Nginx config, no?

Nginx fails to start

Maybe there is a way to make it so that containers where Nginx fails to start never fully comes up, and a future better upgrade process can be made more "rolling" and smart enough to never let such a container even reach serving state? #124 could be a piece of this. Or even #126?

Just my 2 cents! I may well still be missing much of the bigger architectural picture invovled here, of course.

This feature goes around the versioning system. As such, it should only be used as a last resort to favor node stability.

+1. (Or perhaps this shouldn't be needed at all?)

[joaosa]

This is particularly relevant in case a bad config is pushed to the network.

By/through what - a new container image?

Yap 👍

provide a hook for nodes to fetch a previous config in case the current one is faulty

Where would you want Nodes to fetch such previous configs from? Would the Orchestrator "push" Nginx config to Nodes? (Although that may be a "slippery slope", no?) If Nodes would do this automatically in update.sh, maybe; that adds complexity, best avoided? For Node operators to have to manually do this, that's probably hard, at scale?

Perhaps an alternative take on this could be to think about if version updates could be made faster? (E.g. with #125?)

I agree that applying updates faster is the desired approach here.

Then a Nginx config "hotfix" is "just a new container image". Because why are Nginx configs more special than any other file in the container? Problems WILL happen. That's normal. But they will appear in various places, not just the Nginx config, no?

Nginx fails to start

Maybe there is a way to make it so that containers where Nginx fails to start never fully comes up, and a future better upgrade process can be made more "rolling" and smart enough to never let such a container even reach serving state? #124 could be a piece of this. Or even #126?

Yap good point!

Just my 2 cents! I may well still be missing much of the bigger architectural picture invovled here, of course.

This feature goes around the versioning system. As such, it should only be used as a last resort to favor node stability.

+1. (Or perhaps this shouldn't be needed at all?)

I agree that this is not exactly "desired" or necessary. I wanted to put the idea out there and discuss how applicable it would be. There are certainly other cleaner approaches.

[joaosa]

Closing this as there are better options