Support proxy canary in Envoy gateway

[nezdolik]

Description:
It would beneficial for users to be able to rollout proxy config change to canary service first prior to wide fleet rollout. EG could be managing multiple envoyproxy services (stable and canary) without resource duplications (users dont need to explicitly configure & duplicate resources for canary gateway class, gateway, envoyproxy config etc). Another idea is to have both canary EG (control plane) service talking to canary envoyproxies and stable EG+envoyproxies and deployment/rollout that progressively deploys to canary first and then to stable.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[arkodg]

1. Gradual rollout of xDS config into Envoy Proxy fleet when applying Gateway API config
2. Gradual rollout of Envoy Proxy fleet versions during a version upgrade
3. Gradual rollout of xDS config if/when Envoy Gateway version is updated (which also results in 2.)

@nezdolik are you referring to 3. ?

would also help highlight the motivation (e.g. unsure how new envoy proxy or envoy gateway will affect prod traffic) as well as any preferred workflow

[nezdolik]

@arkodg the use case is to be able to rollout config changes as well doing version upgrades to a small subset of canary pods first (so all of them? #1, #2, #3). I have experimented with canary strategy in argo rollouts and it is not compatible with xds config propagation (config propagated to canary and stable at the same time). The motivation is reliable/safe rollouts to production.