Scaling down routes not scaling down memory usage

[arkodg]

          ### Metrics

Test Name	Envoy Gateway Memory (MiB)	Envoy Gateway CPU (s)	Envoy Proxy Memory (Avg) (MiB)	Envoy Proxy CPU (Avg) (s)
scale-up-httproutes-10	107.11	0.74	25.85	30.44
scale-up-httproutes-50	136.46	1.42	31.97	60.84
scale-up-httproutes-100	136.98	2.75	38.13	91.64
scale-up-httproutes-300	141.98	14.64	60.31	124.75
scale-up-httproutes-500	157.41	26.76	80.52	158.63
scale-up-httproutes-1000	179.77	57.89	130.77	197.64
scale-down-httproutes-500	164.67	85.83	130.91	233.84
scale-down-httproutes-300	152.14	96.66	131.07	266.26
scale-down-httproutes-100	146.73	106.85	127.11	298.33
scale-down-httproutes-50	161.01	107.92	127.04	329.02
scale-down-httproutes-10	202.45	108.52	126.36	359.48

Originally posted by @arkodg in #4287 (comment)

[arkodg]

is this an issue with the measurement or with EG ?
cc @shawnh2

[qicz]

imo. we should provide routes with different services or backend for benchmark.

nowly, just for one service, the EG metrics report show the minimal scene