Inconsistent TLS Verify behavior between Nginx and Kong Service level TLS configuration

[Vikash08Mishra]

Kong version

3.7.0

Current Behavior
To Configure TLS certs verification we have a global level config at Nginx Level, additionally we have Kong service level configuration where we can provide ca certificates and use tls_verify configuration. Global setting at Nginx level can be configured using below environment variable:
KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE
KONG_NGINX_PROXY_PROXY_SSL_VERIFY
whereas service level tls_verify setting can be configured either by UI or API and it's definition says "Whether to enable verification of upstream server TLS certificate. If set to null, then the Nginx default is respected."

Below is a summary of scenario's I tried along with visible behavior

1. Scenario-1

For a given service-1:
KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE : added ca cert for service-1 to trust store path.
KONG_NGINX_PROXY_PROXY_SSL_VERIFY: "on"
Service level TLS Verify, i.e. tls_verify: 2 sub scenario's
- a) true
- b) unset which falls back to "use default system settings"
response to my service-1 endpoints: success
As per expectation: Yes

2. Scenario-2

For same given service-1:
KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE : added ca cert for service-1 to trust store path.
KONG_NGINX_PROXY_PROXY_SSL_VERIFY: "off"
Service level TLS Verify, i.e. tls_verify: true
response to my service-1 endpoints: Fails with server verify error mentioning no local issuer certificate found.
As per expectation: maybe
derivation from scenario-2 behavior: Even if service level tls_verify is true, it doesn't look for certificate in PROXY_SSL_TRUSTED_CERTIFICATE store path if KONG_NGINX_PROXY_PROXY_SSL_VERIFY is off. Or KONG_NGINX_PROXY_PROXY_SSL_VERIFY must be on to use KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE.

I went ahead with Scenario-1 configuration which works for TLS verification for my service-1. Then, I had another service-2 registered in my gateway for which I don't want to verify server cert.
3. Scenario-3
For a given service-2:
KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE : service-2 ca_cert not added to trust store path as I don't want to verify service-2 server certificate. It still contains service-1 ca_cert as expected.
KONG_NGINX_PROXY_PROXY_SSL_VERIFY: "on" (I need this for service-1 as concluded from scenario-1 && 2).
Service-2 TLS Verify, i.e. tls_verify: "false"

response to my service-2 endpoints: Fails with server verify error mentioning no local issuer certificate found.
2024/09/10 12:51:12 [error] 1335#0: *95829721 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream,
As per expectation: No because tls_verify is explicitly set to "false" for service-2 and as per definition of service level tls_verify it should not even look to verify but seems it does fall back to KONG_NGINX_PROXY_PROXY_SSL_VERIFY which is "on".

Expected Behavior
When a service level TLS Verify, i.e. tls_verify is set explicitly to "false" then it should not try to look for server certificate and should skip server cert verification. It should fall back to Nginx default only when it's it's unset/null.

Steps To Reproduce
deploy Kong gateway(OSS) 3.7.0.
Have 2 backend service supporting https calls.
Add ca cert for only 1 of above service to KONG_NGINX_PROXY_PROXY_SSL_TRUSTED_CERTIFICATE
Follow scenario's details provided above to verify behavior.