A suite of tools to assist kubernetes test-writing and investigate test coverage.
APISnoop is a collection of programs intended to run in a kubernetes cluster, that help with multiple aspects of conformance testing. These programs include:
- snoopdb
- a postgres database for running queries about the state of conformance coverage. This db starts up with audit events from a recent test run and the latest kubernetes api spec. This allows you to run queries to see exactly which endpoints in the spec are covered by exactly which tests. It can also be connected to auditlogger to create a more data-driven live-testing-environment.
- auditlogger
- a simple server that, when attached to a kubernetes cluster configured with an auditsink, will listen to all events in that cluster and insert them into the snoopdb’s testing tables. This allows you to perform mock tests against a cluster and then query the results immediately to ensure that your mock test is hitting the endpoints you intended.
We use snoopdb to generate the data for the site https://apisnoop.cncf.io.
We use APISnoop currently in a prow job to check if a kubernetes release includes any new, conformance endpoints that do not have conformance tests. the intention is to help reduce technical debt, by ensuring all conformant endpoints have conformant coverage.
The testgrid for the job can be found here: https://testgrid.k8s.io/sig-arch-conformance#apisnoop-conformance-gate
The job uses our snoopdb image. This image, on startup, runs through a set of sql initialization scripts including one that gathers any new, untested conformant endpoints. If the image detects it’s being run as part of prow, it will then do a simple check if its list of untested endpoints is greater than 0. If so, the job fails and we output the latest endpoints. Otherwise, it passes and the db shuts down, completing the job.
Snoopdb uses the following sources for determining tested conformant endpoints:
- The Kubernetes OpenAPI spec, to determine all endpoints
- The list of conformance tests at kubernetes/kubernetes/test/conformance/testdata/conformance.yaml
- The list of endpoints ineligible for conformance in the same directory as above
- The latest conformance suite test runs available via prow/testgrid (example).
We define “new conformance endpoints” as the endpoints in the most recent openAPI spec that are stable, not deprecated, and are not included in the ineligible-endpoints.yaml.
We define “tested” to mean the endpoint shows up in the audit logs of the conformance test suite as a target of a conformance test(e.g. a test that is included in the above conformance.yaml)
The logic for this work is present among the sql files in snoopdb’s initdb. Mainly:
- 505: script to output untested endpoints
- 207: new conformance endpoint view
- 204: eligible endpoint coverage view
To learn more about the database, you can check out its README: ./apps/snoopdb/README.md Or, for auditlogger:./apps/auditlogger/README.org