Restart of snap.k8s.kube-apiserver.service causes deployment of pods to fail

[amc94]

Summary

After kube-apiservice restarts, cluster becomes unable to deploy pods, with all newly created pods hitting an issue of connect: operation not permitted when trying to talk to api-server
I suspect that's what happened with ##592

note: it also causes issues with sudo snap remove k8s, where the networking of the machine seems to be impacted

What Should Happen Instead?

pods should still be able to deploy, I should still be able to ssh to the machine without having to restart it first

Reproduction Steps

sudo snap install k8s --classic --edge
sudo k8s bootstrap
sudo k8s status --wait-ready
sudo systemctl restart snap.k8s.kube-apiserver.service
kubectl rollout restart deployment <deployment> -n kube-system

System information

inspection-report-20240830_180046.tar.gz
k8s-syslog.log

Can you suggest a fix?

No response

Are you interested in contributing with a fix?

No response

[mateoflorido]

Hello @amc94 ,
We are aware of this issue and are currently working on a fix. As commented in #592 , here are a couple of workarounds we've tested to temporarily fix the issue:

• Run the following command:

  /opt/cni/bin/cilium-dbg cleanup --all-state --force

• Restart the affected node.