Let's Encrypt on Kubernetes with cert-manager: webhook failure when creating ClusterIssuer
While following this Linode HOW-TO on setting up Let's Encrypt TLS on Linode Kubernetes Engine, I get the following error when I try to create a ClusterIssuer CRD:
$ kubectl create -f acme-issuer-staging.yaml Error from server (InternalError): error when creating "acme-issuer-staging.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: dial tcp 10.128.124.39:443: i/o timeout
Has anybody else encountered this?
In trying to recreate this issue, I was able to successfully create a ClusterIssuer CRD following the linked instructions - I didn't run into the same error as above.
First, if you haven't already, I would check that your cert-manager pods are running:
kubectl get pods --namespace cert-manager
The output should look similar to the below:
NAME READY STATUS RESTARTS AGE cert-manager-7747db9d88-htjnk 1/1 Running 0 30m cert-manager-cainjector-87c85c6ff-wkg68 1/1 Running 0 30m cert-manager-webhook-64dc9fff44-ggjhj 1/1 Running 0 30m
If that checks out, you may want to consider checking the logs for that cert-manager (replace "64dc9fff44-ggjhj" with your correct input), as this may help pinpoint the issue:
kubectl logs cert-manager-webhook-64dc9fff44-ggjhj -n cert-manager
It seems as though others have run into this issue, too - some were able to find resolutions, some not - it would largely depend on your configuration. Many have pinpointed firewall rules as the main problem, others have cited outdated versions of various applications, which could be worth looking into:
I'm having the same issue.
The webhook pod is never ready
NAME READY STATUS RESTARTS AGE cert-manager-75dbbd5d6-g5bnd 1/1 Running 0 2m15s cert-manager-cainjector-85c559fd6c-qs2ms 1/1 Running 3 2m15s cert-manager-webhook-6c77dfbdb8-546tq 0/1 Running 0 2m15s
and the logs of the pod are like so:
E1119 11:30:45.504379 1 dynamic_source.go:88] cert-manager/webhook "msg"="Failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"=1000000000
Also noticing that these other pods have issues (unsure if it's related):
kube-system pod/calico-kube-controllers-6c5ccf657c-g2dfn 0/1 CrashLoopBackOff 444 38h kube-system pod/csi-linode-controller-0 0/4 Init:0/1 330 38h kube-system pod/csi-linode-node-xkdxv 0/2 Init:CrashLoopBackOff 330 38h
If you also want to support DNS01 (e.g. to have wildcard certificates), you might want to have a look at our webhook adapter: https://github.com/monostream/cert-manager-linode
there is a similar project here: https://github.com/slicen/cert-manager-webhook-linode but has not worked in my tests using sub-domains.