Cannot access site from kubernetes pod (SSL_ERROR_SYSCALL)

This error (SSL_ERROR_SYSCALL) suggests a problem with the SSL/TLS handshake. To troubleshoot, check if you're using the correct hostname/IP address and if there are any firewall rules blocking the traffic. I would also check the logs of the NGINX ingress controller for error messages and ensure the SSL certificate is valid and trusted by the client.

The first of the two commands will check your NGINX ingress controller logs while the second checks the SSL certificate.

kubectl logs -n <namespace> <nginx-ingress-controller-pod-name>

openssl s_client -connect <hostname>:443

Hope that helps!

--Eric

Thanks for the reply

Yes i am using the correct hostname and ip addresses, there is also no firewall rules blocking the traffic.

i tried the two commands you suggested.

This is the log i get from nginx ingress when i curl from another pod inside my cluster.

2023/04/18 14:57:46 [error] 176#176: *28779 broken header: "????????4?��KN@gjAc?)10l��?? m?$??`q?֝v?X�?1>??????,0̨̩̪+/$(k" while reading PROXY protocol, client: 10.2.2.1, server: 0.0.0.0:443

This is what i get when i do openssl s_client -connect hostname:443 from a pods inside the cluster

CONNECTED(00000005)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 316 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

.

HOW TO RECREATE

Here are the steps i took to get this error.

Firstly i installed nginx ingress

helm install my-nginx-ingress ingress-nginx/ingress-nginx -n nginx

secondly i activated proxy protocol on linode Gui for the nginx ingress controller.
Image of activated linode proxy protocol

Thirdly i added this configuration to the nginx configmap data field

apiVersion:v1
data:
   use-proxy-protocol: "true"

also added this configuration to the nginx service annotation

annotations:
   service.beta.kubernetes.io/linode-loadbalancer-enable-proxy- 
     protocol: "true"
   service.beta.kubernetes.io/linode-loadbalancer-proxy-protocol: v2

Finally i created a ssl certificate and a secret for the nginx ingress service

SSL CERTIFICATE

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout nginx-selfsigned.key -out nginx-selfsigned.crt

SECRET

kubectl create secret tls test-app-tls   --namespace test  --key nginx-selfsigned.key  --cert nginx-selfsigned.crt -o yaml

After these steps, i then created a nginx ingress service for the service i want to expose

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: store-ingress
  namespace: test
spec:
  ingressClassName: nginxingress
  rules:
  - host: xxx.xx.xxx.xxx
    http:
      paths:
      - backend:
          service:
            name: store-service
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
      - xxx.xx.xxx.xxx
    secretName: test-app-tls

RESULT

Now i am able to view the client ip when someone try to access the website from the public domain via the nginx ingress and i can also access the site from outside the cluster

FROM OUTSIDE CLUSTER

curl https://x.x.x.x -k

#-> HTTP 404 NOT FOUND

But when i try to access the site from any pod inside my cluster i get this error:

FROM INSIDE CLUSTER

image:

kubectl run -it --image debian sh

apt-update
apt upgrade
apt install curl -y

command:

curl -k https://x.x.x.x

#-> curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to x.x.x.x:443

Thank you for helping me as well.