How do I use a private Docker registry in LKE with Proxy Protocol?

Hi,

I have changed my LKE config to work with Proxy Protocol following this guide. For external traffic this is working fine now, but I'm running into some problems trying to get our in-cluster private Docker registry to work.
Pushing images to the registry from an external connection is working fine, but creating pods fails as K8s is not able to pull images from the registry.

The docker registry is in namespace default. My services are in different namespaces.

I've tried various different configs (and set the --docker-server in the registry-credentials secret accordingly):

image: registry.mydomain.com/foo-bar:latest results in a 503 Bad Gateway

image: docker-registry.default.svc.cluster.local/foo-bar:latest
errors with ImagePullBackOff and
dial tcp: lookup docker-registry.default.svc.cluster.local: no such host

Using the Pod IP (which I rather not use directly) seems to get me somewhere but fails because TLS is offloaded in the NGINX Ingress controller.
image: 10.2.0.15:5000/foo-bar:latest

http: server gave HTTP response to HTTPS client

Any advice on how to get this set up properly is much appreciated.

Edit: I managed to work around this by setting up a dedicated NodeBalancer for our Docker registry without the Proxy Protocol. This is working fine now but feels a bit unnecessary.

2 Replies

Hey there -

This might help you out. We have a new annotation in for the CCM for "hostname-only-ingress" that you can add which should work:

https://github.com/linode/linode-cloud-controller-manager/blob/master/README.md

The full annotation would look like this:

service.beta.kubernetes.io/linode-loadbalancer-hostname-only-ingress: true

What this will do is prevent kube-proxy from circuiting the requests directly to the backend pods. When using TLS and it is terminating on the NodeBalancer, the requests will need to go to the NodeBalancer - and adding that annotation should get this working for you.

@rgerke

Hey there I need the annotation you mentioned, however I set up a Linode LKE cluster through rancher. The loadbalancer is automatically generated and every time I edit the yaml to add the annotation it does not save. Can I still add the annotation?

Edit: Nevermind, I see you can add an annotation by clicking edit on the nodebalancer through the UI and adding the annotation there. Directly editing the YAML does not work.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct