✓ Solved

Use a specific nodebalancer id with ingress-nginx?

I want to select a pre-existing nodebalancer for ingress-nginx instead of creating one when I install the ingress-nginx chart.

The nodebalancer annotations reference for LKE indicates that I can do this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/linode-loadbalancer-nodebalancer-id: "301323"

…but the external IP is always <pending> for the ingress-nginx controller's LoadBalancer service. It's as if CCM isn't noticing the annotation ending "-nodebalancer-id", or there's some kind of mismatch between the configuration of the nodebalancer I created externally with Terraform and the requirements of the ingress-nginx controller service.

6 Replies

✓ Best Answer

I was able to work around this by discovering the public IP of the Nodebalancer through another resource.

data "kubernetes_service" "ingress_nginx_controller" {
  depends_on   = [helm_release.ingress_nginx]
  metadata {
    name = "ingress-nginx-controller"
    namespace = "ingress-nginx"
  }
}

And subsequently create the Linode domain record with this awkward reference.

resource "linode_domain_record" "wildcard_record" {
    domain_id = linode_domain.cluster_zone.id
    name = "*"
    record_type = "A"
    target = data.kubernetes_service.ingress_nginx_controller.status[0]["load_balancer"][0]["ingress"][0]["ip"]
    ttl_sec = var.wildcard_ttl_sec
}

The reason I need to pre-create the nodebalancer is so that I can discover the IPv4 in the Terraform plan so I can use it to also create a DNS record.

The data resource for nodebalancers requires the numeric ID of the nodebalancer to lookup the IPv4, so I can't discover the IPv4 by label or some other criteria after it's created by the ingress-nginx chart.

Now I see these errors in the LoadBalancer service's describe view.

  Type     Reason                    Age                   From                Message
  ----     ------                    ----                  ----                -------
  Normal   EnsuringLoadBalancer      9m (x9 over 24m)      service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed    9m (x9 over 24m)      service-controller  Error syncing load balancer: failed to ensure load balancer: [403] Unauthorized
  Warning  UpdateLoadBalancerFailed  4m20s (x13 over 24m)  service-controller  Error updating load balancer with new hosts map[lke92205-139651-63e7cc9f7da4:{} lke92205-139651-63e7cc9fa724:{}]: [403] Unauthorized

I also noticed that the LoadBalancer Service becomes stuck in this pending state. Uninstalling the ingress-nginx chart doesn't remove the Service, and a new chart release with the same name can't be installed while it's occupying the {release}-controller name in the same namespace.

I worked around this by getting the external IP from the Kubernetes Service resource after installing ingress-nginx and waiting for the LoadBalancer to be provisioned instead of pre-creating the Nodebalancer in the TF plan.

I am also facing an issue with:

service.beta.kubernetes.io/linode-loadbalancer-nodebalancer-id:

BTW, I am creating Nodebalancer via Traefik's service as defined in the helm chart.
However, upon it's creation I want to use it by referencing via annotation.

BTW, I am using the below script to get IP.

traefik-get-loadbalancer-ip.sh

!/bin/bash

take the json input and extract the CONTEXT variable

eval "$(jq -r '@sh "CONTEXT=(.context)"')"

CMD="kubectl -n traefik get services traefik -o jsonpath={.status.loadBalancer.ingress[0].ip} --context ${CONTEXT}"

keep checking until the loadbalancer IP (EXTERNAL-IP) is assigned to the traefik service.

NODEBALANCER_IP=$(${CMD})
until [[ ${NODEBALANCER_IP} != "" ]]
do
sleep 5
NODEBALANCER_IP=$(${CMD})
done

return json with the nodebalancer ip

jq -n --arg nodebalancer_ip ${NODEBALANCER_IP} '{nodebalancer_ip: $nodebalancer_ip}'

And calling it via:

// nodebalancer_ip output
data "external" "nodebalancer_ip" {
program = ["/bin/bash", "${path.module}/files/traefik-get-loadbalancer-ip.sh"]

query = {
context = "${var.kubectx}"
}

depends_on = [null_resource.traefik_install_wait]
}

Can someone from Linode support look into the issue with:
service.beta.kubernetes.io/linode-loadbalancer-nodebalancer-id:

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct