CoreDNS configured incorrectly in LKE beta?

I've been bashing my head on a number of issues trying out the LKE beta. It seems like DNS is once again the root cause… twice… once being my fault and the second time (the point of this question) looks like it may be a configuration issue in LKE

As an aside, it would be great if you could publish a guide on setting up LKE with an ingress (e.g. nginx-ingress or traefik), external-dns so that you can configure the external hostnames and certmanager with ACME via LetsEncrypt. This would seem to be a basic minimum that most people using LKE for hosting would want. (Also is there anything you can do to speed up your hosted DNS update… 30 minutes is a long time to wait to see a new hostname)

Anyway, after much diagnosis (and finally trying helm upgrade --namespace nginx nginx stable/nginx-ingress --set controller.publishService.enabled=true) I was able to get the Ingress external IP to be that of the LoadBalancer assigned to the nginx-ingress and that unblocked the biggest stumbling block I had with cert-manager… namely that it was failing the self-check…

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://...redacted.../.well-known/acme-challenge/bD...redacted...j0': Get http://...redacted.../.well-known/acme-challenge/bD...redacted...j0: dial tcp: lookup ...redacted... on 10.128.0.10:53: no such host
  State:       pending

Given that the external-dns had been pointing the hostname at the Nodes rather than the LoadBalancer, I had believed the issue to be DNS… (plus it's always DNS)

When I configured Nginx Ingress to publish the service end-point the DNS entries were changed to the correct IP… of course then I have to wait for Linode's DNS servers to update (30 minutes WAT!)

Same error!!!

So I say, let's go digging DNS:

apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

followed by a kubectl apply and then:

$ kubectl exec -ti dnsutils -- nslookup kubernetes.default
Server:        10.128.0.10
Address:    10.128.0.10#53

Name:    kubernetes.default.svc.cluster.local
Address: 10.128.0.1

That looks ok… This however…

$ kubectl exec -ti dnsutils -- nslookup ...redacted...
Server:        10.128.0.10
Address:    10.128.0.10#53

** server can't find ...redacted...: NXDOMAIN

command terminated with exit code 1

That's not great…

So I tried to brute force it and changes the CoreDNS configmap from

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap

To

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap

Restarted the coredns pods and w00t DNS was working… but this feels wrong.

Why was DNS not working with forward . /etc/resolv.conf?

(I have the history of all the setup commands I have run on the cluster as I was recording them as notes… but I'm not comfortable making those notes public)

LKE using Kubernetes 1.17