got `lost connection to pod` with `kubectl port-forward`

I need to connect directly to the svcs in staging namespace in the development process.

So I use kubectl port-forward --namespace staging svc/xxxx 1234:1234
Then I found that it would prompt lost connection to pod every 1 minute or so. Time is not fixed. This is true even if the connection is not idle.

Now i can only use

while true; do kubectl port-forward --namespace staging svc/xxxx 1234:1234; done

I have actively investigated this matter. Some people say it is because of the configuration problem of streamingConnectionIdleTimeout. Then I checked this value, it is 4h0m0s.

So I think this should be a limitation of linode?
This affects my development process, because I always encounter errors like EOF.

9 Replies

Linode Staff

This is related to an issue on our side. Our Linode Kubernetes Engine team has a good understanding of why you're encountering this timeout limit, and they're working on a solution so that you don't see frequent timeouts moving forward. We'll make an announcement once this issue has been resolved, so keep an eye on our blog for updates.

Any update of this issue? It's extremely frustrating :(

Linode Staff

We're still looking into adjusting the timeout limit — the consideration is a bit involved, though I can attest to how frustrating it is to have kubectl port-forward connections dropped after a brief period of inactivity.

In the meantime, you can keep the connection alive by using netcat (nc) in a while loop instead. Here's the sequence that I ended up using (borrowing from @smile's example):

  • Run kubectl port-forward:
kubectl port-forward --namespace staging svc/xxxx 1234:1234
  • In a separate session, run nc to the port every 10 seconds:
while true ; do nc -vz 127.0.0.1 1234 ; sleep 10 ; done

The regular "pings" will keep the connection from timing out, so you should never see a "lost connection to pod" or related EOF error.

@bbigger
Thank you for your solution which helped me a lot.

I found another situation.
The solution may not be feasible for some services.
eg: postgresql

@smile Indeed, this approach may not work for every service. Although, for postgresql, the pg_isready utility can be used in the same way to keep a kubectl port-forward to a postgresql service from timing out. For example:

kubectl port-forward svc/postgresql 5432:5432 &
while true ; do pg_isready -d postgres -h 127.0.0.1 -p 5432 -U postgres ; sleep 10 ; done

(This will run the kubectl port-forward command in the background.)

@bbigger

You are simply amazing. Just solved my problem perfectly, and let me learn knowledge. thank you very much. respect!

@bbigger Is there any updates on this issue?

@man-from-geelong Yes! We've extended the timeout limit to 1 hour.

However, if you're still seeing lost connections before 1 hour, a timeout outside of Linode's systems (such as one caused by an intermediate VPN configuration) may be the cause.

If you suspect that lost connections are caused by an issue on our end, the best route would be to open a Support ticket so that we can investigate.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct