Hours Delay "Linode Events Notification"

Hi there :) Anybody else noticed a recurring hours long delays with the automated "Linode Events Notification". Usually those are receive within seconds or minutes. How long is the delay to receive your notifications? Any proposal to resolve this challenge?

Steps to reproduce:

1. If not already done, enable “Events Email Notification” at https://manager.linode.com/profile/notifications

2. Reboot any Linode server

3. Wait 15 minutes. The Linode’s automated email notification is not receive from. This is the challenge. We first notice this challenge four weeks ago. This challenge can be reproduce with all our Linodes.

4. The expected result is that it’s receive between a few seconds and 5 minutes. This was working for the last few years.

Two concerns with this challenge:

• Security risk. Because in the unlikely event that a linode is compromised, hours is a very big window for unauthorized actions and abuses. Then for us to take appropriate actions.

• False alarms and waste budget. Because the whole point of receiving automated event alerts is to be alerted in a timely fashion. When we receive an email hours letter sometime trigger false alarms because we did not rebooted the linode. We did hours ago but not now not 5 minutes ago. So at first is feels like an unauthorized event. But after using budget investigating, it turn out to be a false alarm from Linode.

Below is a copy of the last challenge with the Linode’s email notification we’re referring to. We did a manual reboot on January 28, 2018 around 2am PT. Then we received this email notification January 28 at 12:15pm PT. Roughly 6 hours delay.
> Linode Events Notification - Linode Alerts <noreply@linode.com> Sun, Jan 28, 2018

To: Hello ! The following activity has recently occurred:

  • - () Lassie initiated boot: Profile - Completed Sun, 28 Jan 2018 10:04:58 GMT

  • * * You can change your notification settings via https://manager.linode.com/profile.

Related documentation at https://www.linode.com/docs/security/li … -controls/">https://www.linode.com/docs/security/linode-manager-security-controls/

What we tried that did not work:

• Whitelisted domains:

linode.com

mail[0-9].linode.com

www[0-9].linode.com

• Wait for a few email notifications. In case it’s a temporary workload challenge with Linode mail servers.

• Review the configuration of our email servers. It seems appropriate. Including whitelisted linode email addresses. We do frequently receive automated email notifications from other suppliers. No delay with those.

• Whitelisted this domain webserver1.linode.com (webserver[0-9].linode.com)

Linode's reply "This is something that we are aware of and are working toward a solution presently. Just a little insight into the unevenness; sometimes we have to facilitate the mailing manually, which obviously does not happen with the same precision as if the Linodes were facilitating the sending."

10 Replies

Thank you for the detailed report, we appreciate it. We're looking into this issue, both from the perspective of alleviating the problem in the short-term, and to make sure this doesn't come up going forward.

Thanks nmelehan :)

I would be happy to contribute testing if needed

I noticed the same problem, event notification emails being delayed for many hours.

For example, a reboot was initiated at 8:22 in the morning and the notification email came at 18:30.

My own status notifications via icinga2 come instantly.

I am also experiencing log delays. Linode should absolutely look into this.

I get my notifications delivered instantly (and boy are they annoying)

You should check if your mailserver's queue hasn't grown to silly sizes.

I have noticed Linode doesn't DKIM sign their mail. They should probably start doing that to maximise deliverability.

We received the last two Linode Events Notifications were received in seconds. So speaking for myself and the Ubertus team, this challenge is resolved.

@dubidubno :) +1 for adding DKIM to Linode Events Notification. I created a new ticket about this at https://forum.linode.com/viewtopic.php?f=11&t=15791

Sorry to bump a month-old thread, but I was doing some work on my Linode today and saw the same thing. At first I thought it might be that the mail wasn't going out until local time (CDT, GMT-7) matched the GMT time the event happened at, but the times do not line up quite that perfectly. In any case, it's taking in the neighborhood of 7-8 hours for the notifications to come through. Wondering if I should open a support ticket, but since this was already being discussed (albeit last month), thought I'd start here.

Thanks!

Terry

@drzow:

Sorry to bump a month-old thread, but I was doing some work on my Linode today and saw the same thing. At first I thought it might be that the mail wasn't going out until local time (CDT, GMT-7) matched the GMT time the event happened at, but the times do not line up quite that perfectly. In any case, it's taking in the neighborhood of 7-8 hours for the notifications to come through. Wondering if I should open a support ticket, but since this was already being discussed (albeit last month), thought I'd start here.

Thanks!

Terry

Hey Terry -

Thanks for reporting this, and no worries on bumping an old thread. It's still an issue, and we're still working on a permanent fix. Until that's in place, job event notifications are going to be a bit inconsistent.

  • Jim

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct