Outbound email through smtp.gmail.com stopped working Sep 23
Hi,
We have an old Rails app that has been working just fine for many (10) years.
We use Rails action mailer to send emails to customers to confirm their orders. This has been working just fine for years.
Things stopped working on Sep 23 2021. We now get a 30s timeout attempting to connect to smtp.gmail.com :
Timeout::Error: execution expired
from /usr/local/lib/ruby/1.9.1/net/smtp.rb:540:in `initialize'
from /usr/local/lib/ruby/1.9.1/net/smtp.rb:540:in `open'
from /usr/local/lib/ruby/1.9.1/net/smtp.rb:540:in `tcp_socket'
from /usr/local/lib/ruby/1.9.1/net/smtp.rb:549:in `block in do_start'
from /usr/local/lib/ruby/1.9.1/net/smtp.rb:549:in `do_start'
from /usr/local/lib/ruby/1.9.1/net/smtp.rb:519:in `start'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/mail-2.4.1/lib/mail/network/delivery_methods/smtp.rb:144:in `deliver!'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/mail-2.4.1/lib/mail/message.rb:2033:in `do_delivery'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/mail-2.4.1/lib/mail/message.rb:229:in `block in deliver'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.2/lib/action_mailer/base.rb:414:in `block in deliver_mail'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/notifications.rb:123:in `block in instrument'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/notifications.rb:123:in `instrument'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.2/lib/action_mailer/base.rb:412:in `deliver_mail'
from //somewhere-on-my-server/shared/bundle/ruby/1.9.1/gems/mail-2.4.1/lib/mail/message.rb:229:in `deliver'
We've seen that newly created linodes have various ports (including 587) disable by default, but our linodes have been running pretty much untouched for years.
Have also seen that there's possibly an issue with gmail having issues with our linode's SLAAC ipV6 addresses but I don't understand what I can do to change this.
Simply trying to telnet to smtp.gmail.com from our linode goes into suspended animation for ages trying with ipV6 address then gives up and tries an ipV4 address and connects immediately.
$ telnet smtp.gmail.com 587
Trying 2a00:1450:400c:c09::6c..
<< Takes a few minutes before coming back with >>
Trying 74.125.23.108...
Connected to smtp.gmail.com.
Escape character is '^]'.
220 smtp.gmail.com ESMTP k9sm18021890pfi.86 - gsmtp
Any help or areas to investigate would really be appreciated.
Many thanks
Graham
7 Replies
My guess is that Google changed one of their policies about using smtp.gmail.com with a non-Google client. They're masters of their own domain so they can do that. Whether or not they tell you is not really important to them.
My guess is that they did tell you but it was buried in 4-point type at the bottom of some other content-free notice so you missed it…by design.
Your theory about IPv6 is plausible. It could also be your positively ancient Ruby client (try upgrading your Ruby on a test machine to 2.7.4 or 3.0.2 and see if that changes things … 1.9.1 has been EOL for a very long time!).
The real reason they probably did this is because we can. Google and customer service are not exactly synonymous…
-- sw
Hi chinthe,
we have the same situation and facing the same problem. Did you find any solution? Even, I have tried to change to Yandex email server (not gmail) but ping smtp.yandex.ru" command not executed. It takes about 1 minute and "Internet Server Error" Page appears on the browser.
Please share if you find any solution.
Thanks.
Even, I have tried to change to Yandex email server (not gmail) but ping smtp.yandex.ru command not executed.
This could be a local firewall issue. Do you know if you block outbound ICMP echo requests (aka ping). It works fine for me on my Linode in Fremont, CA for both IPv4 and IPv6. Ditto for my home network in Oregon (using a Seattle, WA ISP).
Your corporate firewall may be blocking Yandex's ASN because it has such a horrible reputation (see below).
It takes about 1 minute and "Internet Server Error" Page appears on the browser.
Of course you're going to get an Internet Server Error from a web browser! smtp.yandex.ru is not a web server. Try http://yandex.ru …you'll have better luck. I hope you can read Russian.
yandex.ru is widely viewed in the US as a safe haven for spammers and phishers. Like most large Russian companies, it has ties to the Russian government and the Russian State Security services. For those reasons, many ISPs, edge networks and IP/email reputation services routinely block or blacklist them.
I block ALL email showing up at my mail server that has been routed through ANY yandex network…no matter what the originating domain/IP is. yandex is just not trustworthy enough for me. Flames to /dev/null…
Regarding yours and @chinthe's specific problem(s) with Gmail… Gmail requires use of ports 465/587, OAuth authentication and TLS for email submission. Any app that was written 10+ years ago is probably using port 25 (not for submission, no TLS and no authentication) to try to submit mail. Gmail specifically disallows this.
Also, Gmail requires that your domain conform to specific email authentication/security standards -- SPF and DMARC in particular (and DKIM will probably be required soon). Most of this didn't exist 10 years ago. If your apps are really 10 years old, it's time to replace them with something more modern. You're not going to get Google to budge on this. The Sept 23, 2021 date is probably the date they started enforcing all this stuff on your account. As I said before, they probably told you but the notice was buried in some other inconsequential notice that you ignored.
You can thank your friendly neighborhood spammer/phisher and a zillion script kiddies worldwide for this state of affairs.
Shoes for Industry!
-- sw
Gmail and Outlook have caused me endless problems with silently undelivered email and other completely non-debuggable and non-compliant behaviour. This began to cause us real-world problems, so we had to make it company policy not to use them for anything where a realistic alternative exists.
We're now investigating using Mailgun as an SMTP provider. So far things look promising; they have a really good interface; they make sure you authenticate and have a valid payment method on file, and get you to add a DKIM security key to your domain record to ensure you are really the domain owner. This means they know and can trust who you are, and thus have few spammers. As a result their outgoing IPs should have a good reputation. They also provide a straightforward API which simplifies things considerably, since you can take sendmail out of the loop if you want to. (In case you're wondering, it's also extremely cheap.)
My only reservation with Mailgun is that it is aimed at high mail volumes, and it's unclear whether sender reputation can be established at the relatively low volumes I'll be dealing with (order confirmations, user account creation, and so on).
Sendgrid is another option, which I plan to look at if Mailgun doesn't pan out. It may be some time before I figure that out, but I'll try to come back and leave an update if anything changes.
Gmail and Outlook have caused me endless problems with silently undelivered email and other completely non-debuggable and non-compliant behaviour.
Gmail and Outlook are run by for-profit companies. The free Gmail/Outlook services are offered as a marketing ploy to drive customers to their pay-for platforms. Both have some (state secret) criteria for determining when a particular user/account has ceased to be a "casual" or "individual" user eligible for free service and become a "commercial" user who is not.
Unfortunately, once you (unknowingly) cross that line, you can't ever go back and the consequences are what you describe. You is doing the right thing. @chinthe and @exist should probably follow suit…
Although, even with a new platform, you're probably not going to be able to do email submission on port 25. So, some app modification is probably going to be in order as well.
-- sw
Gmail and Outlook have caused me endless problems with silently undelivered email and other completely non-debuggable and non-compliant behaviour.
Gmail and Outlook are run by for-profit companies. The free Gmail/Outlook services are offered as a marketing ploy to drive customers to their pay-for platforms. Both have some (state secret) criteria for determining when a particular user/account has ceased to be a "casual" or "individual" user eligible for free service and become a "commercial" user who is not.
Unfortunately, once you (unknowingly) cross that boundary, you can't ever go back and the consequences are what you describe. You are doing the right thing. @chinthe and @exist should probably follow suit…
Although, even with a new platform, you're probably not going to be able to do email submission on port 25. So, some app modification is probably going to be in order as well.
-- sw
My only reservation with Mailgun is that it is aimed at high mail volumes, and it's unclear whether sender reputation can be established at the relatively low volumes I'll be dealing with (order confirmations, user account creation, and so on).
I can vouch for Postmark - they specialise in transactional-only e-mail like you suggested, and personally verify all accounts to keep spam as low as possible. Their delivery rates are much higher than Mailgun, who have very little in the way of verification.
If you do decide to sign-up, contact me through my website and I can refer you through their referral scheme, and we both get a free month :)