My server crashes at 12:01am on the first of every month. What is going on?

Linode Staff

On the first of every month, my Linode stops working. How can I find out what is going on?

11 Replies

A good place to take a look for something that is happening regularly is your /etc/crontab file. If you run the command sudo cat /etc/crontab you will want to look for something that runs on the first day of every month. It will look something like:

Run a command at the 0th minute of the 0th hour of the first day of every() month of every() year

0 0 1 * * /usr/bin/some_command

or

Run a command at the 1st minute of the 0th hour of the first day of every month of every year

1 0 1 * * /usr/bin/some_command

Whatever command is running at this time is probably directly or indirectly responsible for crashing your system. Please let us know what you find out.

I hope this helps!

Ah. Bingo.

I have a script that backs up my mysqls every midnight.

Thank you SO much for this.

Here is my output:

m h dom mon dow user command

17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

Do I need to delete the report /etc/cron.monthly? I just looked… there is only a placeholder file saying:

DO NOT EDIT OR REMOVE

This file is a simple placeholder to keep dpkg from removing this directory

Sounds like you don't have any monthly cron jobs.

I'd check systemd timers then

systemctl list-timers --all

It also isn't clear what you meant by "stops working":

  • Whole operating system shutting down?
  • Kernel crash?
  • Web server stopping?
  • Any other services?
    ….

If it's say the web server - you should be able to check web server logs, and so on…

Or if it's maybe this

script that backs up my mysqls every midnight

Then it's your script and you should be able to debug it (e.g. make it redirect its output to a file to see what goes wrong…)

Here is what I get when listing all monthly crons:

root@ubuntu:~# ls -la /etc/cron.monthly/
total 12
drwxr-xr-x 2 root root 4096 Mar 23 2016 .
drwxr-xr-x 85 root root 4096 Jul 18 2018 ..
-rw-r--r-- 1 root root 102 Feb 9 2013 .placeholder

And when I say "stops working"… my website would go offline - and my monitoring service would email me saying my site was down.

Maybe your backup script:

Has trouble creating a new month specific directory to store the backup?

Shuts down the site's database and fails to bring it back up because of that?

The script is the first thing I'd check.

Here is what I get when listing all monthly crons:

Right you already posted above that there are no monthly cron jobs. And if there were, they wouldn't run at 12:01 (but at 6:52).

I really appreciate your help on this. I am not very "techie"…

There should only be one backup script - daily at midnight. There should not be a weekly or monthly backup.

Is there any way I can turn off the monthly script?

There should only be one backup script - daily at midnight

There should not be a weekly or monthly backup

Well like I said - and I can only take guesses, because that's your system not mine - maybe this daily backup script which runs at midnight runs into some sort of trouble once (and every time) when the month changes?

Does this backup script:

  • Shut down the database or the web site before backing up (and expected to bring them back up when done)?
  • Use any "month" dependent directory (say, where it saves its output)?
  • Can it run into trouble creating that directory or maybe trying to use wrong directory - or something, anything, which could go wrong first time the month is different from before?

Those are the things I'd check…

I have no idea (and the person who set up these backups is no longer online).

Is there anyone I can PayPal to help me out with this?

I don't normally do this kind if thing but …

… I can try (just for fun, not money).

You can contact me at my nickname at gmail dot com.

You got to have offsite backup.

Backups should not impact your server performance.

Great news. I am almost 100% sure we fixed the problem.

I owe a BIG thank you to @kmansoft for helping me out. It looks like a bad cron call to Lets Encrypt took down my server every month at midnight.

I appreciate everyone who reached out to help me out.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct