My server crashes at 12:01am on the first of every month. What is going on?
On the first of every month, my Linode stops working. How can I find out what is going on?
11 Replies
A good place to take a look for something that is happening regularly is your /etc/crontab file. If you run the command sudo cat /etc/crontab
you will want to look for something that runs on the first day of every month. It will look something like:
Run a command at the 0th minute of the 0th hour of the first day of every() month of every() year
0 0 1 * * /usr/bin/some_command
or
Run a command at the 1st minute of the 0th hour of the first day of every month of every year
1 0 1 * * /usr/bin/some_command
Whatever command is running at this time is probably directly or indirectly responsible for crashing your system. Please let us know what you find out.
I hope this helps!
Ah. Bingo.
I have a script that backs up my mysqls every midnight.
Thank you SO much for this.
Here is my output:
m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
Do I need to delete the report /etc/cron.monthly? I just looked… there is only a placeholder file saying:
DO NOT EDIT OR REMOVE
This file is a simple placeholder to keep dpkg from removing this directory
Sounds like you don't have any monthly cron
jobs.
I'd check systemd timers then
systemctl list-timers --all
It also isn't clear what you meant by "stops working":
- Whole operating system shutting down?
- Kernel crash?
- Web server stopping?
- Any other services?
….
If it's say the web server - you should be able to check web server logs, and so on…
Or if it's maybe this
script that backs up my mysqls every midnight
Then it's your script and you should be able to debug it (e.g. make it redirect its output to a file to see what goes wrong…)
Here is what I get when listing all monthly crons:
root@ubuntu:~# ls -la /etc/cron.monthly/
total 12
drwxr-xr-x 2 root root 4096 Mar 23 2016 .
drwxr-xr-x 85 root root 4096 Jul 18 2018 ..
-rw-r--r-- 1 root root 102 Feb 9 2013 .placeholder
And when I say "stops working"… my website would go offline - and my monitoring service would email me saying my site was down.
Maybe your backup script:
Has trouble creating a new month specific directory to store the backup?
Shuts down the site's database and fails to bring it back up because of that?
The script is the first thing I'd check.
Here is what I get when listing all monthly crons:
Right you already posted above that there are no monthly cron jobs. And if there were, they wouldn't run at 12:01 (but at 6:52).
I really appreciate your help on this. I am not very "techie"…
There should only be one backup script - daily at midnight. There should not be a weekly or monthly backup.
Is there any way I can turn off the monthly script?
There should only be one backup script - daily at midnight
There should not be a weekly or monthly backup
Well like I said - and I can only take guesses, because that's your system not mine - maybe this daily backup script which runs at midnight runs into some sort of trouble once (and every time) when the month changes?
Does this backup script:
- Shut down the database or the web site before backing up (and expected to bring them back up when done)?
- Use any "month" dependent directory (say, where it saves its output)?
- Can it run into trouble creating that directory or maybe trying to use wrong directory - or something, anything, which could go wrong first time the month is different from before?
Those are the things I'd check…
I have no idea (and the person who set up these backups is no longer online).
Is there anyone I can PayPal to help me out with this?
I don't normally do this kind if thing but …
… I can try (just for fun, not money).
You can contact me at my nickname at gmail dot com.
You got to have offsite backup.
Backups should not impact your server performance.