Postgres seems to keep getting shutdown after a while
I restart postgresql and then browse around in my app but then after a while it seems to shutdown and I get this message:
exception: "Illuminate\Database\QueryException" file: "/root/maketube/api/vendor/laravel/framework/src/Illuminate/Database/Connection.php" line: 664 message: "SQLSTATE  could not connect to server: Connection refused↵ Is the server running on host "localhost" (::1) and accepting↵ TCP/IP connections on port 5432?↵could not connect to server: Connection refused↵ Is the server running on host "localhost" (127.0.0.1) and accepting↵ TCP/IP connections on port 5432? (SQL: select * from "tags")"
I have it installed on my production server: psql (PostgreSQL) 11.2 (Ubuntu 11.2-1.pgdg18.04+1)
My setup is: Laravel as the backend and node as the frontend. With nginx used to route.
My linode instance is: Ubuntu 18.04 LTS, Nanode 1GB: 1 CPU, 25GB Storage, 1GB RAM
My postgres log does not say much more:
2019-04-12 22:20:55.678 UTC  LOG: database system is ready to accept connections 2019-04-13 10:11:27.727 UTC  LOG: listening on IPv6 address "::1", port 5433 2019-04-13 10:11:27.729 UTC  LOG: listening on IPv4 address "127.0.0.1", port 5433 2019-04-13 10:11:27.732 UTC  LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5433" 2019-04-13 10:11:27.788 UTC  LOG: database system was interrupted; last known up at 2019-04-12 22:20:55 UTC 2019-04-13 10:11:28.747 UTC  LOG: database system was not properly shut down; automatic recovery in progress 2019-04-13 10:11:28.784 UTC  LOG: redo starts at 0/165B6C0 2019-04-13 10:11:28.784 UTC  LOG: invalid record length at 0/165B6F8: wanted 24, got 0 2019-04-13 10:11:28.784 UTC  LOG: redo done at 0/165B6C0 2019-04-13 10:11:28.843 UTC  LOG: database system is ready to accept connections 2019-04-13 10:11:29.391 UTC  [unknown]@[unknown] LOG: incomplete startup packet
Anyone know whats happening?
The log you posted is for postgre starting up… and it complains that it wasn't previously shut down properly…
… but that doesn't tell us why it shut down (or was killed)?
Can you look for that in the logs?
Maybe something like this?
LOG: server process (PID 13904) was terminated by signal 9
LOG: terminating any other active server processes
in postgresql-11-main.log? There is literally nothing more. I delete that file and the restarted postgres and it shows the same messages and I posted initially. Is there any other log I dont know about?
I have a read stuff about OOM Kill but not sure how to check this
Maybe try systemd logs?
journalctl -f -u postgresql to watch output as it happens
journalctl --no-pager -u postgresql for history
As far as OOM goes - you can use e.g.
htop to watch memory usage (
htop is not installed by default,
apt-get install htop).
And if you do find a memory leak - it could be your web app leaking connections (a fairly common type of bug).
To check that you can use
SELECT usesysid, usename FROM pg_stat_activity and keep an eye on it over time if you can.
( pg command taken from here: https://stackoverflow.com/a/464641/2342806 )
To check for OOM kills that already happened - and I suppose there should be some, if that theory is right - you can do this:
grep -i killed /var/log/messages
Hmm so upgrading my linode instance to 2GB ram seems to work. lets see after some usage if the same issue still happens
Can you try
dmesg | grep -i killed
( my servers are Debian, close but not identical to Ubuntu )
That returned nothing, but I guess this is now after I upgraded my linode, so maybe it was a memory issue