POST requests fail with Gunicorn

I have a Flask app that I deployed to Linode that works great for about an hour and then after that the pages keep loading but any database interactions fail for user logins or other post requests (like editing profiles), while get requests work fine (like queries that return data for pages). Is there anything that would impact post requests like that in terms of configuration? I don't have any problems like this when I'm running locally. When I run sudo reboot on the server, it works again, but only for another hour or so.

When checking the error logs, Gunicorn does seem to be running into problems, but as far as I can tell from reading documentation, "signal 15" is pretty vague and makes diagnosis a challenge. I've included a snapshot of the logs below.

I'm running supervisor with the following config (note I'm just testing the single core, hence the three workers for gunicorn):

[program:sims-portal]
directory=/home/jonathan/flask_app
command=/home/jonathan/flask_app/venv/bin/gunicorn -w 3 run:app
user=jonathan
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stderr_logfile=/var/log/flaskapp/flaskapp.err.log
stdout_logfile=/var/log/flaskapp/flaskapp.out.log 

Any help would be appreciated! If it helps to see the code itself, it's stored here: https://github.com/JonathanGarro/SIMS-Portal-Neo

[2022-07-19 16:39:38 +0000] [733] [WARNING] Worker with pid 737 was terminated due to signal 15
[2022-07-19 16:39:38 +0000] [733] [WARNING] Worker with pid 738 was terminated due to signal 15
[2022-07-19 16:39:38 +0000] [733] [WARNING] Worker with pid 736 was terminated due to signal 15
[2022-07-19 16:39:38 +0000] [733] [INFO] Shutting down: Master
[2022-07-19 16:41:37 +0000] [734] [INFO] Starting gunicorn 20.1.0
[2022-07-19 16:41:37 +0000] [734] [INFO] Listening at: http://127.0.0.1:8000 (734)
[2022-07-19 16:41:37 +0000] [734] [INFO] Using worker: sync
[2022-07-19 16:41:37 +0000] [735] [INFO] Booting worker with pid: 735
[2022-07-19 16:41:37 +0000] [736] [INFO] Booting worker with pid: 736
[2022-07-19 16:41:38 +0000] [737] [INFO] Booting worker with pid: 737
[2022-07-19 21:50:12 +0000] [737] [INFO] Worker exiting (pid: 737)
[2022-07-19 21:50:12 +0000] [735] [INFO] Worker exiting (pid: 735)
[2022-07-19 21:50:12 +0000] [736] [INFO] Worker exiting (pid: 736)
[2022-07-19 21:50:12 +0000] [734] [INFO] Handling signal: term
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 735 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 736 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 737 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [INFO] Shutting down: Master
[2022-07-19 21:51:51 +0000] [833] [INFO] Starting gunicorn 20.1.0
[2022-07-19 21:51:51 +0000] [833] [INFO] Listening at: http://127.0.0.1:8000 (833)
[2022-07-19 21:51:51 +0000] [833] [INFO] Using worker: sync
[2022-07-19 21:51:51 +0000] [834] [INFO] Booting worker with pid: 834
[2022-07-19 21:51:51 +0000] [835] [INFO] Booting worker with pid: 835
[2022-07-19 21:51:51 +0000] [836] [INFO] Booting worker with pid: 836
[2022-07-20 20:20:43 +0000] [833] [INFO] Handling signal: term
[2022-07-20 20:20:43 +0000] [834] [INFO] Worker exiting (pid: 834)
[2022-07-20 20:20:43 +0000] [836] [INFO] Worker exiting (pid: 836)
[2022-07-20 20:20:43 +0000] [835] [INFO] Worker exiting (pid: 835)
[2022-07-20 20:20:43 +0000] [833] [WARNING] Worker with pid 835 was terminated due to signal 15
[2022-07-20 20:20:45 +0000] [833] [INFO] Shutting down: Master
[2022-07-20 20:20:47 +0000] [4706] [INFO] Starting gunicorn 20.1.0
[2022-07-20 20:20:47 +0000] [4706] [INFO] Listening at: http://127.0.0.1:8000 (4706)
[2022-07-20 20:20:47 +0000] [4706] [INFO] Using worker: sync
[2022-07-20 20:20:47 +0000] [4707] [INFO] Booting worker with pid: 4707
[2022-07-20 20:20:47 +0000] [4708] [INFO] Booting worker with pid: 4708
[2022-07-20 20:20:47 +0000] [4709] [INFO] Booting worker with pid: 4709

4 Replies

There's a high chance your Linode might be OOMing looking at the logs you provided. Have you considered resizing your Linode to a bigger plan?

Run dmesg to check for any OOMing errors on your Linode that may be killing your processes. A user from this post reported similar issues with Flask and it does have some good troubleshooting information that should be helpful.

[FIXED] Gunicorn worker terminated with signal 9

Why are Workers Silently Killed?

Thanks for the tips. I ran dmesg and got a pretty long return, but didn't find any alerts like in post your link to (nothing about being OOM). I did just resize the Linode but the problem persists.

I also want to know why POST requests fail with Gunicorn. I have tried so many time but facing the same problem. I was searching for this https://writinguniverse.com/free-essay-examples/to-kill-a-mockingbird/ website online and when I was searching for it online, I am glad I found your post.

I don't know anything about python, Flask or Gunicorn. I do know about signal-handling (generally) though…

When checking the error logs, Gunicorn does seem to be running into problems, but as far as I can tell from reading documentation, "signal 15" is pretty vague and makes diagnosis a challenge. I've included a snapshot of the logs below.

 

[2022-07-19 21:50:12 +0000] [734] [INFO] Handling signal: term
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 735 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 736 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 737 was terminated due to signal 15

The Gunicorn doc at https://docs.gunicorn.org/en/stable/signals.html describes SIGTERM processing (for both master & workers) as "graceful shutdown."

This would be what I would expect if you send SIGTERM (signal 15) to a master process with a bunch of children.

-- sw

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct