Django slower and slower, network issue ?
I'm facing performance issues since a few weeks and it became huge since 1 week.
I run an Ecommerce website with Django.
My set up is :
1 DB Linode 8GB with Postgres & Redis
1 Web Linode 8GB with Supervisor+Gunicor+Nginx+Django and Celery
1 Elastic Linode 8GB with elasticsearch
On June 13th I installed elasticsearch on the DB Linode and upgraded django from 1.11.5 to 2.0.6.
Everything was working fin exept on the linode manager the Web Linode stats was high on CPU. (between 80/100%)
Then it started to get a bit slower. My Web server was a 4GB plan at the time, so I decided to upgrade it to the 8GB on July 3rd.
It was better but still a bit slow.
Then it slowly became slower and slower, until next week when it was too much (15/20s to display cart, sometime 504 errors …)
I restarted everything (supervisor, nginx, postgress, elastic, redis …)
I purged every caches …
I took a look at my traffic, to see if a bot was crawling …
I rebooted every linode.
Still the same.
I took a look at my git history, if I pushed a big feature, but nothing, only minor fixes since june.
I tried to put elastic on his own Linode 8GB. But it done nothing. And it’s the web one that is slow anyway.
So I later on removed elasticsearch. It got a bit better for a few hours and now it’s still slower and slower.
I took screenshots of my WEB linode manager stats. (others seems to doing well)
We can see CPU evolution and a strang Net IPV6 behavior. I don’t know how to interpret it.
top - 12:07:06 up 6 days, 2:00, 2 users, load average: 1.14, 0.95, 0.68
Tasks: 274 total, 1 running, 119 sleeping, 0 stopped, 0 zombie
%Cpu(s): 9.2 us, 0.8 sy, 0.0 ni, 82.5 id, 0.1 wa, 0.3 hi, 1.0 si, 6.1 st
KiB Mem: 8058372 total, 3193624 used, 4864748 free, 93424 buffers
KiB Swap: 524284 total, 0 used, 524284 free. 1434932 cached Mem
I’ve no more idea to fix this. Can it be a network issue ?
I would suspect the Django upgrade first. any way to roll back the upgrade? (at the very least to the latest 1.11.x instead of 2.x)
You should also be able to run tools like htop and atop on your servers to hopefully narrow down what is taking up the CPU so you can narrow things down a bit more.
Did you review the release notes for every release between 1.11.5 and 2.0.6 for breaking changes and make sure you don't have anything in your code that runs into it? (same for the dependencies outside of core Django that your app uses.)
Are you using any python 2.x code? Python 2.x is no longer supported with Django 2.x onwards.
Just throwing some ideas out there.