[Resolved] CentOS 7: NetworkManager Connection Restart

I'm not seeing this question asked any where else that I've looked. Long story short, NetworkManager brings down eth0 1 minute and 1 second after every hour for the reason of 'connection-removed'. Most of the time, the connection comes back up–although it obviously disconnects everything--within that same second. In some few and far between cases, it does not recover and despite being set with a static IPv4 address, will not recover until the network service is restarted.

This is a recently installed stock, minimal install of CentOS 7 (trying it out for the first time since CentOS 4) with only one additional application installed: Cockpit.

Here are the logs of that second:

journalctl

http://pastebin.com/sAGEEkrx

/var/log/cron

Feb  3 08:01:01 hyaena CROND[1094]: (root) CMD (run-parts /etc/cron.hourly)
Feb  3 08:01:01 hyaena run-parts(/etc/cron.hourly)[1094]: starting 0anacron
Feb  3 08:01:01 hyaena run-parts(/etc/cron.hourly)[1103]: finished 0anacron
Feb  3 08:01:01 hyaena run-parts(/etc/cron.hourly)[1094]: starting docker-cleanup
Feb  3 08:01:01 hyaena run-parts(/etc/cron.hourly)[1111]: finished docker-cleanup

/etc/cron.hourly

/0anacron

#!/bin/sh
# Check whether 0anacron was run today already
if test -r /var/spool/anacron/cron.daily; then
    day=`cat /var/spool/anacron/cron.daily`
fi
if [ `date +%Y%m%d` = "$day" ]; then
    exit 0;
fi

# Do not run jobs when on battery power
if test -x /usr/bin/on_ac_power; then
    /usr/bin/on_ac_power >/dev/null 2>&1
    if test $? -eq 1; then
    exit 0
    fi
fi
/usr/sbin/anacron -s

/docker-cleanup

#!/bin/bash

# Do nothing if neither docker nor docker-latest service is running
if ! systemctl --quiet is-active docker-latest && ! systemctl --quiet is-active docker; then
  exit 0
fi

# If there are no dead containers, exit.
DEAD_CONTAINERS=`docker ps -aq -f status=dead`

[ -z "$DEAD_CONTAINERS" ] && exit 0

# Try to cleanup dead containers
docker rm $DEAD_CONTAINERS

7 Replies

Primarily to see if it could be done/learning more about KVM abilities. I've not had the need to do anything other than essentially click Play on a virtual machine and automagically have a machine. Essentially, I wanted to see if it could be done. auditd, SELinux, and then an unknown (to me) number of changes made to the Linode kernel versus Cent/RH, I wanted to stick with the "RedHat" way as closely as possible.

You must have messed something up, because Linode's images for CentOS do not use NetworkManager.

The "Network helper" feature is enabled by default and should generate the proper ifcfg-eth0 file, which does not use NetworkManager, so most people remove it with:

systemctl stop NetworkManager wpa_supplicant
systemctl disable NetworkManager wpa_supplicant
yum remove NetworkManager wpa_supplicant

I appreciate your candid response.

You are correct, images provided by Linode do not use NetworkManager. I am running stock CentOS installed on Linode via CentOS 7 installation material, therefore it does include NetworkManager and more importantly, the things I was looking for when choosing CentOS in the first place, namely SELinux implemented in a way I can actually live with without pulling out my hair.

Having run into this problem, I wanted to fix it rather than patch it by removing it–which I'll likely do anyway. I believe I have resolved it (surprisingly nearly the moment you posted). I'm still waiting for one last test. If it succeeds, I'll be able to notify developers of the precise scenario and to others experiencing a similar problem on how to fix it while maintaining recommended and supported configurations by upstream.

I went with CentOS 7 specifically due to SELinux, but there was no need to use a custom installation method, because its easy to just boot the vanilla stock kernel for SELinux support (by using GRUB2).

I'm not sure NetworkManager is useful on a server environment (without wifi), its a waste of resources having it running and memory-resident. So I always encourage others to remove it, unless you know what you need it (eg, when you need wpa_supplicant, that sort of thing).

Other than SELinux, could you please explain why you needed to perform your own installation? just out of curiosity.

This can be marked as solved.

I wiped out the ifcfg files created by the CentOS installation media and used nmcli to create them from scratch. The old an new ifcfg files ended up being almost exactly the same. The differences are the order of the content, a new UID automatically generated, and I gave the connection a custom name of ext0 rather than being generated. And now it works perfectly fine.

So, in the end, who the hell knows. The flip desk emoticon would be useful here.

From what I've found, NetworkManager fully separates the device and the connection. In my case, every hour for some reason NetworkManager determined that eth0 was not configured despite it being so and would generate a temporary new connection which would then fail. Therefore, dropping the current connection entirely, failing the temporary one, and repeat.

Perhaps either of those two items are references elsewhere that I'm not aware of, therefore informing NetworkManager it is indeed in use and controlled by NetworkManager (NM_CONTROLLED=yes did nothing, for the record).

Alright, fully tested, confirmed, working. Still without a true understanding the difference between the two produced files. This is documented in a few places as bugs that should have since been fixed.

In the end, this is my ifcfg-ext0:

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=yes
NAME=ext0 # Arbitrary name given
UUID= <uuid generated="" by="" networkmanager="">DEVICE=eth0
ONBOOT=yes
ZONE=drop
DNS1= <ipv4 dns="" server="">DNS2= <ipv4 dns="" server="">IPADDR= <static ipv4address="">PREFIX=24
GATEWAY= <static ipv4="" gateway="">DOMAIN= <my domain="">DNS3= <ipv6 dns="" server="">DNS4= <ipv6 dns="" server="">IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no</ipv6></ipv6></my></static></static></ipv4></ipv4></uuid>

FAILUREFATAL=yes is not default as I purposely want the connection to fail if it does not have both an IPv4 and 6 address.

Nice to know that you fixed the problem!

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct