firewalld doesn't work with default Linode kernel on Fedora 30

While it looks like there's a few workarounds for this, I'll also check to see if it's feasible to make a change to our Fedora 30 image so that firewalld works 'out of the box' with the Linode-supplied kernel.

In the meantime, you could try booting your Fedora 30 Linodes with Grub 2 to use the distro-supplied kernel instead. As Grub 2's the default for a new Fedora 30 Linode, you won't have to make any configuration changes to Grub on your Linode. Just change over to Grub 2 in your Linode's configuration profile and reboot it.

Presumably this is because Fedora 30 switched from iptables to native nftables?

@mjones I actually had to switch back to the Linode kernel after upgrading to Fedora 30 because of another issue (question 18148). It's supposed to be fixed now, but when I tried it again I got stuck on the GRUB screen as before.

Ran into this issue and also got stuck trying to boot the non-Linode kernel.

Is it possible to get either an update or links/details of some of the work-arounds that @mjones suggested?

@kmansoft I've been running into this mod probe install failure issue on Fedora since 25 or 26. I'm on 29 right now.

firewalld[9282]: ERROR: Failed to load nf_conntrack module: modprobe: ERROR: could not find module by name='nf_conntrack'
                                        modprobe: ERROR: could not insert 'nf_conntrack': Unknown symbol in module, or unknown parameter (see dmesg)
                                        modprobe: ERROR: Error running install command for nf_conntrack
                                        modprobe: ERROR: could not insert 'nf_conntrack': Operation not permitted
firewalld[9282]: ERROR: Raising SystemExit in run_server

I try moving /etc/modprobe.d/firewalld-sysctls.conf so it's ignored, but I guess firewalld wants it:

firewalld[9320]: ERROR: Failed to load nf_conntrack module: modprobe: FATAL: Module nf_conntrack not found in directory /lib/modules/5.1.2-x86_64-linode124
firewalld[9320]: ERROR: Raising SystemExit in run_server

I guess I could try to use a custom kernel, but haven't tried it. I'm not sure what else to try. Been living without FirewallD for quite a while now. I tried managing iptables myself, but stopped when I kept breaking my host by adding bad rules that blocked too many things.

Just to follow-up, this seems to be an issue with firewalld's reliance on modprobe responding in specific ways re: builtins when queried, which Linode's kernel changes by not providing /lib/modules/$(uname -r)/modules.builtin.

A fix is constructing this file by hand, which the firewalld GitHub issue linked above does address.

mkdir /lib/modules/$(uname -r)
touch /lib/modules/$(uname -r)/modules.{builtin,order}
for i in /sys/module/*; do echo kernel/${i##**/}.ko; done >> /lib/modules/$(uname -r)/modules.builtin
depmod -a

Running this allowed firewalld to start for me on Fedora 30, 5.1.16-300.fc30.x86_64.

It'd be nice if Linode was able to eventually fix this; it's a little weird to have to maintain module index files by hand.

Matthew's fix (above) also works on CentOS 7.7 with firewalld-0.6.3-2.el7_7.1.

The above workaround-- using the Linode kernel and manually creating files beneath /lib/modules/$(uname -r)/ did not work for me on CentOS 7.7-- lsmod (implausibly) claimed that there were no kernel modules loaded.

Instead, I used the below steps to switch over to the stock CentOS kernel, being booted by GRUB 2.

Not only does the below method work just fine, but it also should be more reliable: there is no manual intervention required when a new Linode kernel appears.

1. Modify the Linode's boot settings to boot from "GRUB 2" instead of "Latest 64 bit (…-linode…)"

2. Update all packages, including any stock-CentOS kernels:

yum clean expire-cache && yum -y update

1. Manually update the GRUB 2 config.

If the GRUB 2 config had not been getting updated while the Linode kernel was in use, it may not refer to any of the kernels currently installed, rendering the node unbootable:

grub2-mkconfig -o /boot/grub2/grub.cfg

1. Disable SELinux.

The Linode-standard kernel boots with SELinux disabled; even after switching to the stock CentOS kernel and relabelling filesystems, there were still SELinux denials causing broken functionality.

sed -i -r -e 's/^SELINUX=.*$/SELINUX=disabled/g' /etc/selinux/config

1. Power down the Linode, power it back on, and validate that it comes up running the stock CentOS kernel and functioning correctly.

I've just realised a whole group of my servers were running without the firewall too! Matthews Fix worked for me too, but I dread to think of how long my servers were running without the firewall.

I've opened a ticket with Linode.

Is there anyway to bump this with Linode @mjones

I have 22 linodes that were all running FirewallD

Hey @silverark,

We're glad to here that you found a fix! Phone support is available day and night for existing customers, account support, and general inquiries so please feel free to reach out if you are in need of urgent support or want to be sure your ticket was received:

U.S.: 855-454-6633
Intl.: +1-609-380-7100

Cloud firewalls that help make this easier to manage are on our roadmap, but we don't have an ETA for the feature becoming available currently. You can be the first to know when this is coming available by following along with our blog.

Thanks @Ttaylor, but the firewalls on the server are sufficient enough. Tha main issue is that you're custom kernal brought down the firewalls of 17 out of the 23 servers I have with you. I wonder how many people on Centos7 simply haven't noticed that their firewall is down. Lets hope none of those services are for whitelisted IP's only.

Here's the permanent fix, not only for Linode. Will work for other non-stock kernels as well: https://www.getpagespeed.com/server-setup/fix-firewalld-in-centos-7

I registered only to thank @Matthew for the provided solution.

Today i ran an update on 11 linodes only to find out their firewall was gone and they were open to public access, which for this particular client is not acceptable.

Why is it taking you so long to fix this, linode?

Thanks for letting us know, @twisted1919. I'm glad to see that the responses to this post were helpful for you.

I understand how frustrating this may be, but I want you to know that our team has been monitoring this thread, as well as looking into best ways to address this. I don't have an ETA on when that might happen at this time.

@watrick @mjones @Ttaylor

Please prioritize fixing this issue as it is causing the Linode network to be used for DDoS reflection attacks against other networks. Many Linode users have local services such as memcached running that listen on 0.0.0.0/0 and do not have authentication enabled by default.

Over the last few days, I discovered a CentOS 7.7 server on Linode that was running memcached and getting spikes of > 50GB of outbound IPv4 traffic sporadically every few hours, pushing CPU usage as high as > 90%. Firewalld was configured, which should've limited inbound traffic to web and SSH. Until the incompatiblity with Linode's custom kernel emerged, the server would use no more than 5 - 10GB of outbound bandwidth a month and CPU usage is typically < 30%.

Upon further inspection, like the other users in this thread, I noticed that the Firewalld service was dead. @Matthew's
solution of specifying the modules manually is not the complete solution. It would allow the Firewalld service to start after rebooting the server, but would die again soon thereafter - within a few minutes.

Only switching to a standard, non Linode provided kernel as @jtlinode did works. So far I haven't noticed any more reflection attacks on the server after the Firewalld (as originally configured) is back up.

In the meantime while you are working on a fix for the kernel, I suggest blocking UDP and TCP traffic on the public interface for port 11211 as DigitalOcean has done. https://www.digitalocean.com/community/tutorials/how-to-secure-memcached-by-reducing-exposure

Hello @jackieliu and everyone else!

I would like to follow up on this long-standing issue to announce that we have released a new guest kernel today (5.4.10) which should fix this issue for all users, provided that the "modules.dep" helper is also enabled (which is the default). This new kernel is not currently set as the "latest" kernel (we will update that in a few days), but for now it can be manually selected in the kernel list.

Please let us know if you are still experiencing the issue after updating to this new kernel.

Hello @lblaboon,

kernel 5.4.10 fix this issue on CentOS 7 too.

Thanks

Fixes the issue in CentOS 7. Thanks to @lblaboon and the Linode team for the efforts in resolving this. Pleased to switch back to a Linode-provided kernel to benefit from any provider specific performance optimizations.