no longer able to use SSH key-pair login
Considering making new authentication key-pair:
This afternoon. Cannot SSH from the Comcast address. At the same time email is working.
Ran sudo fail2ban-client status. Check blocked IP addresses in SSHD and SSHD-DDOS jails. Comcast address not there.
sudo cat /var/log/secure | grep Comcast IP address
no hits today
sudo cat /var/log/maillog | grep Comcast IP address
no hits today
sudo cat /var/log/dovecot.log | grep Comcast IP address
looking fine
sudo cat /var/log/sogo/sogo.log | grep Comcast IP address
no hits today
CentOS 7 email server had been fine for months.
Issues:
June 29th had Postfix (iRedMail) block an IP address from a customer. I was on-site . Used LISH access to remove the iptables blocks on INPUT and FORWARD each time.
June 30th:
4 AM yum-cron updated the kernel.
6:40 AM Emergency Maintenance. Linode restarted the VM after host issue.
July 1st to 3rd. That customer’s IP address and my residential Comcast address have been periodically blocked by the firewall.
Added addresses to IP address whitelist for OSSEC and Fail2Ban. Most has fixed the problems.
OSSEC is host-only install. Have email alerts active at level 7.
Have LogWatch installed.
Saw such as:
sudo iptables -L INPUT -v -n | grep "IP address"
506 30304 DROP all -- * * IP address 0.0.0.0/0
sudo cat /var/ossec/logs/active-responses.log| grep IP address
Sun Jul 2 17:42:48 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete – IP address 1499014544.273703 604
Sun Jul 2 17:42:48 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete – IP address 1499014544.273703 604
Sun Jul 2 17:42:51 EDT 2017 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete – Comcast IP address 1499014544.273703 604
No more hits since.
Have not seen what action caused the blocks in the first place days ago.
Searched:
/var/log/secure
/var/log/maillog
/var/log/messages
/var/ossec/logs/active-responses.log
Rebooted VM before posting. No change.
11 Replies
Check the following files in CentOS: /etc/sysconfig/iptables-config & /etc/sysconfig/iptables6-config, especially the options:
IPTABLESSAVEON_STOP="no"
IPTABLESSAVEON_RESTART="no"
Based on the above messages, it seems like one or more scripts, wrongly parsed the logs and generated/run a broken iptables rule. My suggestion, is to disable ossec and fail2ban, then clear all the firewalld/iptables rules, then you can check your connectivity status.
Changed the two options in /etc/sysconfig/iptables-config and /etc/sysconfig/ip6tables-config.
From no to yes.
Nothing in iptables blocking the Comcast address. At same time cannot SSH. Keeps timing out.
Email works.
Change to no to disable tunnelled clear text passwords
PasswordAuthentication no to yes
sudo systemctl restart sshd
[Tue Jul 04 10:28:47 :~ ] $ ssh Linode IPv4 address -l user
ssh: connect to host Linode IPv4 address 22: Connection timed out
[Tue Jul 04 10:31:26 :~ ] $ ssh Linode IPv4 address -l user
ssh: connect to host Linode IPv4 address 22: Connection timed out
[Tue Jul 04 10:34:33 :~ ] $ ssh Linode IPv6 -l user
Last login: Mon Jul 3 13:33:35 2017
And then blocked by firewall?
sudo iptables -L INPUT -v -n | grep "Comcast IPv4 address"
nothing
sudo ip6tables -L INPUT -v -n | grep "Comcast IPv4 address"
nothing
[Tue Jul 04 10:42:23 :~ ] $ cat /etc/hosts.deny
ALL:145.236.80.46
ALL:186.178.173.28
ALL:46.148.18.163
sudo netstat -tulpn
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 26072/sshd
tcp6 0 0 :::22 :::* LISTEN 26072/sshd
PING from Comcast. Guessing Comcast has an issue.
[Tue Jul 04 10:45:59 :~ ] $ ping -c 3
PING
64 bytes from 2600:3c00::22 (2600:3c00::22): icmp_seq=1 ttl=52 time=78.8 ms
64 bytes from 2600:3c00::22 (2600:3c00::22): icmp_seq=3 ttl=52 time=473 ms
–-
3 packets transmitted, 2 received, 33% packet loss, time 2030ms
rtt min/avg/max/mdev = 78.886/276.307/473.729/197.422 ms
[Tue Jul 04 10:46:05 :~ ] $ ping -4c 3
PING
–-
3 packets transmitted, 0 received, 100% packet loss, time 2029ms
[Tue Jul 04 10:46:20 :~ ] $ ping -c 3
PING
64 bytes from qm-in-x93.1e100.net (2607:f8b0:400d:c02::93): icmp_seq=1 ttl=42 time=52.6 ms
64 bytes from qm-in-x93.1e100.net (2607:f8b0:400d:c02::93): icmp_seq=2 ttl=42 time=47.1 ms
64 bytes from qm-in-x93.1e100.net (2607:f8b0:400d:c02::93): icmp_seq=3 ttl=42 time=44.7 ms
–-
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 44.714/48.193/52.683/3.340 ms
[Tue Jul 04 10:46:38 :~ ] $ ping -4c 3
PING
–-
3 packets transmitted, 0 received, 100% packet loss, time 2024ms
You can delete all rules in a chain (for example INPUT) like this:
iptables -F INPUT
Believe there is something also going on with the Comcast stack.
Anyway, LISH via SSH. Ran:
sudo iptables -F INPUT
sudo iptables -F FORWARD
sudo ip6tables -F INPUT
sudo ip6tables -F FORWARD
sudo iptables-save
sudo ip6tables-save
ssh Linode IPv4 address -l username
Port 22 timeout. That command usually works.
Called Comcast, may take an in-home visit to sort out.
Ran:
sudo iptables -L INPUT -v -n | grep "IPv4 customer address"
693 41580 DROP all – * * IPv4 customer address 0.0.0.0/0
sudo iptables -D INPUT -s IPv4 customer address -j DROP
sudo iptables -D FORWARD -s IPv4 customer address -j DROP
sudo ip6tables -D INPUT -s IPv4 customer address -j DROP
address not found
sudo ip6tables -D FORWARD -s IPv4 customer address -j DROP
address not found
sudo iptables-save
sudo firewall-cmd --zone=public --add-rich-rule 'rule family="ipv4" source address= IPv4 customer address accept'
See if that is enough.
SSH to Linode's IPv6 address works. IPv4 address times out, coming from Comcast.
Finally, why not try traceroute between the two systems? maybe there is something wrong with IPv4 routing.
Also last night ran:
sudo firewall-cmd –zone=iredmail --add-rich-rule 'rule family="ipv4" source address=40.132.185.99 accept'
Customer’s IPv4 address blocked again after 1:30 EST.
sudo cat /var/ossec/logs/active-responses.log | grep Customer’s IPv4 address
Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete - Customer’s IPv4 address 1499189996.357265 604
Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604
Wed Jul 5 01:41:06 EDT 2017 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604
Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/host-deny.sh add - Customer’s IPv4 address 1499233267.56991 604
Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh add - Customer’s IPv4 address 1499233267.56991 604
Wed Jul 5 01:41:07 EDT 2017 Unable to run (iptables returning != 1): 2 - /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604
Maybe you should contact whoever installed and configured OSSEC.
Last night ran:
sudo /var/ossec/bin/ossec-control stop
sudo iptables -F INPUT
sudo iptables -F FORWARD
sudo ip6tables -F INPUT
sudo ip6tables -F FORWARD
sudo iptables-save
sudo ip6tables-save
sudo /var/ossec/bin/ossec-control start
Customer’s emails continued to arrived overnight. First time in about a week.
Found the XML file unchanged.
Ran: sudo cat /var/ossec/logs/active-responses.log | grep Customer’s IPv4 address. Better than before.
Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete - Customer’s IPv4 address 1499189996.357265 604
Wed Jul 5 01:41:06 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604
Wed Jul 5 01:41:06 EDT 2017 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499189996.357265 604
Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/host-deny.sh add - Customer’s IPv4 address 1499233267.56991 604
Wed Jul 5 01:41:07 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh add - Customer’s IPv4 address 1499233267.56991 604
Wed Jul 5 01:41:07 EDT 2017 Unable to run (iptables returning != 1): 2 - /var/ossec/active-response/bin/firewall-drop.sh delete - 40.132.185.99 1499189996.357265 604
Wed Jul 5 10:55:30 EDT 2017 /var/ossec/active-response/bin/host-deny.sh delete - Customer’s IPv4 address 1499233267.56991 604
Wed Jul 5 10:55:31 EDT 2017 /var/ossec/active-response/bin/firewall-drop.sh delete - Customer’s IPv4 address 1499233267.56991 604