Setting IPv6 slow down ssh login

I configured the firewall for IPv4 (http://bencane.com/2012/09/17/iptables-linux-firewall-rules-for-a-basic-web-server/) according to this article. Access via SSH, Filezilla are lightning fast into my Linode server. When I add the setting for IPv6 with exact the settings as IPv4 firewall (apart from changing iptables to ip6tables), here are the outcome:

1. SSH into my server delays between 2 - 4 minutes (slow)

2. Filezilla access totally failed

3. Web can still be viewed successfully in the browser

Then, I changed the IPv6 setting to DROP all INPUT chain … yet, the same thing happened.

The conditions only improved when:

1. Just config the IPv4 firewall only, and default (do nothing) to IPv6 config OR

2. Totally remove the IPv4 and IPv6 firewall

My question is why? … because I do not foresee any difficulty in writing iptables for IPv4 and IPv6 - they are similar

Please provide suggestion on how to overcome this trivial issue.

TQ