bandwidth usage levels

Did a little bit of investigating:

I ran ifconfig twice, one minute apart, then diff'd the output of the two :

5,6c5,6

< RX packets:9306054 errors:0 dropped:0 overruns:0 frame:0

< TX packets:37061 errors:0 dropped:0 overruns:0 carrier:0

–-

RX packets:9307948 errors:0 dropped:0 overruns:0 frame:0

TX packets:37072 errors:0 dropped:0 overruns:0 carrier:0

8c8

< RX bytes:401100222 (382.5 Mb) TX bytes:2010823 (1.9 Mb)

RX bytes:401178913 (382.5 Mb) TX bytes:2011773 (1.9 Mb)

So in one minute, i recieved 78000 bytes and 2000 packets.

During that one minute i also ran tcpdump capturing packets that were desinted for or from my IP and only collected 33 packets a handful of which was my ssh session.

I then ran tcpdump capturing all packets, and watched a steady stream of packets go past. This stream seemed to come and go. For a while there was a lot of HTTP activity, second time around alot of netbios scans and HSRPVo-hello packets being broadcast to 224.0.0.2

Since i wrote my original post apparently i've consumed another 3MB! At this rate i'm guessing that in one month i'd consume 5GB of bandwidth even if i wasn't running ANY listening services and never logged into the box. How does this compare with other linode stats ?

I've turned off all listening services, except for the DHCP client, and i'm now upto 514MB. hmmm ….

@s:

How does this compare with other linode stats ?
The control panel says I'm up to 1.53GB Total, that's 1.49GB Incoming and 41.2MB Outgoing.

IPTables shows a lot of traffic hit TCP port 135. What's interesting is that it is almost all coming from 64.2/16 through 64.7/16

$ cat /var/log/messages |grep "SRC" |wc -l
619
$ cat /var/log/messages |grep "DPT=135" |wc -l
600
$ cat /var/log/messages |grep "SRC=64\..*DPT=135" |wc -l
581

I'd assume this is MSBlaster (or related worms) coming from infected machines inside ThePlanet's datacenter. If I turn on logging for echo request (Nachi worm) it logs a lot of traffic from the same networks… looks like ThePlanet has a good number of infected windows servers.

Kinda related and probably a dumb question, but am I corrected that broadcast traffic does not count against the linode's incoming network quota? I'm silently dropping most of it but I do remember seeing a fair amount from dhcp, samba, etc.

kenny

Is blocked traffic counting toward our bandwidth usage?

There are a few issues:

A user on our network who is running smb/nbd and broadcasting UDP packets to everyone on our subnet. I've contacted them to ask them to quit it.

Second is the broadcast traffic problem; right now it is counted in the ip accounting. This is wrong. Ive logged this as a bug and we'll correct it.

Third issue is the network monitoring we do. Check "man nmap", the description for -sP when run as root. It has three methods it uses to see if a host is still up.

The correct solution is for us to update the bandwidth accounting to not include broadcast and local IP traffic.

Until we can get it fixed, don't worry so much about the extra bandwidth; if you go over I'll calculate that in manually.

-Chris

> The correct solution is for us to update the bandwidth accounting to not include broadcast and local IP traffic.

By "local IP traffic" I assume you mean traffic which does not leave the host cluster, i.e. from one linode to another. Is that correct?

Right – anything that doesn't go through our gateways (out to the net) shouldn't be counted.

-Chris

Basically it does not cost you money if it does not leave the gateway, therefore there is no reason why we should be billed for it.

It does mean one good thing though.

If I was to have 2 linodes, 1 with apache and the other with mysql I would not be billed for traffic between the two linodes.

Adam

Thanks for the respones.

That idea of having two linodes ( one mysql + one apache ) is a bit of smart thinking. You could get a whole bunch of linodes and run your own cluster! The possibilities are endless… ;)

The only real advantage of having them seperate is for security issues.

But that would only really help if Chris would make the IPs assigned to a linode non-routable, so the data would never leave the internal network.

Adam

@adamgent:

The only real advantage of having them separate is for security issues.

There would be a performance benefit from separate Linodes, as well. Even if they were on the same host – since our hosts are dual processor, two Linodes can occupy processor time at the same time.

@adamgent:

But that would only really help if Chris would make the IPs assigned to a linode non-routable, so the data would never leave the internal network.
I'm still on my first cup of coffee, but how would this be different than using the route-able IP range? The traffic would still be switched anyway… Also assume that local-routed traffic isn't counted towards bandwidth usage.

-Chris

It is more that the node can not be accessed via the internet.

The traffic isnt that much of a problem, becuase as you say the internal traffic would not be counted.

It is more the security aspect of it.

Under a standard server set-up, if you wanted a DB server that can not be accecss by any method over the internet, you would set-up a seperate lan for internal traffic between the web servers and the database serves. The web servers been accessible over the internet via the external network.

Under the linode set-up, if the IPs where non-routable the linode could not be accesssed from the internet, but only via another linode or through the console access, thus increasing the security around the node.

Make any more sense?

Adam

@adamgent:

It is more that the node can not be accessed via the internet.

(snip)

Make any more sense?

Adam
Ok, I must have missed that we were talking about Linodes on a private LAN. Good idea :-)

(caker runs for more coffee)

-Chris

Non-routing IPs with a good set of IPTables is just as good as a private LAN.

Private LANs only really come into effect when you have many of your own servers, either dedi or colo, much like what you have for the linnode servers.

Beside private lans came up under the central storage and distributions of linodes.

Non-routing IPs are the cheapest and quickest way of doing things, although private lans are just as useful, it also means that you can issue many more IPs than arin have allowed you to have.

Adam

@adamgent:

It is more that the node can not be accessed via the internet.

…

Under a standard server set-up, if you wanted a DB server that can not be accecss by any method over the internet, you would set-up a seperate lan for internal traffic between the web servers and the database serves. The web servers been accessible over the internet via the external network.

If that seperate lan is still attached to the same physical network, how is this any different then just removing the routes on your DB server and a using good set of iptables? They both would be accessible from any device plugged into the lan (other linodes), and both wouldn't be accessible from any device outside the lan (internet). Am I missing something?

kenny

It isn't

The standard method is to usual set-up a private lan, between just your servers, so only you would have access to them, if you where colo servers, or had a bunch of dedicated servers, it is also to do with bandwidth usage.

The way the set-up is at linode.com it would not matter either way.

Adam

Just another datapoint: I created my first Linode this morning, booted it briefly to check I could ssh into it, and then shut it down again. Nothing in the outside world knows it exists yet.

But still I have 514KB of incoming traffic in a few hours, to a brand new linode that's 'off' and no-one knows about. Clearly that's not right :)

Chris, do you have an e.t.a for a fix for this?

Paul

@PaulC:

Chris, do you have an e.t.a for a fix for this?

Paul
I can't give you an ETA yet, but I have been messing around with a solution…

-Chris

In an effort to minimize traffic I have set up shorewall, to DROP packets I don't want, this should half 'idle' traffic as it doesn't send a responce. It's really weird, I was getting heaps of pings from *.hotmail.com hosts. Now I don't reply to them. Im also using ipfm to log traffic, by source.

I added some network filtering so the UDP port 137 traffic should no longer be running wild.

I also improved the network monitoring. You should only get ICMP ping (and possibly a packet or two to port 80) from the host your Linode is on every minute. Previously, every host was blindly monitoring every IP, regardless of where the Linode was.

Still working on the "non-local-only traffic accounting" fix.

-Chris