Linode Voluntary distcc Community
What do you think about a voluntary distcc network where anyone can connect up Linodes for sharing CPU time for compiling. Let me clear up some issues that came to mind before you throw this one out the window.
1. This would not be restricted to Gentoo users, anyone could set their system up for it.
2. This could in fact help others not using the network by softening the blow of Gentoo users on the same host as well as making the time the host is caught up in compiling much shorter.
There's 3 other issues to be addressed that I can think of right now.
1. There would have to be 2 separate networks, one for each DC (they could be one network, but then you'd be using internet bandwidth).
2. Building on 1, Chris would have to setup something for not counting cross-linode-same-DC traffic as part of the used bandwidth.
3. Since CPU time is burstable, it would be pointless to have someone with a Linode on the same host compile anything. So there would have to be a design to prevent that. I'm not sure what could be done with that.
Has anyone brought this up before?
10 Replies
@tierra:
What do you think about a voluntary distcc network where anyone can connect up Linodes for sharing CPU time for compiling.
If someone else wants to work out the details of working with Chris so that the bandwidth doesn't count in a DC and work out the details of ensuring two hosts on the same machine are not working on the same problem then I would be happy to contribute my CPU towards the goal. I'm having a couple of problems on my machine right now but when/if you get this going feel free to contact me about joining in. I could even host a webpage or something that would contain instructions/software to show others how to join in.
Technically speaking though, this is Linode, where your system resources are divided up using UML, and your resources are yours to use however you want. And technically speaking, if I and any other Gentoo user wanted to link up hosts with distcc, nothing about Linode restricts us from doing that already (except that it would count against our bandwidth, which wouldn't be entirely fair since we wouldn't be charging that bandwidth on the TP/HE bills).
I've 2 questions :
wouldn't you need the same gcc version on all nodes in a distcc farm (distcc newbie here) ?
is the bandwidth between linodes really counted by caker : I would have guessed that the bandwidth counter was setup on the network borders (maybe on a firewall). Technically it can be tricky to monitor bw between hosts in the same DC…
We could organize ourselves without bothering caker if this is the case :
we already know on which host each linode is hosted (the linode owner's knows),
we already know in which DC each host is installed (this is on
www.linode.com ).
The only thing left is to setup a page (Wiki ?) with :
two lists of linodes (one for each DC) willing to share computing power grouped by hosts (we might need one list per gcc version too),
groups of linodes already setup to cooperate.
In the list, just add the group a linode is in to help people choose free linodes.
When a new linode wants to use distcc, it adds itself to the appropriate list, choose an existing linode group if there's one with no linode on the same host as its one or just create another group with only itself waiting for others to come in.
There should be a "note" section below signaling linodes being unresponsive in order to clean the lists and the groups when people stop using distcc or go away, have temporary problems, …
What we lack could be an automatic way to firewall distcc.
Distcc users should setup their firewall to allow only hosts in the same group.
This can be bothersome as new users should wait for people to update their firewall rules before using distcc (is distcc able to ignore non responding hosts cleanly ?). Using a Wiki's ability to send notices when a page is updated could help greatly. You could setup a separate page for each group to make things easier.
I'm a Gentoo user with 2 linodes, and I don't.
@gyver:
wouldn't you need the same gcc version on all nodes in a distcc farm (distcc newbie here) ?
Yes, from what I understand, you have to have the same major and minor version, so for example, mines is currently 3.3.4, I could use distcc with anyone that has version 3.3.x. With most everyone using Gentoo though, most everyone's would be the same stable version (I hope no-one is using masked ebuilds of gcc, shouldn't be right now anyway as 3.3.4 is I believe the latest version out including masked ebuilds).
@gyver:
is the bandwidth between linodes really counted by caker : I would have guessed that the bandwidth counter was setup on the network borders (maybe on a firewall). Technically it can be tricky to monitor bw between hosts in the same DC…
I assumed it was a per Linode BW count… I figured UML per host handled that, but only Chris knows.
@gyver:
The only thing left is to setup a page (Wiki ?) with :
two lists of linodes (one for each DC) willing to share computing power grouped by hosts (we might need one list per gcc version too),
groups of linodes already setup to cooperate.
In the list, just add the group a linode is in to help people choose free linodes.
This is a little more complicated than initially thought with GCC versions coming into play, thanks for mentioning it. It would require a little more scripting to automatically assign Linodes to groups so groups were fairly balanced, and it ensured both that Linodes were on different hosts, and also that they have compatible versions of GCC, which brings me to another point: With GCC in play, it almost makes it required (well, it would make it a lot easier to solve a number of our issues all at once) that another service be coded up to run on each participating Linode that would handle a number of things you pointed out.
The first and main reason for a service, would be to handle any firewall configurations that would need to be change/updated with new hosts or remove any non-responsive (whether it be that they stopped using the service, or are trying to cheat the system by blocking access to their distcc, but using everyone else's). Second, it could inform the main control of what version of GCC it's using (which I also just realized that even if distcc has support to check those versions and not use them if they aren't compatible, that it would still be necessary to make good judgements on which groups would work best together). Third, it would listen for group changes from the main control for which hosts it needs to configure itself with. And finally, a fourth reason, to handle any optimizations with GCC settings depending on host settings, like -j#.
Where ever and how ever the main control would be run (whether it be a simple web script, or otherwise), it could also handle reports on service status, keeping information on reports of Linodes abusing the system, etc.
Ok, now that I'm done theorizing on solutions, back to thoughts going on in the back of my head…
This is starting to look a little more complicated than what it's worth. I'm still up for the challenge, but that's a lot of work for a small benefit. I don't even compile that much to make this worth it on my Gentoo Linode. And like dmuench said, I haven't even covered the issue of security with a system like this. It was well worth scratching the itch though and satisfying my curiosity by bringing up a debate on it though.
@tierra:
@gyver:is the bandwidth between linodes really counted by caker : I would have guessed that the bandwidth counter was setup on the network borders (maybe on a firewall). Technically it can be tricky to monitor bw between hosts in the same DC…
I assumed it was a per Linode BW count… I figured UML per host handled that, but only Chris knows.
Bandwidth between linodes within the same DC is not counted, anything which leaves the linode switches is counted.
Adam
Thanks for the info.
That means it would be possible to setup distcc between two (trusted) Linodes on different hosts without Chris's help, and it would be beneficial for all involved including the people on those hosts.
If someone comes up with some good ideas for the major problems discussed earlier in this thread, you could start some work on this page:
Whatever work that gets done on it could turn around and turn into a pretty big open source project to handle a big secure distcc network, which would be very cool, but would require a lot of work and time that I don't have.
I can't think of any method to prevent bad binary data being sent back to requesting machines apart from compiling locally and comparing the resulting file, but this would render distcc useless
Some proposed to send the same source to two or more distcc servers and compare the results. It would certainly decrease a little the risks. But it is not a serious solution as it would eat resources for nothing and not help that much in terms of security (the servers can be all compromised/maintened by "bad boys"). Working with one non-trusted system or hundreds of untrusted systems is the same.
I still think this is really a pity but all we can do is to focus on this big issue before anything else (but as I said, I think there is no real solution).