My New Computer

[Rikku]

Just invested in a new number cruncher, let me know what you guys think and any ideas on what I can do to it. So far I'm planning on painting them all black, putting black mesh in the empty bays and somehow having a red led light chaser in the front of each machine.

[Kitz E Kat]

Better elaborate on your setup dude. Exactly what is the spec on that beast ?
I want one !

[Rikku]

It's all PIII's, various spec's but most are dual proscessor 1ghz machines with between 512MB and 1024MB RAM, redundant PSU's and dual NIC's. Still need some network switches and power leads and stuff like that. It's gonna run OpenSSI probably on a heavily customised version of Fedora Core 3, we've tested the setup on Knoppix/OpenSSI and that seems to work okay! Let's just say its main use is going to be 'cryptology' haha

[Kitz E Kat]

Awesome :-)
I actually got some books on said subject ( cryptography), but the maths is mad! So I had to ditch em, and go study the maths first :-(
It's an area that I would love to "crack" , get it ? Ah forget it ......

Dude, keep us informed on what your doing and how your doing it, it would make an awesome tutorial, one which you should write up as your going along.

[Rikku]

Yeah cheers, never considered that! It'd be a fine mix of software, hardware, networking and modding all in one package! Haha

[Rikku]

Today we made a kinda prototype for hat the others are going to eventually look like, I just spent a while trying things and making sure it is gooing to look alright, I think it's going to look pretty sweet when it's done!

[matt138]

looking sweet as a nut, when you get some LED chasers in there it will look awesome!!

[Rikku]

Todays update- Today we stripped down and painted another 4 machines ready to go back together tomorrow, getting the hang of spray painting now! I also made up half of the network cables ready for linking them all up.

Todays side project- Will broke the VOIP headset in a kinda gamer rage so I decided it would be fun to make a new VOIP phone out of a cardboard box and assorted bits of hardware...

(we also drank copious amounts of caffeine drinks...)

[FrontDesk]

looking sweet as a nut, when you get some LED chasers in there it will look awesome!!

Hmmm, sweet as a nut.... WHAT?

[TreeFrog]

Hmmm, sweet as a nut.... WHAT?

Its an expression FrontDesk. It refers to an aspect of being tight and together.
The laundry that is thrown all over the floor before it goes in the washing machine is not so sweet but when its all in the machine, tided up and the wash is on you could say in conclusion "as sweet as a nut"

Its nice to see everyone is still here and doing stuff.

Good luck with the monster Rikku. keep writing it up.. Its inspiring.

[Rikku]

Got a few more completed now, however Im running low on funding and Im on an exercise with work for the next few days so not much is gonna get done til next week providing I can get some funding! Fother Mucker!!

[Will4764]

I don't think the photos do this thing justice.
It's big, aha!

It's just shy of 6' when stacked properly and 38" wide.
So round it up, it's 4 foot wide and 6 feet high and sat in my front room.

It's still OS less at the moment, and running live, side projects keep taking my mind off it, I'll try on Sunday and spare some time up to get it working properly, or at least a few nodes.

For now I'm trying to make my phone actually ring.........which is another side project aha.

[Kitz E Kat]

Great work guys :-)
Are you powering that off an Asus Eee ?

More technical details on the setup please, and the black looks the dogs :-)

[Will4764]

Well all in all there are 23 machines.

One started off as a project of mine, still a work in progress.
It's got a Dual Core 3.0GHz Skt 771 Xeon, though I plan on putting another in, so that'll be 2x3.0GHz. 1 x 1GB FBDIMM though plans allow for it to have 4. Which will be the overall master when all nodes are running.

There are 2 x (2 x 1 GHz and 1 GB RAM) Which are the masters for each stack and each stack consistes of machines ranging from 1 x 800MHz to 2x1GHz with as little as 128MB ram and as much as 1GB.

Each stack has 11 machines (including masters) with +1 overall master.
That's 23 machines in total, with 42GHz overall in the stacks +3GHz(Dual Core) for the overall master with plans for another 3(Dual) adds to 54GHz CPU and in total there is 12GB RAM.

I hope that makes sense, but each machine has such different specs that it would be difficult to sum them up and I really can't be bothered checking each machine to see what exact specs we have, haha.

It's written on the top of them all but they weight a ton and it takes to long to shift them all. I will eventually make a list of each spec though, when they're done.

The plan is to have them set up to cluster as a 23 Server Cluster or as 2 x 11 Server clusters. Though I also intend on installing HDDs in them all and setting them up as two networks for wargaming.


The OS is going to be FC3 with OpenSSI 1.9.2 (Devel). I think that is tomorrows task if I get chance. Apart from that I can't think of any other details haha.

[Rikku]

Well I'm glad you typed all that up, as I really couldn't be bothered! Haha, but that's summed up pretty well!

[Will4764]

Yeah man, that's about as much as I could be bothered putting.
I'll give more details later on the OS when I get home to actually build it.

Stagg Onn.

[GrowMoreWeed]

Awesome project guys! Looks like a hole pile of fun you got there. It all might look like a nice tidy install but Rikku didn't show us the back: :D

http://i286.photobucket.com/albums/ll111/26BB420/network_cabling.jpg

I've just started studying to try and get certified in VMWare so I could use a stack like that myself to practise on as my poor lone server is starting to creak running 4 VMs at a time :(

[Will4764]

Wow, that's bad for air flow, I'll get Rikku to take a picture of the back of the system and post if for you.

It's all nice and stacked in piles for different things happening to different machines. Like one stack is geting it's OS sorted out, one stack is done and ready to be uploaded to the cluster and one stack is stripped waiting to be sprayed.

So for a picture of the back it's gonna need stacking again...effort.
We'll get one for you though. Tested airflow yesterday with 10 nodes running, this thing sounds like an apache, we're trying to get the bios hacked up so that we can get the fan speeds down and the thing a little quieter.

[Rikku]

Today I configured the server management utility and managed to get the fan speeds brought down to a reasonable speed so it isn't as loud as before. I also configured the first node and had a basic 2 node cluster in Fed Core.

Right now I'm having second thoughts on an OS, we can configure it all to run off Fed Core and have it cluster okay, the only problem I'm having is the lack of SMP applications, applications such as John the Ripper aren't ideal applications for parallelization unless you are maybe using some kind of load balancing kernel that can split a single process over multiple cpu's, and with the relatively low speed of cat5e this isn't practical. I'm going to do more research on symmetric multiprocessor systems and parallelization of applications.

Out.

[Will4764]

Still, it was nice to see the thing actually running as a cluster, albeit that slight bit slower and that slight bit quieter.

It's getting there dude, it's getting there.

[Rikku]

The past couple of days have been non-stop coffee and software, but it's paid off...

I found an MPI version of john the ripper which parallelizes some of its functions over MPI software running on your cluster, I downloaded and installed OpenMPI and after numerous uninstalls and re-installs got it working okay! Progress! john the ripper has a benchmarking feature so it is ideal for testing MPI is set up correctly and my first real cluster application!

Tonight I finished the network boot images for the nodes so they all start up and the server management kicks in and proceeds in "SHUTTING THE FANS THE FUCK UP!!" so they all run nice and quiet! Also they have ssh running to a good degree so they are all centrally managed! And OpenMPI, my message passing interface is working and running between nodes!

To test it I ran a john the ripper benchmark on a single 1ghz PIII and got a result for Traditional DES at roughly 380000c/s, I then passed jtr through MPI on localhost to benchmark using the other CPU in the node, to my joy it worked perfectly and ran at roughly 800000c/s on DES! Great stuff! So I set away and started up a cluster of 4 nodes, totaling 6 CPU's. After some tweaks to ssh and numerous reboots, it worked! I haven't got the exact benchmark results but for a standard raw MD5 we had something like 8,400,000c/s!

I can't wait to get more nodes up and running, but right now I'm so so SO very poor I can't afford any more hardware, we are limited to a 5 port switch and we haven't got enough power leads etc. to get the whole lot up and running!

So for now, I'll be working with what I've got and perfecting the software a little bit and tidying up the OS because right now things are all over the place to get the thing working! It's rough for the time being but we have progress!

Out.

[Daremo]

OUT FUCKING RAGEOUS

Rikku that is very cool. I am impressed!

[Rikku]

Oh good :D payday soon so maybe I can buy myself a new network switch and get this thing running at full steam! In the meantime I have a LOT of RJ45's to fit haha and I'm also working on getting a decent set of hashes to test the rig on, nothing better than benchmarking using the real thing!

There's still some software I want to work on, I'd like to get wake-on-lan configured so I can boot up the nodes I need from the master and also there is some software called OpenSSI webview which allows graphical management of the cluster on a web-page. Which leads me to maybe someday the idea of connecting into the master node over the internet and using WOL to wake up the nodes I need and remotely crack hashes! Mega!

Out.

[FrontDesk]

um... *shuffling toe in dirt*.. um.. ok.. uh. maybe I understood one or at best two words of the whole thing. Now I'm spending all my time looking all that stuff up. GEEZ

[Will4764]

The whole network this thing is running on is pretty ninja, and it'll all be remotely controlled. In as secure a way as possible of course, I'll post a little more about how the cluster works in the network as a whole when it's a little less proof-of-concept and a little more set-in-stone.

[Daremo]

Well it seems that old Daremo's spy network has uncovered the secret location of Rikku & Will's cluster.

Therefore in the interest of free speech and full disclosure I would like to present this candid photo of Rikku & Will hard at work bringing their creation to life: (Just kidding guys....I saw this and couldn't resist )

[relik]

That's me on the other side by the way... :)

[Rikku]

Hahaha that's a classic :D

More pics... one stack painted up and I've started to build the OS and apps onto another master node.

[Rikku]

Oh and as Will requested, a picture of some cable management!

[Kitz E Kat]

Keep the updates coming please :-)
This is one awesome project, well done so far, total awesomeness !!

One small technical question springs to mind , who is picking up the bill for the electricity consumption ? :-)

[Rikku]

-cough- something about clocking the electric meter? -cough-

[Kitz E Kat]

-cough- something about clocking the electric meter? -cough-
You wanna get something for that cough before it get's worse :-)

[Will4764]

It's getting worse people,
Not only do I have a cluster server in my front room.
I've not got an uber monitor. Well actually, four not so uber monitors, but it looks cool none the less.

It's a row of three 19" monitors, wide standard wide, with another 19" wide above the middles standard. If that makes sense. Making a sort of upside down T shape.

The plan is to have the three bottom ones hooked into my main machine with the furthest left on a KVM to the cluster (not that it should ever need direct KVM connection as 99% of the time it's all done with ssh, it's just easier because getting behind the thing to move KVM cables that 1% of the time is an absolute nightmare) with the top monitor hooked into the router showing information about what's happening on the network and showing graphical output coming from the IDS.

It looks cool, 'nough said.
Photos to follow.


P.S. Oh and I couldn't get hold of a VESA mount to mount the top monitor above the other three so it's sort of.....cable tied to the wall....it's a must see! haha

[Rikku]

Right, so I'm picking up my new 24 port switch today so we should soon be able the unleash the power of the cluster!

I've been running some tests using john to see how much adding nodes actually improves performance of cracking hashes, I first tried john using it's benchmarking facility on RAW-MD5 encryption on 1 node using 1 cpu and got a result of 1330K c/ps, I then added 4 nodes and ran the same test over OpenMPI using all 5 nodes and all 10 cpu's and got a result of 13,000K c/ps, almost perfectly 10 times more than running on 1 cpu! Great stuff! We set it going on a hash file of raw MD5's taken from an active environment in the wild, containing around 45 hashes, within the first hour we had over 20 of the hashes cracked so we left it running over night to churn away some more.

I've found that running john over a cluster isn't actually too network intensive, it allocates each cpu with a certain keyspace to go over and the cpu works at it and reports back to the master node after it's completed it's work unit and gets another work unit to work on, so although when the nodes report back they all seem to do it at the same time, it isn't creating that much traffic, thus not really slowing the process down too much at all!

Attatched is a picture of the state of the "ComCen" as it was in last night, enjoy!

Out.

[Rikku]

Right so here's the latest, I went and picked up the new switch and the very VERY kind man upon collection gave me another one free! How chuffed was I!? I was pleased enough to be getting a switch worth 130 for 36 anyway, never mind buy one get one free! So I got all the kit back and started working on what I was going to do, I decided after running a few tests that the master node was working at full capacity and it only has a very basic hard drive, and 10/100 LAN, so I wheeled in my big vapo pc and set away at making that my master node as it has that rather nice raptor drive and 10/100/1000 LAN. I didn't realise at first how much of a task this would be, up until now the master nodes have been of pretty much the same spec as all the nodes, making system imaging and network booting a pinch, however my new master node was an entirely different beast. However after a few sleepless nights of fiddling around I got it booting the nodes nicely and got all of my software installed and working!

Time to test it!

So now we have 11 nodes, each with 2x 1Ghz cpu's and my new master with a 3.4Ghz overclocked athlon FX55, schweeeeet. So, back to john, 23cpu's, I ran the benchmark a few times and got an average result of 44,500,000c/s, that's pretty damn quick! (as was the electric meter at this point o.O) To get a better idea of it's ability we gave it a hash we had set it working on before, before now we had some of the machines working on this hash for over 24hrs to no avail, so i was interested in seeing how the new beast would handle it! Sure enough we cracked it, in under 12 minutes! Now all the effort is seeming to pay off!

I took a break from sitting in awe at the benchmarks and started getting the cluster more manageable, I added a web interface so you can log into the cluster and view the status and workload of all the nodes at any given time and also manage some of the processes running on the cluster. I also got X working properly on the master node, before now adding nodes into the cluster caused my X to stop working, I found out this was caused by the nodes sharing the master nodes filesystem and causing problems with things in my /tmp, so I gave all the nodes their own /tmp directories on the master so there was no conflicts. Will gave up one of his 19" monitors so I spent a good while getting all my graphics drivers set up so I have full 3D support over 2 screens, making it that little bit more fancy.

For now the cluster is going to stay at the size it is, we still have a lot of machines left over that, in time, I will add to the cluster. It's just a matter of painting them all and getting the hardware specs up to standard. I'm going to spend some time perfecting the software and seeing what else I can make them do, I'd really like to get something like cowpatty running over the cluster as well as john does, but hey, I'm no amazing programmer so it could take some time!

[Will4764]

Will gave up one of his 19" monitors so I spent a good while getting all my graphics drivers set up so I have full 3D support over 2 screens, making it that little bit more fancy.

I think I can go one better than that, now that I've got my monitor setup working properly! Three 19" Widescreen monitors boasting a res of 4320x900 running Xinerama so that I can drag windows between and now with actual hardware support for the graphics so my CPU can cool off a little and the GPU can kick in for some multihead games,

Also, I can now have windows open on 1, 2 or all of the monitors, and with the monitor bezels stripped off the distance between screens is about 10mm, though due to the curve it's barely noticable. This has got the added benefit of allowing me to play videos on fullscreen over two monitors and still being able to surf on the third, or going *ultra* widescreen and playing games over all three heads.

I'll get some photos of the thing setup and I'll get some screenshots of in game action with Quake3 because that's pretty much the only game I still play nowadays,



As per usual, photos to follow...

[GrowMoreWeed]

2 words: KICK ASS! :D

[usb]

w0w great work

[SyntaxHeir]

I used to work on a stack of Compaq Proliants HALF that size. I served 3000 users email, documents, web proxy, backup the whole works with half that hardware.

Are you looking to start your own Internet? Overthrow a small country? Launch an ICBM??

[Rikku]

O rly? Maybe you can advise me on BIOS settings and the likes, setting boot order and all the usual stuff! It all seems to be software driven and a bit of a nightmare to be honest!

[SyntaxHeir]

Dude... you've come to the right place.

What you need is the "Compaq Insight Manager" for that model. HP does an OUTSTANDING job at archiving and making available old Softpaqs (bios updates) and old utilities.

Dig around a bit looking for your model number and using the terms "Compaq Insight Manager" and Softpaq and you're in the money.

The Compaq Insight Manager is a great piece of free software that will give you details on your drives, fans, memory usage, NIC utilization etc... however... now that I think of it you're not running an MS OS are you.

Oh well, the softpaqs will still work!

[Will4764]

Yeah it's running Fed Core,
and would the insight manager work when it's running as a cluster?
It should, but heck, cluster computing isn't exactly average.

I'm sure the WebView stuff we've got on there shows us stats too,

[Rikku]

The webview is just for monitoring the stats of the cluster status, ie. nodes up etc. I do have one of the hp apps just to slow the bastard fans down but although it's the correct software it doesnt function right, I cant set my onboard NIC to PXE, which is the main thing I wanna do if I can

[TreeFrog]

This is really FUCKING AMAZING stuff guys.. :cool:
More power to you. !

It looks really interesting too.. nice to be able to work on a project like that with someone too.

A real inspiration..

Thanks.
:-)

[usb]

Some info that might be of use:

cbench - Scalable Cluster Benchmarking and Testing (http://cbench.sourceforge.net/#Cbench-ScalableClusterBenchmarkingandTesting)
http://cbench.sourceforge.net/

[Will4764]

Haha,

Benchmarks are always good, so far the main benchmark has been the MD5 benchmark built into John. I think we're kicking out about 52,000,000c/s with that now, which is not bad.

[SyntaxHeir]

Are you certain those onboard NICs even support PXE? The Pre eXecution Environment was still indevelopment (at Intel) when those old Proliants were released to manufacture.

[Rikku]

I actually don't know, I thought they did! I could do with getting etherboot images and doing it that way if I can...

[Will4764]

Dude, I thought that was the reason that we couldn't do etherboot, because they were PXE supported?

[Rikku]

I just couldn't find a boot image that worked!

[Rikku]

How it stands now...

Not been booted in like 3 weeks! Hope it still works haha! Stupid military exercise shit....

[SyntaxHeir]

Does it sound like a jet turbine?

[Kitz E Kat]

Looking for something to keep that beast busy ?
Try this out for fun , http://www.mersenne.org/freesoft.htm
Find a new prime number, become rich :-)

[Will4764]

It sounded like a damn apache originally, but it's a little quieter now, still shakes the house on boot though, because the fans start at fully speed until the software throttles them and kindly outputs on boot...
"Shutting the fans the fuck up". It sounds ace. I love this thing =]

[Rikku]

Okay so... bad bad me... I bought another 4 new servers today, this time HP Proliant DL140's... 2x dual core 3.06Ghz Xeons, 2GB RAM, dual 1000MB ethernet... couldn't resist the cheap cheap price! So I got them home, got them rigged into my cluster master node and set them up for cracking, ran a few tests and then added them to the whole cluster... I think we have a new record! Just short of 87,000,000c/s would have been a bit faster but I had some problems with 2 of the DL380's so we were 2 nodes down!

Gotta break that 100,000,000c/s barrier...

[Will4764]

Yeah we HAVE to break that 100Million mark, before you go away man. Have to.
Even if I've got to buy a start of those DL140s,

*must resssssist eBay*

[Will4764]

Rikku, we definately need to get a collection of Will's flat photos. Just because we keep modding it and I don't think we have a home for all the photos of that cluster and the desk, which seems to some how take up 97% of my home! haha!