Join PrimeGrid
Returning Participants
Community
Leader Boards
Results
Other
drummers-lowrise
|
Message boards :
General discussion :
Turning to the cloud
Author |
Message |
mackerel Volunteer tester
 Send message
Joined: 2 Oct 08 Posts: 2584 ID: 29980 Credit: 550,654,016 RAC: 9,807
                             
|
I have a continuing itch for more hardware, but I was reminded not long ago that cloud services might be viable to a degree now. This would get around space, heat, noise problems I'd have running it myself.
The economics I've yet to go through in detail, and while I think running your own will still work out cheaper in the long term, I'm more looking at cloud based for short term use (like challenges).
For starters, I've looked at Amazon c4.8xlarge instances, which give you 36 vCPUs at around US$1.675/h. The lower tiers scale with the number of vCPUs so for practical purposes this will do. These are run on E5-2666v3 CPUs, which are Haswell generation so should be fast per clock, although the base clock of 2.6 GHz to turbo at 3.3 GHz is less remarkable. Also I'm concerned about the vCPU, which I presume has hyper threading enabled. That is, I'm only getting half the number of physical cores than I think. Is that the case? As far as I can tell those Xeons can be run in 2 socket systems, for a total of 20 cores/40 threads, hence the maximum of 36 offered per instance I assume. Still, assuming 18 physical cores isn't bad, although I wonder if throughput might be better if I went for multiple smaller ones, since even with quad channel ram, I can't but fear it will be severely ram bandwidth limited.
Then there's actually running it... I have no experience in this. If I understand correctly, you can use the supplied EBS to provide persisting storage. So that, with minimal configuration, I could retain state between running it? I don't want to have to set everything up every time I fire it up. I think I'll have to sign up for the free tier to get some experience.
Is there anything other than Amazon I should look at also? I believe some have used other services not strictly within their TOS, and I don't want to go there. I'd like something that will likely be around for the long term. | |
|
mackerel Volunteer tester
 Send message
Joined: 2 Oct 08 Posts: 2584 ID: 29980 Credit: 550,654,016 RAC: 9,807
                             
|
Bit of a reply problem up there?
Did a little more digging and just found out about the spot pricing, which isn't what I was looking at earlier. Right now for example, the same c4.8xlarge instance is under $0.30/h! When I looked yesterday, it was around $0.50/h, so variable but a fair portion less than above. My question then is... how does it work? Does your instance just die when you get outbid, or do you get time for a graceful shutdown? Can it be automated in some way e.g. run only if price is <$xx. I've still got more reading up to do. | |
|
|
This article might help?
http://searchaws.techtarget.com/definition/AWS-Spot-instance
And this one.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
____________
| |
|
|
Any update? I'm interested in your findings.
____________
| |
|
mackerel Volunteer tester
 Send message
Joined: 2 Oct 08 Posts: 2584 ID: 29980 Credit: 550,654,016 RAC: 9,807
                             
|
Haven't had time to look further at this. I would comment that you're supposed to get a 2 minute warning before spot instances are terminated if you should get outbid, so those better at linux than me could automate that to shut down boinc gracefully before then. | |
|
mackerel Volunteer tester
 Send message
Joined: 2 Oct 08 Posts: 2584 ID: 29980 Credit: 550,654,016 RAC: 9,807
                             
|
I did a rough calculation on the costs of the cloud. Very roughly, running c4.8xlarge at $0.30/h for one year equivalent, would cost in the same ball park as buying an overclocked Skylake computer-only system (no fancy GPU, or other user comforts). I haven't factored in electricity running the Skylake system, nor if any taxes are due on the Amazon pricing (since it is in US$), but even with these crude estimates, I will get more work out of buying systems than giving money to Amazon.
I'm probably too generous on the capabilities of the Amazon offering anyway, since it will have much less memory bandwidth per core than an overclocked consumer CPU. It might offer more L3 cache per core, but it is unclear if this might offer a benefit.
Funnily enough, I almost ordered the parts from Amazon, but they didn't have the CPU in stock so I found elsewhere.
So as far as I'm concerned, going cloud is on hold indefinitely as the economics for long term running don't stack up. I haven't ruled it out as something for challenges, but I probably wont be motivated enough to have a go at that until closer to the next specific challenge. | |
|
Tyler Project administrator Volunteer tester Send message
Joined: 4 Dec 12 Posts: 1077 ID: 183129 Credit: 1,365,637,185 RAC: 1
                        
|
I did a rough calculation on the costs of the cloud. Very roughly, running c4.8xlarge at $0.30/h for one year equivalent, would cost in the same ball park as buying an overclocked Skylake computer-only system (no fancy GPU, or other user comforts). I haven't factored in electricity running the Skylake system, nor if any taxes are due on the Amazon pricing (since it is in US$), but even with these crude estimates, I will get more work out of buying systems than giving money to Amazon.
I'm probably too generous on the capabilities of the Amazon offering anyway, since it will have much less memory bandwidth per core than an overclocked consumer CPU. It might offer more L3 cache per core, but it is unclear if this might offer a benefit.
Funnily enough, I almost ordered the parts from Amazon, but they didn't have the CPU in stock so I found elsewhere.
So as far as I'm concerned, going cloud is on hold indefinitely as the economics for long term running don't stack up. I haven't ruled it out as something for challenges, but I probably wont be motivated enough to have a go at that until closer to the next specific challenge.
For reference, this is a c4.8xlarge: http://www.primegrid.com/show_host_detail.php?hostid=509007
And this is a g2.8xlarge: http://www.primegrid.com/show_host_detail.php?hostid=508817
For setup to use on BOINC, it isn't difficult especially running smaller tasks where you don't mind when it shuts down because your spot price is exceeded..
When you create the instance requests, you supply "user data" I believe it's called, and it runs that as a bash script when the instance is created.. I found that sometimes it doesn't attach to the project correctly because BOINC isn't finished setting up, so I added a wait in there before the boinccmd project attach command. Here's what I've used:
(Works on debian-based images I've used, including ubuntu server)
#!/bin/sh
apt-get -y --force-yes install boinc-client
cd /var/lib/boinc-client/
rm global_prefs_override.xml
rm cc_config.xml
wget http://mywebsite.com/global_prefs_override.xml
wget http://mywebsite.com/cc_config.xml
sleep 30
boinccmd --read_cc_config
boinccmd --read_global_prefs_override
boinccmd --project_attach http://www.primegrid.com/ [full account key]
boinccmd --project http://www.primegrid.com/ update
And if it's a GPU instance (I've used on g2.2xlarge and g2.8xlarge), I add this to enable GPU Computing:
boinccmd --set_gpu_mode always
boinccmd --project http://www.primegrid.com/ update
For those GPU instances, I use AMI ID 2cbf3e44 (pre-installed Nvidia drivers), and that has yet to fail me.
global_prefs_override.xml contains:
<global_preferences>
<disk_min_free_gb>1</disk_min_free_gb>
</global_preferences>
And cc_config.xml contains the following, depending on how many CPUs I want to run:
<cc_config>
<log_flags>
</log_flags>
<options>
<ncpus>30</ncpus>
</options>
</cc_config>
Hopefully this was helpful!
~Golfer
____________
275*2^3585539+1 is prime!!! (1079358 digits)
Proud member of Aggie the Pew
| |
|
Crun-chi Volunteer tester
 Send message
Joined: 25 Nov 09 Posts: 3114 ID: 50683 Credit: 76,797,694 RAC: 7,338
                       
|
Look at here
https://developer.ibm.com/linuxone/registration/
It is IBM linux one server: you have 90 days of absolutely free, you dont even need to have credit card.
It works under Linux ( you have few Images of Linux) and can be setup in 10 minutes.
Currently I use it for sieving ,so my cores can do LLR.
____________
92*10^1439761-1 NEAR-REPDIGIT PRIME :) :) :)
4 * 650^498101-1 CRUS PRIME
314187728^131072+1 GENERALIZED FERMAT
Proud member of team Aggie The Pew. Go Aggie! | |
|
|
Look at here
https://developer.ibm.com/linuxone/registration/
It is IBM linux one server: you have 90 days of absolutely free, you dont even need to have credit card.
It works under Linux ( you have few Images of Linux) and can be setup in 10 minutes.
Currently I use it for sieving ,so my cores can do LLR.
How do get boinc running on this? it says it's the 390x architecture only and is being a giant pain :( | |
|
Crun-chi Volunteer tester
 Send message
Joined: 25 Nov 09 Posts: 3114 ID: 50683 Credit: 76,797,694 RAC: 7,338
                       
|
Look at here
https://developer.ibm.com/linuxone/registration/
It is IBM linux one server: you have 90 days of absolutely free, you dont even need to have credit card.
It works under Linux ( you have few Images of Linux) and can be setup in 10 minutes.
Currently I use it for sieving ,so my cores can do LLR.
How do get boinc running on this? it says it's the 390x architecture only and is being a giant pain :(
After you deploy image, you must compile boinc to run under 390x architecture.
I cannot say is that can be or cannot be done, but even compiling sr1sieve and sr2sieve need knowledge since it must be changed small part of code.
I would not try to run Boinc on those two cores you get, it is pretty slow ( even for sieving)
So even you compile Boinc, you must compile and rest , since there is no LLR for 390x architecture ( as I know)
____________
92*10^1439761-1 NEAR-REPDIGIT PRIME :) :) :)
4 * 650^498101-1 CRUS PRIME
314187728^131072+1 GENERALIZED FERMAT
Proud member of team Aggie The Pew. Go Aggie! | |
|
|
Look at here
https://developer.ibm.com/linuxone/registration/
It is IBM linux one server: you have 90 days of absolutely free, you dont even need to have credit card.
It works under Linux ( you have few Images of Linux) and can be setup in 10 minutes.
Currently I use it for sieving ,so my cores can do LLR.
How do get boinc running on this? it says it's the 390x architecture only and is being a giant pain :(
After you deploy image, you must compile boinc to run under 390x architecture.
I cannot say is that can be or cannot be done, but even compiling sr1sieve and sr2sieve need knowledge since it must be changed small part of code.
I would not try to run Boinc on those two cores you get, it is pretty slow ( even for sieving)
So even you compile Boinc, you must compile and rest , since there is no LLR for 390x architecture ( as I know)
BOINC has been compiled! Might be able to compile something to do some pps work in the future :)
| |
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13804 ID: 53948 Credit: 345,369,032 RAC: 4,797
                              
|
Look at here
https://developer.ibm.com/linuxone/registration/
It is IBM linux one server: you have 90 days of absolutely free, you dont even need to have credit card.
It works under Linux ( you have few Images of Linux) and can be setup in 10 minutes.
Currently I use it for sieving ,so my cores can do LLR.
I suspect that IBM and the universities providing this free service will probably not look kindly upon this use (misuse?) of their resources.
____________
My lucky number is 75898524288+1 | |
|
mackerel Volunteer tester
 Send message
Joined: 2 Oct 08 Posts: 2584 ID: 29980 Credit: 550,654,016 RAC: 9,807
                             
|
I had a quick look at the TOS, and basically they want you to develop on their system. So arguably this could fit in that (you need to port and test it). Still looks like too much effort for the gain than I have time for. | |
|
Message boards :
General discussion :
Turning to the cloud |