| Author |
Message |
|---|
 Michael Goetz Volunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
                              
|
|
Yes, it's deja vu all over again. We did this last February.
You may have noticed there was some unscheduled downtime today. For a few weeks now, we've been working on migrating the system over to Rackspace. There's two reasons for this move. The first is that in June our previous hosting company imposed SEVERE disk I/O limits on all their users. This crippled the previously excellent server performance, and the server was having trouble keeping up with the load. The Perseid's challenge greatly exacerbated the problem. It was completely impossible to run the challenge statisitics on the server, and the system was having great difficulty keeping up with the daily load given the inflated size of the database in the aftermath of the challenge. I had serious concerns about the whole thing crashing and burning. There was an urgent desire to on our part to get this move done.
The second reason is that Rackspace likes to support open source projects by donating server resources to those projects. Not only are we running on their nice big and fast servers, but Rackspace is footing the bill. This is exceptionally generous of them, and we thank them for their generosity.
For both those reasons, we wanted this move to happen sooner rather than later, but a move like this requires a big chunk of time from both Jim and myself. Remember that this is a volunteer organization, and we've got "real life" family and professional obligations. Finding a big block of time when we're both free isn't easy. This morning, we realized that we not only had all the pieces in place to make the move, but we both had nothing special to do today -- so it was a good day to make the move. But it meant doing so without giving anyone more than a few minutes warning, and for that we apologize.
Unfortunately, those same disk IO limits they put on the old server also made the backup procedure REALLY slow. In fact, it's still going on. Although the new server is up, it's not fully functional yet. The "uploaded" directory -- where your computer uploads result files -- contains several million files and is still being backed up on the old server. That backup started some 5 or 6 hours ago. You can upload new results to the new server, but until we get all those old results copied over, we can't turn the validator on, so nothing's going to get validated yet. Everything will be "pending" for a little while.
Thank you for your patience and understanding.
____________
My lucky number is 75898524288+1 |
|
|
|
|
|
Thank you Michael & Jim for providing your time and ongoing dedication to oversee Primegrid's day to day operations.
Also, a BIG THANK YOU goes out to rackspace for providing the server & other related resources to keep this project alive!
____________
 |
|
|
|
|
|
Yes a BIG thanks to rackspace for their generosity and helping this and other volunteer projects.
Does this move have anything to do with all the lost points on the Stats sites?
Everyone who has been contributing work recently has lost points (I have lost over 46,000 but others have lost many millions).
Hoping all gets sorted out soon.
Also a BIG thanks to Jim and Michael for all their own free contributed time and energy into this project.
Conan
____________
 |
|
|
|
|
Does this move have anything to do with all the lost points on the Stats sites?
Everyone who has been contributing work recently has lost points (I have lost over 46,000 but others have lost many millions).
Yes there was an old set of stats files on the new server. This should correct itself once the stats cron is run on the new server.
____________
 |
|
|
|
|
|
Nice work to everyone involved with the server move - I imagine it's an absolute nightmare completely transferring something like PG. A little bit of unannounced downtime is nothing, here's to a long time of Rackspace-based PrimeGrid. |
|
|
|
|
|
Yeah, I dropped ~2.3M PG credits on Free-DC, which sounds like about a week's worth of work for my GPU. A similar thing happened recently when climateprediction brought up thier new server. and the credit rollback sorted itself out within a few hours. (Though CPDN has ongoing issues with their crons, and new credits still aren't being applied even on their own site.) That doesn't look to be the case here.
Thanks to all involved for their hard work.
____________
  |
|
|
|
|
|
From a very minor user, I also thank you guys for making this happen,
Jim |
|
|
darkclownVolunteer tester
Send message
Joined: 3 Oct 06
Posts: 323
ID: 3605
Credit: 1,219,748,617
RAC: 342,652
                    
|
|
You guys rock, Mike & Jim. Thank you.
____________
My lucky #: 60133106^131072+1 (GFN 17-mega) |
|
|
|
|
|
Thanks to mike et al, and thanks to Rackspace. If I ever do need a cloud I know where I would look first!
____________
Member team AUSTRALIA
My lucky number is 9291*2^1085585+1 |
|
|
|
|
|
This is a wonderful contribution for us from Rackspace! Many thanks to all of those who made this migration possible.
I'd also like to take this moment to send some well deserved appreciation out to Mike and Jim for their continued personal sacrifices and management of the day-to-day operations of this project!
____________
Team North Carolina
My lucky number is 341112^524288+1 |
|
|
|
|
|
This is great! Thanks very much to Mike, Jim and Rackspace for taking such good care of this project.
____________
Proud member of Team Aggie the Pew
"Wir müssen wissen. Wir werden wissen."
"We must know, we shall know."
- David Hilbert, 1930 |
|
|
|
|
|
Good work rackspace. Much appreciated.
May your hovercraft ever be filled with eels.
Err, do you like rats?
____________
Oh Bondage? Up Yours.
http://www.youtube.com/watch?v=ogypBUCb7DA
 |
|
|
Crun-chiVolunteer tester
Send message
Joined: 25 Nov 09
Posts: 3114
ID: 50683
Credit: 76,797,694
RAC: 4,051
                  
|
|
New, better server.
Primes, here we come :)
Thanks Rackspace!!
____________
92*10^1439761-1 NEAR-REPDIGIT PRIME :) :) :)
4 * 650^498101-1 CRUS PRIME
314187728^131072+1 GENERALIZED FERMAT
Proud member of team Aggie The Pew. Go Aggie! |
|
|
|
|
|
Just dropped a note to Rackspace to thank them. Consider doing so yourself; it's the least we can do to acknowledge their generosity.
And keep up the terrific work, Michael and Jim! |
|
|
|
|
|
Echoing all of the posts above: very big thanks to Mike, Jim and Rackspace for the quality management and hosting! I'm planning to drop a note to Rackspace to say thanks for their generousity. |
|
|
|
|
|
Thank you Mike, Jim and Rackspace! |
|
|
|
|
|
Thanks to Rackspace! I'm looking forward to more reliable servers.
____________
 |
|
|
|
|
|
wackspace. |
|
|
|
|
|
Do you have any estimate on when we will be able to upload files? I'm still getting the 'Project communication failed: attempting access to reference site' errors and can't report or request any new tasks.
____________
~It only takes one bottle cap moving at 23,000 mph to ruin your whole day~
 |
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
Do you have any estimate on when we will be able to upload files? I'm still getting the 'Project communication failed: attempting access to reference site' errors and can't report or request any new tasks.
Approximately 50 hours ago. You should have been able to connect to the system as of around 21:00 on Sunday.
Either you have a proxy server which is ignoring the TTL on the DNS records, which is unlikely because you wouldn't be able to post anything here if that were happening, or your BOINC client is still using the old IP address.
Stopping and restarting BOINC, or rebooting your computer, should fix the problem.
Edit: I'm not sure why you're having this problem. My BOINC client had no trouble connecting to the new server as soon as it was up, and I didn't have to reboot or restart the BOINC client.
____________
My lucky number is 75898524288+1 |
|
|
darkclownVolunteer tester
Send message
Joined: 3 Oct 06
Posts: 323
ID: 3605
Credit: 1,219,748,617
RAC: 342,652
|
|
Was likely DNS related. I was able to access the website just fine from one of my linux boxes runing boinc, but the client was getting that message. Restarting boinc-client cleared it up for me.
____________
My lucky #: 60133106^131072+1 (GFN 17-mega) |
|
|
|
|
Was likely DNS related. I was able to access the website just fine from one of my linux boxes runing boinc, but the client was getting that message. Restarting boinc-client cleared it up for me.
Some of the older Linux BOINC clients have problems with caching DNS entries. I know 6.10 series had the problem. I think it was fixed in the 6.12 series but it might not have been until 7.0 series.
Windows BOINC clients had the same DNS cache issue at some point but I think it was fixed for them before Linux. It has been too long to remember.
____________
|
|
|
|
|
|
Exiting BOINC and then restarting it appears to have corrected the problem as it did with darkclown. Thanks. Strange, though. It's never had that problem before with server changes.
[Edit]: Now that I look at some of the wingmen on my reported tasks, it looks like at least some other users are having the same problem I was.
____________
~It only takes one bottle cap moving at 23,000 mph to ruin your whole day~
|
|
|
ich_eben*Volunteer tester
Send message
Joined: 14 Oct 10
Posts: 101
ID: 70063
Credit: 366,065,128
RAC: 101,586
        
|
wackspace.
can you clarify this for me? ;-)
I know you like aws but rackspace sound like a good alternative - also because they are covering at least a fraction of the bill ...
____________
|
|
|
Ken_g6Volunteer developer
Send message
Joined: 4 Jul 06
Posts: 929
ID: 3110
Credit: 236,990,205
RAC: 41,641
                    
|
|
So what does this mean for "the project cover running costs for this month" as described at the top of the page? Are they zero now?
____________
|
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
So what does this mean for "the project cover running costs for this month" as described at the top of the page? Are they zero now?
It's means we won't be continuously running by the fiscal skin of our teeth any more.
It means we can have a reserve fund to handle unexpected expenses. Remember a stats site called "stats and stones"? I liked that site. A motherboard failed and the operator had no funds with which to replace it. He started an emergency fund raising drive, which fell short. The site vanished.
Beyond that, I think it would be best if Rytis answers, because I really don't know.
____________
My lucky number is 75898524288+1 |
|
|
|
|
I know you like aws but rackspace sound like a good alternative - also because they are covering at least a fraction of the bill ...
And I myself didn't even know that Rackspace had a cloud compute platform! Looks like they don't offer a free tier like Amazon does, nor do they have cheaper "spot instances." AWS has 8-way CPUs for USD$0.07 / hour on the spot market. Rackspace's cheapest offering (Linux) is $0.06-$0.12 per core per hour. So for ad hoc prime crunching, probably not the ideal platform. But for high-availability web application hosting, I'm looking forward to seeing how well Primegrid works from it's new home! |
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
I know you like aws but rackspace sound like a good alternative - also because they are covering at least a fraction of the bill ...
And I myself didn't even know that Rackspace had a cloud compute platform! Looks like they don't offer a free tier like Amazon does, nor do they have cheaper "spot instances." AWS has 8-way CPUs for USD$0.07 / hour on the spot market. Rackspace's cheapest offering (Linux) is $0.06-$0.12 per core per hour. So for ad hoc prime crunching, probably not the ideal platform. But for high-availability web application hosting, I'm looking forward to seeing how well Primegrid works from it's new home!
Yes, as similar as they might seem in some respects, AWS and Rackspace are very different, and each is better suited to some needs than others.
____________
My lucky number is 75898524288+1 |
|
|
|
|
wackspace.
can you clarify this for me? ;-)
I know you like aws but rackspace sound like a good alternative - also because they are covering at least a fraction of the bill ...
An insiders joke.
Has nothing to do with AWS. Or hosting differences.
pay no heed. |
|
|
ich_eben*Volunteer tester
Send message
Joined: 14 Oct 10
Posts: 101
ID: 70063
Credit: 366,065,128
RAC: 101,586
|
wackspace.
can you clarify this for me? ;-)
I know you like aws but rackspace sound like a good alternative - also because they are covering at least a fraction of the bill ...
An insiders joke.
Has nothing to do with AWS. Or hosting differences.
pay no heed.
noted ;-)
A total different question for Mike and/or Jim:
On the old server there where some interesting additional statistics shown under "Server Status", e.g. network load, ram etc.
Are there plans to activate them again?
____________
|
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
wackspace.
can you clarify this for me? ;-)
I know you like aws but rackspace sound like a good alternative - also because they are covering at least a fraction of the bill ...
An insiders joke.
Has nothing to do with AWS. Or hosting differences.
pay no heed.
noted ;-)
A total different question for Mike and/or Jim:
On the old server there where some interesting additional statistics shown under "Server Status", e.g. network load, ram etc.
Are there plans to activate them again?
Those were a function of the hosting provider, so those exact charts can't just be "turned on" here.
We're looking into alternatives, but it's a matter of priorities. This doesn't appear to fall into the "low hanging fruit" category, so it's a question of what else doesn't get done in order to get that working.
____________
My lucky number is 75898524288+1 |
|
|
ich_eben*Volunteer tester
Send message
Joined: 14 Oct 10
Posts: 101
ID: 70063
Credit: 366,065,128
RAC: 101,586
|
noted ;-)
A total different question for Mike and/or Jim:
On the old server there where some interesting additional statistics shown under "Server Status", e.g. network load, ram etc.
Are there plans to activate them again?
Those were a function of the hosting provider, so those exact charts can't just be "turned on" here.
We're looking into alternatives, but it's a matter of priorities. This doesn't appear to fall into the "low hanging fruit" category, so it's a question of what else doesn't get done in order to get that working.
ok - if you have enough free time look at "munin".
its a nice tool to generate graphs with rrdtool - it will generate similiar graphs like your previous hoster did, which you can then include in the html file.
one example i know is torservers.net:
https://www.torservers.net/munin/
____________
|
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
ok - if you have enough free time look at "munin".
its a nice tool to generate graphs with rrdtool - it will generate similiar graphs like your previous hoster did, which you can then include in the html file.
one example i know is torservers.net:
https://www.torservers.net/munin/
Munin was actually referenced somewhere in Rackspace's documentation. Your recommendation is noted. Thanks.
____________
My lucky number is 75898524288+1 |
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
|
I've got munin running on a test server. I don't think it has enough charts. ;)
Seriously though, I like that package and will probably install it on both servers (boinc and prpnet).
____________
My lucky number is 75898524288+1 |
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
|
Charts are back on the server status page, and, as an added bonus, it shows charts for both the BOINC and PRPNet servers.
Some of the disk charts aren't available for the PRPNet server because that data isn't available from the VM on that system. The Apache statistics are not included because PRPNet serves up its own web pages directly without using Apache.
____________
My lucky number is 75898524288+1 |
|
|
ich_eben*Volunteer tester
Send message
Joined: 14 Oct 10
Posts: 101
ID: 70063
Credit: 366,065,128
RAC: 101,586
|
|
its really impressive how fast you handle such minor requests mike.
thanks a lot
____________
|
|
|
RogerVolunteer developer
Volunteer tester
Send message
Joined: 27 Nov 11
Posts: 1138
ID: 120786
Credit: 268,621,444
RAC: 0
           
|
Charts are back on the server status page, and, as an added bonus, it shows charts for both the BOINC and PRPNet servers.
What about this one?
http://primegrid.com/charts.php
Big thanks to rackspace! |
|
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13804
ID: 53948
Credit: 345,369,032
RAC: 2,648
|
What about this one?
Gone forever.
____________
My lucky number is 75898524288+1 |
|
|
|
|
|
sorry, I'm a little late to the party but would like to add my thanks to Mike and Jim for their work on their project and Rackspace for their extremely generous donation. |
|
|
|
|
|
I've been away for just on two weeks and am pleasantly surprised to find the whole system migrated, apparently without any significant disruption to the user community.
Well done to all concerned, especially Jim, Michael and the generosity of Rackspace.
____________
Warped
  |
|
|
