Author |
Message |
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Beginning October 12th, one week from today, access to the statistics files exported by PrimeGrid will be restricted to authorised statistics sites and other authorised users.
We're still working on the final plan, but if you are currently using our statistics files, please contact me either on the forums, by private message, or at the email address listed on our contacts page.
Michael Goetz
____________
My lucky number is 75898524288+1 |
|
|
|
What URLs does this affect? Does this include the "subproject status" links and such at the bottom of the left panel of this page, or are these different files hidden elsewhere?
____________
1 PPSE (+2 DC) & 5 SGS primes |
|
|
|
I am occasionally downloading the statistics files (the nice XMLs) for my private stats engine.
Would like to be able to continue to download these
____________
Member of the Dutch Power Cows
My Stats |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
What URLs does this affect? Does this include the "subproject status" links and such at the bottom of the left panel of this page, or are these different files hidden elsewhere?
This affects NO web pages. There are several bulk XML files, all of which are gzipped and end in a .gz extension, that are used by stats websites. Access to those files is being restricted.
The subproject statistics and any page that is linked to is not affected.
If you're viewing the content in your web browser, it's not affected. If you need to download and unzip an XML file, you'll need to talk to me and justify why you need to access information on all of our users.
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
I am occasionally downloading the statistics files (the nice XMLs) for my private stats engine.
Would like to be able to continue to download these
I'm going to send you a PM to discuss this in private.
____________
My lucky number is 75898524288+1 |
|
|
|
Why is this being done? Just curious. |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Why is this being done? Just curious.
Several reasons, but the immediate trigger was that someone was unintentionally downloading thousands of copies of the stats file at once, effectively performing a DDOS attack on the web server.
I won't mention them by name, but even if you figure out who it was, don't hate on them. It wasn't intentional, and they were very helpful in resolving the problem.
But it did bring to light that we want to control access to that file, both because of bandwidth and also because of privacy concerns.
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Some of you may find this useful.
One person had been downloading the entire user_work.gz file just so he could read his own sub-project stats.
I've made that easier now. If you want an XML version of your subproject stats, use this URL:
https://www.primegrid.com/home.php?format=xml
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Another thing that's been confusing people.
Only the big stats files in the /stats/ and /stats_work/ directories are affected.
If you're not downloading a massive gzip file, or your the URL you're using doesn't start with https://www.primegrid.com/stats/ or https://www.primegrid.com/stats_work/ then you're not affected by this change.
If you're running a stats site with a webserver for the whole world, you're probably using the big stats files.
If you're running something like gridcoin or charity engine, you're probably using the big stats files.
If you're getting information only for yourself, you're probably not using the big stats files.
If you're getting information for your team, you're probably not using the big stats files.
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Some of you may find this useful.
One person had been downloading the entire user_work.gz file just so he could read his own sub-project stats.
I've made that easier now. If you want an XML version of your subproject stats, use this URL:
https://www.primegrid.com/home.php?format=xml
This now includes special badge information as well.
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
I've removed the need to use your account key. This url will now return your subproject and badge information in XML format:
https://www.primegrid.com/home.php?format=xml
(Earlier posts saying that an account key was needed have been corrected to avoid confusion.)
____________
My lucky number is 75898524288+1 |
|
|
|
I've removed the need to use your account key. This url will now return your subproject and badge information in XML format:
https://www.primegrid.com/home.php?format=xml
(Earlier posts saying that an account key was needed have been corrected to avoid confusion.)
Nice !
____________
"Accidit in puncto, quod non contingit in anno."
Something that does not occur in a year may, perchance, happen in a moment. |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
I've removed the need to use your account key. This url will now return your subproject and badge information in XML format:
https://www.primegrid.com/home.php?format=xml
(Earlier posts saying that an account key was needed have been corrected to avoid confusion.)
After a confusing couple of days, this has changed once again. This is how it works now:
home.php?format=xml
If you're logged in, this will show the XML stuff. If you're not logged in, you see an XML-style error.
home.php?format=xml&weak=<your-weak-key-here>
If you're not logged in -- and that includes if you're using a script or a command line instead of a browser -- as long as you supplied a valid weak key -- will show you the XML stuff.
Note that if you don't use format=xml the weak= parameter is ignored.
Also note that if you use this in a script (the intended usage) if you change your password, your weak key also changes and you'll need to update your script.
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
As of right now, only authorized users can access the stats and stats_work folders.
____________
My lucky number is 75898524288+1 |
|
|
|
I don't use them, but I'm curious.
How often are these xml files updated ?
____________
"Accidit in puncto, quod non contingit in anno."
Something that does not occur in a year may, perchance, happen in a moment. |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
I don't use them, but I'm curious.
How often are these xml files updated ?
They're produced by different processes, so they're not all on the same schedule. Most are generated 6 times per day. Some are done once an hour.
____________
My lucky number is 75898524288+1 |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
I'm guessing some people might be having trouble with the passwords because there are symbols in there. If you're using something like wget, there's two things you need to be aware of:
1) Although it's not the most secure method you can put the username and password directly in the URL as such:
wget https://username:password@www.primegrid.com/stats/
You can also use the --http-user=user and --http-password=password parameters. This isn't any more secure than putting it in the URL, however. (Both would be visible via ps, for example.)
More secure would be to put it in .wgetrc or .netrc. You would use http-user=user and http-password=password.
2) Bash is probably going to mangle the symbols in the password, so replace them with their hex equivalent:
Assume your username is user and your password is !password!.
"!" is hex 21, so !password! becomes %21password%21
wget https://user:%21password%21@www.primegrid.com/stats/
____________
My lucky number is 75898524288+1 |
|
|
|
Thanks for that info!!
Helps me a lot! :D
____________
|
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Thanks for that info!!
Helps me a lot! :D
You're welcome. I'm watching tha apache log and many people are having trouble. It's a a bit of a nuisance using passwords in scripts if you've never done it before. Once you know how to do it it's easy, of course. :)
If it makes you feel any better, I had a heck of a time testing the XML I added to the private user account page.
It took me forever to figure out why home.php?format=xml&weak=<key> was working in the browser but not with wget. I neglected to change the & to \&.
____________
My lucky number is 75898524288+1 |
|
|
|
Thats what I want to do for the personal userstats, where i use @file_get_contents.
but the & will get replaces with & and so the home.php xml file could not be fetched....
____________
|
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Just a note for anyone using home.php?format=xml from a script:
I messed up the installation of the code on www, so it wasn't working before. It is now.
Also, in my instructions yesterday I said to change the "&" to "%26". That's incorrect. Change the "&" to "\&".
____________
My lucky number is 75898524288+1 |
|
|
Dad Send message
Joined: 28 Feb 18 Posts: 284 ID: 984171 Credit: 182,080,291 RAC: 0
                 
|
Hi Michael,
Have you noticed a drop off in the Gridcoin team stats since the restricted access went in?
Gridcoin is no longer accepting PrimeGrid as a 'whitelisted' project
Dad
____________
Tonight's lucky numbers are
555*2^3563328+1 (PPS-MEGA)
and
58523466^131072+1 (GFN-17 MEGA) |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Hi Michael,
Have you noticed a drop off in the Gridcoin team stats since the restricted access went in?
Gridcoin is no longer accepting PrimeGrid as a 'whitelisted' project
Dad
I don't track team or user activity as a function of time, so I don't know. What I do know is that there's no noticeable difference in overall activity.
The Gridcoin devs are working hard to fix the problem. It's Gridcoin that caused this problem; locking down the stats was necessary to protect PrimeGrid. The ball's in their court to fix their architecture.
I just realized I've told the Gridcoin community much more about what's going on than I did the PrimeGrid community. Here's the scoop on what happened:
In early September I noticed that PrimeGrid was virtually unreachable for a few minutes. It turns out there was a DDOS attack against us, but it was a short lived attack. It only lasted a few minutes. Some investigation determined that this had been happening about once a day for a while, but since it was short in duration nobody really noticed.
By now you probably have guessed where this story is going.
Gridcoin needs our statistics, and collects them by having a large number of host computers each collect our stats files. At pretty much the exact same time. And these are really large files. While we have a fairly hefty server, and a decent pipe to the Internet, Gridcoin *is* a significant player at PrimeGrid, and all those hosts (over a thousand) downloading those large files at the same time was completely saturating our Internet connection.
PrimeGrid isn't the largest BOINC project, but we're up there, and our servers are more powerful than most. I'm not saying that to brag. The point is that if Gridcoin can effectively, albeit unintentionally, mount a successful DDOS attack against PrimeGrid's servers, imagine what Gridcoin is doing to all the other BOINC sites with much smaller hardware. Their only saving grace is that their stats files are almost certainly smaller than ours, and they may have fewer Gridcoiners (is that a word?) on their system. Nevertheless, this method of gathering the statistics is undoubtedly affecting many BOINC sites adversely. What is certain is that it has been making PrimeGrid's server unreachable for periods of time. That's unacceptable. Except for the fact that this is clearly unintentional, it would literally be criminal.
Back in September I reached out to Gridcoin. There's a fix in the pipeline. Things will hopefully be back to normal soon, minus the DDOS part.
We're not the bad guys here. Nor is Gridcoin. Everyone is working together to get this working.
____________
My lucky number is 75898524288+1 |
|
|
Dad Send message
Joined: 28 Feb 18 Posts: 284 ID: 984171 Credit: 182,080,291 RAC: 0
                 
|
Hi Michael,
Any news on how this is travelling with the Gridcoin team?
Thanx
____________
Tonight's lucky numbers are
555*2^3563328+1 (PPS-MEGA)
and
58523466^131072+1 (GFN-17 MEGA) |
|
|
Dad Send message
Joined: 28 Feb 18 Posts: 284 ID: 984171 Credit: 182,080,291 RAC: 0
                 
|
No need, the Gridcoin team replied to me to say by end of next week they hope to have finished testing a new DLL
____________
Tonight's lucky numbers are
555*2^3563328+1 (PPS-MEGA)
and
58523466^131072+1 (GFN-17 MEGA) |
|
|
Michael Goetz Volunteer moderator Project administrator
 Send message
Joined: 21 Jan 10 Posts: 13871 ID: 53948 Credit: 382,930,516 RAC: 110,294
                              
|
Hi Michael,
Any news on how this is travelling with the Gridcoin team?
Thanx
Just to be clear, everything that needs to be done has to be done on the Gridcoin side.
I can tell you that I see in the logs that Gridcoin's servers have been retrieving the stats successfully for at least several days (possibly a lot longer than that).
No need, the Gridcoin team replied to me to say by end of next week they hope to have finished testing a new DLL
"It should be done by the end of the week" is pretty much what I've heard from them every time I've spoken to them, going back to mid-September. I wouldn't hold my breath if I were you. I've stopped asking. Pretty much everyone else who uses the stats had it working in a day.
____________
My lucky number is 75898524288+1 |
|
|
Dad Send message
Joined: 28 Feb 18 Posts: 284 ID: 984171 Credit: 182,080,291 RAC: 0
                 
|
:(
____________
Tonight's lucky numbers are
555*2^3563328+1 (PPS-MEGA)
and
58523466^131072+1 (GFN-17 MEGA) |
|
|