PrimeGrid
Please visit donation page to help the project cover running costs for this month

Toggle Menu

Join PrimeGrid

Returning Participants

Community

Leader Boards

Results

Other

drummers-lowrise

Advanced search

Message boards : Number crunching : Benchmarking or post-mortem analysis?

Author Message
Profile composite
Volunteer tester
Send message
Joined: 16 Feb 10
Posts: 733
ID: 55391
Credit: 572,922,564
RAC: 285,936
Discovered 2 mega primesFound 1 prime in the 2018 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (5,477,467)Cullen LLR Gold: Earned 500,000 credits (776,297)ESP LLR Ruby: Earned 2,000,000 credits (3,120,351)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,056,207)PPS LLR Jade: Earned 10,000,000 credits (12,244,808)PSP LLR Turquoise: Earned 5,000,000 credits (5,000,271)SoB LLR Sapphire: Earned 20,000,000 credits (20,233,229)SR5 LLR Turquoise: Earned 5,000,000 credits (5,978,312)SGS LLR Ruby: Earned 2,000,000 credits (2,940,587)TRP LLR Turquoise: Earned 5,000,000 credits (7,025,303)Woodall LLR Amethyst: Earned 1,000,000 credits (1,693,614)321 Sieve Jade: Earned 10,000,000 credits (18,635,342)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,571,178)Generalized Cullen/Woodall Sieve (suspended) Emerald: Earned 50,000,000 credits (50,009,610)PPS Sieve Double Silver: Earned 200,000,000 credits (243,063,205)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Jade: Earned 10,000,000 credits (10,165,888)TRP Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,071,454)AP 26/27 Turquoise: Earned 5,000,000 credits (6,037,979)GFN Emerald: Earned 50,000,000 credits (50,062,736)PSA Double Bronze: Earned 100,000,000 credits (102,762,384)
Message 113838 - Posted: 27 Jan 2018 | 16:29:44 UTC

Which is better? (convenient, productive, less tedious)

a) Run benchmarks of various task configurations (multithreading, etc) at the FFT sizes being handed out by PG, or

b) Run a bunch of PG work in various configurations over time and do a post-mortem analysis of the tasks.

Either way, you need to know what you are doing, so this is mostly for PG diehards.

Benchmarking might be a better way to go, considering the long running times involved for many projects, and the fact that option (b) is useful only where CPU time has already been committed to PrimeGrid. Option(b) may be more attractive to people having a variety of computer hardware.

Option (b) doesn't save any configuration work, it just spreads it out over time. At least, some of the bookkeeping is taken care of by the PG tasks list, and no CPU time is wasted doing unproductive computation. You could argue however, that running a CPU inefficiently for an extended period of time is unproductive, and there is extra work to determine how a computer was running given an inconsistent stream of work.

For option (b):
It takes 9 spreadsheet functions in multiple columns coupled with miscellaneous arithmetic to parse each timestamp in the 'sent' and 'reported' columns of the task list. I just wish timestamps in the tasks list could be switched on demand to ISO 8601 format. (Even better, have a data download button for all completed tasks, sort of like a monthly banking statement download option.) And then another formula to compute how much overlap there is between a set of tasks to see how many tasks were running concurrently. Then plot a scatter graph with credit on the X axis and run-time on the Y axis, so that each configuration shows up as a collection of points on a distinct line. Credit is the only reliable way of determining how much work was done for a task. Equivalently to graphing, perform a linear regression on groups of data. The constant and slope are what matter.

Furthermore, the post-mortem method requires that you manually scan the stderr logs to determine what implementation was used (FMA3, etc), the FFT size, the number of threads used, and whether the task was completed consistently and without interruption due to errors or changed configuration, to be able to classify groups of tasks into configurations and reject outliers.

My alternative is to write a web page scraper, but whatever I do on my computer won't be available to others who may also be tuning their PG workloads. With enough motivation I could put this on a cheap VPS so you just supply your PG username and hit the download button if your computers are not hidden, but if it gets swamped with CPU work the service provider will shut 'er down. I see some value in aggregating data for a wide variety of configurations, and that might be motivation enough!

Profile Michael GoetzProject donor
Volunteer moderator
Project administrator
Project scientist
Avatar
Send message
Joined: 21 Jan 10
Posts: 13036
ID: 53948
Credit: 202,626,665
RAC: 115,417
The "Shut up already!" badge:  This loud mouth has mansplained on the forums over 10 thousand times!  Sheesh!!!Discovered the World's First GFN-19 prime!!!Discovered 1 mega primeFound 1 prime in the 2018 Tour de PrimesFound 1 prime in the 2019 Tour de Primes321 LLR Ruby: Earned 2,000,000 credits (2,822,730)Cullen LLR Ruby: Earned 2,000,000 credits (2,005,249)ESP LLR Turquoise: Earned 5,000,000 credits (5,009,577)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,145,754)PPS LLR Turquoise: Earned 5,000,000 credits (8,939,736)PSP LLR Turquoise: Earned 5,000,000 credits (5,065,592)SoB LLR Sapphire: Earned 20,000,000 credits (34,221,148)SR5 LLR Turquoise: Earned 5,000,000 credits (8,293,415)SGS LLR Ruby: Earned 2,000,000 credits (2,014,138)TRP LLR Ruby: Earned 2,000,000 credits (2,737,347)Woodall LLR Ruby: Earned 2,000,000 credits (2,195,123)321 Sieve Turquoise: Earned 5,000,000 credits (5,439,488)Cullen/Woodall Sieve (suspended) Ruby: Earned 2,000,000 credits (4,170,256)Generalized Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,059,304)PPS Sieve Sapphire: Earned 20,000,000 credits (20,110,788)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Amethyst: Earned 1,000,000 credits (1,035,522)TRP Sieve (suspended) Ruby: Earned 2,000,000 credits (2,051,121)AP 26/27 Jade: Earned 10,000,000 credits (10,114,260)GFN Emerald: Earned 50,000,000 credits (66,792,341)PSA Jade: Earned 10,000,000 credits (12,404,447)
Message 113839 - Posted: 27 Jan 2018 | 17:04:58 UTC
Last modified: 27 Jan 2018 | 17:31:37 UTC

I always benchmark. In fact, I spent a good chunk of yesterday benchmarking 9 different configurations of running PPSE on 5 different computers to see what actually worked best.

I never trust live tasks. There's too many variables. Not only may the tasks be different, but your computer may be operating under a different environment.

When I *REALLY* want good benchmarks, for example when I'm doing the LLR benchmarks used to set PrimeGrid's credit system, I go a step further and put the computer in safe mode to reduce the outside influences on the benchmarks.
____________
Please do not PM me with support questions. Ask on the forums instead. Thank you!

My lucky number is 75898524288+1

Profile composite
Volunteer tester
Send message
Joined: 16 Feb 10
Posts: 733
ID: 55391
Credit: 572,922,564
RAC: 285,936
Discovered 2 mega primesFound 1 prime in the 2018 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (5,477,467)Cullen LLR Gold: Earned 500,000 credits (776,297)ESP LLR Ruby: Earned 2,000,000 credits (3,120,351)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,056,207)PPS LLR Jade: Earned 10,000,000 credits (12,244,808)PSP LLR Turquoise: Earned 5,000,000 credits (5,000,271)SoB LLR Sapphire: Earned 20,000,000 credits (20,233,229)SR5 LLR Turquoise: Earned 5,000,000 credits (5,978,312)SGS LLR Ruby: Earned 2,000,000 credits (2,940,587)TRP LLR Turquoise: Earned 5,000,000 credits (7,025,303)Woodall LLR Amethyst: Earned 1,000,000 credits (1,693,614)321 Sieve Jade: Earned 10,000,000 credits (18,635,342)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,571,178)Generalized Cullen/Woodall Sieve (suspended) Emerald: Earned 50,000,000 credits (50,009,610)PPS Sieve Double Silver: Earned 200,000,000 credits (243,063,205)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Jade: Earned 10,000,000 credits (10,165,888)TRP Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,071,454)AP 26/27 Turquoise: Earned 5,000,000 credits (6,037,979)GFN Emerald: Earned 50,000,000 credits (50,062,736)PSA Double Bronze: Earned 100,000,000 credits (102,762,384)
Message 113840 - Posted: 27 Jan 2018 | 17:21:41 UTC

OK, benchmarking for standards, where you don't want results to be biased.

How about for the real world in which credit is generated? Is a benchmark a fair representation of what task configuration will produce the maximum credit in a given amount of time?

Holding other variables constant, the credit value of a task compensates very well for small variability in the size of the task. But a change of FFT size produces a step in the amount of work. Do you calibrate the credit system for different sizes of FFT? Does the FFT size key into a table of constants for credit score calculation or do you run a regression line through all the benchmarks for a given subproject to come up with a single credit score formula?

Profile Michael GoetzProject donor
Volunteer moderator
Project administrator
Project scientist
Avatar
Send message
Joined: 21 Jan 10
Posts: 13036
ID: 53948
Credit: 202,626,665
RAC: 115,417
The "Shut up already!" badge:  This loud mouth has mansplained on the forums over 10 thousand times!  Sheesh!!!Discovered the World's First GFN-19 prime!!!Discovered 1 mega primeFound 1 prime in the 2018 Tour de PrimesFound 1 prime in the 2019 Tour de Primes321 LLR Ruby: Earned 2,000,000 credits (2,822,730)Cullen LLR Ruby: Earned 2,000,000 credits (2,005,249)ESP LLR Turquoise: Earned 5,000,000 credits (5,009,577)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,145,754)PPS LLR Turquoise: Earned 5,000,000 credits (8,939,736)PSP LLR Turquoise: Earned 5,000,000 credits (5,065,592)SoB LLR Sapphire: Earned 20,000,000 credits (34,221,148)SR5 LLR Turquoise: Earned 5,000,000 credits (8,293,415)SGS LLR Ruby: Earned 2,000,000 credits (2,014,138)TRP LLR Ruby: Earned 2,000,000 credits (2,737,347)Woodall LLR Ruby: Earned 2,000,000 credits (2,195,123)321 Sieve Turquoise: Earned 5,000,000 credits (5,439,488)Cullen/Woodall Sieve (suspended) Ruby: Earned 2,000,000 credits (4,170,256)Generalized Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,059,304)PPS Sieve Sapphire: Earned 20,000,000 credits (20,110,788)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Amethyst: Earned 1,000,000 credits (1,035,522)TRP Sieve (suspended) Ruby: Earned 2,000,000 credits (2,051,121)AP 26/27 Jade: Earned 10,000,000 credits (10,114,260)GFN Emerald: Earned 50,000,000 credits (66,792,341)PSA Jade: Earned 10,000,000 credits (12,404,447)
Message 113842 - Posted: 27 Jan 2018 | 17:31:16 UTC - in response to Message 113840.

How about for the real world in which credit is generated? Is a benchmark a fair representation of what task configuration will produce the maximum credit in a given amount of time?


At PrimeGrid, yes. Elsewhere, it's a mixed bag because the standard BOINC credit system has significant limitations.

The assumption, of course, is that when you're doing benchmarks you're taking the steps necessary to accurately replicate that actual environment you'll be running. That means running both the GPU and CPU as you would when you're running BOINC. If you use TThrottle, not only should you use it during the benchmarks but you also need to set set it to control non-BOINC tasks, and you need to run benchmarks at different ambient temperatures.
____________
Please do not PM me with support questions. Ask on the forums instead. Thank you!

My lucky number is 75898524288+1

mackerelProject donor
Volunteer tester
Avatar
Send message
Joined: 2 Oct 08
Posts: 2242
ID: 29980
Credit: 350,165,749
RAC: 376,931
Discovered 2 mega primesEliminated 1 conjecture "k"Found 3 primes in the 2018 Tour de PrimesFound 1 mega prime in the 2018 Tour de PrimesFound 5 primes in the 2019 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (8,774,878)Cullen LLR Turquoise: Earned 5,000,000 credits (5,149,818)ESP LLR Turquoise: Earned 5,000,000 credits (6,454,573)Generalized Cullen/Woodall LLR Turquoise: Earned 5,000,000 credits (5,122,074)PPS LLR Emerald: Earned 50,000,000 credits (57,553,368)PSP LLR Jade: Earned 10,000,000 credits (14,910,335)SoB LLR Jade: Earned 10,000,000 credits (17,319,914)SR5 LLR Jade: Earned 10,000,000 credits (18,479,395)SGS LLR Turquoise: Earned 5,000,000 credits (6,702,766)TPS LLR (retired) Bronze: Earned 10,000 credits (34,130)TRP LLR Jade: Earned 10,000,000 credits (15,154,737)Woodall LLR Turquoise: Earned 5,000,000 credits (5,715,464)321 Sieve Turquoise: Earned 5,000,000 credits (8,383,104)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,383,853)Generalized Cullen/Woodall Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,626,419)PPS Sieve Emerald: Earned 50,000,000 credits (74,140,875)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Ruby: Earned 2,000,000 credits (2,293,882)TRP Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,012,757)AP 26/27 Sapphire: Earned 20,000,000 credits (21,918,894)GFN Sapphire: Earned 20,000,000 credits (48,107,585)PSA Ruby: Earned 2,000,000 credits (2,939,755)
Message 113845 - Posted: 27 Jan 2018 | 18:15:04 UTC - in response to Message 113838.

I've done both generic benchmarks and live unit ones.


To do a generic benchmark, I use the benchmark functionality in Prime95. I ramp up FFT size from 64k to 8192k, although you can go to a slightly smaller range if you limit it to current PrimeGrid tasks. Vary the number of cores optionally. One per core or multi-thread. Results need normalising if you want to compare across FFT sizes. Using FFT size and number of tasks, you can work out effective ram working set too. Regardless of settings, the ultra-short version is: everything fits within CPU cache = fast, otherwise ram performance will affect results. Faster ram generally better.

When age of multi-thread came in, the optimal point seems to be when the tasks substantially fill the CPU L3 cache without exceeding it. Smaller tasks have trouble scaling. Bigger tasks, ram limit again, but still worth it to have on fast task than many slow ones.

Above alone gives a very rough rule of thumb if a planned workload will be ram limited or not. How limited is another question... as a starting point, say you have a modern quad core Intel CPU running at X MHz, matched with dual channel ram at Y MT/s. If Y is comparable to X, that's great and you're close to practically "ram unlimited". Y around 75% of X is quite good, but you will notice it scale more with ram much below that. For most people, ram will be more of a limiting factor than CPU clock. You can also imagine, for higher core counts the ram bandwidth shortage can get more severe unless you move to quad channels or more.

Note the above looks at relative performance only, I haven't found a way to convert that into a time estimate, particularly as I understand the FFT type used in LLR differs form that in P95. The more recent bench screen in P95 even gives an option to change FFT type, but I've not investigated its use.


On using live tasks, as much as Michael repeatedly warns against it, I think it can still be used with caution. At a minimum, for projects with multiple FFT sizes, you need to sort them. For me, that is a manual process. I do a lot of copy and paste of the results, as well as looking up interesting task FFT sizes. Not something I like to do often! In essence, if you average enough units, you will see a trend. In the recent SoB challenge, in one of the FFT sizes, I observed obvious grouping. Probably depending on the k tested which is unknown to me. You could sub-group it into a "slow" and "fast" group in that scenario, but it is extra work again.


For me, the battle is not what to do with the data, but getting it in a usable format the first place.

Profile Michael GoetzProject donor
Volunteer moderator
Project administrator
Project scientist
Avatar
Send message
Joined: 21 Jan 10
Posts: 13036
ID: 53948
Credit: 202,626,665
RAC: 115,417
The "Shut up already!" badge:  This loud mouth has mansplained on the forums over 10 thousand times!  Sheesh!!!Discovered the World's First GFN-19 prime!!!Discovered 1 mega primeFound 1 prime in the 2018 Tour de PrimesFound 1 prime in the 2019 Tour de Primes321 LLR Ruby: Earned 2,000,000 credits (2,822,730)Cullen LLR Ruby: Earned 2,000,000 credits (2,005,249)ESP LLR Turquoise: Earned 5,000,000 credits (5,009,577)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,145,754)PPS LLR Turquoise: Earned 5,000,000 credits (8,939,736)PSP LLR Turquoise: Earned 5,000,000 credits (5,065,592)SoB LLR Sapphire: Earned 20,000,000 credits (34,221,148)SR5 LLR Turquoise: Earned 5,000,000 credits (8,293,415)SGS LLR Ruby: Earned 2,000,000 credits (2,014,138)TRP LLR Ruby: Earned 2,000,000 credits (2,737,347)Woodall LLR Ruby: Earned 2,000,000 credits (2,195,123)321 Sieve Turquoise: Earned 5,000,000 credits (5,439,488)Cullen/Woodall Sieve (suspended) Ruby: Earned 2,000,000 credits (4,170,256)Generalized Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,059,304)PPS Sieve Sapphire: Earned 20,000,000 credits (20,110,788)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Amethyst: Earned 1,000,000 credits (1,035,522)TRP Sieve (suspended) Ruby: Earned 2,000,000 credits (2,051,121)AP 26/27 Jade: Earned 10,000,000 credits (10,114,260)GFN Emerald: Earned 50,000,000 credits (66,792,341)PSA Jade: Earned 10,000,000 credits (12,404,447)
Message 113846 - Posted: 27 Jan 2018 | 18:32:41 UTC - in response to Message 113845.

At a minimum, for projects with multiple FFT sizes, you need to sort them.


FFT-size alone is not enough. LLR will also run different types of FFTs, e.g., zero-padded, all-complex, etc. Those affect execution speed as well. You need to separate those out too.

If all I cared about was "how fast is it?", that wouldn't be that significant. But if I'm deciding on which strategy to use, this could cause me to choose incorrectly.

It's one of the many reasons I do benchmarks with a single test number.
____________
Please do not PM me with support questions. Ask on the forums instead. Thank you!

My lucky number is 75898524288+1

mackerelProject donor
Volunteer tester
Avatar
Send message
Joined: 2 Oct 08
Posts: 2242
ID: 29980
Credit: 350,165,749
RAC: 376,931
Discovered 2 mega primesEliminated 1 conjecture "k"Found 3 primes in the 2018 Tour de PrimesFound 1 mega prime in the 2018 Tour de PrimesFound 5 primes in the 2019 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (8,774,878)Cullen LLR Turquoise: Earned 5,000,000 credits (5,149,818)ESP LLR Turquoise: Earned 5,000,000 credits (6,454,573)Generalized Cullen/Woodall LLR Turquoise: Earned 5,000,000 credits (5,122,074)PPS LLR Emerald: Earned 50,000,000 credits (57,553,368)PSP LLR Jade: Earned 10,000,000 credits (14,910,335)SoB LLR Jade: Earned 10,000,000 credits (17,319,914)SR5 LLR Jade: Earned 10,000,000 credits (18,479,395)SGS LLR Turquoise: Earned 5,000,000 credits (6,702,766)TPS LLR (retired) Bronze: Earned 10,000 credits (34,130)TRP LLR Jade: Earned 10,000,000 credits (15,154,737)Woodall LLR Turquoise: Earned 5,000,000 credits (5,715,464)321 Sieve Turquoise: Earned 5,000,000 credits (8,383,104)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,383,853)Generalized Cullen/Woodall Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,626,419)PPS Sieve Emerald: Earned 50,000,000 credits (74,140,875)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Ruby: Earned 2,000,000 credits (2,293,882)TRP Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,012,757)AP 26/27 Sapphire: Earned 20,000,000 credits (21,918,894)GFN Sapphire: Earned 20,000,000 credits (48,107,585)PSA Ruby: Earned 2,000,000 credits (2,939,755)
Message 113850 - Posted: 27 Jan 2018 | 19:41:10 UTC - in response to Message 113846.

FFT-size alone is not enough. LLR will also run different types of FFTs, e.g., zero-padded, all-complex, etc. Those affect execution speed as well. You need to separate those out too.

It depends on the level of optimisation. If I was publishing a peer reviewed paper, sure. I'm not. I just want to know, in like for like situations, how fast is something compared to something else. Most of the time FFT size alone is a good enough approximation. The example where I saw within one FFT size two groupings, that was enough for me to either take that into consideration, or use something else. I did the latter, and picked a different FFT size which did not have that variation. Again, with big enough numbers to average it out, it is "good enough".

I find manually testing fixed numbers actually harder than to do the above, especially when multiple units are involved and you have to be careful about loading and timing. Plus unless you run multiple times to check it, you can't know if there might be an external factor affecting it. With the real work method, it is pretty much set and forget until analysis. Outliers are easily spotted.

If it were easily extracted, I could try to make use of more information when looking at real work. But it either isn't accessible to ordinary users, or rather fiddly and manual to do so, so I have to look for enough.

Profile composite
Volunteer tester
Send message
Joined: 16 Feb 10
Posts: 733
ID: 55391
Credit: 572,922,564
RAC: 285,936
Discovered 2 mega primesFound 1 prime in the 2018 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (5,477,467)Cullen LLR Gold: Earned 500,000 credits (776,297)ESP LLR Ruby: Earned 2,000,000 credits (3,120,351)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,056,207)PPS LLR Jade: Earned 10,000,000 credits (12,244,808)PSP LLR Turquoise: Earned 5,000,000 credits (5,000,271)SoB LLR Sapphire: Earned 20,000,000 credits (20,233,229)SR5 LLR Turquoise: Earned 5,000,000 credits (5,978,312)SGS LLR Ruby: Earned 2,000,000 credits (2,940,587)TRP LLR Turquoise: Earned 5,000,000 credits (7,025,303)Woodall LLR Amethyst: Earned 1,000,000 credits (1,693,614)321 Sieve Jade: Earned 10,000,000 credits (18,635,342)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,571,178)Generalized Cullen/Woodall Sieve (suspended) Emerald: Earned 50,000,000 credits (50,009,610)PPS Sieve Double Silver: Earned 200,000,000 credits (243,063,205)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Jade: Earned 10,000,000 credits (10,165,888)TRP Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,071,454)AP 26/27 Turquoise: Earned 5,000,000 credits (6,037,979)GFN Emerald: Earned 50,000,000 credits (50,062,736)PSA Double Bronze: Earned 100,000,000 credits (102,762,384)
Message 113852 - Posted: 27 Jan 2018 | 19:54:09 UTC - in response to Message 113850.

I've had enough of copying and pasting HTML tables into spreadsheets.
I'm creating a scraper in Perl and I'll run it on a VPS so you can try it out too.

Profile Michael GoetzProject donor
Volunteer moderator
Project administrator
Project scientist
Avatar
Send message
Joined: 21 Jan 10
Posts: 13036
ID: 53948
Credit: 202,626,665
RAC: 115,417
The "Shut up already!" badge:  This loud mouth has mansplained on the forums over 10 thousand times!  Sheesh!!!Discovered the World's First GFN-19 prime!!!Discovered 1 mega primeFound 1 prime in the 2018 Tour de PrimesFound 1 prime in the 2019 Tour de Primes321 LLR Ruby: Earned 2,000,000 credits (2,822,730)Cullen LLR Ruby: Earned 2,000,000 credits (2,005,249)ESP LLR Turquoise: Earned 5,000,000 credits (5,009,577)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,145,754)PPS LLR Turquoise: Earned 5,000,000 credits (8,939,736)PSP LLR Turquoise: Earned 5,000,000 credits (5,065,592)SoB LLR Sapphire: Earned 20,000,000 credits (34,221,148)SR5 LLR Turquoise: Earned 5,000,000 credits (8,293,415)SGS LLR Ruby: Earned 2,000,000 credits (2,014,138)TRP LLR Ruby: Earned 2,000,000 credits (2,737,347)Woodall LLR Ruby: Earned 2,000,000 credits (2,195,123)321 Sieve Turquoise: Earned 5,000,000 credits (5,439,488)Cullen/Woodall Sieve (suspended) Ruby: Earned 2,000,000 credits (4,170,256)Generalized Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,059,304)PPS Sieve Sapphire: Earned 20,000,000 credits (20,110,788)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Amethyst: Earned 1,000,000 credits (1,035,522)TRP Sieve (suspended) Ruby: Earned 2,000,000 credits (2,051,121)AP 26/27 Jade: Earned 10,000,000 credits (10,114,260)GFN Emerald: Earned 50,000,000 credits (66,792,341)PSA Jade: Earned 10,000,000 credits (12,404,447)
Message 113853 - Posted: 27 Jan 2018 | 20:10:13 UTC - in response to Message 113852.
Last modified: 27 Jan 2018 | 20:11:02 UTC

I've had enough of copying and pasting HTML tables into spreadsheets.
I'm creating a scraper in Perl and I'll run it on a VPS so you can try it out too.


Official Administrator Position:

Scapers are fine as long as it's single threaded, i.e., don't start scraping a second page until you're done scraping the first page. As long as you're single threaded, the scraper won't affect server performance because it's self limiting and won't try to go faster than the server can handle.
____________
Please do not PM me with support questions. Ask on the forums instead. Thank you!

My lucky number is 75898524288+1

Profile composite
Volunteer tester
Send message
Joined: 16 Feb 10
Posts: 733
ID: 55391
Credit: 572,922,564
RAC: 285,936
Discovered 2 mega primesFound 1 prime in the 2018 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (5,477,467)Cullen LLR Gold: Earned 500,000 credits (776,297)ESP LLR Ruby: Earned 2,000,000 credits (3,120,351)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,056,207)PPS LLR Jade: Earned 10,000,000 credits (12,244,808)PSP LLR Turquoise: Earned 5,000,000 credits (5,000,271)SoB LLR Sapphire: Earned 20,000,000 credits (20,233,229)SR5 LLR Turquoise: Earned 5,000,000 credits (5,978,312)SGS LLR Ruby: Earned 2,000,000 credits (2,940,587)TRP LLR Turquoise: Earned 5,000,000 credits (7,025,303)Woodall LLR Amethyst: Earned 1,000,000 credits (1,693,614)321 Sieve Jade: Earned 10,000,000 credits (18,635,342)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,571,178)Generalized Cullen/Woodall Sieve (suspended) Emerald: Earned 50,000,000 credits (50,009,610)PPS Sieve Double Silver: Earned 200,000,000 credits (243,063,205)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Jade: Earned 10,000,000 credits (10,165,888)TRP Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,071,454)AP 26/27 Turquoise: Earned 5,000,000 credits (6,037,979)GFN Emerald: Earned 50,000,000 credits (50,062,736)PSA Double Bronze: Earned 100,000,000 credits (102,762,384)
Message 113869 - Posted: 28 Jan 2018 | 1:25:02 UTC
Last modified: 28 Jan 2018 | 1:25:24 UTC

Well, this code works for post-mortem analysis, but only for users whose computers are not hidden. I'm not going to request PrimeGrid login credentials, so I'm not setting up a web service. It produces a CSV format file on STDOUT.

Output columns

subproject,hostid,resultid,method,fft_length,threads,sent,received,runtime,cputime,credit


The sent and received columns are in a text format that LibreOffice DATEVALUE and TIMEVALUE functions can use, rather than strict ISO 8601. You can't figure out from this data exactly how many tasks are running concurrently, these are just task download and upload times. So it's up to you to control your BOINC task execution to make sense out of the data.

Since I'm only interested in LLR right now, it filters out Genefer, AP Search, and Sieve. I wrote this Perl code and hereby release it to the public domain, without warranty as to function and fitness. Play as desired for your own purposes. Michael has warned you in this thread about acceptable usage.

The code sets the user agent string to 'composite'. Feel free to change that or anything else.

#!/usr/bin/perl -w use strict; use LWP::Simple qw( $ua get ); $ua->timeout(15); $ua->agent('composite'); sub iso8601 { my $primegrid_format = shift; my ($d, $m, $y, undef, $hms, undef) = split / /, $primegrid_format; return "$y-$m-$d $hms"; } sub scrape { my $url = shift; my $html = get $url || die "timed out on $url\n"; return $html; } sub result { my ($hostid, $resultid) = @_; my $html = scrape "https://www.primegrid.com/result.php?resultid=$resultid"; my ($head, $log, $tail) = split(/&lt;[\/]?stderr_txt&gt;/, $html); my @head = split("\n", $head); my @log = split("\n", $log); my @tail = split("\n", $tail); my %data; for my $field (grep {/class=fieldname/} @head) { $field =~ m/fieldname>(.*?)<.*fieldvalue>(.*?)</; next unless $2; $data{$1} = $2; } my $sent = iso8601 $data{"Sent"}; my $received = iso8601 $data{"Received"}; my $runtime = $data{"Run time"}; my $cputime = $data{"CPU time"}; my $credit = $data{"Credit"}; my $subproject = $data{"Application version"}; # remove commas from numbers $runtime =~ s/,//g; $cputime =~ s/,//g; $credit =~ s/,//g; # trim spaces $subproject =~ s/^\s+//; $subproject =~ s/\s+$//; my @command_line = grep {/command line/i} @log; die "oops no command for result $resultid\n" unless scalar(@command_line) > 0; $command_line[0] =~ m/-oThreadsPerTest=(.*?) /; my $threads = $1 || "1"; my @FFT_line = grep {/^Using /} @log; $FFT_line[1] =~ m/^Using (.*?) FFT length (.*?),/; my $method = $1 || "none"; my $fft_length = $2 || "none"; print "\"$subproject\",$hostid,$resultid,\"$method\",$fft_length,$threads,$sent,$received,$runtime,$cputime,$credit\n"; } sub host { my $hostid = shift; my $offset = 0; my @next; do { my $html = scrape "https://www.primegrid.com/results.php?hostid=$hostid&offset=$offset&show_names=0&state=4&appid="; my @lines = split("\n", $html); my @results = grep {/result.php/} @lines; for my $result (@results) { next if $result =~ /Genefer/i; next if $result =~ /Sieve/i; next if $result =~ /Search/i; $result =~ m/resultid=(.*?)"/; die "no result\n" unless $1; result $hostid, $1; } $offset += 20; @next = grep {/Next 20/} @lines; } until (scalar(@next) == 0); } sub user { my $userid = shift; my $html = scrape "https://www.primegrid.com/hosts_user.php?userid=$userid"; my @lines = split("\n", $html); my @hosts = grep {/results.php\?hostid=/} @lines; for my $hostline (@hosts) { $hostline =~ m/hostid=(.*?)>/; die "no host\n" unless $1; host $1; } } sub main { my $userid = shift @ARGV; die "syntax: $0 useridnum\n" unless defined $userid; # untaint argument $userid =~ m/(^\d+$)/; $userid = $1; die "syntax: $0 useridnum\n" unless defined $userid; warn "data for user $userid\n"; print "subproject,hostid,resultid,method,fft_length,threads,sent,received,runtime,cputime,credit\n"; user($userid); } main;

Profile Michael GoetzProject donor
Volunteer moderator
Project administrator
Project scientist
Avatar
Send message
Joined: 21 Jan 10
Posts: 13036
ID: 53948
Credit: 202,626,665
RAC: 115,417
The "Shut up already!" badge:  This loud mouth has mansplained on the forums over 10 thousand times!  Sheesh!!!Discovered the World's First GFN-19 prime!!!Discovered 1 mega primeFound 1 prime in the 2018 Tour de PrimesFound 1 prime in the 2019 Tour de Primes321 LLR Ruby: Earned 2,000,000 credits (2,822,730)Cullen LLR Ruby: Earned 2,000,000 credits (2,005,249)ESP LLR Turquoise: Earned 5,000,000 credits (5,009,577)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,145,754)PPS LLR Turquoise: Earned 5,000,000 credits (8,939,736)PSP LLR Turquoise: Earned 5,000,000 credits (5,065,592)SoB LLR Sapphire: Earned 20,000,000 credits (34,221,148)SR5 LLR Turquoise: Earned 5,000,000 credits (8,293,415)SGS LLR Ruby: Earned 2,000,000 credits (2,014,138)TRP LLR Ruby: Earned 2,000,000 credits (2,737,347)Woodall LLR Ruby: Earned 2,000,000 credits (2,195,123)321 Sieve Turquoise: Earned 5,000,000 credits (5,439,488)Cullen/Woodall Sieve (suspended) Ruby: Earned 2,000,000 credits (4,170,256)Generalized Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,059,304)PPS Sieve Sapphire: Earned 20,000,000 credits (20,110,788)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Amethyst: Earned 1,000,000 credits (1,035,522)TRP Sieve (suspended) Ruby: Earned 2,000,000 credits (2,051,121)AP 26/27 Jade: Earned 10,000,000 credits (10,114,260)GFN Emerald: Earned 50,000,000 credits (66,792,341)PSA Jade: Earned 10,000,000 credits (12,404,447)
Message 114056 - Posted: 1 Feb 2018 | 18:04:15 UTC - in response to Message 113869.

Well, this code works for post-mortem analysis, but only for users whose computers are not hidden. I'm not going to request PrimeGrid login credentials, so I'm not setting up a web service. It produces a CSV format file on STDOUT.


Very nice, thanks!
____________
Please do not PM me with support questions. Ask on the forums instead. Thank you!

My lucky number is 75898524288+1

Profile composite
Volunteer tester
Send message
Joined: 16 Feb 10
Posts: 733
ID: 55391
Credit: 572,922,564
RAC: 285,936
Discovered 2 mega primesFound 1 prime in the 2018 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (5,477,467)Cullen LLR Gold: Earned 500,000 credits (776,297)ESP LLR Ruby: Earned 2,000,000 credits (3,120,351)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,056,207)PPS LLR Jade: Earned 10,000,000 credits (12,244,808)PSP LLR Turquoise: Earned 5,000,000 credits (5,000,271)SoB LLR Sapphire: Earned 20,000,000 credits (20,233,229)SR5 LLR Turquoise: Earned 5,000,000 credits (5,978,312)SGS LLR Ruby: Earned 2,000,000 credits (2,940,587)TRP LLR Turquoise: Earned 5,000,000 credits (7,025,303)Woodall LLR Amethyst: Earned 1,000,000 credits (1,693,614)321 Sieve Jade: Earned 10,000,000 credits (18,635,342)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,571,178)Generalized Cullen/Woodall Sieve (suspended) Emerald: Earned 50,000,000 credits (50,009,610)PPS Sieve Double Silver: Earned 200,000,000 credits (243,063,205)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Jade: Earned 10,000,000 credits (10,165,888)TRP Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,071,454)AP 26/27 Turquoise: Earned 5,000,000 credits (6,037,979)GFN Emerald: Earned 50,000,000 credits (50,062,736)PSA Double Bronze: Earned 100,000,000 credits (102,762,384)
Message 114103 - Posted: 2 Feb 2018 | 1:55:01 UTC - in response to Message 114056.

Well, this code works for post-mortem analysis, but only for users whose computers are not hidden. I'm not going to request PrimeGrid login credentials, so I'm not setting up a web service. It produces a CSV format file on STDOUT.


Very nice, thanks!

That was the first cut. I have a nicer one now that counts the number of programs starts and command line option changes to distinguish outliers from steady-state work.

dukebg
Volunteer tester
Avatar
Send message
Joined: 21 Nov 17
Posts: 235
ID: 950482
Credit: 22,083,013
RAC: 0
Found 1 prime in the 2018 Tour de Primes321 LLR Gold: Earned 500,000 credits (506,942)Cullen LLR Gold: Earned 500,000 credits (500,523)ESP LLR Gold: Earned 500,000 credits (655,642)Generalized Cullen/Woodall LLR Gold: Earned 500,000 credits (539,100)PPS LLR Gold: Earned 500,000 credits (688,232)PSP LLR Gold: Earned 500,000 credits (561,629)SoB LLR Ruby: Earned 2,000,000 credits (3,659,676)SR5 LLR Gold: Earned 500,000 credits (505,810)SGS LLR Gold: Earned 500,000 credits (506,024)TRP LLR Gold: Earned 500,000 credits (963,625)Woodall LLR Gold: Earned 500,000 credits (506,044)321 Sieve Gold: Earned 500,000 credits (511,694)Generalized Cullen/Woodall Sieve (suspended) Gold: Earned 500,000 credits (549,916)PPS Sieve Gold: Earned 500,000 credits (701,168)AP 26/27 Gold: Earned 500,000 credits (525,590)GFN Turquoise: Earned 5,000,000 credits (9,675,760)PSA Gold: Earned 500,000 credits (525,639)
Message 114437 - Posted: 6 Feb 2018 | 16:58:36 UTC

composite,
While discussing tasks purging in Discord recently, I realized that your script, while it scans all the tasks pages until there's no "Next 20" link, will have skewed results near the end.

Tasks are purged on a timer that starts ticking when the last wingman completes their run.* That means that oldest tasks that you still see are most likely the ones that the wingman finished recently, but you long ago. Thus you'll see yourself 1st on the last pages more. The counterparts from those times, where your partner finished around the same time you did (and especially before you) are already purged, skewing the statistics.

The purge timer for PPSE is just 3 days. Don't count any PPSE tasks finished longer than 3 days ago into statistics. 4 days for GFN-16. Source: Mike :]

_____
* - with some exceptions, like tasks belonging to a challenge and tasks resulting in a prime. Those hang around longer.

Profile composite
Volunteer tester
Send message
Joined: 16 Feb 10
Posts: 733
ID: 55391
Credit: 572,922,564
RAC: 285,936
Discovered 2 mega primesFound 1 prime in the 2018 Tour de Primes321 LLR Turquoise: Earned 5,000,000 credits (5,477,467)Cullen LLR Gold: Earned 500,000 credits (776,297)ESP LLR Ruby: Earned 2,000,000 credits (3,120,351)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,056,207)PPS LLR Jade: Earned 10,000,000 credits (12,244,808)PSP LLR Turquoise: Earned 5,000,000 credits (5,000,271)SoB LLR Sapphire: Earned 20,000,000 credits (20,233,229)SR5 LLR Turquoise: Earned 5,000,000 credits (5,978,312)SGS LLR Ruby: Earned 2,000,000 credits (2,940,587)TRP LLR Turquoise: Earned 5,000,000 credits (7,025,303)Woodall LLR Amethyst: Earned 1,000,000 credits (1,693,614)321 Sieve Jade: Earned 10,000,000 credits (18,635,342)Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,571,178)Generalized Cullen/Woodall Sieve (suspended) Emerald: Earned 50,000,000 credits (50,009,610)PPS Sieve Double Silver: Earned 200,000,000 credits (243,063,205)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Jade: Earned 10,000,000 credits (10,165,888)TRP Sieve (suspended) Sapphire: Earned 20,000,000 credits (20,071,454)AP 26/27 Turquoise: Earned 5,000,000 credits (6,037,979)GFN Emerald: Earned 50,000,000 credits (50,062,736)PSA Double Bronze: Earned 100,000,000 credits (102,762,384)
Message 114487 - Posted: 7 Feb 2018 | 3:24:53 UTC - in response to Message 114437.

composite,
While discussing tasks purging in Discord recently, I realized that your script, while it scans all the tasks pages until there's no "Next 20" link, will have skewed results near the end.

Tasks are purged on a timer that starts ticking when the last wingman completes their run.* That means that oldest tasks that you still see are most likely the ones that the wingman finished recently, but you long ago. Thus you'll see yourself 1st on the last pages more. The counterparts from those times, where your partner finished around the same time you did (and especially before you) are already purged, skewing the statistics.

The purge timer for PPSE is just 3 days. Don't count any PPSE tasks finished longer than 3 days ago into statistics. 4 days for GFN-16. Source: Mike :]
_____
* - with some exceptions, like tasks belonging to a challenge and tasks resulting in a prime. Those hang around longer.

EXCELLENT OBSERVATION! I had not thought about the systematic bias. We are after all looking for a statistical treatment.

I saw some hangers-on from September in the long tasks. The purging policy isn't transparent so I didn't even think about cutting off results.

I had to think about this twice. It means throwing away all validated work units where you or the wingman completed a task more than 3 days ago (for PPSE), even if you just finished your task a minute ago. Why? Because, if you don't you do this, your recent results are skewed toward a higher loss rate. If you don't believe that, then realize that the wingman's statistics are symmetric, and if his task is biased enough to toss out, then so too is yours.

What does that leave us with? Only workunits where the tasks were returned less than 3 days apart. That isn't fair either. My 1.5 GHz Pentium Pro could be losing all its competition by a margin longer than that. Tossing out all the data leaves nothing to be measured, and that's absurd because clearly the computer is doing some work.

NB. The script once got an incomplete line while scanning the tasks list. I think it may have been a partly purged task, since I'm guessing that task purging is not done as a transaction, in the interest of higher database speed. The script failed to match some of the columns of data and as a result one row of output had undefined variables. The taskID and hostID were there but the dates were all gone. As usual, Michael can provide some insight from the inside.

Profile Michael GoetzProject donor
Volunteer moderator
Project administrator
Project scientist
Avatar
Send message
Joined: 21 Jan 10
Posts: 13036
ID: 53948
Credit: 202,626,665
RAC: 115,417
The "Shut up already!" badge:  This loud mouth has mansplained on the forums over 10 thousand times!  Sheesh!!!Discovered the World's First GFN-19 prime!!!Discovered 1 mega primeFound 1 prime in the 2018 Tour de PrimesFound 1 prime in the 2019 Tour de Primes321 LLR Ruby: Earned 2,000,000 credits (2,822,730)Cullen LLR Ruby: Earned 2,000,000 credits (2,005,249)ESP LLR Turquoise: Earned 5,000,000 credits (5,009,577)Generalized Cullen/Woodall LLR Ruby: Earned 2,000,000 credits (2,145,754)PPS LLR Turquoise: Earned 5,000,000 credits (8,939,736)PSP LLR Turquoise: Earned 5,000,000 credits (5,065,592)SoB LLR Sapphire: Earned 20,000,000 credits (34,221,148)SR5 LLR Turquoise: Earned 5,000,000 credits (8,293,415)SGS LLR Ruby: Earned 2,000,000 credits (2,014,138)TRP LLR Ruby: Earned 2,000,000 credits (2,737,347)Woodall LLR Ruby: Earned 2,000,000 credits (2,195,123)321 Sieve Turquoise: Earned 5,000,000 credits (5,439,488)Cullen/Woodall Sieve (suspended) Ruby: Earned 2,000,000 credits (4,170,256)Generalized Cullen/Woodall Sieve (suspended) Turquoise: Earned 5,000,000 credits (5,059,304)PPS Sieve Sapphire: Earned 20,000,000 credits (20,110,788)Sierpinski (ESP/PSP/SoB) Sieve (suspended) Amethyst: Earned 1,000,000 credits (1,035,522)TRP Sieve (suspended) Ruby: Earned 2,000,000 credits (2,051,121)AP 26/27 Jade: Earned 10,000,000 credits (10,114,260)GFN Emerald: Earned 50,000,000 credits (66,792,341)PSA Jade: Earned 10,000,000 credits (12,404,447)
Message 114488 - Posted: 7 Feb 2018 | 3:35:24 UTC - in response to Message 114487.

NB. The script once got an incomplete line while scanning the tasks list. I think it may have been a partly purged task, since I'm guessing that task purging is not done as a transaction, in the interest of higher database speed. The script failed to match some of the columns of data and as a result one row of output had undefined variables. The taskID and hostID were there but the dates were all gone. As usual, Michael can provide some insight from the inside.


All the results are deleted in a single SQL statement, which is itself inside a transaction.
____________
Please do not PM me with support questions. Ask on the forums instead. Thank you!

My lucky number is 75898524288+1

Message boards : Number crunching : Benchmarking or post-mortem analysis?

[Return to PrimeGrid main page]
DNS Powered by DNSEXIT.COM
Copyright © 2005 - 2019 Rytis Slatkevičius (contact) and PrimeGrid community. Server load 1.95, 2.02, 2.03
Generated 7 Dec 2019 | 2:35:38 UTC