Join PrimeGrid
Returning Participants
Community
Leader Boards
Results
Other
drummers-lowrise
|
Message boards :
Number crunching :
PrimeGrid Task Summary Generator
| Author |
Message |
|---|
|
|
|
Greetings All,
I was interested in finding out how many tasks for certain sub-projects individual computers have done, ran into some issues separating certain sub-projects, figured those out, messed around a bunch and ended up with a batch file to generate a text file summary of everything. After some initial testing within team Storm, the sub-projects from Stream's private server were also added. This has now been tested by several people on a number of computers running Windows 7 and Windows 10, and can be downloaded at the following link.
https://drive.google.com/open?id=17IyXbGtsB5tWJexuKeSH6h2XJwA-RfJa
Here is an example of the output (which should open as soon as the batch is finished):
PrimeGrid Task Summary for DESKTOP-V97EL52 Sat 06 08 2019 16:39:43.19
321 Prime Search: 0
Cullen Prime Search: 15
Extended Sierpinski Problem: 0
Generalized Cullen/Woodall: 52
Prime Sierpinski Problem: 0
Proth Prime Search: 875
Proth Prime Search Extended: 198
Proth Mega Prime Search: 0
Seventeen or Bust: 22
Sierpinski/Riesel Base 5: 172
Sophie Germain Search: 23614
The Riesel Problem: 382
Woodall Prime Search: 76
321 Sieve: 12451
GCW Sieve: 26299
PPS Sieve: 5535
GFN-13 (Stream's Server): 1028599
GFN-14 (Stream's Server): 30481
GFN-15: 216510
GFN-16: 6238
GFN-16 Mega (Stream's Server): 0
GFN-17 Low: 441
GFN-17 Mega: 11268
GFN-18: 225
GFN-19: 2
GFN-20: 3
GFN-21: 2
GFN-22: 1
GFN-DYFL: 0
AP27: 334
The sub-projects and numbers of each tasks should be in columns when viewed on Windows 10 and encapsulated by "code" tags here, but the spacing is a bit funny on Windows 7. Either way; the numbers all seem correct.
To run it; plop the .bat file into your main BOINC Data directory (not the PrimeGrid one, but the directory where you find cc_config and the other main BOINC files) and double-click. Then wait. If you have large job log files it can take some time to run. It took about a minute on the computer which produced the output above, but I have a 30MB PrimeGrid and 110MB Stream's server job log, so those are kinda big compared to what most job logs will look like. Some of you will have much larger logs than that, so it may take quite a while.
I can't guarantee that this is bug free, but it doesn't seem to have done anything bad to anyone who has run it, and it is returning the correct results for all cases where I actually know what the correct result should be. As always, only run it if you are comfortable with running stuff like this. If you find any bugs, feel free to let me know as well.
Enjoy :)
Kellen | |
|
|
|
|
Simply a fine piece of programming there. I love it!
Thank you for this Kellen!
Seeing the GFN subproject breakdown is so nice! | |
|
|
|
|
Thanks Penguin; glad you like it! It was the GFN subproject breakdown (specifically GFN-15) that got me thinking about this in the first place ;) | |
|
|
|
|
Very nice !
Thanks for sharing.
____________
"Accidit in puncto, quod non contingit in anno."
Something that does not occur in a year may, perchance, happen in a moment. | |
|
|
|
|
Thanks for sharing!
____________
5912891284485*2^1290000-1
(Sophie Germain Prime Search) | |
|
 Michael Goetz Volunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13955
ID: 53948
Credit: 392,890,430
RAC: 184,027
                               
|
|
Really nice.
FWIW, I made a few changes in my copy.
I changed the first two lines to be this:
echo PrimeGrid Task Summary for %computername% %Date:/= % %Time:/= % > Tasks.txt
echo off
and the last two lines to be this:
type "PrimeGrid Tasks %Date:/= %.txt"
echo on
That's just a matter of personal taste.
I also added a few old projects that some people may have (these probably *should* be added to your batch file):
for /f %%c in ('find /c /i "gcwsieve" ^< job_log_www.primegrid.com.txt') do set /a cwsieve=%%c
echo CW Sieve: %cwsieve% >> Tasks.txt
for /f %%c in ('find /c /i "psp_sr2sieve" ^< job_log_www.primegrid.com.txt') do set /a pspsieve=%%c
echo PSP Sieve: %pspsieve% >> Tasks.txt
for /f %%c in ('find /c /i "trp_sr2sieve" ^< job_log_www.primegrid.com.txt') do set /a trpsieve=%%c
echo TRP Sieve: %trpsieve% >> Tasks.txt
It's possible someone may have some of those tasks still in the log.
Thanks for making this!
____________
My lucky number is 75898524288+1 | |
|
|
|
|
My pleasure everyone!
Thank you for the lines Michael! I will make those changes and re-issue the file shortly. Before doing that though, just curious; is there an ESP and SoB Sieve as well, or do those tasks get counted under the PSP Sieve? If they are counted under PSP I will change the name to "ESP/PSP/SoB Sieve" to reflect the naming on the Task Status page. Otherwise I will leave it as-is.
Thanks!
Kellen | |
|
dthononVolunteer tester
Send message
Joined: 6 Dec 17
Posts: 435
ID: 957147
Credit: 1,737,342,032
RAC: 13,828
                               
|
|
Similar code for Linux. It only prints the counters for tasks found in the log file. I tested it on Ubuntu and Debian, but not on CentOS/Fedora... distributions.
#!/usr/bin/env python3
import os
import os.path
import re
from collections import Counter
from datetime import datetime
if os.path.exists('/var/lib/boinc-client/'):
# Debian/Ubuntu/.. path
path = '/var/lib/boinc-client/'
elif os.path.exists('/var/lib/boinc/'):
# CentOS/Fedora/... path
path = '/var/lib/boinc/'
else:
print('BOINC path not found')
raise FileNotFoundError
logf1 = path + 'job_log_www.primegrid.com.txt'
logf2 = path + 'job_log_boincvm.proxyma.ru_30080_test4vm.txt'
cnt = Counter()
p = re.compile(r'(\w*)_\d*_\d*')
nb_lines = 0
def count(logf):
global nb_lines
# Read lines from log file
try:
with open(logf) as f:
for line in f:
nb_lines += 1
# Split by whitespace and take 9th item
task = line.split()[8]
# Pattern match for application name
app = p.match(task).group(1)
#print('{} -> {}'.format(task, app))
# Update counter
cnt.update([app])
except FileNotFoundError:
print('File {} not found'.format(logf))
return None
count(logf1)
count(logf2)
print('PrimeGrid Task Summary for {}, at {}'.format(os.uname()[1],
datetime.now()))
print('Total tasks = {} | Lines in log file = {}'.format(sum(cnt.values()),
nb_lines))
for (k, v) in sorted(cnt.items()):
print('{0:15s}: {1:8d}'.format(k, v))
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 13955
ID: 53948
Credit: 392,890,430
RAC: 184,027
|
My pleasure everyone!
Thank you for the lines Michael! I will make those changes and re-issue the file shortly. Before doing that though, just curious; is there an ESP and SoB Sieve as well, or do those tasks get counted under the PSP Sieve? If they are counted under PSP I will change the name to "ESP/PSP/SoB Sieve" to reflect the naming on the Task Status page. Otherwise I will leave it as-is.
Thanks!
Kellen
ESP/PSP/SOB were all the same sieve project.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
Thanks!
It has now been updated and I ended up with two versions based on different user preferences (and probably different job log file sizes). My job logs are pretty big and it takes a while to run through everything, so with echo off I was prone to thinking that something broke, but I can definitely see how having echo off would be desirable in many cases, so I made two :)
For the version that runs the same way the original did (ie. shows everything in the command window, then opens the file at the end) the link is here;
https://drive.google.com/open?id=1bHaAa_pfiwiftTTvlkxpKi8gglGbRK_y
For the version that runs a little "quieter" and doesn't pop the file open in the end, the link is here;
https://drive.google.com/open?id=17Lv2CwFhqjrWJzS3IADD1MZzS2uOd9Zv
Both now include the older sieves that I was missing. Thanks again Michael! | |
|
darkclownVolunteer tester
Send message
Joined: 3 Oct 06
Posts: 328
ID: 3605
Credit: 1,422,865,129
RAC: 250,840
              
|
|
Awesome little script! Thanks Kellen!
____________
My lucky #: 60133106^131072+1 (GFN 17-mega) | |
|
compositeVolunteer tester
Send message
Joined: 16 Feb 10
Posts: 1141
ID: 55391
Credit: 1,024,759,473
RAC: 1,703,156
                
|
|
The spirit of shell usage in Linux is to reuse existing tools and write little new software.
This bash two-liner will work on any flavour of Linux.
You'll notice that I threw in plenty of unnecessary spaces for readability.
echo PrimeGrid Task Summary for $(uname -n), at $(date); cut -d ' ' -f 9 /var/lib/boinc*/job_log_www.primegrid.com.txt | sed 's/_[0-9].*//' | perl -e 'chomp(@a=<>); $cnt{$_}++ for @a; printf "%-15s%8d\n", $_, $cnt{$_} for sort keys %cnt;' | |
|
|
|
The spirit of shell usage in Linux is to reuse existing tools and write little new software.
This bash two-liner will work on any flavour of Linux.
You'll notice that I threw in plenty of unnecessary spaces for readability.
echo PrimeGrid Task Summary for $(uname -n), at $(date); cut -d ' ' -f 9 /var/lib/boinc*/job_log_www.primegrid.com.txt | sed 's/_[0-9].*//' | perl -e 'chomp(@a=<>); $cnt{$_}++ for @a; printf "%-15s%8d\n", $_, $cnt{$_} for sort keys %cnt;'
That's what always puzzled me about bash-types. If you are sliding to perl anyway, why would you keep cut, sed, and whatnot. Do it with perl!
perl -nle '
$aa=(split / /)[8];
$aa=~s{_[0-9Me._]+$}{};
$ab{$aa}++;
END { printf "%-16s%8d\n", $_, $ab{$_} foreach sort keys %ab }' \
/var/lib/boinc*/job_log_www.primegrid.com.txt
Yes, it can be made shorter but that's not the point. Yes, you can do multiline quoted strings, even with bash. Also, '[0-9Me._]' bears 'M', 'e', and '.' for a reason.
p.s. Also, worth noting. You don't need perl for this. Modern shells, even bash, provide associative arrays and printf for eternity.
____________
I'm counting for science,
Points just make me sick. | |
|
compositeVolunteer tester
Send message
Joined: 16 Feb 10
Posts: 1141
ID: 55391
Credit: 1,024,759,473
RAC: 1,703,156
|
The spirit of shell usage in Linux is to reuse existing tools and write little new software.
This bash two-liner will work on any flavour of Linux.
You'll notice that I threw in plenty of unnecessary spaces for readability.
echo PrimeGrid Task Summary for $(uname -n), at $(date); cut -d ' ' -f 9 /var/lib/boinc*/job_log_www.primegrid.com.txt | sed 's/_[0-9].*//' | perl -e 'chomp(@a=<>); $cnt{$_}++ for @a; printf "%-15s%8d\n", $_, $cnt{$_} for sort keys %cnt;'
That's what always puzzled me about bash-types. If you are sliding to perl anyway, why would you keep cut, sed, and whatnot. Do it with perl!
perl -nle '
$aa=(split / /)[8];
$aa=~s{_[0-9Me._]+$}{};
$ab{$aa}++;
END { printf "%-16s%8d\n", $_, $ab{$_} foreach sort keys %ab }' \
/var/lib/boinc*/job_log_www.primegrid.com.txt
Yes, it can be made shorter but that's not the point. Yes, you can do multiline quoted strings, even with bash. Also, '[0-9Me._]' bears 'M', 'e', and '.' for a reason.
p.s. Also, worth noting. You don't need perl for this. Modern shells, even bash, provide associative arrays and printf for eternity.
That's the ticket. I've never used an END block so I didn't see how to get the summary from perl with -nle.
| |
|
Message boards :
Number crunching :
PrimeGrid Task Summary Generator |
