Join PrimeGrid
Returning Participants
Community
Leader Boards
Results
Other
drummers-lowrise
|
Message boards :
Number crunching :
Better multi-threading
| Author |
Message |
|---|
 Michael Goetz Volunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
                               
|
|
The new multi-threading system is now live.
IMPORTANT UPDATE: If you currently use app_config.xml for LLR tasks, you need to change your app_config.xml file if you want it to work after the new multi-threading system is turned on. Please see this message for more information.
UPDATE 2: (Item #5) The hyperthreading control isn't possible at this time.
UPDATE 3: (Item #6) Max jobs will apply to all apps. Max cpus only applies to LLR apps.
I was working this evening on testing the built-in BOINC multi-threading mechanism. This will allow you to select multi-threading from the project preferences web page. No more app_config.xml.
Although there's several coding changes that need to be done, the basic functionality seems to work correctly, and I don't foresee any major obstacles.
When this is implemented, expect it to work like this:
1) In the preferences selection, there will be selections for "max jobs" and "max cpus", similar to the settings in app_config.
2) Unlike app_config, these two settings apply to ALL apps. You can't chose 1 thread for SGS and 4 for SoB. When you change apps, you need to change your multithreading settings if you want to run a different number of threads.
3) There will be individual settings for each venue (location).
4) This will eliminate the problem of BOINC downloading 1 task for every core.
5) I'm considering adding a third setting, for turning hyperthreading on or off. This is a relatively new BOINC feature and requires BOINC client 7.14 or later. While most people should turn HT off (unless you believe it's beneficial for you), if you're using an older version of the BOINC client and have a non-HT CPU (such as a desktop core-i5), the server can't tell if you have HT or not, and will assume you do and only use half the cores. In this scenario, you'll need to set HT to "yes" in the preferences to use all cores. With 7.14 or later, it should work as you would expect it to work. (I still need to test this part.) Update: The BOINC client part of this doesn't work at all. There's no way to work around this, so this idea is dead.
6) The "max cpus" control will only apply to LLR apps. The "max jobs" control applies to all apps.
____________
My lucky number is 75898524288+1 | |
|
dthononVolunteer tester
Send message
Joined: 6 Dec 17
Posts: 435
ID: 957147
Credit: 1,764,269,087
RAC: 57,488
                               
|
|
That sounds good, as it will simplify remote control of multi threading.
I guess we could dedicate venues for different number of threads (Mercury 1, Venus 2,Earth 4...) and switch computers between venues. That would make sense, as venues are also where you select applications. | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
           
|
That sounds good, as it will simplify remote control of multi threading.
I guess we could dedicate venues for different number of threads (Mercury 1, Venus 2,Earth 4...) and switch computers between venues. That would make sense, as venues are also where you select applications.
Which would also simplify testing to see which setting is best for my computer as opposed to your computer.
| |
|
HonzaVolunteer moderator
Volunteer tester
Project scientist
Send message
Joined: 15 Aug 05
Posts: 1963
ID: 352
Credit: 6,410,029,624
RAC: 2,750,979
                                 
|
5) I'm considering adding a third setting, for turning hyperthreading on or off. This is a relatively new BOINC feature and requires BOINC client 7.14 or later. While most people should turn HT off (unless you believe it's beneficial for you), if you're using an older version of the BOINC client and have a non-HT CPU (such as a desktop core-i5), the server can't tell if you have HT or not, and will assume you do and only use half the cores. In this scenario, you'll need to set HT to "yes" in the preferences to use all cores. With 7.14 or later, it should work as you would expect it to work. (I still need to test this part.)
Particularly there, but overall usage - what is the percentage of 7.14 version connected to PG?
And older versions?
____________
My stats | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
That sounds good, as it will simplify remote control of multi threading.
I guess we could dedicate venues for different number of threads (Mercury 1, Venus 2,Earth 4...) and switch computers between venues. That would make sense, as venues are also where you select applications.
It may be a little easier than you expect.
The website preferences specify "maximum" CPU cores. So you can set it to 10, and all your computers will use 10 cores -- or less, if they have less that 10 cores. It won't be ideal for everyone, but it will work for a lot of people.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
5) I'm considering adding a third setting, for turning hyperthreading on or off. This is a relatively new BOINC feature and requires BOINC client 7.14 or later. While most people should turn HT off (unless you believe it's beneficial for you), if you're using an older version of the BOINC client and have a non-HT CPU (such as a desktop core-i5), the server can't tell if you have HT or not, and will assume you do and only use half the cores. In this scenario, you'll need to set HT to "yes" in the preferences to use all cores. With 7.14 or later, it should work as you would expect it to work. (I still need to test this part.)
Particularly there, but overall usage - what is the percentage of 7.14 version connected to PG?
And older versions?
I don't know. I suspect it will go up. :)
But it's 0.00% on Linux.
It's really only a problem for Linux on a non-HT CPU. The problem is that 7.14 tells the server the number of logical cores AND the number of physical cores, while older clients only tell the server the number of logical cores. The server, to be conservative, will always assume you have a HT CPU if you have an old client, and use half the number of cores (what it thinks are the physical cores) if it's told not to use hyperthreads. There's a number of ways to work around this problem, but it's a nuisance. I'll probably see if I can build a 7.14 linux boinc client just for my own use -- and, of course, it will be available for everyone to download. Just like the 7.13 linux boinc client is available for download.
____________
My lucky number is 75898524288+1 | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
That sounds good, as it will simplify remote control of multi threading.
I guess we could dedicate venues for different number of threads (Mercury 1, Venus 2,Earth 4...) and switch computers between venues. That would make sense, as venues are also where you select applications.
It may be a little easier than you expect.
The website preferences specify "maximum" CPU cores. So you can set it to 10, and all your computers will use 10 cores -- or less, if they have less that 10 cores. It won't be ideal for everyone, but it will work for a lot of people.
This could also require people to move their AMD cpu's to different venues than their Intel cpu's even though they could be crunching for the same sub-project. And maybe Linux and Windows pc's away from each other too depending on if the new version of Boinc works on their distro or not. I hope you give us a few days to do that as some people have more than 25 or 50 pc's and don't have time to do this every day. I count 14 venues here so it should be doable though.
Do you have a link to the 7.13 version you made? | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
That sounds good, as it will simplify remote control of multi threading.
I guess we could dedicate venues for different number of threads (Mercury 1, Venus 2,Earth 4...) and switch computers between venues. That would make sense, as venues are also where you select applications.
It may be a little easier than you expect.
The website preferences specify "maximum" CPU cores. So you can set it to 10, and all your computers will use 10 cores -- or less, if they have less that 10 cores. It won't be ideal for everyone, but it will work for a lot of people.
This could also require people to move their AMD cpu's to different venues than their Intel cpu's even though they could be crunching for the same sub-project. And maybe Linux and Windows pc's away from each other too depending on if the new version of Boinc works on their distro or not. I hope you give us a few days to do that as some people have more than 25 or 50 pc's and don't have time to do this every day. I count 14 venues here so it should be doable though.
If at all possible, I'll make it so that it's seamless -- if you take no action, nothing changes. I can't promise that will be possible, but that's my goal, and I think I know how it might be done.
Do you have a link to the 7.13 version you made?
https://www.primegrid.com/download/boinc_client/boinc_linux_x64_7.13.0
That's the BOINC client. Rename it to "boinc" and replace the boinc executable in your BOINC installation.
This is the associated boinccmd, the command line interface: https://www.primegrid.com/download/boinc_client/boinccmd_linux_x64_7.13.0 Likewise, rename it to boinccmd and replace the existing one.
____________
My lucky number is 75898524288+1 | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
That sounds good, as it will simplify remote control of multi threading.
I guess we could dedicate venues for different number of threads (Mercury 1, Venus 2,Earth 4...) and switch computers between venues. That would make sense, as venues are also where you select applications.
It may be a little easier than you expect.
The website preferences specify "maximum" CPU cores. So you can set it to 10, and all your computers will use 10 cores -- or less, if they have less that 10 cores. It won't be ideal for everyone, but it will work for a lot of people.
This could also require people to move their AMD cpu's to different venues than their Intel cpu's even though they could be crunching for the same sub-project. And maybe Linux and Windows pc's away from each other too depending on if the new version of Boinc works on their distro or not. I hope you give us a few days to do that as some people have more than 25 or 50 pc's and don't have time to do this every day. I count 14 venues here so it should be doable though.
If at all possible, I'll make it so that it's seamless -- if you take no action, nothing changes. I can't promise that will be possible, but that's my goal, and I think I know how it might be done.
Do you have a link to the 7.13 version you made?
https://www.primegrid.com/download/boinc_client/boinc_linux_x64_7.13.0
That's the BOINC client. Rename it to "boinc" and replace the boinc executable in your BOINC installation.
This is the associated boinccmd, the command line interface: https://www.primegrid.com/download/boinc_client/boinccmd_linux_x64_7.13.0 Likewise, rename it to boinccmd and replace the existing one.
Thank you VERY much on all counts!!!! | |
|
|
|
|
Will an app_config.xml still work normally? In other words, will it override any preferences set at the project?
____________
Reno, NV
 | |
|
|
|
|
This is an excellent development. I fully support this, it will make my life easier.
____________
 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Will an app_config.xml still work normally? In other words, will it override any preferences set at the project?
Yes it will, but there's a wrinkle.
The LLR apps currently do not have a plan_class. The new LLR apps will have a plan class of "mt" (or something similar to that.) This means that although you can still use app_config to override the website preferences, you will need to modify your current app_config.xml file to add the <plan_class>mt</plan_class> line.
I am hopeful that I can create a transition whereby your existing, unmodified app_config.xml file (without <plan_class>) can continue to work. Undoubtedly, no matter how many forum posts I write, or news items I create, or how many warnings I put on the website, some people won't see it.
The best way to be prepared for the change is to include two <app_version> blocks in your app_config.xml, one each for the current apps (without <plan_class>) and for the new upcoming apps (with <plan_class>). I'll have more information on this once I work out all the details.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
The idea of having a webpage control for hyperthreading is dead. The 7.14 BOINC client doesn't do what the documentation says it does. There's no way to work around this, so there's no way to make this work.
The rest of the multi-threading stuff will still work.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
This is a nice idea. They have this on several other BOINC projects. it will be interesting to see how it works here. Looking forward to trying it.
| |
|
KEPSend message
Joined: 10 Aug 05
Posts: 303
ID: 110
Credit: 13,001,669
RAC: 31,288
        
|
The idea of having a webpage control for hyperthreading is dead. The 7.14 BOINC client doesn't do what the documentation says it does. There's no way to work around this, so there's no way to make this work.
The rest of the multi-threading stuff will still work.
Can´t it be done in the WU creation system?
If you, have a selection similar to the subproject selection part of PG, just where there is only a list of all LLR projects, with a field next to the subproject where the user can then type in/or select the number of cores per WU for each LLR project. Then when a user request new work, the server creates the code that is now in the app_config.xml file and sent out a WU wich runs using the userprefered amount of cores. I think it should be possible to do, since -t (number of threads) is afterall understood by LLR. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
The idea of having a webpage control for hyperthreading is dead. The 7.14 BOINC client doesn't do what the documentation says it does. There's no way to work around this, so there's no way to make this work.
The rest of the multi-threading stuff will still work.
Can´t it be done in the WU creation system?
If you, have a selection similar to the subproject selection part of PG, just where there is only a list of all LLR projects, with a field next to the subproject where the user can then type in/or select the number of cores per WU for each LLR project. Then when a user request new work, the server creates the code that is now in the app_config.xml file and sent out a WU wich runs using the userprefered amount of cores. I think it should be possible to do, since -t (number of threads) is afterall understood by LLR.
Unfortunately, BOINC doesn't work like that. I do need to work within the BOINC framework.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
Must implement the LLR app the plan_class or is it the wrapper that has to implement this feature?
Will an app_config.xml still work normally? In other words, will it override any preferences set at the project?
Yes it will, but there's a wrinkle.
The LLR apps currently do not have a plan_class. The new LLR apps will have a plan class of "mt" (or something similar to that.) This means that although you can still use app_config to override the website preferences, you will need to modify your current app_config.xml file to add the <plan_class>mt</plan_class> line.
I am hopeful that I can create a transition whereby your existing, unmodified app_config.xml file (without <plan_class>) can continue to work. Undoubtedly, no matter how many forum posts I write, or news items I create, or how many warnings I put on the website, some people won't see it.
The best way to be prepared for the change is to include two <app_version> blocks in your app_config.xml, one each for the current apps (without <plan_class>) and for the new upcoming apps (with <plan_class>). I'll have more information on this once I work out all the details.
____________
DeleteNull | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Must implement the LLR app the plan_class or is it the wrapper that has to implement this feature?
It works like this...
You make some changes on the PrimeGrid preferences page, which sets two tags in your preferences (which are XML): <max_jobs> and <max_cpus>.
When your computer asks for work, the server's scheduler picks up those two tags from your preferences. It also picks up <min_ncpus> and <max_threads> from the plan_class definition. These two set minimum and maximum bounds on the number of threads. I'm planning on making the allowable range 1-80.
There's also a tag <nthreads_cmdline> in the plan class which instructs the server to tell the client on your computer to pass "--nthreads N" on the command line to the application.
In the task that the server sends to the client, included in the XML is the number of threads to run, which comes from the <max_cpus> tag you set on the preferences page.
The BOINC client invokes the app -- the LLR wrapper -- with "--nthreads N". The new version of the wrapper will understand --nthreads as well as -t, and will pass that along to LLR.
Does that make sense?
____________
My lucky number is 75898524288+1 | |
|
|
|
The BOINC client invokes the app -- the LLR wrapper -- with "--nthreads N". The new version of the wrapper will understand --nthreads as well as -t, and will pass that along to LLR.
Does that make sense?
Yes that is the reason for my question. --nthreads should be transformed into -t and into the parameter in the llr.ini (ThreadsPerTest=N) that is copied into the slots folder.
____________
DeleteNull | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
Must implement the LLR app the plan_class or is it the wrapper that has to implement this feature?
It works like this...
You make some changes on the PrimeGrid preferences page, which sets two tags in your preferences (which are XML): <max_jobs> and <max_cpus>.
When your computer asks for work, the server's scheduler picks up those two tags from your preferences. It also picks up <min_ncpus> and <max_threads> from the plan_class definition. These two set minimum and maximum bounds on the number of threads. I'm planning on making the allowable range 1-80.
There's also a tag <nthreads_cmdline> in the plan class which instructs the server to tell the client on your computer to pass "--nthreads N" on the command line to the application.
In the task that the server sends to the client, included in the XML is the number of threads to run, which comes from the <max_cpus> tag you set on the preferences page.
The BOINC client invokes the app -- the LLR wrapper -- with "--nthreads N". The new version of the wrapper will understand --nthreads as well as -t, and will pass that along to LLR.
Does that make sense?
I don't know if you Project level folks talk to each or not but at Cosmology they have the settings you are talking about:
Primary (default) preferences (Switch View)
Resource share ---
Use CPU
Is it OK for Cosmology@Home and your team (if any) to email you?
Should Cosmology@Home show your computers on its web site?
Default computer location ---
Run only the selected applications camb_legacy: yes
camb_boinc2docker: no
planck_param_sims: yes
If no work for selected applications is available, accept work from other applications? yes
Max # jobs 2
Max # CPUs 2
Edit preferences
With one of those kinds of tasks being made as MT(multi-thread) tasks and then going with the settings in the preferences. Perhaps you can contact them and figure out what and how they do it and if that will work for you guys here at PrimeGrid.
Those are my default settings for the project and I do get MT tasks at times. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Perhaps you can contact them and figure out what and how they do it and if that will work for you guys here at PrimeGrid.
That's standard BOINC functionality now, and I know how to do it. (Being as you quoted me explaining how the whole process works, that should be obvious.)
We just have to build some web code and crank out a modified LLR wrapper. I already have tested the guts of it on the dev server.
____________
My lucky number is 75898524288+1 | |
|
Nick
Send message
Joined: 11 Jul 11
Posts: 2301
ID: 105020
Credit: 10,120,453,899
RAC: 32,581,662
             
|
|
Thanks heaps Michael. This change will be awesome! | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
Perhaps you can contact them and figure out what and how they do it and if that will work for you guys here at PrimeGrid.
That's standard BOINC functionality now, and I know how to do it. (Being as you quoted me explaining how the whole process works, that should be obvious.)
We just have to build some web code and crank out a modified LLR wrapper. I already have tested the guts of it on the dev server.
I was not trying to imply you didn't I was just giving an example of one place that did get it working, but as I said working there doesn't mean working here.
I hope you can get it working, that would be sooo much easier than playing with app_config files. It would also eliminate the error lines saying "unknown application" for a sub-project it's never seen before. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
I hope you can get it working, that would be sooo much easier than playing with app_config files.
It will work.
Not only, as you point out, is it in use by many other BOINC sites, but I've already tested the important parts on our dev system.
And, yes, it will be much simpler to use. I'm looking forward to it.
____________
My lucky number is 75898524288+1 | |
|
Nick
Send message
Joined: 11 Jul 11
Posts: 2301
ID: 105020
Credit: 10,120,453,899
RAC: 32,581,662
|
|
I think the difficulty could be in transition to the new way | |
|
KEPSend message
Joined: 10 Aug 05
Posts: 303
ID: 110
Credit: 13,001,669
RAC: 31,288
|
I hope you can get it working, that would be sooo much easier than playing with app_config files.
It will work.
Not only, as you point out, is it in use by many other BOINC sites, but I've already tested the important parts on our dev system.
And, yes, it will be much simpler to use. I'm looking forward to it.
Cool :)
Are you also testing how this works if one has different computers with different numbers of cores?
Nice work :)
Also even though it might not have been formulated that way, it actually was more or less what I had in mind and tried to suggest, without knowing that it was actually already in use on other platforms :) | |
|
robishVolunteer moderator
Volunteer tester
Send message
Joined: 7 Jan 12
Posts: 2223
ID: 126266
Credit: 7,951,168,997
RAC: 5,401,041
            
|
I'm looking forward to it.
Me too. Going to be great! ;)
____________
My lucky number 10590941048576+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
IMPORTANT:
PrimeGrid will be switching to a preferences-based method for setting up multi-threading in the near future.
If you are using app_config.xml (probably to set up multi-threading) it's going to stop working when the multi-threading system goes live. If you want your app_config.xml to continue working, you should make modifications now that will allow it to work both before and after the change.
Only LLR projects are affected.
Every <app_version> block for an LLR project should be duplicated, with the duplicate having a <plan_class>mt<plan_class> line added to it. For example, this is the before and after for ESP, set up for 4 threads:
<app>
<name>llrESP</name>
<fraction_done_exact>1</fraction_done_exact>
<report_results_immediately>1</report_results_immediately>
</app>
<app_version>
<app_name>llrESP</app_name>
<cmdline>-t 4</cmdline>
<avg_ncpus>4</avg_ncpus>
</app_version>
<app_version>
<app_name>llrESP</app_name> <plan_class>mt</plan_class> <cmdline>-t 4</cmdline>
<avg_ncpus>4</avg_ncpus>
</app_version>
You can do this today, and the original app_version block will continue to work. Once we install the new multi-tasking, that app_version block will be ignored and the second app_version block for the mt plan class will take over. At that point, most people can erase app_config.xml, or at least the parts for LLR projects, and start using the multi-threading controls on the website.
I don't have an exact time yet for this change, but it won't happen in August. I'm hoping for early September.
____________
My lucky number is 75898524288+1 | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
|
Thank you very much!!! | |
|
|
|
|
Like this then?
<app_config>
<app>
<name>llrESP</name>
<fraction_done_exact>1</fraction_done_exact>
<max_concurrent>2</max_concurrent>
<report_results_immediately>1</report_results_immediately>
</app>
<app_version>
<app_name>llrESP</app_name>
<cmdline>-t 6</cmdline>
<avg_ncpus>1</avg_ncpus>
</app_version>
<app_version>
<app_name>llrESP</app_name>
<plan_class>mt</plan_class>
<cmdline>-t 6</cmdline>
<avg_ncpus>1</avg_ncpus>
</app_version>
<app>
</app_config>
And we keep Resource share set to 0 | |
|
|
|
|
Are these errors normal for now?
PrimeGrid: Notice from BOINC
Entry in app_config.xml for app 'llrSOB', plan class 'mt' doesn't match any app versions
| |
|
|
|
Are these errors normal for now?
PrimeGrid: Notice from BOINC
Entry in app_config.xml for app 'llrSOB', plan class 'mt' doesn't match any app versions
I see the same Error as well for llrMEGA.
Entry in app_config.xml for app 'llrMEGA', plan class 'mt' doesn't match any app versions | |
|
dh1sajVolunteer tester
Send message
Joined: 13 Jul 08
Posts: 49
ID: 25532
Credit: 3,636,918,553
RAC: 6
               
|
|
Same here.
Guess thats OK for the time being.
22.08.2019 20:29:34 | | Re-reading cc_config.xml
22.08.2019 20:29:34 | | cc_config.xml not found - using defaults
22.08.2019 20:29:34 | | log flags: file_xfer, sched_ops, task
22.08.2019 20:29:34 | PrimeGrid | Found app_config.xml
22.08.2019 20:29:34 | PrimeGrid | Entry in app_config.xml for app 'llr321', plan class 'mt' doesn't match any app versions
22.08.2019 20:29:34 | PrimeGrid | Entry in app_config.xml for app 'llrPPS', plan class 'mt' doesn't match any app versions
22.08.2019 20:29:34 | PrimeGrid | Entry in app_config.xml for app 'llrESP', plan class 'mt' doesn't match any app versions
22.08.2019 20:29:34 | PrimeGrid | Entry in app_config.xml for app 'llrMEGA', plan class 'mt' doesn't match any app versions
22.08.2019 20:29:34 | PrimeGrid | Entry in app_config.xml for app 'llrCUL', plan class 'mt' doesn't match any app versions
22.08.2019 20:29:34 | PrimeGrid | Entry in app_config.xml for app 'llrSOB', plan class 'mt' doesn't match any app versions
22.08.2019 20:29:36 | PrimeGrid | General prefs: from PrimeGrid (last modified 08-Dec-2016 20:02:03)
22.08.2019 20:29:36 | PrimeGrid | Computer location: home
22.08.2019 20:29:36 | | General prefs: using separate prefs for home
22.08.2019 20:29:36 | | Reading preferences override file
| |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
Are these errors normal for now?
PrimeGrid: Notice from BOINC
Entry in app_config.xml for app 'llrSOB', plan class 'mt' doesn't match any app versions
Since we've never had any "mt" tasks yes it's normal, once we get some it won't be an error anymore. It's the same as if you put SR5 task in the app_config the first time, you get the "unknown application valid ones are...." then when you get some SR5 units they work just fine and the error is also gone. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
I wish that error didn't exist. It's really quite useless.
Yes, it's normal. Just ignore it.
____________
My lucky number is 75898524288+1 | |
|
tngSend message
Joined: 29 Aug 10
Posts: 500
ID: 66603
Credit: 50,857,378,573
RAC: 30,980,247
                                         
|
I wish that error didn't exist. It's really quite useless.
+1 -- it had me hyperventilating a bit until I figured it out.
____________
 | |
|
|
|
But it's 0.00% on Linux. But not 0, as I run 7.14.2 on Linux. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
But it's 0.00% on Linux. But not 0, as I run 7.14.2 on Linux.
Awesome.
Unfortunately... it seems that the hyperthread control feature was documented as working in 7.14 but wasn't actually implemented. So this part isn't going to happen anytime soon.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
ANNOUNCEMENT:
I've got the multithreading working on the dev system (CompositeGrid). Feel free to try it out and tell me what you think. A few caveats:
* Windows and Linux only, so far. Mac tasks aren't done yet, and will crash and burn if you try to run them. Or they'll just ignore the multi-threading and run single-threaded. Update: The Mac app is now available for testing.
* Only SGS tasks are loaded. There's no work available for any other sub-projects.
* The tasks aren't being refilled automatically, so they may run out.
* The server (connect at https://dev.primegrid.com/) isn't up 24/7. I may take down for various reasons.
* Your login credentials at PrimeGrid should work on the dev system, unless you signed up within the last few days.
____________
My lucky number is 75898524288+1 | |
|
HonzaVolunteer moderator
Volunteer tester
Project scientist
Send message
Joined: 15 Aug 05
Posts: 1963
ID: 352
Credit: 6,410,029,624
RAC: 2,750,979
|
|
Initial try.
I had no limits for Max # of jobs for this project nor Max # of CPUs for this project and attached Windows host with 2-CPUs.
It donwloaded 7 SGS tasks thinking each will take 25 secs...a bit of wishfull thinking even on this machine but we know how BOINC behaves considering time estimation.
First task started using 30 CPUs. Damn, cores.
I must says I consider it confusing on 2-CPUs system.
In preferences, I would rather see "Max # of CPU cores for this project"
Letting first task complete, will take 16 minutes.
Now, setting Max # of CPUs for this project to 7.
Later...BOINC just couple minutes before finishing first task downloaded another. Good.
It was set to use 7 CPUs. Good.
It started along the first 30CPUs, NOT good.
Machines has "only" 32 core but BOINC used 37 and is set to use ast most 94% of the CPUs (which is 30).
After finishing first SGS 30-cores task, it resumed other 7-thread PG tasks (yes, I have PG attached as well with task in cache). Good.
BOINC downloaded another 4 SGS and started one and pause on of PG. Total 5x7 is again overcomming with 35 cores and limit is set to 30 cores.
____________
My stats | |
|
VatoVolunteer tester
Send message
Joined: 2 Feb 08
Posts: 861
ID: 18447
Credit: 876,568,914
RAC: 1,448,223
          
|
|
running mt tests:
2c machine with no cache and non-zero project priority - configured for max 4 cores via web
downloaded a single task (WIN)
task is 2-CPU (WIN)
ran immediately even though there was a single-core task in progress from PG (not ideal, but cross-project is hard in BOINC - makes me think website default should be no job limit, but core limit of 1 to ensure quiet transition from old to new scheme)
changed web settings to max 1 job, 1 core
server now limits how many tasks can be outstanding rather than client - only downloaded a single task once the upload was done, not at the 3 minute marker - task correctly runs 1c
so this doesn't seem to use max_concurrent (or not only max_concurrent)
changing to max 2 job, 1c, works as expected
nothing wrong with the wrapper, nor website afaict
possibly stuff wrong with my understanding and assumptions though
tbh the idea of max_concurrent being used was probably broken in the first place
why should one task impact another, via config provided to client by a task
it should only stand a chance of working that way if config is provided project-wide
____________
 | |
|
HonzaVolunteer moderator
Volunteer tester
Project scientist
Send message
Joined: 15 Aug 05
Posts: 1963
ID: 352
Credit: 6,410,029,624
RAC: 2,750,979
|
|
Later, with only one (with only o PG LLR task), dev PG seems to honor Max # of jobs for this project and Max # of CPUs.
When settings dev PG to NNW, BOINC goes again crazy downloading dozens of 7-threads tasks from PG.
It will be a relief when this all gets implemented.
Transitions periods are painfull.
Any other scenario to test?
____________
My stats | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Later, with only one (with only o PG LLR task), dev PG seems to honor Max # of jobs for this project and Max # of CPUs.
When settings dev PG to NNW, BOINC goes again crazy downloading dozens of 7-threads tasks from PG.
It will be a relief when this all gets implemented.
Transitions periods are painfull.
Any other scenario to test?
You were getting 7-thread tasks on a 2-core machine?
If so, was there anything in cc_config (ncores) or app_info/app_config that might be causing this?
Any idea why BOINC would think a 2 core machine would have 7 cores?
____________
My lucky number is 75898524288+1 | |
|
HonzaVolunteer moderator
Volunteer tester
Project scientist
Send message
Joined: 15 Aug 05
Posts: 1963
ID: 352
Credit: 6,410,029,624
RAC: 2,750,979
|
You were getting 7-thread tasks on a 2-core machine?
If so, was there anything in cc_config (ncores) or app_info/app_config that might be causing this?
Any idea why BOINC would think a 2 core machine would have 7 cores?
Nope, sorry for not being accurate.
I change Max # CPUs to 2 and it was doing what was supposed to.
BOINC started downloading 2CPUs tasks.
It is a 2-CPUs systems with 16-cores each, hostid=530837
BOINC is set to use 94% of CPU so 30 should be available to BOINC.
____________
My stats | |
|
|
|
In preferences, I would rather see "Max # of CPU cores for this project"
I think "threads" would be most accurate. BOINC counts threads, not cores. If you have a 4 core machine, with HT turned off, BOINC sees 4 threads. If you have a 4 core machine, with HT turned on, BOINC sees 8 threads.
____________
Reno, NV
| |
|
|
|
|
Hi.
Atm I'm running an SOB with added plan_class, but without having duplicated the app_version part, and without having restarted Boinc.
Working fine so far with multi-threading still being used.
So, duplicating might not be necessary.
I'll be keeping you updated after restarting Boinc.
For security reasons I won't further test this while running this SOB. ;-)
If that hyper-threading detection stuff would run I'd be quite happy.
Well, nearly flawless Boinc documentation and stuff that should work and doesn't are nothing new to me...
____________
Greetings, Jens
147433824^131072+1 | |
|
|
|
IMPORTANT:
PrimeGrid will be switching to a preferences-based method for setting up multi-threading in the near future.
If you are using app_config.xml (probably to set up multi-threading) it's going to stop working when the multi-threading system goes live. If you want your app_config.xml to continue working, you should make modifications now that will allow it to work both before and after the change.
Only LLR projects are affected.
Every <app_version> block for an LLR project should be duplicated, with the duplicate having a <plan_class>mt<plan_class> line added to it. For example, this is the before and after for ESP, set up for 4 threads:
<app>
<name>llrESP</name>
<fraction_done_exact>1</fraction_done_exact>
<report_results_immediately>1</report_results_immediately>
</app>
<app_version>
<app_name>llrESP</app_name>
<cmdline>-t 4</cmdline>
<avg_ncpus>4</avg_ncpus>
</app_version>
<app_version>
<app_name>llrESP</app_name> <plan_class>mt</plan_class> <cmdline>-t 4</cmdline>
<avg_ncpus>4</avg_ncpus>
</app_version>
You can do this today, and the original app_version block will continue to work. Once we install the new multi-tasking, that app_version block will be ignored and the second app_version block for the mt plan class will take over. At that point, most people can erase app_config.xml, or at least the parts for LLR projects, and start using the multi-threading controls on the website.
I don't have an exact time yet for this change, but it won't happen in August. I'm hoping for early September.
A bit late to be able to edit the post but there is a [ B ] that looks to be ignored by a [ PRE ]. I've never see a PRE function before in a forum.
I went to copy the post to my team and noticed the formatting was goofy. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
A bit late to be able to edit the post but there is a [ B ] that looks to be ignored by a [ PRE ]. I've never see a PRE function before in a forum.
I went to copy the post to my team and noticed the formatting was goofy.
If you mean the bold formatting for the line with the <plan_class> tag, that's intentional.
It's
[ pre ]1
... stufff ... [ /pre ]1 [ b ]2 [ pre ]3 bold line 'pre' text [ /pre ]3 [ /b ]2 [ pre ]4 next line of normal 'pre' text
... more stuff...
[ /pre ]4
What I did there is actually use 3 "pre" blocks, with the middle "pre" block being surrounded by B tags.
1 is the opening pre block
2 is the B block surrounding the middle (3) pre block.
4 is the pre block for the end portion
____________
My lucky number is 75898524288+1 | |
|
|
|
|
hello
Windows 10
BOINC 7.14.2(X64)
not running as a service.
I'm not very computer literate :(
how do I find app_config.xml ?
I think it is:
right click on computer icon desktop
this PC - -> OS C - -> Program Files - -> BOINC --> then ?
any help would be greatly appreciated :)
Byron.
my screen shot:
any help would be greatly appreciated :)
Byron.
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
how do I find app_config.xml ?
I think it is:
right click on computer icon desktop
this PC - -> OS C - -> Program Files - -> BOINC --> then ?
Very close...
this PC - -> OS C - -> ProgramData - -> BOINC --> projects --> www.primegrid.com
Note that you may need to change your Windows Explorer settings to even be able to see ProgramData since it is hidden by default. Also note there's no space between Program and Data.
Please heed the part about "if you don't know what this is, you don't need to do anything."
Once the new system goes live (probably in about two weeks) everything you're going to do with app_config will likely be rendered obsolete. My recommendation is just to wait until the new system goes live, and then use the new web-based controls.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
Byron, app_config.xml files you make your self and put into the directory of the project you are running. If you did not make one and put it there, then there won't be one there.
You can make one for every project you run if you want/need to.
You have to have show hidden folders and files checked. You can find that by typing
show hidden files and folders in the start area lower left corner.
programdata/BIONC/projects/www.primegrid.com
/seti
/WCG
Then the same for the other projects you run | |
|
|
|
|
For me, the new 8.07(mt) applications seem to be running just fine. No errors, at least on the PCs with no app_config files.
____________
My lucky number is 6219*2^3374198+1
 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
For me, the new 8.07(mt) applications seem to be running just fine. No errors, at least on the PCs with no app_config files.
That's good!
Note: Don't get too used to calling this 8.07. When it gets to the live server, it will probably be 8.04.
____________
My lucky number is 75898524288+1 | |
|
|
|
how do I find app_config.xml ?
I think it is:
right click on computer icon desktop
this PC - -> OS C - -> Program Files - -> BOINC --> then ?
Very close...
this PC - -> OS C - -> ProgramData - -> BOINC --> projects --> www.primegrid.com
Note that you may need to change your Windows Explorer settings to even be able to see ProgramData since it is hidden by default. Also note there's no space between Program and Data.
Please heed the part about "if you don't know what this is, you don't need to do anything."
Once the new system goes live (probably in about two weeks) everything you're going to do with app_config will likely be rendered obsolete. My recommendation is just to wait until the new system goes live, and then use the new web-based controls.
If you type %ProgramData% in the explorer path bar, it will take you to it. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
The Mac MT app (64 bit only) is now available on the dev system. Feel free to test it!
____________
My lucky number is 75898524288+1 | |
|
Chooka
Send message
Joined: 15 May 18
Posts: 335
ID: 1014486
Credit: 1,312,034,212
RAC: 4,550,817
    
|
|
Oh dear... app_configs are confusing for me at the best of times. Looks like I've got something else to figure out now.
Oh well.
____________
Слава Україні! | |
|
|
|
|
Try PRPNet if you can.
____________
My lucky number is 6219*2^3374198+1
| |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
Oh dear... app_configs are confusing for me at the best of times. Looks like I've got something else to figure out now.
Oh well.
The new way should be easier in the long run, no more snytax mistakes or using the wrong kind of program to create or edit the app_config files etc. You just set how many workunits you want to run at a time, then how many cpu cores to use for each workunit and Boinc does the rest. It works really well over at Cosmology right now and it's also working on the test server here according to those that are testing it for the rest of us. Michael Goetz has done a great job as usual!! | |
|
|
|
|
:)
Also, can we have different CPU usage for different venues?
____________
My lucky number is 6219*2^3374198+1
| |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
:)
Also, can we have different CPU usage for different venues?
That's what Michael said he is hoping to accomplish. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
:)
Also, can we have different CPU usage for different venues?
That's what Michael said he is hoping to accomplish.
Everything I said in the first post in this thread, except for one item that is crossed out, is working on the test system.
So, yes, there's separate controls for each venue.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
ANNOUNCEMENT:
I am basically ready to put the new multi-threading system online. Two people said they needed some time to update app_config on their systems. Scott Brown was one of them, but I can't remember the other person. How much time do you need? September 2nd (a week from today) was mentioned.
If anyone needs time to prepare for the change, let me know. I'll delay the rollout so as to give you enough time to get ready. Just let me know what you need.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
The MAX-JOBS control on the web affects ALL apps, including those that are not affected by multi-threading.
MAX-CPUs only affects LLR apps (i.e., those running with the mt plan_class.)
____________
My lucky number is 75898524288+1 | |
|
|
|
|
Good to go Michael.
____________
| |
|
tngSend message
Joined: 29 Aug 10
Posts: 500
ID: 66603
Credit: 50,857,378,573
RAC: 30,980,247
|
|
I'll get them done by the 2d.
____________
| |
|
|
|
|
good to go, but anytime would be ok for me
____________
My lucky number is 6219*2^3374198+1
| |
|
GregCSend message
Joined: 12 Nov 18
Posts: 57
ID: 1077873
Credit: 2,300,051,701
RAC: 6,855,667
  
|
|
When you select the number of tasks you want to do (I see it ranges from 1-256), is it done on a First In, First Out basis? For example:
I want to download 4 workunits.
I want to dedicate 8 cores to the job.
Is it split so that it's 2 workunits per 4 cores, or just one at a time working across all 8 cores (one finished, the next one is downloaded)? | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
When you select the number of tasks you want to do (I see it ranges from 1-256), is it done on a First In, First Out basis? For example:
I want to download 4 workunits.
I want to dedicate 8 cores to the job.
Is it split so that it's 2 workunits per 4 cores, or just one at a time working across all 8 cores (one finished, the next one is downloaded)?
Assuming that your computer has 8 cores:
The MAX_CPUs=8 setting will cause it to run -t8 tasks.
The MAX_JOBS=4 setting won't have any effect, because on an 8 core machine, if the tasks consume 8 cores each, you can only run 1 job at a time. It will (or should1) download a single task to run on all 8 cores.
Assume that the computer has 4 cores:
The MAX_CPUs=8 setting will cause it to run -t4 tasks, because there's only 4 cores in the computer.
The MAX_JOBS=4 setting won't have any effect because you can only run one 4-core job at a time.
Assume that the computer has 64 cores:
The MAX_CPUs=8 setting will cause it to run -t8 tasks.
Normally, BOINC could run eight 8-cores tasks on a 64 core machine, but the MAX_JOBS=4 setting will cause it to only run 4 tasks at once.
Now, you said you want to "download four tasks". These controls affect how BOINC runs the tasks. It does not directly affect how many are downloaded. That's controlled primarily by the settings in the BOINC client for how many days of work to download.
1I say "should", because BOINC sometimes downloads more tasks than it should.
____________
My lucky number is 75898524288+1 | |
|
GregCSend message
Joined: 12 Nov 18
Posts: 57
ID: 1077873
Credit: 2,300,051,701
RAC: 6,855,667
|
When you select the number of tasks you want to do (I see it ranges from 1-256), is it done on a First In, First Out basis? For example:
I want to download 4 workunits.
I want to dedicate 8 cores to the job.
Is it split so that it's 2 workunits per 4 cores, or just one at a time working across all 8 cores (one finished, the next one is downloaded)?
Assuming that your computer has 8 cores:
The MAX_CPUs=8 setting will cause it to run -t8 tasks.
The MAX_JOBS=4 setting won't have any effect, because on an 8 core machine, if the tasks consume 8 cores each, you can only run 1 job at a time. It will (or should1) download a single task to run on all 8 cores.
Assume that the computer has 4 cores:
The MAX_CPUs=8 setting will cause it to run -t4 tasks, because there's only 4 cores in the computer.
The MAX_JOBS=4 setting won't have any effect because you can only run one 4-core job at a time.
Assume that the computer has 64 cores:
The MAX_CPUs=8 setting will cause it to run -t8 tasks.
Normally, BOINC could run eight 8-cores tasks on a 64 core machine, but the MAX_JOBS=4 setting will cause it to only run 4 tasks at once.
Now, you said you want to "download four tasks". These controls affect how BOINC runs the tasks. It does not directly affect how many are downloaded. That's controlled primarily by the settings in the BOINC client for how many days of work to download.
1I say "should", because BOINC sometimes downloads more tasks than it should.
Thank you! This answers my questions and hopefully a few other folks'.
| |
|
GregCSend message
Joined: 12 Nov 18
Posts: 57
ID: 1077873
Credit: 2,300,051,701
RAC: 6,855,667
|
|
Ah, something I forgot to ask: How would the MAX-JOBS and MAX-CPUs be set if I want to run, say, a single SOB across all 8 cores at the same time? | |
|
dthononVolunteer tester
Send message
Joined: 6 Dec 17
Posts: 435
ID: 957147
Credit: 1,764,269,087
RAC: 57,488
|
|
python app_config generator. Save in file $HOME/gen_app_config.py, tailor to your threads and cores preferences and execute (On Linux. I did not try on WIndows).
sudo pip3 install jinja2
sudo python3 $HOME/gen_app_config.py > /var/lib/boinc-client/projects/www.primegrid.com/app_config.xml
sudo systemctl restart boinc-client
#!/usr/bin/env python3
"""Generating app_config.xml."""
from jinja2 import Template
t_llr = Template("""
<app>
<name>{{app}}</name>
<fraction_done_exact>1</fraction_done_exact>
<report_results_immediately>1</report_results_immediately>
</app>
<app_version>
<app_name>{{app}}</app_name>
<cmdline>-t {{threads}}</cmdline>
<avg_ncpus>{{cores}}</avg_ncpus>
</app_version>
<app_version>
<app_name>{{app}}</app_name>
<plan_class>mt</plan_class>
<cmdline>-t {{threads}}</cmdline>
<avg_ncpus>{{cores}}</avg_ncpus>
</app_version>
""")
t_sieve = Template("""
<app>
<name>{{app}}</name>
<fraction_done_exact>1</fraction_done_exact>
<report_results_immediately>1</report_results_immediately>
</app>
<app_version>
<app_name>{{app}}</app_name>
<avg_ncpus>{{cores}}</avg_ncpus>
</app_version>
""")
print("<app_config>")
print(t_llr.render(app="llrTPS", threads="1", cores="1"))
print(t_llr.render(app="llrWOO", threads="4", cores="4"))
print(t_llr.render(app="llrCUL", threads="4", cores="4"))
print(t_llr.render(app="llr321", threads="4", cores="4"))
print(t_llr.render(app="llrPSP", threads="4", cores="4"))
print(t_llr.render(app="llrPPS", threads="2", cores="2"))
print(t_llr.render(app="llrSOB", threads="4", cores="4"))
print(t_llr.render(app="llrTRP", threads="4", cores="4"))
print(t_llr.render(app="llrPPSE", threads="1", cores="1"))
print(t_llr.render(app="llrSR5", threads="4", cores="4"))
print(t_llr.render(app="llrESP", threads="4", cores="4"))
print(t_llr.render(app="llrMEGA", threads="2", cores="2"))
print(t_llr.render(app="llrGCW", threads="4", cores="4"))
print(t_sieve.render(app="pps_sr2sieve", cores="1"))
print(t_sieve.render(app="321_sr2sieve", cores="1"))
print("</app_config>")
| |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
Probably JOBS=1&CPUS=8
dthonon: that's great! somehow i never thought of doing that
Remember though if half of those 8 cores are hyper-threaded they will slow your crunching down here at PrimeGrid on these LLR tasks. So in that case you would want 1 job done using 4 cpu cores.
I have an AMD 1920x cpu and ran 1 task using 23 cpu cores to run some SOB tasks and they only took 23 hours to run each one. I did not try other settings as it was waaaaaay faster than the predicted 311 hours per task that's listed when you select to run them. On one of my Intel pc's I'm running some GCW tasks using 6 non HT cores and it's takes about 7 hours instead of the listed 60 hours. On another Intel machine that only has 4 non HT cores it's taking about 16 hours instead of the again 60 listed hours for the same GCW tasks. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Ah, something I forgot to ask: How would the MAX-JOBS and MAX-CPUs be set if I want to run, say, a single SOB across all 8 cores at the same time?
That's the easiest question of all!
Leave both controls set to "no-limit" -- which is the default setting.
BOINC will run 1 task using all available cores.
Here's how I personally intend to set the controls (after erasing app_config.xml):
SGS, PPSE, and PPS:
MAX-JOBS: no-limit
MAX-CPUs: 1
All other LLR:
MAX-JOBS: no-limit
MAX-CPUs: no-limit
That should work for most people who do not have a computer with a lot of cores.
____________
My lucky number is 75898524288+1 | |
|
Scott BrownVolunteer moderator
Project administrator
Volunteer tester
Project scientist
Send message
Joined: 17 Oct 05
Posts: 2417
ID: 1178
Credit: 20,070,256,029
RAC: 21,654,260
                              
|
|
Just to clarify this a bit for myself. Let's say I am using a common i7 CPU with 4-cores and 8-threads. HT is turned on, but I want to run LLR only at 50%. I would set the following:
MAX-JOBS: 4
MAX-CPUs: 1
That would be the -t4 equivalent. Is that correct? And this will have no effect on the GPUs that are also running non-LLR PG tasks in the system. | |
|
streamVolunteer moderator
Project administrator
Volunteer developer
Volunteer tester
Send message
Joined: 1 Mar 14
Posts: 1051
ID: 301928
Credit: 563,881,725
RAC: 1,034
  
|
Leave both controls set to "no-limit" -- which is the default setting.
BOINC will run 1 task using all available cores.
These defaults settings may lead to drop of overall PG performance because MT is not optimal for SGS, PPSE, and PPS, and sub-optimal for other LLR projects. I did some benchmarks and many low-level customer CPUs are working worse in MT LLR mode. First reason is an inefficient implementation of MT in LLR - threads are waiting for each other, causing only 80-90% of CPU load per thread, and losses are getting worse as a number of thread grows. Additionally, in real Windows world, where Windows itself often load the CPU with some internal background work, LLR will wait for these delayed threads which will lead to further decrease of performance. Even 1 delayed thread will also stop all other threads. This will not happen if we have few independent LLR processes.
I think default setting must be 1 core / no limits, to emulate old behavior. LLR is not an app which could be blindly scaled to N cores, it MT implementation is far from ideal. An user must intentionally enable this mode after test and benchmarks.
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Just to clarify this a bit for myself. Let's say I am using a common i7 CPU with 4-cores and 8-threads. HT is turned on, but I want to run LLR only at 50%. I would set the following:
MAX-JOBS: 4
MAX-CPUs: 1
That would be the -t4 equivalent. Is that correct?
That's... perfectly BACKWARDS. :)
What I would do is first set BOINC to use 50% of the CPUs. Same as you do right now.
Then leave both controls set to "no-limit":
MAX-JOBS: no-limit
MAX-CPUs: no-limit
Because the default behavior (with both set to "no-limit") is to run a single task using all the cores, setting CPUs to 50% effectively creates a 4 core machine and produces your desired result.
If, however, for some reason you want to leave BOINC set to use 100% of the CPUs, you can do this:
MAX-JOBS: 1
MAX-CPUs: 4
MAX-CPUs tells BOINC it's allowed to use 4 CPUs. MAX-JOBS tells BOINC it can only run 1 task at a time. It will therefore run a single -t4 task.
And this will have no effect on the GPUs that are also running non-LLR PG tasks in the system.
You bring up a good point.
This is where it gets complicated. MAX-JOBS is a GLOBAL setting. It affects all of that venue's PrimeGrid tasks, not just the LLR tasks. Setting it to 1 says "only run one PrimeGrid task at a time." That will affect your GPU jobs, and that's not what you want.
To be honest, in my opinion, most people should leave "MAX-JOBS" set to "no-limit" all the time. Use the BOINC manager's "Use xx% of the CPUs" setting to control hyperthreading, same as today. Use the new MAX-CPUs setting to control multi-threading.
____________
My lucky number is 75898524288+1 | |
|
RafaelVolunteer tester
Send message
Joined: 22 Oct 14
Posts: 918
ID: 370496
Credit: 608,531,089
RAC: 718,919
  
|
Leave both controls set to "no-limit" -- which is the default setting.
BOINC will run 1 task using all available cores.
I think default setting must be 1 core / no limits, to emulate old behavior. LLR is not an app which could be blindly scaled to N cores, it MT implementation is far from ideal. An user must intentionally enable this mode after test and benchmarks.
Agreed with default behaviour.
At any rate, how does these new settings play if you change your preferences midway through jobs? Will it behave like appconfig and only aplly upon a restart or will it update as soon as it connects to server? And assuming it does on connect, will it recalculate the number of jobs to run and stop / start new ones immediately, but without changing the command line for ongoing ones, or will it be smarter and stop everything and restart with the new thread count?
I ask because if the current appconfig's behaviour is kept, we are bound to get confused people that update their preferences but see no actual change happening, so it might be a good idead to either change it (if possible) or at the very least put some explanation as to when your new settings will actually take effect. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
At any rate, how does these new settings play if you change your preferences midway through jobs? Will it behave like appconfig and only aplly upon a restart or will it update as soon as it connects to server? And assuming it does on connect, will it recalculate the number of jobs to run and stop / start new ones immediately, but without changing the command line for ongoing ones, or will it be smarter and stop everything and restart with the new thread count?
I ask because if the current appconfig's behaviour is kept, we are bound to get confused people that update their preferences but see no actual change happening, so it might be a good idead to either change it (if possible) or at the very least put some explanation as to when your new settings will actually take effect.
I'm not sure about most of that. The only part I'm sure of is that in order to actuall change the multi-threading behavior of a running task, e.g., change -t4 to -t2, the LLR task must be paused and resumed, "including removing it from memory". That's the same as it is with app_config.
As for the rest of it, it's likely it also behaves the same as app_config (with "update" replacing "read config files"), but I have not tested that.
____________
My lucky number is 75898524288+1 | |
|
HonzaVolunteer moderator
Volunteer tester
Project scientist
Send message
Joined: 15 Aug 05
Posts: 1963
ID: 352
Credit: 6,410,029,624
RAC: 2,750,979
|
These defaults settings may lead to drop of overall PG performance because MT is not optimal for SGS, PPSE, and PPS, and sub-optimal for other LLR projects. I did some benchmarks and many low-level customer CPUs are working worse in MT LLR mode. First reason is an inefficient implementation of MT in LLR - threads are waiting for each other, causing only 80-90% of CPU load per thread, and losses are getting worse as a number of thread grows. Additionally, in real Windows world, where Windows itself often load the CPU with some internal background work, LLR will wait for these delayed threads which will lead to further decrease of performance. Even 1 delayed thread will also stop all other threads. This will not happen if we have few independent LLR processes.
I think default setting must be 1 core / no limits, to emulate old behavior. LLR is not an app which could be blindly scaled to N cores, it MT implementation is far from ideal. An user must intentionally enable this mode after test and benchmarks.
I tend to agree.
Simulate old behaviour as default and let participants change their settings.
With multi-core CPUs, you get greater drop using all cores on single task. Even worst scenario with 2-CPUs systems. And with my 2x16cores, ouch.
(I don't my baby-sit or prepare in advance, just pointing out weak spots).
____________
My stats | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
These defaults settings may lead to drop of overall PG performance because MT is not optimal for SGS, PPSE, and PPS, and sub-optimal for other LLR projects. I did some benchmarks and many low-level customer CPUs are working worse in MT LLR mode. First reason is an inefficient implementation of MT in LLR - threads are waiting for each other, causing only 80-90% of CPU load per thread, and losses are getting worse as a number of thread grows. Additionally, in real Windows world, where Windows itself often load the CPU with some internal background work, LLR will wait for these delayed threads which will lead to further decrease of performance. Even 1 delayed thread will also stop all other threads. This will not happen if we have few independent LLR processes.
I think default setting must be 1 core / no limits, to emulate old behavior. LLR is not an app which could be blindly scaled to N cores, it MT implementation is far from ideal. An user must intentionally enable this mode after test and benchmarks.
I tend to agree.
Simulate old behaviour as default and let participants change their settings.
With multi-core CPUs, you get greater drop using all cores on single task. Even worst scenario with 2-CPUs systems. And with my 2x16cores, ouch.
(I don't my baby-sit or prepare in advance, just pointing out weak spots).
Unfortunately... I have to agree with this.
I'll look into changing the default to single core.
____________
My lucky number is 75898524288+1 | |
|
GregCSend message
Joined: 12 Nov 18
Posts: 57
ID: 1077873
Credit: 2,300,051,701
RAC: 6,855,667
|
|
Another newbie question: can/will we be able to ditch app_config? | |
|
RafaelVolunteer tester
Send message
Joined: 22 Oct 14
Posts: 918
ID: 370496
Credit: 608,531,089
RAC: 718,919
|
Another newbie question: can/will we be able to ditch app_config?
For the majority of users, yes.
But if you want some more granular control (say, set up different thread counts for each subproject once and never deal with it again) or add extra flags like -no_avx512 when it gets implemented, you will still need to use app_config. | |
|
GregCSend message
Joined: 12 Nov 18
Posts: 57
ID: 1077873
Credit: 2,300,051,701
RAC: 6,855,667
|
Another newbie question: can/will we be able to ditch app_config?
For the majority of users, yes.
But if you want some more granular control (say, set up different thread counts for each subproject once and never deal with it again) or add extra flags like -no_avx512 when it gets implemented, you will still need to use app_config.
Super, thank you kindly!! | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
These defaults settings may lead to drop of overall PG performance because MT is not optimal for SGS, PPSE, and PPS, and sub-optimal for other LLR projects. I did some benchmarks and many low-level customer CPUs are working worse in MT LLR mode. First reason is an inefficient implementation of MT in LLR - threads are waiting for each other, causing only 80-90% of CPU load per thread, and losses are getting worse as a number of thread grows. Additionally, in real Windows world, where Windows itself often load the CPU with some internal background work, LLR will wait for these delayed threads which will lead to further decrease of performance. Even 1 delayed thread will also stop all other threads. This will not happen if we have few independent LLR processes.
I think default setting must be 1 core / no limits, to emulate old behavior. LLR is not an app which could be blindly scaled to N cores, it MT implementation is far from ideal. An user must intentionally enable this mode after test and benchmarks.
I tend to agree.
Simulate old behaviour as default and let participants change their settings.
With multi-core CPUs, you get greater drop using all cores on single task. Even worst scenario with 2-CPUs systems. And with my 2x16cores, ouch.
(I don't my baby-sit or prepare in advance, just pointing out weak spots).
Unfortunately... I have to agree with this.
I'll look into changing the default to single core.
MAX_CPUS should now default to 1.
____________
My lucky number is 75898524288+1 | |
|
streamVolunteer moderator
Project administrator
Volunteer developer
Volunteer tester
Send message
Joined: 1 Mar 14
Posts: 1051
ID: 301928
Credit: 563,881,725
RAC: 1,034
|
At any rate, how does these new settings play if you change your preferences midway through jobs?
At least number of threads cannot be changed in progress.
If you get a task with 'x' threads, you cannot change this later. You should complete task with this setting or cancel it. Even if you didn't started it yet.
As far as I can see from the source, MT support in server is a ugliest hack. All what it does is to append "--nthreads x" to task command line before sending this task to client. The command line is opaque for client which passes it to the application/wrapper as is. Of course it cannot be changed on the fly.
Oh crap. What happens if you'll use app_config.xml? Then full app command line will contain both "--nthreads" (from server) and "-t" (from app_config). It will not work because wrapper is way too simple and can accept only one argument.
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Oh crap. What happens if you'll use app_config.xml? Then full app command line will contain both "--nthreads" (from server) and "-t" (from app_config). It will not work because wrapper is way too simple and can accept only one argument.
Let's discuss this on Discord.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
Unless something changes, I plan on turning on the new multi-threading controls on September 3rd.
____________
My lucky number is 75898524288+1 | |
|
streamVolunteer moderator
Project administrator
Volunteer developer
Volunteer tester
Send message
Joined: 1 Mar 14
Posts: 1051
ID: 301928
Credit: 563,881,725
RAC: 1,034
|
At any rate, how does these new settings play if you change your preferences midway through jobs?
At least number of threads cannot be changed in progress.
If you get a task with 'x' threads, you cannot change this later. You should complete task with this setting or cancel it. Even if you didn't started it yet.
I was wrong. There are two parts of command line used to run a program. First part is assigned to specific application version. In our case, it contain '--nthreads X' and could be changed by server. So, if this setting is updated somehow, existing tasks could be (re)started with new setting.
Also, it's absolutely safe to use app_config.xml because app_config will completely replace (not append) command line sent by server with user-specified value.
The second part of command line is workunit-specific. It indeed could not be changed and will be appended to previous part of command line, which may have side effects. But it's not used for LLR.
The only unknown thing now is when server sends new application-specific information (including possibly changed number of threads) to client. Look like a simple click on "update" button (with "got 0 new tasks") does not work. You must get at least one new task from server to obtain new information about their application (including new number of threads) as well.
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
I have made some purely cosmetic changes to the text on the project preferences pages. Hopefully this will make the job and thread controls more intuitive.
For example,
Max # Jobs
becomes
Max # of simultaneous PrimeGrid tasks
(Sets the maximum # of PrimeGrid tasks, including GPU tasks)
____________
My lucky number is 75898524288+1 | |
|
|
|
|
It would be nice to have an option to "use app_config settings" (or an explanation text next to new web controls) for those who wish to continue using app_config and not the new mt-controlling interface. With this setting the server would not send anything extra to the client, like today, and the client will solely rely on local configs. Thank you. | |
|
|
|
|
Hi.
For me nothing seems to change apart from the fact that I have to go over my app_config files to apply the plan_class, as I use hyper-threading, but not on all of my machines.
Just switching the Boinc manager to only run on the number of physical cores is an option I don't want to use on anything than my main machine because it limits the throughput of other projects' tasks that actually benefit from running on all threads.
Having my machines running unattended has me needing to tweak things to the best and not to the options the server gives me.
Thanks anyway for making things easier for users that don't want to fiddle with complicated stuff!
____________
Greetings, Jens
147433824^131072+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
It would be nice to have an option to "use app_config settings" (or an explanation text next to new web controls) for those who wish to continue using app_config and not the new mt-controlling interface. With this setting the server would not send anything extra to the client, like today, and the client will solely rely on local configs. Thank you.
Since app_config overrides the new web settings, that's implicitly how it works all the time.
____________
My lucky number is 75898524288+1 | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
I have made some purely cosmetic changes to the text on the project preferences pages. Hopefully this will make the job and thread controls more intuitive.
For example,
Max # Jobs
becomes
Max # of simultaneous PrimeGrid tasks
(Sets the maximum # of PrimeGrid tasks, including GPU tasks)
The 'max jobs' part is a problem for me too...are you saying that if I use 'max jobs' and set it to 7 for instance on an 8 core pc, and then set 'max cpu's' to 4 that I can then run one LLR task using 4 cpu cores, 1 gpu task using 1 cpu core and 2 other non LLR tasks as well?
My problem stems from the 'max jobs' being set in the current app_config file to one for example and the same 8 core pc running 1 LLR task on 4 cpu cores and then no other cpu cores are used, for non LLR tasks, because of the 'max jobs set to one' setting.
Another quirk in there someplace my gpu's are crunching even though the 'max jobs' is set to only one, I don't know why and don't want to lose that if possible. Could it be because I do not run the same Project on the gpu's and cpu's in the same pc? And the other Project doesn't 'see' the app_config file in the PG folder? Maybe...but then why don't other Projects cpu tasks fill up the rest of the cpu cores left empty? My settings in Boinc are to use 99% of the cpu cores, leaving one free for the gpu to use and then I manage the MT tasks thru an app_config file. But not having to do that, along with the resultant changes every time I go from LLR app to LLR app, without having a very long app_config file would be very nice and easy. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
I have made some purely cosmetic changes to the text on the project preferences pages. Hopefully this will make the job and thread controls more intuitive.
For example,
Max # Jobs
becomes
Max # of simultaneous PrimeGrid tasks
(Sets the maximum # of PrimeGrid tasks, including GPU tasks)
The 'max jobs' part is a problem for me too...are you saying that if I use 'max jobs' and set it to 7 for instance on an 8 core pc, and then set 'max cpu's' to 4 that I can then run one LLR task using 4 cpu cores, 1 gpu task using 1 cpu core and 2 other non LLR tasks as well?
Yes, since you would be setting the limit at 7 tasks and you're only running 4 tasks. However, I would recommend simply leaving "max jobs" set to "no limit".
My problem stems from the 'max jobs' being set in the current app_config file to one for example and the same 8 core pc running 1 LLR task on 4 cpu cores and then no other cpu cores are used, for non LLR tasks, because of the 'max jobs set to one' setting.
If that's what you want to do (reserve half the CPU for one type of task), then you'll need to continue using app_config. The website controls work best when you're running only one type of CPU task.
Another quirk in there someplace my gpu's are crunching...
Nobody can answer that without seeing your complete app_config file. However, please don't post that here. If you want to pursue this, create a new thread and put it there.
____________
My lucky number is 75898524288+1 | |
|
mikey
Send message
Joined: 17 Mar 09
Posts: 1899
ID: 37043
Credit: 827,008,081
RAC: 650,056
|
I have made some purely cosmetic changes to the text on the project preferences pages. Hopefully this will make the job and thread controls more intuitive.
For example,
Max # Jobs
becomes
Max # of simultaneous PrimeGrid tasks
(Sets the maximum # of PrimeGrid tasks, including GPU tasks)
The 'max jobs' part is a problem for me too...are you saying that if I use 'max jobs' and set it to 7 for instance on an 8 core pc, and then set 'max cpu's' to 4 that I can then run one LLR task using 4 cpu cores, 1 gpu task using 1 cpu core and 2 other non LLR tasks as well?
Yes, since you would be setting the limit at 7 tasks and you're only running 4 tasks. However, I would recommend simply leaving "max jobs" set to "no limit".
My problem stems from the 'max jobs' being set in the current app_config file to one for example and the same 8 core pc running 1 LLR task on 4 cpu cores and then no other cpu cores are used, for non LLR tasks, because of the 'max jobs set to one' setting.
If that's what you want to do (reserve half the CPU for one type of task), then you'll need to continue using app_config. The website controls work best when you're running only one type of CPU task.
Another quirk in there someplace my gpu's are crunching...
Nobody can answer that without seeing your complete app_config file. However, please don't post that here. If you want to pursue this, create a new thread and put it there.
I have enough answers for now and will wait until after the 3rd when you release the new settings to see what happens next, and before starting a new thread. Thank you very much for you help!! | |
|
mackerelVolunteer tester
Send message
Joined: 2 Oct 08
Posts: 2652
ID: 29980
Credit: 570,442,335
RAC: 7,565
      
|
|
To check my understanding, when the change goes live:
1, if there is no valid app_config, settings will be controlled by the project settings
2, if I have NOT modified my existing app_config with the new plan classes, it will stop working, like it is not there
Personally, it will be nice not to have to mess around with app_config any more. If it does no harm to keep out of date ones, can I just set accordingly when the new system goes live and continue? | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
2, if I have NOT modified my existing app_config with the new plan classes, it will stop working, like it is not there
For LLR, yes, it will stop working. App_config for other apps will continue to work since those plan classes are not changing. Only the plan class for LLR apps is changing.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
It would be cool to have a global job control multi-threading preference option and one for each project. That way we would just set everything form the preferences page and never have to mess with app_config again if we didn't want to. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
It would be cool to have a global job control multi-threading preference option and one for each project. That way we would just set everything form the preferences page and never have to mess with app_config again if we didn't want to.
Yes, yes it would.
And I could EASILY do *half* of that. The half where you tell LLR to use -t4 when running SoB and -t1 when running SGS. That I can do.
The part we can't do is tell the BOINC client that the SoB app is using 4 cores and the SGS app is using 1 core. It would end up scheduling either 4 -t4 SoB tasks running at once, or only one -t1 SGS task. The only way to do that, as far as I know, is via app_info.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
I think I'm in the minority here where I didn't mind updating the app_config. :)
I wrote an application a while ago that creates unique config files. The appropriate file gets copied to each machine.
I fully support what's better for the PrimeGrid/BOINC community. Reading about CPU core/thread/HT misinterpretations on Linux machines makes me nervous. I will duplicate the plan_class blocks as suggested prior to the update. Then I will play with one Linux machine to see if the settings properly reflect the number of cores and threads. I will post anything unusual for fellow Linux users to read.
Cheers Michael on another solid enhancement! | |
|
|
|
|
I've been away, giving a shot at Mersenne prime search in GIMPS.
It took me about 9 days to find out that M91740643 is not prime :(
I've just seen those new to be configs for better multi-threading.
Thank you Michael, for spending time to make our lives easier here at PG !
____________
"Accidit in puncto, quod non contingit in anno."
Something that does not occur in a year may, perchance, happen in a moment. | |
|
|
|
|
I am running 5 copies of llrCUL with 4 threads each on my 18/36 cores/thread Intel 9980xe. I have 6 tasks set in "Job Control and Multi-threading" setting and would like to insure that 1 of the WU is allocated to the RTX 2080 Ti GPU.
PrimeGrid keeps allocating a 6th llrCUL WU leaving the GPU idle for the hours that it takes to finish the llrCUL.
Max # of simultaneous PrimeGrid tasks 6
Multi-threading: Max # of threads for each task 4
What knobs would I set on my 36 thread machine to consistently use 21 threads?
5 WU x -t 4 of llrCUL
1 WU x GFN
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
I am running 5 copies of llrCUL with 4 threads each on my 18/36 cores/thread Intel 9980xe. I have 6 tasks set in "Job Control and Multi-threading" setting and would like to insure that 1 of the WU is allocated to the RTX 2080 Ti GPU.
PrimeGrid keeps allocating a 6th llrCUL WU leaving the GPU idle for the hours that it takes to finish the llrCUL.
Max # of simultaneous PrimeGrid tasks 6
Multi-threading: Max # of threads for each task 4
What knobs would I set on my 36 thread machine to consistently use 21 threads?
5 WU x -t 4 of llrCUL
1 WU x GFN
1) Pretend the "Max jobs" button doesn't exist and set it to "no limit". (This is good advice for anyone unless you're trying to allocate some tasks to PG and some tasks to a different project. That's what Berkeley designed this control to do.)
2) Set "max cpus" to 4.
3) In the BOINC manager, set "use at most nn% of the CPUs" to 53% (for 20 /38 cores).
In general terms, always set max_jobs to "no limit", set "max CPUs" to the number of threads per task, and use the BOINC manager's "use nn%" of the CPUs" (or the BIOS HT control) to limit the total number of threads to be used.
____________
My lucky number is 75898524288+1 | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
The new multi-threading system is live!
____________
My lucky number is 75898524288+1 | |
|
|
|
|
and now I have Quad cores with no app_config file running a singe prime grid task on all 4 cores and pausing all the other projects. Better multi threading? not so much for me. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
and now I have Quad cores with no app_config file running a singe prime grid task on all 4 cores and pausing all the other projects. Better multi threading? not so much for me.
There's an unexpected side effect.
If you go and change your "PrimeGrid Preferences", make sure the max threads setting is set to 1, and then save the settings.
Then hit update. Once the current task completes, things will return to normal.
____________
My lucky number is 75898524288+1 | |
|
GDBSend message
Joined: 15 Nov 11
Posts: 304
ID: 119185
Credit: 4,284,170,186
RAC: 1,753,837
|
|
I used to have control of PG apps via app_config file.
Now you went 737MAX on me, and I no longer have
the ability to EASILY manage PG LLR apps.
If I run any PG apps now, it will be only be apps
that I control, like PRPNET or manual sieving only. | |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
|
I've applied a global fix that should correct the problem for those who were affected.
GDB, you always DID have control. The problem only affected people who haven't changed their preferences recently. All you needed to do to fix the problem was go to preferences and save them. Even that's no longer necessary. Essentially, I've done that for everyone.
All should be good now -- or at least will be as soon as your computer fetches more tasks or you hit the update button. The fix only affects tasks downloaded from this point forward.
If anyone is still having problems, please let me know.
____________
My lucky number is 75898524288+1 | |
|
GregCSend message
Joined: 12 Nov 18
Posts: 57
ID: 1077873
Credit: 2,300,051,701
RAC: 6,855,667
|
|
Working perfectly for me now, once new tasks were downloaded. Easy to tweak without app_config. Thanks!! | |
|
|
|
|
Thanks for this update Michael, and all others who were involved behind the scenes.
It seems that most of my PCs are memory constrained rather than CPU constrained, so having fewer tasks in progress, but processed quicker, leads to some dramatic speed increases even for relatively small workunits :)
____________
 | |
|
|
|
|
Okidoki.... From reading this thread and others, I am seeing that an APP_Config file regardless of the setting in Primegrid Preferences should override and independantly control individuall LLR Apps.
So for PPS, PPSE, PPS Mega and TPS LLR's i have the following excerpt from App_Config
<app>
<app_name>llrPPS</app_name>
<max_concurrent>4</max_concurrent>
</app>
<app_version>
<app_name>llrPPS</app_name>
<plan_class>mt</plan_class>
<cmdline>-t 1</cmdline>
<avg_ncpus>1</avg_ncpus>
<max_ncpus>1</max_ncpus>
</app_version>
<app>
<app_name>llrTPS</app_name>
<max_concurrent>4</max_concurrent>
</app>
<app_version>
<app_name>llrTPS</app_name>
<plan_class>mt</plan_class>
<cmdline>-t 1</cmdline>
<avg_ncpus>1</avg_ncpus>
<max_ncpus>1</max_ncpus>
</app_version>
<app>
<app_name>llrPPSE</app_name>
<max_concurrent>4</max_concurrent>
</app>
<app_version>
<app_name>llrPPSE</app_name>
<plan_class>mt</plan_class>
<cmdline>-t 1</cmdline>
<avg_ncpus>1</avg_ncpus>
<max_ncpus>1</max_ncpus>
</app_version>
<app>
<app_name>llrMEGA</app_name>
<max_concurrent>2</max_concurrent>
</app>
<app_version>
<app_name>llrMEGA</app_name>
<plan_class>mt</plan_class>
<cmdline>-t 2</cmdline>
<avg_ncpus>2</avg_ncpus>
<max_ncpus>2</max_ncpus>
</app_version>
I would assume based on past working experience that this would allow 4 concurrent instances of llrTPS to run on an individual core each.... This has worked in the past.
Since the change All Sophie Germaine tasks and only Sophie Germaine Tasks are now running on 4 cores... (I7 3770K running 4 cores - 50% CPU Usage in settings to avoid HT.)
This does not happen when running PPSE or PPS.... 4 tasks run concurrently with one task per core.... PPSMega i have 2 tasks running with two cores per task but the TPS (SGS) is allocating 4 cores to a single task despite a virtually identical <APP></APP> reference as PPS... (Basically the same as PPS but with P removed and a T replaced in a copy / replace process)
The PPS's are all running to plan... as expected... just not the TPS?
Any thoughts.... Currrently in Prime Grid Preferences i have no limit on work units, with a maximum of 4 Threads per Task set.? I have set Boinc to use 50% of CPUS' to avoid HT on an I7-3770K
Any suggestions would be appreciated.
Cheers
Glen
| |
|
Michael GoetzVolunteer moderator
Project administrator
Send message
Joined: 21 Jan 10
Posts: 14037
ID: 53948
Credit: 478,439,679
RAC: 361,518
|
Any suggestions would be appreciated.
Cheers
Glen
I'm not really sure what's going on there either, but I did notice two things.
The first is that the most recent SGS task ran -t1, as you wanted.
The second is that this computer appears to be assigned to the "Home" venue, but you don't seem to have a "Home" venue defined. I'd try either creating "Home", or moving the computer to either "---". Judging by your expectations, "---" (the default venue) is where this computer should be.
____________
My lucky number is 75898524288+1 | |
|
|
|
|
I am running a 9700k with 8 cores.
Everything was working fine with my app_config file with the adde |
|
