Effective Immediately:

LLR v3.8.20 is live for SoB. (It's v8.00 in BOINC)

Other LLR projects will be upgraded in the near future.

You may use app_config to enable multithreading if you so desire.

We can't support 32 bit Macs anymore due to compiler incompatibilities, so once 8.00 goes live for a particular app, there will be no more 32 bit mac tasks available. (There was only a single 32-bit Mac successfully running tasks at PrimeGrid.)
____________
My lucky number is 75898⁵²⁴²⁸⁸+1

3.8.20 is now live for all LLR projects.
____________
My lucky number is 75898⁵²⁴²⁸⁸+1

You may use app_config to enable multithreading if you so desire.

You may use app_config to enable multithreading if you so desire.

How would I go about that? I got an app_info.xml working, but then I couldn't do Genefer at the same time. With a simple app_config, it said it couldn't find the apps named llrPPS or llrPPSE.

Thu 16 Mar 2017 04:58:33 PM MDT | PrimeGrid | Entry in app_config.xml for app 'llrPPSE', plan class '' doesn't match any app versions

Hello,
i have a question:
I get LLR 3.8.20 to work under Ubuntu 17.04, but it seems i make something wrong.
http://www.primegrid.com/show_host_detail.php?hostid=537602

I can't seem to get it to run for llrMEGA. Created an app_config.xml file just as Mike said and placed in the correct directory for Win 7. Help!
[/img]

http://www.primegrid.com/forum_thread.php?id=7348&nowrap=true#106105

Thank you. That was exactly what I was thinking about. This method will kill the competition on LLR tests, but it will cost efficiency too./OT

Surprisingly, you may find it increases efficiency.

Hi, I'm experiencing a strange behaviour on my host (id: 518288). I set the app_config.xml for llrMEGA app. Should I not do it? My host downloads 4 tasks and it finishes the first one in 1000s, the second one in 2000s, the third one in 3000s and the fourth one in 3700s. The expected time should be 1000s per multithreaded task, that is a worse time than the run time (~3600s) for 4 single-threaded tasks.

1000*4 > 3600.

http://www.primegrid.com/forum_thread.php?id=7348&nowrap=true#106105

Thank you. That was exactly what I was thinking about. This method will kill the competition on LLR tests, but it will cost efficiency too./OT

Surprisingly, you may find it increases efficiency.

Hi, I'm experiencing a strange behaviour on my host (id: 518288). I set the app_config.xml for llrMEGA app. Should I not do it? My host downloads 4 tasks and it finishes the first one in 1000s, the second one in 2000s, the third one in 3000s and the fourth one in 3700s. The expected time should be 1000s per multithreaded task, that is a worse time than the run time (~3600s) for 4 single-threaded tasks.

1000*4 > 3600.

http://www.primegrid.com/forum_thread.php?id=7348&nowrap=true#106105

Thank you. That was exactly what I was thinking about. This method will kill the competition on LLR tests, but it will cost efficiency too./OT

Surprisingly, you may find it increases efficiency.

Hi, I'm experiencing a strange behaviour on my host (id: 518288). I set the app_config.xml for llrMEGA app. Should I not do it? My host downloads 4 tasks and it finishes the first one in 1000s, the second one in 2000s, the third one in 3000s and the fourth one in 3700s. The expected time should be 1000s per multithreaded task, that is a worse time than the run time (~3600s) for 4 single-threaded tasks.

1000*4 > 3600.

I see. I didn't notice that the server-side run times were not the real run times (~990s). I thought my tasks were lasting more time than needed one for some unknown reason or bug. Actually, it was too strange to be real.

Why do you not reset the run time counter for every task? Is it not possible?

Can someone make up a reference app_config.xml with ALL LLR projects in it, or at least point to a list of the correct names to use to reference subprojects? I'm trying to patch one up and searching the forum isn't productive.

Edit: am I misunderstanding something here, that applications do not correlate with subprojects? If so, which applications apply to which subprojects?

Edit2: more by chance than any knowledge on my part I seem to have got it working, but would still appreciate if it could be cleared up how to configure subprojects separately as far as practical.

point to a list of the correct names to use to reference subprojects?

Thanks both. Looks like my guesses weren't far off, but incomplete and will have to update later. Doing thread scaling testing on PSP now.

I also think I got confused when boinc client appeared to report an error, now I suspect it only does so if a subproject hasn't been run on a particular system before.

I also think I got confused when boinc client appeared to report an error, now I suspect it only does so if a subproject hasn't been run on a particular system before.

Some early PSP multi-thread results, which are interesting...

PSP work is currently 1280k FFT, or near enough 10MB of ram required per task, so even a single task wont fit in the cache of consumer level Intel CPUs. Times listed below in kiloseconds are total CPU time (roughly cores multiplied by wall clock time). Times listed in hours are elapsed time.

i5-4570S which runs at 3.2 GHz all cores loaded, and has 6MB L3 cache, and dual channel dual rank DDR3 at 2400. Running units one per core took 92.2ks average, going down to 90.8ks (6.3h) for 4 threads. So 4t is 4.06x faster than single. So, not a significant boost, but no loss either. Overall a benefit as you return tasks faster.

i5-5675C OC to 3.5 GHz all cores loaded, and has 4MB L3 cache and 128MB L4 cache. Running units one per core took 77.0ks average, going UP to 81.9ks (5.7h) for 4 threads. So 4t is only 3.76x faster than single. I think this is a unique situation with the smaller L3, and huge L4 cache, that running singles isn't really limited, and going 4t overheads were not insignificant.

i7-6700k OC to 4.2 GHz all cores loaded, with 8MB L3 cache, and dual channel dual rank DDR4 2666. Running units one per core took 71.1ks average, going down to 61.0ks (4.2h) for 4 threads. So 4t is 4.67x faster than single. That is a nice boost.

I need to get some tests going on my 6600k to see if the cache helps the i7 or not.

For fun:
Xeon E5-2683v3 14 cores @ 2.3 GHz, estimating around 152ks total but that is only 3 hours in real time! I suspect there is some scaling problem as the CPU is only loaded around 85%. I want to try 2x7t later and see if that helps, but it will be a tradedoff of throughput and return time.
Ryzen 1700 8 cores @ 3.2, estimating 152ks (5.3h). That puts it comparable throughput to the i5 systems before but IPC is way down.

For fun:
Xeon E5-2683v3 14 cores @ 2.3 GHz, estimating around 152ks total but that is only 3 hours in real time! I suspect there is some scaling problem as the CPU is only loaded around 85%. I want to try 2x7t later and see if that helps, but it will be a tradedoff of throughput and return time.

For fun:
Xeon E5-2683v3 14 cores @ 2.3 GHz, estimating around 152ks total but that is only 3 hours in real time! I suspect there is some scaling problem as the CPU is only loaded around 85%. I want to try 2x7t later and see if that helps, but it will be a tradedoff of throughput and return time.

Once you see CPU usage below what you would expect from the number of threads running, backing off thread count may actually improve return time--so perhaps no tradeoff.

Can you try 13 cores?
Or in general, one core idle scenario (Ryzen).

What is the formula to check if it can fit?

My Phenom II X6 1100T CPU has 6 cores and 6 MB of L3 Cache.
https://en.wikipedia.org/wiki/List_of_AMD_Phenom_microprocessors#.22Thuban.22_.28E0.2C_45_nm.2C_Hexa-core.29
Based on FFT size I would expect SR5, PPS and SGS to scale well, but others not so much.
Mike has conveniently posted a table of FFT size vs Candidate Digits:
http://www.primegrid.com/forum_thread.php?id=7093&nowrap=true#105501
I'll have to do some testing to confirm this; 1x6 vs 2x3 vs 3x2 vs 6x1 (threads x instance)

Mike has conveniently posted a table of FFT size vs Candidate Digits:
http://www.primegrid.com/forum_thread.php?id=7093&nowrap=true#105501

I'm running the new LLR on three cores out of four (i56600) and run two GFN on my two GPUs, 960 & 1060.
I noticed the following:
1) The GFN CPU jobs takes together about 10% of the CPU and the GPU load is below 90%.
2) When I suspend the three LLR jobs the CPU used by the GFN jobs drops to about 2%-3% and the GPU load rises into the nineties.
Does the new llr uses the GPU?

Assuming that memory bandwidth is a limiting factor that impairs the GPUs how does that explain why the genefer jobs consume more cpu time in the presence of LLR jobs.

I monitored the machine closely since I installed the 1060 a while back and didn't notice anything until the new LLR version came along.

I will however keep testing.

Question: Can I use LLR v3.8.20 also on PRPNet ?

Question: Can I use LLR v3.8.20 also on PRPNet ?

@echo off IF "%~1"=="" ( echo. No command lines arguments passed cllr64.exe -t2 ) ELSE ( IF "%~2"=="" ( REM echo. One command line argument passed %1 cllr64.exe %1 -t2 ) ELSE ( IF "%~3"=="" ( REM echo. Two command line arguments passed %1 %2 cllr64.exe %1 %2 -t2 ) ELSE ( IF "%~4"=="" ( REM echo. Three command line arguments passed %1 %2 %3 cllr64.exe %1 %2 %3 -t2 ) ELSE ( IF "%~5"=="" ( REM echo. Four command line arguments passed %1 %2 %3 %4 cllr64.exe %1 %2 %3 %4 -t2 ) ELSE ( echo. Five or more command line arguments passed ) ) ) ) )

llrexe=cllr64.bat

I haven't tried it, but I think this much simpler batch file will do the same thing:

Quick data point comparing i7-6700k 4.2 GHz 2666 ram, to i5-6600k 4.2 GHz 3000 ram, running PSP with 4 threads, the i5 took about 4% longer. It isn't entirely clear but it doesn't seem to suggest either a strong influence of ram or cache quantity.

The 14 core Xeon is more interesting.
Running 14 threads, PSP tasks took 129.1k CPU seconds in 11.0ks real time, for 84% utilisation.
Running 12 threads, PSP tasks took 126.0k CPU seconds in 11.9ks real time, for 88% utilisation. So less CPU time in more real time than 14.
Running 10 threads, a PSP task took 124.1k CPU seconds in 13.5ks real time, for 92% utilisation.
Running 7 threads, a PSP task took 121.3k CPU seconds in 18.1ks real time, for 96% utilisation.
Running 2x7 threads, PSP tasks took 121.7k CPU seconds in unknown real time due to reporting weirdness.

Maybe more data points would help, but this confirms my observation that beyond some point, more threads aren't able to saturate the allocated cores. For comparison, the quad core Intels were above 97% utilisation, and Ryzen 8t was about 93%.

Multi-threading is always going to have some inherent inefficiencies due to the need for threads to pause while they wait to synchronize with the other threads. The more threads you run, the larger the overhead. The tradeoff is whether you gain more speed because of improved cache efficiencies from running a single task than you lose from thread synchronization.

On my Haswell i5-4670K, -t4 only runs at about 90% CPU utilization, but it's still usually more efficient than running 4 separate tasks. (Task size plays a factor here, of course.) I may lose 10% to the multithreading, but running 4 separate tasks loses even more although CPU utilization "rises" to 100%. The CPU may be fully occupied, but in reality it's running slower due to memory bottlenecks.

Here's some numbers from a test I ran on one of our 8-core servers:

I haven't tried it, but I think this much simpler batch file will do the same thing:

@echo off cllr64.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 -t2

I witnessed the same thing. However, with CPUs that support hyperthreading, I have been successful turning that on and limiting CPU threads to the physical cores. The "free" virtual cores seem to be enough to feed the GPUs.

On both a 6700K and a 5930K I have 4 and 6 threads worth of LLR, with HT on, and there is no discernable impact to the GPU tasks (GFN20) on the 1080 in the former and the 2 W9100s in the latter. If I ran 4 and 6 threads with HT off, it crushed the GPU tasks.

This also seems to work on an i3-4130 with a 290x.

I kept on testing, trying various configuration and I found out that a 4coreLLR task interferes severely with the GPUs, 3coreLLR task leaves the GPUs alone but chokes the 1coreLLR task that uses the 4th core.

It is probably Voodoo but the following configuration works fine for me, running 4 BOINC clients:

Client #1: GPU #0 (1060), genefer20, No CPU task

Client #2: GPU #1 (960), AP27 | genefer17mega, No CPU task

Client #3: No GPU task, LLR SOB (2 cores)

Client #4: No GPU task, LLR MEGA (2 cores) | LLR SGS (2 cores)

All clients use zero queue.

Did you do that via processor affinity and are you under Linux or Windows?

Not sure whether it has been reported already, or should be reported elsewhere:

Multithreaded LLR tasks are reported with wrong runtime. The runtime which is listed in the results tables on this web site are all over the place, anywhere between the actual runtime, and the actual runtime times number of threads.

Obviously there is a race condition in the runtime accounting. Is this done in the application, or in the boinc client, or...?

As an example, see this results list: http://www.primegrid.com/results.php?hostid=530208

This host with 2 sockets, 14 cores per socket, 2.9 GHz all-core AVX turbo, has been working on PSP-LLR with <cmdline>-t 7</cmdline> for more than two weeks now. Timestamp of app_config.xml is April 4, 06:00 UTC. Right now, the boincmgr UI is showing an estimated task duration of 8h40m = 31,000 s. (Average runtimes earlier in April were 5h30m...6h on this host with -t 7, but WUs have gotten bigger since then.)

Is there any more information on which PG tasks do not benefit from MT?
I've seen SGS mentioned in several threads, that it's never more effiient in MT mode than on discrete cores.
Which other tasks will not benefit from a -t4 on a bread and butter four core CPU?
Other than being able to push out a few more WUs before the end of a challenge, of course..
Cheers
Holger

There are a lot of variables and it is hard to say what is "best". There are two slightly different goals: doing units as fast as possible, doing as many units in a given time as possible.

In the following, when I refer to cores, I mean physical cores, not any extra threads from HT or SMT. While I have not tested it, it is assumed they still don't provide a significant benefit apart from possibly helping drive a GPU at the same time if you need to.

Doing LARGE units as fast as possible seems to be as simple as use all the cores you have. There seems to be some difficulty scaling beyond 8 cores where each core seems unable to be fully loaded.

In observation from the PSP challenge, it looks like ram bandwidth is far less important than it was running a separate task on each core. Similarly, it doesn't look like cache size plays much of a role, so a value sweet spot for LLR may have just moved from i3s to i5s.

For small tasks, there seems to be some inefficiency. Personally I wouldn't run 4 core on anything MEGA or smaller. It might be worth it on a dual, but I haven't tested it.

The one issue with i5s is the lack of HT, if you are trying to drive a GPU. I had to keep my i5s restricted to 3 threads or the GPU would starve.

I took an even more conservative approach, I didn't scale any units past six cores. I noticed the same drop in utilization if I went beyond that.

I just reported my first SoB using multithreading (-t4) and the run time was not what I was expecting (i7 6700k) with the unit tooking a bit over 7 hrs to complete (FFT length 1280K). I have units also running on my two other Skylakes so as to build a better overall picture.

HT is beneficial to throughput with multithreading because of better CPU utilization, typically when a thread stalls due to a cache miss. Here's a demonstration of 8.4% throughput increase using HT and multithreading just 1 task, compared to running 1 task per core with HT off.

5820K (Haswell-E) 3.3 GHz no OC; 6 physical cores, HT on (12 threads), 15 MB L3 cache; 16 GB RAM
Stock fan, speed is set in the BIOS to 100% to keep higher core temperatures from throttling the speed.

6 physical cores, HT on, 1 task, 10 threads

BOINC computing preference 100% (so 12 threads allowed).
appconfig: llr321 -t 10

CPU loading is one task of LLR321 and one task of GFN22 simultaneously. No CPU affinity setting, letting Linux manage threads. The speed of GFN 22 is not impacted, as there are 2 threads available for GFN. GPU utilization is 99% even with only 1 thread for GFN.

LLR321 FMA3 FFT length 768k
According to BOINC task properties: virtual memory size ~ 153 MB, working set size ~ 47 MB. There is plenty of RAM to hold the task, but the working set (memory used by data to solve the problem) does not fit in the CPU cache. One FFT fits in the L3 cache but there will be frequent cache misses for the rest of the working set.

Task CPU time 77,528s = 21h32m8s
Actual run time 8,901s = 2h28m21s
Average thread loading = run time / CPU time = 8.71 threads
Thread efficiency = thread loading / threads occupied = 8.71 / 10 = 87%.
Multithreading overhead (by Micheal's definition) = 10 / 8.71 - 1 = 14.8%

But look at this:
Core efficiency = thread loading / physical cores = 8.71 / 6 = 145%

The increase in core efficiency more than compensates for the multithreading overhead, for a net increase in throughput of 8.4%, calculated here...

6 physical cores, HT off, 6 tasks, 1 thread per task

This configuration was not directly measured, but uses mackerel's estimate of a 10% performance drop when HT is left on.

HT on
BOINC computing preference at 50% (so 6 threads allowed)
no multithreading

BOINC ran a GFN22 task and 6 LLR321 tasks in parallel. The LLR average run time (also average CPU time) was 18h2m. The average LLR task completion time 3h. Now depending on how you measure a 10% performance drop, the computed runtime with HT off would be either 10,800s / 1.1 = 9,818s or 10,800s * 0.9 = 9,720s. Just to please the skeptics, let's take the more aggressive benefits of turning off HT, with a runtime of 9,720s.

Change in Throughput

The reduction in average runtime by using multithreading and more threads than cores with HT on is 9,270s - 8,901s = 819s. This is an increase in throughput of 819 / 9,720 = 8.4%, compared to running 1 task per core on all cores with HT off.

As Micheal said, the cost of using more threads to get faster task completion is lower CPU efficiency. Other effects are higher CPU temperature and higher energy usage per task. With only physical cores, using more threads incurs a performance penalty in terms of lower throughput.

When a hyperthread stalls, the other hyperthread on the core may be able to execute, and the result can be higher throughput.

I haven't yet explored the best number of threads for highest throughput or better energy efficiency with the same throughput, but so far I'm pretty happy with -t 10. I can't play with other LLR loads right now since Rafael is getting his payback with this computer for helping me on the TRP sieve. I invite others to turn on HT, try running a single LLR with more threads than cores, and relay their result back here.

I suppose now is as good a time as any for me to try and test this...

I've got a 6700k idle, will set up a batch file to run (with HT on) 8, 6, 4, 3, 2, 1 on the same TBD task. I'll probably try one of the previous SR5 units as an initial test, then expand to bigger projects once I got the above working as intended.

Edit: for my interests I'm only concerned about CPU performance. Adding a GPU complicates matters somewhat...

Tune down the number of threads in app_config.xml so that your system runs cooler. If you have the option of CPU fan speed control in your BIOS, crank it up to the highest tolerable noise level.

Summer is indeed just around the block. It's snowing again as I look out the window right now.

Tune down the number of threads in app_config.xml so that your system runs cooler. If you have the option of CPU fan speed control in your BIOS, crank it up to the highest tolerable noise level.

I said...

Average thread loading = run time / CPU time = 8.71 threads

-t time (s) relative
1 8168 1.00
2 4410 1.85
3 3016 2.71
4 2347 3.48
5 2337 3.50
6 2245 3.64
7 2191 3.73
8 2108 3.87

This is for the previously "bug" SR5 unit 64598*5^2318694-1. Ran 1 through 8 threads on i7-6700k fixed at 4.2 GHz, HT enabled, 2666 dual rank dual channel ram. It does look like running on those extra threads does claw towards the ideal scaling for 4 cores. Running 8 threads over 4 gains about 11%. Those feeding GPUs can probably spare one without much impact to CPU, but someone else can test that.

I'll set up 4 tasks in parallel to run while I'm at work, for comparison against the old case of running one task per core. I can't quickly set HT off or affinity on the system so it'll likely have the HT penalty in this state.

I could also repeat the scaling on an i5-6600k too... smaller cache, faster ram, no HT.

Edit: I don't believe the 11% I saw was the HT penalty, as running with -t makes LLR appear to set affinity of each to a pair of threads. Let's see if the i5 result scales similar up to 4 cores which would suggest this, or if it scales better, it could still be HT penalty.

No, and I wouldn't expect it to do 4 in that time either. That is why I'm running 4 in parallel to see how long that takes, and we can work out a benefit from that. While I expect the tasks to complete long before then, I wont be able to see the results for another 7 hours or so.

We are exploring the benefits of HT with multithreading one task. So far it's looking like this will produce a winning combination: minimizing a task's runtime as well as maximizing the CPU's throughput.

I'd like to see the impact on throughput of HT and multithreading in other combinations of tasks and threads on 4 cores:
2 tasks with -t 4
3 tasks with -t 2
4 tasks with -t 2
and the wildcard: 3 tasks with -t 3

I also have an interesting observation:

For LLR 1 task and using CPU affinity for all threads, the task's master thread has more % CPU occupancy than all the other threads of the task (~ 97% versus ~ 82% for 10 threads). If other CPU-intensive work occupies the other hyperthread on the same core as the master thread, then all the LLR task's threads have reduced CPU occupancy (and probably longer runtime). This tells me that the master thread of each simultaneous task should be pinned to its own physical core for maximum throughput. However, I don't think there's a mechanism for BOINC to handle CPU affinity. Too bad. Can we fork BOINC?
Edit; or maybe a better idea would be to run a PG coprocess that communicates with wrappers to hand out CPU assignments.

For small tasks, the scaling of multiple smaller tasks with thread number is potentially interesting. In this scenario, multiple -t 2 or -t 4 might be tested. I might have a go at this once I'm done with big tasks.

For big tasks, I doubt multiple numbers of fewer threads would help in that. I'll leave that as an exercise for someone else.

As for benefit from manual affinity setting, I'll await the i5 test results before speculating in depth. It does look like multithread LLR already does some affinity itself.

Ok, some more data in.

Running 4 single thread tasks it took 18% longer than running four of tasks with 4 threads, or 31% longer than running four of tasks with 8 threads. Remember this is with HT on without affinity on the single tasks, so past experience suggests about 10% penalty. I'm re-running -t 4 now with HT off to check this.

The 6600k has also completed a set. Like for like against the i7, it is 1 to 2% faster for -t 1 through 3. The -t 4 result is obviously too long and I suspect some windows background task has interfered with that, and am re-running it now.

For now, it looks like there is no significant benefit to be gained from extra affinity tinkering when running more than one thread per task.

The 6600k is set to same clock speed as the 6700k, but has smaller cache, and faster ram. I think ram speed influence is what I'll look at after the current tests are done.

Once I have the file in place, what are the steps to get started with it? (At the moment I am running 4 tasks in parallel.)

"place app_config.xml" -- where exactly?

For default Windows install it is C:\ProgramData\BOINC\projects\www.primegrid.com

Guys, I am on Debian 8 -- not Windows. Where on this do I put app_config.xml?

I have tried ".BOINC" and I have tried "/var/lib/boinc-client/" and "/etc/boinc-client/"

<app_config> <app> <name>llr321</name> <fraction_done_exact/> </app> <app_version> <app_name>llr321/app_name> <cmdline>-t 4</cmdline> <avg_ncpus>4</avg_ncpus> <max_ncpus>4</max_ncpus> </app_version> </app_config>

Is this correct XML content for app_config.xml for 321?

I am running Debian 8. Where do I put the file?

Once I have the file in place, what are the steps to get started with it? (At the moment I am running 4 tasks in parallel.)