EVGA Forum Thread https://forums.evga.com/Geforce-Drivers-4xxxx-Drop-more-than-23-in-CUDA-Performance-from-the-3xxxx-Drvers-m2884225.aspx

It looks like all the NVIDIA 4xx.xx Drivers on the GTX 1080 Ti Cards are not doing so well on this CUDA Application.
My Testing results are shown here on this Host https://www.primegrid.com/results.php?hostid=502676&offset=0&show_names=0&state=0&appid=11
More info on our Team Forum above.

Yeah I've been staying on Driver 399.07 after hearing about this issue from others on my 1080ti.

Nice job getting attention to this issue -- looks like the nvidia dev forum picked it up. I wonder if a direct link to the ap26 executable and some simple instructions for how to run a reasonable test from a command line would help the devs to look at the issue more closely. I'll bet someone here knows both of those things.

the nvidia forum link is:

https://devtalk.nvidia.com/default/topic/1044539/cuda-programming-and-performance/geforce-drivers-4xx-xx-drop-more-than-2-3-in-cuda-performance-from-the-3xx-xx-drvers/

Posting the link to the source code repository might help them too. https://github.com/ibethune/ap26

We've known about this for a while now. I posted about it in this thread on Oct 20, 2018. http://www.primegrid.com/forum_thread.php?id=8249.

I've reproduced it over and over as well as several other Primegrid members.

We've known about this for a while now. I posted about it in this thread on Oct 20, 2018. http://www.primegrid.com/forum_thread.php?id=8249.

I've reproduced it over and over as well as several other Primegrid members.

We've known about this for a while now. I posted about it in this thread on Oct 20, 2018. http://www.primegrid.com/forum_thread.php?id=8249.

I've reproduced it over and over as well as several other Primegrid members.

Thanks for also posting over on the NVIDIA Sit.

Testing on the same host https://www.primegrid.com/results.php?hostid=502676&offset=0&show_names=0&state=0&appid=11
Last Tested Driver: 397.93

Driver: 417.01 Run time (sec) 2,222.86 CPU time (sec) 40.41
Driver: 411.70 Run time (sec) 2,619.05 CPU time (sec) 45.12
Driver: 416.16 Run time (sec) 2,356.02 CPU time (sec) 43.10
Driver: 416.81 Run time (sec) 2,507.31 CPU time (sec) 44.83
Driver: 416.69 Run time (sec) 2,609.08 CPU time (sec) 46.11
Driver: 399.24 Run time (sec) 860.20 CPU time (sec) 39.38
Driver: 398.36 Run time (sec) 859.27 CPU time (sec) 38.95
Driver: 397.93 Run time (sec) 866.27 CPU time (sec) 39.38
Driver: 391.35 Run time (sec) 851.45 CPU time (sec) 40.19
Driver: 390.65 Run time (sec) 852.15 CPU time (sec) 38.39

Done Testing Drivers

https://devtalk.nvidia.com/default/topic/1044539/cuda-programming-and-performance/geforce-drivers-4xx-xx-drop-more-than-2-3-in-cuda-performance-from-the-3xx-xx-drvers/

https://developer.nvidia.com/nvidia_bug/2450934
https://developer.nvidia.com/nvidia_bug/2450242


I could use some Support from PG in the below Forum Thread.
Not sure how to provide the info and or data they are asking for.
Thanks
https://devtalk.nvidia.com/default/topic/1044539/cuda-programming-and-performance/geforce-drivers-4xx-xx-drop-more-than-2-3-in-cuda-performance-from-the-3xx-xx-drvers/post/5299990/

I have to ask not really Knowing is AP CUDA or Only OpenCL


Never Mind I found it, Really Wish PG would have taken this NIVDIA

Supported platforms:
•Windows: Nvidia GPU1 (OpenCL): 64 bit, AMD/ATI GPU1 (OpenCL): 64 bit, CPU: 64 bit
•Linux: Nvidia GPU1 (OpenCL): 64 bit, AMD/ATI GPU1 (OpenCL): 64 bit, CPU: 64 bit
•Mac2: Nvidia GPU1 (OpenCL): 64 bit, CPU: 64 bit
1 GPU must have a minimum of 1.5 GB of VRAM.
2 Due to an Apple driver bug, no ATI/AMD GPU application is available for Mac.


Deadline: 7 days (up to 30 days)

Recent average CPU time: 25:17:26
Recent average GPU time: 1:19:50

FYI
https://devtalk.nvidia.com/default/topic/1044539/cuda-programming-and-performance/geforce-drivers-4xx-xx-drop-more-than-2-3-in-opencl-performance-from-the-3xx-xx-drivers/post/5303366/#5303366

Topic: GeForce Drivers 4xx.xx drop more than 2/3 in OpenCL Performance from the 3xx.xx Drivers



There is no point in testing newer drivers; I don't expect any changes in this respect. Changes are required in the application if they want to restore performance with the newer drivers.

Current Scenario in ap26 app:

1. App queries CL_KERNEL_WORK_GROUP_SIZE in order to decide local work group size of either 1024 (seems optimal) or 64 (sub-optimal). If app gets value for query <1024 it reduces local work group size to 64 assuming device doesn't support 1024.

2. Nvidia OpenCL Driver changed return value for CL_KERNEL_WORK_GROUP_SIZE from 1024 to 256.

3. App is not using CL_KERNEL_WORK_GROUP_SIZE returned by driver as is, but just choosing a non-optimal local work-group size (64) based on this query.


What should developers do:

• Query CL_KERNEL_WORK_GROUP_SIZE to get just hint about work group size from driver and use it to launch kernel with that specific value. It need not be optimal for all kernels.

• App is free to choose any value from range [1 , CL_DEVICE_MAX_WORK_GROUP_SIZE] to get best possible work group size for different kernels, irrespective of CL_KERNEL_WORK_GROUP_SIZE returned by driver.


Suggestions specific to ap26:

• App can query CL_DEVICE_MAX_WORK_GROUP_SIZE and set work group size accordingly instead of using CL_KERNEL_WORK_GROUP_SIZE.

• Simplest solution for ap26 would be to use 1024 work group size directly if it comes in range [1 , CL_DEVICE_MAX_WORK_GROUP_SIZE].


I don't know how to best communicate the above information to the developers. If there is a good way to do that, please advise.

Hah, was just about to post the same..

Little application code editing needed... glad they figured out the issue

Hah, was just about to post the same..

Little application code editing needed... glad they figured out the issue

Hah, was just about to post the same..

Little application code editing needed... glad they figured out the issue

The Bug Report is still open
https://developer.nvidia.com/nvidia_bug/2450934
Odd that they had not made a comment.

Yes, thanks Nvidia for finding out what changed in the driver!

They just updated it :) Case closed. Non NVIDIA bug. Application specific. It was cool of them to even tell us how to fix it.

Now we need a Primegrid developer to fix the app.

The Topic Title Should Read OpenCL and Not CDUA but have no way to change it.

AP26 - AP27 Search : Geforce Drivers 4xx.xx Drop more than 2/3 in OpenCL Performance from the 3xx.xx Drvers.
____________

Crunching@EVGA The Number One Team in the BOINC Community. Folding@EVGA The Number One Team in the Folding@Home Community.

Thank you for the Topic Title Change.

Any update as to the Coding being updated to support the Newer Drivers for the GTX Graphics Cards?
Tested 417.58 Today and still double the run time from the 3xx.xx Drivers

Thank you, mfl0p !
____________

"Accidit in puncto, quod non contingit in anno."
Something that does not occur in a year may, perchance, happen in a moment.

Any update as to the Coding being updated to support the Newer Drivers for the GTX Graphics Cards?
Tested 417.58 Today and still double the run time from the 3xx.xx Drivers

Code changes are complete. Just waiting on the application builds.

Any update as to the Coding being updated to support the Newer Drivers for the GTX Graphics Cards?
Tested 417.58 Today and still double the run time from the 3xx.xx Drivers

Code changes are complete. Just waiting on the application builds.

Any update as to the Coding being updated to support the Newer Drivers for the GTX Graphics Cards?
Tested 417.58 Today and still double the run time from the 3xx.xx Drivers

Code changes are complete. Just waiting on the application builds.

Any sign of the application builds being done and distributed ?

Any update as to the Coding being updated to support the Newer Drivers for the GTX Graphics Cards?
Tested 417.58 Today and still double the run time from the 3xx.xx Drivers

Code changes are complete. Just waiting on the application builds.

Any sign of the application builds being done and distributed ?

I was just about to ask that same question.

Sometimes our real lives must take precedence over our hobbies.

At the moment, we're missing some manpower and this is why that app isn't available yet.

Hello!

Still no Ap 26 Ap 27 loaded; can't get any wu's!
My first attempt to run the dev server , so may be something not right on my side?



Ps;;;;;
Forgot to update the new venue,just got one wu!!
____________
MyStats


My Badges

win10, rtx 2080, nvidia driver 417.71 -

old app - ~74x seconds
new app - ~620 seconds


win 10, gtx 1070, nvidia driver 417.71 -

New times - ~1400 sec
old times - at least double that 1400 time


CPU usage also GREATLY reduced!

Awesome work mfl0p! 10 series cards back on track, and faster on both 10 and 20 series cards with the new app and latest nvidia drivers.