I'm using a normal speed GTX 590, is there a reason why I get the maxErr exceeded only when N=32768 (on PRPNet) but OK for the other N values?

What would I need to do to get it to run for this N value?

Any help would be appreciated.
____________
147*2^1392930+1 was my first prime number found, others have followed :)

I'm using a normal speed GTX 590, is there a reason why I get the maxErr exceeded only when N=32768 (on PRPNet) but OK for the other N values?

What would I need to do to get it to run for this N value?

Any help would be appreciated.

Thanks both, the upper "B" limits helps to explain a few things, I had been trying to slow it down by building a slower version with a few tweaks, but also wasn't sure about my final results with an "(err=0.5000)".

Out of interest what would be the max "err" value allowed?
____________
147*2^1392930+1 was my first prime number found, others have followed :)

I returned from work this morning(and had to leave again for another job 10 minutes later)
to find all my recent genfer WUs had errored out.
I expected a few of them to do so ,as some of them had already errored out or been aborted
up to 11 times before.But I didn't expect everyone to do so.
I have already completed quite a few WUs at the same clock speed on the same card(460)
and recently validated a WU which had errored out or been aborted 11 times,so wasn't
expecting this problem when I got home.

I have since reduced clock speeds,but as I said earlier I have validated quite a few
on the clock speeds I was running.

The only major changes were running SOBs and turning off hyper threading to hopefully
get a bit of turbo boost on the i7.

Confused john3760

edit: 1 WU has just validated on the lower clock speed . I will keep an eye on the next few.

edit2: Make that two WUs.I still don't understand why I have had to drop the clocks,when they were working alright beforehand.

Hello,

can you pls have a look a my failed wu's

i use an gigabyte gtx460, never overclocked, but the large wu's break after some time

http://www.primegrid.com/result.php?resultid=356749167
http://www.primegrid.com/result.php?resultid=356362009
http://www.primegrid.com/result.php?resultid=355521013


any idea whts wrong ?

rgds

Tabaluga

I also have some strange behaviours, which I yet could not figured out. I have two hosts with GTX560 running, W7, same driver, same distributor, same stock clock, default Shift.

One host with 1.06 stock app, the other with app_info (because of avx) and 2.3 beta. The host with stock app is running fine and all results get validated. The other have a couple of results returned as valid, but inconclusive. Also One of 20 wu's gets a maxerror during crunching. But also a lot of valid and correct results.

Both are only boinc machines, no productive work on them. The good host have also the higher temps, cpu and gpu.

I have no idea how I can track down the problem. Michael, any ideas?

Regards Odi
____________

Hi Michael,

set down the speeds to recommended values

will come back with result of next wu, but looks like i don't get one

found funny messages in boinc log

08.03.2012 20:54:46 | PrimeGrid | Resent lost task genefer_4194304_72892_12
08.03.2012 20:54:46 | PrimeGrid | [error] No app version found for app genefer platform windows_intelx86 ver 106 class cuda32_13; discarding genefer_4194304_72892_12

any idea ?

rgds

Tabaluga

No having any idea what it took or takes to get the program going for the genefers I do wonder if you can add an error trap routine what would allow the program to maybe restart at some point when it encounters these kinds of errors with a counter to keep it from looping. Of course you may already be doing this.

thanks for the reply.
since downclockIng every.thing seems to be Ok.
I wish I had caught it sooner.

thanks john3760

Try setting the clocks to the stock settings and see if that helps:

Core Clock: 675 MHz
shader clock: 1350 MHz
memory clock: 1800 MHz

If this problem occurs, it will probably continue to reoccur. If a maxErr exceeded error is encountered, you can't just go back one step and try again. You would have to restart from the beginning, and there's no guarantee that the problem won't happen again.

i will say in my opinion, a lot of errors are temperature related however that's caused.

Yeah, that one's a little puzzling.

I think you're already running at stock clocks

fixed his problems by swapping parts (GPUs and power supplies) between two computers, and now both computers are working fine.

Try swapping that GPU with a similar GPU in another computer, and see if the problem moves with the GPU or stays in the same computer.

All problems gone out after downclocking MEMORY clock without changing CORE clock. I had finished enough tasks to claim that GTX460 has enough adequate margin of safety for CORE overclocking, but MEMORY clock needs to be reduced to stock speed or so.

That's not surprising at all. It's underclocking the memory clock that resolved the GTX 550 Ti problem, so it makes sense that it would work on other cards too.

and it's possible you need UAC access to get to that information.

I know, but you still reccomend do CORE downclocking, because all info you possesed is Shader clock.

I changed the post at the top to emphasize the memory clock, and I'll try to remember to mention it in the future.

yes, correct never used app_info

the out of sync problem is solved :

restarting boinc did not help, but just detach from primegrid and
reattach solved it.

rgds

Tabaluga

88C seems far too high. I avoid going over 80 on my 580s. Manually set the fan speed to a speed that brings the temp certainly below 80C & monitor it with SpeedFan.

88C seems far to high. I avoid going over 80 on my 580s. Manually set the fan speed to a speed that brings the temp certainly below 80C & monitor it with SpeedFan.

next wu failed with same error code

core 675
shader 1350
memory 1800

so now i took back core/shader to normal 715/1430
and memory to 1750

will come back with next result.


rgds

Tabaluga

Tabaluga,
try to reduce memory clock down to 1600.
actually, memory clock doesn't almost impact to duration of GeneferCuda execution.
If it will successful on 1600, you can try 1700 and 1750 afterward.
____________

For now I reboot the system. I have the subjective feeling that the errors accumulate the last days. The system seems reliable a week ago.

Update: I figured out it is not the app. I can reproduce this behaviour also with 1.06. I also tested downclocking but this not solve the problem. Meanwhile I suppose it could have a relation with cpu liability. The errors accumulate when the cpu is running 4 SR5 AVX prpnet tasks simultaneously, which stress the CPU a lot.

I switched the machine to Milkyway to take a look if the double precision tasks from then also produces errors or not.

Regards Odi
____________

Is there a GeneferCUDA test or benchmark we can use to test the ability to successfully run a work unit without having to wait for one to be assigned...to just fail? I'm running the windows BOINC software.

http://www.primegrid.com/result.php?resultid=366798079

Ok, I can answer my own question. It might be useful for others too:

start->run->cmd
cd\programdata\boinc\projects\www.primegrid.com
primegrid_genefer_2_3_0_0_1.07_windows_intelx86__cuda32_13.exe -boinc -q "215902^524288+1" --device 0

If it fails, run "type stderr.txt" to see the info. I guess that's better than logging computational errors.

Thanks for the link. I had edited my post before I saw yours with what I had come up with.

The reported speed is "wrong" - that is, nVidia's reference design comes of the box clocked at 1006MHz. This card is just a reference card without any manufacturer overclock (other than the auto overclocking built into these new cards).

I'm sitting at around 73C running the command I had in my previous post, so my temps are good.

I had been overclocking both the GPU and VRAM when getting those errors, and reducing the overclock has helped in my testing. I'm trying to find a setting that is reliable and fast, so I'm still overclocking... But I want the next block I retrieve to actually be completed, so I'm simulating it by running one of the blocks I failed manually.

I was actually surprised that an overclock that showed no artifacts or other instability would actually fail in CUDA, so I had come here looking to see if there was a bug related to the new architecture. But it was just my overclock.

Your general advice in point #6 still holds true for the GTX680. Memory speeds are key to processing the Genefer work units successfully, but there's no reason to underclock the memory on the 680. As a note, according to GPU-Z, memory controller load is about 57% for Genefer versus 0% for PPS.

As can be seen at http://www.primegrid.com/results.php?hostid=250209&offset=0&show_names=0&state=0&appid=16 from April 5 and beyond, all the work units processed by the 680 have been successful (using the settings described below). The ~9500-10000s runs are the 680, and the ~12000s runs are the 560Ti I also have in there.

My particular card seems to handle Genefer running at 1652MHz(1502MHz stock, +300 DDR MHz offset) memory and +135MHz core. There are no separate shader clocks.

+325 memory offset seemed to work but eventually failed as can be seen in the work history, but +300 has worked without failures yet. Failures seem to happen within the first 15 minutes if the memory is clocked too high, which is different from the other computation errors I see on the Fermi cards which tend to fail a lot around an hour into processing. This memory clock is significantly lower than +475 which will work for a game without noticing any corruption.

I did slightly less extensive testing with core speeds, as I wanted to use a speed that would work for Genefer and for games. I've used +140 without problems in for games, but I've kept it to +135.

Heat definitely isn't a problem. 74C/54%/2100RPM is what I see on Genefer work units with a slightly modified fan profile, and this is pretty quiet. There are probably two reasons for such cool and quiet operations... the performance is less, and I'm unable to use any voltages higher than 1.1750 to clock the core higher to make heat more of an issue. Other work unit types definitely generate more heat.

I wanted to be reasonably sure that I found stable settings before I posted describing how I got there. It looks reasonably stable enough. I hope these are the details you were looking for, and I don't see any reason to expect issues with the stock GTX680.

EDIT: About the "+135" on the core speed - the stock clock is 1006MHz but it will automatically dynamically overclock to at least 1059 and usually more depending on TDP and temperature. With a +135 offset to the base clock, there are two frequencies the core will run at depending on temperature. If the GPU is below 70C, the core will run at 1215MHz. If it's 71C or above, it will run at 1202MHz. It runs at 1.175v at 1215MHz and 1.163v at 1202MHz. This is the highest the voltage will go on the reference board.

I am seeing lots of computation errors, it *used* to work just fine, no driver changes but now all the wu i get fail:
http://www.primegrid.com/results.php?hostid=221152

It seems most/all of the work units i am getting that failed also failed multiple times for others so I'm no sure if this a more widespread issue...

So after posting last night I aborted my wu and went back to prpnet.

First I tried GFN262144 which estimated to complete 1:14:00 but it exceeded maxerr and fell back to cpu.

I then tried GFN524288 and each wu took 4 hours, it completed two wu last night on cuda without issue.

This morning I've installed the nvidia monitoring tools, the GPU runs at 65 C (17 C in the room). I'm now trying GFN262144 with the GPU fans set to max which has the temp at 54C. If it completes these I'm going to switch back and try it on primegrid again perhaps on a smaller wu to start and work my way up.

If this keeps failing I'm going to have to tell my wife the video card has failed and I need to upgrade to a dual 580 ;)

I just got this GFN error on a new platform I'm working with:

I am pretty sure that this question was asked many times, but...

I am testing a completely new, out-of-the-box 570 card (Gigabyte GV N570OC-13I REV2.0) and it seems to pass any test except GeneFer/CUDA (in the framework of this BOINC rally) and separately CUDALucas. They fail at similar rates (unsurprisingly, because these are very similar algos): totally wrong data as the card is shipped (memory at 1900MHz) and 10-20% fail rate with memory at, say, 1820MHz. GPU freq doesn't seem to matter; I underclocked it to some degree, too.

I am not sure if the RMA Gigabyte people will listen kindly to an argument as above ("works fine for anything but some obscure (for them) applications"). What could be my options? RMA would probably get the same result. Reseat the cooler (which in itself and by external reviews look pretty decent) on some quality paste? This particular cooler design I believe covers the memory modules, too (even though memory is not factory OCd).

Thanks in advance for any advice.

Yep, that's the one; 3 fans. The airflow is good (the case is Antec 900) and I ran it in the open as well.

Either way the temps are very low, hovering at 61"C under full load.
I am using EVGA Precision (and OC Scanner) left from the previous 560Ti448 that i moved to another comp. It doesn't fail the OC Scanner (it would have been easier if it did).

There may be a defective memory chip that passed the light QC and is somewhat working. What I like the least that even with more underclocking, genefer fails still once every few tasks. That's not good.
When the tasks finish, they finish well - and quite a few are validated already. But with the 1/3 drop off, this is a fairly rotten deal.

I'll start the RMA, or maybe a return if Amazon will take it back. The kids won't be happy :-) less games for them for a while.

Could it be from somewhat silly wiring? (I had the 8-pin PCI-E which I used; for the other 6-pin I used the 2-molex joiner included with the card. This contraption leaves one, middle 12V rail not powered. Are they connected in the circuitry on the card immediately anyway?)

Thanks.

P.S. The PSU is 700W, though.

I'll try to crimple a proper straight 6-to-6 PCI-E YY5700-H connector on Monday and will decide on RMA after that. (The PSU is the modular venerable TT 700W; the proper cables were misplaced after the move to the new house.)

Thanks, it could be the reason, too. (They throw in those Y-cables with every card, sure, that's better than nothing - but for this job, these cables could be the weak link.)

I was able to crimple it tonight. (The TT 700W PSU is modular and has two "red" PCI-E feeders + one hardwired 8-pin PCI-E which I was already using.)

With the first PCI-E port, nothing changed: still shaky even when underclocked.
With diminishing hope I've replugged the PCI-E to the other, never before used port ...et voila, no errors in CUDALucas -r * 10 times with underclock. No errors at stock either (which is a 780 OC/1900 memory). The card's temp crawled to 65"C, which signals that it is now properly fed (and still nicely cool; there's huge room for OC which I am not interested in. I am interested in relative silence).

So it was the PSU, but there's still life in it. It seems that one of the two ports has been burnt.

I will go for a genefer test for a couple days now. And will probably stay with it, because GPU-to-72 is starting to tread water, not much fun.

Thanks, Mike, you've been very helpful! You hit the nail on the head.

P.S. ...Nope, genefer still failed at stock. Will get another PSU, then.

Serge Batalov

I am not sure if the RMA Gigabyte people will listen kindly to an argument as above ("works fine for anything but some obscure (for them) applications"). What could be my options?

It occurred to me not to be lazy and indeed I swapped the cards in the morning.
It was the card -- the bugs go with the card and not with the PSU or environment.
(The other comp is Linux64, so I cannot even underclock it there. It stubbornly does exactly the same err=0.5>0.45 in this different environment which is if anything only better. In the original comp, the 560Ti448 happily picked up the BOINC queue and the last I checked never fouled.)

I am returning the 570 and getting another from Tigerdirect. It will be a different batch than Amazon and I can wait for it to arrive to cross-mail (unlike RMA or replacement).

Thanks, guys! It was educational. Like Tolstoy said, "Happy families are all alike; every unhappy family is unhappy in its own way." That is, when everything works, there's no way to be prepared for how things will go wrong in a new build. :-)


P.S. For Amazon, just like you suggested, I didn't sophisticate or rationalize: just wrote, "defective; runs for a while, then produces artifacts under load."

Followup: the new card came in the mail. No errors.

I don't have a good BOINC binary (it's OpenSUSE 12.1), and even when I built it, the uploaded binary (1.07; cuda 3.2) didn't run b/c of some dependencies... so I'll post the residue here, if that's alright? I am pretty sure that his WU is already done and validated.

If you could help me out and confirm that this residue is correct? (No credit requested.)

372886^524288+1 is a probable composite. (RES=bbe7b88c2c01efe3) (2921111 digits) (err = 0.0859) (time = 4:33:39)

This is with own built genefercuda.1.061, sm20, cuda 4.2.

Is there any way to block computers like this from receiving any new GFN work?
http://www.primegrid.com/show_host_detail.php?hostid=236684

Users like this are very frustrating for every serious user.

____________

I think it is causing problems for other users.
Since the units are held of a few hours/days the system generates a lot more units then needed. So the overall time for all these units to validate increases.

If less units are generated the time for validation will be shorter.

____________

Is there a testing program available that will detect double precision issues or memory issues such as with GeneFerCUDA without wrecking someone else's scores ?

I am becoming a bit frustrated with days and days of wasted GPU time.
No doubt my wingmen/women will be frustrated with me for having errored out on them.

MaxErr exceeded... This to me sounds like a rounding issue somewhere in the code.

GTX295 which ran milkyway and oter projects fine for a long time.
Something faster and more modern is on the near horizon...


Found an anser in this thread:
"Ok, I can answer my own question. It might be useful for others too:

start->run->cmd
cd\programdata\boinc\projects\www.primegrid.com
primegrid_genefer_2_3_0_0_1.07_windows_intelx86__cuda32_13.exe -boinc -q "215902^524288+1" --device 0

If it fails, run "type stderr.txt" to see the info. I guess that's better than logging computational errors."

Thank you for sharing that !

Thank you Michael,

In fact, that's designed just for that purpose. Not to catch a coding error, but to detect when the number being tested is too large. That's what determines the "b limit" you might read about in other threads.

GeneferCUDA, on the other hand, performs sanity checks at every step of the calculation and usually (but not always) will detect computational errors and abort the processing immediately, rather than letting it run to the end only to eventually be invalidated on the server when it doesn't match the wingman's results.

As for more modern hardware, if you're upgrading specifically to run GeneferCUDA, please be aware that the 600 series GPUs are slower with this app than are the older 500 series GPUs. A GTX 680 is about 20% slower than a GTX 580.

Thanks for the elaborate answer, much appreciated.

Although the program's maxErr checks are designed to catch loss of precision errors, what you see happening is a hardware failure. Using smaller elements (and larger FFTs) won't fix that.

Tesla cards are high-reliability for compute critical applications.

They're also VERY expensive. Nevertheless, some people have tried running on them -- I suspect in most cases, by renting them on AWS. This lead us to discover that...

They're not very fast. Their clock speeds aren't all that high, and they are out-performed by the much less expensive consumer GeForce cards.