Strange problem in PB/CC 4

[RC Santore]

A PB/CC (4.04.0042) program I've been using for many years has developed some weird problem I have so far not been able to debug.

First a code snippet:

PercHA! = VAL(DField$(j)) / 100!

The origin of the contents of DField$() are from a text input file. I read the input file one line at a time, then parse the columns into individual elements of DField$() before using them to set the values of several dozen variables including this one. In this case DField$(j) will have a value of something like "1.000000E-002" before being converted to a number and stored in a floating point variable.

This has worked fine for many hundreds of thousands of lines of input over several years.

Recently I found a situation where PercHA! is assigned a value of zero, even though DField$(j) has an appropriate value (such as "1.000000E-002").

For those rare times when PercHA! is assigned a value of zero, the following code will not detect the zero value:

IF (PercHA! = 0.0) THEN
    PRINT "PercHA! is zero"; PercHA!
ELSE
    PRINT "PercHA! is not zero"; PercHA!
END IF

When executed it will report:

PercHA! is not zero 0


I should point out that this only happens with one or two lines of input from an input file many hundreds of lines long. The way this program works is it reads a line of input, performs some complex math, writes the output, then reads the next line of input and does the whole process again. For the case when this problem occurs, the problem seems to be unrelated to the specific lines of input. By that I mean, if I run it and see the problem with the 37th and 62nd lines of input, and if re-run it I will always see the same problem with those two lines of input. If I then rearrange the input file so that those two lines are now the 1st and 2nd line of input, then the problem will not occur on those two lines, but will occur on some other lines (and not the 37th and 62nd again). So I don't think the problem is related to the input data.


Finally, another example of the odd behavior on the rare occasions when this problem occurs:

PercHA! = VAL(DField$(j)) / 100! 

IF (PercHA! = 0.0) THEN
    PRINT "PercHA! is zero"; PercHA!
ELSE
    PRINT "PercHA! is not zero"; PercHA!
END IF

PercHA! = PercHA! + 1

IF (PercHA! = 0.0) THEN
    PRINT "PercHA! is zero"; PercHA!
ELSE
    PRINT "PercHA! is not zero"; PercHA!
END IF

When executed will report:


PercHA! is not zero 0
PercHA! is not zero 0

If I replace the first line with

PercHA! = 0.001

it still has the same behavior.

At this point I am guessing that this code is not the problem, but only the symptom of some other problem, like a memory overrun, or something . . . ? This is a large program with many variables, and it is very actively used (by me and many others over the past 8 yrs, although I wrote it) and we do not normally see anything like this happening.

Does anyone have any ideas about what I should try next? I'm really puzzled.

- Robert

[Kev Peel]

IIRC, a floating point value is only an approximation, so checking for equality with another floating point value will never be exact - there are more figures after the .00.

A hex comparison might be better?

[RC Santore]

Right, and normally I wouldn't be doing it this way, I am here just trying to debug why this variable has a value of zero.

But I do believe the third and forth code snippets are showing that this is not just due to unexpected results due to floating point comparisons. How can both "PercHA! = 0" and "PercHA! + 1 = 0"?

I can also replace

IF (PercHA! = 0.0)

with

IF (PercHA! < 0.00001)

and it has the same result.

[Arthur Gomide]

Are you sure that the input text file was obtained in a continuous way? I ask because there may be some non-ASCII characters in the middle of the lines.

[RC Santore]

Arthur thank you for your reply (and thank you too Kev, I'm sorry for not saying that previously in my reply to you).

Yes, I am sure there are no extended characters in the input file, and as I mentioned I can move lines around in the input file and see that the problem will appear on different lines.

In fact, when this is happening, I cannot assign any value to PercHA! as all. If I do any of the following:

PercHA! = 0.1
PercHA! = PercHA! + 1

and then "PRINT PercHA!" it will always print as "0"

- R

[Kev Peel]

Any chance you could draw up an example program to demonstrate? Maybe something that will also auto-generate some sample data. If the problem cannot be reproduced in a sample then more than likely (IMO) you have an issue occurring elsewhere which is corrupting or overwriting the stack. If that's the case then there is no easy option to resolve it, except combing over the code. Cutting down on the amount of globals and using parameters may also make this easier to do.

[RC Santore]

I suspect you're right Kev and I am currently in the process of checking all dims - but there are a lot of variables and a lot of code (about 12,000 lines in 78 subs and functions - not sure if that's really a lot, but it isn't small either).

This behavior is so bizarre that I suspect it won't be reproducible in any smaller program.

Thanks again,

- R

[John Gleason]

RC, try making the variables EXT (possibly DOUBLE would work too but I'd try EXT first), and to be sure your values are exactly entered, eg. for .000001 use either VAL(".000001") or 1 / 1000000. Perhaps some internal calc needs the extra precision.

[Michael Bell]

RC
I've had similar problems before with apparently impossible program behaviour, and they've (almost) always turned out to be related to over- or underflows occurring somewhere else entirely. You mention "complex math" - therein may lie the problem.

If checking your dims gives no joy, then I would suggest locating all the mathematical operations that may be sensitive to inadmissible arguments - such as divisions and logs - and replace the relevant lines with code that: (i) checks the argument; (ii) performs the operation if the argument is admissible; and (iii) flags inadmissible values.

For example, replace

x = y / z

with

IF z <> 0## THEN
  x = y / z
ELSE
  PRINT "ERROR: z is zero!"
  WAITKEY$
  EXIT FUNCTION ' or whatever other action is appropriate
END IF

Obviously, I'm only suggesting this for debugging rather than a final solution...

As John says, it would also be a good idea to work with EXT rather than single precision - some arguments may become inadmissible simply because of precision. Working with EXT would in any case be better for complex math.

[RC Santore]

Your comment about under- and over-flows is in line with my suspicion that the behavior I'm seeing here is just the symptom of some as yet undiscovered issue.

I generally try to use a sensible mix of single and double precision based on the range of expected values. But I suppose that tendency comes from a background in fortran, at a time when memory wasn't as plentiful as it is now. Although memory might not be an issue there is still a considerable performance hit when comparing single to double (about 4x the time required) to extended (about 7x to 8x) that would argue for not using more precision than needed.

However, I am so far at a loss to figure out what's going wrong here and I appreciate your suggestions for new approaches to try out. I'll start converting to extended and see if something turns up.

Although I generally keep current in my PBCC and PBWIN versions, I am currently using version 4. Does anyone think version 5 would provide any new debugging aids or diagnostics for this sort of problem?

Again, I appreciate everyone's help so far.

- R

[Michael Mattias]

always turned out to be related to over- or underflows occurring somewhere else entirely. You mention "complex math"

I note the original code uses an array subscript. You don't need arithmetic overflow to get funky results if you refer to an out-of-bounds subscript somewhere and don't 'do something' about that.

Or, an out-of-bounds reference (trappable) could be causing overflows (not trappable using only compiler intrinsics) because the compiler skips statements causing same (#DEBUG ERROR ON), or provides "undefined" results (#DEBUG ERROR OFF).

[RC Santore]

Thank you Michael. I should have specified that I have run this with #DEBUG ERROR ON and #DIM ALL as part of this debugging effort.

Maybe I don't understand your last sentence, but should #DEBUG ERROR ON catch out-of-bounds subscripts? This code uses many arrays of many different types of variables. I am not sure the distinction between "trappable" and "not trappable". Can you elaborate?

- R

[Michael Mattias]

#DEBUG ERROR ON will do what it says it will... on bounds violation set error 9 and not attempt the statement. You still have to create code to to actually "do something" when this occurs.

#DEBUG OVERFLOW ON is not supported. i.e, "Let Z = X/0" ==> protection fault and nothing you can do about it using only compiler intrinsics.

(See SetUnhandledExceptionFilter() winapi function if you want to trap. There is a current thread here started by Cliff Nichols on this very topic).

MCM

[RC Santore]

Yes of course, with "ON ERROR GOTO" in each subroutine.

But the first trap that trips occurs after the above, and the error doesn't make sense to me (Error 52 - Bad file name or number) so I think it to be unrelated (just another symptom but still not the disease?).

The way I found the above is tracing backwards (figuratively speaking - although a debugger that ran backwards would be pretty neat) from where I see the Error 52. Then after looking at the above behavior for a while I started thinking that there really must be something happening even prior to all of that.

I'll look for the Cliff Nichols post.

Thanks again,

- R

[RC Santore]

Ok, I found it.

So do I understand that you are suggesting that a numeric overflow such as what could result from:

   LOCAL B AS BYTE
   DO WHILE B  < 256 
        B = B + 1
   LOOP

might be causing this behavior (or just general bad things), and hence moving to EXT . . .?

- R

[Kev Peel]

That would cause an infinite loop - B would never get to 256, the range is 0-255.

Use the following:

 
DO
    B = B + 1
LOOP UNTIL (B = 255)

[RC Santore]

Kev - yes I know, that's not my code, the example was from the other post that Michael mentioned and was intended to create an overflow. I was just trying to see if I was following what he was trying to tell me.

- R

[Michael Mattias]

If you get a numeric overflow, who knows what happens?

However, since the compiler does not provide the intrinsic options to even know if an overflow occurred, it's probably beating a dead horse to pursue it.

Stick with the 'usual suspects' - the kinds of errors you CAN trap by using TRY..CATCH, ON ERROR GOTO or just testing the system ERR variable.

MCM

[Michael Bell]

Precision and performance

I generally try to use a sensible mix of single and double precision based on the range of expected values. But I suppose that tendency comes from a background in fortran, at a time when memory wasn't as plentiful as it is now. Although memory might not be an issue there is still a considerable performance hit when comparing single to double (about 4x the time required) to extended (about 7x to 8x) that would argue for not using more precision than needed.
- R

Your comment about performance interested me, as I have performance issues of my own (programming performance, that is...!). I've been running some fishery simulations, written in PB/CC5.01, and so far it's taken ten days to run 12 scenarios out of 26 - not much use when I want to explore a much wider parameter space. I tend to work in extended precision for numerical processing, but I wondered how much quicker it might run if I could get away with single precision. No much, as it turns out. I didn't let it run for long, but I couldn't notice the difference in terms of the speed of results reported to the screen.

I tried writing some test code using different floating point types (SQR, ^, LOG and EXP operations within loops of 100,000,000 values), and I find that the same operations are in fact running up to 2% faster in extended than in single precision, which is not what I was expecting...

Any thoughts? I'd certainly appreciate any hints for faster running code. I'm aware of #OPTIMIZE SPEED, not updating the screen too often, not having unnecessary function calls, ... As background, I use PB/CC as a tool for scientific modelling and analysis, and I'm certainly no professional programmer, so it's quite easy for comments in these forums to go right over my head.

-MCB

[RC Santore]

Michael,

I suspect that exponentiation and transcendental functions do not depend as much on the variable type being used. I thought I'd do a quick test to verify that, and found that is indeed the case.

However I also discovered a real eye opener for me - for multiplication extended precision is the fastest (not slowest) variable type in PBCC. This probably will sound like an obvious statement to many others, but I will say it anyway to remind myself - it pays to do benchmarks with the compiler you are using and don't assume behavior will be consistent from one to another.

Depending on the values being multiplied, using extended precision was either much faster or just slightly faster.

As noted above, I did also find that there was much less difference with exponentiation than with multiplication operations.

So I take back everything I said about a performance penalty from using extended precision.

By the way, I am also use PBCC for scientific programming (coincidentally also somewhat related to fisheries).

- R