Single-precision float precision?

[Mark Newman]

Sorry for the less-than-precise subject, but...

I'm reading a binary data file created with VB6 using the normal
PUT statements. The file contains test data, stored in single-precision
format. I sometimes need to export this data file to plain-text
format, so I wrote a utility in PB6 to do this. The core of this
utility is just a simple loop:

' Read the data
For I = 1 To lNumPoints
                                 
   ' Read the data from the binary file, single-precision float
   Get #hFileInput, , fAverage
   Get #hFileInput, , fOneSigma
                  
   ' Save the data to the text file
   Print #hFileOutput, Format$(fAverage) & "," & Format$(fOneSigma)
                     
Next

The problem is that when I look at the text data file, there's
way too many digits; here's what the data looks like:

-16.6229248046875,10.6029329299927
-9.305419921875, 7.90202713012695
-7.869873046875, 6.12179470062256
-6.3330078125, 5.09605121612549
-5.333251953125, 4.58390331268311

This is too many digits for a single-precision float. So I ported
the PB app to VB6 and here's the data it converted:

-16.62292, 10.60293
-9.30542, 7.902027
-7.869873, 6.121795
-6.333008, 5.096051
-5.333252, 4.583903

Which is what I'd expect for single-precision float.

So, what's up? Looking at the PB docs for the Format$
function it appears that using it without a formatting string
does output a lot of trailing digits when used with a real number.

My concern is that the Format$ function is showing more
data than is really there, at least according to IEEE-754. It
seems like Format$ is converting values to Doubles first
and then formatting the output.



------------------
Mark Newman

[Eric Pearson]

> My concern is that the Format$
> function is showing more data
> than is really there

Actually, the value that PowerBASIC is producing is more accurate than the VB number, at least statistically speaking.

For example, if you try to store the result of 2/3 in a SINGLE variable, the best you can do is .6666667. If that value is displayed by FORMAT$ you get .666666686534882. The extra data is a result of the way floating point values are stored in memory. But the "real" value of 2/3 is 0.6666666666666...etc. Notice that the "extra data" results in a value that is closer to the "real" value than .6666667...

.6666666666666666666.... (actual value)
.6666666865348820000 off by approx. 0.0000000199
.6666667000000000000 off by approx. 0.0000000334

To summarize, the extra digits are not accurate, but they are more accurate than rounding.

If you want to round your output values to a certain number of digits you can use the ROUND function, but you will be losing a certain amount of accuracy.

-- Eric


------------------
Perfect Sync: Perfect Sync Development Tools
Email: mailto:[email protected][email protected]</A>

[Mark Newman]

Hi Eric,

Thanks for the reply. I understand your point but the problem I'm
having is that according to IEEE-754, there's a certain resolution
available with single-precision format numbers. The data output by
Format$() violates that resolution, so the digits beyond this limit
are meaningless.

Interestingly, I changed to using Str$ in my app and the
output matched VB's (except for missing some leading zeros).



------------------
Mark Newman

[Eric Pearson]

This will (probably) match VB's output exactly...

FORMAT$(spSingle,"0.#####")

-- Eric


------------------
Perfect Sync: Perfect Sync Development Tools
Email: mailto:[email protected][email protected]</A>