Announcement

Collapse
No announcement yet.

Single-precision float precision?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Eric Pearson
    replied
    This will (probably) match VB's output exactly...

    Code:
    FORMAT$(spSingle,"0.#####")
    -- Eric


    ------------------
    Perfect Sync: Perfect Sync Development Tools
    Email: mailto:[email protected][email protected]</A>

    Leave a comment:


  • Mark Newman
    replied
    Hi Eric,

    Thanks for the reply. I understand your point but the problem I'm
    having is that according to IEEE-754, there's a certain resolution
    available with single-precision format numbers. The data output by
    Format$() violates that resolution, so the digits beyond this limit
    are meaningless.

    Interestingly, I changed to using Str$ in my app and the
    output matched VB's (except for missing some leading zeros).



    ------------------
    Mark Newman

    Leave a comment:


  • Eric Pearson
    replied
    > My concern is that the Format$
    > function is showing more data
    > than is really there

    Actually, the value that PowerBASIC is producing is more accurate than the VB number, at least statistically speaking.

    For example, if you try to store the result of 2/3 in a SINGLE variable, the best you can do is .6666667. If that value is displayed by FORMAT$ you get .666666686534882. The extra data is a result of the way floating point values are stored in memory. But the "real" value of 2/3 is 0.6666666666666...etc. Notice that the "extra data" results in a value that is closer to the "real" value than .6666667...

    Code:
    .6666666666666666666.... (actual value)
    .6666666865348820000 off by approx. 0.0000000199
    .6666667000000000000 off by approx. 0.0000000334
    To summarize, the extra digits are not accurate, but they are more accurate than rounding.

    If you want to round your output values to a certain number of digits you can use the ROUND function, but you will be losing a certain amount of accuracy.

    -- Eric


    ------------------
    Perfect Sync: Perfect Sync Development Tools
    Email: mailto:[email protected][email protected]</A>

    Leave a comment:


  • Mark Newman
    started a topic Single-precision float precision?

    Single-precision float precision?

    Sorry for the less-than-precise subject, but...

    I'm reading a binary data file created with VB6 using the normal
    PUT statements. The file contains test data, stored in single-precision
    format. I sometimes need to export this data file to plain-text
    format, so I wrote a utility in PB6 to do this. The core of this
    utility is just a simple loop:

    Code:
    ' Read the data
    For I = 1 To lNumPoints
                                     
       ' Read the data from the binary file, single-precision float
       Get #hFileInput, , fAverage
       Get #hFileInput, , fOneSigma
                      
       ' Save the data to the text file
       Print #hFileOutput, Format$(fAverage) & "," & Format$(fOneSigma)
                         
    Next
    The problem is that when I look at the text data file, there's
    way too many digits; here's what the data looks like:

    -16.6229248046875,10.6029329299927
    -9.305419921875, 7.90202713012695
    -7.869873046875, 6.12179470062256
    -6.3330078125, 5.09605121612549
    -5.333251953125, 4.58390331268311

    This is too many digits for a single-precision float. So I ported
    the PB app to VB6 and here's the data it converted:

    -16.62292, 10.60293
    -9.30542, 7.902027
    -7.869873, 6.121795
    -6.333008, 5.096051
    -5.333252, 4.583903

    Which is what I'd expect for single-precision float.

    So, what's up? Looking at the PB docs for the Format$
    function it appears that using it without a formatting string
    does output a lot of trailing digits when used with a real number.

    My concern is that the Format$ function is showing more
    data than is really there, at least according to IEEE-754. It
    seems like Format$ is converting values to Doubles first
    and then formatting the output.



    ------------------
    Mark Newman
Working...
X