Tweet
Originally posted by Steve Rossell
View Post
In the meantime, however, the following code illustrates a simple method for obtaining higher precision (18 decimal digits) for your floating point numeric literals. It involves using the VAL function to convert your literals to extended precision instead of casting them to extended precision; i.e, using VAL("literal") instead of literal##.
In using this method, note the following:
(1) If you have a floating point literal which has more than 18 significant digits, round the literal to 18 digits before using it in the string argument of VAL. If you don’t, VAL will truncate any digits beyond the 18 digits (instead of rounding to 18 digits), resulting in less precision.
(2) You do not have to add zeroes to a floating point literal with fewer than 18 significant digits. (In other words -- with VAL -- using 0.1 will give you the same result as 0.100000000000000000.
(3) If calculation time matters, do not use the VAL function within a loop with many iterations. It does have considerable overhead. Instead, assign the value of VAL(“literal”) to an extended precision variable and use the variable within the loop.
(4) If you have any doubt about the value of your literal as obtained from VAL(“literal”), check it out with STR$(VAL(“literal”), 18).
(5) This procedure has not been fully tested (or vetted), so please use good judgment in its use.
Any constructive comments relative to this post will be greatly appreciated.
--WH
Code:
[FONT=Courier New]#COMPILE EXE[/FONT]
[FONT=Courier New]#DIM ALL[/FONT]
[FONT=Courier New]FUNCTION PBMAIN() AS LONG[/FONT]
[FONT=Courier New]DIM z AS LOCAL EXT, y AS LOCAL EXT, counter AS LOCAL LONG, timecheck AS LOCAL DOUBLE[/FONT]
[FONT=Courier New]PRINT "Assigning 0.1 to z with 16 digits of precision"[/FONT]
[FONT=Courier New]z = 0.1##[/FONT]
[FONT=Courier New]PRINT:PRINT "z = 0.1## = ";STR$(z, 18)[/FONT]
[FONT=Courier New]PRINT:PRINT:PRINT "Assigning 0.1 to z with 18 digits of precision"[/FONT]
[FONT=Courier New]z = VAL("0.1")[/FONT]
[FONT=Courier New]PRINT:PRINT "z = VAL(""0.1"") = ";STR$(z, 18)[/FONT]
[FONT=Courier New]PRINT:PRINT:PRINT "Calculation of 0.1 - 1/10. (Exact result = 0)"[/FONT]
[FONT=Courier New]z = 0.1## - 1/10[/FONT]
[FONT=Courier New]PRINT:PRINT "z = 0.1## - 1/10 = "; STR$(z,18)[/FONT]
[FONT=Courier New]PRINT:PRINT:PRINT "Calculation of 0.1 - 1/10. (Exact result = 0)"[/FONT]
[FONT=Courier New]z = VAL("0.1") - 1/10[/FONT]
[FONT=Courier New]PRINT:PRINT "z = VAL(""0.1"") - 1/10 = "; STR$(z,18)[/FONT]
[FONT=Courier New]PRINT:PRINT:PRINT "Iteration loop (1 million iterations) using VAL(""0.1"") within the loop"[/FONT]
[FONT=Courier New]timecheck = TIMER[/FONT]
[FONT=Courier New]FOR counter = 1 TO 1000000[/FONT]
[FONT=Courier New] z = VAL("0.1") - 1/10[/FONT]
[FONT=Courier New]NEXT counter[/FONT]
[FONT=Courier New]PRINT "Result: "; STR$(z, 18)[/FONT]
[FONT=Courier New]PRINT "Elapsed time (seconds): ";ROUND(TIMER - timecheck, 4)[/FONT]
[FONT=Courier New]PRINT:PRINT:PRINT "Iteration loop (1 million iterations) using EXT y (equal to VAL(""0.1"")) within the loop"[/FONT]
[FONT=Courier New]timecheck = TIMER[/FONT]
[FONT=Courier New]y = VAL("0.1")[/FONT]
[FONT=Courier New]FOR counter = 1 TO 1000000[/FONT]
[FONT=Courier New] z = y - 1/10[/FONT]
[FONT=Courier New]NEXT counter[/FONT]
[FONT=Courier New]PRINT "Result: "; STR$(z, 18)[/FONT]
[FONT=Courier New]PRINT "Elapsed time (seconds): ";ROUND(TIMER - timecheck, 4)[/FONT]
[FONT=Courier New]WAITKEY$[/FONT]
[FONT=Courier New]END FUNCTION[/FONT]
Comment