One more try, then I promise I'll stop...

[Patrick Mills]

OK, I am still trying to understand why my variant assignment statement causes subsequent POKE$ operations to silently fail. I think my first program might have been too long to expect people to look at it. I've tried again with the simplest possible program that demonstrates what I am now suspecting is a bug in PBCC 5.0 with variant array assignment.

BTW, Yes - I *know* variant assignment is slow and not a good way to copy arrays, but I believe this code demonstrates a problem with its implementation. Such a problem could easily affect legitimate uses of it, causing programs to silently fail, producing incorrect results that are very difficult to trace in production code.

Please, could someone try running this program on their system and see if they also always get to the line indicating "Arrays not equal."? Thanks.

#DIM ALL

%LAST = 98664440
    
GLOBAL gArray1() AS LONG, gArray2() AS LONG, gVar AS VARIANT

FUNCTION PBMAIN() AS LONG

       DIM i AS LONG

       REDIM gArray1(%LAST), gArray2(%LAST)

       FOR i = 0 TO %LAST
           gArray1(i) = RND(1,999)
       NEXT i

       'NOTE: Once this line executes, the POKE$ silently fails.
       gVar = gArray1() : gArray2() = gVar

       REDIM gArray2(%LAST)

       'NOTE: With %LAST >= 98664440, this line will silently fail.
       POKE$ VARPTR(gArray2(0)), PEEK$(VARPTR(gArray1(0)), 4 * (%LAST+1))

       FOR i = 0 TO %LAST

           IF gArray1(i) <> gArray2(i) THEN
               STDOUT "Arrays not equal.  Press any key..."
               WAITKEY$
               EXIT FUNCTION
           END IF

       NEXT i

       STDOUT "Arrays were equal.  Press any key..."
       WAITKEY$
       
END FUNCTION

[Michael Mattias]

IWH (it works here)

Code run:

' FILE: ARRAY_VT.BAS
' 01/10/09

#DIM ALL

%LAST = 98664440

GLOBAL gArray1() AS LONG, gArray2() AS LONG, gVar AS VARIANT

FUNCTION PBMAIN() AS LONG
 LOCAL Serr  AS STRING


       DIM i AS LONG
       ON ERROR GOTO ErrorTrap
       
       sErr = "REDIMing gArrary1"
       
       STDOUT sErr
       REDIM gArray1(%LAST)
       sErr =  "REDIMing GArray2
       STDOUT sErr
       REDIM gArray2(%LAST)
       
       sErr = "Filling GArray1"
       STDOUT sErr
       FOR i = 0 TO %LAST
           gArray1(i) = RND(1,999)
       NEXT i

       'NOTE: Once this line executes, the POKE$ silently fails.
       sErr = "Assigning GArray1 TO gVar"
       STDOUT sErr
       gVar = gArray1()
       sErr =  "Assigning Garray2 from gvar"
       STDOUT sErr
       gArray2() = gVar
       
       sErr = "REDIMing GArray2"
       STDOUT sErr
       REDIM gArray2(%LAST)   ' This voids the content of gArray2

       'NOTE: With %LAST >= 98664440, this line will silently fail.
       
       sErr = "POKE'ing new values into Garray2"
       STDOUT sErr
       POKE$ VARPTR(gArray2(0)), PEEK$(VARPTR(gArray1(0)), 4 * (%LAST+1))
       
       sErr = "Comparing Arrays"
       STDOUT sErr
       FOR i = 0 TO %LAST
           IF gArray1(i) <> gArray2(i) THEN
               STDOUT "Arrays not equal.  Press any key..."
               WAITKEY$
               EXIT FUNCTION
           END IF

       NEXT i

       STDOUT "Arrays were equal.  Press any key..."
ExitPoint:
       WAITKEY$
       EXIT FUNCTION
       
ErrorTrap:
 STDOUT "Entered error trap on Error " & FORMAT$(ERR)
 RESUME  ExitPoint
        

END FUNCTION

OUTPUT:

REDIMing gArrary1
REDIMing GArray2
Filling GArray1
Assigning GArray1 TO gVar
Assigning Garray2 from gvar
REDIMing GArray2
POKE'ing new values into Garray2
Comparing Arrays
Arrays were equal.  Press any key...

PB/CC 5.0.0, Win XP/Pro SP3 all updates.

MCM

[Michael Mattias]

BTW, I think you need to set the variant to NOTHING (EMPTY?) when you are done with it. Otherwise it just sucks up memory. True, it's released when the process ends, but not erasing "work" variables which are eating up this much memory when your program has more to do is IMO quite careless.

[Michael Mattias]

And frankly, in your other demo program, I wonder if you are running out of memory, since you don't clear out your workspace. But we wouldn't know since you don't have any error trapping in that code.

Here I used ON ERROR, but simply testing the system ERR variable is plenty good enough.

FWIW, Error 7 is returned if a DIM/REDIM or any other allication/reallocation (eg, the temporary string created by PEEK$) fails for want of memory.

The compiler has so many tools... or maybe I should say so many unused tools.

[Michael Mattias]

FWIW.. although I did not see anywhere it could possibly come into play, I recompiled and ran with the #DEBUG ERROR ON directive.

Same output.

And just in case you have not guessed yet... five bucks says your program "failures" are the result of willy-nilly proceeding after an error you failed to trap.

MCM

[Brian Chirgwin]

On my laptop Win XP with 3Gigs memory arrays are equal

On VM Win 2000 set for 256 MB memory arrays are not equal

Seems the number of elements is directly related to the amount of memory available.

OK, I am still trying to understand why my variant assignment statement causes subsequent POKE$ operations to silently fail. I think my first program might have been too long to expect people to look at it. I've tried again with the simplest possible program that demonstrates what I am now suspecting is a bug in PBCC 5.0 with variant array assignment.

BTW, Yes - I *know* variant assignment is slow and not a good way to copy arrays, but I believe this code demonstrates a problem with its implementation. Such a problem could easily affect legitimate uses of it, causing programs to silently fail, producing incorrect results that are very difficult to trace in production code.

Please, could someone try running this program on their system and see if they also always get to the line indicating "Arrays not equal."? Thanks.

#DIM ALL

%LAST = 98664440
    
GLOBAL gArray1() AS LONG, gArray2() AS LONG, gVar AS VARIANT

FUNCTION PBMAIN() AS LONG

       DIM i AS LONG

       REDIM gArray1(%LAST), gArray2(%LAST)

       FOR i = 0 TO %LAST
           gArray1(i) = RND(1,999)
       NEXT i

       'NOTE: Once this line executes, the POKE$ silently fails.
       gVar = gArray1() : gArray2() = gVar

       REDIM gArray2(%LAST)

       'NOTE: With %LAST >= 98664440, this line will silently fail.
       POKE$ VARPTR(gArray2(0)), PEEK$(VARPTR(gArray1(0)), 4 * (%LAST+1))

       FOR i = 0 TO %LAST

           IF gArray1(i) <> gArray2(i) THEN
               STDOUT "Arrays not equal.  Press any key..."
               WAITKEY$
               EXIT FUNCTION
           END IF

       NEXT i

       STDOUT "Arrays were equal.  Press any key..."
       WAITKEY$
       
END FUNCTION

[Patrick Mills]

FWIW.. although I did not see anywhere it could possibly come into play, I recompiled and ran with the #DEBUG ERROR ON directive.

Same output.

And just in case you have not guessed yet... five bucks says your program "failures" are the result of willy-nilly proceeding after an error you failed to trap.

MCM

Michael,

I'm afraid I got a bit hasty with my sample code and didn't error trap. I should obviously have done that. Oops.

I pasted your error-trapped code into my PBCC window and ran it. Here is the output:

REDIMing gArrary1
REDIMing GArray2
Filling GArray1
Assigning GArray1 TO gVar
Assigning Garray2 from gvar
REDIMing GArray2
POKE'ing new values into Garray2
Comparing Arrays
Arrays not equal. Press any key...

Bingo - silent fail of POKE, no error triggered.

I'm stumped as to why I'm experiencing this problem.

We have different OS's, and, apparently, different versions of PBCC.

I'm running Vista Ultimate SP1 (all updates) and PBCC 5.00.0085.

EDIT:

I tried running this code under Windows XP SP3 (running as a guest OS in Vista, using VPC 2007), and it ran without errors. The failure does seem to be related to the OS, and not to the version of PBCC.

Is anyone running Vista SP1 willing to try Michael's error-trapped version of the code and report their results back to this thread?

[Patrick Mills]

On my laptop Win XP with 3Gigs memory arrays are equal

On VM Win 2000 set for 256 MB memory arrays are not equal

Seems the number of elements is directly related to the amount of memory available.

@Brian,

I tried running Michael's error-trapped version on my Vista SP1 (4 gigs) machine and still get the problem with arrays not being equal, although no errors are detected by the error trap.

On my VPC Win XP Pro SP3 (1 gig) the code runs correctly.

Therefore, I remain stumped. We have my copy of Vista (4 gigs) failing, Michael's copy of XP SP3 (memory unspecified) working, your copy of XP (3 gigs) working, your copy of virtual Win2000 (256 megs) failing, and my copy of virtual XP (1 gig) working.

The two failures are on Vista (4 gigs) and 2000 (265 megs) - not much similarity there. If you were running my original non-error-trapped code, your windows 2000 failure could certainly be due to an untrapped memory allocation error, but that isn't what's happening on my 4-gig version of Vista using Michael's error-trapped version.

[Michael Mattias]

>Therefore, I remain stumped

One thing to look at: adequate swap space. I had one of these with a client a little over a year ago. Never any error codes from anything, but seemingly random failures, about three times per week, usuaully on "RPC server not responding."

We upped the swap space (had been set at 4 Gb max), never had the problem again.

Methinks there's probably lots of software out there that says, "this allocation can't fail because this program is only using <X> of our allowed 2 Gb" - but that allocation WILL fail if inadequate swap space is available.

In this program, however, because you are asking the compiler to make an allocation (as opposed to the compiler doing things behind the scenes for you), you SHOULD be getting a memory ERR value. In my experience the PB compilers have always been very good about that.

If swap space is not the problem ===> [email protected]

MCM

[Michael Mattias]

>Bingo - silent fail of POKE, no error triggered

BTW, this is a dangerous assumption on your part. As your code is written, true, POKE may have failed; however, it's also possible the PEEK failed; or that your source data array may have been corrupted BEFORE it was PEEK'ed.

Besides, neither PEEK nor POKE will never set an application error (the system ERR value). All these functions know to do is read or write "N" bytes at addresss "Y" and it does as it is told. They assume the programmer owns the N bytes at address Y, and there really is no (effficient) way to add some kind of automatic checking.

If you want to detect attempted bad accesses of memory DURING one of these operations you will have to use something like structured exception handling. Search here, I know there are examples.

[Edwin Knoppert]

What's the purpose of this code?
Why use the variants, to proof a 'bug'?

Variants being set as you do become safearray handles.
Memory created not similar as ordinary arrays.
Pokeing to an array element as you seem to try is imo a serious thing to avoid.

Using a variants having a safearray is usually used for data exchange.
Not really 'work memory'.

[Patrick Mills]

@Edwin,

The sample code I've posted has no "purpose" other than to demonstrate the point I'm asking about.

I originally was not (and am still not) at all interested in using variants as an array copying method. The possibility simply occured to me as I was testing other methods, so I gave it a try since it was trivial to code. When I noticed that another technique I was trying at the time (the Poke/Peek method) began to fail after I added the variant assignment method, I tried to code a simple example of the problem to post to the forum.

Clearly, using proper error traps and assuring that resources are released when not needed would be something I (hopefully) not neglect in production code, but these samples are not production code. The general error trap added by Michael was something I should have done before posting, since an allocation error could certainly have resulted in some of the problems I was posting about, and my code as posted wouldn't have caught it. Mea Culpa. However, as we've seen, error trapping still does not catch or prevent what was happening in my code.

You can question whether using the variant to proof a "bug" is legitimate, but if there is a problem with PBCC's implemention of variants, what assurance would we have that it won't show up when we're not copying arrays, but exchanging data with COM objects? Perhaps I should have coded up a COM object data exchange example using variants that behaves unpredictably, but I'm not working with COM right now - I'm working with moving large arrays about, and just happened onto the possibility of a variant oddity in PB.

As far as Poke'ing to an array, yes it's a dangerous thing to do (as is calling the MoveMemory api from your code, and for the same reasons). But my application needs to quickly copy large arrays, or portions of them, back and forth, and I needed to find a faster way to do it than looping through them with indices, or even looping through them with pointers. The MoveMemory APIwill meet my needs speedwise, but the dangers of altering arrays outside the safety of the compiler's bounds checking remains. I must either accept the danger or put up with slower performance, as an alternative to moving memory around doesn't exist for my application. Someone suggested memory-mapped files, but I need to reference the copied arrays with the same algorithms as the original arrays (and to compare them, as well), so storing them in a different form is not really a good answer, either.

[Edwin Knoppert]

>As far as Poke'ing to an array, yes it's a dangerous thing to do (as is calling the MoveMemory api from your code, and for the same reasons)

In this case there is a difference.
Wrong calls on the variant i mean, go low level for safearrays.
But then, there now better ways to pass data back and forth by not using late bound com interfaces and not using variants.

[Patrick Mills]

>Bingo - silent fail of POKE, no error triggered

BTW, this is a dangerous assumption on your part.

I suppose so, but I actually did realize that the problem wasn't necessarily the POKE (even though that *is* what I wrote As far as the source array being corrupted - even though my posted code didn't show this, I have other versions which allow me to see the before/after contents of each array. After the variant assignment completes, not only is array1 equal to array2, but they are both still equal to an untouched third array with the same data. After the peek/poke failure, array1 and the (unshown in the posted code) third array are still equal and untouched, and array2 is in its initialized state from the REDIM. Not a single byte of it is changed by the PEEK/POKE code. As far as my program is concerned, the PEEK/POKE line effectively becomes a NOOP (at least with respect to the running code - perhaps 400 million bytes of *something* *somewhere* are being changed, but there is no GPF or other visible failure apparent).

The odd thing is that I can add other Peek/Poke code to the example and it continues to work, right alongside the code that fails.

If I add a loop that uses Peek/Poke to move data 4-bytes at a time from the first array to the second, that Peek/Poke code succeeds right after the bulk copy peek/poke fails. I can begin increasing the sizes of the moves from 4 bytes to 8 to 16, etc. and things still work. By process of elimination, I could probably come up with the point at which it stops working, but I'm losing interest in doing that, since this is just experimental code.

Anyway, thanks for everyone's indulgence. I apologize if you all think this was a waste of time. If at some point in the future I really need to do a large variant assignment followed by an Peek/Poke that mysteriously fails, and I can't think of any other way to do what I need, I guess I'll contact PB support. Otherwise, I think my need to understand this problem has reached its limit, due to lack of any practical impact.

[Michael Mattias]

Three quick points:

• Test code which does not employ all the error-testing mechanisms available is not really test code.
• I have over the years encountered 'flaky' performance with ALL software - not just software I have created with the PB compilers - when operating with "really big" data sets or "near" some documented limit.
• PB's support department is where you should go if it appears the compiler is doing something naughty, including the naughtiness of not catching and reporting an error condition. Surely you would like your users to report difficulties to you, regardless if they post them in some peer support forum or not.

MCM

[Michael Mattias]

> Not a single byte of it is changed by the PEEK/POKE code

That's the key. One of those two statements (the PEEK or the POKE) failed.

Since they are in an expression now, you can't separate what's happening.

Try:

Instead of ...

       POKE$  address, PEEK$(address,size)

Change to two lines:

 S = PEEK$ (address, size )  
 POKE$  address, S 
 ' should add at this point: 
 S= ""   ' deallocate string

If that fails to set an error code, you can analyze the string S to see if it "PEEKed" correctly. If S is incorrect, you have found a bug in the compiler.

(Yes, it IS the same code. When you use an expression like this, the compiler is creating "S" anyway, it just does it behind the scenes).

But take a look at my suggested design in your other thread. That uses a LOT less system resources.

I was thinking about that some more and now realize I only went halfway. There's no reason you could not also use an MMF for the "working" array. That you can map only PART of the file suggests that by working in pieces you can accomodate even larger sets of data without breaking the memory bank.

MCM