Is this safe?

[William Burns]

On a multi-thread application, is it safe to use a global array of strings across the threads if they are reading only and nothing is changing the data while it runs? In other words if the string array is filled before the threads run and nothing is adjusting the data in anyway while it runs. The only thing the threads are doing is array scans and reading the data. Could that cause memory corruption? And if so, how? :think:

PS. I know this is not ideal and I would love to find a way to avoid globals and statics. But speed is critical on this app and so far in my testing nothing else has come close to the same speed.
:daz:

[Roger Garstang]

I don't see anything wrong with it. I actually prefer globals in my threaded apps rather than find a less efficient way using other types.

[Mike Trader]

Read this
http://www.powerbasic.com/support/pb...ghlight=thread

Its hard to get a thread based GPF, but as Michael points out, data corruption is allways a threat.

My personal solution du jour is to put any global memory access inside a function and enclose it in a Critical Section.

In this way it is not possible for two threads to be operating on the globals at once.

The function is a handy centralized device to ensure that you remember to enclose all relevant code in the critical section.

[Michael Mattias]

I have an application in which I do exactly this. It's perfectly safe as long you you control the update of that data correctly.

[William Burns]

Mike, I also like to put these types of shared things inside a function and use criticalsections around that function. (or on number globals use Interlocked__ functions) But in this case, speed is a must and when I tried putting the array searches in a criticalsection I noticed a big speed decrease. (I guess I should have said this program runs millions of threads in an 8 hour timeframe... each one does searches on this memory array) So if it is safe, it would be better to leave it out of a criticalsection.

I know that multi-threads which are only reading global DWORD or LONG variables is generally safe (if nothing is changing them). But I was not sure if strings and arrays would have more catches and things to watch out for. For example I am not sure what "Array Scan" is doing behind the scenes and if it is thread safe. (the helpfile does not say... at least that I can find)

FYI, I am using PBWin 7.04.

[Bob Zale]

It is safe.

Bob Zale
PowerBASIC Inc.

[William Burns]

Thanks MCM. That is what I was hoping to hear.


As a side question. Would it still be safe on global LONG values if I use Interlocked__ functions to update them, but a normal check routine to read them at the same time?

Example: One thread is updating the global using InterlockedIncrement gValue and another thread is reading it at the same time with something like: If gValue > 50 then ... I think it would, but maybe there is a way to use the InterlockedCompareExchange to do the comparision instead?

[Michael Mattias]

Example: One thread is updating the global using InterlockedIncrement gValue and another thread is reading it at the same time with something like: If gValue > 50 then ... I think it would, but maybe there is a way to use the InterlockedCompareExchange to do the comparision instead?

Sure, that's "safe" - if "safe" means "No protection faults."

It is, however, "silly." How can your application possibly produce correct output if the data you are using in FUNCTION "A" is subject to change without notice by actions occuring in FUNCTION "B" during your calculations?

[William Burns]

Thanks Bob, I feel much better about that part now.

Unfortunately I have something causing GPFs in this program and I am still hunting the culprit. I have done all the normal things like #DEBUG ERROR ON and putting every function inside a TRY/CATCH (which I have never gotten any errors trapped so far) I also test all pointers to make sure they are not null before using. And I never change any buffer that was created by an outside API. I also check all API return codes to see if anything is failing. I made sure that all buffers sent to outside APIs are correctly sized etc.
Unfortunately the GPFs are not recreatable. In other words I sometimes can run the full 8 hours without any problems, and sometimes it will bomb several times. Which I realize is the nature of the problem, but it still makes it a nightmare to track down. It would probably cause a GPF if I tried to use TRACE since it would likely fill up the drive before crashing. I have been over the code soo many times that now I am looking in the less likely causes. For example I tried #REGISTER NONE at the top thinking it might be a fluke thing. I have tried putting all globals in critical sections.(the same critical section for anything to do with that variable. read or write) That only slowed things down to a crawl, but still GPF'd.

[William Burns]

It is, however, "silly." How can your application possibly produce correct output if the data you are using in FUNCTION "A" is subject to change without notice by actions occuring in FUNCTION "B" during your calculations?

It might sound silly, but it is useful in some cases. For example I use this as a method to control the number of running threads at the same time. I have a main thread that takes a list to process and then when it runs a new thread it uses InterlockedIncrement giThreadCount to count the thread and before it runs another thread it checks the count to make sure it is not too high. Then in the thread function it will use InterlockedDecrement giThreadCount to decrease the count when it exits. Can you think of a better way to control a thread count? I can't let it run a million threads at the same time.

[Michael Mattias]

>Can you think of a better way to control a thread count

CreateSemaphore() ?

(We have no demos of using semaphores in Source Code Forum. Maybe I'll do one just so we do.. unless someone else wants to volunteer).

[Michael Mattias]

it runs a new thread it uses InterlockedIncrement giThreadCount to count the thread and before it runs another thread it checks the count

BTW, this is not accurate anyway. You do not control when Windows effects a thread switch. Eg in your thread function...

FUNCTION MyThreadFunction (...
  Yadda: Yadda: Yadda
  Badda Bing: Badda Bang: Badda Boom
 ....
 InterlockedDecrement GlobaThreadCountVar
END FUNCTION

... there is no assurance that Windows won't effect a thread switch between the conclusion of the InterlockedDecrement call and the final "ret" instruction marking the end of the function.

You think debugging is hard now? Just try to re-create that error!!

MCM

[Mike Trader]

I also test all pointers to make sure they are not null before using.

Also try using:
IF IsBadWritePtr(BYVAL pWrite, BYVAL LenBytes) THEN ...

This helps, but it is NOT foolproof. I have seen situations where I will feed it a bad pointer and it does not catch it (or perhaps it does but the app GPFs anyway)

(We have no demos of using semaphores in Source Code Forum. Maybe I'll do one just so we do..

That would be great Michael. I study your examples very carefully.

[Roy Cline]

Are you interacting with other programs?
Is there a low memory condition?
What O/S are you using?

[Michael Mattias]

have seen situations where I will feed it a bad pointer and it does not catch it (or perhaps it does but the app GPFs anyway)

IsBadXXXXPtr functions work perfectly.

Remember, all those functions tell you is if you own that memory and can write to it. Those functions should have named "IsInvalidxxxPtr, because that's what they test... they don't test that it is the correct value.

Eg, you own your stack and can write to it.. so if you ask Windows if that's a good write pointer, Windows will say, "sure, no problem." (Hint: do not try this at home).

PS: Any LOCAL scalar variable is on your stack, in case you do want to try this at home anyway.

[William Burns]

Sorry for the delay, I could not post replies because I updated my email on my profile and the forum had me locked until I confirmed it via my home email account. (I was at work so I did not have access to my home email)

Mike, I was using the IsBadReadPtr() already. (I don't have any pointers that I write to, but I do have some that I read from.)

Roy, I am running XP pro SP2. I monitored the memory and handles and it is not getting low. (I have 4gb, so its not likely to either ) I use the ProcessExplorer from SysInternals to monitor the programs to make sure I don't have mem or handle leaks. (very cool program in case you haven't tried it) The program does some netbios queries to outside PCs, but it does not interact with any outside programs.

MCM, as for the global used for the thread count, you are correct, there is a possibility that it will context switch after the decrement and before the function ends which will result in starting an extra thread. But since this is only for throttling the number of threads, it will not really matter if it goes up a few past the limit. (not something that has to be exact) I have used this method for years and it has always worked great.

[William Burns]

Found the problem!

And now for the best part. I found the problem. I have been searching for this bug for months with no success. But today I finally found a way to consistently get it to GPF, which made it much easier to narrow down the code and find it. It was a stupid mistake. (the ones that cause memory corruption usually are)

See if you can spot it: (I will make is easy by just posting the few lines of offending code, which is much easier than looking for it in the 26,000+ lines around it :coffee3: )

sGroup = Space$(17)
For iCount = 0 To iNumNames
   If ((@pMyData.name_array(iCount).name_flags And %GROUP_NAME_FLAG) = %GROUP_NAME_FLAG) _
   And (@pMyData.name_array(iCount).nb_name(15) = %NAME_TYPE_DOMAIN) _
   And (IsBadReadPtr(ByVal VarPtr(@pMyData.name_array(iCount).nb_name(0)),15) = 0) Then
      MoveMemory ByVal StrPtr(sGroup), ByVal VarPtr(@pMyData.name_array(iCount).nb_name(0)), ByVal 15
      sGroup = Trim$(sGroup)
   End If
Next iCount

Does any one else see it? :whistle:

Like I said, it was a dumb mistake. :dang:

[Michael Mattias]

See if you can spot it....Does any one else see it?

sGroup = Space$(17)
For iCount = 0 To iNumNames
   If ((@pMyData.name_array(iCount).name_flags And %GROUP_NAME_FLAG) = %GROUP_NAME_FLAG) _
   And (@pMyData.name_array(iCount).nb_name(15) = %NAME_TYPE_DOMAIN) _
   And (IsBadReadPtr(ByVal VarPtr(@pMyData.name_array(iCount).nb_name(0)),15) = 0) Then
      MoveMemory ByVal StrPtr(sGroup), ByVal VarPtr(@pMyData.name_array(iCount).nb_name(0)), ByVal 15
      sGroup = Trim$(sGroup)
   End If
Next iCount

Of course we can't spot it: Insufficient code shown.

We can't see the definition of pMyData or its underlying UDTs. Because we can't see the underlying UDTs we have no clue if the nb_name member table's elements are indexed from 0 to 15 (or greater) and the name_array table member of @pMydata at least in the range zero to inumnames (which value we don't know, either, because the assignment statement is not shown).

Unless you are referring to the test for the bad read ptr. This test is unecessary, since the syntax used if only valid for UDT variables and therefore are automatically valid. Unless of course the nbname member is some kind of PTR, in which case you should be testing the pointer itself, not it's address (VARPTR). BUt that's not really an error, unless the nbName member is some kind of PTR.

MCM

[William Burns]

MCM, I am surprised you didnt see it. I admit the pointers and UDTs add to the confusion, but the actual error is not related to them and there is enough code listed to show the problem.

I will give you a hint. (to trim or not to trim)

:shhh:

[Michael Mattias]

OK, I see it now.

Well, that's what you get .. no, make that that's what you deserve for using inline numeric literals instead of functions or variables.

MCM