Network collision of a read and a write

[Mark Hunter]

Here’s the setup:
... A network.
... One plaintext file on the server.
... One fixed client who can write to that file.
... Several clients who can read it.

Because only one client is allowed (enforced by the program) to write to the file, I do not have the program put a write lock on the file when writing.

I do not have it put a read lock on it either.

Question (really ignorant, but I’m not really a programmer):
... Is it always safe for a reader client to read the file, including when the writer client is writing to it?

By “safe” I mean that the reader gets the file contents uncorrupted and entire, either what it was before the write or afterwards, it doesn’t matter which.

(At present I’m using VB on this, but presumably the answer for PBWin is the same. Also, eventually I’m converting to PBWin.)

[Michael Mattias]

To answer your specific question, that is can you get the entire contents of the file and you don't care if before or after.... maybe. It's possible you have only written half the file when you read. But other than that, if you don't care about the data, you're OK.

That's not to say this is a good design.

If you'd like a primer on handling shared disk files, you might find this of interest:

"Fundamentals Of Multi-User Programming." Article published in December 1995 issue of "BASICally Speaking" magazine discussing the principles of writing multi-user programs; code samples in BASIC. Rich Text format; placed in the Public Domain June 2005
http://www.talsystems.com/tsihome_ht...rogramming.rtf

[Mike Doty]

The writer could write while reader is reading.
You should lock the file while writing and the reader would retry.

'Writer
h = FreeFile
Open "c:\junk.txt" For Binary Lock Write As #h
MyString = " "
Lock #h
Put #h, 1, MyString$
Unlock #h
Close #h
 
'Reader
On Error Resume Next
h = FreeFile
Open "c:\junk.txt" For Binary As #h
Count = 0
MyString = " "
Do
  ERR.CLEAR
  Count = Count + 1
  Get #h, 1, MyString$
  If Err Then Sleep 250
Loop Until Err = 0 Or Count > 3
If Count > 3 Then
  MsgBox "Unable to read"
End If
Close #h

Many variations

Count = 0
MyString = " "
Do
  Err.Clear
  Get #h, 1, MyString$
  If Err = 0 Then Exit Do
  Count = Count + 1
  If Count > 3 Then Exit Do
  Sleep 250
Loop
If Err Then
  MsgBox "Unable to read"
End If
CLOSE #h

[Mark Hunter]

MCM,

Interesting article from the time of DOS and SHARE. The principles are the same today and your article contains some ideas I might be able to use one day. Thanks.

Still my specific question remains unanswered. Is there a problem? Can one program read a file while it’s being written to by another?

I don’t see why just doing it is necessarily bad design, as you intimate. The O/S may well address the type of collision we’re talking about, and the programs don’t have to. I’d like to know.

Mike D.,

Yes, that’s how to deal with the problem, if there is a problem.

In practice – actually setting up the situation described in my first post and doing many random writes and reads – the answer appears to be that there’s no problem. (I say appears because it’s impossible, at least at my level of expertise, to know when there’s been a collision.) But it would be nice to know the official answer.

[Mike Doty]

Yes, there is a problem.

The writer could change information at any position while others are reading. The writer could be ahead or behind of the readers.
"My dog has fleas" could be read as "My dog how are you".
"1 + 1 = 2" could be read "1 + dog has fleas .

The default open (for other processes) is LOCK READ WRITE so without seeing your open statement on the writer you may be locking.

[Mike Doty]

Here is a multi-threaded PB/CC program that might be modified to test
opening in different modes.

#COMPILE EXE
#DIM ALL
GLOBAL gs AS STRING
FUNCTION PBMAIN () AS LONG
  CLS
  ? "Press [Esc] key to end program"
  LOCATE 10,1:PRINT "WRITER:";
  LOCATE 15,1:PRINT "READER:";
  LOCAL hReader AS LONG
  LOCAL hWriter AS LONG
  LOCAL x       AS LONG
  LOCAL sKey    AS STRING
  
  'Switch these lines around for different test results
  THREAD CREATE Writer (x), TO hWriter: SLEEP 500 'allow to allocate and give a head start
  THREAD CREATE Reader (x), TO hReader :SLEEP 50
  
  DO
    sKey = UCASE$(WAITKEY$)
    IF sKey = $ESC THEN gs = "END":EXIT DO
  LOOP
  BEEP
  LOCATE 2,1: ? "Closing all threads, ending in 5 seconds";
  
  THREAD CLOSE hWriter TO hWriter: IF hWriter = 0 THEN ?:? "Error closing hWriter thread"
  THREAD CLOSE hReader TO hReader: IF hReader = 0 THEN ?:? "Error closing hReader thread"
  SLEEP 5000
END FUNCTION
  
THREAD FUNCTION Writer(BYVAL something AS LONG) AS LONG
  LOCAL x AS LONG, h AS LONG, s AS STRING
  h = FREEFILE
  OPEN "\junk.txt" FOR OUTPUT AS #h  'exclusive access
  'OPEN "\junk.txt" for output lock WRITE AS #h  'only lock others from writing
  IF ERR THEN
    LOCATE 4,1
    ? "Writer: Unable to open exclusive, exiting thread with error" + STR$(ERRCLEAR)
    EXIT FUNCTION
  END IF
  
  DO
    INCR X
    s = "This is line" + STR$(x)
    PRINT #h, s
    LOCATE 10,10
    ? s,
    IF ERR THEN EXIT DO
    SLEEP 100
  LOOP UNTIL LEN(gs)
  LOCATE 11,1:? "Writer last error" + STR$(ERRCLEAR)
  CLOSE #h
END FUNCTION
THREAD FUNCTION Reader(BYVAL something AS LONG) AS LONG
   LOCAL h AS LONG
   LOCAL s AS STRING
   h = FREEFILE
   OPEN "\JUNK.TXT" FOR INPUT SHARED AS #h
   IF ERR THEN
     LOCATE 19,1
     ? "Reader: Unable to read file, error" + STR$(ERRCLEAR);
     EXIT FUNCTION
   END IF
   DO
     LINE INPUT #h, s
     LOCATE 15,10
     ? s,
     SLEEP 100
   LOOP UNTIL ERR OR LEN(gs)
   LOCATE 16,1:? "Reader last error" + STR$(ERRCLEAR);
   CLOSE #h
END FUNCTION

[Joseph Cote]

With small files like the ones given as examples, there seems to be a low likelihood of collisions for two reasons. First, read and write happens so quickly there's little chance for collision. Second, the OS buffers reads and writes even if you don't; I'd expect the buffer size to be the cluster size or a multiple of it, 16k or 32k, I don't even know anymore. Almost no chance of a mixed up file here.

Could be a problem when you're dealing with files longer than the internal buffer length. Perhaps the Reader is stil working on the first buffer's worth, Writer steps on the file and the second buffer comes from the new edition.

Safest thing is to write lock. For short files, the reader won't be held up for long. For longer files, you eliminate any possibility of mixed data.

And one question re the examples already given: Would you not want to let the Writer also have retries on file open? Suppose a Reader is taking their sweet time about reading and releasing the file?

[Mark Hunter]

JC,

The file referred to in my last post was about 60K, so the buffer would not have prevented the problem.

As you suggest maybe a collision never happened. File operations are so fast that a collision would be like two bullets at Gettysburg hitting one another. (Seems I read somewhere that it happened.)

Here’s what was at the back of mind when I first asked the question we’re discussing. If I were writing an operating system I would have it automatically lock out reads – lock out but hold the request – while a file was being written to – then automatically remove the lock after the write was finished and then service the request. (There would be a timeout value, an error would be returned to the reader if the write took too long.)

Why not? Well, every time you read a file (on a network) you would have to check if it failed from the writer taking too long. But isn’t this easier than always having the Writer program do a lock write and the Reader program a persistent open?

You worry about the Writer as well as the Reader:

Would you not want to let the Writer also have retries on file open? Suppose a Reader is taking their sweet time about reading and releasing the file?

I think I understand. In Mike’s examples the Reader opened the file as shared. I gather you’re saying that the Reader as well as the Writer needs to lock the file, so the Writer can’t write in the middle of the read.

Mike D.,

Thanks for the suggestion. The test I described in my last post was on a network, not a simulation on one PC. I thought of trying something like your code on my (all alone) PC, but worried that a negative result wouldn’t mean anything because the fact that the reader and writer were the same PC might prevent the very problem I was trying to find.

Threads are only an illusion. They don’t really run simultaneously.

If you have a multi-core PC a number of threads (up to the number of cores) might be able to really run simultaneously as far as computing, but when it comes to writing to one hard drive ...

[Michael Mattias]

>Suppose a Reader is taking their sweet time about reading and releasing the file?

Then the reader's programmer needs to re-take "Multiuser Applications 101" , since he obviously did not pay attention in class the first time.

In the article I discuss the proper sequence for doing these things. (OK, so that is subjective, but darned if don't work). This sequence does not include holding a lock on the file for a 'sweet time' either when reading or writing.

Of course, ny sequence assumes there is some semblance of caring about what data one reads, which apparently is not a requirement here. (I won't even ask. )

MCM

[Mark Hunter]

MCM,

Doubtless JC was engaging in hyperbole – exaggeration to make a point. After all, these physical process do take some positive interval of time. Instants do not exist.

Thanks for describing that five-step sequence in your article. I think I understand how it works. One possible problem might be in the collation of fields when two users edit the same record. What if they both change the same field? Or the reasonable options for one field depends on the value of another, and one user edits one and the other the other? There must be a protocol for handling that situation.

Anyway, per my original question I don’t have simultaneous Writers. To return to that question, I’ve decided to be conservative and have both Reader and Writer lock out everything, and consequently use a “persistent” open routine. I might be violating “if it ain’t broke don’t fix it.”

[Mike Doty]

What does your OPEN statement look like on the writer?

Your open statement on the writer takes care of everything.
If you OPEN for OUTPUT you get exclusive permission.
Just check that you get err = 0 after the open.
The readers do close the file, right?

[Michael Mattias]

One possible problem might be in the collation of fields when two users edit the same record. What if they both change the same field?

He who goes last wins.

Or the reasonable options for one field depends on the value of another, and one user edits one and the other the other?

Then the screen and/or the edit was poorly designed and/or implemented. Very poorly. If two fields are dependent like this, they must be read and written as a matched pair.

As I point out.... the reason you pay more for the "multiple user version" of an application is because there is a lot more to writing a multi-user application than throwing a couple of "SHARED" clauses into the OPENs.

MCM

[Guy Dombrowski]

Michael,

I have read your 101 text and it is still just as good today as before. I have found that a TRY/CATCH/EXIT TRY loop is a very good way to implement those principles.
I add a SLEEP 1 to the loop and if needed, a message to give a warning about the locked field.
Works like a charm and so simple to write.

[Mark Hunter]

MCM’s article about multi-user access of a file considers only random access files. In the situation posited at the start of this discussion (one Writer, many Readers) it’s possible to use a sequential file if it’s not too large. I’ll describe how and discuss the advantages and disadvantages compared with a random access file in a separate thread one of these days.

Mike D.,

The Writer opens the database as a sequential file, like this:

Open database For Output Lock Write As #

The Lock Write is unnecessary because there’s only one Writer. I put it in just in case something goes wrong and there’s not.

A Reader opens the database as a sequential file too, like this:

Open database For Input Shared As #

>.. Your open statement on the writer takes care of
>.. everything. If you OPEN for OUTPUT you get
>.. exclusive permission. Just check that you get
>.. err = 0 after the open.

Yes it checks. Does this mean that the lock business you recommended earlier is unnecessary?

>.. The Readers do close the file, right?

Yes, naturellement.

[Michael Mattias]

MCM’s article about multi-user access of a file considers only random access files. In the situation posited at the start of this discussion (one Writer, many Readers) it’s possible to use a sequential file if it’s not too large

Huh?

There are no such things as 'random access' or 'sequential' files. A file is a file is a file. 'RANDOM' or 'SEQUENTIAL' is how you choose to access the data therein.

Using BASIC, you have your choice of two random access open modes (RANDOM and BINARY) and three sequential access open modes (INPUT, OUTPUT and APPEND).

MCM

[Mark Hunter]

MCM,

Yes (though see below). To make an analogy, in electronics you have “pull up resistors,” “current limiting resistors,” etc. But they’re all just resistors. The adjective refers not to a difference in the resistor itself but rather to how it is used.

The jargon can be confusing to the novice in electronics. The adept is used to it.

Similarly with files and how they are used. The PB help sometimes refers to random access files etc. and other times to files opened for random access etc.

My analogy isn’t perfect though, because you can almost always tell a file that will be opened as random access from one that will be opened as sequential. The latter have commas and quote delimiters. In the former you can usually make out the records and fields by inspection with a byte editor and the fields will be the same length from record to record. Typically a binary file – excuse me, a file to be opened as binary – has neither delimiters nor a uniform structure.

[Michael Mattias]

>Typically a binary file – excuse me, a file to be opened as binary – ..

Excellent Catch!

The difference between sequential access mode and random access mode is simple: using a random access mode, you can go to any point in the file at any time and read or write as much or as little as you want; with sequential access modes, you have your choice of starting at the beginning or starting at the beginning and going forward only.

That said, within the BASIC language the "file reading and writing" statements used are different depending on the BASIC-language OPEN mode; these statements are nicely organized to work with the types of data typically found in files opened in that particular access mode; eg, LINE INPUT (sequential access) works perfectly for getting the file data "from current file pointer to next CRLF" and PUT (random access) is perfectly suited to reading or writing "a fixed number of bytes at the current file pointer."

To put this back on track a bit... I think you will find that disk files which must be shared among multiple users are almost always easier to work with when designed for random access. Not that a 'text' file (variable-length CRLF-delimited records) cannot be shared, but it's a lot harder to program file sharing with this structure than is programming fixed-length records.

MCM

[Mark Hunter]

Mike D.,

Another question: in your code
... Open "c:\junk.txt" For Binary Lock Write As #h
... MyString = " "
... Lock #h
... Put #h, 1, MyString$
... Unlock #h
... Close #h
why bother with Lock Write in the Open statement when there is Lock / Unlock about Put ? Alternatively, couldn’t you just use Lock Read Write in the Open statement and dispense with the later Lock / Unlock ?

[Mark Hunter]

MCM,

> Excellent Catch!

You’re being sarcastic. It’s obvious (my keyboard has backspace and delete keys) that my “excuse me” was a bit of whimsy, intended to show that the term “binary file” is in common use and everybody but the neophyte knows what it means.

I don’t see why you’re arguing about this. Look at the PB help, as I’ve suggested already. It uses the terminology I use and that everybody uses. You seem to be arguing against the dictionary about the meaning of a word.

Why do people do it? Why does an electronics engineer say “this is a current limiting resistor” instead of “this is a resistor I’m using to limit the current?”

Why turn usage into an attribute?

I can think of two reasons:
1. It saves words. In the example, three versus six. Brevity is a virtue
2. It compresses two concepts into one, which allows more efficient thinking. (As long as you know what you’re doing, so you can separate the concepts when necessary.)

[Michael Mattias]

why bother with Lock Write in the Open statement when there is Lock / Unlock about Put ?

LOCK WRITE will prevent another process from even OPENing the file in any output-capable mode.

Using LOCK/UNLOCK statements without LOCK WRITE in the OPEN will allow other processes to open the file, and to read and write except when another process has locked the 'region of interest.'

Normally with multiuser applications, you lock the file or region of the file when need it exclusively. However, you only 'hold' that lock for the minimum time required. If you try to read or lock a file, you can detect that you can't because another user has locked that region of the file (error code 70 permission denied), and you 'wait' a little bit and try again. Hopefully the other user is done and has unlocked the region, and now it's your turn.

MCM