Announcement

**Paul Purvis** · 23 Sep 2007, 05:33 AM

paul, i too have been working on file comparisons

how do you like those first 3 words, ha ha ha.
i have just been working on such a thing too, here is the best i can do as of yet, the code has some testing features.
i have just lost a lot of time runing this code in comparing a large file(982 megabytes) on the hard drive to a cdrom/dvd drive and this program would lockup.
well come to find out, the cdrom/dvd had some bad track errors on it, or at least this program could not read the file. i am still doing some testing on what happened, as another computer could read the file, when i was reading the disk 1 byte at a time.
so if you are doing any file comparisons on a removeable drive, i found out that first, it would be nice to always know the file is readable,period.
i will never get that time back.

anyways here is some starter code, maybe somebody knowing more about how pb compiles(creates machine code) can make this code faster if possible.
i am working with win2k
my intention is turn this code into a callable function inside a program to compare directories.

Code:

'COMPILER PBCC 4.03

#COMPILE EXE
'#DIM ALL
#REGISTER NONE

FUNCTION PBMAIN () AS LONG
#REGISTER NONE
REGISTER TBLOCKS AS LONG
REGISTER BLOCKSIZE AS LONG
DIM x1 AS ASCIIZ PTR,x2 AS ASCIIZ PTR

filematch&=0&
BLOCKSIZE=32768&*8&
file1$="temp1.iso"
file2$="temp2.iso"
hfile1&=FREEFILE
hfile2&=FREEFILE


'TRY
'OPEN file1$ FOR INPUT LOCK SHARED AS #hfile1&
'CATCH
'filematch&=-1
'EXIT TRY
'FINALLY
'CLOSE #hfile1&
'END TRY

'TRY
'OPEN file2$ FOR INPUT LOCK SHARED AS #hfile2&
'CATCH
'filematch&=filematch&-2
'EXIT TRY
'FINALLY
'CLOSE #hfile2&
'END TRY

'IF filematch& THEN GOTO ENDOFFUNCTION

ERRCLEAR
OPEN file1$ FOR BINARY LOCK SHARED AS #hfile1&  'LEN=65536&
IF ERR THEN filematch&=4:GOTO ENDOFFUNCTION
OPEN file2$ FOR BINARY LOCK SHARED AS #hfile2&  'LEN=65536&
IF ERR THEN filematch&=4:GOTO ENDOFFUNCTION

loftemp1&=LOF(hfile1&)
loftemp2&=LOF(hfile2&)

IF loftemp1&=0 THEN
    IF loftemp1&=loftemp2& THEN GOTO ENDOFFUNCTION
    IF loftemp1&<loftemp2& THEN filematch&=1:GOTO ENDOFFUNCTION
    filematch&=2
    GOTO ENDOFFUNCTION
END IF

IF loftemp1&<>loftemp2& THEN
   IF loftemp1&<loftemp2& THEN filematch&=1:GOTO ENDOFFUNCTION
   filematch&=2
   GOTO ENDOFFUNCTION
END IF

blockextra&=loftemp1&  MOD BLOCKSIZE
tblocks=loftemp1&\BLOCKSIZE

t1&=TIMER

STDOUT "processing"

IF tblocks=0 THEN GOTO READLAST

TEMP1$=SPACE$(BLOCKSIZE)
TEMP2$=TEMP1$
x1=STRPTR(temp1$)
x2=STRPTR(temp2$)
WHILE TBLOCKS
GET$ #hfile1&, BLOCKSIZE,@x1
GET$ #hfile2&, BLOCKSIZE,@x2
'PRINT  ASC(TEMP1$)
'PRINT  ASC(TEMP2$)
IF TEMP1$<>TEMP2$ THEN filematch&=3&:GOTO endoffunction
IF TBLOCKS MOD 600 = 0 THEN SLEEP 16
DECR TBLOCKS
WEND


READLAST:
IF blockextra& THEN
TEMP1$=SPACE$(blockextra&)
TEMP2$=TEMP1$
x1=STRPTR(temp1$)
x2=STRPTR(temp2$)
GET$ #hfile1&, blockextra&,@x1
GET$ #hfile2&, blockextra&,@x2
'PRINT ASC(TEMP1$)
'PRINT ASC(TEMP2$)
IF TEMP1$<>TEMP2$ THEN filematch&=3&:GOTO endoffunction
END IF


IF ERR THEN
     filematch&=-4
     PRINT ERR
END IF

ENDOFFUNCTION:

CLOSE #hFile1&,#hfile2&
ERRCLEAR

t2&=TIMER
FUNCTION=filematch&
STDOUT STR$(filematch&)
STDOUT STR$(t2&-t1&)
WAITKEY$
END FUNCTION

**Michael Mattias** · 23 Sep 2007, 10:40 AM

Could be optimized, but this ain't too bad...
Compare Two Files and return location of first mismatch

MCM

**Michael Mattias** · 23 Sep 2007, 11:26 AM

Code:

DIM x1 AS ASCIIZ PTR,x2 AS ASCIIZ PTR
...
GET$ #hfile1&, BLOCKSIZE,@x1
GET$ #hfile2&, BLOCKSIZE,@x2

I have no clue how this could even come close to working. X1 and X2 have no size. It's also a type mismatch so I don't think this should even compile.

**John Gleason** · 23 Sep 2007, 02:37 PM

Paul (Johnson),
This is a windows program I did a couple years ago and have been using since then with no known issues yet. It is probably way more than you need--it has drag n drop, send to/command line functionality, timers etc.--but I see it does the actual comparison in MMX which is fast. It's a ton of spaghetti but you may be able to use the inner core algorithm, so FWIW:

Code:

#PBFORMS Created


#COMPILE EXE
#DIM ALL
#REGISTER NONE
'--------------------------------------------------------------------------------
'   ** Includes **
'--------------------------------------------------------------------------------
#PBFORMS Begin Includes
#RESOURCE "compareUm.pbr"
#IF NOT %DEF(%WINAPI)
    #INCLUDE "WIN32API.INC"
#ENDIF
#INCLUDE "PBForms.INC"
#PBFORMS End Includes
'--------------------------------------------------------------------------------

'--------------------------------------------------------------------------------
'   ** Constants **
'--------------------------------------------------------------------------------
#PBFORMS Begin Constants
%IDR_IMGFILE1   = 102
%IDD_DIALOG1    = 101
%IDC_BUTTON1    = 1001
%IDC_TEXTBOX1   = 1002
%IDC_TEXTBOX2   = 1003
%IDC_LABEL1     = 1004
%IDC_LABEL2     = 1005
%IDC_BUTTON2    = 1006
%IDC_LABEL3     = 1007
%IDC_BUTTON3    = 1008
%IDC_CHECKBOX1  = 1009
%IDC_CHECKBOX2  = 1010
%IDC_BUTTON4    = 1011
%IDC_BUTTON5    = 1012
%IDC_BUTTON6    = 1013
%IDC_BUTTON7    = 1014
%IDC_BUTTON8    = 1015
%IDC_BUTTON9    = 1016
%IDC_LISTBOX1   = 1017
%IDC_COMBOBOX1  = 1018
%IDC_LABEL4     = 1019
%IDC_LABEL5     = 1020
#PBFORMS End Constants
'%TENMEGS    = 1000000
%SLEEPSEC    = 150
%DISPLAYMOD = 0
$TEXT1       = " " & CHR$(126) & "File #"
$TEXT2       = CHR$(126) & "  "
$TEXT3       = " Bytes copied to "
%MSGBOXLABEL = 3001
%SIXTEEN     = 16
'%SLEEP2      = 100
%COMPARECHUNK = 12000000
%COMPARECHUNKASM = %COMPARECHUNK / 4 - 1
'--------------------------------------------------------------------------------

'--------------------------------------------------------------------------------
'   ** Declarations **
'--------------------------------------------------------------------------------
DECLARE CALLBACK FUNCTION ShowDIALOG1Proc()
DECLARE FUNCTION ShowDIALOG1(BYVAL hParent AS DWORD) AS LONG
DECLARE SUB showTimedMsgBox(lText AS STRING)
DECLARE FUNCTION Exists(File AS ASCIIZ * %MAX_PATH) AS LONG
DECLARE FUNCTION setFileDatesEqual() AS LONG
DECLARE FUNCTION directoryError(fileNam AS ASCIIZ * %MAX_PATH, hWindow AS DWORD) AS LONG

#PBFORMS Declarations
'--------------------------------------------------------------------------------
GLOBAL lpMsg AS TAGmsg
GLOBAL hMainDlg AS DWORD
GLOBAL oneMeg AS LONG
GLOBAL delay2() AS STRING
GLOBAL delay2oh AS LONG
GLOBAL commandFile AS STRING
GLOBAL comFile() AS ASCIIZ * %MAX_PATH
GLOBAL tenMegs AS STRING * %COMPARECHUNK, tenMegs2 AS STRING * %COMPARECHUNK

'--------------------------------------------------------------------------------
FUNCTION PBMAIN()
    commandFile = COMMAND$
    ShowDIALOG1 %HWND_DESKTOP
END FUNCTION
'--------------------------------------------------------------------------------

'--------------------------------------------------------------------------------
'   ** CallBacks **
'--------------------------------------------------------------------------------
CALLBACK FUNCTION ShowDIALOG1Proc()
STATIC fromFile AS ASCIIZ * %MAX_PATH
STATIC toFile   AS ASCIIZ * %MAX_PATH
STATIC toDir    AS ASCIIZ * %MAX_PATH
STATIC  dropFileArr()  AS ASCIIZ * %MAX_PATH
'DIM     dropFileArr(5) AS STATIC ASCIIZ * %MAX_PATH
DIM dropFileArr(1) AS STATIC ASCIIZ * %MAX_PATH
'LOCAL tenMegs AS STRING * %COMPARECHUNK, tenMegs2 AS STRING * %COMPARECHUNK
LOCAL secondPart AS DWORD, secondPart2 AS DWORD, lastChunk AS LONG
LOCAL lastChunk2 AS LONG, ii AS LONG, ix AS LONG
LOCAL remainingLoops AS LONG, remainingLoops2 AS LONG, xMegs AS STRING, xMegs2 AS STRING
STATIC totMegs AS QUAD
STATIC thisFileMegs AS DWORD
STATIC exitLoop AS LONG
STATIC totFilesDropped AS LONG
LOCAL mb AS LONG, mbC AS LONG
LOCAL blankFile AS STRING
STATIC usb1or2 AS LONG
LOCAL badFilename AS LONG
STATIC warnOverwrite AS LONG, totErrors AS LONG
LOCAL singleDroppedFile AS ASCIIZ * %MAX_PATH
LOCAL totFilesDroppedTmp AS LONG, qq AS LONG
STATIC sCharWrit AS STRING, loopRunning AS LONG
STATIC sTime AS DWORD, rotator AS LONG
LOCAL hours AS LONG, minutes AS LONG, secs AS LONG
LOCAL copySuccess AS LONG
STATIC totFilesSize AS QUAD, totFilesSize2 AS QUAD
LOCAL fileInfom AS WIN32_FILE_ATTRIBUTE_DATA
LOCAL elapsedSecs AS LONG, totSecs AS DWORD
STATIC misMatch AS LONG
LOCAL parsCnt AS LONG
LOCAL dropPoint AS POINTAPI
LOCAL checked AS LONG, q1zz AS QUAD

'static running as long


    SELECT CASE CBMSG
        CASE %WM_INITDIALOG
            DragAcceptFiles CBHNDL, %TRUE
            oneMeg = %COMPARECHUNK
            CONTROL SET CHECK CBHNDL, %IDC_CHECKBOX2, 1
            setWindowPos CBHNDL, %HWND_TOPMOST, 0,0,0,0, %SWP_NOSIZE OR %SWP_NOMOVE OR %SWP_SHOWWINDOW
            IF commandFile > "" THEN
                REPLACE CHR$(34) WITH "" IN commandFile
                parsCnt = PARSECOUNT(commandFile, " ")
                REDIM comFile(parsCnt) AS ASCIIZ * %MAX_PATH
                PARSE commandFile, comFile(), " "
                IF (GETATTR(comFile(0)) AND %SIXTEEN) = %SIXTEEN THEN
                    directoryError comFile(0), CBHNDL
                    comFile(0) = ""
                END IF
                IF (GETATTR(comFile(1)) AND %SIXTEEN) = %SIXTEEN THEN
                    directoryError comFile(1), CBHNDL
                    comFile(1) = ""
                END IF
                CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX1, comFile(0)
                CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX2, comFile(1)
            END IF

         '   DragAcceptFiles hTbox1, %TRUE
        '    DragAcceptFiles hTbox2, %TRUE
        CASE %WM_TIMER
            IF sTime = 0 THEN sTime = getTickCount - 1000&   'take away extra second
            totSecs = getTickCount
            elapsedSecs = (totSecs - sTime) \ 1000
           ' IF elapsedSecs < 1 THEN EXIT FUNCTION
            totSecs = elapsedSecs * (totFilesSize / totMegs) - elapsedSecs
            IF rotator < 5 THEN
                hours   = elapsedSecs \ 3600
                minutes = (elapsedSecs - hours * 3600) \ 60
                secs    = elapsedSecs MOD 60
                IF hours > 0 THEN
                CONTROL SET TEXT CBHNDL, %IDC_LABEL5, FORMAT$(hours, "00\:") & FORMAT$(minutes, "00\:") & _
                                     FORMAT$(secs, "00") & ":e"
                ELSE
                CONTROL SET TEXT CBHNDL, %IDC_LABEL5, FORMAT$(minutes, "00\:") & FORMAT$(secs, "00") & ":e"
                END IF
                INCR rotator
            ELSEIF rotator < 10 THEN
                hours   = totSecs \ 3600
                minutes = (totSecs - hours * 3600) \ 60
                secs    = totSecs MOD 60
                IF hours > 0 THEN
                CONTROL SET TEXT CBHNDL, %IDC_LABEL5, FORMAT$(hours, "00\:") & FORMAT$(minutes, "00\:") & _
                                     FORMAT$(secs, "00") & ":r"
                ELSE
                CONTROL SET TEXT CBHNDL, %IDC_LABEL5, FORMAT$(minutes, "00\:") & FORMAT$(secs, "00") & ":r"
                END IF
                INCR rotator
            ELSE
                INCR rotator
                IF rotator > 11 THEN   'this IF...THEN makes avg. appear for 2 secs w/o update
                    rotator = 0
                ELSE
                    CONTROL SET TEXT CBHNDL, %IDC_LABEL5, FORMAT$((totMegs / elapsedSecs / 1000000) * 2, "0.00") & "MB/s"
                END IF
            END IF
            DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0

        CASE %WM_SYSCOMMAND
            IF CBCTL = %SC_CLOSE THEN
            IF loopRunning = %TRUE THEN
                DIALOG DISABLE CBHNDL
                mbC = MSGBOX("Oopsie, compare is not yet complete. Exit anyway?", %MB_ICONQUESTION + _
                                                 %MB_SYSTEMMODAL + %MB_YESNO, "CompareUm")
                DIALOG ENABLE CBHNDL
                setForegroundWindow CBHNDL
                IF mbC = %IDYES THEN
                    exitProcess(0)
                ELSE
                    FUNCTION = 1     'App won't terminate, even with ALT-F4
                    'Function = 0    'App will terminate
                    EXIT FUNCTION
                END IF
            ELSE
                exitProcess(0)
            END IF
            END IF

        CASE %WM_COMMAND
            SELECT CASE CBCTL
                CASE %IDC_CHECKBOX2
                    SELECT CASE CBCTLMSG
                        CASE %BN_CLICKED
                            CONTROL GET CHECK CBHNDL, %IDC_CHECKBOX2 TO checked
                            IF checked = 1 THEN
                                setWindowPos CBHNDL, %HWND_TOPMOST, 0,0,0,0, %SWP_NOSIZE OR %SWP_NOMOVE OR %SWP_SHOWWINDOW
                            ELSE
                                setWindowPos CBHNDL, %HWND_NOTOPMOST, 0,0,0,0, %SWP_NOSIZE OR %SWP_NOMOVE OR %SWP_SHOWWINDOW
                            END IF
                    END SELECT
                CASE %IDC_COMBOBOX1
                    COMBOBOX GET TEXT CBHNDL, %IDC_COMBOBOX1 TO sCharWrit
                    delay2oh = VAL(sCharWrit)
                    FUNCTION = 1

                CASE %IDC_CHECKBOX1
                    SELECT CASE CBCTLMSG
                        CASE %BN_CLICKED
                            CONTROL GET CHECK CBHNDL, %IDC_CHECKBOX1 TO usb1or2
                            IF usb1or2 = 1 THEN oneMeg = %COMPARECHUNK ELSE oneMeg = %COMPARECHUNK
                    END SELECT

                CASE %IDC_BUTTON2
                    IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN
         '           if running = %FALSE THEN
                        setTimer CBHNDL, 1, 1000, 0
                        CONTROL GET TEXT CBHNDL, %IDC_TEXTBOX1 TO blankFile
                        IF blankFile = "" THEN
                            CONTROL SET FOCUS CBHNDL, %IDC_TEXTBOX1
                            FUNCTION = 1
                            EXIT FUNCTION
                        END IF

                        CONTROL GET TEXT CBHNDL, %IDC_TEXTBOX1 TO fromFile
                        CONTROL GET TEXT CBHNDL, %IDC_TEXTBOX2 TO toFile
                        fromFile = TRIM$(fromFile)
                        toFile = TRIM$(toFile)
                        'msgbox format$(exists(fromFile))
                            IF exists(fromFile) = %FALSE THEN
                                badFilename = %TRUE
                                DIALOG DISABLE CBHNDL
                                MSGBOX "Oopsie, " & fromFile & " doesn't exist. Try again", %MB_ICONERROR + _
                                                                       %MB_SYSTEMMODAL, "CompareUm"
                                DIALOG ENABLE CBHNDL
                                setForegroundWindow CBHNDL
                                GOSUB resetAllStatics
                                FUNCTION = 1
                                EXIT FUNCTION
                            END IF
                            IF exists(toFile) = %FALSE THEN
                                badFilename = %TRUE
                                DIALOG DISABLE CBHNDL
                                MSGBOX "Oopsie, " & toFile & " doesn't exist. Try again", %MB_ICONERROR + _
                                                                       %MB_SYSTEMMODAL, "CompareUm"
                                DIALOG ENABLE CBHNDL
                                setForegroundWindow CBHNDL
                                GOSUB resetAllStatics
                                FUNCTION = 1
                                EXIT FUNCTION
                            END IF
                        TRY
                            OPEN fromFile FOR BINARY ACCESS READ LOCK SHARED AS #1
                        CATCH
                            DIALOG DISABLE CBHNDL
                            MSGBOX "Oopsie, problem with " & fromFile & "." & "  Try again", %MB_ICONERROR + _
                                                                        %MB_SYSTEMMODAL, "CompareUm"
                            DIALOG ENABLE CBHNDL
                            setForegroundWindow CBHNDL
                            GOSUB resetAllStatics
                            FUNCTION = 1
                            EXIT FUNCTION
                        END TRY
                        TRY
                            OPEN toFile FOR BINARY ACCESS READ LOCK SHARED AS #2
                        CATCH
                            DIALOG DISABLE CBHNDL
                            MSGBOX "Oopsie, problem with " & fromFile & "." & "  Try again", %MB_ICONERROR + _
                                                                        %MB_SYSTEMMODAL, "CompareUm"
                            DIALOG ENABLE CBHNDL
                            setForegroundWindow CBHNDL
                            GOSUB resetAllStatics
                            FUNCTION = 1
                            EXIT FUNCTION
                        END TRY
                        secondPart = LOF(1)
                        totFilesSize = secondPart
                        remainingLoops = secondPart \ oneMeg
                        lastChunk = secondPart MOD oneMeg

                        secondPart2 = LOF(2)
                        totFilesSize2 = secondPart2
                        remainingLoops2 = secondPart2 \ oneMeg
                        lastChunk2 = secondPart2 MOD oneMeg

                        IF secondPart2 <> secondPart THEN
                            DIALOG DISABLE CBHNDL
                            MSGBOX "Oopsie, file sizes are not the same!", %MB_ICONWARNING + _
                                                                        %MB_SYSTEMMODAL, "CompareUm"
                            DIALOG ENABLE CBHNDL
                            setForegroundWindow CBHNDL
                            DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0
                            mismatch = %TRUE
                        END IF

                        IF remainingLoops > remainingLoops2 THEN remainingLoops = remainingLoops2
                        IF remainingLoops > 0 THEN
                        exitLoop = %FALSE
                       ' running = %TRUE
                        CONTROL DISABLE CBHNDL, %IDC_CHECKBOX1
                        CONTROL DISABLE CBHNDL, %IDC_BUTTON2
                        FOR ii = 1 TO remainingLoops
                            loopRunning = %TRUE
                            GET  #1,       , tenMegs
                            IF delay2oh > 0 THEN SLEEP delay2oh
                            IF usb1or2 = 1 THEN SLEEP %SLEEPSEC
                            GET  #2,       , tenMegs2
                           ' if tenMegs <> tenMegs2 THEN goto notTheSame else goto theyAreEqual
                            'following asm = "if tenMegs <> tenMegs2 THEN"
                            q1zz = &hffffffffffffffff
                            !lea eax, tenMegs
                            !lea edx, tenMegs2
                            !movq mm6, q1zz           ;pronouced "Q ones" (all bits set to 1)
                            !mov ecx, %COMPARECHUNKASM
                          repCompare:
                            !movq mm0, [eax+ecx*4-4]
                            !movq mm1, [edx+ecx*4-4]
                            !movq mm2, [eax+ecx*4-12]
                            !pcmpeqd mm0,  mm1
                            !movq mm3, [edx+ecx*4-12]
                            !pand    mm6,  mm0
                            !movq mm4, [eax+ecx*4-20]
                            !pcmpeqd mm2,  mm3
                            !movq mm5, [edx+ecx*4-20]
                            !pand    mm6,  mm2
                            !pcmpeqd mm4,  mm5
                            !movq mm0, [eax+ecx*4-28]    ;start of next series of four, put here avoids dependency
                            !pand    mm6,  mm4

                            !movq mm1, [edx+ecx*4-28]
                            !movq mm2, [eax+ecx*4-36]
                            !pcmpeqd mm0,  mm1
                            !movq mm3, [edx+ecx*4-36]
                            !pand    mm6,  mm0
                            !pcmpeqd mm2,  mm3
                            !sub ecx, 10                 ;avoid dependency
                            !pand    mm6,  mm2
                            '------------------------
                            !jns short repCompare

                            !movd eax,  mm6
                            !psrlq mm6, 32
                            !movd ecx,  mm6
                            !emms
                            !test eax,   ecx
                            !jz short notTheSame
                            !jmp theyAreEqual
                          notTheSame:
'                            !pop esi
'                            !pop ebx
                                DIALOG DISABLE CBHNDL
                                MSGBOX "Oopsie, here the files do not match!", %MB_ICONWARNING + _
                                                                         %MB_SYSTEMMODAL, "CompareUm"
                                DIALOG ENABLE CBHNDL
                                setForegroundWindow CBHNDL
                                DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0
                                mismatch = %TRUE
                          theyAreEqual:
'                            !pop esi
'                            !pop ebx
'                          compareDone:

                          ' IF tenMegs <> tenMegs2 THEN
                          '     DIALOG DISABLE CBHNDL
                          '     MSGBOX "Oopsie, here the files do not match!", %MB_ICONWARNING + _
                          '                                              %MB_SYSTEMMODAL, "CompareUm"
                          '     DIALOG ENABLE CBHNDL
                          '     setForegroundWindow CBHNDL
                          '     DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0
                          '     mismatch = %TRUE
                          ' END IF
                            totMegs = totMegs + oneMeg
                          ' IF (ii AND %DISPLAYMOD) = 0 THEN
                                CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX2, "  " & FORMAT$(totMegs, "0,") & _
                                                         " Bytes compared to " & toFile
                                DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0
                          ' END IF
                            IF exitLoop = %TRUE THEN
                              '  CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX1, ""
                                CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX2, toFile
                                DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0
                                CLOSE
                                CONTROL SET FOCUS CBHNDL, %IDC_TEXTBOX1
                                GOSUB resetAllStatics
                                FUNCTION = 1
                                EXIT FUNCTION
                            END IF
                        NEXT
                        END IF
                  'tack on the last part of file
                        GET$ #1, lastChunk, xMegs
                            IF delay2oh > 0 THEN SLEEP delay2oh
                            IF usb1or2 = 1 THEN SLEEP %SLEEPSEC
                        GET$ #2, lastChunk2, xMegs2
                        IF xMegs <> xMegs2 THEN
                                mismatch = %TRUE
                        END IF
                        CLOSE
                        'else
                            killTimer CBHNDL, 1
                            totSecs = getTickCount
                            elapsedSecs = (totSecs - sTime) \ 1000
                            hours   = elapsedSecs \ 3600
                            minutes = (elapsedSecs - hours * 3600) \ 60
                            secs    = elapsedSecs MOD 60
                            CONTROL SET TEXT CBHNDL, %IDC_LABEL5, FORMAT$(hours, "00\:") & FORMAT$(minutes, "00\:") & _
                                     FORMAT$(secs, "00")
                        DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, CBHNDL, 0, 0, %PM_NOREMOVE) = 0
                        IF misMatch = %FALSE THEN
                                DIALOG DISABLE CBHNDL
                                MSGBOX "The files are IDENTICAL", %MB_ICONINFORMATION + _
                                                                         %MB_SYSTEMMODAL, "CompareUm"
                                DIALOG ENABLE CBHNDL
                                setForegroundWindow CBHNDL
                        ELSE
                                DIALOG DISABLE CBHNDL
                                MSGBOX "Oopsie, files did not match!", %MB_ICONWARNING + _
                                                                         %MB_SYSTEMMODAL, "CompareUm"
                                DIALOG ENABLE CBHNDL
                                setForegroundWindow CBHNDL
                        END IF
                        CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX1, fromFile
                        CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX2, toFile
                        GOSUB resetAllStatics
                        CONTROL SET FOCUS CBHNDL, %IDC_BUTTON2
                       ' END IF
'                    END IF
                    END IF

                CASE %IDC_BUTTON3
                    IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN
                        CONTROL GET TEXT CBHNDL, %IDC_TEXTBOX1 TO blankFile
                        IF blankFile = "" THEN
                            CONTROL SET FOCUS CBHNDL, %IDC_TEXTBOX1
                            killTimer CBHNDL, 1
                            RESET totSecs: RESET sTime
                            FUNCTION = 1
                            EXIT FUNCTION
                        END IF
                        CONTROL SET FOCUS CBHNDL, %IDC_TEXTBOX1
                        exitLoop = %TRUE
                        FUNCTION = 1
                    END IF
                    EXIT SELECT

resetAllStatics:
            RESET dropFileArr(): RESET exitLoop: RESET totFilesDropped
            RESET fromFile: RESET toFile: RESET toDir: RESET loopRunning
            RESET totSecs: RESET sTime: RESET totMegs: RESET rotator
            RESET thisFileMegs: RESET totErrors: RESET totFilesSize: RESET totFilesSize2: RESET misMatch
            CONTROL ENABLE CBHNDL, %IDC_BUTTON2
            CONTROL ENABLE CBHNDL, %IDC_CHECKBOX1
            killTimer CBHNDL, 1
RETURN

            END SELECT

        CASE %WM_DROPFILES
            totFilesDropped = DragQueryFile(CBWPARAM, &hFFFFFFFF, fromFile, SIZEOF(fromFile))
            IF totFilesDropped = 2 THEN
                GOSUB resetAllStatics
                DragQueryFile(CBWPARAM, 0, dropFileArr(0), SIZEOF(dropFileArr(0)))
                DragQueryFile(CBWPARAM, 1, dropFileArr(1), SIZEOF(dropFileArr(1)))
                DragFinish CBWPARAM
                IF (GETATTR(dropFileArr(0)) AND %SIXTEEN) = %SIXTEEN THEN   'it's a directory
                    directoryError dropFileArr(0), CBHNDL
                    dropFileArr(0) = ""
                END IF
                IF (GETATTR(dropFileArr(1)) AND %SIXTEEN) = %SIXTEEN THEN   'it's a directory
                    directoryError dropFileArr(1), CBHNDL
                    dropFileArr(1) = ""
                END IF
                CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX1, dropFileArr(0)
                CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX2, dropFileArr(1)

            ELSE
                DragQueryFile(CBWPARAM, 0, singleDroppedFile, SIZEOF(singleDroppedFile))
                dragQueryPoint CBWPARAM, dropPoint
                DragFinish CBWPARAM
         '       MSGBOX FORMAT$(dropPoint.x) & "   " & FORMAT$(dropPoint.y)
                IF (GETATTR(singleDroppedFile) AND %SIXTEEN) = %SIXTEEN THEN   'it's a directory
                    directoryError singleDroppedFile, CBHNDL
                    GOSUB resetAllStatics

                ELSE
                    GOSUB resetAllStatics
                    dropFileArr(0) = singleDroppedFile
                    GetFileAttributesEx dropFileArr(0), 0, fileInfom
                    totFilesSize = fileInfom.nFileSizeLow
                    IF dropPoint.y < 60 THEN
                        CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX1, dropFileArr(0)
                    ELSE
                        CONTROL SET TEXT CBHNDL, %IDC_TEXTBOX2, dropFileArr(0)
                    END IF
                END IF
            END IF


    END SELECT

END FUNCTION
'--------------------------------------------------------------------------------

'--------------------------------------------------------------------------------
'   ** Dialogs **
'--------------------------------------------------------------------------------
FUNCTION ShowDIALOG1(BYVAL hParent AS DWORD) AS LONG
    LOCAL lRslt AS LONG
    DIM delay2(13) AS STRING
            delay2(0) = "0 msec"
            delay2(1) = "1"
            delay2(2) = "2"
            delay2(3) = "4"
            delay2(4) = "8"
            delay2(5) = "16"
            delay2(6) = "32"
            delay2(7) = "40"
            delay2(8) = "60"
            delay2(9) = "80"
            delay2(10) = "100"
            delay2(11) = "120"
            delay2(12) = "140"
            delay2(13) = "160"
 #PBFORMS Begin Dialog %IDD_DIALOG1->->
    LOCAL hDlg AS DWORD
    LOCAL hFont1 AS DWORD
    LOCAL hFont2 AS DWORD



    DIALOG NEW hParent, "CompareUm", 94, 155, 490, 110, %WS_POPUP OR _
        %WS_BORDER OR %WS_DLGFRAME OR %WS_SYSMENU OR %WS_MINIMIZEBOX OR _
        %WS_CLIPSIBLINGS OR %WS_VISIBLE OR %DS_MODALFRAME OR %DS_3DLOOK OR _
        %DS_NOFAILCREATE OR %DS_SETFONT, %WS_EX_WINDOWEDGE OR _
        %WS_EX_CONTROLPARENT OR %WS_EX_LEFT OR %WS_EX_LTRREADING OR _
        %WS_EX_RIGHTSCROLLBAR, TO hDlg
    DIALOG SET ICON hDlg, "#" + FORMAT$(%IDR_IMGFILE1)
    CONTROL ADD TEXTBOX, hDlg, %IDC_TEXTBOX1, "", 49, 16, 398, 13
    CONTROL ADD TEXTBOX, hDlg, %IDC_TEXTBOX2, "", 49, 45, 398, 13
    CONTROL ADD LABEL, hDlg, %IDC_LABEL1, "File 1:", 5, 14, 38, 15, %WS_CHILD OR _
        %WS_VISIBLE OR %SS_CENTER OR %SS_CENTERIMAGE, %WS_EX_LEFT OR _
        %WS_EX_LTRREADING
    CONTROL ADD BUTTON, hDlg, %IDC_BUTTON2, "Go", 178, 69, 138, 16
    CONTROL ADD LABEL, hDlg, %IDC_LABEL3, "File 2:", 5, 44, 38, 15, %WS_CHILD OR _
        %WS_VISIBLE OR %SS_CENTER OR %SS_CENTERIMAGE, %WS_EX_LEFT OR _
        %WS_EX_LTRREADING
    CONTROL ADD BUTTON, hDlg, %IDC_BUTTON3, "Stop", 371, 69, 56, 16
    CONTROL ADD CHECKBOX, hDlg, %IDC_CHECKBOX1, "Use USB 1.1", 65, 66, 80, 9
    CONTROL ADD CHECKBOX, hDlg, %IDC_CHECKBOX2, "Keep on Top", 65, 77, 80, 9
  '  CONTROL ADD CHECKBOX, hDlg, %IDC_CHECKBOX2, "Warn if Overwrite", 65, 77, 80, _
  '      8
  '  CONTROL ADD BUTTON, hDlg, %IDC_BUTTON4, "C:", 5, 62, 12, 12
  '  CONTROL ADD BUTTON, hDlg, %IDC_BUTTON5, "D:", 19, 62, 12, 12
  '  CONTROL ADD BUTTON, hDlg, %IDC_BUTTON6, "E:", 33, 62, 12, 12
  '  CONTROL ADD BUTTON, hDlg, %IDC_BUTTON7, "F:", 5, 75, 12, 12
  '  CONTROL ADD BUTTON, hDlg, %IDC_BUTTON8, "G:", 19, 75, 12, 12
  '  CONTROL ADD BUTTON, hDlg, %IDC_BUTTON9, "H:", 33, 75, 12, 12
    CONTROL ADD COMBOBOX, hDlg, %IDC_COMBOBOX1, delay2(), 65, 89, 30, 210, %WS_CHILD OR _
        %WS_VISIBLE OR %WS_TABSTOP OR %WS_VSCROLL OR %CBS_DROPDOWNLIST, _
        %WS_EX_LEFT OR %WS_EX_LTRREADING OR %WS_EX_RIGHTSCROLLBAR
    CONTROL ADD LABEL, hDlg, %IDC_LABEL4, "USB Delay", 97, 90, 45, 12, %WS_CHILD _
        OR %WS_VISIBLE OR %SS_LEFT OR %SS_CENTERIMAGE, %WS_EX_LEFT OR _
        %WS_EX_LTRREADING
    CONTROL ADD LABEL, hDlg, %IDC_LABEL5, "00:00", 227, 90, 41, 12, %WS_CHILD _
        OR %WS_VISIBLE OR %SS_CENTER OR %SS_LEFT OR %SS_SUNKEN OR %SS_CENTERIMAGE, %WS_EX_LEFT OR _
        %WS_EX_LTRREADING
    CONTROL SET COLOR hDlg, %IDC_LABEL5, %WHITE, %BLUE

    hFont1 = PBFormsMakeFont("MS Sans Serif", 12, 700, %FALSE, %FALSE, %FALSE, %ANSI_CHARSET)
    hFont2 = PBFormsMakeFont("MS Sans Serif", 8, 700, %FALSE, %FALSE, %FALSE, %ANSI_CHARSET)

    CONTROL SEND hDlg, %IDC_LABEL1, %WM_SETFONT, hFont1, 0
    CONTROL SEND hDlg, %IDC_LABEL3, %WM_SETFONT, hFont1, 0
    CONTROL SEND hDlg, %IDC_BUTTON4, %WM_SETFONT, hFont2, 0
    CONTROL SEND hDlg, %IDC_BUTTON5, %WM_SETFONT, hFont2, 0
    CONTROL SEND hDlg, %IDC_BUTTON6, %WM_SETFONT, hFont2, 0
    CONTROL SEND hDlg, %IDC_BUTTON7, %WM_SETFONT, hFont2, 0
    CONTROL SEND hDlg, %IDC_BUTTON8, %WM_SETFONT, hFont2, 0
    CONTROL SEND hDlg, %IDC_BUTTON9, %WM_SETFONT, hFont2, 0

#PBFORMS End Dialog
 '   CONTROL SET TEXT hDlg, %IDC_TEXTBOX2, "F:\"
    DIALOG SHOW MODAL hDlg, CALL ShowDIALOG1Proc TO lRslt
    hMainDlg = hDlg
    DeleteObject hFont1

    FUNCTION = lRslt
END FUNCTION
'--------------------------------------------------------------------------------

SUB showTimedMsgBox(lText AS STRING)
 DIM llT AS LONG, iiTime AS LONG
 LOCAL hDlgAlarm AS DWORD
 LOCAL hMenuCopy AS DWORD
 llT = LEN(lText) * 3.6
 DIALOG NEW hMainDlg, "USB 2.0 copy", , , llT, 47, %WS_SYSMENU OR %WS_CAPTION TO hDlgAlarm
 DIALOG SET ICON hDlgAlarm, "#" + FORMAT$(%IDR_IMGFILE1)
 CONTROL ADD LABEL, hDlgAlarm, %MSGBOXLABEL, lText, 1, 13, llT, 30, %SS_CENTER
           'get system menu copy handle
          hMenuCopy = getSystemMenu(hDlgAlarm, %FALSE)
          'gray out (and disable) the close box
          enableMenuItem hMenuCopy, %SC_CLOSE, %MF_BYCOMMAND OR %MF_GRAYED
 DIALOG SHOW MODELESS hDlgAlarm 'call showBmsgbox
' bringWindowToTop hDlgAlarm
' SetWindowPos hDlgAlarm, hMainDlg, 0, 0, 0, 0, %SWP_NOMOVE OR %SWP_NOSIZE 'OR %SWP_SHOWWINDOW OR %SWP_NOACTIVATE
      DO
          DO: DIALOG DOEVENTS: LOOP UNTIL peekMessage(lpMsg, hDlgAlarm, 0, 0, %PM_NOREMOVE) = 0
          SLEEP 35
          INCR iiTime
      LOOP UNTIL iiTime = 25
      DIALOG END hDlgAlarm
END SUB

FUNCTION Exists(File AS ASCIIZ * %MAX_PATH) AS LONG
  LOCAL Dummy AS LONG
  ERRCLEAR                       'same as err = 0
  Dummy = GETATTR(File)          'error 53(file not found) will result if it doesn't exist
  FUNCTION = (ERRCLEAR = 0)      'if errclear = 0 then function = -1 else if there is error, function = %FALSE
END FUNCTION

FUNCTION setFileDatesEqual() AS LONG
  '    DIM qtime(2) AS QUAD
  '    QueryPerformanceCounter qtime(1)
    LOCAL hOrigFile AS DWORD, hCopyFile AS DWORD
    LOCAL CREATED AS FILETIME
    LOCAL accessed AS FILETIME
    LOCAL modified AS FILETIME
    hOrigFile = FILEATTR(#1, 2)
    hCopyFile = FILEATTR(#2, 2)
    getFileTime hOrigFile, CREATED, accessed, modified
    setFileTime hCopyFile, CREATED, accessed, modified
 '  QueryPerformanceCounter qtime(2)
 '  msgbox FORMAT$(qtime(2) - qtime(1), "0,")
END FUNCTION

FUNCTION directoryError(fileNam AS ASCIIZ * %MAX_PATH, hWindow AS DWORD) AS LONG
    DIALOG DISABLE hWindow
    MSGBOX "Oopsie, " & fileNam & " is a directory, not a file. Try again", _
                       %MB_ICONERROR + %MB_SYSTEMMODAL, "CompareUm"
    DIALOG ENABLE hWindow
    setForegroundWindow hWindow

END FUNCTION

**Ian Cairns** · 24 Sep 2007, 10:34 AM

DOS rears its ugly head yet again.

How about using the File Comparison utility, FC, included with windows since DOS days? That way, you only have to bother with a (slower) programming solution if you find a problem.

Syntax:
FC: Compares two files or sets of files and displays the differences between them

FC [/A] [/C] [/L] [/LBn] [/N] [/OFF[LINE]] [/T] [/U] [/W] [/nnnn]
[drive1:][path1]filename1 [drive2:][path2]filename2
FC /B [drive1:][path1]filename1 [drive2:][path2]filename2

/A Displays only first and last lines for each set of differences.
/B Performs a binary comparison.
/C Disregards the case of letters.
/L Compares files as ASCII text.
/LBn Sets the maximum consecutive mismatches to the specified
number of lines.
/N Displays the line numbers on an ASCII comparison.
/OFF[LINE] Do not skip files with offline attribute set.
/T Does not expand tabs to spaces.
/U Compare files as UNICODE text files.
/W Compresses white space (tabs and spaces) for comparison.
/nnnn Specifies the number of consecutive lines that must match
after a mismatch.
[drive1:][path1]filename1
Specifies the first file or set of files to compare.
[drive2:][path2]filename2
Specifies the second file or set of files to compare.

**Paul Purvis** · 24 Sep 2007, 12:12 PM

more on subject

i wish i had seen michael's code earlier, for some reason i missed it, an wanted to try his approach by reading data into an array.

on the code program i wrote,
there are a couple of notes.
i had taken this approach also:
read file 1 then read file 2 then compare data,
then read file 2 then read file 1 then compare data,
then read file 1 then read file 2 then compare data.
i was hoping to minimize head movement over the platters, and on a laptop, which i though might be a good test machine(slower drives and cpu cycles), but i was not testing data on a separate drive which might have proved faster.
it seemed the extra code was more not worth what it actually provided in speed.
and on speed, i was actually trying my best to lower cpu cycles(less code) and also provide more cpu time for other programs and hot hog that cpu.

i am not sure whether my approach is the fastest way to fill a variable or not,
i was actually looking to see only if both files where equal.

also my code should be programmed so that it compares the last xx bytes of the file first then starts back at the top comparing the bytes.
i believe that would give the best results.

also where would the number of blocks be best set at for all windows os
and memory sizes.
i find the string comparison in pb to be fast enough, and reading the file and placing the data in a variable to be areas of improvement in my program if it can be done. once again, i have not tried michaels version.
the code in my program to check for existence of a file, should be removed and code made that does not create a file if the file does not exist.

**Roger Garstang** · 24 Sep 2007, 12:26 PM

Not code, but I still use WinDiff for all my compares. It is fast enough for me, and M$ just updated it in the recent Service Packs for WinXP 64bit and Win2003. It is pretty good at finding patterns and where things are different and when chucks are just moved, etc.

**Michael Mattias** · 24 Sep 2007, 12:38 PM

> was hoping to minimize head movement over the platters..

This is a waste of your time. The amount of time you can save doing this is insignificant compared to what you can gain by better use of your existing resources. Between Windows' caching and changes to disk drive architecture there's just nothing there today which you can count on being there tomorrow.

Besides, what if file1 and file2 are on different disk drives?

> also where would the number of blocks be best set at for all windows os
>and memory sizes.

This IS worth thinking about. However, it is not so much going to be operating-system dependent as it is 'network' dependent if the files to be compared are on one or more network disks. Installed memory and number of other processes currently running and how much memory they take is going to make tuning this impossible.

>i have not tried michaels version

Then now is the time. It is FAST on my LOCAL disk drives......

.... ooh, just thought of something... the guy was in last Friday and set up my new laptop to network into my desktop and share all disks both ways. I should try that demo program across the network to see what effect that has!

**John McWilliams** · 24 Sep 2007, 03:50 PM

Originally posted by Michael Mattias View Post

>
.... ooh, just thought of something... the guy was in last Friday and set up my new laptop to network into my desktop and share all disks both ways. I should try that demo program across the network to see what effect that has!

That probably won't give you a very good "real world" evaluation. Your network has no load on it. Try hanging a couple of hundred workstations on the same network, all of them accessing a server based database and you'll get a much better "real world" idea of how it's going to perform.

You're "network test" will probably show about the same response time as a local disk.

**John Gleason** · 24 Sep 2007, 06:56 PM

MCM & MMX

My post above is not exactly "programmer friendly" and MCM posted code that is very programmer friendly, so I took the core asm from my "noodle surprise" and added it to Michael's code.

Code:

These were some of the speeds I got comparing the same 2 big files:
MCM           35.0 sec
MCM&MMX       21.8 sec
FC            69   sec
winDiff       126  sec

Below is the code with the added MMX function.

Code:

' large_file_compare.bas
' Purpose   : Quickly test two files for equality
' Author    :  Michael Mattias Racine WI
' Date      : September 22 2007
' Copyright : Placed in public domain by author 9/22/07
' Compiler  : PB/Windows 8.03
' ----------------------------------------------------------------------------------------------------
' Notes     :
' a. Added Sept.24, 07, MMX optimized function to do most of file compare. ~60% faster overall.
' b. ...   Sept.24, 07, No longer gives exact byte of difference, only gives 12MB chunk position. John Gleason
' 1. Could be optimized by better use of REGISTER variables.
' 2. The bigger the %BLOCKSIZE, the faster the compare; but remember you will be allocating
'    2*%BLOCKSIZE bytes of user memory.
' 3. Not tested with files > 2 Gb but "should" work.
' -----------------------------------------------------------------------------------------------------

#COMPILE  EXE
#DEBUG    ERROR ON
#REGISTER NONE
#DIM      ALL
#TOOLS    OFF
'=====[Windows API Header Files] ============================
'  If you don't need all of the functionality supported by these APIs
'  (and who does?), you can selectively turn off various modules by putting
'  the following constants in your program BEFORE you #include "win32api.inc":
'
'  %NOGDI = 1     ' no GDI (Graphics Device Interface) functions
'  %NOMMIDS = 1   ' no Multimedia ID definitions
%NOMMIDS = 1
#INCLUDE "WIN32API.INC"       ' 21 Feb 2005
'==[End Windows API Header Files]============================

#IF NOT %DEF (%INVALID_HANDLE_VALUE_LONG)
   %INVALID_HANDLE_VALUE_LONG    = -1&
#ENDIF


FUNCTION WINMAIN (BYVAL hInstance     AS LONG, _
                  BYVAL hPrevInstance AS LONG, _
                  BYVAL lpCmdLine     AS ASCIIZ PTR, _
                  BYVAL iCmdShow      AS LONG) AS LONG

  LOCAL szFile1 AS ASCIIZ * %MAX_PATH, szFile2 AS ASCIIZ * %MAX_PATH
  LOCAL iRet AS LONG, w AS STRING, t1 AS DOUBLE


  szFile1 =  "C:\MVI_0641.AVI"
  szFile2 =  "C:\MVI_0642.AVI"

  t1 = TIMER
  CALL FileCompare ( szFile1, szFile2) TO iret

  SELECT CASE iRet
      CASE 0&
          w = "Files are a match in " & FORMAT$(TIMER - t1, "0.00") & " secs"
      CASE -1&
          w = "One or more files not found or files not same size"
      CASE ELSE
          W = USING$ ("Files same size but mismatch in 12MB segment starting at #,", iRet - 1)
  END SELECT

  MSGBOX   W, %MB_APPLMODAL OR %MB_ICONINFORMATION, "File Comparison Demo"

END FUNCTION  ' WinMain

' --------------------------------------------------------
' FILE COMPARISON
' RETURNS: 0    ==> Files are identical
'          >0   ==> Files are same size, first non-matching character at this offset
'          -1&  ==> Files are not the same size or one or more not found
'                   or open error so no character comparison even attempted
' ----------------------------------------------------------

%BLOCKSIZE    =   12000000&  ' only 12 million bytes because I don't
                             ' I don't have files big enough to compare using larger numbers
FUNCTION FileCompare (sz1 AS ASCIIZ, sz2 AS ASCIIZ) AS QUAD

    LOCAL h1 AS LONG,h2 AS LONG
    LOCAL pb1 AS BYTE PTR, pb2 AS BYTE PTR
    LOCAL b1() AS BYTE, b2() AS BYTE
    LOCAL nBlock AS LONG, nOddByte AS LONG
    LOCAL fSize AS QUAD, nOb AS LONG, ii AS LONG

    LOCAL iBlock  AS LONG
    LOCAL iCmp    AS LONG
    LOCAL MisMatchPos AS QUAD

    ' --------------------------------------------------------
    ' are files the same size?
    ' (kind of required for them to be equivalent, you know?)
    ' --------------------------------------------------------

    CALL AreFileSizesEqual (sz1, sz2) TO icmp


    SELECT CASE AS LONG iCmp
        CASE -1&, 0&
            FUNCTION = -1&   ' return 'no byte comparisons' value
            EXIT FUNCTION
    END SELECT

    ' --------------------------------------------------------------
    ' Ok, the files are the same size, so we'll have to open 'em up
    ' and use brute force to compare the content.
    ' --------------------------------------------------------------

    h1 = FREEFILE
    OPEN sz1 FOR BINARY ACCESS READ AS h1 BASE = 0

    h2 = FREEFILE
    OPEN sz2 FOR BINARY ACCESS READ AS h2 BASE = 0

    ' --------------------------------------------
    ' should really test for error on the OPENs here.
    ' Exercise left to user.
    ' --------------------------------------------

    ' allocate buffers for the file data
    REDIM    B1 (%BLOCKSIZE-1)
    REDIM    B2 (%BLOCKSIZE-1)
    pb1    = VARPTR(b1(0))          ' point at first character of the file data blocks
    pb2    = VARPTR(b2(0))


    fsize       = LOF (h1)                ' size of files (already tested for same size)
    nBlock      = fsize \ %BLOCKSIZE
    nOddByte    = fSize MOD %BLOCKSIZE
    mismatchpos = 0

    FOR iBlock= 1 TO nBlock             ' never executes if fSize < %BLOCKSIZE
        GET h1,,b1()                    ' get bytes from files into arrays
        GET h2,,b2()
        CALL   CompareMemoryMMX (pb1, pb2, %BLOCKSIZE) TO icmp  ' returns 1=based mismatch postion
        IF iCmp THEN
            ' we have a mismatch
            MisMatchPos = (iBlock-1) * %BLOCKSIZE  + iCmp
            EXIT FOR    ' no sense checking any more , is there?
        END IF

    NEXT iBlock

    ' ----------------------------------------------------------------------------
    ' If files still match after all full blocks (if there were any) have been
    ' tested, compare the oddbytes at end of file
    ' ----------------------------------------------------------------------------

    IF ISFALSE MisMatchPos THEN

       IF nOddByte THEN

          REDIM b1 (nOddbyte-1)  ' resize arrays to receive rest of file only
          REDIM b2 (nOddbyte-1)
          GET   h1, ,b1()
          GET   h2, ,b2()
          pb1    = VARPTR (b1(0))
          pb2    = VARPTR (b2(0))
          CALL  CompareMemory (pb1, pb2, NOddbyte) TO iCmp
          IF iCmp THEN
              MisMatchPOs = (nBlock * %BLOCKSIZE) + iCmp
          END IF
       END IF

    END IF

    CLOSE      h1, h2
    FUNCTION = MisMatchPos

END FUNCTION

' compare'cb' bytes of memory at addresses pb1 and pb2
' returns:    0  blocks are identical
'            >0  blocks mismatch at this byte position (1-based offset)
FUNCTION CompareMemory (BYVAL A1 AS BYTE PTR, BYVAL A2 AS BYTE PTR, BYVAL cb AS LONG ) AS LONG

    REGISTER I AS LONG, bMismatch AS LONG

    LOCAL pb1 AS BYTE PTR, pb2 AS BYTE PTR
    bMismatch = %FALSE

    pb1 = A1
    pb2 = A2
    FOR I = 0 TO cb-1
        IF @pb1 XOR @pb2  THEN   ' there is a mismatch
          bMismatch = %TRUE
          EXIT FOR
        ELSE                     ' the bytes match so we keep going
           INCR pb1
           INCR pb2
        END IF
    NEXT
    IF bMismatch THEN
        FUNCTION = I + 1        ' return 1-based mismatch position
    ELSE
      ' if we get this far all bytes matched
       FUNCTION = 0
    END IF

END FUNCTION

FUNCTION CompareMemoryMMX (BYVAL A1 AS BYTE PTR, BYVAL A2 AS BYTE PTR, BYVAL cb AS LONG ) AS LONG
       LOCAL q1zz AS QUAD
       'takes 40 bytes per file each loop--80 bytes total--and compares them in ~23 ticks per loop. That's ~4 bytes per tick!
                            q1zz = &hffffffffffffffff
                            !mov ecx, cb              ;compare chunk
                            !mov eax, A1
                            !shr ecx, 2               ;divide by 4
                            !mov edx, A2
                            !sub ecx, 1               ;to line up properly
                            !movq mm6, q1zz           ;pronouced "Q ones" (all bits set to 1)
                          repCompare:
                            !movq mm0, [eax+ecx*4-4]
                            !movq mm1, [edx+ecx*4-4]
                            !movq mm2, [eax+ecx*4-12]
                            !pcmpeqd mm0,  mm1
                            !movq mm3, [edx+ecx*4-12]
                            !pand    mm6,  mm0
                            !movq mm4, [eax+ecx*4-20]
                            !pcmpeqd mm2,  mm3
                            !movq mm5, [edx+ecx*4-20]
                            !pand    mm6,  mm2
                            !pcmpeqd mm4,  mm5
                            !movq mm0, [eax+ecx*4-28]    ;start of next series of four, put here avoids dependency
                            !pand    mm6,  mm4

                            !movq mm1, [edx+ecx*4-28]
                            !movq mm2, [eax+ecx*4-36]
                            !pcmpeqd mm0,  mm1
                            !movq mm3, [edx+ecx*4-36]
                            !pand    mm6,  mm0
                            !pcmpeqd mm2,  mm3
                            !sub ecx, 10                 ;avoid dependency
                            !pand    mm6,  mm2
                            '------------------------
                            !jns short repCompare

                            !movd eax,  mm6
                            !psrlq mm6, 32
                            !movd ecx,  mm6
                            !emms
                            !test eax,   ecx
                            !jz short notTheSame
                            !mov FUNCTION, 0             ;matched
                            !jmp short finish
                           notTheSame:
                            !mov FUNCTION, 1             ;no match
                           finish:

END FUNCTION

' returns +1& files are same size, 0 they ain't -1& one or more files not found
' requires Win anything
FUNCTION AreFileSizesEqual (sz1 AS ASCIIZ, sz2 AS ASCIIZ) AS LONG

    LOCAL hSearch AS LONG

    LOCAL w32A AS WIN32_FIND_DATA, W32B AS WIN32_FIND_DATA

    hSearch = FindFirstFile (sz1, W32A)
    IF hSearch = %INVALID_HANDLE_VALUE_LONG THEN
       FUNCTION = -1&
    ELSE
        FindClose    hSearch   ' close the search handle we go what we need
        hSearch  =   FindFirstFile(sz2, W32B)
        IF hSearch = %INvALID_HANDLE_VALUE_LONG THEN
            FUNCTION = -1&
        ELSE
            FindClose hSearch
            IF (W32A.nFileSizeHigh = W32B.nFilesizeHigh) AND (W32A.nFilesizeLow = W32B.nFileSizeLow) THEN
                FUNCTION = 1&   ' return "both found and same size" value
            ELSE
                FUNCTION = 0&   ' return "both found but not same size" value
            END IF
         END IF'
     END IF

END FUNCTION
'/// END OF FILE

**Michael Mattias** · 25 Sep 2007, 08:42 AM

These were some of the speeds I got comparing the same 2 big files:
MCM 35.0 sec
MCM&MMX 21.8 sec
FC 69 sec
winDiff 126 sec

This is not really apples and apples; or even apples and oranges; there are at least three fruits in play.

First, I don't know WinDiff, so throw that out. Let's just say it's a Windows port of FC.

Second, FC.EXE by default writes the mismatched bytes to STDOUT AND compares the entire file, unless command line says to stop after 1 mismatch (command-line used not shown).

MMX assembly language version of function does not report offset of mismatch whereas pure BASIC code does.

And I'm not an assembly-language guy, but what happens to the assembly-language code if 'cb' comes in less than 40 ( It could come in anywhere from 1 to (%BLOCKSIZE-1))? Looks to me like there would be some address overrun or underrun somewhere.

Not to mention..... the BASIC code is not optimized to do QUAD compares on sub-blocks of size 8, then again doing it a byte-at-a-time enabled simplified reporting of the location of the first mismatch. The designer (moi)made a conscious 'trade off' of speed versus features - something all designers do every day.

Are there optimization opportunities within the BASIC code? Sure! Lots of 'em! And I encourage them!

But 'performance comparisons' when the software being compared is not doing the same thing are invalid.

MCM

**David Roberts** · 25 Sep 2007, 09:53 AM

John

Have a look at

Code:

IF ISFALSE MisMatchPos THEN
 
  IF nOddByte THEN
 
    REDIM b1 (nOddbyte-1)  ' resize arrays to receive rest of file only
    REDIM b2 (nOddbyte-1)
    GET   h1, ,b1()
    GET   h2, ,b2()
    IF (nOddbyte MOD 4) > 0 THEN
      nOddbyte = nOddbyte + 4 -(nOddbyte MOD 4) ' DWORD padding
      REDIM b1 (nOddbyte-1)
      REDIM b2 (nOddbyte-1)
    END IF
    pb1    = VARPTR (b1(0))
    pb2    = VARPTR (b2(0))
    CALL  CompareMemory (pb1, pb2, NOddbyte) TO iCmp
    IF iCmp THEN
      MisMatchPOs = (nBlock * %BLOCKSIZE) + iCmp
    END IF
  END IF
 
END IF

Code:

FUNCTION CompareMemory (BYVAL A1 AS BYTE PTR, BYVAL A2 AS BYTE PTR, BYVAL cb AS LONG ) AS LONG
 
    REGISTER I AS LONG, bMismatch AS LONG
 
    LOCAL pb1 AS BYTE PTR, pb2 AS BYTE PTR
    bMismatch = %FALSE
 
'    pb1 = A1
'    pb2 = A2
'    FOR I = 0 TO cb-1
'        IF @pb1 XOR @pb2  THEN   ' there is a mismatch
'          bMismatch = %TRUE
'          EXIT FOR
'       ELSE                     ' the bytes match so we keep going
'           INCR pb1
'           INCR pb2
'        END IF
'    NEXT
'    IF bMismatch THEN
'        FUNCTION = I + 1        ' return 1-based mismatch position
'    ELSE
'     ' if we get this far all bytes matched
'       FUNCTION = 0
'    END IF
 
    LOCAL counter AS LONG
    counter= cb\4
 
    ! mov esi, A1
    ! mov edi, A2
    ! mov ecx, counter
    ! cld
    ! repe cmpsd
    ! jz match
    ! mov counter, ecx
    FUNCTION = cb\4 - counter
    match:
 
END FUNCTION

I am now getting the mismatch at the DWORD position rather than the BYTE position but I'm not concerned about the position, just the speed, so

Code:

CASE ELSE
          W = USING$ ("Files same size but mismatch at dword offset #,", iRet)

I agree with MCMs comments - I was just interested to see if the olde worlde (

) 'repe cmpsd' was coming in quicker then MMX as I seem to be getting MCM/2.

**Michael Mattias** · 25 Sep 2007, 10:06 AM

I wasn't trying to be single out any person or any specific code here.

But I have seen so many "comparisons" here where apples were being compared to submarines I just could not resist.

Then again, I "may" have mentioned once or twice that optimization is always "application-specific" and all code posted here for those purposes should be considered an example only and optimizers (?) should be prepared to modify the code.

e.g in this case, a good application-specific optimization would be to start the comparison at the END of the file instead of the beginning. (See post # 7 this thread).

MCM

**Pierre Bellisle** · 25 Sep 2007, 12:01 PM

John,

The MMX code you posted is real fast, I love it,
it's a clear winner. If you have any more of this kind of stuff
and want to post it, then, be assured that I will appreciate.

Also, if you ever feel like adding more comments to the ASM code
to explain the logic, then, again, it will be much appreciated.

Thank

**David Roberts** · 25 Sep 2007, 03:01 PM

Actually, there isn't much in it.

I copied a 700MB iso image from my USB 2.0 external to my SATA internal and restarted on each run and got

MMX 53.91 sec
Olde Worlde 56.52

and from the filecache

MMX 2.91 sec
Olde Worlde 2.92

To eliminate the 'apples and submarines' I added a tad to the above to return the offending byte.

Code:

  LOCAL counter, bcounter AS LONG
  counter= cb\4
    
  ! mov esi, A1
  ! mov edi, A2
  ! mov ecx, counter
  ! cld
  ! repe cmpsd
  ! jz match
  ! mov counter, ecx
  A1 = A1 + cb - 4*(counter+1) ' target the offending DWORD
  A2 = A2 + cb - 4*(counter+1)
  bcounter = 4
  ! mov esi, A1
  ! mov edi, A2
  ! mov ecx, bcounter
  ! cld
  ! repe cmpsb
  ! mov bcounter, ecx
  ' FUNCTION = cb - 4*(counter+1) + (4 - bcounter)
  ' ie
  FUNCTION = cb - 4*counter - bcounter
  match:

**Paul Dixon** · 25 Sep 2007, 05:05 PM

David,
The speed limit for the compare of large files is not the software that does the compare, it's the speed at which you can read the data from the medium.
Two 700MB files on a fast hard disk will take about 14 seconds to read.

If you have the RAM for them and the files are already stored in RAM then it'll be quicker.
My 3 year old reasonably-fast-in-its-day-but-now-a-bit-slow PC has a memory bandwidth of about 2.5GB/s so under ideal circumstances it could read the 2 files in about 0.6s.

If we accept the figure in John's code of 4 bytes/tick then the time taken to do the comparison, excluding fetching the data, will be nearer 0.2s on my 1.9GHz Athlon.

So the 2.9s shown in the last post is still quite slow. The problem is that the data is too big to fit in the CPU cache so it must be fetched from slow main RAM.
The CPU asks the memory for a byte then hangs around for hundreds of CLKs waiting for that byte to arrive from the slow memory.
The next few bytes are then read very quickly because the system caches a big block of bytes around the one requested so the next few bytes are in fast cache. When the last byte in the cache is read the next read is again very slow while the next block of data is retrieved from main memory.

You can improve things by prefetching data. Check out the PREFETCH instruction in the Intel Pentium manual. They were designed to do this. Of course, it'll only work if you explicitly code the loop yourself, not with REP instructions.

Paul.

**John Gleason** · 25 Sep 2007, 06:26 PM

Thanks Pierre, yes, it definitely needs that documentation,

so here it is for just that function.

Code:

FUNCTION CompareMemoryMMX (BYVAL A1 AS BYTE PTR, BYVAL A2 AS BYTE PTR, BYVAL cb AS LONG ) AS LONG
       LOCAL q1zz AS QUAD
       'takes 40 bytes per file each loop--80 bytes total--and compares them in ~23 ticks per loop. That's ~4 bytes per tick!

       'The general idea is to compare 8-byte quads from each of the two files, and if they don't match, the first
       'quad's bits (eg.!pcmpeqd mm0 below) will be changed to all 0's, ELSE they'll be changed to all 1's. Next we AND that
       'result with a register set to all 1 bits (mm6 below) and if there was a match in the previous compare, mm6 remains
       'all 1's, ELSE it turns to all 0 bits, and will remain all zero bits thru to the end of the function. So, if register
       'mm6 = 0 at the end if the loop, somewhere in the loop at least one bit didn't match. MMX speed derives from both the
       'quad size and the fact that each command supposedly takes only 1 tick. With new 64-bit processors, MMX days may be numbered.
                            q1zz = &hffffffffffffffff
                            !mov ecx, cb              ;total memory compare size. Must be a multiple of 40, (8 bytes * 5 per loop).
                            !mov eax, A1              ;file1 memory ptr
                            !shr ecx, 2               ;divide by 4 because below we multiply this offset by 4, (ecx*4). Why? I'd already written loop below and this was easy fix.
                            !mov edx, A2              ;file2 memory ptr
                            !sub ecx, 1               ;to line up properly: eg. top 8 bytes will start at eax+ecx*4-4. Here we just made ecx=cb/4-1
                            !movq mm6, q1zz           ;pronouced "Q ones" (all bits set to 1)
                          repCompare:                 'the interlaced order of commands that follow is to avoid dependencies for speed. But, it does look more confusing.
                            !movq mm0, [eax+ecx*4-4]  ;load mmx register file1
                            !movq mm1, [edx+ecx*4-4]  ;      "           file2
                            !movq mm2, [eax+ecx*4-12] ;      "           file1
                            !pcmpeqd mm0,  mm1        ;compare 8 bytes from each of the 2 files
                            !movq mm3, [edx+ecx*4-12] ;                  file2
                            !pand    mm6,  mm0        ;AND with mmx register 6--our tracking register, once it's zero it stays zero--to see if there is a mismatch
                            !movq mm4, [eax+ecx*4-20]
                            !pcmpeqd mm2,  mm3        ;compare 8 bytes from each of the 2 files
                            !movq mm5, [edx+ecx*4-20]
                            !pand    mm6,  mm2        ;AND with mmx register 6--our tracking register
                            !pcmpeqd mm4,  mm5        ;compare 8 bytes from each of the 2 files
                            !movq mm0, [eax+ecx*4-28] ;start of next series of four, put here avoids dependency
                            !pand    mm6,  mm4        ;AND with mmx register 6--our tracking register

                            !movq mm1, [edx+ecx*4-28]
                            !movq mm2, [eax+ecx*4-36]
                            !pcmpeqd mm0,  mm1        ;compare 8 bytes from each of the 2 files
                            !movq mm3, [edx+ecx*4-36]
                            !pand    mm6,  mm0        ;AND with mmx register 6--our tracking register
                            !pcmpeqd mm2,  mm3        ;compare 8 bytes from each of the 2 files
                            !sub ecx, 10              ;avoid dependency, don't put just before !jns
                            !pand    mm6,  mm2        ;AND with mmx register 6--our tracking register
                            '------------------------
                            !jns short repCompare     ;do 40 more

                            !movd eax,  mm6           ;move low 32 bits of mm6 to eax
                            !psrlq mm6, 32            ;shift right 32
                            !movd ecx,  mm6           ;move hi 32 bits of mm6 to ecx
                            !emms                     ;clear MMX state
                            !test eax,   ecx          ;eax and ecx are either both all 1's or both all 0's
                            !jz short notTheSame      ;if all 0's, mismatch
                            !mov FUNCTION, 0          ;all bits 1, so matched
                            !jmp short finish
                           notTheSame:
                            !mov FUNCTION, 1          ;no match
                           finish:

END FUNCTION

**David Roberts** · 25 Sep 2007, 07:54 PM

Thanks for that Paul. I am aware of much of what you wrote and why I quoted the filecache figures. I've often noted here that some apps may be so disk intensive that it wouldn't matter much whether they were written in PPCC or cscript.

**John Gleason** · 25 Sep 2007, 08:00 PM

Dave, I got almost the same result:
Ye Olde Worlde: 23.4
prev. Poste I did: 21.8
But! You used just half a dozen lines of code whereas I more properly filled dang near a whole screen.

Paul, tho it probably won't matter much for this app with all the disk action, thx for the PREFETCH tip. Also, my arithmetic on the (never actually tested it) 4 bytes/tick? Yes, optimism incarnate. It's Price Rollback time to 3 bytes/tick.

Michael, agreed on most points. I think tho that, like Dave's code showed he could do, one could quickly derive the exact difference point once narrowed to a 10 or 12 MB chunk of a big file.
The timings I posted were meant just as a general guideline of what one might expect using, in binary mode, the programs previously mentioned. A starting point, if you will, for Paul Johnson (whom we haven't heard from in a while btw, where'd you go Paul?).
Lastly, I added to the fully documented version of the MMX function the point you made that cb needs to be a multiple of 40--an important safety tip.

Announcement

Need very fast way of comparing binary files

Need very fast way of comparing binary files

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment