MD5 Hash Questions

[John Strasser]

Hi Folks,

I've come across a situation where I find an MD5 hash would be extremely usefull - seeing if 2 (or more) files are the same.

I figured it should be simple to find source code for this (and there is plenty here in the forums) BUT almost all of it uses assembly for speed.

I can apreciate this from an encrypton POV, BUT it makes it much harder for me to muddle through.

Speed isn't the issue for me(pe se), so I'm obviously going to translate a c version myself whilst referencing the awesome work that our resident gurus have already done.

While I've got my *current* problem solved, my questions for the gurus are:

(Based upon the Florent Heyworth version from pbcrypto)

Why break everything up into chunks? If I can load an entire file into a memory buffer, why can't I just feed that buffer? Is there something inherent in the algorithm limiting the input string size? Or is that related more to the particular implementation (CPU register sizes,etc)?

In that particular implementation, is there a reason why it seems as if you have to call 15 different fuctions just to get the final value?

Or is that again limited to the fact that the conversion was done in 2000?

Obviously I'm going to be translating without really understanding...

<sigh> in php you just go $somevar = md5(file($filename));

Btw - HIME looks great, it's just way overpowered for what I want. And since I have no need of anything else, I can't justify the $150 to get rid of the nag screen...

Thanks in advance,

JS

[Scott Turchin]

Haven't learned how to use this collection of code - but it's been floating around for a few years.

Dunno if it helps any..

'MD5 stuff
Declare Function  md5_init( ByRef ctx As MD5_CTX )As Long
Declare Function  md5_Transform ( ByRef ctx As MD5_CTX, ByVal bBuf As Long Ptr )As Long
Declare Function  md5_Update(ByRef ctx As MD5_CTX, ByVal bBuf As Byte Ptr, ByVal dwLen As Dword )As Long
Declare Function  md5_Final( ByVal bDigest As Byte Ptr, ctx As MD5_CTX)As Long
Declare Function md5_Digest( ByRef ctx As MD5_CTX) As String
Declare Function md5_readFile( ByRef ctx As MD5_CTX,  szFile As Asciiz )As Long 



Here's a couple of others too

'*************************************************************************
'MD5
'* Begin Functions for MD5
'*************************************************************************
Function md5_shiftRight(ByVal lThis As Long, ByVal lBits As Long)Export As Long
Shift Right lThis, lBits
Function = lThis
End Function
'----------------------------------------------------------------------------------------
Function md5_shiftLeft(ByVal lThis As Long, ByVal lBits As Long) Export As Long
Shift Left lThis, lBits
Function = lThis
End Function
'----------------------------------------------------------------------------------------
Function md5_init( ByRef ctx As MD5_CTX )Export As Long
   'MD5 initialization. Begins an MD5 operation, writing a new context

   ctx.lCount(0) = 0
   ctx.lCount(1) = 0

   ' Load magic initialization constants.
   ctx.lState(0) = &H67452301
   ctx.lState(1) = &HEFCDAB89
   ctx.lState(2) = &H98BADCFE
   ctx.lState(3) = &H10325476

End Function
'----------------------------------------------------------------------------------------
Function md5_Transform ( ByRef ctx As MD5_CTX, ByVal bBuf As Long Ptr )Export As Long
   Local a As Long
   Local b As Long
   Local c As Long
   Local d As Long

   a = ctx.lState(0)
   b = ctx.lState(1)
   c = ctx.lState(2)
   d = ctx.lState(3)

  'ROUND 1
  a = a + ((((b)) And ((c))) Or ((Not (b)) And ((d)))) + (@bBuf[ 0]) +&Hd76aa478
  Rotate Left a, %S11: a = a + b
  d = d + ((((a)) And ((b))) Or ((Not (a)) And ((c)))) + (@bBuf[ 1]) +&He8c7b756
  Rotate Left d, %S12: d = d + a
  c = c + ((((d)) And ((a))) Or ((Not (d)) And ((b)))) + (@bBuf[ 2]) +&H242070db
  Rotate Left c, %S13: c = c + d
  b = b + ((((c)) And ((d))) Or ((Not (c)) And ((a)))) + (@bBuf[ 3]) +&Hc1bdceee
  Rotate Left b, %S14: b = b + c
  '
  a = a + ((((b)) And ((c))) Or ((Not (b)) And ((d)))) + (@bBuf[ 4]) +&Hf57c0faf
  Rotate Left a, %S11: a = a + b
  d = d + ((((a)) And ((b))) Or ((Not (a)) And ((c)))) + (@bBuf[ 5]) +&H4787c62a
  Rotate Left d, %S12: d = d + a
  c = c + ((((d)) And ((a))) Or ((Not (d)) And ((b)))) + (@bBuf[ 6]) +&Ha8304613
  Rotate Left c, %S13: c = c + d
  b = b + ((((c)) And ((d))) Or ((Not (c)) And ((a)))) + (@bBuf[ 7]) +&Hfd469501
  Rotate Left b, %S14: b = b + c
  '
  a = a + ((((b)) And ((c))) Or ((Not (b)) And ((d)))) + (@bBuf[ 8]) +&H698098d8
  Rotate Left a, %S11: a = a + b
  d = d + ((((a)) And ((b))) Or ((Not (a)) And ((c)))) + (@bBuf[ 9]) +&H8b44f7af
  Rotate Left d, %S12: d = d + a
  c = c + ((((d)) And ((a))) Or ((Not (d)) And ((b)))) + (@bBuf[10]) +&Hffff5bb1
  Rotate Left c, %S13: c = c + d
  b = b + ((((c)) And ((d))) Or ((Not (c)) And ((a)))) + (@bBuf[11]) +&H895cd7be
  Rotate Left b, %S14: b = b + c
  '
  a = a + ((((b)) And ((c))) Or ((Not (b)) And ((d)))) + (@bBuf[12]) +&H6b901122
  Rotate Left a, %S11: a = a + b
  d = d + ((((a)) And ((b))) Or ((Not (a)) And ((c)))) + (@bBuf[13]) +&Hfd987193
  Rotate Left d, %S12: d = d + a
  c = c + ((((d)) And ((a))) Or ((Not (d)) And ((b)))) + (@bBuf[14]) +&Ha679438e
  Rotate Left c, %S13: c = c + d
  b = b + ((((c)) And ((d))) Or ((Not (c)) And ((a)))) + (@bBuf[15]) +&H49b40821
  Rotate Left b, %S14: b = b + c


  'ROUND 2
  a = a + ((((b)) And ((d))) Or (((c)) And (Not (d)))) + (@bBuf[ 1]) +&Hf61e2562
  Rotate Left a, %S21: a = a + b
  d = d + ((((a)) And ((c))) Or (((b)) And (Not (c)))) + (@bBuf[ 6]) +&Hc040b340
  Rotate Left d, %S22: d = d + a
  c = c + ((((d)) And ((b))) Or (((a)) And (Not (b)))) + (@bBuf[11]) +&H265e5a51
  Rotate Left c, %S23: c = c + d
  b = b + ((((c)) And ((a))) Or (((d)) And (Not (a)))) + (@bBuf[ 0]) +&He9b6c7aa
  Rotate Left b, %S24: b = b + c
  '
  a = a + ((((b)) And ((d))) Or (((c)) And (Not (d)))) + (@bBuf[ 5]) +&Hd62f105d
  Rotate Left a, %S21: a = a + b
  d = d + ((((a)) And ((c))) Or (((b)) And (Not (c)))) + (@bBuf[10]) +&H2441453
  Rotate Left d, %S22: d = d + a
  c = c + ((((d)) And ((b))) Or (((a)) And (Not (b)))) + (@bBuf[15]) +&Hd8a1e681
  Rotate Left c, %S23: c = c + d
  b = b + ((((c)) And ((a))) Or (((d)) And (Not (a)))) + (@bBuf[ 4]) +&He7d3fbc8
  Rotate Left b, %S24: b = b + c
  '
  a = a + ((((b)) And ((d))) Or (((c)) And (Not (d)))) + (@bBuf[ 9]) +&H21e1cde6
  Rotate Left a, %S21: a = a + b
  d = d + ((((a)) And ((c))) Or (((b)) And (Not (c)))) + (@bBuf[14]) +&Hc33707d6
  Rotate Left d, %S22: d = d + a
  c = c + ((((d)) And ((b))) Or (((a)) And (Not (b)))) + (@bBuf[ 3]) +&Hf4d50d87
  Rotate Left c, %S23: c = c + d
  b = b + ((((c)) And ((a))) Or (((d)) And (Not (a)))) + (@bBuf[ 8]) +&H455a14ed
  Rotate Left b, %S24: b = b + c
  '
  a = a + ((((b)) And ((d))) Or (((c)) And (Not (d)))) + (@bBuf[13]) +&Ha9e3e905
  Rotate Left a, %S21: a = a + b
  d = d + ((((a)) And ((c))) Or (((b)) And (Not (c)))) + (@bBuf[ 2]) +&Hfcefa3f8
  Rotate Left d, %S22: d = d + a
  c = c + ((((d)) And ((b))) Or (((a)) And (Not (b)))) + (@bBuf[ 7]) +&H676f02d9
  Rotate Left c, %S23: c = c + d
  b = b + ((((c)) And ((a))) Or (((d)) And (Not (a)))) + (@bBuf[12]) +&H8d2a4c8a
  Rotate Left b, %S24: b = b + c


  'ROUND 3
  a = a + (((b)) Xor ((c)) Xor ((d))) + (@bBuf[ 5]) +&Hfffa3942
  Rotate Left a, %S31: a = a + b
  d = d + (((a)) Xor ((b)) Xor ((c))) + (@bBuf[ 8]) +&H8771f681
  Rotate Left d, %S32: d = d + a
  c = c + (((d)) Xor ((a)) Xor ((b))) + (@bBuf[11]) +&H6d9d6122
  Rotate Left c, %S33: c = c + d
  b = b + (((c)) Xor ((d)) Xor ((a))) + (@bBuf[14]) +&Hfde5380c
  Rotate Left b, %S34: b = b + c
  '
  a = a + (((b)) Xor ((c)) Xor ((d))) + (@bBuf[ 1]) +&Ha4beea44
  Rotate Left a, %S31: a = a + b
  d = d + (((a)) Xor ((b)) Xor ((c))) + (@bBuf[ 4]) +&H4bdecfa9
  Rotate Left d, %S32: d = d + a
  c = c + (((d)) Xor ((a)) Xor ((b))) + (@bBuf[ 7]) +&Hf6bb4b60
  Rotate Left c, %S33: c = c + d
  b = b + (((c)) Xor ((d)) Xor ((a))) + (@bBuf[10]) +&Hbebfbc70
  Rotate Left b, %S34: b = b + c
  '
  a = a + (((b)) Xor ((c)) Xor ((d))) + (@bBuf[13]) +&H289b7ec6
  Rotate Left a, %S31: a = a + b
  d = d + (((a)) Xor ((b)) Xor ((c))) + (@bBuf[ 0]) +&Heaa127fa
  Rotate Left d, %S32: d = d + a
  c = c + (((d)) Xor ((a)) Xor ((b))) + (@bBuf[ 3]) +&Hd4ef3085
  Rotate Left c, %S33: c = c + d
  b = b + (((c)) Xor ((d)) Xor ((a))) + (@bBuf[ 6]) +&H4881d05
  Rotate Left b, %S34: b = b + c
  '
  a = a + (((b)) Xor ((c)) Xor ((d))) + (@bBuf[ 9]) +&Hd9d4d039
  Rotate Left a, %S31: a = a + b
  d = d + (((a)) Xor ((b)) Xor ((c))) + (@bBuf[12]) +&He6db99e5
  Rotate Left d, %S32: d = d + a
  c = c + (((d)) Xor ((a)) Xor ((b))) + (@bBuf[15]) +&H1fa27cf8
  Rotate Left c, %S33: c = c + d
  b = b + (((c)) Xor ((d)) Xor ((a))) + (@bBuf[ 2]) +&Hc4ac5665
  Rotate Left b, %S34: b = b + c


  'ROUND 4
  a = a + (((c)) Xor (((b)) Or (Not (d)))) + (@bBuf[ 0]) +&Hf4292244
  Rotate Left a, %S41: a = a + b
  d = d + (((b)) Xor (((a)) Or (Not (c)))) + (@bBuf[ 7]) +&H432aff97
  Rotate Left d, %S42: d = d + a
  c = c + (((a)) Xor (((d)) Or (Not (b)))) + (@bBuf[14]) +&Hab9423a7
  Rotate Left c, %S43: c = c + d
  b = b + (((d)) Xor (((c)) Or (Not (a)))) + (@bBuf[ 5]) +&Hfc93a039
  Rotate Left b, %S44: b = b + c
  '
  a = a + (((c)) Xor (((b)) Or (Not (d)))) + (@bBuf[12]) +&H655b59c3
  Rotate Left a, %S41: a = a + b
  d = d + (((b)) Xor (((a)) Or (Not (c)))) + (@bBuf[ 3]) +&H8f0ccc92
  Rotate Left d, %S42: d = d + a
  c = c + (((a)) Xor (((d)) Or (Not (b)))) + (@bBuf[10]) +&Hffeff47d
  Rotate Left c, %S43: c = c + d
  b = b + (((d)) Xor (((c)) Or (Not (a)))) + (@bBuf[ 1]) +&H85845dd1
  Rotate Left b, %S44: b = b + c
  '
  a = a + (((c)) Xor (((b)) Or (Not (d)))) + (@bBuf[ 8]) +&H6fa87e4f
  Rotate Left a, %S41: a = a + b
  d = d + (((b)) Xor (((a)) Or (Not (c)))) + (@bBuf[15]) +&Hfe2ce6e0
  Rotate Left d, %S42: d = d + a
  c = c + (((a)) Xor (((d)) Or (Not (b)))) + (@bBuf[ 6]) +&Ha3014314
  Rotate Left c, %S43: c = c + d
  b = b + (((d)) Xor (((c)) Or (Not (a)))) + (@bBuf[13]) +&H4e0811a1
  Rotate Left b, %S44: b = b + c
  '
  a = a + (((c)) Xor (((b)) Or (Not (d)))) + (@bBuf[ 4]) +&Hf7537e82
  Rotate Left a, %S41: a = a + b
  d = d + (((b)) Xor (((a)) Or (Not (c)))) + (@bBuf[11]) +&Hbd3af235
  Rotate Left d, %S42: d = d + a
  c = c + (((a)) Xor (((d)) Or (Not (b)))) + (@bBuf[ 2]) +&H2ad7d2bb
  Rotate Left c, %S43: c = c + d
  b = b + (((d)) Xor (((c)) Or (Not (a)))) + (@bBuf[ 9]) +&Heb86d391
  Rotate Left b, %S44: b = b + c

   ctx.lState(0) = ctx.lState(0) + a
   ctx.lState(1) = ctx.lState(1) + b
   ctx.lState(2) = ctx.lState(2) + c
   ctx.lState(3) = ctx.lState(3) + d

End Function
'----------------------------------------------------------------------------------------

' Update context TO reflect the concatenation of another buffer full
' of bytes.

Function md5_Update(ByRef ctx As MD5_CTX, ByVal bBuf As Byte Ptr, ByVal dwLen As Dword )Export As Long
    Local t As Dword

    ' Update bitcount
    t = ctx.lCount(0)
    ctx.lCount(0) = t + md5_shiftLeft( dwLen, 3 )
    If ctx.lCount(0) < t Then
       Incr ctx.lCount(1) 'carry from low to high
    End If

    ctx.lCount(1) = ctx.lCount(1) + md5_shiftRight( dwLen, 29 )

    t = ( md5_shiftRight( t, 3) And &H3F ) 'Bytes already IN shsInfo->DATA

    'handle any leading odd-sized chunks
    If IsTrue(t) Then
       Local p As Byte Ptr

       p = VarPtr(ctx.bBuf)
       p = p + t
       t = 64 - t
       If dwLen < t Then
          Call MoveMemory( ByVal p, ByVal bBuf, dwLen )
          Exit Function
       End If
       Call MoveMemory( ByVal p, ByVal bBuf, t )
       Call md5_Transform( ctx, VarPtr(ctx.bBuf) )
       bBuf = bBuf + t
       dwLen = dwLen - t
    End If

    'Process DATA IN 64-BYTE chunks
    Do While (dwLen >= 64)
       Call MoveMemory( ByVal VarPtr(ctx.bBuf), ByVal bBuf, 64)
       Call md5_Transform( ctx, VarPtr(ctx.bBuf) )
       bBuf = bBuf + 64
       dwLen = dwLen - 64
    Loop

    'handle any remaining bytes of data
    Call MoveMemory( ByVal VarPtr(ctx.bBuf), ByVal bBuf, dwLen )
End Function

' Final wrapup - pad TO 64-BYTE boundary WITH the BIT pattern
' 1 0* (64-BIT count of bits processed, MSB-first)


'----------------------------------------------------------------------------------------
Function md5_Final( ByVal bDigest As Byte Ptr, ctx As MD5_CTX)Export As Long
    Local lCount As Dword
    Local p As Byte Ptr
    Local bBuffer As Byte Ptr

    bBuffer = VarPtr(ctx.bBuf)
    'Compute number of bytes MOD 64
    lCount = (md5_shiftRight(ctx.lCount(0), 3) And &H3F )

    'set the first char of padding to &H80.  This is safe since there is
    'always AT least one BYTE free
    p = VarPtr(ctx.bBuf)
    p = p + lCount
    @p = &H80
    Incr p

    ' Bytes of padding needed TO make 64 bytes
    lCount = 64 - 1 - lCount

    ' Pad out TO 56 MOD 64
    If lCount < 8 Then
       'two lots of padding: pad the first block to 64 bytes
       Call FillMemory( p, lCount, 0 )
       Call md5_Transform( ctx, VarPtr(ctx.bBuf) )
       Call FillMemory( ByVal VarPtr(ctx.bBuf), 56, 0 )
    Else
       'pad block to 56 bytes
       Call FillMemory( ByVal p, (lCount - 8), 0)
    End If

    'append length IN bits and transform
    Call MoveMemory( ByVal VarPtr(@bBuffer[14 * SizeOf(lCount)]), ByVal VarPtr(ctx.lCount(0)), SizeOf(lCount))
    Call MoveMemory( ByVal VarPtr(@bBuffer[15 * SizeOf(lCount)]), ByVal VarPtr(ctx.lCount(1)), SizeOf(lCount))

    Call md5_Transform( ctx, VarPtr(ctx.bBuf) )
    Call MoveMemory( ByVal bDigest, ByVal VarPtr(ctx.lState(0)), 16)
    'Zeroise sensitive information
    Call FillMemory( ByVal VarPtr(ctx), SizeOf(ctx), 0)

End Function

'----------------------------------------------------------------------------------------
Function md5_Digest( ByRef ctx As MD5_CTX) Export As String
   'returns the MD5 digest as a hex string
   Dim sHex As String
   Dim szDigest As Asciiz * 16
   Dim bDigest As Byte Ptr
   Dim sCheck As String
   Dim i As Long

   bDigest = VarPtr(szDigest)
   Call md5_Final( bDigest, ctx)

   If bDigest <> %NULL Then
      For i = 0 To 15
        sHex = sHex + Hex$(@bDigest[i],2)
      Next
   End If
   Function = sHex

End Function
'----------------------------------------------------------------------------------------
Function md5_readFile( ByRef ctx As MD5_CTX,  szFile As Asciiz )Export As Long
   'reads a file and calls md5_Update on the buffer
   'md5_Init() MUST have been called prior to calling
   'this function - call md5_Digest() afterwards to
   'get the fingerprint
   Local hFile As HHANDLE
   Local szBuffer As Asciiz * 1024
   Local lBufLen As Long
   Local lLen As Long
   Local lResult As Long
   Local lToRead As Long
   Local lToReadHigh As Long
   Local lpBuffer As Long

   lBufLen = SizeOf(szBuffer)

   lpBuffer = VarPtr(szBuffer)
   hFile = CreateFile( szFile, %GENERIC_READ, %FILE_SHARE_READ, _
                          ByVal %NULL, %OPEN_EXISTING, %FILE_ATTRIBUTE_NORMAL, 0& )

   If hFile <> %INVALID_HANDLE_VALUE Then
       lToRead = GetFileSize(hFile, lToReadHigh)
       md5_readFile_loop:
        lResult = ReadFile( hFile, ByVal lpBuffer, lBufLen, lLen , ByVal %NULL )
        If lResult = 0  Then GoTo md5_readFile_close
        Call md5_Update(ctx, lpBuffer, lLen)
        lToRead = lToRead - lLen
        If lToRead = 0 Then GoTo md5_readFile_close
        GoTo md5_readFile_loop
       md5_readFile_close:
       Call CloseHandle(hFile)
   End If
End Function


'*************************************************************************
'END MD5 CODE
'*************************************************************************

'----------------------------------------------------------------------------------------------------------------
Function KeyExchange(KeyA As String,KeyB As String) Export As String
On Error Resume Next
Local A As Long, B As Long
Local A2 As Long, B2 As Long
Local DecryptKeyA As Long, DecryptKeyB As Long
Local Function1Way As Long
'A = Alice, B = Bob

'STAGE 1 - Alice and Bob choose a secret number.
A = 3
B = 6

'STAGE 2 - Alice and Bob put their secret numbers into
'          the one-way function.
A2 = OneWayFunc(A)  ' = 2
B2 = OneWayFunc(B)  ' = 4

'STAGE 3 - Alice and Bob tell each other their STAGE2 numbers.
'Ordinarily, this would be a crucial moment, because Alice and Bob are
'exchanging information, and therefore this is an opportunity for Eve
'to eavesdrop and find out the details of the information. However, it
'turns out that Eve can listen in without it affecting the ultimate
'security of the system. Alice and Bob could use the same telephone line
'that they used to agree the values for Y and P, and Eve could intercept
'the two numbers that are being exchanged, 2 and 4. However, these numbers
'are not the key, which is why it doesn't matter if Eve knows them.
'[no code required - imagine Alice and Bob simply telling each other their STAGE2 numbers.

'STAGE 4 - Alice takes Bob's STAGE2 number (B2) and works out the result using the one-way function
DecryptKeyB = (((B2 ^ A) Mod 11) Mod 11)
'Bob takes Alice's STAGE2 number (A2) and works out the result using the one-way function
DecryptKeyA = (((A2 ^ B) Mod 11) Mod 11)

'Display the results...
'STDOUT "Alice has ended up with: " & Str$(DecryptKeyB)
'STDOUT "  Bob has ended up with: " & Str$(DecryptKeyA)
'Miraculously, both Alice and Bob have ended up with the same key - 9.
'They can now proceed with secure encryption, knowing that their key (9) is safe.
End Function

'----------------------------------------------------------------------------------------------------------------
'For use above, do not export
Function OneWayFunc(LongIn As Long) As Long
On Error Resume Next
'// The general one-way function is Y^x (mod P).
'// Alive and Bob have chosen values for Y and P, and hence
'// have agreed on the one-way function 7^x (mod 11)
Function = (((7 ^ LongIn) Mod 11) Mod 11)
End Function

[John Strasser]

thanks Scott

John