Tweet
This is a sibling thread to SHA256/384/512 in XP SP3.
There we looked at how to access the higher ordered SHAs. AES is accessed in the same way - via the new Cryptographic Service Provider CSP).
Quote from Eddy Van Esch
Only two of the methods Eddy mentions are fully implemented in the MS CryptoAPI, namely ECB and CBC. CBC is the default but the following code allows ECB as an alternative. The weakness of ECB is that any repetitions in the plaintext will see repetitions in the ciphertext and that may be advantageous to an attacker. Whilst there will be many situations where ECB is fine, we don't have to concern ourselves when employing CBC as repetitions in the plaintext are not reflected in the ciphertext and hence CBCs popularity.
With regard to IV the MS default is all zeros. They could have chosen all 255, or all 99 or whatever. They could have chosen random bytes. The old DES block cipher had a weakness in that if an attacker got hold of a sufficient number of ciphertexts using the same IV then a successful attack was possible. I should add that this weakness has not been found in AES but I must confess to an unease of using the same IV forever even though I have no justification for such unease. The following code allows us to supply our own IV, as a hex string, and we may change it as many times as we wish. The obvious criteria is that the same IV must be used for both encryption and decryption. If the supplied IV is all zeros the result will be the same as the MS default IV. This may be simplified by passing an empty string. In practice those of us uneasy about using the same IV forever would probably not change it as often as we would like. Most of us tend to be lazy about changing passwords.
Another approach is to generate a random IV for each and every encryption. The likelihood of a IV ever repeating is 2^128 to 1 against with an AES block size of 16 bytes. This totally removes any unease of using the same IV all the time even though, as mentioned, I have no justification for such unease. The random IV is got by implementing the CryptGenRandom API, a cryptographic pseudo random number generator.
So, what we do with the generated IV? The method employed here is to dump the IV to disk and follow on with the encryption. The IV then becomes a header for the output file. On decryption we simply read the header first and then decrypt the following data. The ciphertext will then be 16 bytes longer than the plaintext.
Using a random IV is the default for the following code. A user IV is got by an optional string parameter in the main functions parameter list.
Naturally, a password is required of us. This may be ascii. In some other situations we may have no option but to use an ascii password but if a binary string is allowed then, I for one, would advocate that. In the following code if a password is wholly hexadecimal and has an even number of characters then it will be treated as a binary string. We don't have a choice here. "ABCD", for example, will be treated as a binary string whether we like it or not.
Even though our main purpose here is AES the ciphers used with the older CSP are still available. The block size then cannot be 'hard wired' and rather than have a look up table we can use the CryptGetKeyParam API to determine, amongst several other parameters, the block size for a given cipher. I have used the 56/112/168 bit variants of the DES block cipher, which uses a block size of 64 bit as opposed to 128 bit for AES. CryptGetKeyParam has been added to WinCrypt.inc along with the equates required for AES and found in post #3. This updates the WinCrypt.inc used in the opening sentences link. Setting the cipher mode, ECB or CBC, and a user IV is done via the CryptSetKeyParam API and this has been added to WinCrypt.inc as well. CryptGenRandom is also there.
Rather than read a file in its entirity we read blocks of 256KB. A 100MB test file peaked at just over 4MB usage in Task Manager. I have looked at buffer sizes many times before and found that using large buffers does not improve the speed of operation for this type of application. By 'type of application' I mean one primarily involved in reading data and 'crunching' it. Hashing applications are of this type. So, you could, for example, encrypt a 700MB iso image without having to worry about RAM usage.
The core code for encryption/decryption is done via the CryptEncrypt/CryptDecrypt APIs. The third parameter for both is called Final. If Final is true then this is a trigger to pad the block for encryption or remove the padding for decryption. I had looked a VirtualAlocc/CreateFile combo plus %FILE_FLAG_NO_BUFFERING for reading and writing but whilst reading was OK, writing was a pain as we are restricted to sector size. I am sure that we could work around this but at what cost so the PB Open was used to fill a string. The padding aspect is all done for us but we have to intervene in the event of a file to be encrypted being a multiple of the input buffer size. If we do not increase the buffer to accomodate the padding then although we have said Final is true the CryptEncrypt API will fail reckoning that more data is available.
We save the day in the Do/Loop processing the input buffer with
pdwDatalen tells the CryptEncrypt/CryptDecrypt APIs the amount of data to 'crunch'. Before the Final section this will be the whole input buffer. It will also be the whole input buffer if the Final section is equal to the input buffer. This is why we have to intervene. I should add that since the initial input buffer is an integral multiple of the block size then if it was fixed in size padding would not create an over run of the input buffer when the data read is less than the input buffer. However, this is academic as the 'saving' code adjusts the input buffer according to pdwDatalen whether or not an adjustment is needed. I didn't want to add extra code saying don't adjust the input buffer if the final section is less than the input buffer.
The interesting statement is in the provision for padding. If the block size is zero then pbdata is not changed, and hence the input buffer, that is no padding is applied. This is a result of BlockSize being zero and 'p Mod q' evaluating to zero if either or both expressions are zero. So, although the code is for block ciphers it will work for stream ciphers such as RC4 since Final is ignored and, as we have seen, padding does not occur.
So, the following code will work for all of the symmetric-key ciphers in the CryptoAPI.
I have not done extensive tests on timing. What I have found is that on my system AES 128 is about 20% faster than AES 256. This is significant but not as much as I would have thought. With AES 128 half of the time was spent reading and writing data with the other half 'crunching' the data. So, even if the AES implementation could be halved the total time would fall by only 25% as the bottleneck would then be the hard drive. My system comprises an Intel E6700 and a SATA hard drive. It follows then that with a slower hard drive the less of that 20% will be seen compared with my system. A SSD will see more of that 20%. Only a drive with an infinite speed will see all of that 20%.
The bottom line for me is that a 100MB file is read, crunched and written in about five seconds with AES 256 and, as mentioned earlier, RAM got hit for only 4MB. It would be pointless to give tables of timings as your bottom line will depend upon how fast your system can Read/Write and how fast can it 'crunch'.
Here is a summary of the main functions input/output.
Input:
sInfile: Filepath of target file. For decryption the extension must be enc.
WhichAlgorithm: %CALG_AES_128/192/256 and others previously available.
WhichMode: If 0 then default CBC will be used else ECB.
sPassword: If wholly hexadecimal and even number of characters then sPassword will be treated as a binary string else ASCII.
sInitialValues: If option not taken then IV will be cryptographic pseudo random bytes. If option taken then user supplied IV. For AES this will normally be 32 hex characters for a 128 bit block size. Having said that, if an empty string is supplied then the MS default IV will be used ie all zeros. If more than required is supplied then excess will be ignored. If less than required then the shortfall will be filled with zeros. sInitialValues is not required for decryption as IV is in the header of the decrypted file. If the ECB cipher mode has been chosen then any IV is ignored as ECB does not use any feedback.
Output: Empty string if failure or "OK" if success.
With such functions output is usually false or true for failure or success respectively but my departure is gearing up for a follow up project.
In a few days I will post a dll.
As always, have fun.
There we looked at how to access the higher ordered SHAs. AES is accessed in the same way - via the new Cryptographic Service Provider CSP).
Quote from Eddy Van Esch
The simplest method is named ECB (Electronic Code Book):
You add a number of bytes ('padding') so that your plaintext length becomes a multiple of the block size (16 bytes for AES).
You split up the plaintext in n blocks of each 16 bytes long. Every block is encrypted and concatenated to form the ciphertext.
After decryption, the padded bytes are removed from the plaintext.
So, the ciphertext must contain extra data to know how many bytes were padded.
Every encrypted block is independent from any other encrypted block.
There are several other methods, where the plaintext blocks are combined with other datablocks, usually the ciphertext of the previously encrypted plaintext block.
Most know methods are:
- CIPHER BLOCK CHAINING MODE (CBC)
- CIPHER FEEDBACK MODE (CFB)
- OUTPUT FEEDBACK MODE (OFB)
- COUNTER MODE (CTR)
These methods have the benefit over ECB mode that the ciphertext of a certain block is influenced by the previously encrypted blocks.
The tricky part for the first 3 methods is that, for the first encryped block, the plaintext is combined with a so-called 'Initial Vector' or IV.
The user can freely determine what this IV is. It does not even have to be kept a secret.
Depending on what IV you choose, your entire ciphertext looks different.
Which of the different block cipher methods (except ECB) is the 'best', is usually a matter of personal preference. I have never seen any real objective arguments why one should be better than the other. Most experts agree that CBC is very safe.
So, even if 2 people use the same block cipher mode for AES (CBC, ... all except ECB) and the same password and the same plaintext, still the ciphertext will look different if they use a different IV... This is the 'if' bit for all cipher block modes except for ECB.
You add a number of bytes ('padding') so that your plaintext length becomes a multiple of the block size (16 bytes for AES).
You split up the plaintext in n blocks of each 16 bytes long. Every block is encrypted and concatenated to form the ciphertext.
After decryption, the padded bytes are removed from the plaintext.
So, the ciphertext must contain extra data to know how many bytes were padded.
Every encrypted block is independent from any other encrypted block.
There are several other methods, where the plaintext blocks are combined with other datablocks, usually the ciphertext of the previously encrypted plaintext block.
Most know methods are:
- CIPHER BLOCK CHAINING MODE (CBC)
- CIPHER FEEDBACK MODE (CFB)
- OUTPUT FEEDBACK MODE (OFB)
- COUNTER MODE (CTR)
These methods have the benefit over ECB mode that the ciphertext of a certain block is influenced by the previously encrypted blocks.
The tricky part for the first 3 methods is that, for the first encryped block, the plaintext is combined with a so-called 'Initial Vector' or IV.
The user can freely determine what this IV is. It does not even have to be kept a secret.
Depending on what IV you choose, your entire ciphertext looks different.
Which of the different block cipher methods (except ECB) is the 'best', is usually a matter of personal preference. I have never seen any real objective arguments why one should be better than the other. Most experts agree that CBC is very safe.
So, even if 2 people use the same block cipher mode for AES (CBC, ... all except ECB) and the same password and the same plaintext, still the ciphertext will look different if they use a different IV... This is the 'if' bit for all cipher block modes except for ECB.
With regard to IV the MS default is all zeros. They could have chosen all 255, or all 99 or whatever. They could have chosen random bytes. The old DES block cipher had a weakness in that if an attacker got hold of a sufficient number of ciphertexts using the same IV then a successful attack was possible. I should add that this weakness has not been found in AES but I must confess to an unease of using the same IV forever even though I have no justification for such unease. The following code allows us to supply our own IV, as a hex string, and we may change it as many times as we wish. The obvious criteria is that the same IV must be used for both encryption and decryption. If the supplied IV is all zeros the result will be the same as the MS default IV. This may be simplified by passing an empty string. In practice those of us uneasy about using the same IV forever would probably not change it as often as we would like. Most of us tend to be lazy about changing passwords.
Another approach is to generate a random IV for each and every encryption. The likelihood of a IV ever repeating is 2^128 to 1 against with an AES block size of 16 bytes. This totally removes any unease of using the same IV all the time even though, as mentioned, I have no justification for such unease. The random IV is got by implementing the CryptGenRandom API, a cryptographic pseudo random number generator.
So, what we do with the generated IV? The method employed here is to dump the IV to disk and follow on with the encryption. The IV then becomes a header for the output file. On decryption we simply read the header first and then decrypt the following data. The ciphertext will then be 16 bytes longer than the plaintext.
Using a random IV is the default for the following code. A user IV is got by an optional string parameter in the main functions parameter list.
Naturally, a password is required of us. This may be ascii. In some other situations we may have no option but to use an ascii password but if a binary string is allowed then, I for one, would advocate that. In the following code if a password is wholly hexadecimal and has an even number of characters then it will be treated as a binary string. We don't have a choice here. "ABCD", for example, will be treated as a binary string whether we like it or not.
Even though our main purpose here is AES the ciphers used with the older CSP are still available. The block size then cannot be 'hard wired' and rather than have a look up table we can use the CryptGetKeyParam API to determine, amongst several other parameters, the block size for a given cipher. I have used the 56/112/168 bit variants of the DES block cipher, which uses a block size of 64 bit as opposed to 128 bit for AES. CryptGetKeyParam has been added to WinCrypt.inc along with the equates required for AES and found in post #3. This updates the WinCrypt.inc used in the opening sentences link. Setting the cipher mode, ECB or CBC, and a user IV is done via the CryptSetKeyParam API and this has been added to WinCrypt.inc as well. CryptGenRandom is also there.
Rather than read a file in its entirity we read blocks of 256KB. A 100MB test file peaked at just over 4MB usage in Task Manager. I have looked at buffer sizes many times before and found that using large buffers does not improve the speed of operation for this type of application. By 'type of application' I mean one primarily involved in reading data and 'crunching' it. Hashing applications are of this type. So, you could, for example, encrypt a 700MB iso image without having to worry about RAM usage.
The core code for encryption/decryption is done via the CryptEncrypt/CryptDecrypt APIs. The third parameter for both is called Final. If Final is true then this is a trigger to pad the block for encryption or remove the padding for decryption. I had looked a VirtualAlocc/CreateFile combo plus %FILE_FLAG_NO_BUFFERING for reading and writing but whilst reading was OK, writing was a pain as we are restricted to sector size. I am sure that we could work around this but at what cost so the PB Open was used to fill a string. The padding aspect is all done for us but we have to intervene in the event of a file to be encrypted being a multiple of the input buffer size. If we do not increase the buffer to accomodate the padding then although we have said Final is true the CryptEncrypt API will fail reckoning that more data is available.
We save the day in the Do/Loop processing the input buffer with
Code:
Incr SectionCount
Get$ hIn, dwBuflen, pbdata
If SectionCount = FinalSection Then ' note the Final data & prepare for encryption padding
pdwDatalen = Len(pbdata)
If IsFalse IsInputEncrypted Then
pbdata = pbdata + Space$(BlockSize - pdwDatalen Mod BlockSize) ' Provide for padding
dwBuflen = Len(pbdata)
' Note that dwBuflen may be reduced from its initial size but it is not neccessary
' An increase is necessary if this pdwDatalen = initial buffer size
End If
Final = %True
End If
The interesting statement is in the provision for padding. If the block size is zero then pbdata is not changed, and hence the input buffer, that is no padding is applied. This is a result of BlockSize being zero and 'p Mod q' evaluating to zero if either or both expressions are zero. So, although the code is for block ciphers it will work for stream ciphers such as RC4 since Final is ignored and, as we have seen, padding does not occur.
So, the following code will work for all of the symmetric-key ciphers in the CryptoAPI.
I have not done extensive tests on timing. What I have found is that on my system AES 128 is about 20% faster than AES 256. This is significant but not as much as I would have thought. With AES 128 half of the time was spent reading and writing data with the other half 'crunching' the data. So, even if the AES implementation could be halved the total time would fall by only 25% as the bottleneck would then be the hard drive. My system comprises an Intel E6700 and a SATA hard drive. It follows then that with a slower hard drive the less of that 20% will be seen compared with my system. A SSD will see more of that 20%. Only a drive with an infinite speed will see all of that 20%.
The bottom line for me is that a 100MB file is read, crunched and written in about five seconds with AES 256 and, as mentioned earlier, RAM got hit for only 4MB. It would be pointless to give tables of timings as your bottom line will depend upon how fast your system can Read/Write and how fast can it 'crunch'.
Here is a summary of the main functions input/output.
Input:
sInfile: Filepath of target file. For decryption the extension must be enc.
WhichAlgorithm: %CALG_AES_128/192/256 and others previously available.
WhichMode: If 0 then default CBC will be used else ECB.
sPassword: If wholly hexadecimal and even number of characters then sPassword will be treated as a binary string else ASCII.
sInitialValues: If option not taken then IV will be cryptographic pseudo random bytes. If option taken then user supplied IV. For AES this will normally be 32 hex characters for a 128 bit block size. Having said that, if an empty string is supplied then the MS default IV will be used ie all zeros. If more than required is supplied then excess will be ignored. If less than required then the shortfall will be filled with zeros. sInitialValues is not required for decryption as IV is in the header of the decrypted file. If the ECB cipher mode has been chosen then any IV is ignored as ECB does not use any feedback.
Output: Empty string if failure or "OK" if success.
With such functions output is usually false or true for failure or success respectively but my departure is gearing up for a follow up project.
In a few days I will post a dll.
As always, have fun.
Comment