API problem

[David Roberts]

I have a problem with the CryptGenRandom function either within itself or what it relies upon as I am getting a 'The parameter is incorrect' error.

I've read the SDK docs and checked out a lot of C code got from a Google Search Code and cannot figure where the problem is.

I am assuming that the problem is with my code but after kicking it around for a few days, trying various pointer methods and declarations I am now wondering about the function itself; found in the NT kernel starting at Win2000. There is a security issue here which Microsoft will be fixing in XP SP3 but that is not a problem within the context of my use.

I've put CryptGenRandom into a code snippet, below, to illustrate the error generated.

Perhaps someone can spot my mistake?

The code should work with Vista so perhaps someone could try it out with that for me?

The intended output, in this case, is a 16 byte hex string.

#Compile Exe
#Register None
#Dim      All

%NOMMIDS = 1
%NOGDI = 1

#Include "WIN32API.inc" ' 27 January 2005

$App = "CGR_Problem"
$MS_DEF_PROV = "Microsoft Base Cryptographic Provider v1.0"
%PROV_RSA_FULL = 1
%CRYPT_VERIFYCONTEXT = &hF0000000

Macro SysErrMsg(API, Number) = MsgBox SystemErrorMessage( WinErr, API, FuncName$, Number ), %MB_ICONERROR Or %MB_TOPMOST, $App

Declare Function crGetDefaultRSAHandle( ByVal Flags As Dword ) As Long
Declare Function CryptAcquireContext Lib "advapi32.dll" Alias "CryptAcquireContextA" _
      ( hProv As Long, zContainer As Asciiz, zProvider As Asciiz, _
         ByVal dwProvType As Dword, ByVal dwFlags As Dword ) As Long
Declare Function CryptReleaseContext Lib "advapi32.dll" Alias "CryptReleaseContext" _
      ( ByVal hProv As Long, ByVal dwFlags As Dword ) As Long
Declare Function CryptGenRandom Lib "advapi32.dll" Alias "CryptGenRandom" _
      ( hProv As Long, ByVal dwLen As Dword, ByVal pbBuffer As Byte Ptr ) As Long
Declare Function SystemErrorMessage( ByVal Long, String, String, ByVal Long ) As String

Function PBMain

  Local hProv, WinErr, i As Long, hexSize As Dword
  Dim hexByte( ) As Byte
  Dim strHex As String

  hProv = crGetDefaultRSAHandle( %CRYPT_VERIFYCONTEXT )
  If hProv = 0 Then Exit Function

  hexSize = 16 ' Will actually vary but 16 is one such value
  ReDim hexByte( hexSize - 1 ) As Byte

  If CryptGenRandom( hProv, ByVal hexSize, hexByte( 0 ) )= 0 Then
    WinErr = GetLastError : SysErrMsg( "CryptGenRandom", 10 )
    GoTo TidyUp
  End If

  strHex = ""
  For i =0 To hexSize - 1
    strHex = strHex + Hex$( hexByte(i),2 )
  Next
  MsgBox strHex

  TidyUp:
  CryptReleaseContext hProv, 0

End Function

Function crGetDefaultRSAHandle( ByVal Flags As Dword ) As Long

  Local hProv, WinErr As Long

  If CryptAcquireContext( hProv, ByVal %Null, ByVal %Null, %PROV_RSA_FULL, Flags ) = %FALSE Then
    If CryptAcquireContext( hProv, ByVal %Null, $MS_DEF_PROV, %PROV_RSA_FULL, Flags ) = %FALSE Then
      WinErr = GetLastError : SysErrMsg( "CryptAcquireContext", 10 )
      Function = %FALSE
    Else
      Function = hProv
    End If
  Else
    Function = hProv
  End If

End Function

Function SystemErrorMessage( ByVal WinErr As Long, API As String, sFunction As String, ByVal Number As Long ) As String

  Local pBuffer As Asciiz Ptr, ncbBuffer As Dword
  Dim sText As String

  ncbBuffer = FormatMessage( %FORMAT_MESSAGE_ALLOCATE_BUFFER Or %FORMAT_MESSAGE_FROM_SYSTEM Or %FORMAT_MESSAGE_IGNORE_INSERTS, _
  ByVal %Null, WinErr, %Null, ByVal VarPtr(pBuffer), 0, ByVal %Null )
  If ncbBuffer Then
    sText = Peek$(pBuffer, ncbBuffer)
    sText = Remove$(sText, Any Chr$(13) + Chr$(10))
    Function = $CrLf + "Error" + Format$(WinErr, " #####") + " during " + API + $CrLf + $CrLf + _
      Chr$(34) + sText + Chr$(34) + $CrLf + $CrLf + "Location : " + sFunction + IIf$(Number, Format$(Number, " ###"), "")
    LocalFree pBuffer
  Else
    Function = "Unknown error - Code:" + Format$(WinErr, " #####")
  End If

End Function

[Kev Peel]

Right off the bat

 [B]BYVAL [/B]hProv As Long

I can try it on Vista later, I'm on my Win2K laptop at the moment..

[Kev Peel]

Addendum: Here's the original C declare:

BOOL WINAPI CryptGenRandom(
  __in     HCRYPTPROV [I]hProv[/I],
  __in     DWORD [I]dwLen[/I],
  __inout  BYTE* [I]pbBuffer[/I]
);

[jcfuller]

What Kev said :
Declare Function CryptGenRandom Lib "advapi32.dll" Alias "CryptGenRandom" _
( BYVAL hProv As Long, ByVal dwLen As Dword, ByVal pbBuffer As Byte Ptr ) As Long

And:

Also needed VARPTR(hexByte( 0 ))
If CryptGenRandom( BYVAL hProv, ByVal hexSize, VARPTR(hexByte( 0 )) )= 0 Then

James

[Kev Peel]

Oh yes, how could I miss that? A definite GPF cause for sure

[David Roberts]

Thanks gentlemen.

I've messed about with so many combinations of logic that I gave up and tried brute force - sometimes works.

I normally use 'BYVAL hProv as Long' as you can see in CryptReleaseContext. Sorry, I should have put that back to my starting point. I do not with CryptAcquireContext, of course, since hProv is passed by reference with the SDK as 'HCRYPTPROV* phProv'.

With an invalid parameter the function will return without further ado so a GPF cannot be on the cards.

What I cannot understand is that VARPTR(hexByte(0)) now works.

I had tried

phexByte = VarPtr( hexByte( 0 ) )
If CryptGenRandom( ByVal hProv, ByVal hexSize, ByVal phexByte )= 0 Then
  WinErr = GetLastError : SysErrMsg( "CryptGenRandom", 10 )
  GoTo TidyUp
End If

to no avail.

Since it now works I am now wondering what the hell else I was up to causing it not to.

Anyway, I came to the forum with broken code and am now leaving with it fixed. Thanks again.

[Kev Peel]

David,

It works because the parameter is declared as requiring a BYVAL address (must be a numeric address [pointer] to the variable). The call would have also worked if you had it declared as BYREF pbBuffer as BYTE, where the compiler will translate the parameter for you, removing the need for VARPTR. Both parameter types will work.

[David Roberts]

Thanks Kev.

This is very strange.

Since BYREF is implicit then simply 'pbBuffer as BYTE' will do and with the latest compilers, forget which version introduced it, we can drop the variable name in the declaration and use just 'BYTE'.

I had been down all these paths and posted my last shot at it.

I can now only find a few ways where it doesn't work!

I'll leave my system on a few days, work as normal and keep an eye on things.