Help Using Pointers

[Doug Gamble]

I have a routine that reads data from a disk file from one file
8 bytes at a time into a quad variable. It then encrypts the
quad variable and then writes it to a different file. I would
like to speed the process by reading the data 32000 bytes at a
time and then process through the data using pointers, only I
can't get any code to work.

Does anyone have an experience with this or better yet a piece
of example code demonstrating something like this. I can read
and write the data into strings using GET# and Put# on a file
OPEN in binary mode. The problem is I haven't found a way of
getting real quad data variables out of the strings to
manipulate.

------------------

[Lance Edmonds]

Disclaimer: I'm writing this online without having a compiler handy to test this with.

'Psuedo code
DIM pQ1 as QUAD PTR, pQ2 as QUAD PTR
OPEN "FILE.TXT" FOR BINARY AS #1
FOR x& = 1 TO LOF(1) STEP 32000
  GET$ #1, 32000, a$
  b$ = a$         ' create output buffer of same length
  pQ1 = STRPTR(a$) ' use STRPTR32() with PB/DOS.
  pQ2 = STRPTR(b$)
  FOR y& = 1 to (LEN(a$) \ 8)    ' for example purposes... optimize for efficiency
    CALL MyEncryptFunction(@pQ1) ' function takes a BYREF QUAD 
    @pQ2 = @pQ1
    INCR @pQ1
    INCR @pQ2
  NEXT y&
  CALL WriteEncryptedString(b$)
NEXT x&

This should give you the idea. For most efficiency, dont place the encryption code in a sub/function, but make it LOCAL within the same sub/function as this code, then use GOSUB to call it. Alternatively, place the encryption code directly in the FOR/NEXT loop.

Another variation is to use Indexed Pointers:

'Psuedo code
DIM pQ1 as QUAD PTR, pQ2 as QUAD PTR
OPEN "FILE.TXT" FOR BINARY AS #1
FOR x& = 1 TO LOF(1) STEP 32000
  GET$ #1, 32000, a$
  b$ = a$         ' create output buffer of same length
  pQ1 = STRPTR(a$) ' use STRPTR32() with PB/DOS.
  pQ2 = STRPTR(b$)
  FOR y& = 0 to (LEN(a$) \ 8) - 1   ' for example purposes... optimize for efficiency
    CALL MyEncryptFunction(@pQ1[y&])   ' function takes a BYREF QUAD 
    @pQ2[y&] = @pQ1[y&]
  NEXT y&
  CALL WriteEncryptedString(b$)
NEXT x&

I hope this helps!

PS: If you are using PB/CC or PB/DLL (32-bit) then you can read the file in much larger chunks, say 1Mb or more. Personally, I would use 8192 byte chunks as a tradeoff for speed VS network traffic generation (large cunks tend to swamp a network more than smaller packets... 8192 bytes is a good "rule of thumb" chunk size for network based apps).

------------------
Lance
PowerBASIC Support
mailto:[email protected][email protected]</A>

[Steve Hutchesson]

Doug,

Unless you have good reason to encrypt 8 bytes at a time, there are much
more powerful and much faster ways of encrypting information at a file
level, the size limit will only be dictated by available memory.

The sub below is a rotating XOR type where you supply the string to be
encrypted (bString$) and a key string to encrypt it with (KeyString$). I
posted a variation of this algo some time ago

KeyString$ can be any length and the longer and more varied the better, an
image file is a very good source of key string and if its very long, it
will break the heart of any brute force attack.

The advantage of a very large key is that there are no repeat cycles on
small blocks that can be easily cracked, there is no problem with using a
key that is even larger than the file being encrypted.

What you need to avoid is a short repeat of the key string as the data can
be broken down to a short sample and cracked that way. If you use a key
over a couple of K in size, this becomes a lot more difficult. The SUB
below will use any length key and when it gets to the end, it loops and
starts again.

It is used as follows,

XorString MyFile$, MyKey$ ' encrypt file

' MyFile$ is now encrypted.

XorString MyFile$, MyKey$ ' decrypt file

' MyFile is now decrypted

This one should be fast enough to blow the hubcaps of most encryption
routines and powerful enough to break the heart of most forms of attack.

Hope its useful to you.

Regards,

[email protected]

' ###########################################################################
  
  SUB XorString(bString$,KeyString$)
  
      #REGISTER NONE
  
      LOCAL ln   as LONG      ' source length
      LOCAL lkey as LONG      ' key length
      LOCAL lref as LONG      ' counter reference for key char position
      LOCAL src  as LONG
      LOCAL lpBt as LONG
      LOCAL pcnt as LONG
      LOCAL bvar as BYTE
  
      ! push ebx
      ! push esi
      ! push edi
  
      ln   = len(bString$)
      lkey = len(KeyString$)
  
      lpBt = StrPtr(KeyString$)   ' get starting address of key string
      pcnt = lpBt                 ' copy it to another variable
      lref = pcnt + lkey
  
      ! mov esi, lpBt
      ! mov al,[esi]
      ! mov bvar, al      ; put 1st byte in bvar
  
      src = StrPtr(bString$)
  
      ! mov ecx, ln
      ! mov esi, src
      ! mov edi, src
  
    xsSt:
      ! mov al,[esi]      ; copy 1st byte of source into al
      ! inc esi
      
      ! xor al, bvar      ; xor al with next byte in bvar
  
      ! push eax
      ! push esi
  
    ' ====== This code gets the next byte in the key string ======
    '        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      ! inc pcnt          ; increment byte address
      ! mov esi, pcnt     ; put it in esi
      ! mov al,[esi]
      ! inc esi
  
      ! mov ebx, lref
      ! cmp pcnt, ebx
      ! jne xsNxt
  
      ! mov edx, lpBt
      ! mov pcnt, edx     ; reset pcnt to ariginal address
      ! mov esi, pcnt     ; put original address in esi
  
      ! mov al,[esi]
      ! inc esi
  
    xsNxt:
      ! mov bvar, al
    ' ============================================================
  
      ! pop esi
      ! pop eax
  
      ! mov [edi], al
      ! inc edi
  
      ! dec ecx
      ! cmp ecx, 0
      ! jne xsSt
  
      ! pop edi
      ! pop esi
      ! pop ebx
  
  END SUB
  
' ###########################################################################

------------------

[Doug Gamble]

Thanks for the feed back. I'm using a form of the Blowfish
encryption algorhythm which is much more secure than an XOR
method but thanks for the suggestion. I'll try the code for
using pointers and see what comes of it.

------------------

[Gafny Jacob]

You can try this method.

    OPEN "FILE.TXT" FOR BINARY AS #1
    X& = LOF(1)
    B$ = SPACE$(X&)
    GET #1,, B$
    CLOSE #1
    X& = X& \ 8 
    DIM Pq(X&) AS QUAD AT STRPTR(B$)
    FOR i = 1 TO X&
       CALL MyEncryptFunction(Pq(i)) 
    NEXT

HTH




------------------

[Tom Hanlin]

Originally posted by Steve Hutchesson:
The sub below is a rotating XOR type where you supply the string to be
encrypted (bString$) and a key string to encrypt it with (KeyString$). I
posted a variation of this algo some time ago

KeyString$ can be any length and the longer and more varied the better, an
image file is a very good source of key string and if its very long, it
will break the heart of any brute force attack.

As a matter of common sense, this would seem to be a fast and secure technique.
Unfortunately, when it comes to mathematics, common sense is not always a very
accurate guide. This type of XOR technique has a number of severe weaknesses
that make it a poor choice for secure encryption. It's also particularly
sensitive to choice of passwords. It would be inadvisable to use a text
password at all, here: you'd want to run it through a secure hash first, at
least.

I'd recommend a web search on "Bruce Schneier" (and reading any of his
excellent books) for more background on encryption.

------------------
Tom Hanlin
PowerBASIC Staff

[Doug Gamble]

So far I have had the best success reading data in 4096 byte
batches instead of 32000. This may have something to do with
the 4096 cluster size on my NTFS partition or something about
the encryption algorythm.

Thakns to everyone for the feedback...I'll let everyone know
what I find out regarding maximizing the file I/O.

------------------

[Steve Hutchesson]

Doug,

The actual encryption technique is not where the action is, you can have
rotations, displacements, XORs etc... , its the key size and repeat of
that key that make it weak or strong. Your problem will be that an 8 byte
long chunk is easy to break by brute force technique which is the normal
type of attack on an algo of that type.

56 bit DES was broken publically over 2 years ago and some of the cracking
groups have already broken 512 bit keys so if the data is high security,
you will need a stronger method.

The technique I suggested to you differs in so much as instead of using a
64 bit key, it can use ANY size key so if you are encrypting megabyte size
files, you can easily use a megabyte size key to encrypt it that has no
repeats in it at all. Put simply, a megabyte key is many magnitudes more
difficult to attack than a 64 bit key.

A simple XOR on one byte can be broken in 256 attempts, thats why the algo
uses a large key file so that there is no sequence that can be discovered.
The algo XORs each byte with a different byte from the key string that
cannot be predicted as it is a seperate file not contained in the source.

The largest single weakness in any encryption method is the security of the
key, no method can protect from leaked key data.

Regards,

[email protected]


------------------

[Steve Hutchesson]

Tom has an interesting point here, plaintext keys are dangerous for a
couple of reasons, they have a character distribution of about 70 or less,
(upper & lower case, numbers and some punctuation) which is a lot less
than the 256 possible, the other is that it leaves the data open to what
is called dictionary attack on the password.

Make an encryption system that will use short plain text passwords and you
will get "Bill", Fred" and other easy to remember words. A dictionary
attack lists a large range of common words, names and just brute force
tests them.

My own preference is to use a binary file with a simple one step
encryption process, a bitmap of a meg or so in 24 bit colour is well out
of the range of dictionary attacks and it does not have the depleted
character range of a plain text password.

Tom's suggestion of pre-processing the plain text password with one of the
standard hash algorithms solves the problem of character distribution so
if the password/phrase is large enough, the password is much harder to
attack.

For seriously secure data, I prefer multiple processed hybrids using very
large keys. Writing the individual algos in asm makes them fast enough to
increase the complexity to time ratio in this manner.

Code breaking is an interesting area, especially as modern computers can
be set up to test massive numbers of combinations in a relatively short
time. Relying on sheer quantity is becoming less effective as modern
computers become more powerful. From memory, the publically set up system
that broke DES 56 bit was testing some billions of keys per second and it
was by no means a smart method, sheer brute force can deliver results if
the combinations can be calculated in mathematical terms.

Where there is no smart method is in the protection of the password, you
may have a system that can wear out an array of Cray supercomputers but
if someone has a method of getting at the actual password/key text, it is
wasted.

Regards,

[email protected]

------------------