unicode and dir$

[andrea d'ottavio]

I have this problem: can't get the filename when it contains non asci character.
i was just writing a program that retrived all the filenames of a directory using the dir$ command
when the filename contains non asci character the dir command converts them to asci. the result is that you get a non existing filename.
can anybody suggest a way to get the correct filenames (that i suppose in unicode) of a directory avoiding this problem?

[Mel Bishop]

Would be appreciated if you can give an example of a "non-ascii" character. All file names can only contain, with some exceptions, ASCII characters (0-255).

[Marco Pontello]

Not really.
Here's a sample file with a non-ASCII filename.

P.S.
It's just a renamed PNG image.

Bye!

[Michael Mattias]

Assuming the name can be expressed in Unicode, you can use the WinAPI function FindNextFileW() to find files.

BTW, are you sure your problem is FINDING a file? or is it DISPLAYING/PRINTING a filename?

[Michael Mattias]

Darned if I know how to display that one file with the funky characters, but here's a demo of using the Unicode version of FindFirstFile with PB/CC 4.03 ...

' CC_UNICODE_DIRS.bas
' 05-30-08
' PB/CC 4.03
' MCM (public domain) 

#COMPILE   EXE
#DIM       ALL
#REGISTER  NONE
#DEBUG     ERROR  ON
#TOOLS     OFF

' functions we'll be using


#INCLUDE "WIN32API.INC"

%MAX_PATH_W    = %MAX_PATH * 2
$UNICODE_NULL  =  CHR$(0,0)

TYPE WIn32_FIND_DATA_W
  dwFileAttributes AS DWORD
  ftCreationTime     AS FILETIME
  ftLastAccessTime   AS FILETIME
  ftLastWriteTime    AS FILETIME
  nFileSizeHigh      AS DWORD
  nFileSizeLow       AS DWORD
  dwReserved0        AS DWORD
  dwReserved1        AS DWORD
  cFileName          AS STRING * %MAX_PATH_W
  cAlternateFileName AS ASCIIZ * 14
END TYPE

    
DECLARE FUNCTION FindFirstFileW LIB "KERNEL32.DLL" ALIAS "FindFirstFileW" _
       (BYVAL lpFileName AS DWORD, W32W AS WIN32_FIND_DATA_W) AS DWORD

DECLARE FUNCTION FindNextFileW LIB "KERNEL32.DLL" ALIAS "FindNextFileW" _
      (BYVAL hFindFile AS DWORD, W32W AS WIN32_FIND_DATA_W) AS LONG
      
' ** FindClose is not ANSI/UNICODE sensitive can use the standard DECLARE in WIN32API.INC ***
       

FUNCTION PBMAIN() AS LONG
    
    LOCAL sADirList () AS STRING , sWDirList() AS STRING
    LOCAL I AS LONG
    
    REDIM  sADirList(0), sWDirList(0)
    
    CALL GetAnsiDirList ("D:\Work", sADirList ())
    
    CALL DisplayFileList (sADirList())
    STDOUT "End of Ansi File list, any key"
    WAITKEY$
    
    CALL GetUnicodeDirList ("D:\work", sWDirList())
    
    CALL DisplayFileList (sWDirList())
    STDOUT "End of Unicode File list, any key"
    WAITKEY$
    
    FOR I = 1 TO UBOUND (swDirList,1)
         STDOUT USING$ ("A:'&'    W:'&'", LSET$(sADirList(I), 30), sWDirList(I))
    NEXT
    STDOUT "End of Side-by-Size File list, any key"
    WAITKEY$
    
    
    
    'STDOUT "Any Key"
    'WAITKEY$
    
END FUNCTION

' DIR$ does not return the "." and ",," entries; Findfirst/next file does.
FUNCTION GetAnsiDirList (Folder AS STRING, sA () AS STRING) AS LONG
    
    LOCAL sF AS STRING, mask AS STRING
    LOCAL nFile AS LONG
    
    mask = Folder & "\*.*"
    sf   = DIR$(Mask, %SUBDIR)
    WHILE LEN (sf)
        INCR nFile
        sf = DIR$
    WEND
    
    DIR$ CLOSE
    
    REDIM SA(nFile)
    nFile   = 0
    sf      = DIR$(Mask, %SUBDIR)
    WHILE LEN (sf)
        INCR nFile
        sA(nFile) = sf
        sf = DIR$
    WEND
    
    
END FUNCTION
    
      
FUNCTION GetUnicodeDirList (Folder AS STRING, sW () AS STRING) AS LONG
    
    LOCAL W32W AS  WIN32_FIND_DATA_W, hSearch AS LONG
    LOCAL mask AS STRING, wMask AS STRING
    LOCAL nFile AS LONG, bFound AS LONG
    LOCAL nDot  AS LONG
    LOCAL wFile AS STRING
    
        
    mask = folder & "\*.*" & $NUL   ' add explicit null so it gets to uncode with double-null
    
    wMask = UCODE$(Mask)   ' convert to unicode
    
    hSearch = FindFirstFileW (BYVAL STRPTR (wMask), W32W)
    IF hSearch <> %INVALID_HANDLE_VALUE THEN
        bFound = %TRUE
    END IF
    
    WHILE ISTRUE bFound
             INCR nFile
             bFound = FindNextFileW (hSearch, W32W)
    WEND
    FindClose hSearch
    
    REDIM SW (nFile)
    nDot    = 0&

    hSearch = FindFirstFileW (BYVAL STRPTR (wMask), W32W)
    IF hSearch <> %INVALID_HANDLE_VALUE THEN
        bFound = %TRUE
    END IF
    nFile = 0
    
    WHILE ISTRUE bFound
             ' get entire returned filename
             wFile      = W32W.cFileName
             'get the portion of the filename up to the unicode terminator
             ' add a null so PB can convert to ANSI correctly
             wFile = EXTRACT$ (wFile, $UNICODE_NULL) & $NUL
             ' convert to ansi for display purposes
             WFile  = ACODE$ (wFile)
             ' if not the "." or ".." entry include
             IF LEFT$(wFile,1) <> "." THEN
                 INCR nFile
                 sw (nFile) =  wFile
             ELSE
                 INCR nDot
             END IF
             bFound      = FindNextFileW (hSearch, W32w)
    WEND
    FindClose hSearch
    IF nDot THEN
         REDIM PRESERVE sW (UBOUND (sw,1) - nDot)
    END IF
    
    
    
END FUNCTION

FUNCTION DisplayFileList (sF() AS STRING) AS LONG

 LOCAL i AS LONG
 
 FOR I =  1 TO UBOUND (Sf,1)
     STDOUT sf(i)
 NEXT
 
END FUNCTION

I just DL'd the linked file above to my "D:\Work" folder; both MS-DOS 'dir' and ACODE$ return all question marks for the name.

Win XP/Pro SP2 all updates.


MCM

[Kev Peel]

JFYI: the "?" mean that the converted UNICODE character(s) have no ASCII equivalent character. Using an app like this one will enable you to display the characters in their intended format.

[Michael Mattias]

1. Oh, you mean the only way to "see" these 'Funky' characters is to display them in a Unicode control? I could add that to the above.

2. PB has a CALL DWORD option which would obviate the need for the assembly language calls in your demo. Eg my demo of using the MSLU at PB/CC: IsTextUnicode with Microsoft Unicode Layer for Win95/98/ME April 11, 2002

[Kev Peel]

Oh, you mean the only way to "see" these 'Funky' characters is to display them in a Unicode control? I could add that to the above.

Yup

PB has a CALL DWORD option which would obviate the need for the assembly language calls in your demo

I know, we discussed this a while ago in a related thread. I prefer to use the ASM statements, as opposed to CALL DWORD. Just a personal preference.

[Michael Mattias]

Well... now I just get different funky characters...

' CC_UNICODE_DIRS.bas
' 05-30-08
' PB/CC 4.03
' MCM (public domain)
' Update: display in UNicode Cotnrol.
' Still Can't get the name of the funky file. I just get crap characters in unicode window.

#COMPILE   EXE
#DIM       ALL
#REGISTER  NONE
#DEBUG     ERROR  ON
#TOOLS     OFF

' functions we'll be using

#INCLUDE "WIN32API.INC"

%MAX_PATH_W    = %MAX_PATH * 2
$UNICODE_NULL  =  CHR$(0,0)

TYPE WIn32_FIND_DATA_W
  dwFileAttributes AS DWORD
  ftCreationTime     AS FILETIME
  ftLastAccessTime   AS FILETIME
  ftLastWriteTime    AS FILETIME
  nFileSizeHigh      AS DWORD
  nFileSizeLow       AS DWORD
  dwReserved0        AS DWORD
  dwReserved1        AS DWORD
  cFileName          AS STRING * %MAX_PATH_W
  cAlternateFileName AS ASCIIZ * 14
END TYPE

    
DECLARE FUNCTION FindFirstFileW LIB "KERNEL32.DLL" ALIAS "FindFirstFileW" _
       (BYVAL lpFileName AS DWORD, W32W AS WIN32_FIND_DATA_W) AS DWORD

DECLARE FUNCTION FindNextFileW LIB "KERNEL32.DLL" ALIAS "FindNextFileW" _
      (BYVAL hFindFile AS DWORD, W32W AS WIN32_FIND_DATA_W) AS LONG
      
' ** FindClose is not ANSI/UNICODE sensitive can use the standard DECLARE in WIN32API.INC ***
       

FUNCTION PBMAIN() AS LONG
    
    LOCAL sADirList () AS STRING , sWDirList() AS STRING
    LOCAL I AS LONG
    
    REDIM  sADirList(0), sWDirList(0)
    
    CALL GetAnsiDirList ("D:\Work", sADirList ())
    
    CALL DisplayFileList (sADirList())
    STDOUT "End of Ansi File list, any key"
    WAITKEY$
    
    CALL GetUnicodeDirList ("D:\work", sWDirList())
    
    CALL DisplayFileList (sWDirList(), %TRUE)
    STDOUT "End of Unicode File list, any key"
    WAITKEY$
    
    FOR I = 1 TO UBOUND (swDirList,1)
         STDOUT USING$ ("A:'&'    W:'&'", LSET$(sADirList(I), 30), ACODE$(sWDirList(I) & $NUL))
    NEXT
    STDOUT "End of Side-by-Size File list, any key"
    WAITKEY$
    
    CALL DisplayArraysSideBySide (sADirList(), sWDirList())
    
    
    STDOUT "End of Demo, Any Key"
    WAITKEY$
    
END FUNCTION

' DIR$ does not return the "." and ",," entries; Findfirst/next file does.
FUNCTION GetAnsiDirList (Folder AS STRING, sA () AS STRING) AS LONG
    
    LOCAL sF AS STRING, mask AS STRING
    LOCAL nFile AS LONG
    
    mask = Folder & "\*.*"
    sf   = DIR$(Mask, %SUBDIR)
    WHILE LEN (sf)
        INCR nFile
        sf = DIR$
    WEND
    
    DIR$ CLOSE
    
    REDIM SA(nFile)
    nFile   = 0
    sf      = DIR$(Mask, %SUBDIR)
    WHILE LEN (sf)
        INCR nFile
        sA(nFile) = sf
        sf = DIR$
    WEND
    
    
    
END FUNCTION

' TO DISPLAY IN A CONTROL, CANNOT CONVERT TO ANSI HERE... must return
' the array as a bunch of UNicode Strings.
    
      
FUNCTION GetUnicodeDirList (Folder AS STRING, sW () AS STRING) AS LONG
    
    LOCAL W32W AS  WIN32_FIND_DATA_W, hSearch AS LONG
    LOCAL mask AS STRING, wMask AS STRING
    LOCAL nFile AS LONG, bFound AS LONG
    LOCAL nDot  AS LONG
    LOCAL wFile AS STRING
    
        
    mask = folder & "\*.*" & $NUL   ' add explicit null so it gets to uncode with double-null
    wMask = UCODE$(Mask)   ' convert to unicode
    
    hSearch = FindFirstFileW (BYVAL STRPTR (wMask), W32W)
    IF hSearch <> %INVALID_HANDLE_VALUE THEN
        bFound = %TRUE
    END IF
    
    WHILE ISTRUE bFound
             INCR nFile
             bFound = FindNextFileW (hSearch, W32W)
    WEND
    FindClose hSearch
    
    REDIM SW (nFile)
    nDot    = 0&

    hSearch = FindFirstFileW (BYVAL STRPTR (wMask), W32W)
    IF hSearch <> %INVALID_HANDLE_VALUE THEN
        bFound = %TRUE
    END IF
    
    nFile = 0               ' reset
    WHILE ISTRUE bFound
             ' get entire returned filename
             wFile      = W32W.cFileName
             'get the portion of the filename up to the unicode terminator
             wFile = EXTRACT$ (wFile, $UNICODE_NULL)
             WFile = wFile & $NUL   ' needs this extra null here.
             
             'STDOUT USING$ ("Filename found LEN # ==> '&'", LEN(wFile), ACODE$(wFile))
             'WAITKEY$
             ' Look for the "." entries whilst still in Unicode...
             
             IF LEN(Wfile) > 1     THEN ' I was getting a null entry
                IF ACODE$(LEFT$(WFile, 2) & $NUL) <> "." THEN
                    INCR  nFile
                    sw (nFile) =  wFile
                ELSE
                    INCR nDot
                END IF
             ELSE
                 STDOUT "Got NUll Back (should not happen)"
                 WAITKEY$
             END IF
             bFound      = FindNextFileW (hSearch, W32w)
    WEND
    FindClose hSearch
    IF nDot THEN
         REDIM PRESERVE sW (UBOUND (sw,1) - nDot)
    END IF
    
    
    
END FUNCTION

FUNCTION DisplayFileList (sF() AS STRING, OPTIONAL BYVAL IsUnicode AS LONG) AS LONG

 LOCAL i AS LONG
 
 FOR I =  1 TO UBOUND (Sf,1)
     IF ISTRUE IsUnicode THEN
         STDOUT ACODE$(sf(i))
     ELSE
         STDOUT sf(i)
     END IF
 NEXT
 
END FUNCTION

DECLARE FUNCTION CreateWindowExW LIB "USER32.DLL" ALIAS "CreateWindowExW" _
(BYVAL dwExStyle AS DWORD, BYVAL lpClassName AS DWORD, BYVAL lpWindowName AS DWORD, BYVAL dwStyle AS DWORD, BYVAL x AS LONG, BYVAL y AS LONG, _
    BYVAL nWidth AS LONG, BYVAL nHeight AS LONG, _
    BYVAL hWndParent AS DWORD, BYVAL hMenu AS DWORD, BYVAL hInstance AS DWORD, lpParam AS ANY) AS DWORD
    
DECLARE FUNCTION SetWindowTextW LIB "USER32.DLL" ALIAS "SetWindowTextW" (BYVAL hWnd AS DWORD, BYVAL lpString AS DWORD) AS LONG



FUNCTION DisplayArraysSideBySide (sA() AS STRING, sw() AS STRING) AS LONG
    
  
  ' Create Two Multiline Edit controls and launch with contents (no parent)
  ' Keep track of how many Windows are open can't Post WM_QUIT until down to one window.
  
   LOCAL hWnd() AS LONG
   LOCAL dwStyle AS DWORD, sWClass AS STRING, sWTitle AS STRING
   LOCAL hParent AS LONG
   LOCAL i AS LONG
   LOCAL X AS LONG, Y AS LONG, cx AS LONG, Cy AS LONG
   LOCAL sWindowText AS STRING
   LOCAL hInst AS LONG
   
   
   
   ' ---------------------------------
   ' Stuff the same for both windows
   ' ---------------------------------
   hInst   =   GetModuleHandle (BYVAL %NULL)
   sWClass =  "edit" & $NUL
   swClass =  UCODE$(swClass)
   dwstyle =  %WS_OVERLAPPEDWINDOW OR %ES_MULTILINE OR %WS_BORDER OR %WS_VISIBLE OR %WS_THICKFRAME OR %WS_SYSMENU
   
   dwStyle =  DWSTYLE OR %ES_WANTRETURN OR %ES_READONLY OR %WS_VSCROLL OR %WS_HSCROLL
   
   hParent =  %NULL
   x       =  %CW_USEDEFAULT
   y       =  %CW_USEDEFAULT
   cx      =  %cW_USEDEFAULT
   cy      =  %CW_USEDEFAULT
   
   
   REDIM hWnd(1)
   
   FOR I = 0 TO 1
       sWindowText = ""
       IF I = 0 THEN
           swTitle      = "ANSI Directory List" & $NUL
           sWindowText  = JOIN$(sA(), $CRLF) & $NUL
           sWindowText  = UCODE$(sWindowText)
       ELSE
           swTitle     = "Unicode Directory List" & $NUL
           sWindowText = JOIN$(sW(), UCODE$($CRLF)) & $UNICODE_NULL
       END IF
       swTitle      = UCODE$(sWTitle)

       
       hWnd (i) = CreateWindowExW ( %NULL, STRPTR(swClass), STRPTR(swTitle), _
                   dwStyle, X, Y, Cx, Cy, hParent, %NULL, hInst, BYVAL %NULL)
                   
       SetWindowTextW hWnd(I), BYVAL STRPTR(sWindowText)
                   
       ShowWindow hWnd(i), %SW_SHOWNORMAL
                   
  NEXT

  LOCAL msg AS TagMsg
  LOCAL iMsg AS LONG
  
  DO
     iMsg =  GetMessage ( msg, %NULL, 0,0)
     IF ISTRUE iMSg THEN
        TranslateMessage msg
        DispatchMessage  msg
     END IF
  LOOP UNTIL ISFALSE ISwindow(hWnd(0)) AND ISFALSE IsWindow (hWnd(1))
  
END FUNCTION

[Kev Peel]

The second window displays it correctly here on my Vista system. Are you sure your system has a unicode supported font? Setting the font for the edit controls with a unicode-compatible font should display the text correctly.

[Michael Mattias]

Well, I think I'll try that font thing later today. If that works (Win/XP Pro SP2 all updates), I'll clean up the code and put it in the source code section, since by then it will certainly be useful. (I'll make it compilable with either PB/WIN or PB/CC at the same time). (Not that that will be a whole lot of effort).

[Michael Mattias]

>Attached Files Программы.zip
..
>It's just a renamed PNG image.

Actually, it IS a ZIP file, containing the "CALLDLL for MS-DOS" software I first saw posted I think on the PB forum back in the Compuserve days.

Sheesh, and I thought *I* kept things forever!

[Marco Pontello]

>Attached Files Программы.zip
..
>It's just a renamed PNG image.

Actually, it IS a ZIP file, containing the "CALLDLL for MS-DOS" software I first saw posted I think on the PB forum back in the Compuserve days.

Sheesh, and I thought *I* kept things forever!

I'm sorry, but you have probably confused the downloaded file with something different you already had on the HD.

This is the hexdump of the file I have attacched, just re-downloaded:

File size: 4411 Bytes

0000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52   .PNG........IHDR
0010: 00 00 00 64 00 00 00 64 08 06 00 00 00 70 E2 95   ...d...d.....p..
0020: 54 00 00 00 01 73 52 47 42 00 AE CE 1C E9 00 00   T....sRGB.......
0030: 00 04 67 41 4D 41 00 00 B1 8F 0B FC 61 05 00 00   ..gAMA......a...
0040: 00 20 63 48 52 4D 00 00 7A 26 00 00 80 84 00 00   . cHRM..z&......
0050: FA 00 00 00 80 E8 00 00 75 30 00 00 EA 60 00 00   ........u0...`..
0060: 3A 98 00 00 17 70 9C BA 51 3C 00 00 00 18 74 45   :....p..Q<....tE
0070: 58 74 53 6F 66 74 77 61 72 65 00 50 61 69 6E 74   XtSoftware.Paint
0080: 2E 4E 45 54 20 76 33 2E 32 32 B7 91 17 7D 00 00   .NET v3.22...}..
0090: 0F E4 49 44 41 54 78 5E ED 9D 85 97 53 57 17 C5   ..IDATx^....SW..
00A0: 43 F9 17 3E D7 BA 53 77 2F 75 77 77 A1 4A 95 2A   C..>..Sw/uww.J.*
00B0: 55 EA EE DE 52 A5 EE 54 A9 52 77 AA D4 9D 2A 75   U...R..T.Rw...*u
00C0: F7 EE EF 9E CC 3B C9 79 27 79 73 5F F6 BC BC 24   .....;.y'ys_...$
00D0: 5D 5F D7 62 75 86 81 00 B3 D7 BB BF EC 7D E4 0E   ]_.bu........}..
00E0: 9A F2 4F 40 A5 8D FF 0D 1A 54 A9 4C 31 45 A5 32   [email protected]
00F0: 78 70 A5 32 28 FC 5F 3E 9E 42 7E 2E 7C 5E FD 38   xp.2(._>.B~.|^.8

Pretty sure it's a PNG.

Bye!

[Michael Mattias]

Hey, setting the font works!!! I can read that filename now!!!

Also, I re-downloaded that file and sho' 'nuff, that's a PNG. Darned if I know how, when I downloaded it the first time, it came up a ZIP file. But I have this same file name now in two folders, and the first one I D'Ld is absolutely a ZIP file containing the CALLDLL stuff.

I swear on my mother's grave I did not rename "Calldll.zip" to that other name because I have no clue how to type those characters.

What do you think for the demo... just cut out all the STDOUT stuff and just let the program create the two windows?

I kind of like the STDOUT displays, at least for instructional purposes.

Oooh, here's an idea... maybe I should include those functions in the posted code, but just not call them? Then users can code 'em in at their leisure if they want!

How does that sound?

MCM

[Marco Pontello]

Darned if I know how, when I downloaded it the first time, it came up a ZIP file. But I have this same file name now in two folders, and the first one I D'Ld is absolutely a ZIP file containing the CALLDLL stuff.

Uhm... never had seen|downloaded|etc. that file, so I surely haven't uploaded it!

Bye!

[Michael Mattias]

Demo is at:

http://www.powerbasic.com/support/pbforums/showthread.php?t=37555

Compiles and runs "as is" using either PB/CC 4.03 or PB/Wn 8.03
Except: you may have to tinker a tad if using PB/Windows on Win 98/ME. But I put notes in the source code which 'should' do the trick in this case.

(Should run on prior versions of compilers, too, but I did not test that. )

I did not attach an EXE because
A. It compiles and runs 'as is' using either PB/CC 4.03 or PB/Wn 8.03
B. The equate $FOLDER_TO_TEST must be changed for your system anyway, unless you are interested in folder D:\Work.


MCM

[Paul Purvis]

thanks for the discussion on unicode guys.
i have not had any experience in it and this kind of explains a lot of it and problems that can be encountered and how to solve some of those issues.

MCM
after running the program i noticed that directories where included.
excuse me on my little knowledge in the use of findfirstfile and findnextfile api functions. is there a way to tell if the returned filename is a directory or file



MCM
i probably misunderstood some thoughts of your code dealing with $nul character but i saw this line on another forum and if it could be of any help, there it is.

http://www.experts-exchange.com/Programming/Languages/Visual_Basic/Q_21775294.html

"Like I said earlier - the filenames are in Unicode - you will need to convert to ANSI string before testing for Null"


so is it to my understanding to make any programs i write more global, i need to write code that sees files in unicode format.

paul

[Kev Peel]

I disabled the demo's STDOUT calls before compiling in PB/WIN. I could never get unicode to display in the console anyhow (codepage and other issues abound).

To test for a directory, a simple statement like this will do it (in the code above):

IsFolder = (w32w.dwFileAttributes AND %FILE_ATTRIBUTE_DIRECTORY)

The additional NULs are for double-null string termination, which is required for unicode strings, as opposed to the single-nul terminator in "traditional" ASCIIZ strings. Some functions need to detect the string length manually when it is not passed as a parameter.

[Michael Mattias]

What we could use for unicode handling is a new datatype: "UNICODESTRING, " using double-null the same way "ASCIIZ" uses single-null.

[Michael Mattias]

Some functions need to detect the string length manually when it is not passed as a parameter..

from Win32API.INC, factory edition..

DECLARE FUNCTION lstrlenw LIB "KERNEL32.DLL" ALIAS "lstrlenW" (lpString AS ANY) AS LONG

I guess I could have used that in the demo, but EXTRACT$() was so much more straightforward.