Max number of directories on XP

[BOB MECHLER]

We've added document imaging to our product and a prospect will have 13,000 customers needing to store statement pdf's, copies of checks sent and received etc. How many directories can be handled from a single node in XP, Server 2003 etc. I might have to create additional subdirectories under the root directory for the account is why I'm asking.

Bob Mechler

[Eddy Van Esch]

.. and how many files per directory are allowed ?

(sorry to hijack your thread, Bob)

Kind regards

[Steve Rossell]

I was interested in this so I searched MSDN and found an answer for FAT32 drives, saying the maxiumum number of files per directory is 65535. I found this at http://www.microsoft.com/technet/tec...sConfidential/.

Don't know about NTFS, I will see if I can find anything on this.

[Steve Rossell]

From what I can tell the max number of directories or folders on a NTFS file system is 4,294,967,295 (an unsigned 32-bit integer).

[Steve Rossell]

You can find these answers at
http://technet2.microsoft.com/Window....mspx?mfr=true and http://technet.microsoft.com/en-us/l.../bb457112.aspx

[BOB MECHLER]

Thanks, we should be fine then.

Bob Mechler

[Eric Pearson]

>>> a prospect will have 13,000 customers

>> the maxiumum number of files per [FAT] directory is 65,535

> Thanks, we should be fine then.

Ah yes, the Y2K Syndrome.

-- Eric

[Theo Gottwald]

And in short the answer is:

4,294,967,295 (2^32 minus 1 file) files or folders.

Maximum file size
Architecturally: 16 exabytes minus 1 KB (2^64 bytes minus 1 KB)
Implementation: 16 terabytes minus 64 KB (2^44 bytes minus 64 KB)



PS:If you use large numbers of files in an NTFS folder (300,000 or more), disable short-file name generation for better performance, and especially if the first six characters of the long file names are similar.

[Paul Purvis]

In the experience i have had, anytime you get more than about 500 files in a single directory, the process to manage those files really slows down and specially over 1000 files in a directory. i do not know how you are going to name your files, but i keep files in separate directories by their last character in the file name prior to the extension name.
so file abcd1.txt was saved in directory 1, then file abcd2.txt was saved in directory 2, and so on.

this made for faster backups and coping files and accessing files and defraging.
i was not using any dedicated microsoft servers.
i have moved much of my files on a *ix servers and have not had time to test speed improvements, but as i understand it, the file systems on a *ix servers are suppose to be less management, no defraging, and faster access times and more efficient network transfers, but still no proof in the pudding.

[Peter Lameijn]

I don't know if this still applies, but under NT4 / W2000 the maximum usable number of files in a directory was ~3000 files. Any higher number would result in incredibly slow operations performed on that directory. For 3000 files, a file search operation took very short (much less as 1 second). When we tried it with 30,000 files in one directory, the time needed could reach 45 seconds!

For some reason the OS got upset by the large number of files...

I don't know if this is resolved on XP...

[Kev Peel]

I would use a database for each customer that can store each file as a record, indexed by file name. It would probably be faster and you have the option of encryption/compression as well.

[BOB MECHLER]

The layout would be a single directory below which the user could have up to 13,000 directories off this one directory. The number of files in the directory would be no more than 100. The 13,000 directories off the one defined root directory could be on the same server or addon harddrive. The contents of the directory would pdf's, word docs, scanned images etc. The idea was to have a place for each financial customer to have all their original documents scanned into the computer for easy display. Just clicking on them in an open file dialog would load them in their native application for viewing. By using a directory structure the user could determine what went in the directory and how it got there. Based on volume, I'm guessing the user might want to create subdirectories by type of file, legal documents, checks, statements etc.

When a customer record is on screen and they click the Imaging menu item, an Open File Dialog opens directly showing their private directory. If one hasn't been created yet it is. From there they can move documents from other places or create them in place by right clicking and selecting the type of file to create. If a customer's directory ends up being empty, the directory is deleted as the master record is saved. We are making this ability available from any screen where a single record is displayed.

I think I've answered my own question. The customer needs to decide how to set up their own menu structure below the root directory named for the account number. Given the number of files supported, I don't have a problem. I still remember the days when you couldn't have more than 255 files off the root directory for the drive.

Being a Trust company they would not have as many checks coming or going out as a bank would but if the volume got too great, check images would be a good candidate for and SQL binary or blob field with an account/check number as key fields. The legal documents and correspondence could just be in their directory as individual files.

Bob Mechler

[Michael Mattias]

....the user could have up to 13,000 directories ...the number of files in the directory would be no more than 100..

Why is this application screaming "database, database" at me?

I can see the SINGLE table already (space permitting)

user_id               VARCHAR (24) NOT NULL,
user_doc_name    vARCHAR (24)
doc_type            INTEGER            value 1= BMP, 2 = MS-Word, etc....
data                   LONGBINARY     ' actual type name DBMS dependent; 
 Sql types:  SQL_LONGVARBINARY, SQL_VARBINARY or SQL_BINARY

If you wanted, you could save the image data compressed, which sounds like a really good idea given the kind of volumes you are talking about. Unless you happen to own stock in a disk drive manufacturer, that is.

Heck, you could even create a separate table for each user, that might actually work out better security-wise, as it would be pretty easy to GRANT access by user IDs.

[Mark Strickland]

Windows Behavior

Even if it is allowed sometimes you just should not do it.

Don't forget about the default behaviour of Windows to try and cache "everything". When you browse a folder it puts every item in the folder in a list in memory. This browsing behavior is found lots of places in Windows so dealing with long lists or things that are not found (mapped but missing network items) seems to make Windows feel sluggish even on a fast machine.

I would suggest BEFORE "banking" on it will work like you want just write a little program to generate this massive amount of files and directories as test data. From there you will have your real answer.

Michael is probably on the right track, although a little more work, that a database is a better method of dealing with this many items.

"Never jump out of an airplane without two parachutes unless it is parked on the ground."

[BOB MECHLER]

Trial run is something I should have thought of. I can emulate what they'll be doing here.

We have a professional document storage and retrieval company that we've made some basic agreements with to take over if it turns out an unsatisfactory answer.

Many of our customers just need a place for the monthly statement going back twelve months and a couple of trust documents stored. Our approach should be fine for them.

I appreciate you guys pounding away at the database idea. I initially shyed away from that idea because our other product used an SQL database with the document embedded in a field. This other customer got into the bad habit of modifying the documents and saving them back to the hard drive without updating the internal stored version. This was due to slowness and the huge size of the daily backup with all the documents in the database. Needless to say the stored documents no longer matched the documents stored in a folder on the disk.

I really need to verify the scenario can be done that way we thought.

Good thoughts everyone. Thanks,

Bob Mechler

[Michael Mattias]

This was due to slowness and the huge size of the daily backup with all the documents in the database

Incremental Backup?

CREATE TABLE Backupinfo (
   doc_key     VARCHAR (30), 
   last_backup  TIMESTAMP);

????

MCM

[Roger Garstang]

I'd have to agree with MCM on a database being a much better solution. Take it from someone having to deal with an app (wrote by someone else) that puts 4000 files in a TEMP folder to be processed everyday...not very efficient. Windows of course scans and shows it fine, but of course you have to consider Explorer has 9-18 threads always flopping around reading things. Process that folder on your own with one thread and you have a mess.

FWIW, the 4G limit for files sounds right since it is a 32bit system. NTFS maybe a little more though since it is 64bit isn't it (I have files on my system that are 6GB plus, and I thought it was a binary tree kind of like some of the Linux File Systems)? I've always thought only the root had limits and the subfolders only limited by space/memory. You also have to worry about path lengths like %MAX_PATH and path depth...especially if you are backing up to things like CD/DVD.

[BOB MECHLER]

I think I'm seeing where you are going with this Michael. Let me add a wrinkle to it.

Instead of an open file dialog come up, have a listbox with the types of documents stored. Once selected , populate a listview of DOC_KEY's with a description. Clicking on the desired item in the listview would unarchive (get a copy) it from, say a zip file. Since most items are for review only, they would not have to be put back. Let the zip file keep track of where it's at. Already compressed and ready to back up. If it hasn't been touched since the last backup, then why backup this account's zip file in the latest backup. That would accomplish the incremental backup strategy. The zip file would only be backed up again if something were added to it or changed.

The database here is Btrieve, which is why I'm dancing around an SQL implementation. Most of the companies are still on 6.15 and see no reason to change.

I'll keep my mind open though.

Bob Mechler

[Paul Purvis]

you can also create a small file to partner it with the actual file and let the partner file retain information in a formated way you can index on.
Index on?
create a index(database) from all these partner files.
therefore write a program to scan all directories(search paul d purvis and directory in the forum) for files and where a partner file does not exist, have a program request information from the user and make the information standard/uniform in nature then placed that information in the partner file.
you can have your cake and eat it too.
your database can also be rebuilt by reading these partner files that you will standardize with file extension names.
you can also always scan through partner files for searches on key words as a backup.

So where ever you decide in the future to place these files or move the files or combine the files in other folders together or split up the files the files or remove files, you can still have the partner file with standardized information to give you key information on the master file.

you can also scan the partner files to validate the file being a partner file is actually a partner file, because you are going to place some data in the partner file that identifies it as being a partner file and what version of partner file it is, because there will be changes to your format in the future to your partner files, i guarantee that.

and unless you have a large amount of files now, do not worry about file size and number of files now, technology will probably grow faster than your application.
i would try to keep the partner file to a size of the file block size used on a hard drive.

if you plan on keeping backups or running this application on a fat32 hard drive, i would probably format the hard drive to 32k blocks because you probably would be able to place a lot more more files on that kind of partition than say block sizes of 16k or smaller. On fat32 formated partitions, you can run out of fat space much faster that hard drive space in the partition on smaller files.

if you plan on a ntfs for backup or where the application is running, a 4k is probably best, win2k does not defrag any ntfs partitions with a block size larger than 4k.

[Peter Lameijn]

In the past we have used a combination of both. This was a huge number of voicefiles for a dating / voice response system.
The files were stored in a directory structure, and for search / retrieve purposes stored in a (btrieve) database.
The advantage is both fail-safe and high speed. Even if the database gets corrupted or lost, you can just re-create it by walking through the directory structure...