Announcement

Collapse
No announcement yet.

Max number of directories on XP

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Michael Mattias
    replied
    I think I see what you mean.

    "Here is an upgrade... FREE to you good customers on maintenance.

    "By the way, to install this free upgrade you'll need MS SQL Server.... and I think we can get you into that for about 25 kilobucks."

    http://www.microsoft.com/sql/howtobuy/default.mspx

    MCM
    (But I think I'd still be 'preparing' the users for this step. After all, if they want to "GO BIG" then getting a premium DBMS is probably not too unreasonable a request.

    But interesting enough, my closest direct competitor for the Provider Payment Partner(tm) System DOES require you have either M/S SQL Server or Oracle.... but in fairness, the Rycan Remittance Reporter has a LOT more features than does PPPS.

    Rycan is for like really big hospitals and service bureaus... PPPS is for SMALL hospitals, nursing homes and hospices, plus "reasonably small" group practices (1-30 or so professionals).

    Leave a comment:


  • BOB MECHLER
    replied
    Michael,

    We've been building a multi-currency version of our financial software that uses MS SQL for database tables and will use TSUNAMI tables for local work area private files. TSUNAMI has a call method almost exactly like btrcall. All this is encapsulated in the include files that do data retrieval etc. We opted to for the most part map the codes used by our btrieve app and converted them to dynamically built sql statements with SQL Tools.

    I used the DDF files to auto create the CREATE TABLE, INSERT, UPDATE, DELETE etc statements to replace the btrieve actions get greater than or equal and get next. The codes in btrieve are 9 and 6. With the 9 I generated an SQL cursor for all records in the MSSQL >= the key and the code 6 is just a fetch next record. It's more involved than that of course. I also use soft-locks so I can inform other users which user has their record locked. All cursors are forward only. If a generated SQL statement proves to be slow, I'll just write an optimized, parameterized query or stored procedure for the SQL version.

    Before doing all the SQL stuff I relied heavily on all the fine examples by Jose Roca for ADO, ODBC etc. Simply could not have done it without his examples.

    Once we've got the routines down for the multi-currency product, we'll convert our main product to MS SQL.

    New customers are turned off by btrieve and/or Pervasive but our existing customers would not have it any other way.

    All our Web products are fully VB.NET and MS SQL already. I just fill the SQL tables from our btrieve based financial product using SQL Tools.

    The way I'm writing all of this requires only an ini file with a connection string to signal the app that we're using SQL instead of btrieve, same include files.

    It's more work, but we don't feel we can force our existing customers to go to the extra expense of buying MS SQL. Also doing it this way allows to move incrementally to SQL without terminating the btrieve option should people want that route.

    Bob Mechler

    Leave a comment:


  • Michael Mattias
    replied
    The database here is Btrieve, which is why I'm dancing around an SQL implementation. Most of the companies are still on 6.15 and see no reason to change.
    I have always said applications software has a finite life, and that sooner or later you have to 'bite the bullet' and rewrite it starting with a blank page.

    After years and years of adding new features, you reach a point where adding still more features - features never envisioned when the system was first designed - becomes so difficult and expensive that 'rewrite' makes the most economic sense.

    Maybe you are not there yet, but it sure sounds like you are in the home stretch.


    MCM

    Leave a comment:


  • Peter Lameijn
    replied
    In the past we have used a combination of both. This was a huge number of voicefiles for a dating / voice response system.
    The files were stored in a directory structure, and for search / retrieve purposes stored in a (btrieve) database.
    The advantage is both fail-safe and high speed. Even if the database gets corrupted or lost, you can just re-create it by walking through the directory structure...

    Leave a comment:


  • Paul Purvis
    replied
    you can also create a small file to partner it with the actual file and let the partner file retain information in a formated way you can index on.
    Index on?
    create a index(database) from all these partner files.
    therefore write a program to scan all directories(search paul d purvis and directory in the forum) for files and where a partner file does not exist, have a program request information from the user and make the information standard/uniform in nature then placed that information in the partner file.
    you can have your cake and eat it too.
    your database can also be rebuilt by reading these partner files that you will standardize with file extension names.
    you can also always scan through partner files for searches on key words as a backup.

    So where ever you decide in the future to place these files or move the files or combine the files in other folders together or split up the files the files or remove files, you can still have the partner file with standardized information to give you key information on the master file.

    you can also scan the partner files to validate the file being a partner file is actually a partner file, because you are going to place some data in the partner file that identifies it as being a partner file and what version of partner file it is, because there will be changes to your format in the future to your partner files, i guarantee that.

    and unless you have a large amount of files now, do not worry about file size and number of files now, technology will probably grow faster than your application.
    i would try to keep the partner file to a size of the file block size used on a hard drive.

    if you plan on keeping backups or running this application on a fat32 hard drive, i would probably format the hard drive to 32k blocks because you probably would be able to place a lot more more files on that kind of partition than say block sizes of 16k or smaller. On fat32 formated partitions, you can run out of fat space much faster that hard drive space in the partition on smaller files.

    if you plan on a ntfs for backup or where the application is running, a 4k is probably best, win2k does not defrag any ntfs partitions with a block size larger than 4k.
    Last edited by Paul Purvis; 19 Nov 2007, 11:09 PM.

    Leave a comment:


  • BOB MECHLER
    replied
    I think I'm seeing where you are going with this Michael. Let me add a wrinkle to it.

    Instead of an open file dialog come up, have a listbox with the types of documents stored. Once selected , populate a listview of DOC_KEY's with a description. Clicking on the desired item in the listview would unarchive (get a copy) it from, say a zip file. Since most items are for review only, they would not have to be put back. Let the zip file keep track of where it's at. Already compressed and ready to back up. If it hasn't been touched since the last backup, then why backup this account's zip file in the latest backup. That would accomplish the incremental backup strategy. The zip file would only be backed up again if something were added to it or changed.

    The database here is Btrieve, which is why I'm dancing around an SQL implementation. Most of the companies are still on 6.15 and see no reason to change.

    I'll keep my mind open though.

    Bob Mechler

    Leave a comment:


  • Roger Garstang
    replied
    I'd have to agree with MCM on a database being a much better solution. Take it from someone having to deal with an app (wrote by someone else) that puts 4000 files in a TEMP folder to be processed everyday...not very efficient. Windows of course scans and shows it fine, but of course you have to consider Explorer has 9-18 threads always flopping around reading things. Process that folder on your own with one thread and you have a mess.

    FWIW, the 4G limit for files sounds right since it is a 32bit system. NTFS maybe a little more though since it is 64bit isn't it (I have files on my system that are 6GB plus, and I thought it was a binary tree kind of like some of the Linux File Systems)? I've always thought only the root had limits and the subfolders only limited by space/memory. You also have to worry about path lengths like %MAX_PATH and path depth...especially if you are backing up to things like CD/DVD.
    Last edited by Roger Garstang; 19 Nov 2007, 04:51 PM.

    Leave a comment:


  • Michael Mattias
    replied
    This was due to slowness and the huge size of the daily backup with all the documents in the database
    Incremental Backup?

    Code:
    CREATE TABLE Backupinfo (
       doc_key     VARCHAR (30), 
       last_backup  TIMESTAMP);
    ????

    MCM

    Leave a comment:


  • BOB MECHLER
    replied
    Trial run is something I should have thought of. I can emulate what they'll be doing here.

    We have a professional document storage and retrieval company that we've made some basic agreements with to take over if it turns out an unsatisfactory answer.

    Many of our customers just need a place for the monthly statement going back twelve months and a couple of trust documents stored. Our approach should be fine for them.

    I appreciate you guys pounding away at the database idea. I initially shyed away from that idea because our other product used an SQL database with the document embedded in a field. This other customer got into the bad habit of modifying the documents and saving them back to the hard drive without updating the internal stored version. This was due to slowness and the huge size of the daily backup with all the documents in the database. Needless to say the stored documents no longer matched the documents stored in a folder on the disk.

    I really need to verify the scenario can be done that way we thought.

    Good thoughts everyone. Thanks,

    Bob Mechler

    Leave a comment:


  • Mark Strickland
    replied
    Windows Behavior

    Even if it is allowed sometimes you just should not do it.

    Don't forget about the default behaviour of Windows to try and cache "everything". When you browse a folder it puts every item in the folder in a list in memory. This browsing behavior is found lots of places in Windows so dealing with long lists or things that are not found (mapped but missing network items) seems to make Windows feel sluggish even on a fast machine.

    I would suggest BEFORE "banking" on it will work like you want just write a little program to generate this massive amount of files and directories as test data. From there you will have your real answer.

    Michael is probably on the right track, although a little more work, that a database is a better method of dealing with this many items.

    "Never jump out of an airplane without two parachutes unless it is parked on the ground."

    Leave a comment:


  • Michael Mattias
    replied
    ....the user could have up to 13,000 directories ...the number of files in the directory would be no more than 100..
    Why is this application screaming "database, database" at me?

    I can see the SINGLE table already (space permitting)

    Code:
    user_id               VARCHAR (24) NOT NULL,
    user_doc_name    vARCHAR (24)
    doc_type            INTEGER            value 1= BMP, 2 = MS-Word, etc....
    data                   LONGBINARY     ' actual type name DBMS dependent; 
     Sql types:  SQL_LONGVARBINARY, SQL_VARBINARY or SQL_BINARY
    If you wanted, you could save the image data compressed, which sounds like a really good idea given the kind of volumes you are talking about. Unless you happen to own stock in a disk drive manufacturer, that is.

    Heck, you could even create a separate table for each user, that might actually work out better security-wise, as it would be pretty easy to GRANT access by user IDs.

    Leave a comment:


  • BOB MECHLER
    replied
    The layout would be a single directory below which the user could have up to 13,000 directories off this one directory. The number of files in the directory would be no more than 100. The 13,000 directories off the one defined root directory could be on the same server or addon harddrive. The contents of the directory would pdf's, word docs, scanned images etc. The idea was to have a place for each financial customer to have all their original documents scanned into the computer for easy display. Just clicking on them in an open file dialog would load them in their native application for viewing. By using a directory structure the user could determine what went in the directory and how it got there. Based on volume, I'm guessing the user might want to create subdirectories by type of file, legal documents, checks, statements etc.

    When a customer record is on screen and they click the Imaging menu item, an Open File Dialog opens directly showing their private directory. If one hasn't been created yet it is. From there they can move documents from other places or create them in place by right clicking and selecting the type of file to create. If a customer's directory ends up being empty, the directory is deleted as the master record is saved. We are making this ability available from any screen where a single record is displayed.

    I think I've answered my own question. The customer needs to decide how to set up their own menu structure below the root directory named for the account number. Given the number of files supported, I don't have a problem. I still remember the days when you couldn't have more than 255 files off the root directory for the drive.

    Being a Trust company they would not have as many checks coming or going out as a bank would but if the volume got too great, check images would be a good candidate for and SQL binary or blob field with an account/check number as key fields. The legal documents and correspondence could just be in their directory as individual files.

    Bob Mechler
    Last edited by BOB MECHLER; 17 Nov 2007, 10:48 PM.

    Leave a comment:


  • Kev Peel
    replied
    I would use a database for each customer that can store each file as a record, indexed by file name. It would probably be faster and you have the option of encryption/compression as well.

    Leave a comment:


  • Peter Lameijn
    replied
    I don't know if this still applies, but under NT4 / W2000 the maximum usable number of files in a directory was ~3000 files. Any higher number would result in incredibly slow operations performed on that directory. For 3000 files, a file search operation took very short (much less as 1 second). When we tried it with 30,000 files in one directory, the time needed could reach 45 seconds!

    For some reason the OS got upset by the large number of files...

    I don't know if this is resolved on XP...

    Leave a comment:


  • Paul Purvis
    replied
    In the experience i have had, anytime you get more than about 500 files in a single directory, the process to manage those files really slows down and specially over 1000 files in a directory. i do not know how you are going to name your files, but i keep files in separate directories by their last character in the file name prior to the extension name.
    so file abcd1.txt was saved in directory 1, then file abcd2.txt was saved in directory 2, and so on.

    this made for faster backups and coping files and accessing files and defraging.
    i was not using any dedicated microsoft servers.
    i have moved much of my files on a *ix servers and have not had time to test speed improvements, but as i understand it, the file systems on a *ix servers are suppose to be less management, no defraging, and faster access times and more efficient network transfers, but still no proof in the pudding.

    Leave a comment:


  • Theo Gottwald
    replied
    And in short the answer is:

    4,294,967,295 (2^32 minus 1 file) files or folders.

    Maximum file size
    Architecturally: 16 exabytes minus 1 KB (2^64 bytes minus 1 KB)
    Implementation: 16 terabytes minus 64 KB (2^44 bytes minus 64 KB)



    PS:If you use large numbers of files in an NTFS folder (300,000 or more), disable short-file name generation for better performance, and especially if the first six characters of the long file names are similar.

    Leave a comment:


  • Eric Pearson
    replied
    >>> a prospect will have 13,000 customers

    >> the maxiumum number of files per [FAT] directory is 65,535

    > Thanks, we should be fine then.

    Ah yes, the Y2K Syndrome.

    -- Eric

    Leave a comment:


  • BOB MECHLER
    replied
    Thanks, we should be fine then.

    Bob Mechler

    Leave a comment:


  • Steve Rossell
    replied
    You can find these answers at
    http://technet2.microsoft.com/Window....mspx?mfr=true and http://technet.microsoft.com/en-us/l.../bb457112.aspx

    Leave a comment:


  • Steve Rossell
    replied
    From what I can tell the max number of directories or folders on a NTFS file system is 4,294,967,295 (an unsigned 32-bit integer).

    Leave a comment:

Working...
X