Announcement

Collapse
No announcement yet.

fragmentation of a file on disk

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • David Roberts
    replied
    Nope, right clicking is too much hard work - he has gone for a silent mode schedule.

    Leave a comment:


  • David Roberts
    replied
    > Is there any possibility you could translate some of the C into a simple working demo?

    I'd struggle with C. There are others here much better equipped than I in this respect.

    I and another forum member have written to the WinContig's author, Marco D'Amato, with some issues we had and Marco has quickly attended to my issues. There is a new build available to download.

    A useful tip was also given me in putting WinContig into the SendTo folder. This will be useful for a friend of mine who has a particular file which tends to fragment badly. He'll be able to right click and analyse and then right click to defrag as required.

    Added: Only a right click to analyze is needed. If fragmentation then a 'Defragment' button becomes enabled in the report.
    Last edited by David Roberts; 12 Jan 2008, 03:00 PM.

    Leave a comment:


  • Mike Trader
    replied
    Thx David.
    I wrote to the author asking him about APIs etc
    Hello Mike,

    You can take a look at http://ultradefrag.sourceforge.net/.
    I suspect he based his work on this.

    Is there any possibility you could translate some of the C into a simple working demo?

    Leave a comment:


  • David Roberts
    replied
    The following does not answer Mike's original query but nonetheless it is worth mentioning in this thread. I came across it in a UK magazine's RSS feed.

    The app will analyse a group of files/folders down to a single file, report the fragmentation and defrag if wanted.

    It has more command line switches than we can shake a stick at.

    It is 'portable' so we can drop it onto a flash drive without further ado.

    The link is to the English page.

    WinContig

    I should add:

    "WinContig is released as freeware for personal and commercial use.

    WinContig is an application designed for Windows 2000, Windows XP, Windows Vista operating systems.

    As a user of Windows 2000, Windows XP or Windows Vista you have to be logged in with full administration rights."

    More: I've just analysed C:. This app is on steroids.
    Last edited by David Roberts; 7 Jan 2008, 09:07 PM.

    Leave a comment:


  • Mike Trader
    replied
    David,
    Good detective work, thank you.
    Now all we need is a little help converting the C!



    Defrag API C# wrappers

    For my own amusement, I decided to wrap the NT Defrag APIs in C#.

    It isn't great, but... I'll give it the classic dev guarantee of "it works on my machine". http://www.thenetworkadministrator.com/top20.htm

    The MSDN documentation uses lots of very scary words like "IOCTL" and "DeviceIoControl" and the such. Hopefully, I've abstracted some of that away...



    First, you can get a bitmap of the cluster usage on the device...

    static public BitArray GetVolumeMap(string DeviceName)

    Then you can get a map of a file - virtual clusters and logical clusters

    static public Array GetFileMap(string path)

    Finally, you can move sections of a file around on a volume...

    static public void MoveFile(string deviceName, string path, Int64 VCN, Int64 LCN, Int32 count)



    Here's the whole thing, nearly guaranteed to break your file system if used.



    Code:
    //
    // a set of simple C# wrappers over the NT Defragmenter APIs
    //
    
    //
    // Refrences
    
    //
    // http://www.sysinternals.com/ntw2k/info/defrag.shtml
    //
    // msdn how-to
    // ms-help://MS.MSDNQTR.2004JUL.1033/fileio/base/defragmenting_files.htm
    // http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/defragmenting_files.asp
    //
    // FSCTL_GET_VOLUME_BITMAP
    // http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/fsctl_get_volume_bitmap.asp
    //
    // interesting structures...
    // FSCTL_MOVE_FILE
    // FSCTL_GET_RETRIEVAL_POINTERS
    // RETRIEVAL_POINTERS_BUFFER
    // FSCTL_GET_RETRIEVAL_POINTERS
    //
    // DeviceIoControl
    // http://msdn.microsoft.com/library/default.asp?url=/library/en-us/devio/base/deviceiocontrol.asp
    //
    
    using System;
    using System.Diagnostics;
    using System.Collections;
    using System.Runtime.InteropServices;
    
    namespace defraglib
    {
        public class IOWrapper
        {
    
            //
            // CreateFile constants
            //
            const uint FILE_SHARE_READ = 0x00000001;
            const uint FILE_SHARE_WRITE = 0x00000002;
            const uint FILE_SHARE_DELETE = 0x00000004;
            const uint OPEN_EXISTING = 3;
    
            const uint GENERIC_READ = (0x80000000);
            const uint GENERIC_WRITE = (0x40000000);
    
            const uint FILE_FLAG_NO_BUFFERING = 0x20000000;
            const uint FILE_READ_ATTRIBUTES = (0x0080);
            const uint FILE_WRITE_ATTRIBUTES = 0x0100;
            const uint ERROR_INSUFFICIENT_BUFFER = 122;
    
            [DllImport("kernel32.dll", SetLastError = true)]
            static extern IntPtr CreateFile(
                string lpFileName,
                uint dwDesiredAccess,
                uint dwShareMode,
                IntPtr lpSecurityAttributes,
                uint dwCreationDisposition,
                uint dwFlagsAndAttributes,
                IntPtr hTemplateFile);
    
            [DllImport("kernel32.dll", SetLastError = true)]
            static extern int CloseHandle(IntPtr hObject);
    
            [DllImport("kernel32.dll", SetLastError = true)]
            static extern bool DeviceIoControl(
                IntPtr hDevice,
                uint dwIoControlCode,
                IntPtr lpInBuffer,
                uint nInBufferSize,
                [Out] IntPtr lpOutBuffer,
                uint nOutBufferSize,
                ref uint lpBytesReturned,
                IntPtr lpOverlapped);
    
            static private IntPtr OpenVolume(string DeviceName)
            {
                IntPtr hDevice;
                hDevice = CreateFile(
                    @"\\.\" + DeviceName,
                    GENERIC_READ | GENERIC_WRITE,
                    FILE_SHARE_WRITE,
                    IntPtr.Zero,
                    OPEN_EXISTING,
                    0,
                    IntPtr.Zero);
                if ((int)hDevice == -1)
                {
                    throw new Exception(Marshal.GetLastWin32Error().ToString());
                }
                return hDevice;
            }
    
            static private IntPtr OpenFile(string path)
            {
                IntPtr hFile;
                hFile = CreateFile(
                            path,
                            FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES,
                            FILE_SHARE_READ | FILE_SHARE_WRITE,
                            IntPtr.Zero,
                            OPEN_EXISTING,
                            0,
                            IntPtr.Zero);
                if ((int)hFile == -1)
                {
                    throw new Exception(Marshal.GetLastWin32Error().ToString());
                }
                return hFile;
            }
    
    
            /// <summary>
            /// Get cluster usage for a device
            /// </summary>
            /// <param name="DeviceName">use "c:"</param>
            /// <returns>a bitarray for each cluster</returns>
            static public BitArray GetVolumeMap(string DeviceName)
            {
                IntPtr pAlloc = IntPtr.Zero;
                IntPtr hDevice = IntPtr.Zero;
    
                try
                {
                    hDevice = OpenVolume(DeviceName);
    
                    Int64 i64 = 0;
    
                    GCHandle handle = GCHandle.Alloc(i64, GCHandleType.Pinned);
                    IntPtr p = handle.AddrOfPinnedObject();
    
                    // alloc off more than enough for my machine
                    // 64 megs == 67108864 bytes == 536870912 bits == cluster count
                    // NTFS 4k clusters == 2147483648 k of storage == 2097152 megs == 2048 gig disk storage
                    uint q = 1024 * 1024 * 64; // 1024 bytes == 1k * 1024 == 1 meg * 64 == 64 megs
    
                    uint size = 0;
                    pAlloc = Marshal.AllocHGlobal((int)q);
                    IntPtr pDest = pAlloc;
    
                    bool fResult = DeviceIoControl(
                        hDevice,
                        FSConstants.FSCTL_GET_VOLUME_BITMAP,
                        p,
                        (uint)Marshal.SizeOf(i64),
                        pDest,
                        q,
                        ref size,
                        IntPtr.Zero);
    
                    if (!fResult)
                    {
                        throw new Exception(Marshal.GetLastWin32Error().ToString());
                    }
                    handle.Free();
    
                    /*
                    object returned was...
              typedef struct
              {
               LARGE_INTEGER StartingLcn;
               LARGE_INTEGER BitmapSize;
               BYTE Buffer[1];
              } VOLUME_BITMAP_BUFFER, *PVOLUME_BITMAP_BUFFER;
                    */
                    Int64 StartingLcn = (Int64)Marshal.PtrToStructure(pDest, typeof(Int64));
    
                    Debug.Assert(StartingLcn == 0);
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    Int64 BitmapSize = (Int64)Marshal.PtrToStructure(pDest, typeof(Int64));
    
                    Int32 byteSize = (int)(BitmapSize / 8);
                    byteSize++; // round up - even with no remainder
    
                    IntPtr BitmapBegin = (IntPtr)((Int64)pDest + 8);
    
                    byte[] byteArr = new byte[byteSize];
    
                    Marshal.Copy(BitmapBegin, byteArr, 0, (Int32)byteSize);
    
                    BitArray retVal = new BitArray(byteArr);
                    retVal.Length = (int)BitmapSize; // truncate to exact cluster count
                    return retVal;
                }
                finally
                {
                    CloseHandle(hDevice);
                    hDevice = IntPtr.Zero;
    
                    Marshal.FreeHGlobal(pAlloc);
                    pAlloc = IntPtr.Zero;
                }
            }
    
            /// <summary>
            /// returns a 2*number of extents array -
            /// the vcn and the lcn as pairs
            /// </summary>
            /// <param name="path">file to get the map for ex: "c:\windows\explorer.exe" </param>
            /// <returns>An array of [virtual cluster, physical cluster]</returns>
            static public Array GetFileMap(string path)
            {
                IntPtr hFile = IntPtr.Zero;
                IntPtr pAlloc = IntPtr.Zero;
    
                try
                {
                    hFile = OpenFile(path);
    
                    Int64 i64 = 0;
    
                    GCHandle handle = GCHandle.Alloc(i64, GCHandleType.Pinned);
                    IntPtr p = handle.AddrOfPinnedObject();
    
                    uint q = 1024 * 1024 * 64; // 1024 bytes == 1k * 1024 == 1 meg * 64 == 64 megs
    
                    uint size = 0;
                    pAlloc = Marshal.AllocHGlobal((int)q);
                    IntPtr pDest = pAlloc;
                    bool fResult = DeviceIoControl(
                        hFile,
                        FSConstants.FSCTL_GET_RETRIEVAL_POINTERS,
                        p,
                        (uint)Marshal.SizeOf(i64),
                        pDest,
                        q,
                        ref size,
                        IntPtr.Zero);
    
                    if (!fResult)
                    {
                        throw new Exception(Marshal.GetLastWin32Error().ToString());
                    }
    
                    handle.Free();
    
                    /*
                    returned back one of...
         typedef struct RETRIEVAL_POINTERS_BUFFER { 
         DWORD ExtentCount; 
         LARGE_INTEGER StartingVcn; 
         struct {
             LARGE_INTEGER NextVcn;
          LARGE_INTEGER Lcn;
            } Extents[1];
         } RETRIEVAL_POINTERS_BUFFER, *PRETRIEVAL_POINTERS_BUFFER;
        */
    
                    Int32 ExtentCount = (Int32)Marshal.PtrToStructure(pDest, typeof(Int32));
    
                    pDest = (IntPtr)((Int64)pDest + 4);
    
                    Int64 StartingVcn = (Int64)Marshal.PtrToStructure(pDest, typeof(Int64));
    
                    Debug.Assert(StartingVcn == 0);
    
                    pDest = (IntPtr)((Int64)pDest + 8);
    
                    // now pDest points at an array of pairs of Int64s.
    
                    Array retVal = Array.CreateInstance(typeof(Int64), new int[2] { ExtentCount, 2 });
    
                    for (int i = 0; i < ExtentCount; i++)
                    {
                        for (int j = 0; j < 2; j++)
                        {
                            Int64 v = (Int64)Marshal.PtrToStructure(pDest, typeof(Int64));
                            retVal.SetValue(v, new int[2] { i, j });
                            pDest = (IntPtr)((Int64)pDest + 8);
                        }
                    }
    
                    return retVal;
                }
                finally
                {
                    CloseHandle(hFile);
                    hFile = IntPtr.Zero;
    
                    Marshal.FreeHGlobal(pAlloc);
                    pAlloc = IntPtr.Zero;
                }
            }
    
            /// <summary>
            /// input structure for use in MoveFile
            /// </summary>
            private struct MoveFileData
            {
                public IntPtr hFile;
                public Int64 StartingVCN;
                public Int64 StartingLCN;
                public Int32 ClusterCount;
            }
    
            /// <summary>
            /// move a virtual cluster for a file to a logical cluster on disk, repeat for count clusters
            /// </summary>
            /// <param name="deviceName">device to move on"c:"</param>
            /// <param name="path">file to muck with "c:\windows\explorer.exe"</param>
            /// <param name="VCN">cluster number in file to move</param>
            /// <param name="LCN">cluster on disk to move to</param>
            /// <param name="count">for how many clusters</param>
            static public void MoveFile(string deviceName, string path, Int64 VCN, Int64 LCN, Int32 count)
            {
                IntPtr hVol = IntPtr.Zero;
                IntPtr hFile = IntPtr.Zero;
                try
                {
                    hVol = OpenVolume(deviceName);
    
                    hFile = OpenFile(path);
    
    
                    MoveFileData mfd = new MoveFileData();
                    mfd.hFile = hFile;
                    mfd.StartingVCN = VCN;
                    mfd.StartingLCN = LCN;
                    mfd.ClusterCount = count;
    
                    GCHandle handle = GCHandle.Alloc(mfd, GCHandleType.Pinned);
                    IntPtr p = handle.AddrOfPinnedObject();
                    uint bufSize = (uint)Marshal.SizeOf(mfd);
                    uint size = 0;
    
                    bool fResult = DeviceIoControl(
                        hVol,
                        FSConstants.FSCTL_MOVE_FILE,
                        p,
                        bufSize,
                        IntPtr.Zero, // no output data from this FSCTL
                        0,
                        ref size,
                        IntPtr.Zero);
    
                    handle.Free();
    
                    if (!fResult)
                    {
                        throw new Exception(Marshal.GetLastWin32Error().ToString());
                    }
                }
                finally
                {
                    CloseHandle(hVol);
                    CloseHandle(hFile);
                }
            }
        }
    
    
        /// <summary>
        /// constants lifted from winioctl.h from platform sdk
        /// </summary>
        internal class FSConstants
        {
            const uint FILE_DEVICE_FILE_SYSTEM = 0x00000009;
    
            const uint METHOD_NEITHER = 3;
            const uint METHOD_BUFFERED = 0;
    
            const uint FILE_ANY_ACCESS = 0;
            const uint FILE_SPECIAL_ACCESS = FILE_ANY_ACCESS;
    
            public static uint FSCTL_GET_VOLUME_BITMAP = CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 27, METHOD_NEITHER, FILE_ANY_ACCESS);
            public static uint FSCTL_GET_RETRIEVAL_POINTERS = CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 28, METHOD_NEITHER, FILE_ANY_ACCESS);
            public static uint FSCTL_MOVE_FILE = CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 29, METHOD_BUFFERED, FILE_SPECIAL_ACCESS);
    
            static uint CTL_CODE(uint DeviceType, uint Function, uint Method, uint Access)
            {
                return ((DeviceType) << 16) | ((Access) << 14) | ((Function) << 2) | (Method);
            }
        }
    
    }

    Leave a comment:


  • Mike Doty
    replied
    Me? Not working on defrag.

    Leave a comment:


  • David Roberts
    replied
    Found this which got me to this.

    I think that you have your work cut out Mike.

    Leave a comment:


  • David Roberts
    replied
    > Can I ship it with my app tho?

    Doesn't look like it.

    > I would much rather "roll my own"

    Which is what you indicated in your first post.

    I've just run Dependency Walker and, boy, defrage.exe depends upon a heck of a lot of libraries. Your answer is in one of them.

    Leave a comment:


  • Mike Doty
    replied
    Another way without fear of power outages is to XXCOPY one drive to a second formatted drive. This is also pretty quick. These days with
    tons of extra space it doesn't seem like defragging is needed that often. Looks like the professional version is needed to boot.
    Last edited by Mike Doty; 15 Dec 2007, 04:42 PM.

    Leave a comment:


  • Mike Trader
    replied
    Contig has potential short term. Can I ship it with my app tho?


    You may not:

    * work around any technical limitations in the binary versions of the software;
    * reverse engineer, decompile or disassemble the binary versions of the software, except and only to the extent that
    applicable law expressly permits, despite this limitation;
    * make more copies of the software than specified in this agreement or allowed by applicable law, despite this limitation;
    * publish the software for others to copy;
    * rent, lease or lend the software;
    * transfer the software or this agreement to any third party; or
    * use the software for commercial software hosting services.


    "Contig uses standard Windows defragmentation APIs so it won't cause disk corruption, even if you terminate it while its running."
    http://www.microsoft.com/technet/sys...sk/Contig.mspx

    at the risk of
    I would much rather "roll my own"
    Surely this can't be too hard?
    Last edited by Mike Trader; 15 Dec 2007, 02:36 PM.

    Leave a comment:


  • David Roberts
    replied
    If you in the Task Manager, Eddy, you'll see that it operates below normal priority. Any other disk access and DK backs off. In the last 10 hours it has done only 9 minutes work. On average it eliminates a few hundred fragments a day. I had a bit off a tidy up last month and it eliminated 2500 fragments, when it got the chance. It is one job less we have to do.

    Leave a comment:


  • Eddy Van Esch
    replied
    Originally posted by David Roberts View Post
    I have been using DiskKeeper Pro, which defrags during idle time,
    Davids post reminded me of Diskeeper (played with it some years ago). I have now installed the 30 day trial version and it seems to perform really well ! Certainly better than the default XP defragmenter. It also defragments free disk space and operates in the background.
    I haven't noticed any negative influence on overall system performance (while it is defragging I mean).

    Kind regards
    Last edited by Eddy Van Esch; 12 Dec 2007, 03:58 PM.

    Leave a comment:


  • Mike Doty
    replied
    1) Fragmentation can be eliminated by creating a partiition for the file.
    2) Another method is write blank records to the end of the file then defrag
    will keep the file from becoming fragmented. Using this method just keep track
    of the highest record number written so a new record is updated 1 record past it.
    Using this method records are never inserted, just updated. New blank records
    might be added say every 1000 records.
    Last edited by Mike Doty; 12 Dec 2007, 03:31 PM.

    Leave a comment:


  • David Roberts
    replied
    Whilst reading about the prefetch folder and layout.ini and how the windows defrag is brought in to defrag just layout.ini I looked for a command switch but couldn't find one.

    During the search I came across SysInternals contig.exe.

    SysInternals is a pig to get to sometimes!

    I quickly dropped contig onto C: and tried

    Shell "cmd /c c:\contig.exe -v -a C:\PBWin\CmpSig\HMACWP\Big.txt > c:\frag.txt"

    and c:\frag.txt gave me

    Contig v1.54 - Makes files contiguous
    Copyright (C) 1998-2007 Mark Russinovich
    Sysinternals - www.sysinternals.com

    ------------------------
    Processing C:\PBWin\CmpSig\HMACWP\Big.txt:
    Scanning file...
    File size: 104857600 bytes

    C:\PBWin\CmpSig\HMACWP\Big.txt is in 1 fragment
    ------------------------
    Summary:
    Number of files processed : 1
    Average fragmentation : 1 frags/file

    All we have to do is parse frag.txt and get the filesize and fragmentation.

    Admittedly an olde worlde way of doing things but it works and it is better than a poke in the eye with a sharp stick.
    Last edited by David Roberts; 12 Dec 2007, 02:11 PM.

    Leave a comment:


  • Robert DeBolt
    replied
    Defrag-A-File

    Originally posted by Mike Trader View Post
    I'm just asking for a little help with the code...
    You have indicated that you only want programming advice on determining the percent of defragmentation of a given file.

    I just saw this program listed, so I post this link for all interested parties.

    About the author:
    Mark Sweeney, the author of Top Stats, DiskAction 2, and many other PC Magazine utilities is the vice president of software at Clearview Software International, a company specializing in client/server and Internet technologies. He is also owner of Seamist Software which creates the awarding winning PassCrypt password manager.
    Perhaps, if you contact Mr. Sweeney, you could arrange for consulting with him on how to fulfill your requests.

    Leave a comment:


  • Knuth Konrad
    replied
    Originally posted by Mike Trader View Post
    Jim,
    Knuth,
    >I would strongly suggest to STAY AWAY from defragmentation within your own applications
    I can just copy the flat databse file.
    What for? As Eric and Joe already pointed out, by copying the flat file you still can't assure that the OS writes it back in one piece. It might it write back even more fragmented than before, if, for example, the user hasn't done a defrag for a long time and the remaining free space is scattered across the whole HD.

    While I understand that you're trying to do your users a favor (which I applaud), my feelings are that you create more hassle for them than you solve.

    Take this advise from a systems administrator that manages 200+ machines: Do whatever you think will improve *your* software and *your* (data) files internally (like, what Michael has suggested), but stay away from trying to force/fool the OS doing things. You may never know what kind of (OS) management/maintenance procedures are already in place on that particular machine. Even if it's a "dumb user" who never does a defrag by himself, he might have purchased some "cool tool that improves your machines performance by factor x", which does a defrag from time to time.

    Leave a comment:


  • Mike Trader
    replied
    I'm just asking for a little help with the code...
    Last edited by Mike Trader; 6 Dec 2007, 06:48 AM.

    Leave a comment:


  • Michael Mattias
    replied
    > I really need to just figure out how to determine a files fragmentation

    >>In using several database products that utilize a flat file, the issue of file >>fragmentation crops up in one form or another

    Are you talking about internal fragmentation.. eg, "gaps" in the physical file where no records exist, having been deleted; or Windows fragmentation, where the file is not currently located in contiguous sectors (clusters?)?

    If the former, Count (valid records)*recordsize / Filesize; if the latter, don't bother as previously suggested by several.

    Except in the former case, your figure will be off due to any overhead requirements of the "database product" in use.. such figure more than likely proprietary and therefore unavailable to you, so you may as well ignore it.

    Pray, what is the question to which you believe the answer is some kind of programmatic defragmentation, with which you require some assistance?

    Leave a comment:


  • Mike Trader
    replied
    Jim,
    That would mean re-writing all my database functionality for the this purpose. It is much simpler to just copy the file.

    Knuth,
    >I would strongly suggest to STAY AWAY from defragmentation within your own applications
    I can just copy the flat databse file.

    David,
    Defraggler is cool, but I probably cant ship it and install it with my app. I really need to just figure out how to determine a files fragmentation...

    Leave a comment:


  • David Roberts
    replied
    I have been using DiskKeeper Pro, which defrags during idle time, since February so fragmentation ceased to be an issue then.

    However, I did spot Defraggler recently which allows the defragging of one or more files up to the whole shooting match. Needless to say I have not tried it and it should be noted that it is still beta. It is by the same outfit as Crap Cleaner and is free.

    Leave a comment:

Working...
X