Announcement

Collapse
No announcement yet.

Set Copy Priority

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Paul Purvis
    replied
    Gary, you did not ask for this but you might be interested in the program "mkisofs.exe" and you can get it from the wusuoffline updater in the ususoffline114.zip file that is recent as of now.
    You can find a cmd file that uses the command. I just copied over my windows 7 directory to another directory and created an iso file from about 62500 files totaling about 12.5 gig of file space.
    There was no compression involved and my system is a windows 7 32 bit with 1TB ssd.
    I really think a non ssd for preparing these kind of files is probably best on saving your ssd from wear.
    I took off the joliet parameter due to large file names.
    I cannot give you much information on this but i have a use for making iso files.
    The program was easy on the cpu, memory, and fast too.
    I used the vcdcontroltool to view the files of the created iso but i really need a better iso to drive emulator that is smaller and faster and more friendly to a command line.
    I did not do any testing of extracting the files from the iso and comparing to the original files either.

    Leave a comment:


  • John Petty
    replied
    My experience is it depends on whether it is the size of files or number of files.
    Where file size is the problem then not much you can do about it as it will be limited by the slowest connection link between the two locations being speed of the hardware or network link.
    If it is the number of files then do them in small batches.When Windows copies via a simple "drag and drop" it builds an entire "to do" work list including reversal information if cancelled. With copying (or deleting) thousands of files this uses up massive amounts of temporary memory and time just planning the operation. Simple test, create a folder wit say 100,000 files, they can be small,named so easy to select in groups. Time copying all at once (by file name not folder name) then do again 10,000 at a time and the again 1,000 at a time.

    Leave a comment:


  • Paul Purvis
    replied
    Make it hard to turn ON Windows Defender
    https://www.raymond.cc/blog/how-to-d...nder-in-vista/

    Leave a comment:


  • Paul Dixon
    replied
    Compress the files.
    The CPU is way faster than the hardware interface so program/thread priority is not relevant as the CPU is not the bottleneck.

    Don't ZIP all the files, then copy them, then unzip them all.
    Instead, do them in batches.
    Zip 1 second's worth of files, start the copy of that zipped file and while that copy is taking place, zip the next second's worth of files.
    During the copy the CPU is almost idle anyway, waiting for the slow interface, so even a single core CPU has plenty of time to compress the next block before the first block is written.


    For copying over a network I use RtlCompressBuffer and RtlDecompressBuffer and it's 5-10x faster than a straight copy of the files although I'm only dealing with files of around 2MB or less which compress to around 200k.
    RTLcompress might not be the best compression technique but it works well enough for me.

    Leave a comment:


  • Mike Doty
    replied
    Make it hard to turn off Windows Defender
    https://docs.microsoft.com/en-us/win...nder-antivirus

    Leave a comment:


  • Paul Purvis
    replied
    Gary, not what you original posted for but a few things to consider.
    You can check the SMB version with a powershell command in some windows oases.
    You can disable the signatures in SMB packets. Encryption in each SMB packet I believe.
    You might consider a direct Ethernet pic to pic by passing a router or switch. If you don't have a switch but only use a router, add a switch box unless you are using wireless on the equipment.
    You can get ride of netbios over TCP/IP if you connect ip addresses only. That should help increase your packets payload.
    If you are using scoot. Consider braking down the copy files routine to multiple routines that run at the same time.
    As i type even breaking the xcopy into file date ranges might help to produce a multi instance condition.
    Use anothe program or another protocol or both.
    Is usb networkable also.


    https://www.quora.com/What-is-the-fa...etween-two-PCs

    https://ipmsg.org/tools/fastcopy.html.en

    Linux uses a different default protocol that windows that i hear is much faster but never triedmit.
    I set up a FileZilla ftp server recently which was pretty easy to do. FTP iprotocol is suppose to be fast.

    There re is also a small web server program here on the form. For serving files. It sends the whole data contents in one tcp send line. No looping. So if you can get your file names in from your in house server that might be a way to go.

    I have swen linux zip files pretty fat and large outcomes too.
    Who says a zipped file also has to include compression.
    What I do like about sending a single file if possible is for comparing for file mismatch.
    Look at some other zipping functions. You prograblem could zip,files to a USB 3 non SSD device to help keep you local SSD in better condition or place a non SSD in your computer
    You can also off load the task like I do in my offices to another windows machine and you can also use that other machine as a backup server to your main working computers.
    But I do love my Linux servers with RAID 1 and I don't love much equipment anytime.
    One thing I love about it is that I have been able to pull the drive out of one computer and put it in another if the other broke. All I have to do is make an easy menu driven change to setup,the new network card by telling it the new or same IP address and off I go.

    Leave a comment:


  • Michael Mattias
    replied
    I'm doing a drag and drop kind of Copy, using the built in Window copy function from one folder to another. Your suggestions do not apply in this case, do they?
    No, not when you are using a third-party program like this... UNLESS... I think you could create a Shell Extension to run your own (copy) program which DOES allow for his prioritization.

    Code:
      IF GetFileSize (szFile) > %SOME_VALUE  THEN   ' If big file launch copy thread and return immediately
             Alloc Memory size of file NAME and copy name to it.
            THREAD CREATE     CopyMyFile  (integer_handle_obtained above)
      END IF
    Doing this will make it "look like" the copy ran faster because the calling program is ready for more action essentially immediately.

    FWIW..
    When I'm in a hurry, , I'd like to have it copy at one of the higher rates that it seems capable of doing.
    1. "When I am in a hurry" is not a detectable condition
    2. As long as you relying on a third-party copy, if the 3P utility does not support some kind of option, forget it.

    You might experiment with various forms of "effectively copy a file" in your target environment.

    MCM

    Leave a comment:


  • Paul Purvis
    replied
    There is no doubt that "Sneaker Net USB v3" is the fastest but it does require you being there.
    I have stayed up all night many times treansferring files at locations were the only comfort was a hard floor.
    Did that last weekend after been gone there for too long.
    I don't think nothing will beat a good thought out approach.
    Odd now we got so many files and large ones too.
    Even the USB approach with multiple USB storage devices on time is a quick fix but I would stay awawy from flash drives. They can be too slow.

    Leave a comment:


  • Mike Doty
    replied
    I'm frequently copying either large files, or many files, from one drive to another (combinations of SSD, Hard Drive or Flash drives). Sometimes I'd like it to finish quickly, sometimes I don't care.
    For USB the performance may be doubled using USB 3.1 rev 2.
    I purchased and use these items.

    Here is a card:
    https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

    Here is a standalone duplicator and 2 drive bay rated at 10gbps
    https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

    Here is an external Samsung T5 USB 3.1 rev 2 drive:
    It does not fit in a bay, it uses its own cable(s) and doesn't need a power supply.

    https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1



    To be sure I can be up an running in a minute to another bootable drive:

    Clone a smaller hard drive to a larger drive (free), but use paid version
    https://www.backup-utility.com/

    If doing a daily clone to a drive as large as source using rapid delta clone (to only clone what has changed, paid version.)
    Clone is about 3-minutes after initial clone.
    https://www.macrium.com/products/home

    Note: Using the Samsung T5 external USB 3.1 rev 2 drive and Windows to Go
    it is possible to run everything on one computer and later plug in and boot on another computer.
    It is pretty cool seeing everything done on one computer running on another computer without modifying it.
    One could easily work off this drive all day and take it with you and plug into another machine.

    Bottom line, if getting a new machine be sure there is a USB 3.1 rev 2 port.



    To convert a drive to Windows to Go, click on tools and click Windows to Go conversion.
    https://www.easyuefi.com/wintousb/index.html



    Turn off anti-virus from above post using Windows Defender
    This will change directory to current directory so multiple commands in batch file will work if batch file is started from a shortcut.
    cd /d %~dp0

    rem something.bat
    cd /d %~dp0
    powershell set-mppreference -disablerealtimemonitoring 1
    rem xcopy /e/r/y/d
    rem robocopy ...
    pause
    powershell set-mppreference -disablerealtimemonitoring 0

    https://www.tenforums.com/tutorials/...a.html#option5

    I'm done.

    Leave a comment:


  • Mike Doty
    replied
    > But, if something can cut a 20 minutes copy down to a fraction of that, then a minute's worth of action should be a good trade off.
    Get rid of the minute by using PowerShell to automate turning anti-virus off and on.

    Windows Defender turn off anti-virus (on both machines if possible)
    If using another anti-virus will need another approach
    If using drag and drop, click a batch file that disables before beginning and enables after finishing
    Could do this with PowerBASIC and SHELL.

    Paul covered all the switches like /D for speed and /C if you want to continue on error with xcopy
    Sometimes it may be better to run without /C a few times to see what might never be copied because of access violations

    This will change directory to current directory so multiple commands in batch file will work if batch file is started from a shortcut.
    cd /d %~dp0

    rem something.bat
    cd /d %~dp0
    powershell set-mppreference -disablerealtimemonitoring 1
    rem xcopy /e/r/y/d
    rem robocopy ...
    pause
    powershell set-mppreference -disablerealtimemonitoring 0

    https://www.tenforums.com/tutorials/...a.html#option5

    Leave a comment:


  • Bob Carver
    replied
    No. CPU priority has no bearing on IO priority.
    Here's Microsoft's document about it - "Recommendations for Application, Driver and Device Developers for Supporting I/O Prioritization in Windows Vista"
    https://download.microsoft.com/downl...Priorityio.doc

    Leave a comment:


  • Gary Beene
    replied
    MCM,
    I'm doing a drag and drop kind of Copy, using the built in Window copy function from one folder to another. Your suggestions do not apply in this case, do they?

    Leave a comment:


  • Michael Mattias
    replied
    See the WinApi functions SetThreadPriority ( PB 'THREAD SET PRIORITY' ) and SetProcessPriority (No PB equivalant).

    Run your copy in a separate thread of execution and raise/lower the priority as desired
    OR
    Induce a delay in the copy sequence after each "n" files or bytes copied. . (See SHFileOperation() winapi function.. might prove really handy).

    No, you cannot control the O/S utility 'COPY" like this. (Didn't I just address this same topic in another thread here? )

    MCM

    Leave a comment:


  • Gary Beene
    replied
    Howdy, Paul!
    In my experience, for a large quantity of files, zipping then copying then unzipping doesn't save any/much time. Zipping/unzipping 50,000 files is not quick! I'll have to go look at XCopy to remember what all of the settings do!

    And Raymond,
    Yep, I use Beyond Compare. But I'm often dealing with multiple PCs, some of which do not have the tool installed.

    Leave a comment:


  • Raymond Leech
    replied
    Gary, Have you thought of one of the syncronization tools? We use Syncovery. It is capable of multi-threaded copy, file change detection, and changed-block copies for large files. On large files with low updates, it saves a ton of time / bandwidth copying only the changed blocks. It has two tons of options for almost any archive, backup, replication option you could think of. Maybe some tool like that would be an solution to your speed problems.

    Leave a comment:


  • Paul Purvis
    replied
    Gary, sometimes I need an inflatable mattress at a location where I might be copying files. I understand the time spent.
    If the files are many, you might consider zipping them into one file on the source.and if you are copying over the same set of files with only some new ones or a few files have new dates. You might consider a command line like this one where we copy over files that are mostly letters, etc from a server to a workstation on each new login.
    I copy over the same set of documents to each workstation and use the workstations as a backup to the server.

    MD M:\BACKUP\H
    "XCOPY H:\*.* M:\BACKUP\H /S /V /E /H /K /C /R /Y /D /Q"
    H: drive is the server drive letter and M: is a local workstation partition I have on each workstation.
    After the initial copying over the full drive off the server, the next time it runs on login is very fast.
    Periodically, when i want, I rename m:\backup\h to something like m:\backup\H-20180510.
    You might want to do as i do also. Have a dedicated machine for handling backups off a linux server that is using RAID-1.
    I am starting to run the dedicated backup machine headless now too, just using a VNC server.
    If you turn off the signing of SMB, you might pick up some speed too on those transfers across a LAN.

    Leave a comment:


  • Paul Purvis
    replied
    Gary,. I bought a USB plugin device that allowed for a flash drive. It was for a bigger card that fit into a Cannon Camera. I think the card cost about 500 bucks for a 5 gig card at the time.
    But i will never forgot how a particular USB adapter plug in was that I was using. Much faster than any other that I used for this same wide SD like plugin card.
    I learned. Adapters are not all built the same in properties.

    Leave a comment:


  • Gary Beene
    replied
    I'm finding plenty of links to utilities that are supposed to provide faster copy (TeraCopy, Supercopier, etc.) but in reviews of the utilities, the speed difference between Windows built-in copy and the utilities is often quoted as only about 10%-15%.

    That's an improvement, but doesn't seem like a very significant improvement - not enough for me to get excited about.

    Leave a comment:


  • Gary Beene
    replied
    An opinion I've run across ...

    ... that no matter what the advertisements tell you, your file transfer speeds are limited by your hardware. No software solution, no matter how sophisticated it may be, will increase those speeds
    I don't know if it's true or not. Still looking ...

    Leave a comment:


  • Gary Beene
    replied
    Other options to know about, but not the easy, magic bullet I hoped to find ...

    1. Turn off anti-virus
    2. Use the command lineWindows 10 utility robocopy (multi-threaded mode)

    But, if something can cut a 20 minutes copy down to a fraction of that, then a minute's worth of action should be a good trade off.

    Still looking ...

    Leave a comment:

Working...
X