Announcement

Collapse
No announcement yet.

Set Copy Priority

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Set Copy Priority

    I'm frequently copying either large files, or many files, from one drive to another (combinations of SSD, Hard Drive or Flash drives). Sometimes I'd like it to finish quickly, sometimes I don't care.

    For those cases where time matters, can I increase the priority given to a copy action?

    I can see in the Copy window that sometimes the speed is high, sometimes low, sometimes jumps around. When I'm in a hurry, I'd like to have it copy at one of the higher rates that it seems capable of doing.

    Task Manager says my RAM and CPU usage is very low, so that doesn't seem to be a factor.

    In Task Manager, "Details", I know you can right click on a process and select "Set Priority", but in Task Manager it's not clear which process is the Copy action.

  • #2
    I did run across this for USB speed improvements, but I've not tried it yet. This seems to require that you do the task each time you insert the USB. I was hoping for a one-time setting change, but if the speed improvement is great enough, I could stand doing a per-copy settings change.


    Note: After Changing the Device Policy for better performance, Your Data Transfer Speed Will Increase But Make Sure You always Click on ” Safely Remove Hardware and Eject Media” Before Removing Your Pendrive.

    To change device policy for Better performance, Insert the USB device to your PC, Open this PC Right click on the USB drive, and select properties. Here on USB device properties move to hardware Tab, select your USB drive and click on the properties button.

    You will see a new Dialog Box, Here under general tab, Click on Change Settings button. One more Dialog Box will appear, Here Click on Policy Tab you will see two options in the USB Device Properties box:
    Last edited by Gary Beene; 10 May 2018, 06:47 PM.

    Comment


    • #3
      Other options to know about, but not the easy, magic bullet I hoped to find ...

      1. Turn off anti-virus
      2. Use the command lineWindows 10 utility robocopy (multi-threaded mode)

      But, if something can cut a 20 minutes copy down to a fraction of that, then a minute's worth of action should be a good trade off.

      Still looking ...

      Comment


      • #4
        An opinion I've run across ...

        ... that no matter what the advertisements tell you, your file transfer speeds are limited by your hardware. No software solution, no matter how sophisticated it may be, will increase those speeds
        I don't know if it's true or not. Still looking ...

        Comment


        • #5
          I'm finding plenty of links to utilities that are supposed to provide faster copy (TeraCopy, Supercopier, etc.) but in reviews of the utilities, the speed difference between Windows built-in copy and the utilities is often quoted as only about 10%-15%.

          That's an improvement, but doesn't seem like a very significant improvement - not enough for me to get excited about.

          Comment


          • #6
            Gary,. I bought a USB plugin device that allowed for a flash drive. It was for a bigger card that fit into a Cannon Camera. I think the card cost about 500 bucks for a 5 gig card at the time.
            But i will never forgot how a particular USB adapter plug in was that I was using. Much faster than any other that I used for this same wide SD like plugin card.
            I learned. Adapters are not all built the same in properties.
            p purvis

            Comment


            • #7
              Gary, sometimes I need an inflatable mattress at a location where I might be copying files. I understand the time spent.
              If the files are many, you might consider zipping them into one file on the source.and if you are copying over the same set of files with only some new ones or a few files have new dates. You might consider a command line like this one where we copy over files that are mostly letters, etc from a server to a workstation on each new login.
              I copy over the same set of documents to each workstation and use the workstations as a backup to the server.

              MD M:\BACKUP\H
              "XCOPY H:\*.* M:\BACKUP\H /S /V /E /H /K /C /R /Y /D /Q"
              H: drive is the server drive letter and M: is a local workstation partition I have on each workstation.
              After the initial copying over the full drive off the server, the next time it runs on login is very fast.
              Periodically, when i want, I rename m:\backup\h to something like m:\backup\H-20180510.
              You might want to do as i do also. Have a dedicated machine for handling backups off a linux server that is using RAID-1.
              I am starting to run the dedicated backup machine headless now too, just using a VNC server.
              If you turn off the signing of SMB, you might pick up some speed too on those transfers across a LAN.
              p purvis

              Comment


              • #8
                Gary, Have you thought of one of the syncronization tools? We use Syncovery. It is capable of multi-threaded copy, file change detection, and changed-block copies for large files. On large files with low updates, it saves a ton of time / bandwidth copying only the changed blocks. It has two tons of options for almost any archive, backup, replication option you could think of. Maybe some tool like that would be an solution to your speed problems.

                Comment


                • #9
                  Howdy, Paul!
                  In my experience, for a large quantity of files, zipping then copying then unzipping doesn't save any/much time. Zipping/unzipping 50,000 files is not quick! I'll have to go look at XCopy to remember what all of the settings do!

                  And Raymond,
                  Yep, I use Beyond Compare. But I'm often dealing with multiple PCs, some of which do not have the tool installed.

                  Comment


                  • #10
                    See the WinApi functions SetThreadPriority ( PB 'THREAD SET PRIORITY' ) and SetProcessPriority (No PB equivalant).

                    Run your copy in a separate thread of execution and raise/lower the priority as desired
                    OR
                    Induce a delay in the copy sequence after each "n" files or bytes copied. . (See SHFileOperation() winapi function.. might prove really handy).

                    No, you cannot control the O/S utility 'COPY" like this. (Didn't I just address this same topic in another thread here? )

                    MCM

                    Comment


                    • #11
                      MCM,
                      I'm doing a drag and drop kind of Copy, using the built in Window copy function from one folder to another. Your suggestions do not apply in this case, do they?

                      Comment


                      • #12
                        No. CPU priority has no bearing on IO priority.
                        Here's Microsoft's document about it - "Recommendations for Application, Driver and Device Developers for Supporting I/O Prioritization in Windows Vista"
                        https://download.microsoft.com/downl...Priorityio.doc

                        Comment


                        • #13
                          > But, if something can cut a 20 minutes copy down to a fraction of that, then a minute's worth of action should be a good trade off.
                          Get rid of the minute by using PowerShell to automate turning anti-virus off and on.

                          Windows Defender turn off anti-virus (on both machines if possible)
                          If using another anti-virus will need another approach
                          If using drag and drop, click a batch file that disables before beginning and enables after finishing
                          Could do this with PowerBASIC and SHELL.

                          Paul covered all the switches like /D for speed and /C if you want to continue on error with xcopy
                          Sometimes it may be better to run without /C a few times to see what might never be copied because of access violations

                          This will change directory to current directory so multiple commands in batch file will work if batch file is started from a shortcut.
                          cd /d %~dp0

                          rem something.bat
                          cd /d %~dp0
                          powershell set-mppreference -disablerealtimemonitoring 1
                          rem xcopy /e/r/y/d
                          rem robocopy ...
                          pause
                          powershell set-mppreference -disablerealtimemonitoring 0

                          https://www.tenforums.com/tutorials/...a.html#option5
                          https://www.tesla.com/roadster

                          Comment


                          • #14
                            I'm frequently copying either large files, or many files, from one drive to another (combinations of SSD, Hard Drive or Flash drives). Sometimes I'd like it to finish quickly, sometimes I don't care.
                            For USB the performance may be doubled using USB 3.1 rev 2.
                            I purchased and use these items.

                            Here is a card:
                            https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

                            Here is a standalone duplicator and 2 drive bay rated at 10gbps
                            https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

                            Here is an external Samsung T5 USB 3.1 rev 2 drive:
                            It does not fit in a bay, it uses its own cable(s) and doesn't need a power supply.

                            https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1



                            To be sure I can be up an running in a minute to another bootable drive:

                            Clone a smaller hard drive to a larger drive (free), but use paid version
                            https://www.backup-utility.com/

                            If doing a daily clone to a drive as large as source using rapid delta clone (to only clone what has changed, paid version.)
                            Clone is about 3-minutes after initial clone.
                            https://www.macrium.com/products/home

                            Note: Using the Samsung T5 external USB 3.1 rev 2 drive and Windows to Go
                            it is possible to run everything on one computer and later plug in and boot on another computer.
                            It is pretty cool seeing everything done on one computer running on another computer without modifying it.
                            One could easily work off this drive all day and take it with you and plug into another machine.

                            Bottom line, if getting a new machine be sure there is a USB 3.1 rev 2 port.



                            To convert a drive to Windows to Go, click on tools and click Windows to Go conversion.
                            https://www.easyuefi.com/wintousb/index.html



                            Turn off anti-virus from above post using Windows Defender
                            This will change directory to current directory so multiple commands in batch file will work if batch file is started from a shortcut.
                            cd /d %~dp0

                            rem something.bat
                            cd /d %~dp0
                            powershell set-mppreference -disablerealtimemonitoring 1
                            rem xcopy /e/r/y/d
                            rem robocopy ...
                            pause
                            powershell set-mppreference -disablerealtimemonitoring 0

                            https://www.tenforums.com/tutorials/...a.html#option5

                            I'm done.
                            https://www.tesla.com/roadster

                            Comment


                            • #15
                              There is no doubt that "Sneaker Net USB v3" is the fastest but it does require you being there.
                              I have stayed up all night many times treansferring files at locations were the only comfort was a hard floor.
                              Did that last weekend after been gone there for too long.
                              I don't think nothing will beat a good thought out approach.
                              Odd now we got so many files and large ones too.
                              Even the USB approach with multiple USB storage devices on time is a quick fix but I would stay awawy from flash drives. They can be too slow.
                              p purvis

                              Comment


                              • #16
                                I'm doing a drag and drop kind of Copy, using the built in Window copy function from one folder to another. Your suggestions do not apply in this case, do they?
                                No, not when you are using a third-party program like this... UNLESS... I think you could create a Shell Extension to run your own (copy) program which DOES allow for his prioritization.

                                Code:
                                  IF GetFileSize (szFile) > %SOME_VALUE  THEN   ' If big file launch copy thread and return immediately
                                         Alloc Memory size of file NAME and copy name to it.
                                        THREAD CREATE     CopyMyFile  (integer_handle_obtained above)
                                  END IF
                                Doing this will make it "look like" the copy ran faster because the calling program is ready for more action essentially immediately.

                                FWIW..
                                When I'm in a hurry, , I'd like to have it copy at one of the higher rates that it seems capable of doing.
                                1. "When I am in a hurry" is not a detectable condition
                                2. As long as you relying on a third-party copy, if the 3P utility does not support some kind of option, forget it.

                                You might experiment with various forms of "effectively copy a file" in your target environment.

                                MCM

                                Comment


                                • #17
                                  Gary, not what you original posted for but a few things to consider.
                                  You can check the SMB version with a powershell command in some windows oases.
                                  You can disable the signatures in SMB packets. Encryption in each SMB packet I believe.
                                  You might consider a direct Ethernet pic to pic by passing a router or switch. If you don't have a switch but only use a router, add a switch box unless you are using wireless on the equipment.
                                  You can get ride of netbios over TCP/IP if you connect ip addresses only. That should help increase your packets payload.
                                  If you are using scoot. Consider braking down the copy files routine to multiple routines that run at the same time.
                                  As i type even breaking the xcopy into file date ranges might help to produce a multi instance condition.
                                  Use anothe program or another protocol or both.
                                  Is usb networkable also.


                                  https://www.quora.com/What-is-the-fa...etween-two-PCs

                                  https://ipmsg.org/tools/fastcopy.html.en

                                  Linux uses a different default protocol that windows that i hear is much faster but never triedmit.
                                  I set up a FileZilla ftp server recently which was pretty easy to do. FTP iprotocol is suppose to be fast.

                                  There re is also a small web server program here on the form. For serving files. It sends the whole data contents in one tcp send line. No looping. So if you can get your file names in from your in house server that might be a way to go.

                                  I have swen linux zip files pretty fat and large outcomes too.
                                  Who says a zipped file also has to include compression.
                                  What I do like about sending a single file if possible is for comparing for file mismatch.
                                  Look at some other zipping functions. You prograblem could zip,files to a USB 3 non SSD device to help keep you local SSD in better condition or place a non SSD in your computer
                                  You can also off load the task like I do in my offices to another windows machine and you can also use that other machine as a backup server to your main working computers.
                                  But I do love my Linux servers with RAID 1 and I don't love much equipment anytime.
                                  One thing I love about it is that I have been able to pull the drive out of one computer and put it in another if the other broke. All I have to do is make an easy menu driven change to setup,the new network card by telling it the new or same IP address and off I go.
                                  p purvis

                                  Comment


                                  • #18
                                    Make it hard to turn off Windows Defender
                                    https://docs.microsoft.com/en-us/win...nder-antivirus
                                    https://www.tesla.com/roadster

                                    Comment


                                    • #19
                                      Compress the files.
                                      The CPU is way faster than the hardware interface so program/thread priority is not relevant as the CPU is not the bottleneck.

                                      Don't ZIP all the files, then copy them, then unzip them all.
                                      Instead, do them in batches.
                                      Zip 1 second's worth of files, start the copy of that zipped file and while that copy is taking place, zip the next second's worth of files.
                                      During the copy the CPU is almost idle anyway, waiting for the slow interface, so even a single core CPU has plenty of time to compress the next block before the first block is written.


                                      For copying over a network I use RtlCompressBuffer and RtlDecompressBuffer and it's 5-10x faster than a straight copy of the files although I'm only dealing with files of around 2MB or less which compress to around 200k.
                                      RTLcompress might not be the best compression technique but it works well enough for me.

                                      Comment


                                      • #20
                                        Make it hard to turn ON Windows Defender
                                        https://www.raymond.cc/blog/how-to-d...nder-in-vista/
                                        p purvis

                                        Comment

                                        Working...
                                        X