Announcement

Collapse
No announcement yet.

3 to 4 gig File Copy

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • 3 to 4 gig File Copy

    My MRP Database is a flatfile database system which has 16 parts.
    Each part ranges from 100 megs to 800 megs in size.

    Last night our main MRP server died due to hardware issues.
    I could of had Business up and running within a few minutes on
    a new server if I had the database copied to a Sandbox Server.

    I'm going to start coping the Database, but it takes SO long to
    copy over the network which we have a 100 bit connection.

    I'm wanting to make a program that will only copy the changes
    of each file, in turn reducing the amount of network bandwidth
    needed.

    Any ideas?

    ------------------
    -Greg
    -Greg
    [email protected]
    MCP,MCSA,MCSE,MCSD

  • #2
    Any way you go over even a 100 network, it's gonna take time.
    What I would consider is getting a multi-speed CD-RW drive and
    backup / restore from that. You can then keep the network up
    and running and take all the time you need to copy the data.



    ------------------
    There are no atheists in a fox hole or the morning of a math test.
    If my flag offends you, I'll help you pack.

    Comment


    • #3
      Mel,

      There is no way I can do that as an automatic procedure, and
      with 3 to 4 gigs of information, that is a lot of cds to deal
      with.



      ------------------
      -Greg
      -Greg
      [email protected]
      MCP,MCSA,MCSE,MCSD

      Comment


      • #4
        The process I use in a similar situation is that I zip it then copy it.
        On a flat file you will recieve good compression. However, if this
        is something you plan on doing daily or more often, consider replication.



        ------------------
        Thanks,

        Bradley Callis
        Thanks,

        Bradley Callis

        Comment


        • #5
          One of the techniques I use is to configure 2.5 inch 20 GB disks
          into removable cartridges and install the receptacle trays in
          each server, my laptop and desktop. I usually configure this
          disk as D or E depending on my local configuration. I can then
          do file copies that are very fast. This is a very efficient
          way to move large amounts of data and even use them in a
          rotating manner as backup media.

          The cost of these disks is about $200, and the trays are about
          $15 Each.

          -Tony

          ------------------

          Comment


          • #6
            Tony,

            That won't work, the power is on the server 24/7.

            ------------------
            -Greg
            -Greg
            [email protected]
            MCP,MCSA,MCSE,MCSD

            Comment


            • #7
              The Filesize of each database file is always the same, even an
              empty database will be 3 or 4 gigs depending on the Page Size I
              specify. I'm sure if I open a before and then an after and
              compare the two files and return the differences and transmit
              that over the network, it will be faster. I don't know... I'll
              have to look into it.

              Thanks

              ------------------
              -Greg
              -Greg
              [email protected]
              MCP,MCSA,MCSE,MCSD

              Comment


              • #8
                Greg --
                Do not compare files directly (to avoid reading through network).
                Divide each file to chunks, let's say 1Mb, calculate CRC and copy a part, if necessary only.


                ------------------
                E-MAIL: [email protected]

                Comment


                • #9
                  I also have another idea, my MRP manufacturer offers a transaction
                  recovery method which uses transaction logs, which are usually
                  less then 50 or 60 megs. I could copy those over and then do
                  a transaction restore on the old database to make it current.

                  I will look into this further

                  ------------------
                  -Greg
                  -Greg
                  [email protected]
                  MCP,MCSA,MCSE,MCSD

                  Comment


                  • #10
                    Hi Greg,
                    There are some Drives and shells that are "Hot Swappable".
                    I use the IBM 2.5” Drives.

                    -Tony



                    ------------------

                    Comment


                    • #11
                      The proper way to do this would be to install an additional NIC in each of the machines.
                      Subnet that out so that it does not cross paths with your local network.
                      While it does take some overhead on the machines, you won't have any overhead on your network.

                      It's always a suggested method of having a backup network for this reason, most major dot-bomb's do this, most major network architects do this as well, *if* it's physically possible....


                      If it's not, your idea MAY work but only if you know what has changed and that may not be a good idea becaues of the uncertainties.

                      But doing a file transfer, through a solitary network, is likely to have the least load on any of the machines.


                      Scott

                      ------------------
                      Scott
                      Scott Turchin
                      MCSE, MCP+I
                      http://www.tngbbs.com
                      ----------------------
                      True Karate-do is this: that in daily life, one's mind and body be trained and developed in a spirit of humility; and that in critical times, one be devoted utterly to the cause of justice. -Gichin Funakoshi

                      Comment


                      • #12
                        PS, fibre channel cards are getting cheaper and cheaper too, maybe expensive in the beginning but it will read/write faster than your hard drive will.
                        I'd be willing to bet a 100 meg file would transfer in under 60 seconds...
                        I'm not sure, if they are gigabyte NIC's maybe...again maybe initially expensive to do this but it's what I would do to make backups daily..
                        THen you could write a PB app or use Win2k backup to copy to the other drive..

                        Scott

                        ------------------
                        Scott
                        Scott Turchin
                        MCSE, MCP+I
                        http://www.tngbbs.com
                        ----------------------
                        True Karate-do is this: that in daily life, one's mind and body be trained and developed in a spirit of humility; and that in critical times, one be devoted utterly to the cause of justice. -Gichin Funakoshi

                        Comment


                        • #13
                          Greg;

                          Some stuff off the top of my head;

                          You didn't mention what platform this server is. I can pretty much guarantee, though, that just about any kind of file copy process will likely be too slow to suit, especially if it's Win98 or NT; file I/O is not their strong suit.

                          What about a good old tape backup? 12/24GB DDS drive are pretty cheap, especially if the company's data is at stake. If you'rereally paranoid, schedule a backup every two hours.

                          Disk mirroring may also be an option, but that assumes you've got a good 'spare' to drop the disks into if the server goes belly-up.

                          Do these files shrink with use? If all new data is simply appended to end of the exiting file, you could perhaps open it in binary mode and only copy off the "new" parts.




                          ------------------
                          [email protected]
                          http://www.northnet.org/bdurland
                          Real programmers use a magnetized needle and a steady hand

                          Comment


                          • #14
                            Tony,

                            I don't like the idea of having to manually pull a hard drive,
                            many issues can arrise

                            Scott,

                            I like that idea, I've never installed a second nic before in an
                            nt box. My subnet I currently use is 192.168.0.1/24 so if I
                            put my sandbox server and my production server on a new subnet
                            such as 192.168.1.1/24 would that work? If I get everything set
                            up, how do I route certain data through one NIC and other
                            data through the other one?

                            Bud,

                            I'm sorry I must of missed something when I brought this topic
                            up. This isn't going to be an absolute backup method, it is just
                            a method I'm using to get us back up and available in the least
                            amount of time. We currently use 15/30GB DLT Tape to backup
                            our MRP and Datafiles everynight, but what happened was the
                            production server died, it doesn't even turn on. So the backup
                            never ran last night (the tape drive is in the production server)
                            and all the MRP files are on the production server. It appears
                            that it is a bad power supply and we have one on order from
                            compaq.



                            ------------------
                            -Greg
                            -Greg
                            [email protected]
                            MCP,MCSA,MCSE,MCSD

                            Comment


                            • #15
                              Greg, for a separate subnet with 100mbit nics,
                              you would just add the cards, assign the ip's to them
                              just like you stated in your message.

                              To move data between the dedicate subnet, just use the IP's for that subnet.
                              copy \\192.168.0.1\c$\Backups\MyFile.dat \\192.168.1.1\c$\backups

                              That copy program I sent you should work fine with a dedicated subnet.

                              Ron

                              Comment


                              • #16
                                http://www.rtpatch.com

                                ------------------
                                How long is an idea?

                                Comment


                                • #17
                                  If you are using 192.168.1.1/24 I'd suggest perhaps going to something different with the two nics, that way it cannot find the machine otherwise.

                                  Ie, Server: 192.168.1.1 *and* 10.10.10.1 (If not on the net this is fine).
                                  Sandbox 10.10.10.2 and 192.168.10.2 or whatever it may be.

                                  Now, as mentioned you may have to use the ip address in the UNC name, 10.10.10.1\Backups\blah blah....

                                  Or, if you are comfortable with routing tables you could do something with them but I suggest the UNC name...

                                  Or, if a piece of software does the work then it may have the option to do it by IP address....
                                  It's by far the cheapest and most reliable method for you to do this....

                                  And, if you are not us ing Disk mirroring, or raid 5, you may wish to consider that in the future, a Raid5 system would have had ya back up in 2 hours or so depending on the SCSI card...

                                  I'm learning more than I ever wanted to about SCSI but it's a good thing, in fact I may go SCSI at home when I get my new house...

                                  Scott

                                  ------------------
                                  Scott
                                  Scott Turchin
                                  MCSE, MCP+I
                                  http://www.tngbbs.com
                                  ----------------------
                                  True Karate-do is this: that in daily life, one's mind and body be trained and developed in a spirit of humility; and that in critical times, one be devoted utterly to the cause of justice. -Gichin Funakoshi

                                  Comment


                                  • #18
                                    Scott,

                                    Thanks for that information... We use raid 5 scsi at work. It
                                    works very well, but the problem with this server is that our
                                    compaq 750watt power supply went out and we had to order it from
                                    compaq. Lets just say we now have 2 redunant power supplies on
                                    that server.

                                    As far as home use, go with the Promise raid 5 controller. They
                                    even have hot swappable IDE chassis. The only manufacturer to
                                    use ATA/IDE drives as a raid option. http://www.promise.com
                                    I think...

                                    ------------------
                                    -Greg
                                    -Greg
                                    [email protected]
                                    MCP,MCSA,MCSE,MCSD

                                    Comment


                                    • #19
                                      I'll soon be using LapLink Gold to keep two 144GB file
                                      systems synced. PCsync from LapLink may also work for you.

                                      Comment

                                      Working...
                                      X