3 to 4 gig File Copy

[Gregery D Engle]

My MRP Database is a flatfile database system which has 16 parts.
Each part ranges from 100 megs to 800 megs in size.

Last night our main MRP server died due to hardware issues.
I could of had Business up and running within a few minutes on
a new server if I had the database copied to a Sandbox Server.

I'm going to start coping the Database, but it takes SO long to
copy over the network which we have a 100 bit connection.

I'm wanting to make a program that will only copy the changes
of each file, in turn reducing the amount of network bandwidth
needed.

Any ideas?

------------------
-Greg

[Mel Bishop]

Any way you go over even a 100 network, it's gonna take time.
What I would consider is getting a multi-speed CD-RW drive and
backup / restore from that. You can then keep the network up
and running and take all the time you need to copy the data.



------------------

[Gregery D Engle]

Mel,

There is no way I can do that as an automatic procedure, and
with 3 to 4 gigs of information, that is a lot of cds to deal
with.



------------------
-Greg

[Bradley Callis]

The process I use in a similar situation is that I zip it then copy it.
On a flat file you will recieve good compression. However, if this
is something you plan on doing daily or more often, consider replication.



------------------
Thanks,

Bradley Callis

[Tony Altwies]

One of the techniques I use is to configure 2.5 inch 20 GB disks
into removable cartridges and install the receptacle trays in
each server, my laptop and desktop. I usually configure this
disk as D or E depending on my local configuration. I can then
do file copies that are very fast. This is a very efficient
way to move large amounts of data and even use them in a
rotating manner as backup media.

The cost of these disks is about $200, and the trays are about
$15 Each.

-Tony

------------------

[Gregery D Engle]

Tony,

That won't work, the power is on the server 24/7.

------------------
-Greg

[Gregery D Engle]

The Filesize of each database file is always the same, even an
empty database will be 3 or 4 gigs depending on the Page Size I
specify. I'm sure if I open a before and then an after and
compare the two files and return the differences and transmit
that over the network, it will be faster. I don't know... I'll
have to look into it.

Thanks

------------------
-Greg

[Semen Matusovski]

Greg --
Do not compare files directly (to avoid reading through network).
Divide each file to chunks, let's say 1Mb, calculate CRC and copy a part, if necessary only.


------------------
E-MAIL: [email protected]

[Gregery D Engle]

I also have another idea, my MRP manufacturer offers a transaction
recovery method which uses transaction logs, which are usually
less then 50 or 60 megs. I could copy those over and then do
a transaction restore on the old database to make it current.

I will look into this further

------------------
-Greg

[Tony Altwies]

Hi Greg,
There are some Drives and shells that are "Hot Swappable".
I use the IBM 2.5” Drives.

-Tony



------------------

[Scott Turchin]

The proper way to do this would be to install an additional NIC in each of the machines.
Subnet that out so that it does not cross paths with your local network.
While it does take some overhead on the machines, you won't have any overhead on your network.

It's always a suggested method of having a backup network for this reason, most major dot-bomb's do this, most major network architects do this as well, *if* it's physically possible....


If it's not, your idea MAY work but only if you know what has changed and that may not be a good idea becaues of the uncertainties.

But doing a file transfer, through a solitary network, is likely to have the least load on any of the machines.


Scott

------------------
Scott

[Scott Turchin]

PS, fibre channel cards are getting cheaper and cheaper too, maybe expensive in the beginning but it will read/write faster than your hard drive will.
I'd be willing to bet a 100 meg file would transfer in under 60 seconds...
I'm not sure, if they are gigabyte NIC's maybe...again maybe initially expensive to do this but it's what I would do to make backups daily..
THen you could write a PB app or use Win2k backup to copy to the other drive..

Scott

------------------
Scott

[Bud Durland]

Greg;

Some stuff off the top of my head;

You didn't mention what platform this server is. I can pretty much guarantee, though, that just about any kind of file copy process will likely be too slow to suit, especially if it's Win98 or NT; file I/O is not their strong suit.

What about a good old tape backup? 12/24GB DDS drive are pretty cheap, especially if the company's data is at stake. If you'rereally paranoid, schedule a backup every two hours.

Disk mirroring may also be an option, but that assumes you've got a good 'spare' to drop the disks into if the server goes belly-up.

Do these files shrink with use? If all new data is simply appended to end of the exiting file, you could perhaps open it in binary mode and only copy off the "new" parts.




------------------
[email protected]
http://www.northnet.org/bdurland

[Gregery D Engle]

Tony,

I don't like the idea of having to manually pull a hard drive,
many issues can arrise

Scott,

I like that idea, I've never installed a second nic before in an
nt box. My subnet I currently use is 192.168.0.1/24 so if I
put my sandbox server and my production server on a new subnet
such as 192.168.1.1/24 would that work? If I get everything set
up, how do I route certain data through one NIC and other
data through the other one?

Bud,

I'm sorry I must of missed something when I brought this topic
up. This isn't going to be an absolute backup method, it is just
a method I'm using to get us back up and available in the least
amount of time. We currently use 15/30GB DLT Tape to backup
our MRP and Datafiles everynight, but what happened was the
production server died, it doesn't even turn on. So the backup
never ran last night (the tape drive is in the production server)
and all the MRP files are on the production server. It appears
that it is a bad power supply and we have one on order from
compaq.



------------------
-Greg

[Guest]

Greg, for a separate subnet with 100mbit nics,
you would just add the cards, assign the ip's to them
just like you stated in your message.

To move data between the dedicate subnet, just use the IP's for that subnet.
copy \\192.168.0.1\c$\Backups\MyFile.dat \\192.168.1.1\c$\backups

That copy program I sent you should work fine with a dedicated subnet.

Ron

[Mike Doty]

http://www.rtpatch.com

------------------

[Scott Turchin]

If you are using 192.168.1.1/24 I'd suggest perhaps going to something different with the two nics, that way it cannot find the machine otherwise.

Ie, Server: 192.168.1.1 *and* 10.10.10.1 (If not on the net this is fine).
Sandbox 10.10.10.2 and 192.168.10.2 or whatever it may be.

Now, as mentioned you may have to use the ip address in the UNC name, 10.10.10.1\Backups\blah blah....

Or, if you are comfortable with routing tables you could do something with them but I suggest the UNC name...

Or, if a piece of software does the work then it may have the option to do it by IP address....
It's by far the cheapest and most reliable method for you to do this....

And, if you are not us ing Disk mirroring, or raid 5, you may wish to consider that in the future, a Raid5 system would have had ya back up in 2 hours or so depending on the SCSI card...

I'm learning more than I ever wanted to about SCSI but it's a good thing, in fact I may go SCSI at home when I get my new house...

Scott

------------------
Scott

[Gregery D Engle]

Scott,

Thanks for that information... We use raid 5 scsi at work. It
works very well, but the problem with this server is that our
compaq 750watt power supply went out and we had to order it from
compaq. Lets just say we now have 2 redunant power supplies on
that server.

As far as home use, go with the Promise raid 5 controller. They
even have hot swappable IDE chassis. The only manufacturer to
use ATA/IDE drives as a raid option. http://www.promise.com
I think...

------------------
-Greg

[eric stewart]

I'll soon be using LapLink Gold to keep two 144GB file
systems synced. PCsync from LapLink may also work for you.