Need strategy for keeping offsite backup updated

Discussion in 'backup, imaging & disk mgmt' started by doveman, Jan 4, 2013.

Thread Status:
Not open for further replies.
  1. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    I'm currently backing up my data to an onsite USB3 2TB HDD and now want to get another 2TB SATA HDD which will be stored offsite.

    I'll obviously backup the USB3 backup to the new SATA HDD when I get it before moving it offsite but what I'm not sure how to do is then update the offsite drive from time to time with any new/changed files I've backed up to the USB3 drive. I'll be doing this on a weekly basis by putting the files on a USB stick (or HDD if there's too many but I don't anticipate this) and taking it to the offsite location physically.

    I use Backup4All currently. What I'm wondering is if I can use this to make the initial backup from USB3 to SATA HDD and then have it identify any new/changed files on the USB3, even though the SATA HDD is not present and back those up (uncompressed) to the USB stick? It would then have to recognise that those files had been backed up and update it's records accordingly so that the next time I ran it, it didn't backup those same files again (unless they've changed since of course).

    Is this going to work or is there some other approach I need to look at?
     
  2. jwcca

    jwcca Registered Member

    Joined:
    Dec 6, 2003
    Posts:
    716
    Location:
    Toronto
    You could use
    FreeFileSync from Sourceforge to make a mirror of the USB to the Sata.
     
  3. kC_

    kC_ Registered Member

    Joined:
    Apr 6, 2007
    Posts:
    452
    robocopy "source" "destination" /MIR

    this will mirror the source with the destination (only changed files need to be copied)
     
  4. layman

    layman Registered Member

    Joined:
    May 20, 2006
    Posts:
    217
    It would be simpler if you were merely to swap drives. No need to shuttle changes in some way. Sync disk #1 with your data and swap it with disk #2 at the remote location, immediately bringing disk #2 current upon your return. Ping pong the drives each time you travel to the remote site.
     
  5. TheWindBringeth

    TheWindBringeth Registered Member

    Joined:
    Feb 29, 2012
    Posts:
    2,084
    I think the more common approach would be to use three storage devices, for example and in your case:

    1) 2TB master storage device
    2) 2TB removable backup storage device
    3) 2TB removable backup storage device

    where one of the backup devices would be onsite, one of the backup devices would be offsite, and you'd periodically rotate (physically swap) the backup devices (what was onsite becomes offsite, what was offsite becomes onsite). You could, for example, perform daily backups (copying new/changed files) to the onsite backup device and swap the onsite and offsite devices weekly. Edit: As layman has mentioned.

    Based on your description it sounds to me as though you might be trying to get by with just two 2TB storage devices (one master, one offsite backup) and using a smaller capacity USB flash drive for a) daily, lets say, backups, and b) shuttling copies of those files created/modified during the week, lets say, to the offsite location where they will be copied to your offsite backup device. If this is what you mean you should confirm it. I'm not familiar with that backup software and its capabilities, but this approach is possible if you use the correct protocols. In this scenario, the backup software doesn't have a "full" destination filesystem to compare the source filesystem against. The protocol has to make up for that in one way or another. For example, by storing state information (filesystem names, attributes, hashes) for the source somewhere so that the source can be compared against previous source state information to determine what has changed and needs to be copied to the USB flash drive.

    I'd go with the first approach rather than this later approach if you can.
     
  6. Keatah

    Keatah Registered Member

    Joined:
    Jan 13, 2011
    Posts:
    853
    Am I correct in assuming the OP wants to use a JumpDrive to shuttle the changes to the offsite disk?
     
  7. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Yes, to clarify the offsite HDD will be stored inside another PC so it's not going to be practical to keep swapping it out, not to mention I can't afford a third HDD. I did consider using a USB3 2TB HDD for the offsite HDD (which is what the onsite backup HDD is) but as it requires an external PSU, not to mention deskspace, I decided it was too cumbersome.

    As TheWindBringeth suggests, the tricky part is that the backup software won't be able to see the offsite HDD, so it will need to store a local record of what has already been backed up it.

    The initial copy will be done locally, so I'll setup the onsite backup HDD then sync or backup that to the HDD destined to be the offsite one, so at that point they'll be identical. Then the offsite HDD will be moved offsite and from then on I'll need to do two steps

    a) backup new/changed files from the live system to the onsite backup

    b) backup new/changed files from the onsite backup to a USB flashdrive, to be taken to the offsite location and copied to the offsite backup

    It may be that there's some way to combine the two steps though, so that I can backup the new/changed files from the live system to the onsite backup and the USB flashdrive at the same time which would make things a lot simpler.

    EDIT: Just to complicate matters, some of the folders (C:\Users, C:\ProgramData) I'll need to backup using a "remove deleted files" strategy, whilst the other folders (such as D:\Software which is where I store downloaded programs) will need to not do that and just copy any new files, as I'll not be keeping all the software I've downloaded on the live PC, only on the backups.

    Backup4all has a Mirror backup type, with "Fast Mirror (copy only new or modified files)" and "Remove excluded or deleted files from backup" options, so I might be able to use that, although I'd have to make two backup profiles and thus do two runs to backup the two different types of folders.

    It also supports local catalogs, which sounds like what I need to use for doing backups when the destination/offsite drive isn't available. I'm still not sure how to achieve what I want though. For the folders where I'm just backing up new/modified files, I thought I could do the backup to the USB flash drive and then copy the files to the onsite backup HDD and then take it to the offsite location and copy them to the offsite backup HDD as well but obviously for the folders where I want to remove deleted files from the backup, that's not going to work.
     
    Last edited: Jan 11, 2013
  8. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Perhaps the only way to sync the backups of the folders where I want the backups to reflect their current live state (i.e. with deleted files removed) will be to backup to the onsite HDD and then copy the entire folders to a portable medium to take to the offsite HDD and overwrite the folders there.

    I might be able to manage with a 32GB flash drive or if not, use a bus-powered USB HDD (I've got a 500GB 2.5" drive I can use, just need to get a new caddy for it as the one I've got is dodgy and causes SMART CRC errors).
     
  9. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Anyone got any ideas?

    One drawbackup of Backup4All is that it doesn't show me what it's going to backup/delete before proceeding, which makes it hard to see if I've set it up to do what I want. Using FreeFileSync it shows me what it's going to do when I click Sync, which is much better but that's not really suitable for backup jobs I don't think.
     
  10. HAN

    HAN Registered Member

    Joined:
    Feb 24, 2005
    Posts:
    2,080
    Location:
    USA
    Let's see if I get this...

    2 TB drives at each end point, but want to use a smaller drive in between to shuttle ONLY changes (additions, deletions and changes to existing.) Right? To my knowledge, it can't be done. Mirroring software I've used needs to mirror everything, not just changes.

    If you could find a way to mirror sync via the Cloud, there are ways of doing that. But that's a different thing than what you've noted thus far.

    If the shuttled drive could be large enough to mirror ALL files, then something like GoodSync2Go would work. It costs $40 (US) but it would do what you want. (Beyond that, it would allow you to verify all file changes prior to committing them, which is something you say you want.)

    I have used GoodSync for several years and it's very reliable.

    http://www.goodsync.com/platforms/win/2go
     
  11. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Yeah, that's pretty much the idea, although there's two different types of backup folders, those which I'll only be adding new files to and those which I wish to sync with the live system's folders. So for the first type, I just need to get new files on the portable drive to take those offsite, whilst for the second type, I'd have to copy the entire folder to take it offsite.

    I get the impression that Backup4All will enable me to deal with the first type, using it's "local catalog" feature, so that I would create a backup job to sync the folders on the two drives before moving one offsite, then on subsequent runs, set the destination to my portable drive. As it has the local catalog telling it which files have already been backed up to the offsite drive, it doesn't need to see it to know which files need to be added. Then I can just copy the files from the portable drive to both the onsite and offsite backup drives.

    For the second type I believe there's no alternative to copying the entire folder(s) to the portable drive to take offsite and then delete the existing folder from the backup drive and copying the new one from the portable. Thankfully the folders where I want to do this generally don't contain tons of data so it might be feasible with a 32GB flashdrive but if not I'll have to use a USB HDD. In theory I think a program could record which files need to be deleted and then be run on the offsite machine to perform those deletions, which would mean I wouldn't have to delete the entire folder, just copy across the new/updated files but I'm not aware of any program that does this unfortunately.

    EDIT: Seems that Backup4All can't do what I want actually, as when changing the destination drive it deletes the local catalog!
     
    Last edited: Jan 16, 2013
  12. HAN

    HAN Registered Member

    Joined:
    Feb 24, 2005
    Posts:
    2,080
    Location:
    USA
    First, I need to clarify terminology I have been using. I was using mirror to mean 2-way sync. After reading the Backup4All manual, it's clear that they are NOT the same. Hopefully my use of mirror has not been confusing.

    Second, I agree that with whatever software you choose, you'll probably need two different syncing profiles/jobs.

    I'm not familiar with Backup4All. But after taking a quick glance through the manual, I agree that the Fast Mirror setting looks like it should work for the new/modified file profile/job. But not for the job that needs to track deletions.



    It looks to me that Backup4All does NOT do 2-way syncing. If you are trying to keep multiple exact folder copies, 2-way sync programs can save so much time. I 2-way sync several drives and depending on the changes that need to be updated, they can sometimes be processed in a matter of a minute or two. This tiny time frame includes ALL changes, additions and deletions. Everything!

    The reason I bring this up again is to have you consider using a large enough shuttle drive that could accommodate full 2-way syncing. It would mean serious time saving for you. (I daily sync two 64 GB flash drives containing around 40,000 files. With 2-way syncing, and an average number of file changes, they typically match each other in under 2 or 3 minutes. If they mirrored each other (using Backup4All's terminology) it would take an hour or two to copy all the files. I could not tolerate that much time on a daily basis!)

    I realize you have a budget to deal with. But if you can swing something like GoodSync2Go, and a large enough shuttle drive, you'll be MUCH happier in the end. You'll have fast, accurate backups and you'll know exactly which files/folders were changed, added and/or deleted. Here's a link to the GoodSync manual http://www.goodsync.com/support/manual


    EDIT
    Just saw your edit. Not good!
     
  13. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    That's OK, I understood mirror to mean 2-way sync. As you say, Backup4All uses the term somewhat differently.

    FreeFileSync works fairly well. I'm really only doing 1-way sync, as I'm just updating the 2nd HDD to match the first, including deletions, rather than copying any files from the 2nd to the 1st HDD. The only issue I've found is that for some reason filenames don't always keep their case and then FFS identifies them as different, despite having the same filesize and date, which leads to clutter in the results and unnecessary overwriting.

    As my onsite backup HDD is a USB3 external, it would be possible to take that to the offsite location to update the HDD there but I want to avoid that if possible, partly because there's obviously additional risks in transporting it but also because it requires an external PSU so is cumbersome to connect up.

    So practically I think I'll have to make do with my 32GB flash drive and/or my 500GB USB2 HDD, which is bus-powered so less of a pain to move (and also doesn't contain anything important, so would be no major loss if something happened to it in transport).

    For the live folders I want to keep synced, I guess I can use FreeFileSync to sync them to both the onsite backup HDD and the portable drive and use FFS again at the offsite location from the portable drive to the offsite backup without having to delete and copy the entire folders.

    For the other folders (the ones where I just want to add new files) perhaps the easiest way to do this is to store any new files in a "To backup" folder on my main HDD or on the portable drive and delete them once I've copied them to both the onsite and offsite backup drives.

    I'll take a look at Goodsync thanks but I'll probably have to make do with the free options for now I think.
     
  14. Sully

    Sully Registered Member

    Joined:
    Dec 23, 2005
    Posts:
    3,719
    If it were me, I would install the files/folders that need protected to the 3rd "portable" drive. I would access them from either end point. I would sync either endpoint right from the 3rd drive. I could take the 3rd drive to either location and utilize the files as needed. I would then have a backup on each end.

    Seems like a good solution to me since you are toting a drive with you anyway.

    Sul.
     
  15. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Yes, that might be a good idea for the new downloaded files if I use the portable 500GB HDD and keep it attached via e-SATA at home and just store the new files on that directly, then update the onsite backup and offsite backup from that.

    It's not really suitable for some of the other folders (i.e. C:\users\data, My Documents) as I'll want to keep those on the main drive but I can just make a sync copy of those to the portable drive to take offsite, as well as syncing to the onsite backup.

    I just need to find a suitable 2.5" USB/e-SATA caddy for the drive as the one I have is obviously faulty as SMART shows increasing CRC errors which doesn't happen with the drive connected by SATA. I found one before Christmas on e-bay that said it supported up to 500GB HDD but it only came with one USB plug on the power lead and I know from using my existing caddy that the drive requires two.
     
  16. dankorzon

    dankorzon Registered Member

    Joined:
    Jan 19, 2013
    Posts:
    1
    Location:
    United States
    Why not just set up an sftp server on a local computer or the backup computer and backup the files on the source computer to the external hard drive and then just copy the files over to the offsite computer via that connection so you don't have to worry about driving there all the time and you still have that redundant external backup. If the computer isn't connected to the internet then you'd probably just have to go with lugging it back and forth. As far as software goes I use Genie Timeline as it is a real time solution and you can just dedicate your external hdd to be the backup location and let it go. The program auto-detects whether the destination drive is backed up and will stop/start on detection and it comes with a handy bootable iso for recovery situations.
     
  17. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    That's certainly an idea for the data folders, where the changes will be fairly small, so I could just setup an sftp server sharing the live folders directly and sync them to the offsite backup when I'm there.

    For the other folders they tend to get rather large though, with things like ISOs and program install files so I'll probably need to carry those on a portable media to the offsite location still, particularly as I don't have great upload bandwidth. Those folders can't be synced as they only contain the new files since making the original backup, so it's just a matter of copying them into the appropriate folders on both backups before deleting them from the live system (or portable drive if I decide to temporarily store them there instead).

    I'll certainly look at Genie Timeline though, thanks for the tip.
     
  18. Kees1958

    Kees1958 Registered Member

    Joined:
    Jul 8, 2006
    Posts:
    5,857
    Easiest would be rotating the disk (aka the 'Nike/Adidas backup system')

    Week 1, disk A is connected, disk B is remote and off-line
    week 2, bring disk A to remote and off-line location, take disk B back and connect to PC/Network, let the data auto update through sync mechanism (Ms Sync Toy or any other synchronization tool like Syncback), do a seperate Image backup for programs.
     
  19. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    That's no good for me as a) the offsite drive is stored inside a PC, so can't be swapped out easily and b) much of the backup data is only on the backup drives, not the live system (one reason for getting the second backup drive was so that I could have an offsite backup but the other reason was so that I could delete stuff from the live system, freeing up space but still have two copies of the files).
     
  20. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Does anyone know of a decent, reasonably cheap, 2.5" SATA to USB and e-SATA caddy?

    The only ones I've found are this which would be fine as it says it supports up to 500GB HDD but despite saying in the description "No external power needed, just USB is enough, please note that sometimes it needs to use 2 USB ports of your computer" only comes with a DC power lead with 1 USB plug, so when using e-SATA there's no way to use 2 USB ports to power it (with my current caddy I can't plug in the USB data lead and the e-SATA at the same time as it just confuses the PC, so I'm presuming this is the same) and in my experience with my drive in my current caddy, it needs 2 USB ports to power it. I did think I might be able to use the lead from my existing caddy but there's no guarantee they use the same size DC socket.

    http://www.ebay.co.uk/itm/2-5-SATA-...es_HardDriveEnclosures_RL&hash=item4aace143f2

    Then there's this USB3 and e-SATA caddy which is too expensive but also has no USB->DC plug facility. I presume it must prefer the e-SATA over USB when both are plugged in, as the USB cable would be needed for power but unless USB3 provides more current than USB2 sockets, I don't see that my drive would work with a single USB lead.

    http://www.ebay.co.uk/itm/USB3-0-eS...es_HardDriveEnclosures_RL&hash=item45fdcc0fb1
     
  21. doveman

    doveman Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    119
    Hmm, I've just realised that whilst I can setup a FTP server using SSL with FileZilla, I don't actually know how to then access that as a source with FreeFileSync, only by connecting with a FTP client like Filezilla, so any advice appreciated.
     
Loading...
Thread Status:
Not open for further replies.