Backing up data, keeping it private

Discussion in 'privacy technology' started by blainefry, Jun 30, 2014.

Thread Status:
Not open for further replies.
  1. blainefry

    blainefry Registered Member

    Joined:
    Jan 25, 2014
    Posts:
    165
    I started this thread to help determine how I might go about encrypting an external hard drive to use for backup:

    Encrypting an external hard drive with TrueCrypt

    Now that I've pretty much got everything underway I thought I might see if there's any suggestions for improvement.

    I initially thought that performing image backups would be the way to go. It would ensure that nothing was left out, that files and structure were kept up to date, and the backups could be compressed and encrypted.

    But the problem I found with imaging backups is that both incremental and differential backups do not absolve you from having to do periodic full backups. They simply allow you to do full backups less often.

    (See "Appendix B: Backup Strategies" in Terabyte Unlimited's Image for Windows manual):
    http://www.terabyteunlimited.com/downloads/ifw_en_manual.pdf

    For me, I simply wanted to be able to backup my data. I have no real need to backup the operating system or the OS partition because all data resides on a different partition. (I just make sure to periodically manually copy a few things that might get saved to the OS partition.)

    So I really seemed to have no need for imaging for my data backup.

    So as I mentioned in that thread above, I went with full device encryption on the external (backup) drive. And I just use FreeFileSync to keep it updated. That's one of the best programs I've ever used in terms of the application getting its specific job done while also being efficient and intuitive about it.

    http://freefilesync.sourceforge.net/
    http://sourceforge.net/projects/freefilesync/

    I will admit that I do like the copy verification that certain dedicated copy programs will do, and in fact, FreeFileSync has verification ability itself:

    http://sourceforge.net/p/freefilesync/feature-requests/390/

    But I'm not too worried about it for most files, so instead of toggling that on and off, for larger files or more important folders I just use FastCopy...mentioned here:

    https://www.techsupportalert.com/best-free-file-copy-utility.htm
    http://lifehacker.com/5280976/five-best-alternative-file-copiers

    I just run a File Compare scan in FreeFileSync, using the Mirror sync method. This shows how the source drive is different from the backup drive. If you're not too concerned about verification (which, odds are pretty low that a simple copy-over will be corrupted), then all you do is just perform the mirror sync. But if you're a little paranoid, you can easily just use the File Compare readout to easily determine if there's anything that you want to copy over using verification.

    FreeFileSync is very intuitive, so you'll really only need to open it and run a File Compare scan to understand all of what I'm saying and get an idea of the flexibility it offers.

    This keeps my data backed up, and, as the entire device is encrypted, keeps it private.

    Does anyone have any ideas for improvement on this? Is there anything I should be aware of or anything else I could implement to make this system better?
     
    Last edited: Jun 30, 2014
  2. MrBrian

    MrBrian Registered Member

    Joined:
    Feb 24, 2008
    Posts:
    6,032
    Location:
    USA
    1. You didn't mention whether you're using FreeFileSync's file versioning feature.

    2. Are you using multiple external drives or just one?

    3. You can also use FreeFileSync to compare by file contents, for peace of mind.
     
  3. blainefry

    blainefry Registered Member

    Joined:
    Jan 25, 2014
    Posts:
    165
    @MrBrian

    Thanks for offering feedback.

    Currently I'm only using one external drive. What are you thinking on that? Did you have some ideas for redundancy?

    I actually wasn't aware of a file versioning feature. That's awesome to know (and given how great FFS is, and how well it's maintained, I'm not surprised that feature is available). But I'm not really worried about backing up past versions. Now that you mention it, I hadn't really consciously thought about it before but I'm actually in the habit of saving new versions of important docs under different successive file names. If I make substantive changes to something I actually do tend to just save it as "filename 2.ext"

    I have used the content comparison a few times on specific folders just to confirm the files within them were duplicates (although I ended up using Duplicate Cleaner instead, since it produces a better-suited output for that (as of course that's what it's designed for). Also on that note, DC is another really great program that I have found to be incredibly well suited to its dedicated task. I highly recommend it.)

    That being said, I simply have way too much data to be doing a byte comparison every time I sync. Possibly when I switch to a faster machine, but even then, it's a bit of an unnecessary time expense for me. I suppose I might do that once a week or so.
     
  4. MrBrian

    MrBrian Registered Member

    Joined:
    Feb 24, 2008
    Posts:
    6,032
    Location:
    USA
    You're welcome :).

    If budget allows, you may wish to consider having at least one external drive kept offsite in case of a local disaster.

    There are other reasons for file versioning:
    1. You delete folder(s)/file(s) by accident and don't realize it until sometime later.
    2. CryptoLocker or similar corrupts your files and you don't realize it until sometime later.

    The fact that CryptoLocker or similar can alter your external drives when they're in use is another good reason to consider having multiple external drives. Here is a thread with more info.
     
  5. Tekhne

    Tekhne Registered Member

    Joined:
    May 23, 2014
    Posts:
    19
    I don't think there is too much I have to contribute with, except in regards to that file compare ability of FFS I'll just mention that I happened to come a cross a program for that as well called 'Beyond Compare 3' recently as well. I have yet to explore it's functionality, just while posting this to you I realized it can sync and do backups apparently as well...(sigh, so many options)
    http://www.scootersoftware.com/index.php

    You might not need it though since you already seem to have something akin to that ability already in FFS.

    Btw, Freefilesync doesn't update folder structures and such/name changes ect right ? cause then I'm sure it's a neat program for just moving stuff and comparing it, but you will then have to mentally keep track of any changes you make, like changing names on a folder or file, and then end up with duplicates/old and revised versions of your files and folder structure right ? If so it seems a bit of a hazzle to me, and I would personally worry about making a mistake or forget to cover something/leave something out.

    In my mind, something that "mirrors" my data drive without me needing to delete or micro manage stuff is most ideal. I still don't know if these more "advanced" backup programs do this, but I'm about to test it by deleting the data I restored, and then restore it again after making a differential backup to build on the first full one. (perhaps I already mentioned this in that 'Other' thread)
     
    Last edited: Jul 1, 2014
  6. blainefry

    blainefry Registered Member

    Joined:
    Jan 25, 2014
    Posts:
    165
    It does, actually.

    Again, you can set it to "Mirror" the source drive, which means the destination drive literally looks exactly like the source. This means not only adding files that appear on the source and not the destination, but also deleting files that appear on the destination and not on the source. (But also, as mentioned, you could set it to keep previous versions files as well.)

    So if you rename a file or a folder, or move a file to a different folder, at the very least, FFS will simply delete it from the old structure (where it appears from the last backup), and copy it to the new structure (where it appears in the source now, at the time of the sync).

    But there is also an option to "detect moved/renamed files."

    http://sourceforge.net/p/freefilesync/feature-requests/280/

    I actually haven't bothered to toggle this on yet, as I haven't had much of a need for it. If I notice in the comparison readout that I have moved something large and it is about to delete it from the destination and recopy it from the source to the new destination, I might open the destination folder (easily done with right-click) and move the file myself. But of course this isn't necessary, as all it's doing is saving the program a delete operation and a copy operation. No big deal, just a little more processing, i.e. time.

    But the bottom line is OF COURSE you aren't expected to keep track of changes. What would be the point of having a sync program otherwise??

    This is exactly what FreeFileSync does (and other options as well). Just try it out. You'll see.
     
  7. blainefry

    blainefry Registered Member

    Joined:
    Jan 25, 2014
    Posts:
    165
    At this point maintaining an off-site drive would be more trouble than it's worth. I simply don't have a convenient place to keep it, not to mention, doing a back and forth trip on a periodic basis would just be too cumbersome to me.

    For offsite I'll be going with an online service. I haven't implemented it yet, but I've done quite a bit of research and will most likely be going with Backblaze or CrashPlan.

    And even though they both claim client-side encyption, I've taken a lot of time to consider the most efficient way to not rely on/trust them for the most sensitive files. Even with a completely private key setup, at the end of the day, you ultimately have no idea what their applications are doing. They could easily alter the app at any time and have it send a copy of files in plaintext to some other server. (This is of course assuming it doesn't do this already.)

    And going back to the offsite drive, of course I'm aware of CrashPlan's offsite option to transfer to a private hard drive at another location, but again in the same way I don't really have a place to bring a HDD, I don't really have anyone I'd want to bother with keeping their machine on and letting me use space and such.

    As for cryptolocker, I'm really just not that worried about it. I only have my external plugged in when I sync it, and more importantly than that, perhaps this is a bit paradoxical from someone who is more on the paranoid side, but trojans just don't really scare me. I know what a spoofed email looks like, I'm probably more suspicious than I need to be, and it's rare that I'll download or run anything even remotely questionable. Not to mention, I reformat my system more or less monthly anyway.

    (Which is where this thread came from: )
    TrueCrypt: encrypting a Windows partition?

    That thread you linked was pretty interesting and good to have available, but it's just way overkill for me.
     
  8. JimmySausage

    JimmySausage Registered Member

    Joined:
    Apr 11, 2010
    Posts:
    53
    FreeFileSync installs malware that you cannot uninstall or not install by unclicking during custom install.
    I used VM to check. Works well though. What makes it Hot is the Mirroring function.
    Can't use it though. Pretty unethical of the developers.
     
  9. blainefry

    blainefry Registered Member

    Joined:
    Jan 25, 2014
    Posts:
    165
    Would you care to offer any sort of actual evidence to support this claim?
    Are you just talking about OpenCandy? Seriously?
     
  10. crawfish

    crawfish Registered Member

    Joined:
    Jul 2, 2014
    Posts:
    24
    I have two sets of hard drives I use for backup, one off-site, one local, and I rotate them monthly. A standard small safe deposit box can hold at least four 3.5" HDs.

    I backup several TBs, much of it essentially archival, so I do a contents comparison every couple of months after rotating the backups and bringing the off-site set home. I've never found a discrepancy, but if I did, how would I know which is the "good" file if it wasn't something like a .rar with built-in validation? This has led me to start keeping a database of file hashes.

    Be sure you go through AppData and ProgramData with a fine-toothed comb. Lots of programs store lots of the data you create in these hidden folders, and most don't document it. It would probably be inappropriate to just copy them wholesale outside a system image, but you would really lose out if you didn't selectively copy the important stuff like Outlook .pst files, Firefox profiles, and so many other things.
     
  11. blainefry

    blainefry Registered Member

    Joined:
    Jan 25, 2014
    Posts:
    165
    Yeah like I said, that's just too much of a hassle for me. Particularly when I consider the likelihood of actually ever needing the offsite backup. (Which, again, I'll be using an online service to handle that.)

    Good point. But as that page linked within one of the links above mentions, there really is no verification of data until you get around to using it.

    http://blogs.msdn.com/b/oldnewthing/archive/2012/09/19/10350645.aspx

    Good catch. That is actually exactly the "few things" I was talking about that get saved to the OS partition that I manually copy over. As I said, I reformat pretty regularly, so I'm definitely aware of everything I need to save to be able to restore everything the way I had it. But that's definitely an important detail that folks should be aware of.
     
  12. crawfish

    crawfish Registered Member

    Joined:
    Jul 2, 2014
    Posts:
    24
    Oh, I'm definitely assuming the data as originally stored is correct, and some of that is verified, and some is not. I'm concerned with detecting changes after that point, which is probably more paranoid than I need to be, but I figure if I'm going to all this trouble, why not cover as many bases as I can. For example, whenever I introduce a new backup or other drive and copy data to it over the network or with it in a USB3 dock, I can verify all that hardware worked correctly by computing the hashes on the destination drives and comparing to my database, which avoids tying up my storage drives in a lengthy file comparison operation. It also indirectly verifies my storage drives, as I just used them as the source for the copy, so I don't need to do file comparisons. I've recently done this with about 8 TB of data, and there were zero errors, and I've never found any sort of file transfer error in years of doing this. It's remarkable how reliable these things are, but all the same, they're not perfect. For my ordinary backups, I do the file comparisons from time to time to test both source and backup drives for bit rot, and if there ever was a difference, the hashes would tell me which was the original "good" file.
     
  13. Tekhne

    Tekhne Registered Member

    Joined:
    May 23, 2014
    Posts:
    19
    Just dropping by to say, I've tried it out, and used it for my backup/syncing needs now, and yes, this was indeed very well ! At least so far.

    Yes I noticed that, and enabled it, it needs to create some database files at both source and destination though in order for that to work just fyi. Their hidden so I guess thats perfectly okay.

    I found out it could even sync to multiple destinations, (even though it probably doesn't do so simultaneously) So thanks for the tip. It's a sure keeper. Even though I used it together with the program I mentioned above 'beyond compare 3' for what I think is a better and more detailed file comparison. I certainly won't bother more with acronisTI, i tried some, but not only did it become unstable and crashing even, I even discovered how utterly unreliable it was when I noticed it changed certain backup settings all on it's own ! without the users consent or knowledge. So I feel I've come out perhaps quite a bit more experienced and less forlorn at the end of this now.

    Take care then, and may everybody's data be safe.
     
    Last edited: Jul 9, 2014
Loading...
Thread Status:
Not open for further replies.