Is differential based solely on the prior full backup?

Discussion in 'Acronis True Image Product Line' started by VanguardLH, Dec 4, 2007.

Thread Status:
Not open for further replies.
  1. VanguardLH

    VanguardLH Registered Member

    Joined:
    Sep 10, 2007
    Posts:
    96
    According to the help, the differential is based only on the prior full backup. Is that really true?

    Back when the archive bit got used to determine which files changed, full and incremental backups reset (cleared) this bit. That meant that differential backups were based on BOTH the full and incremental backups. The differential did not change the state of the archive bit but it still only backed up files that had it set. Well, that meant the archive bit got cleared for both full and incremental backups. So the differential would see only files that changed since the prior full or incremental backup, not just since the prior full backup only.

    The help says TI bases its differential on the prior full backup. Is that true? If a full backup is performed and several times thereafter there have been incremental backups, will the differential backup include all the files that changed and were included in those incrementals (versus how it worked before where differentials didn't include any files in either the full or incrementals)?
     
  2. shieber

    shieber Registered Member

    Joined:
    Oct 27, 2004
    Posts:
    3,710
    It would have all the changes since the full was created, which is why, unlike incs, intermediate diffs aren't needed to restore.

    However, as an odd quirk, intermediate diffs are necessary to validate -- go figure!
     
  3. VanguardLH

    VanguardLH Registered Member

    Joined:
    Sep 10, 2007
    Posts:
    96
    Well, yes, the differentials would be necessary to restore. Since differentials are not based on either the prior full or incremental backups then restore from differentials is required (despite how the program itself may behave) because it captures changes but doesn't record them. Rather than have 29 daily incremental backups hanging on the tit of the prior monthly full backup, I'd like to reduce the count of the backups. I also don't like like a long chain of incrementals to eliminate possible loss of lots of data simply because one incremental, perhaps early after the full backup, got corrupted.

    Under the old scheme of using the archive bit, I had the following setup:

    1 monthly full backup (Monday 3AM)
    3 weekly incremental backups (Monday 3AM)
    6 daily differential backups (Tue-Sun 3AM)

    If I got to the incremental backup and it completed okay then I didn't need to use the daily differentials in a restore. After the month, and until the next monthly backup (and because I like to keep a couple in rotation), I might have 10 backups for the month: 1 full and 3 incrementals, or 1 full, 2 incrementals, and 1 to 6 daily differentials. When the incremental ran, it reset the archive bit so it would capture what changed since the full backup, so I used it to capture what changed per week, not per day. What I really wanted was daily incrementals that would get rolled into a weekly incremental but that option rarely exists in home-grade backup programs. So I had to figure out how to incorporate differentials to reduce the number of backups in the chain (not the total number of backups performed per month).

    I do not like the the liability of chaining togther 29 incrementals onto 1 full backup to have a monthly backup scheme. If one incremental gets corrupted then I cannot restore using it or any of the later incrementals because one link in the chain got broke. By reducing the size of the chain the liability is reduced, plus the daily backups are not as important as the monthly and weekly backups (full backups are most important, then weeklies, and lastly the dailies).

    To have the most reliable backup scheme with TI means that I have to do the 1 full monthly backup and use differentials for everyday thereafter. That gives me a chain with only 2 links. If the differentials were based on both the full and incremental backups (as what happened when using the archive bit) then the a full-month long chain would have 4 to 9 links (1 full + 3 weekly incrementals, or 1 full + 2 weekly + up to 6 daily differentials). Even a 9-link chain is more reliable than a 30-link chain (1 full + 29 incrementals).

    No, the archive bit doesn't get used anymore but there's nothing preventing a program from emulating the same behavior in a catalog. Guess I'll have to rethink my backup scheme and get rid of the daily differentials and instead have 1 monthly full backup and 29 daily differentials. Yes, it means more disk space gets eaten but it also means that I have a short[er] chain of backups for improved reliability. Alternatively, maybe I'll have to forego having monthly full backups and go with weekly full backups and then have 6 daily incrementals.
     
  4. shieber

    shieber Registered Member

    Joined:
    Oct 27, 2004
    Posts:
    3,710
    Um, you meant "would not be required to restore?

    Because they only have the changes since the last full, I think that's exactly why intermediate diffs aren't necessary for a restore. Otherwise, they'd be the same things as incs. You should only need the full and the one diff that was made at the time to which you wish to restore. The intermidaries are irrelevent. If you make a diff every day and want to restore your disk to how it was last Tuesday, you only need that the diff made Tuesday plus the full. The Tuesday diff has all the differences between the full and how the disk was on Tuesday.

    With incs, otoh, each subsequent file only records differences since the last file (inc or full) so to restore to the state the disk was in when the las inc was made, you need the whole chain of files.

    Maybe I'm totally misunderstanding your remarks. If so, I apologize.


     
  5. VanguardLH

    VanguardLH Registered Member

    Joined:
    Sep 10, 2007
    Posts:
    96
    Regardless of how the program automatically links together just the full and incremental backups, you will still need the differentials if you want to restore what they captured. If the differentials are ran between the full and incremental backups then why would you throw away what they contain simply because you haven't yet gotten to the next full or incremental backup at the time you need to restore some files?

    How Acronis handles differentials is different than how I'm used to for behavior with other programs (old ones that use the archive bit and new ones that emulate the same behavior but using a catalog). I'm used to differentials that are based on the last full or incremental backup. Maybe I just got lucky with those other backup programs so I could have a more flexible backup schedule while reducing the number of backup files that had to get used for the restore to a particular day (and not just to a particular backup, like a full or a full with just the incrementals).

    I had based my backup scheme to what I expected would occur for differentials. Because they are only based on full backups, I've had to change from 1 full monthly + 3 weekly incrementals + 6 daily differentials to just saving 1 full weekly + 6 daily incrementals. This means full backups are executed more often than I wanted. Looks like that is the scheme that Acronis figures should be used, too (see http://www.acronis.com/enterprise/resource/solutions/backup/2005/incremental-backups.html, 3rd paragraph). I could go with 1 full weekly + 6 daily differentials to short then backup chain to 2 backup files but a chain of up to 7 links is okay. I definitely don't want a chain with 30 links in it (1 full + 29 daily incrementals).

    Because differentials are based only on the prior full backup (and not also on a prior incremental backup), and because I want a short chain of backups for reliable restores, it looks like I'll have to forego using both incremental and differential backups and go with reduced granularity using:

    1 full weekly + 6 differentials
    restore chain length = 2 backups

    or reduce backup file sizes with reduced reliability by using:

    1 full weekly + 6 incrementals
    restore chain length = 7 backups

    Since I was considering the 2nd setup (that uses incrementals), I figured an increase of just 2 links in the backup wouldn't much hurt reliability and I could use:

    1 full monthly + 3 incrementals weekly + 6 differentials daily

    A restore might take 1 full + up to 3 incrementals + up to 6 differetials (for a max chain length of 9). However, that relied on differentials being based on both full and incremental backups (which clear the changed state of files after they've been backed up whether using an archive bit or catalog).

    My expectation for behavior of differentials is based on historical usage (i.e., that a differential was the same as an incremental but the differential didn't change the archive bit). For example, the NT Backup program (around a lot longer than Acronis), which was the crippled version of Veritas Desktop Backup Exec (sold to Stompsoft and got called Backup MyPC), says in its help that "A differential backup copies files created or changed since the last normal or incremental backup." A normal backup is a full backup. Notice the differentials are also based on incremental backups. NovaBackup is another well-known backup program, and what Stompsoft carries as PC Backup (which replaced the Backup MyPC that they bought before), says in its manual, "Differential backup: Backs up selected files that have either not been backed up previously or have changed. These backed up files are not marked as backed up, however." Since both full and incremental backups clear the flag, differentials were based on both full and incremental backups. Unfortunately there are plenty of other examples where a differential backup is defined as based solely on the prior full backup. So there are 2 classes of differential backups: one that has the differential based solely on the prior full backup, and another that has the differential based on the prior full or prior incremental backup. This means the definition of what is a differential backup is somewhat muddy.

    Since differentials in TI Home are based solely on the prior full backup, I'll have to rethink my backup schedule.
     
  6. Xpilot

    Xpilot Registered Member

    Joined:
    May 14, 2005
    Posts:
    2,318
    VanguardLH,

    Although your dissertation and comparative study of other backup methods is interesting the article will have no effect or change the way True image handles differentials and incremental backups. The words "tablets of stone" spring to mind.

    To work sucessfully with any program one has to accept it as it is and work around its perceived short comings. In fact you summed it up quite well in your last sentence.
    While you are doing the rethinking there are some practical points that you should bear in mind.
    In general each differential will be larger than the previous one while each increment will basically the same size.
    There is however, a big however, when any changes are made to the layout of data on the hard drive. Then an increment or a diff can be nearly as large as the base image following a defragmentation run or partition re-size. TI will see data that has been moved as a change when it is only a position change, it is just the way it works.
    If you go for the backup locations feature you will have to allow a chunk of extra storage space and time to allow for the consolidation process.

    It was these kinds of complications that steered me towards using plain and simple full partition backup images. Once that was decided everything became much easier. I just had size my backup drive to take the number of full images I feel I really need plus plenty of room for expansion.
    With only full images to deal with scheduling them only took moments and because I can work round the shortcomings of the secure zone archive management is automatic and it works.

    Xpilot
     
  7. VanguardLH

    VanguardLH Registered Member

    Joined:
    Sep 10, 2007
    Posts:
    96
    I saw a comment somewhere (FAQ or help) saying that an incremental (or, I guess, a differential) could be as large as a full backup if a disk defragment had been ran prior to the incremental backup. That makes sense (for image or physical backups but not for logical file backups) because a lot of sectors would get moved around.

    The best practice would be to do a restore from the backups (full plus either the incrementals or differentials) to ensure it works okay. That is, to be sure all those resources consumed for backups are actually usable, I would do a restore. There is no option in TI Home to do a restore but not write to a destination (i.e., do everything in a restore operation except write the sectors or files). I have 2 disks: first one is the OS and data and the second one is for TI's Secure Zone. One day I might add a 3rd drive (internal or USB) but I don't have that now to do a test restore. So I have to wonder how reliable are restore operations or, more accurately, how usable are the backups to restore from them. Has anyone done any testing to check reliability of restores?

    Doing only weekly full backups reduces the file chain to just 1 backup file but consumes a lot of disk space. A weekly full backup plus daily differentials would shorten the chain to 2 backup files. A weekly full backup plus daily incrementals could have a chain that was 7 backup files long. (My original plan would've have a max chain length of 9 backup files and spanned a whole month instead of just a week.) Without the availability of an extra drive (or gobs of extra space on my existing drives to create testing partitions), I don't have the means of testing the reliability of restore operations when using TI Home and the Secure Zone. Any case studies available that checked this?
     
  8. Xpilot

    Xpilot Registered Member

    Joined:
    May 14, 2005
    Posts:
    2,318
    Reference to a case study will tell you very little. There are examples in this group were some people say they can never make a valid image, others including myself who find no need to run validations at all, we just go straight for a restore. There is also a set who do regular backups which have never been tested. Sometimes one of the latter will post here to say " wow! my hard drive broke and TI saved my bacon" well of course it should not have been a surprise.
    The thing that really counts is does the the program work on your hardware and the way that you use it?
    The safe and certain way to test this is to invest in a spare hard drive.

    Another principle to bear in mind is to have more that one backup in more than one place at any one time.
    In your case you will have several backup sets in your secure zone but the drive containing those images could fail which would leave you with none.
    When I realised I could not copy a full systems image from the secure zone to other media I used to run a second backup image to a USB drive to give a second restorable set of images. This became a bit of a chore so I eventually got a second spare drive and now have a rotation of three main hard drives. One is in use, the other two being a great form of native backup.

    Xpilot
     
  9. smatofu

    smatofu Registered Member

    Joined:
    Nov 14, 2007
    Posts:
    5
    At the beginning, I thought that not using the Archive flag was an Acronis deficiency. Now, I see it was a great decision: you can do multiple independent backup scenarios if the flag is not used. More backups to more destinations = much more reliability than just one backup. I do 2 different backups using 2 different backup methods (computer, data, sector-by-sector).

    The only disadvantage of not using the Archive flag is that I cannot quickly check if a file was already backed up. I think there should be an option to reset the flag, although it could confuse some users even more. Also, Acronis could have included some kind of tutorial to their archive algorithms in the application. I have to know these algorithms to design a good backup strategy.

    Every user and computer are different. I, myself, divided the HD into 2 partitions (one with system, data, and programs; 2nd with caches, HDTV recordings, less important voluminous applications). Most of the backups are tiny and quick (after excluding pagefile.sys and hiberfil.sys).

    I am mostly happy with Acronis. I still have to do some manual backup work, for example, move family pictures to DVD, because 10 years from now, none of my current disk backups will be available, but it is not Acronis'es fault.
     
Thread Status:
Not open for further replies.