How does DIFFERENTIAL backup consolidation work?

Hi all. I have TI 11.0 (latest build).

I have read this thread: https://www.wilderssecurity.com/showthread.php?t=159812 but it seems inconclusive on the functioning of differential consolidation. It does not seem necessary in the same way as with incremental backups but I guess this is done to stop new differential backups getting too big (although this can be dealt with by doing a full backup every so many incremental backups).

I assume if it consolidates, so the new full backup includes the changes in the oldest available differential backup that it then has to rework all the other differential backups, right?

So once you have hit your quota limit this seems like more work than creating a new full backup as you are now each time creating a consolidated full backup AND reworking all your differentials, right? And this is likely to continue each and every time you do a backup from that time forward unless you manually intervene to delete backups.

Is this right? In which case it would seem senseless to do anything but a full backup each time. But I may well be misunderstanding something. Can anyone put me straight! Thanks.

 

First let me say I think this is a great product. Makes backing up and restoring a whole lot simpler than when I used to use Ghost. I am very happy with my purchase.

But, I've just seen how differential consolidation works and it is very inefficient.

- First it makes a differential backup
- Then it deletes the oldest differential backup needed to meet quota e.g. "D1"
- Then it consolidates the full backup with the next oldest "D2"
- Then it works progressively through each subsequent backup (in my case D3-D7") to update it

First, this takes forever as it has to read each file in and write it back out again, taking twice as long as a normal backup between two drives. I am doing this on a network drive and the original diff backup took 20 mins. The consolidation has now been running for 5 hours and it still has 3 1/2 differential refreshes to do. I anticipate it is going to take about 9 hours in total!! Plus, because ALL the backups have been redone, I should really re-validate ALL of them!! So a backup and validate that could take 40 mins turns into a 13-14 hour marathon.

Second, it is doing all the refreshes before deleting ANY of the original files, so it requires additional free working space approximately the same size as the defined quota (not just the size of a full backup or even one full backup and one differential backup). I guess it is possible that if there was not this much space it would start deleting the backups which it has already refreshed, but there is nothing in the log to say it is doing such checks. It is hard to know what it would do.

This is clearly mad. It seems far more sensible to me that for differential backups it just deletes as many old differential backups as needed to meet quota and leave it at that. My backup, for example would take 20 mins instead of 9 hours and my data would be just as secure, with very little difference in final space required. It will then be up to the administrator to schedule full backups at sensible intervals so the differential backups do not become almost the same size and of little benefit.

........................

Also, to comment on the current approach of creating new backups before deleting any old ones, I have to say I disagree with it, at least when there is not enough free space on the disc. This is my logic:

- I have a main partition ~200GB in size
- I have a secondary 500GB drive for backups
- I would like to have at least 2 full backups (differential backups are not much use because I would not be able to have more than about 1 or 2 of them AND because the above consolidation would make the backup run for literally days).
- The way True Image works right now I can only keep one as I have to have enough free space to create a second, only to immediately delete the first. If I wanted to keep two I would need to have >400GB quota AND an additional 200GB free space, so TI can make the new image before deleting the first.
- If True Image worked differently I could have a 400GB quota with 2 backups. True Image could delete the oldest of the two BEFORE backing up. I would have only one backup for the brief duration of the backup process, but would then have two backups for the rest of the time. This is clearly MORE SECURE than what TI offers at present.

So, the sensible logic to me would be:

- Check the size of the planned backup and the amount of free space BEFORE starting the backup
- If there IS enough free space to create the new backup, reserve it and then backup as it is done now
- However, if there is not enough free space do not error (as happens now). Instead see if you can free up enough space by deleting old backups (other than the most recent). Only error if freeing up enough space would require deleting the most recent backup. Otherwise delete as many of the older backup as needed to free up space and then backup.

I'm convinced these approaches to differential backups and freeing up space make more sense than what is done at present but do join the discussion if you think I have things wrong.

 

Even though I understand some rationale in not deleting old backups, I also see your point.

So you know where my perspective is coming from, I do not do scheduled tasks, use SZ or Backup Locations, and only do Full images of my C partition. It contains only the OS and applications. I prefer to backup my data partitions with SyncBack which keeps the files in their native format.

Is 200GB the size of your partition or the amount of used space in it?

Since differentials capture all the changes since the Full each time, could you not save space with incrementals?

If you want to continue as you are doing now and you do have 200GB backups, I would consider throwing a bit more money at the solution. I would buy another archive drive and alternate backups to the two drives. This actually reduces the risk of a common point of failure for your archive storage.

Can you achive anything by setting your HD up differently? Do you have to image everything every time? I have a partition with large games on it and I rarely ever make an image of it. If I lose it, I still have the installation CDs.

I like having a relatively small C drive since I can very quickly backup and restore it if I am testing some software or other configuration.

My principal concern with backups is to securely capture my personal data files like spreadsheets, letters, photos, etc. Anything else can be reinstalled, a PITA but it can be done.

 

Hi!

At the moment I am backing up entire system to a spare drive in each PC and then also backing up my "valuable data" - photos etc. - to another location, with a file level backup, for a Doomsday scenario! I feel it covers me and offers a quick full recovery in most cases.

I was glad of my full image only this week when my main system went down. First time ever that has happened but great to get it back in a couple of hours.

The main point of my post was to follow on from the discussion in the thread I linked to, which is now closed. The current approach is mad.

I don't understand their rational about deletion of backups or consolidating differential archives.

The latter seems really pointless. At least it gains very little and the cost is to more than obliterate any benefit of doing a differential backup once you hit quota.

With regards to the former, I can understand you would not want to delete the most recent archive to free space for a backup. However, deleting an old backup seems very logical if there is not enough free space available. It has to be better than the new backup failing (as it does at the moment). And, if the worst happens, you just have one fewer backups to fall back on, but still have at least your most recent. In many cases you may only lose one or two old backups out of maybe 5-10.

To follow on my case in point with differential backups, the consolidation I was talking about earlier took 8.5 hours to complete AND still it was over quota, so True Image started ANOTHER consolidation, which is still running, but I expect to complete in about another 2 hours, making a total backup time of 16.5 hours for a differential backup of a folder (this is not my main image) that is only 6.57GB total size (admittedly over an 802.11g network, but running at full speed). I have lost the same number of old backups in this process (two) and gained very little space over the much simpler alternative of simply doing the initial differential backup that took 20 mins then deleting two old differential backups to get back within quota. 20 mins vs 16.5 hours it is amazing. It would of course be quicker on a wired LAN but it would still be ~50 times as long as the alternative.

I do not really like incremental backups as this then increases the number of files that have to be right to get your system back, but differential backups are certainly useless the way they are currently implemented. However, after this learning experience (which of course only happens when you start hitting quotas) I think I will use incremental backups for only slightly changing changing "data" backups but will have to do full system backups each time for my main image. Then I will hit my other point that because of the order True Image does things I will only have one backup at any time, when a simple switch in the order would give me 2 backups for the same amount of allocated space at all times other than while the backup is running and even at that time I will still have one backup - the same number as the current True Image system is able to provide through the whole period between backups.

So I'm very interested in what others think about the current process and whether you can see flaws in my proposed logic for a revised process.