Validation

I’ve been testing various things out on some laptops which means at the moment I’m constantly backing up & restoring images to test out my changes. Can anyone tell me how often I should be validating my images. At present I’ve been validating on the completion of a backup image & then again just before I restore an image. Do I need to validate each time or am I waiting my time doing this too often?

The images I’m restoring may only be a day old, or 1 hour old.

Is there a recommended timescale when the verification is needed?

 

Are you trying to start a war? In other words, there are differing opinions but since I'm writing this, I'll give you mine.

A validation ensures that TI can read the archive properly and successfully reconstruct the 4000 checksums/ GB of archive. It is a good disk subsystem and RAM test since a failure in these 2 components will give a validation error (archive corrupt) - only 1 bit in the archive has to be bad to trigger the error.

I always validate my archives because it makes me feel better and I know that my system is still working as intended and TI is happy with the archive.

Since I have a degree of confidence in my system I usually don't check the archive before restoring. The downside is that if TI has any trouble reading the archive during the restore you will be left with unallocated space instead of your partition since one of the first things TI does when restoring a partition is to delete the partition. So, validating before a restore can be a good thing.

Interestingly, I had 2 images go bad on my portable's HD. I was going to revert to an earlier image due to an IE8 installation issue. TI9 said 2 of them weren't TI image files (it never got to the validate operation). Chkdsk /r found a lot of bad sectors on the partition. These images were validated when first put on several months ago but something has gone bad on the HD.

A lot of the people that say they validated and then the image was corrupt when they went to restore it may have had a situation like mine above but they most likely suffered from WIndows-only validation. When TI restores the active partition it does not use Windows but a Linux environment. You must ensure your system is properly supported by the Linux environment. The best way to do this is to do a test restore to a spare HD (in case it fails).

The next best thing is to boot up the TI rescue CD and make your archive with it. Then validate the archive using the rescue CD. If that works then go through the restore wizard and pretend you are going to restore but cancel out without Proceeding at the final screen. This will give you a pretty good degree of confidence but the test restore is the best.

Once you have determined the TI rescue environment works on your PC then Windows validations are fine.

 

seekforever - that was quite a bit more detail than I expected but its appreciated. Does anyone one else have any other ways of doing things?

 

I also usually validate when I create an image and not before a restore.

If doing testing where a lot of image creation and restoration is taking place, I will sometimes skip all validations. It's not really necessary in these cases because you can start over from your base image.

If you've been doing this for a while and haven't had any problems, I wouldn't worry about validating everything. I would still validate any images you intend to keep, though.

 

Hi

How dependable is the checksum ?

Back in 1978 home computer data transfers to/from tape recorders were validated by 8 bit arithmetic checksums.
A single bit error was guaranteed to be detected.
Multiple bit errors had a 255 in 256 chance of detection,
i.e. a 0.4% possibility that hopelessly corrupt data could be accepted as perfect.

I elected to implement a 16 bit Cyclic Redundancy Polynomial Calculation for which :-
Multiple bit errors within a burst over a restricted range are guaranteed to be detected;
Outside that range the possibility of detection failure was less than 1 in 65,536.

31 years later, I would like to know :-
1. Does Acronis use a puny 8 bit arithmetic checksum, or a vastly superior 16 or 32 bit CRC ?
2. exactly what is validated by "4000 checksums/ GB of archive" - i.e. is this per GB of compressed image in the *.tib archive, or per GB of the Drive C:\ "Used Space" reported by Properties.
3. I assume that a large amount of RAM is used, making validation vulnerable, because it is needed for effective compression when creating the image, and is also needed for effective decompression so the checksums are available - please tell me if my guess is wrong.
4. Given that a specific computer has defective RAM with a 10% chance a new image may fail validation, I would revalidate that image several times.
If it always failed re-validation I assume it was corrupted in RAM before getting into the archive, and that archive would never validate, so I would delete the archive and create another. If I found the archive sometimes validated well, I would keep the archive as being "as good as it gets", and then focus on getting more reliable RAM. I consider myself fortunate that no image has yet failed validation.

IMPORTANT QUESTION :-
When restoring an image there is an option to pre-validate.
If my RAM should degrade with a 10% probability of failure, I assume the restore will be safely aborted upon pre-validation error.
If pre-validation is successful, I assume that the archive is then decompressed with a 10% probability of error. Will Acronis detect and report any such error so I can try again with the Linux Boot Disk, or is there a danger that the restore will finish without reporting any problem, and that the computer may reboot in a functional state with no indication of problems, BUT with major corruption of non-system files (e.g. documents, esppciallialy income/expentiture spreadsheets and DRM protected/crippled media files) ?

I recommend that any image should be validated after creation. If it fails you can try again whilst the system status is still as you wished to preserve - it is too late to try again if one week later you find the archive is unusable.
And DO NOT VALIDATE as part of the creation task. Validation by the task which creates the archive will validate the entire contents of the archive folder. I expected the task to take 10 minutes when I added validation, instead at the end of the day the Acronis log showed about 6 hours validating all 50 archives ! !

Regards
Alan

 

I have no idea what method Acronis uses to calculate the checksum but I would say it is a lot more sophisticated than a bit-add or XOR.

Acronis has said a checksum is added for every 256K bytes of data. I would take this to mean pre-compressed.

TI probably does take a lot of RAM or as much as it can get and this combined with the rather stringent checking of data shows up RAM errors that were undetected previously. A typical PC never checks RAM data, it is always assumed to be correct and this holds for HD data going into RAM since the CRC check on the HD pertains only to data at the connector not as read into RAM although SATA drives do use a parity mechanism.

When doing a restore from the Linux rescue environment without validating the image as part of the restore process the partition being restored will be deleted and you will be left with unallocated space should an error be detected during the restore. If you pre-validate then the restore should be successful since even a 1 bit error will cause the archive to be seen as corrupt and the restore aborted. AFAIK, TI does not provide a restored partition if it has encountered an error - and some users have asked that it be modified to do that since they reason something is better than nothing.

A 10% chance it would fail? If it fails once then something is not right and needs to be investigated. If the machine has worked perfectly in the past then something is not right. If it hasn't been observed to work problem-free with TI then there is also the possibility the system design isn't rock-solid since it is known the TI hammers the system with huge amounts of data at a high-rate. Some users have only been able to fix their PCs by backing off on recommended RAM timings. I'm not going to get into an argument whether that's a PC design problem or a TI design problem!

 

Hi everyone,
Personnally, I never validate before restoring but I do when creating a backup.
So far ,when I have restored the reason was that Windows crashed.Then I said to myself what is the purpose of validating my backup.
The worst thing that can happen to me is that it is corrupted.If it's corrupted ,I will realize it during the restore process.

That's the reason I always keep 2 full backups on hand ,through the Acronis scheduled tasks.
For the same reason,I also keep 2 rescue discs on hand ,in case...

 

When using TI danger to a system and data arises when a restore fails for whatever reason. A failed restore leaves the user with a hard drive that is nothing more than unallocated space because the contents is deleted as a first step in the restore process.
Running a validation is an assurance that the image is good but is by no means a 100% guarantee that a restore will be sucessful.

My way to reduce the risk element to zero has been to implement a simple rule. Never restore to overwrite a current working hard drive and always have a series of backup images available.

Swapping over hard drives for both PCs and laptops is quite a straightforward process if advantage is taken of the hardware odds and ends that make the job as easy as changing a light bulb.

Because I have built regular restores to swapped drives into my backup process I have no need to spend time on running validations. If there is a restore failure it is not a crisis. I have the choice of relacing the now blank drive with the previous one and doing nothing further. Or after the swap I can re-image and swap back again and restore, thus keeping the chain of backup images intact.

Xpilot

 

In my view if you have critical data then store as a backup it in its native format. Use Karen's Replicator or something similar.

1000 jpg files are unlikely to all go bad at once in two different locations but having deleted your partition that had the data you want on it, it only needs one bit to be wrong for a reason and TI will not restore your backup. Result is no files anywhere. You can lessen this risk by having multiple backup files of course.

I only use TI to backup my system drive to prevent me having to reinstall the OS, my tweaks and programmes. I have two backups made a week apart which suits me. Others may need a daily backup.

I validate after a backup to my USB drive but not before a restore. I have successfully restored my system partition many times while booting from a VistaPE USB drive and TI Linux boot CD.

 

Hi

Forty years ago transistors were good or bad - never in-between.

This year semiconductors have reliability issues - good for some of the data some of the time.

In several threads on this forum I have read that failure to validate can be due to corruption as a result of defective RAM. In these threads is typically a recommendation of a memory test to be down-loaded and burnt to CD, and to run the test multiple times - since an error may be shy and not reveal itself on the first run.

A 10% failure rate is a number I pulled out of the air because I have never seen any number quoted in this forum.
After perhaps 100 validation I have never had a single validation error, so I know my RAM is at least 99% reliable.

I realise that restoration commences by deleting the destination partition, so if anything goes wrong there is no going back.

It is not obvious to me that the partition will remain empty if the archive is corrupt.

I do not fully comprehend "you will be left with unallocated space should an error be detected during the restore."
If the partition is deleted and then recreated and half populated until a validation error abort, does "unallocated space" refer to the area not yet populated, or does it mean Acronis will delete the partition a second time ?

It is obvious to me that if a 70 GB archive is being restored to a 100 GB partition, the entire 100 GB image is not all held in my 1 GB RAM.
I therefore assume that when enough 256 kB chunks of data have been decompressed into RAM they are validated by their checksums and then written into the newly deleted and recreated partition. Then the RAM is free for use on the next set of 256 kB data chunks.
I would expect that half way through the archive, about 50 GB of decompressed data will have been stored in the target partition.

I would like to know the result should a single bit RAM error cause a validation error when the target partition holds the first half of the archived image.
I believe the possibilities include :-
1. Completion of the restore operation with a corresponding error in the target partition, and no warning of any error, because Acronis was satisfied with the success of the optional pre-restore validation, and so has not bothered to re-compute as it decompresses for the actual restoration;
2. Discontinuation of the restore, and hopefully an error warning display that never times-out so it will be seen the next morning if the restoration was launched before the user went home. Finally, this could leave the target partition :-
2a. The correct size but holding only 50 GB of the original 100 GB;
2b. The correct size but holding no data because Acronis deleted all data upon failure;
2c. No partition because Acronis has deleted the partition a second time.

 

I am not sure about the detailed technicalities of the restore process.
All I can comment on is from several years of experience using TI to make hard drive images and restores.

To keep life simple I just make full hard drive images and do complete restores. I never run validations because it is quicker and and certain to actually restore to a swapped drive instead.

During a restore TI definitely checks if the restore is valid or not. My evidence for this is that from time to time a restore fails because of some corruption somewhere.

When restoring a disk or partition image there is no such thing as a partial restore. It either completes sucessfully or it fails completely no ifs or buts. A failed restore results in an "empty" hard drive or partition if only a partition is being restored.

It follows from my drive swapping mehod that a failed restore is really of no consequence because the original drive is not deleted and the situation is easy to recover from.

Because I had some rare restore failures I spent some time going through my hardware with a fine toothcomb to find the causes. All the usual software tests of the computer were passed with no errors however I did find three places where there were probable inter connection faults.
1. Over time a hard drive ribbon lead's connection had become partially unseated giving an intermittent fault.
2. The power supply plug to the main hard drive had a poor connection. I started to find my soldering iron to make good when I realised that there was another unused plug available.
3. A dust bunny or rather a fragment of one was picked up in the drive bay swap connector.

I did at one time collect some statistics on failed restores and these worked out at < half a percent over an 18 month period. However this is obviously specific to one particular computer. What I did find was that restore fails occured in slowly growing clutches and this encouraged me to find out the causes.

Xpilot

 

Based on reading this forum for sometime now, I don't think I've ever heard of partial restore. There has been the odd case where the user felt TI didn't restore some modification but it seems that this was often user error (they hadn't restored the right image etc). I have never heard of anything like 75% of the partition was restored. I would guess that there is more of a chance of an error when TI images a partition with filesystem errors and then restores it and the error causes the TI "adjustments" to go wrong. (TI does not necessarily put the contents of sector ABCD back into ABCD when it restores.)
I think it is good practice to run chkdsk /f or /r at some suitable interval.

I have no idea of the inner workings of the program beyond what I have mentioned. I will say that the philosophy on an image restore seems to be all or nothing as demonstrated by the "1 bad bit kills the validation".

I have no reason to be concerned that TI doesn't do its job for me but it does expect that you have good, solid hardware and it will detect some hardware problems. My only problems to date after doing many C drive restores are: marginal SATA cables which were detected in the validation phase and now bad sectors on the disk storing the archive which was detected by TI stating the file wasn't a valid TI archive file.

I also agree with Tatou about backing up data files in their native format rather than stuffing them into a proprietary container file. I use SyncBackSE for my data files.

 

I agree it SHOULD be successful, but fear it will fail.

If pre-validate is successful, that shows the archive is good, and the 90% reliable RAM was lucky this time whilst decompressing the image to extract and evaluate the embedded checksum.
After that the restore again depends upon luck, with a 10% to 20% chance that this time the RAM will be unlucky as the archive is decompressed.
Depending upon whether Acronis recomputes the checksums or is lulled into a false sense of security by a good pre-validate, there may be acceptance of erroneous data into the restored partition, or Acronis may recognise the partition is half populated, and I am wondering what it would then do.

n.b. A 10% RAM failure rate during pre-validation may correspond to 20% during restoration, in that during restoration the vulnerable data has to survive for twice as long in a hostile environment because of the extra time taken to actually write the data to the hard drive.

Regards
Alan

 

alan b

You have a concern that a restore may fail. You refer to hypothetical RAM failure rates. I do not understand what your post is trying to do.

Running a validation sucessfully is probably the best test of your RAM. If the validation fails then running Memtest for several passes overnight will confirm the RAM as good or bad. If it is good the problem is elsewhere and further testing is needed to locate the cause.

Have you actually suffered from any failed validations ? I guess that you have never had a failed restore or if you had you would have known that the failed restore would have left you with an empty hard drive.

The only real practical test to prove 100% that your backup and restore procedures really work is to carry out a restore. The 100% safe way to do this is to remove the current drive and replace it with a spare one before running the restore.
Many users run validations sucessfully and yet do not have sufficient confidence to test restore their hard drive. They would rather wait until they have a real disaster before commiting a restore.
A very high number will not have a problem but an unfortunate few will find out too late that they have lost their OS and data.

Xpilot

 

90% reliable RAM means the PC is broken and should be fixed. The RAM should be 100% reliable and the correct operation of it assumes that it is but unfortunately RAM errors don't often show up as RAM errors but as BSODs, corrupt files or obscure behaviour.

I don't agree with your statement that 40 years ago transistors were either good or bad, never in-between. I have seen a lot of transistors that exhibited intermittent working/not working behaviour or suffering from excessive collector leakage such that they would not go to a proper logical high level all the time. If a typical P4 PC was made of discrete transistors, I would bet its MTBF would be significantly inferior.

I wonder how many people backing up their PCs realize that the amount of data being backed up exceeds what a computer center would have backed up say, 25 years ago.

 

Once I know that ATI works on my hardware I only do a validation once in a blue moon just for the fun of it. They take too long and prove too little for my tastes.

Transistors and other solid state electronic devices certain can fail by degrees although typically, solid state circuits less often exhibit the slow, gradual decline of vacuum tube circuits. Then again, a filament breaks and tube circuit can die in an instant.

Certainly many of us have experienced the slow and inevitable circuit failure that comes from DC leakage across a circuit output or the intermittent oddities of a capacitor seeping electrolyte.

I'd bet that PC memory errors show up intermittently much more often than constantly.

 

Shieber

I concur with your view "that PC memory errors show up intermittently much more often than constantl"

BUT I WISH TO QUALIFY THE CONSEQUENCES.

If the intermittent error occurs whilst the data is being processed and delivered to the archive store drive, that archive is forever permanently corrupt, with only a 0.4% chance of validating if a corresponding error strikes whilst being retrieved from the drive (0.4% assumes an 8 bit checksum.)

If however the data survives the transfer through RAM into the archive drive, we then have an image with a good chance it can be read back and again validated.

My concern is that a good archive has a good chance of validating before the restore, but the subsequent read for the actual restoration WILL be independant - and according to Murphy's law it will go wrong at the least opportune time.


Xpilot

I had one restore failure. It was a non-event. It rebooted harmlessly.
This was after several good restores from my external drive.
After quietly whimpering in a corner I decided that if I was lucky the reason would be that the power saving circuitry/software on the external drive had seen Windows shut-down and cut power to the disc, and the reboot and restoration started before the disc was back up to speed. I never tested this hypothesis - I disabled the software and switched of the circuitry so the drive now always spins whilst power comes out the wall, and this has been a perfect cure - so far.

I have never had a "real" failure to restore.

What do you mean by "the result is an empty drive"?
I have C:\ plus several other partitions on my internal drive.
I assume a failure to restore C:\ will not disturb the other partitions.
Are you saying that C:\ will be its original size but zero contents,
or are you saying that there is no more C:\ ?

I will accept that you have experienced a failure to restore and the result has been "an empty drive".
Have you only experienced this after a "normal" restore without pre-validation ?

My fear is that whilst an independent validate BEFORE restore will avoid grief from an archive that was good but has since become over-written / corrupted, if this validate is NOT a completely separate action, but included as a precursor to the restoration, then POSSIBLY Acronis will remember it has just checked successfully all the checksum and won't bother to repeat the exercise.

I am sorry I am beating this to death, but my career has included design of all the hardware and software for computerised intruder alarm systems that protect Nuclear research facilities and Nuclear submarine sites, and these systems were fully operational 24/7/365 for years on end, whilst more recent system based on Microsoft operating system would BSOD and reboot several times a day.
Now I am retired I cannot shake off old habits.


Seekforever

I apologize for a slight exaggeration for effect.

Please forgive my rambling reminiscences.

With over 40 years experience designing electronic equipment I accept that individual transistors has problems, BUT they were quite consistent under consistent circumstances.
Leakage current remained the same for given voltages and temperature,
with increases in leakage being due to the corresponding wasted power causing an increase in temperature and possibly thermal run-away.

I would say in the good old days the defects were mostly hard defects, with some anomalies such as Safe Operational Area limits.

With the advent of microprocessors and RAM we had soft errors as well as hard errors.
Hard errors like frozen row/column select lines were easy to screen out.
Pattern sensitivity by which data at one address could be disturbed by data changes at another address - less easy.
Then along came cosmic particles to flip the status of memory bits.
Then came schmoo plots and Q.A. departments.

I was responsible for software in video frame store multiplexors that become unreliable after a given production date. This was rapidly identified with a "process shrink" by the micro-processor manufacturer. We obtained alternative devices and resumed production and sale of good equipment.
A large amount of less reliable product was already installed, and the chief share holder / Chief Executive Officer could not afford a factory recall.
I eventually discovered that if a video timing pulse occurred within a 10 nanoSec window of opportunity, and if at that time the processor was reading Register 'A', and if the processor instruction code was at address 0x??FF,
THEN the next instruction that was read was 0x??FF + 1 MINUS the carry,
eg it would move from 0x9CFF to 0x9C00 instead of rolling over to 0x9D00.
Having found how it went wrong, a quick program to identify within the final object code where Read 'A' instructions were sitting at 0x??FF, and then some NOP patches in the source code to shift them out of harms way. Then a new issue of plugin replacement EPROMs for the next scheduled service visit, instead of a major recall factory replacements and accidental breakages.

Sorry, that was 20 years ago, but I still live with the nightmare of tracing cause and effect with a cheap logic analyser, until I found a 10 nanoSecond anomaly within a 1 hour trace.

I still remember when PNP germanium transistors were superseded by NPN silicon, and circuit diagrams reverted to the sensible scheme of positive rail at the top. It never got better than that ! !

I did MTBF calculations for Defence work 40 years ago.
I agree that a P.C. using discrete transistors would have an inferior MTBF.
I think such a P.C. would hardly live long enough to complete a Windows Installation ! !

Regards to all
Alan

 

alan_b

I understand that you have not had a real restore failure which resulted from a corrupt image.
There was some break in the power supply but the good image on the backup drive was not affected , nor should it be as it was only being read. It follows that restarting the restore resulted in sucess. When you restarted the restore did you do this from Windows ? If so it must have failed before you hit the TI proceed button had any real effect because the first step is to delete the target partition/drive.

Otherwise if you had tried to restart the computer before re-running the restore the partition, or if the restore was for the whole disk, would have been unallocated space.

I notice from your description that you have been starting restores while booted in Windows. There is no harm in that but you should at least test your recovery CD, which you made earlier?, to make sure that it can see your backup images and target drives. This is to be prepared for the possibility of main hard drive failure or to use when upgrading to a replacement drive.

To answer you specific questions:
I used the term drive to mean a partition or a complete hard drive. depending on whch sort of restore was being made. I agree I was not explicit enough. Only the target partions will be affected.
In the case of your C:\ there would be unallocated space instead of its original content.
I have never tested this particular scenario as I always run full hard drive restores. It is rather a moot point whether the unallocated space is still called C:\ or not because Windows would not start and so would no be able to mess with the other partition's designations.

I only backup and restore full hard drive images. I always restore to a previous generation hard drive after removing the current drive in its caddy. I have no need to bother with validations.
The tiny number of failed restores I have experienced have always had an Aconis pop up saying that the archive was corrupt. I note that these fails happened at different places during each failed restore. My guess is that TI performs check sum validations during a restore and when some corruption is found the whole process fails.
Because of the way I swap over maindrives and keep several backup images on a second internal hard drive in a secure zone a failed restore is of no consequence.

Incidentally I find that swapping hard drives and running a restore takes far less time than running a validation. I gather that multiple re- validations can be avoided in some newer versions of TI but I am not ready to give up my trusty secure zone.

Xpilot

 

Xpilot

Thank you for the extra information.

I did launch the restore from Windows, and in the usual fashion after PROCEED and a brief delay the PC shut down and rebooted and Acronis took over.
Fortunately Acronis recognised the absence of the archive (because the external drive was not up to speed), and it aborted partition deletion.
After the abort the PC rebooted and I logged on and found no errors - everything was fine.

The restore did not automatically repeat itself.
For a week or so I continued to use the PC as normal, but refrained from risking another restore until I guessed the probable cause of the problem and decided how to avoid a repetition.

I had a 30 GB Hard Drive. Before replacing with a 160 GB drive I created the CD (I never realised till later how dangerous it was to not have a CD).
Upon replacing the drive I was able to use the CD to restore the latest image onto the new blank 160 GB.

I now realise that LINUX drivers may fail to be compatible with the hardware,
and it is essential to determine compatibility before rather than after disaster.
I know that my CD with its LINUX drivers will restore old images to C:\
I ASSUME my CD will restore any new images that can pass validation.
QUESTION - Is this safe, or is this hardware compatibility subject to "data pattern sensitivity" ?
i.e. Could a different image suffer corruption when restored because the drivers / hardware is only perfect with a data sequence that never includes a particular pattern (e.g. 0x55,0x55,0x55,0x55 or 01,02,03,04) ?

I am not happy about swapping drives to ensure full restorability of an image.
My laptop needs a very heavy slap and tickle to remove the battery pack and switch over hard drives, and now I am retired if I break something I no longer have access to surface mount manufacturing / repairing equipment.

I now have several partitions on the internal drive.
C:\ is now 15 GB, and never has less than 50% free space.
R:\ is 10 GB, and every new FULL image of C:\ is now restored to R:\,
and a quick folder/file comparison confirms only the usual differences because Windows cannot sit still but is always shuffling and logging ! !
C:\ to R:\ comparison is often repeated, and I then observe :-
1. Unusual changes due to Malware;
2. Horrible changes because Microsoft knows what is best for me and insists upon stealth updates regardless of my settings;
3. Things I have downloaded and forgotten about.

Actually "1" never happens because my malware protection has not yet failed me.
I regret that "2" does happen - they even planted a EULA for Silverlight which appears to give them and their partners the right to share any information any of them find on my PC ! !
I am sorely tempted to tell my Firewall to block the Microsoft servers ! !

I always do C:\ to R:\ comparison before making a new image so I can reconsider whether to retain forgotten downloads.

Regards and thanks
Alan