Integrity check fails - Comprehensive troubleshooting doesn't fix it

Backup & Recovery 10 Free Edition

Warning: This is long because I tried lots of different methods to troubleshoot this problem.

Paragon Backup & Recover 10 Free Edition
Windows XP Pro SP-3

Did a full backup which completed with no errors. The program option to "Control integrity" was enabled which means checksums of the backed up files get saved into the archive. After the full backup was done, I ran an archive integrity check. It failed with:

Operation failed
Invalid archive file

Error source: Hard Disk Manager
Error code: 0x10025​

No other information was provided in the popup error dialog. So what is the "Hard Disk Manager"? Is that some component of B&R10Free? Just where am I supposed to lookup the error codes (since they're used instead of providing plain language descriptions)?

Note that before I performed this full backup that I already checked my hardware. I had ran memtest86 for 6 passes with no errors, Windows Memory Diagnostics for 5 passes with no errors, and Western Digital's Data LifeGuard Diagnostics found no problems with its quick and full check of the hard disk purposed only for backup storage. So it's not the hardware at fault.

I rebooted into Safe Mode but the integrity check still failed. I did another full backup while in Safe Mode and then did an integrity check on that one. It failed, too, this time noting:

Integrity error has occurred
Archive integrity error between sectors: xxxxxx and xxxxxx!​

So is this an error reading sectors off the disk where the file was copied? Or is this a mismatch between the newly generated hash value for a file within the archive and the checksum that was stored in the backup (i.e., are these sectors from the image in the backup file)?

From what I’ve read, causes for the integrity/verification/validation/or-some-other-name test to fail are:

1. Bad memory: corrupted content in memory could result in invalid data used in the compare. Thorough memory testing has ruled this out.
2. Bad hard disk: sectors/bytes won’t read reliably or at all. Thorough disk testing via “chkdksk /r” and using WD diagnostics rules this out. Also, other file operations on this same hard disk (the one used for the backup store) have not resulted in corrupted or damaged files.
3. The checksum on the file(s) saved in the archive file generated during the backup process were invalid. No way for users to test this cause. If the checksums stored in the archive are wrong, users have no way to fix this. There is no utility provided that will generate a checksum of the file/sectors stored in the archive file that operates separately of B&R10Free to see what it generates and then extract the checksum that got stored in the archive for when the file got backed up and put into the archive file along with generating a checksum on the original (and unchanged) file that got backed up. With these 3 checksum (on original file, on archived file, checksum stored in archive), the user could see if it was the stored checksum in the archive that was wrong or if the checksum for the original and archived files were really different. It is unclear how the integrity check works.
   • It may generate a new checksum from the archived file/sectors and compare it against the checksum value created during the backup that got stored in the archive. This checks only the archive contents are valid but works only if the stored checksum is the actual checksum of the content that got stored in the archive. It also presumes that the EXACT algorithms used to compile the checksum during the backup is the same used when performing an integrity check. This scheme would be faster because only one checksum need be generated: the one for the archived content.
   • It may generate a new checksum against the archived file/sectors against a new checksum generated on the original file. The problem here is the file may have changed since the backup. Even if the file didn’t change, the slack space in the cluster isn’t part of the file but its content could change on the hard disk versus what was in the slack space when the sectors got backed up. This would be a slower integrity check because 2 checksums would have to get generated: one on the archived file and another on the original file.
4. The new checksum generated from the files/sectors stored in the archive file don’t match the stored checksums in the same archive file.
   • Could be the method used to generate the checksum during an integrity check results in a different value than how the checksum was computed during the backup.
   • Could be the file was stored incorrectly into the archive file. That is, the file on the disk is okay but its copy in the archive is different (i.e., corrupted).
   • Could be the archive file got corrupted sometime later; however, I’m doing the integrity check immediately after a backup completes.
   • Could be anti-virus software is interfering with generating correct checksums; however, disabling the AV program or a reboot into safe mode where the AV isn’t active indicates this is not the source of the problem.
   • Another cause could be the compression. If the compression doesn’t permit an accurate calculation for a newly generated checksum on the contents of the archive then they won’t match against the checksums stored in the archive. As noted in #3, the user has no means of separately generating checksums on the original and archived files/sectors and to extract the stored checksum to see which ones match and which don’t.

Although B&R10Free is supposed to revert to using its own snapshotting driver if VSS fails, there is no indication given in the progress window during the backup to indicate that the backup program has discontinued using VSS (Microsoft’s Volume Shadow [Copying] Service) and switched to Paragon’s driver. VSS has been shown to have problems. Perhaps VSS has improved in later NT-based versions of Windows (Vista and 7) but I’m still back on Windows XP. VSS won’t work with FAT16/32 and requires NTFS (on, I believe, at least one partition). On all 3 hard disks, there is just 1 partition that spans the entire hard disk and NTFS is used in each. To test if VSS is the culprit, I configured B&R10Free to use “Paragon Hot Processing” driver instead of VSS. When I started the backup wizard, I checked the option to view that job’s options and “Paragon Hot Processing” was selected. After the full backup completed, a following integrity check still failed.

I noticed that I was selecting the entire archive to include in the integrity check. So instead of picking the archive (which was a disk backup), I just picked the volume (C that spans that entire hard disk. Nope, the integrity check failed, too.

As yet another test, I did backups with no compression. If the compression scheme is unreliable or otherwises causes problems in generating new checksums against the archived files/sectors then there will be a mismatch against the checksum generated at backup time that was stored in the archive. The disk properties showed 28GB was used and the prior full backups were creating 16GB “.000” files (I disabled image splitting since the backup location is a hard disk using NTFS). So an uncompressed backup would take longer but finish in maybe under an hour (the compressed full backups were taking 28 minutes). Without compression, the full backup now only took 18 minutes. Shorter backup time but a larger archive file (but only 20GB versus the 28GB consumed in the volume). The integrity check failed.

I choose not to split up the archive into numerous files because it shouldn’t be required. A separate hard disk was purposed solely to store backups. It had only 1 partition which was NTFS formatted. However, if B&R10Free has problems reading huge files then it might be generating invalid checksums on its archived content. With the image split up into 4GB chunks, a following integrity check still failed.

Another test was to not select the entire partition to backup but just the volume. That is, instead of selecting the disk (with its 1 partition, 1st track, and MBR), I just selected the volume (C to backup. The integrity check failed.

Despite the integrity check failing and claiming the archive is corrupted, I tried pulling all the files out of the archive and saved them to an alternate location. I right-clicked on the archive and mounted it as drive N: and ran “xcopy n: c:\tempx /e /h” (since C: had enough free space to hold all the files). No errors reported by the xcopy operation. The xcopy command failed but not for what appears to be a “cannot read sector” or checksum error. The error after copying over 10,000 files was:

…(other files)…
Nocuments and Settings\lee_hodsdon\My Documents\My Videos\Desktop.ini
Access denied
Unable to create directory - C:\tempx\%userprofile%\My Documents\My Virtual Machines
10837 File(s) copied​

I don’t quite know if the error was bitching about the desktop.ini file or “unable to create directory”. This folder is not really a folder. It is a junction point. That is, it is a link that points somewhere else. VirtualPC 2007 wants to dump its virtual machine definitions under My Documents and I don’t want them there. Even if while defining a VM it is told its store folder is elsewhere, VPC2007 still creates this subfolder under My Documents. To alleviate it into thinking this documents folder already exists but actually have the VMs stored elsewhere, I created a junction point. VPC2007 sees the “folder” but which is really located elsewhere. This junction point on the C: drive under My Documents points to a folder on my D: drive.

%userprofile%\My Documents\Virtual Machines (this is on drive C

is a junction point to:

D:\VirtualMachines\VirtualPC​

I can see why xcopy failed because it cannot create junction points so it cannot create the “folder”. I have to wonder if B&R10Free (and B&R2010FreeAdv that I tried earlier and had the integrity errors) don’t know how to properly handle junction points. For one, they should NOT follow junction points as this can lead to redundant data in the archive. As another example, I have a junction point under My Documents which points to C:\References. If the backup program follows the junction point then it would backup the files under there and later backup again the files it found under C:\References. While the expectation is that a restore could recreate junction points because they are inside of NTFS that doesn’t mean the backup program actually does handle junction points. I know that Acronis True Image does NOT follow junction points (but it does create them successfully on a restore). So I know why the xcopy aborted when it hit a junction point (but it did manage to copy the 3GB of files before it hit this junction point. I then ran xcopy on the rest of the 14GB of files in N:, the mounted archive which copied okay. So other than the junction point problem (something xcopy doesn’t support), all the files from the archive copied out okay. Maybe Paragon’s offerings don’t handle junction points, especially if they span across volumes.

I have no spare drive or another same-hardware host in which to test restoring the entire archive (which is a disk image with 1st track, MBR, and volume). So I can’t do a disk or partition restore, just load the archive as a drive and copy from there to check if there are any errors accessing and copying the files out of it.

As a last resort, I rebooted using the recovery CD. Perhaps the snapshot “technology” (Microsoft’s VSS or Paragon’s Hotshot) really aren’t stable. After all, they are trying to backup an active partition versus making sure it is completely quiescent during the backup. After I booted using the recovery CD, and after a length wait to get past a progress bar and then a blue screen after selecting Normal (i.e., the blue screen has no progress info), I tried to check integrity of the full backup. Got some error about invalid operation code. So I created another full backup using this boot-time version of the program. After it completed, I tried an integrity check but still got the invalid operation code error. Well, that was a wonderfully worthless tool.

Figuring the backup created at boot (outside of Windows) would be of a quiescent partition where no changes were occurring and no snapshotting technology need be used (VSS or Hotshot), I rebooted into Windows and ran an integrity check on that backup. It failed, too.

I’m not getting any errors during the backups. It is only afterward when I later run an integrity check that the product claims the archive is corrupted. Something else that I noticed is that the hard disk is properly identified when I selected it for backup. After the backup completes, its archive is listed as “Basic Hard Disk 1 (WDC WD1600JS-00NCB1 SCSI Disk Device )”. It’s a SATA drive and those are seen as SCSI devices. Yet when I go to run an integrity check, and when selecting the archive to test, it says “Basic Hard Disk 1 (Unknown Model)”. So the disk info is present in the archive but the integrity check suddenly can’t figure out what it was from the archive.

I’m beginning to strongly suspect that either the integrity check doesn’t compute checksums using the exact same mechanism as used when doing the backups and storing the file/sectors inside the archive or the checksums computed at backup time and stored in the archive are wrong. This isn’t new. I forget which backup product it was many years ago but I happened upon one that was storing incorrect checksums in the archive which meant the verification or integrity check would always fail. It didn’t store the wrong checksum all the time. There were specific criteria found under which the wrong checksum got computed and then stored in the archive.

The integrity check appears worthless because it cannot be relied upon to correctly validate an archive. It looks like I can extract all the files out of the archive. I can’t use the product’s own integrity check to verify the archives are usable. Well, I certainly don’t want to find out months later or even the next day that an archive is unusable because the integrity check is unreliable.

 

Found the cause. I have an old Abit NF7-S rev2 motherboard. It has 2 SATA ports but this was when SATA was new. Apparently the PCI bridge to SATA controller really doesn't perform to full SATA specs and something about regenerating checksums from the archives would trigger the integrity errors.

I noticed that I could save my image backups to my IDE hard disk and the integrity check worked okay on each backup. It was when my backups were stored on the SATA disks that the integrity checks failed. So I reconfigured my computer so, as before, it boots to Windows which is on a SATA drive; however, I've moved Windows to the big 500GB SATA drive and will exclude folders in the backup that don't need to be included in the backups, especially since these folders contain huge files (.iso files for CD/DVD images, games, downloads, shared folders, etc) and which can be recreated or re-retrieved.

So my image backups, with some exclusions, will go onto my 120GB IDE drive. That won't give me near as many backups as I had before (but I've also gone back to my paid Acronis True Image 11 backup program that gave me incrementals to minimalize the size of the backup images). I'll later get a bigger IDE hard disk. Yeah, I know, it sucks that going forward in hardware upgrades in new hosts that I'd be migrating over an IDE drive but this host seems to work best with IDE drives. The SATA support is solid enough for me to feel comfortable using it and I have done so for several years without problems. It's just the integrity checks on the backup images that was causing me grief when they were getting stored on a SATA drive.

So I moved Windows from the 160GB SATA disk to the 500GB SATA disk, use the 120GB IDE disk for now to store image backups (and will need to get a bigger IDE drive later), and will keep the 120GB SATA disk for some other types of backups (some of those that I excluded from the image backups of C.

Basically the problem is with the SATA hardware on my motherboard causing the integrity check errors. Despite "chkdsk /r" not finding problems, and Western Digital's bootable and Windows-time hard disk diagnostics not finding any problems with the SATA disks. Whatever the low-level cause in the SATA hardware, the backups seem happy now to get saved on the IDE disk. It's an old computer (probably around 7 years now), single CPU (AMD Athlon XP 3200), 2GB DDR2 memory, and way more disk space than I need and works for what I want. I probably spent over a week trying to resolve this frustrating integrity check error. A hardware reconfiguration looks to have fixed it.

 

CONGRATULATIONS !! I've been trying for m-a-n-y years to backup & validate disk images using several different backup & restore pgms without success on my single HD Toshiba Satellite laptop to external USB. Some of the backup pgms would work occasionally in LINUX mode (booting from CD) but nothing worked consistently.

I have tried (using 2 different pgms) backing up to a second partition on the HD and then copying the image to USB drive. One pgm validated the image but the other did not. Since another internal HD cannot be installed, what are my options ?

Thanx,
Bewildered Bob

 

Other than uninstalling all security software (anti-virus, anti-malware, anti-spam, etc) from your host, disconnecting (from the network), and seeing after a reboot if the backups will complete and also verify on the integrity check, one possible workaround that I can think of is to repartition your single hard disk. Backups to your internal hard disk may work with a following integrity check. Resize your current partition (make it smaller), create another one in the unallocated space (big enough to hold at least one full backup), and specify a backup location on the new drive in the new partition. See if that works. If so, "[x]copy /v" the backup from the new drive onto your USB drive. If concerned that the /v verify option isn't that good, you could follow with a "fc /b" command to do a bitwise binary compare between the original backup file and the copy on the USB drive. Then delete the original file on the internal hard disk partition to make room for the next backup job. I suspect you could write up a batch file that does the copy for you and add it as a post-processing command to the backup job (I don't recall if Paragon lets you define pre- and post-processing commands; if not, use Task Schedule to do the copy script sometime long after the backup should finish).

Of course, this is all predicated upon the backup completing okay and then verifying okay when stored on your internal hard disk. You can't save the backup image into the same partition that you are backing up. Some backup programs will require that you completely lock the source partition during the backup if you are saving to a partition that is on the same hard disk as the source partition so a reboot may be needed. If that's true then scheduled backups would be a pain because the reboot probably would wait until you said it was okay or after the reboot and on login you would get prompted to enter your login credentials (unless you choose to dump security and leave your computer wide open to let anyone boot and use it).

 

Has anything been done for this problem? I can't find an answer to everyone's archive integration problems, including mine.