Corrupt/Can't Verify Corrupt Archives: Let's uncover the problem!

Hi Folks,

The Iomega USB2.0 external HDD that I use is actually supplied with a 'double' USB cable, one of which provides the data connections + power and the other simply provides another auxiliary power connection. In other words, this arrangement uses two USB connectors on the PC. This is to ensure that sufficient peak current is available to the external HDD. In the case of the Iomega drive, this peak current is 900mA, which is more than can be comfortably provided by one USB connection alone.

JPC

 

I think you have to pay careful attention to what the device specifies. An examination of my 2 USB drive cables shows they are about 1M long. Haven't checked to see if they are the maximum permitted though.

 

I was not aware that devices specify max cable lengths. Are these shorter then the USB cable specs ?

The USB 2 specs state that the maximum cable length is 5 metres. There are other constraints such as transfer speeds that it must be capable of performing.
I chose a max length of 2 metres as not all cables are equal but even some of the less good should manage to perform to spec at less than half the maximum length allowed.

 

I think you are right, if the device conforms to USB2 then it should support 5M cables or 3M for USB1. I'll consider my shorter cables as being just a "money-saver" for the supplier until I run across some more definite info.

According to www.usb.org FAQ you can put a device up to 30M away if you use 5M cables and hubs. I wonder if anybody could validate any TI archive with that setup?

 

My max cables for USB 2.0 hard drives are less than 1 meter. I've had no problems with verifications, and I'm beginning to wonder if my overly conservative cables are the reason.

 

One person asked if I could describe the one cable that provides "OK" rather than "corrupt" validations. I measured it, and it is about 6 feet (i.e., roughly 2 meters). There are no markings or labeling, so I cannot provide an IEEE rating or anything else. However, it happens to use a transparent plastic sheath, so I can see the conductors, and it has a very nice braided shield. Also, it is substantially thicker than my other cables, but I cannot tell without cutting into my cable as to how much of that thickness is due to the plastic sheathing and how much is due to the shield.

If anyone wants, I can go down to my shop and get my micrometer and measure its exact thickness and the thickness of my thinner cables.

Another person has stated several times that I should run the MD5 checksum test and also stated that somehow my cable testing confirmed that bad checksums was the problem.

Actually, I proved just the opposite.

First, just to be clear, and to make sure I satisfy that poster, I just downloaded and ran the eXpress checksum calculator. I did a checksum on a TIB file that was one that had caused problems on an external drive. It was residing on an internal drive. I then connected the drive on which I had done my tests, and used the "bad" cable. I then copied the TIB file and used eXpress checksum and, of course, the checksum (all of them--it provides three different types of checksum) was IDENTICAL. I then copied the file from the external USB drive back to a different location on my internal drive and re-ran the eXpress checksum.

Once again, the checksums were identical.

Of course I knew this would be true before I ran the test. Why? Because, as I stated in previous posts, it is IMPOSSIBLE for a file that tests bad ("corrupt") when created and validated on an external drive, to somehow be "healed" merely by copying it to the internal drive. It is true that a file might get corrupted by copying, but it sure as heck isn't going to magically change from being corrupted to being good. Actually, that would be beyond magic.

What's more, Windows uses its own checksum when doing file copies, and if the file fails to copy, you'll get a Retry/Fail type of notification. It happens so rarely that I'm not even sure of what the Windows alert says.

What I've tried to get across is that the problem we are dealing with here isn't an issue of whether the file ultimately gets copied correctly. Instead, what I think is happening is more "under the covers," before the final checksum is calculated, and involves errors, retries, and corrections that are done by the O/S during the copy.

What I need is a utility that would somehow keep track of these errors, much like the Nero "CD-DVD Speed" does for CDs and DVDs. If any of you are familiar with that tool (and Plextor has a similar one for their drives) that is exactly the kind of tool needed here.

I truly believe now that the problem is that True Image is somehow obtaining and using this information on errors and retries, and even if the errors are correctable, if there are too many, it reports the archive as corrupt. I'm just guessing on this, but that's what it feels like. Remember, in MOST cases, the archive is NOT corrupt, because it can be copied to another drive (usually an internal drive) and it validates OK.

 

Thank you. (From one obsessive to another.) This seems dead right. Inappropriately clever code, now become part of a fixed core.

I'd bet that 11 shows the same problem. Not that I'll be testing it.

 

Hello Johnmeyer,

Indeed, it certainly would be magic if copying a file from one location to another changed a corrupt file to a non-corrupt file!! However, the sole purpose of the Large File Copy/MD5 Checksum test is to determine whether your computer's USB sub-system (including the USB cable), in association with the external hard disk enclosure's USB to IDE-bridge chipset, can successfully transfer a very large file (e.g. TI image file) without causing data errors. Believe you me John, many users have found to their surprise that this has not been the case.

A red herring surely? Windows will pop up a Retry/Ignore/Abort warning if it fails to read or write a particular data packet. As far as I'm aware, it has no way of knowing whether the data packets contain faithful copies of the zero's and one's going to and from the drive's embeded electronics. If this isn't the case then I stand corrected.

Regards

Menorcaman

 

That's a great idea. Could you let me know more about that, such as what hardware is involved and where I might purchase it? I believe this issue would all but go away with such a setup and be much faster as well... and still provide easy off site storage by swapping drives periodically.

Also, I wonder if eSata external drives have the same problem. Anyone using those?

 

While self-healing might be nice, I am not at all sure that it would help with the problem that ATI is presenting to some (many?) users. I.e., it isn't really clear that the problem is bad tibs. What's known is that ATI can't read them on some setups and is calling them invalid. That might be because ATI isn't writing them properly and they are actually invalid, or not creating the checksums correctly (and slef helaing might help), or maybe AIT just can't read tibs, even good ones, under certain conditions where other software read/write files just fine. We do know from various reports that some tibs that ATI declares to be invalid are demonstrably actually valid!

I agree that there is a fundamental prgramming problem in ATI and I think this prob is moving customers to other products -- not that others aren't without there probs but if you can't get a ATI to write and restore tibs, it's not any use to you and that's what folks are faced with that see this prob on their machines. I've had to stop recommending the product because it's just too embarrassing when this invalid tib prob pops up.

Before they add features in an "upgrade", they need to figure this prob out and resolve it.

 

This is the brand of racks and removable drawers I chose there are plenty of other maufactures about. I just jumped for this one as soon as I found it because I wanted to get started ASAP.
http://www.startech.com/Product/Category.aspx?MLID=7&WCLID=62&WCID=190&c=UK

Several users have commented that it is a complicated process. It is NOT.
The sense of comfort when one restores to the next hard drive and can then walk off with a ready to go drive to put under your pillow takes some beating!

These racks and drawers are available for all kinds of drives.

Xpilot

 

My system contains two removable SATA HD racks. They're pretty straightforward to install but a couple of things one needs to be aware of:

1. The racks fit into a vacant 5.25" drive bay rather than a 3.5" HD drive bay. Therefore you need to ensure you have sufficient spare 5.25" bays.
2. If the removable rack has one or more cooling fans they can be somewhat noisy (each of my racks contain three ). I would not recommend the use of fanless models unless the rack was a skeletal type (i.e. no top and bottom covers).

Regards

Menorcaman

 

I wish I knew for certain exactly how the various copy/validation(if any) mechanisms work in Windows for a file copy. However, I think your statement is optimistic since there have been reported cases of the checksum validation failing and Windows has been quite happy.

The errors detected appear to be on reading (read after write to do a check is slow so systems typically don't do it by default) when the CRC error is detected. This means the ECC attached to the sectors couldn't correct the error. Note that the ECC exists primarily to handle problems caused by drive problems such as platter defects. If the data gets corrupted on the way to the disk, the ECC is written to match the bad data and thus when the disk is read it will not trigger an error.

 

I have to “throw my hat into the ring” now! There is no doubt that TI is a good program, capable of doing “what it says on the tin”. I have, however, suffered the interminable “E00070020: The archive is corrupted” message. This happens all too often for my liking.

I may be repeating what has already been said in this thread (and if so, I’m sorry,) but here goes…

I contacted Acronis regarding the “archive is corrupted” message. After a few days, they sent the following advice…
____________________________________
Could you please do the following?

• Please download the latest SnapAPI archive file from http://download.acronis.com/support/SnapAPI_l_s_e.zip
• Unpack the archive
• Install the unpacked MSI package and check if that fixes the problem
• If the problem persists, replace C:\WINDOWS\system32\snapapi.dll with the one from the downloaded SnapAPI archive
• Reboot the machine
• Reproduce the problem
• Get the log file without closing any application windows (including the error message windows if there are any). The log file will be created at C:\ . The name of the log file will be snapapi [date-time].log

Could you please also try to make backup archive using Acronis Bootable Rescue Media?

If the issue remains, please let us know what the destination place for the backups is.
____________________________________

This I did, but the problem persisted. I told Acronis that, and I’m still waiting for the reply to that email after five weeks!

In the meantime, I came across this posting on this forum. I thought that it was a good thing to experience other peoples’ findings here. Although I’ve taken a while to do it, I’m actually getting to the point here!

I generally found that I could make a “base backup” that would verify possibly 80% of the time, but when I tried an incremental backup thereafter, the whole thing went “belly up”. On subsequently verifying the image, the dreaded “archive is corrupted” message is all I get. In all this, I appreciate that error checking during read and write processes are critical. Errors are pretty random here though, and can happen at different times during verification (mostly, I have to say, at the very beginning!)

To experiment, I created an “Acronis Secure Zone” partition on my slave hard drive. Initially it appeared okay, but once again, when I added an incremental backup, this also caused the verification problem.

To prove Acronis does work, I’m writing this on a system which was originally created by making an image on my slave drive. I restored this image to another drive, located in a 3.5” drive caddy, using a USB interface (wrote MBR + Track 0 during restore.) I then took this drive out of the caddy and made it a bootable drive inside my main computer case. After a minimal amount of hassle, it could now be my main system drive if I wanted.

Although I’ve not really offered a concrete answer to the original posting on this, I hope that my initial contribution has shown that although the verification thing can be a lot of trouble, there can be positive results.

 

It sounds like something in your PC is on the edge if you can validate 80% of the time or adding an incremental causes problems. Since this seems to happen on a slave drive which I assume is mounted as an internal drive rather than a USB it is less likely your problem is with the disk system - note that I said less likely, not impossible.

Have you done the ususal steps to eliminate/locate the cause?

Did you try making the backup, validate, etc with the bootable rescue CD as Acronis suggested? This removes any possible, if unlikely, Windows/application conflicts when making an image with Windows running. The rescue CD works on a static disk. Also, running the rescue CD is a very good idea since it is the media that must run to do a recovery if your HD fails.

Check the disk surface and structure by running chkdsk X: /r on ALL your partitions. Replace X with the drive letter of the partition being tested. Reboot for C required.

If you overclocking your system or if you are using aggressive memory timings, put them back to the defaults.

Download from www.memtest.org Memtest86+ version 1.7. Let it run overnight. Another thing to try since memory diagnostics aren't perfect, they don't test the memory in the actual conditions used when you are running an application like TI, is to remove part of your memory. If you only have one stick or if you have 2 sticks on a dual-channel motherboard this is not possible.

If you have a second PC, can you validate an image that fails validation on the first PC with it and vice-versa?

 

Some folks have the validation problem with ATI and not other software, including other backup software. So even if some cases are member probs- it's still a prob that is going to hurt ATI in the long run until it finds a way to handle the errors better or handle the hardware better.

 

I am not convinced that corrupt archives has anything much to do with the way the TI imaging/restore program works.
If it were such a basic fault how is my computer able to perform many hundreds of image/restore cycles with virtually no problems at all ? I have had a very few restores fail (now down to about 0.25%).
With this low level of failure it is just not a problem to worry about. Particularly as one should have taken precautions to be able to recover from a corrupt restore however remote the chance of one happening. Reliance on just one image or more than one from just one source is asking for trouble whatever backup program is being used. In other words an HDD with a hundred backup images is worthless if it is broken.

I am convinced that corrupt archives mainly happen for a variety of hardware reasons or software conflicts with other running processes.
A high or regular failure rate makes the investigation process fairly straight forward because as each possible cause is eliminated the effect will soon be seen.

I would start with the easiest things first. Pick your own running order.
1. Eliminate software conflicts by running TI from the rescue CD for imaging validations and restores.
2.Eliminate linux driver problems by running TI through the hoops using a Bart PE CD with TI plug in.
3. Run the latest MEM test overnight.
4.Eliminate external drives by fitting an additional internal one if only on a temporary basis.
5. If there is a lot of RAM fitted remove some to bring it to a minimum workable level. Perhaps 25% of the motherboard's maximum if that is still enough to run the OS.
6. Turn off any over-clocking and indeed reduce the timings to a level below the normal maximum.
7. Run CHKDSK R on all HDD partitions. Examine the results and act accordingly.
8. Contact TI support. Should have been No1?

If these steps have not identified the general area where the problems are occuring I would be disappointed but not surprised because there are probably a few that I have not thought of including.


Once the faulty area has been identified detailed checks can be done mainly by substitution of components.
An example, if USB drives are suspects, having swapped them about and tried various leads etc. I would try fitting a known good make of USB controller in a spare PCI slot. Anyway a set of another4 to 6 USB ports can usually find good uses. External USB hubs shoud not be used. Even those which have their own power supply allocate the bandwidth available from one motherboard connection pro rata to the added hub ports.

Modern RAM is pretty robust but like any other component it can have faults and this affects TI more than many other applications. It should also be remembered ( forgive the pun) that few today have inbuilt checking procedures such as ECC or parity checking. Indeed some PCs will have parity checking turned off as a sacrifice to god of speed. I would rather take a 2% performance hit if it meant reliable restores.

False positive corrupt archive reports are a bit of an annoying side issue. It certainly was an Acronis issue with early builds of TI9 home. They happened when a validation was run from the recovery CD. Subsequent builds corrected this problem. I am not aware that this has been identified as happening with Version 10 or 11.

Xpilot

 

Thanks to all who have responded to my first posting in this forum.

To appease all the “please check that your file system is reliable” folk, I have indeed conducted a chkdsk test, using the /r switch (at great expense of time) on all my source and destination areas! I must admit it does help to prove that your file system is in good order before going any further. Many thanks for pointing this out anyway!

I have also conducted a pretty thorough check of my RAM. I have to say here that I actually replaced one of my memory chips some while ago, for having consistently failed the Memtest86+ program!! Since then, no problems in that area, even leaving it to go all night.

I’m still convinced that my problem is a hardware related thing. As such, I’m giving my most important system properties below. I’ll also post this on “my hardware” in this forum at a later date, if there’s a place for doing that.

Motherboard: Asus A7V266-E
Operating system: Microsoft Windows XP Professional, Service pack 2.
CPU: AMD 1800
Memory: 1GB (2x512mb [generic])
Video: ATI Radeon 8500 series
Soundcard: Creative Audigy 2 ZS

Any other important stuff you feel is missing, please “speak now, or forever hold your peace!”

MF

 

What are your HDs?

If you are interested in testing out a longshot:
Some of the Via chipset (the V in A7V) boards around your vintage had problems transferring large files, IIRC. I seem to remember that from reading the Asus forums back then when I bought my A7A266 which I am using right now.

Run a checksum calculator like the free XCSC or AccuHash on a large archive file and record the results. You only need one checksum recorded, MD5 is a good one and you don't really need to write down all the characters, the last 6 or so is plenty.
Transfer the file to your slave HD and re-run the checksum calc; it must agree. If so, transfer it back to another folder on the original HD and recheck it.
Try with various large files several times since your problem isn't a hard one.

It is a longshot but not too hard to checkout.

 

Thanks for that seek; my Hdd's are:..

Primary: Western digital WD2500JB-00REA0 (one with system)
Secondary: Maxtor 6L250R0 (one creating image to)

Also: Seagate Barracuda - Model ST380021A (used for latest system restoration, originally in caddy as described earlier.)

When I get time, I'll try running a checksum calculator. I’ll keep you posted with the result.

MF

 

Hello matfry,

A few threads refering to ASUS motherboards that might be of interest to you:

https://www.wilderssecurity.com/showthread.php?t=90711
https://www.wilderssecurity.com/showthread.php?p=523028
https://www.wilderssecurity.com/showthread.php?t=71752

Regards

Menorcaman

 

Thanks for your input Menorcaman.

I notice that the motherboards concerned have other chipsets to mine. My motherboard has the VIA chipset.

MF

 

Many of you are chasing ghosts if you are wasting time running Memtest86, looking at chipset and motherboard problems, etc. As I posted in my original posts in this thread, I have verified this problem on a completely diverse set of motherboards and chipsets running on computers that were built from 2003 through 2007. The one common denominator is that they are connected externally, mostly via USB 2.0, but also via 1394. Just to reiterate for those that haven't had the time to read all the earlier posts, backups made to internal drives work, but when those TIB files are copied to the external drive, and that Windows copy operation verified using various compare tools, True Image will then report that the exact copy on the external drive is corrupt, whereas the exact same, bit-for-bit copy, which was verified using the tools recommended in this thread (and by Windows itself, which is quite capable of copying files -- it would be worthless if it couldn't do this) is deemed "corrupt" by True Image.

So while one or two people may, perhaps, have actually had a problem with memory chips, I can say with fairly high certainty that the problem is with True Image, and is specifically with how True Image reads files. Why it occurs predominantly (and perhaps exclusively) with external drives, I cannot say. However, it you want to get a reliable backup and don't want to jump through the extraordinary hoops of finding a blank drive to restore to (which NONE of us should have to do), then I think you can create the backup to an internal drive, and then copy that to an external drive, and I'm 99% certain that you'll have a useful backup for when the time comes to do a restore.

Someday, perhaps, Acronis will fix this, but given the problems it faces in fixing TI 11, I am not expecting this to happen soon.

 

Give these guys a break - they've just released the super-slick TI 11 with fab utilites like Drive Cleanser and File Shredder. You guys are just being picky, whinging about trivialities like corrupt files!

Lon

 

I think I’ve done enough work now to prove my file system is 100% reliable for read/writing. I completely agree with johnmeyer that the operating system should take care of things anyway. A lot of time in data transfer - possibly most - is spent checking the data has been reliably copied. There are special hard disks you can buy for things like PVR recorders, which don’t have any data checking components built into the hardware, as these thing sometimes have to work at blazingly fast speeds (especially if you’re recording two channels, while watching one recorded earlier!!) With PVR’s, it doesn’t matter if you drop the occasional frame etc… Sorry I’ve been rambling here (as I’m sometimes prone to doing!!!)

All red herrings aside, I do now believe there is a basic problem with the Acronis software.

One very interesting point. If I make an incremental backup of recent data, the file created is as large, if not larger than the original. The whole notion of incremental/differential backups is that they save time and space.

Another thing I’ve just discovered (last night) is that if I try to verify the original (base) backup file after making an incremental backup, the verification of the original file always fails. If I then delete the incremental backup file, the original file always gets verified okay!!! Maybe I’m being a complete idiot in all this, and missing something pretty basic in the “preferences” options? I don’t think so though.

In all my experimentation, I’ve been using my slave drive for backing up to, as it’s faster than using an external USB drive. My results with the external drive(s) however do tend to mirror the internal drive results.

One thing I do realise is that it’s best to make any backups to a drive/system that’s easily detachable from the main computer unit. The safest backups of all are normally done to off-site locations, but I don’t have the necessary funds to do this!

I think you must be saying this with “tongue in cheek”, because the very essence of backing up is that you can produce a file that the backup system can verify, and in an emergency, restore.

Also, don’t forget that this is essentially an image creation program as its name suggests. Any “bells and whistles” the programmers add are merely incidental. Possibly good selling points, but let’s get this long-running problem back in focus!

MF