ShadowProtect Desktop 3 is out...

I verified a few days ago that each time I boot from the Recovery CD into the recommended (Vista PE) environment, my PC's clock advances by one hour. It doesn't happen when I boot into the legacy environment. It's not a big deal but one I hope MS addresses soon...

 

For ShadowProtect Desktop/Sever 3.0, HIR functionality will work for any image of an OS volume on which any flavor (trial/full/etc) of ShadowProtect was installed, regardless whether that image was taken from within live windows or from the recovery environment, including any existing older images taken with previous versions of ShadowProtect.

 

This morning I successfully moved an XP partition from a quad-boot DOS6/98SE/XPSP2/XPSP2 hard disk to a different single-boot physical disk that was initially all unallocated space using SPDT3. The source XP was in the third physical position on the source disk. The destination position was the first physical position on the destination disk. I am very pleased (and somewhat surprised) with the result.

My first attempt failed because I thought XP recorded partition location information in the XP registry and moving to a different physical location would require that information to be changed in order for a successful boot. I therefore also used SavePart, a DOS program to patch the restored XP registry with the updated partition location information - or so I thought. It seemed to make some partition boundary adjustments but the resulting partition wouldn't boot, hanging just before the login screen.

Sooo, I again restored the source partition using only SPDT3. To my surprise, it booted right up after restore and everything seems perfect. The boot.ini file was automatically patched by SPDT3 and although it works as is as the default, it also refers to a second, selectable XP boot partition which does not exist on the single-boot destination disk. That is easily corrected with a manual edit. I did not restore the source mbr or the source first head since those would not be appropriate in going from an 11 partition disk with a third party boot manager to a 2 partition disk with no boot manager. The source and destination disk were the same total size (80gb).

Apparently SPDT3 automatically revises any partition location information in the XP registry during a restore to a different partition location. Either that or the information I had about such things in the registry is not valid. Can anyone confirm this action by SPDT3?

I am liking SPDT3 very much. I am eager to try the new HIR function but alas, I have no new hardware to try it out on at the present time.

Note that my system clock also gets sets ahead one hour but ONLY when using the "recommened" boot environment. There is no clock setting change when using the "legacy" boot environment. I don't need the "recommended" environment at this time so a prompt fix is not an important issue for me.

Rod

 

If you never plan on using ShadowProtect for incremental imaging then it does no harm to delete the .idx files in the root directories of your volumes. If you do start using incremental imaging then leave those .idx files alone. Frankly there's never much harm in deleting those .idx files, but they'll just come back. They're written at shutdown time, so each time you shut down they'll reappear on your next boot.

 

I've just installed SPD 3.0 on my machine and have tried to create a Conitnuous Incremental schedule every 15 minutes, but when I set it running SPD "hangs" (ie. the window becomes unresponsive and nothing happens). Eventually, I terminated ShadowProtectSvc.exe but when I started it again it just said "Connecting" for a while, before the GUI became responsive again. Once in, the status of the job was "failed" and the log was as follows (machine info and Volume info removed in case of privacy issues):

Code:

29-Aug-2007 21:17:27	service	100	service (build 40) started job by incremental trigger
29-Aug-2007 21:17:27	service	300	trial period has 23 days left
29-Aug-2007 21:17:27	service	104	machine information: <removed> DEFAULT
29-Aug-2007 21:17:27	service	504	Volume {<removed>} was not found
29-Aug-2007 21:17:27	service	100	no need to get snapshot
29-Aug-2007 21:17:27	service	101	Cannot execute job (Unspecified error 0x80004005(2147500037))

Also, the Continuous Incrementals options has me a bit confused - What is the difference between the top "VSS Incremental Backups" and the bottom "Additional Incremental Backups" ?

Can someone explain how to set it up so I can keep on having continuous backups every 15 minutes, every day of the week, every day of the year ?


PS. The on-line help could do with a few tutorials using backup scenarios.

 

BTW, does verify in 3.0 do a byte-by-byte comparison, or does it just do a simple CRC as in 2.5 ?

 

After readin the help I came across the bit about configuring the imagemanager. However, there is no shortcut for it on the Start Menu so I cannot et it up. Is this only for registered users ?

 

Wow, that's crazy stuff. Volume {<removed>}? machine information: <removed>? I've never seen anything like that. I suspect there's some amazing software on your machine, and it's not happy that you installed ShadowProtect. Hmm. It's also very strange that it apparently (according to your description) took a long time to connect the GUI to the service's DCOM interface. Normally, a local connect to the service is pretty much instantaneous. I'm not sure what to make of this. Do you have some kind of third party firewall installed that caused this? Or did an "Unblock port" dialog pop up an didn't respond to it for a while?

On to ImageManager question - the installer for ImageManager is available on the recovery environment CD. You have to request a full eval to get access to the ISO for this CD if you want to try this out. I'll see about have it posted so you can just download the sucker. Um, also, relating to the VSS vs non-VSS options within Continuous Incremental jobs - the thing is that when you take a VSS incremental backup, it's always at least a few hundred KB in size, whereas if you're not so concerned about VSS we give you the option (you can turn this on or off) to have many of your intra-daily incrementals taken without VSS, which results in smaller incremental image sizes as well as faster imaging (as there's no VSS quiescence going on at the time they're taken). Such non-VSS incrementals still capture the filesystem in a nice flushed state.

The verify uses CRC32 hashing to test every byte in your image file to ensure that nothing has changed. Additionally, if you have used any compression (most users do as compression saves both time and space) and/or encryption, then the decompression and decryption routines will detect any corruption as the entire image file data content is tested. So there are up to three tests that are performed on the data during a verify.

 

I edited the log and put the text "<removed>", as like many on this forum I am paranoid about my privacy/security

I do have Look 'n' Stop as my firewall, and have tried running SPD with LnS disabled, but it didn't work. No LnS alerts were displayed either.

However, just after I replied to Pete's post I rebooted my machine and tried to do a Now->Full backup which failed as the original Continuous backup was already in progress. Now, I haven't got a clue what made it suddenly work that time, but subsequent attempts to run a backup have failed. I checked the log for the time is succeeded, and it kept trying different methods for backing up. It also mentioned the VSS was in a bad state. I will post a full log of the time it was successful once I have gone through it removing Volume and machine information

I do have quite a few Services disabled or set to Manual (to minimize the number of unnecessary processes running on my machine), so can I ask what services SPD relies on and do they have to be set to Automatic ?

I am also running Kaspersky Anti-Virus 7.0.0.125, but I believe Pete is also running that AV, which tends to rule that out as the culprit.

I have a backup from another imaging app (Image For Windows), so I will try reverting to that point in time before installing SPD again to test it, in case something else has got on my system to cause problems.

It would be great if I could try ImageManager. I have already applied for a full eval before version 3 was released, but specifically mentioned I was only interested in version 3. Mike Kunz contacted me to ask how I would be using the software, but I have yet to receive the full eval (maybe he didn't like my reply ).

CRC32 hashing can result in collisions every now and then, which means that an image which passes verification could in fact be corrupted. I understand most people use compression and/or encryption which would help to show up any corruption, but the verification algorithm used is flawed. I would like to see byte-by-byte verification included in a future version.

 

Yeah, if you disable ShadowProtect and VSS services (ShadowProtectSvc.exe - which you should see twice in your process list, vsnapvss.exe, vssvc.exe, etc) then you'll have all kinds of problems and you should expect failures.

 

@Defenestration

Yup, a larger hash would be nice (perhaps we'll add one in the future), but the only hash that avoids collisions altogether is a hash that's as large as the data that's being hashed.... which isn't very useful.

Here's a little discussion on this topic.

In practice, corruption is immediately detected by the decompressor first. If corruption somehow managed to corrupt the data while maintaining the integrity of the compressed data stream (extremely unlikely) then CRC32 will catch it. The combination of these two tests is significantly less likely to miss corruption due to a crc32 collision.

If you're really worried about this, please, by all means, use a different product. Or just flip on compression and encryption and you'll have triple verification.

From what little you posted of your log, I can't really tell much about the error. It does look like something is preventing the backup from occurring but I have no idea what that may be. It's my opinion that this is very likely caused by some other software and a poor interaction between it and ShadowProtect.

You must enable all of the "StorageCraft *" and "ShadowProtect *" services (should be Automatic start - start them after setting to Automatic), as well as the "MS Software Shadow Copy Provider" and "Volume Shadow Copy" services (these two should be Manual start).

 

Yeah, regarding CRC32 collisions, you really have to consider the use case.

For a collision to occur, the following events have to take place:

Actual corruption of the image file has to occur. This event in and of itself is actually fairly rare, especially with journaled file systems which, for the most part, relegate the causes of corruption to hardware. If it was common, none of us would be using hard disks. The odds that corruption will occur depend on the quality and age of the hard disk, the size of the image file (the percentage of the disk that's consumed by image data - the larger the file the more likely that it can become corrupt), environment, and usage (a disk that's being pounded 24x7 is more likely to experience failures than one that's sitting on a shelf). Let's be pessimistic and say that the odds that an image file will become corrupt as it sits on your hard disk are 1 in 100.

The hash of the region of image data that was corrupted would have to generate the same value as the it did before the data was corrupted. Odds of this are roughly 1 in 4,294,967,295 (I say roughly because this assumes CRC32 generates uniformly distributed hashes).

If you don't encrypt or compress the image (which results in extra degrees of verification), then the odds that a hash collision will actually be a problem for you are roughly 1 in 400 billion (4294967295 x 100). In other words, you should start worrying about this around the time that you've actually created 400 billion image files.

However, let's not forget use case, because after all, most users DO compress their image files because it turns out that compressed images are generated FASTER than uncompressed images, and they take up less disk space. In other words, there are few reasons not to compress. And it's the default option in ShadowProtect. Now before CRC32 is calculated, the decryptor (yet another layer of verify) and decompressor consume the data stream, and immediately detect corruption. The odds that the data stream can be corrupted in such a way that the corruption gets past the decryptor and decompressor are pretty remote. I haven't done the math to figure out the exact probability, but it's a gross underestimate to say that it'd be a 1 in 100 chance that such corruption would go unnoticed. Same for the decryptor. So with this incredibly low-ball/pessimistic probility, you're still talking about a 1 in 40 trillion chance of a collision actually being a problem for you, or 1 in 4 quadrillion chance if you're using both compression and encryption. And again, that's a low ball estimate. I'm pretty confident that if I worked the math on the probability of the decompressor and decrypor missing corruption that it'd be a much longer odd.

So, if you're concerned, just flip on compression and encryption, and don't sweat it until you've generated 4 quadrillion image files. Hmm... y'know, I think it might (currently) be impossible to even have 4 quadrillion decent-sized image files - there's not sufficient storage space on this planet for that much data.

But, all that being said, I think it might not be a bad idea to add a SHA-1 or SHA-256 hash, and maybe let the user pick the hash level.

 

Pete - May I ask what AV you use now ?

grnxnm - I never disabled any of the services after installing SPD, but I had previously disabled the MS Software Shadow Copy Provider. However, even after enabling this service I still can't get it to work properly. I think I'm going to go ahead an re-image with a known good image and start testing SPD again from there.

It might be worth including a check in SP to make sure all the required services are enabled (and have correct startup mode).

Re. the hashing algorithm - I'm not saying introduce SHA-1 or SHA-256 (although giving the user the option would be nice), or to get rid of CRC-32, but rather to have an additional byte-by-byte comparison after the image is created (by de-compressing the image stream) to compare the original source with the backed up data. This would guarantee that you have an exact image of the source at the time the image was taken.

Obviously, this byte-by-byte comparison cannot be performed later on as the original source data will have most likely changed, and this is where the CRC-32 checksum comes into play, to verify the image when the original source is no longer present. It would just be an extra layer of security to guarantee the image created is an EXACT match of the snapshot in time.

 

Doing a basic check that the required services are available would be an easy thing to implement and would result in more robust software, with a nice informative error message being displayed instead of it just hanging. While it may be an oversight, and I know no software is perfect, but it's small details like this that make me lose a bit of faith in a product. If they haven't done simple checks like this, what else aren't they checking/doing to make there software more robust.

 

Windows usually requires certain services to run normally. Beyond that, some are complete unnecessary. So, I'm not saying test all services are running, but ones specific to an app and which cause that app to hang if they are not running should be tested IMO, to make the design more robust.

grnxnm stated himself that aside from SP's own services, both "MS Software Shadow Copy Provider" and "Volume Shadow Copy" services should be set to Manual start. So, if this is known to be the case then checks should be made to ensure that these two services are set to Manual start at the very least, and preferably also checked to see they are running (if needed for the particular operation).

 

I'm waiting for the SP ver.3 ISO to be released. Hopefully today.

Question is: How long "should" a restore take of a 35GB System Drive / Image?

System specs in sig.

...screamer

 

I dont know how long it "should" take but I guess it has to do with how fast hard drives and processor one has.
34GB takes 18 minutes for me to image with standard compression for me (when I do a backup in windows Vista and XP)