Introducing AX64 Time Machine - hybrid imaging/snapshot software

There is a subforum for Instant Rescue on this Wilders board - it is not as active as this thread but this is just because it's a mature product that works.

 

Totally agree with this.

 

You are aware that you are talking about version 1. Version 2, when they get the bugs worked out (as froggie says just above "it's a BETA!") will address these issues. Once v2 is officially released the anti-malware issue will disappear, and this issue is the main culprit causing failed restores.

And no imaging or snapshot program works well with defragging apps since the post defrag image or snapshot will be rather larger than it would otherwise be. Apart from this fact AX64 works just fine with defraggers, as long as you are willing to tolerate the larger snapshots.

Now I recently had an issue with cold restore but the experience has in no way turned me off of AX64. In fact I was very grateful today for the hourly snapshot feature. I had been working on a CAD file and had saved it. Then decided to modify the model to see what it would weigh in a few different configurations (I am a jeweller so the finished weight is critical). I tweaked the model a few times, checked the weight projected, made my notes and then intended to close the file without saving it but like a jerk I hit OK when the save window came up. No problem, I went to AX64 browser and opened the last snap, copied the file and pasted it into Win Exp and all was fine (I personally would prefer more frequent snaps along with more frequent merges, say 1 snap every 15 min for the last hour, then 1 snap per hour for 24 hours etc. This would suit my needs better than 1 per hour but,,,,,). Had I been using pretty much any other imaging program the small mistake of clicking OK instead of no would have meant a lot of time repairing my base model. With AX64 it was a few min and all was well again.

 

I don't think cold restore has been show to not work in V1, yes, in V2 its a problem but of course V2 is a beta so dumping on AX64 or TM because of this is inappropriate. Yes, some say the problem extends to V1 (Peter) but others (Froggie) disagree. I have flip flopped on this but after considering what happened in regards to my recent failed cold restore I cannot blame AX64s cold restore failure on AX64s cold restore function. If my tracking file became damaged during an attempted hot restore (and I am convinced that it did) I should not expect it (AX64) to work during a subsequent cold restore. I am convinced that had I gone right to a cold restore ESET (of course) would have been shut down (it was running a weekly system scan at the time I initiated the hot restore) and the restore would have occurred correctly. These kinds of problems are only a factor with hot restores. They should all be resolved with the new warm restore. When testing a new beta release will begin is anyones guess, hopefully it will be sooner rather than later.

 

I disagree. The tracking file is only necessary for hot (or warm) restores. A cold restore does not depend on the tracking file at all, it works just like with any other traditional image based backup software. This is the reason why I am so disturbed about your cold restore failures.

BTW did you read my suggestion to mount one of the snaps which failed to restore and perform a "chkdsk /f" on it? This could reveal if the snapshot itself was already corrupted.


Cheers
manolito

 

When you say mount are you suggesting mounting the snap as a virtual drive? If this is indeed what you suggested then no, I did not do that but I ran chdsk on a restored to snap and sat back an watched as ckdsk went to town on my drive. I fully expected to have the boot fail after the chdsk did its thing it was so active. I think this was my 3rd attempt to get the system back and my 2nd cold restore.

The 2nd to last attempt using AX64 triggered an automatic chdsk (I must have attempted at least a half dozen times in all). It found less issues than the 1st ckdsk but did indeed find issues.

Do you have any suggestion as to how snapshots on an external drive could become corrupted due to a C drive virus scan. I cannot fathom how this can happen. Its clear to me that the scan caused something to go amiss with AX64 and the only thing I can think of that would possibly be affected would be the tracking file, I cannot for the life of me figure out what else it could have been.

Are you sure the tracking file is not used for cold restores. I thought this is where AX64 kept track of all changes to the drive between snaps. If I was rolling back to snap X (or rolling forward to it) would AX64 not need to know what the tracking file has in it in order to limit the process to the appropriate state?

 

Yes, mounting the snap as a virtual drive and then perform a "chkdsk /f" on it is the equivalent of an image verification which other backup software allows. This way you can find out if the snapshot was already corrupted.

And yes, I am absolutely sure that the tracking file is not used for a cold restore. Ask Froggie...


Cheers
manolito

 

I will run the test you suggested and report back.

 

Still on the latest V1 Beta here, I somehow trashed the Windows installation making it unbootable so I tried to do a cold restore from a USB thumbdrive and it got stuck at 44.9% and stayed there, I did a hard shutdown and booted into the recovery media again and instead tried to restore the baseline image and it worked fine and then from there I did a hot restore to the latest snapshot and it worked fine too.. Wondering why the cold restore just out right stopped at 44.9%?

 

Ran the test as you suggested and the chdsk did find errors. I am attaching a screen shot and as you can see we were not very far into the process when problems were discovered. The snapshot I mounted was the one that had the large number of chdsk errors when I manually ran ckdsk on my own. I remember there were a great many orphan files as well as other assorted things that ckdsk had to deal with.

So what does this tell us besides the fact that the snap was corrupt?

 

This is getting interesting...

As I see it there are only two possible explanations:
Either your HDD file system was already corrupted at the time you made the snapshot, or (much worse) AX64 created a corrupt snapshot from a perfect partition.

Some other image backuppers have an option to check the HDD integrity before making a backup, but AX64 does not. So the user has the responsibility to make sure his file system is error free before making backups.

The other possibility that Ax64 might create corrupt snapshots from a healthy partition is too scary to speculate upon...


Cheers
manolito

 

Sorry, I'm no expert but the thought just crossed my mind.
I remember that defrags often used to lock up when files were in use or locked by another program with higher or similar status. Could these lock ups and unfinished writes be anything to do with that sort of scenario?
Sometimes a system monitoring software that requires high level access is a culprit or anti virus software that is active or something that is locking down a file, some kernel level software holding ownership and not letting go as the restore is being read and written leading to an impasse. ?

 

This is absolutely TRUE (although not quite enough for Pete and his requirements) for AXTM v1 and absolutely NOT TRUE for TM v2 BETA. The COLD RESTORE in v2 is actually using its WARM restore algorithm during the BOOT of the COLD restore media... why, I have no idea. I don't know the criteria used by that restore method to determine what to put back, but it's not everything by DEFAULT. If it's trying to do a WARM restore instead (which I believe it is), it must be using the TRACKING mechanism (once it's determined that it thinks it's OK) and/or the on-disk file structure (after making the same determination) to do the restore. If that's the case, as Pete says, it's NOT a COLD restore... which I support.

I have seen v2 do what appears to be a FULL restore operation (mainly due to time and data moved) but I can't be sure that's what's happening based on the observations and information above.

Absolutely agreed. I couldn't tell from BG's failure description whether he was using v1 or v2 when his "COLD" restore failed but I've been beating the hell out of weird restore combinations to try and create this type of failure... so far, no go. Based on BG's failed ChkDsk on his mounted snap, it appears that the COLD restore failure was most likely caused by a bad snap chain rather than the COLD restore operation failure. It's important to try and figure out how THAT actually happened..

Whenever I question a v2 COLD restore, I just do a v1 COLD restore of the same snap chain instead... all does as expected.

 

SDmod... it is believed that this type of interaction is indeed at the root of the HOT retore failures. AX64 tried to elevate its processing priority, during HOT restores, to the highest level possible (probably what MicroSloth calls "Real Time") in order to keep the general system at bey during the restore process. They discovered that some system functions, possibly called by some of those special apps you mention, are not being kept at bey like they thought. I don't think they clearly understand the problem they ran into except that there is a problem and it's probably insurmountable... that's why they headed to what they're calling a WARM RESTORE. Clearly a WARM restore done from outside of the uncontrolled LIVE Windows environment and under a controlled external Windows process should have a much greater chance of success... assuming that all their assumptions about changing file structures and tracking environments have been correctly made about a LIVE Windows system. They've already discovered one major area (the Windows NATIVE API at pre-BOOT time) that is giving them some issues as far as the Perfect Disk application is concerned... who knows, there may be more.

Anyone working at this level within the Windows system is clearly in the Dark Forest as far as unknown creatures are concerned

 

Hi Froggie

Barry was using V 1.307 when his problem with restoring occurred. At the time, I had thought something must have become corrupted especially when he stated later ESET was doing a scan during one of his HOT restore attempts, and to me that was probably not a good thing. Also, I do think his snaps were corrupted, the cause of it is not clear, but I think it possible he had a problem on the disk maybe he was not aware of. The COLD restore failure is the last indicator of a corrupt backup, but how they got corrupted is unclear. When I suggested the easiest thing to try was to restore his Drive Cloner image to see if that restored properly, which it did, proved to me at least that his backup chain from AX64 was corrupt and his chkdsk test results on the mounted backup per manolito instructions would tell me his file system was already corrupt at the time of the snap, at least it would appear so.

I agree with you for sure that it was a bad chain, but the question to me is still HOW did they become corrupt enough to have a COLD restore fail. Do you think it possible that his tracking file became corrupt at some point, and IF so, would not AX64 had decided to create a new baseline?...or would it just go through the steps of creating a snapshot, but take the same time as a full image?
These issues to me are head scratchers, but since his DC image restored fine, definitely points to AX64 corruption but not clear as to how that happened.

Thanks for all your help! When I have done COLD restores with AX64 v1, they have always been flawless, when these issues occur I do wonder if there is a form of conflict that simply has not been exposed at this time. I still think security software is a problem especially for HOT restores, and sometimes it is near impossible to truly "disable" some security software due to how low level it operates and his comment regarding ESET leads me to think perhaps ESET potentially could have caused an issue with his backup chain? Just thinking out loud.

Have a good day TRF!

Jim

 

You folks may remember that I had an issue with AX64 not staying open when I tried to open the browser some time ago (a week or 2). It would flash open for a second and then shut down. I was told how to create exceptions for AX64 as it appeared that ESET had an update that affected AX64s ability to function. In hindsight I probably should of started a new chain at the time but I did not. It seems possible to me that the chain became corrupted at that time. I am going to check back through my posts to determine as closely as I can when this took place and then run a ckdsk on a snap from before this time. If its clean then we will have our culprit.

And Jim is correct Froggie I am using V 1.3.0.7

 

Morning Jim!

Without knowing AXTM's algorithm for determining either TRACKING INFO corruption or whether the volume has been messed with outside of tracking, I'm not sure how it would make a determination to either rescan from scratch and produce an incremental, or question the whole BASELINE and produce anew. Normally in imaging mode if it detects a possible unTRacked change to the file system it just does a rescan and produces an updated tracking file then compares that to the one most recently saved and produces the resultant incremental. This should be more than adequate as it remakes the tracking file from scratch (just like it does when it produces a new baseline)... BUT it has to trust the last snapshot to be able to do this adequately.

The problem is that it doesn't take a "corrupt" tracking file to do this... all it takes is an undetected change to the file system to start down this very questionable road, and the farther down you get, the worse it can get. Basically this type of change doesn't "corrupt" the tracking info, it invalidates it at that point in time. The only one of these that we know about, currently, is the Perfect Disk low level disk input/output operations done during its BOOT time defrag... for all we know, there may be more, and some may be actually running under a LIVE Windows system. Only someone who knows the real available structure of LIVE Windows (Russinovich?) can really answer this question.

My point above exactly...

Jim, don't ever stop "thinking out loud," it's that "out loud" thing that starts others doing the same.

 

Barry, that's an EXCELLENT idea!! Good luck in finding the time period when that change occurred!

 

LOL! Froggie, you always have an excellent way of stating things, but this one take the cake!

Acadia

 

Ran the tests as suggested and there is no correlation with the corrupt snaps and the mentioned ESET not opening problem. I first mentioned the issue on May 16 and I believe it was resolved on the 17th, at the latest 18th. I ran ckdsk on snaps (mounted) for May 11, may 16, may 20, and may 25. All were fine. The problem snaps occurred around the 29th or 30th.

 

Dang it you guys! My head hurts AGAIN......

 

Really no reason for that, I suspect I made it more complicated sounding than it really is. Its like this,,,,,,I have some corrupt snapshots. This was suggested by the fact that after a restore ckdsk was run and it found issues. The snap(s) in question were created May 29/30. A few weeks earlier I had had an issue where every time I tried to open the AX64 browser it would close AX64. One of the forum members (sorry I do not recall who) had the same problem and it turned out to be an ESET update. He provided the fix and all seemed to be well. I thought that perhaps the ESET update and or fix had possibly compromised the snapshot chain causing the restores to the later snaps to fail so I ran tests on the snaps surrounding the ESET issues dates. They proved to not be corrupt so its clear that the source of the corruption of the snaps from the 29th & 30th was not the ESET issue, so there is still no idea as to why my snaps were corrupted,,,,,which is a bummer.

I suspect what this means is that we will never know what caused the corruption unless someone has an idea how I may further test in order to determine this. Some fear that it was AX64 itself but other apps have had corrupt images so I don't see why AX64 would be expected to be immune from them. Still, its a bit disturbing to say the least.

 

"Some fear that it was AX64 itself but other apps have had corrupt images so I don't see why AX64 would be expected to be immune from them. Still, its a bit disturbing to say the least."

One of the reason I was hoping that the new version of TM would have a verify option built in, at least that would tell you if your image was ok at that moment in time.

 

Some thoughts on image verification...

Basically two kinds of "formal" verification have appeared on the scene... one which does a CheckSum of the image data itself and the other which tries to compare, byte for byte, the actual image data which has been taken. The first is much faster but suffers from the fact that if the wrong data has been imaged to begin with, it will checksum beautifully but won't be found to be bad until it's restored and the system attempts to use it. This is badly structured data rather than data that has changed along the way and is very tough to verify early in the process.

The second method suffers from the same "structure" issue but takes a long time to verify that what it thinks should be imaged is what was actually imaged... the result will be the same, the wrong data was taken but it checks perfectly against the list of wrong data that should have be taken.

Both of the above methods can determine if the image data itself has been corrupted from the time it was taken and the time it was checked. This covers failures of storage devices, possibly intermittent RAM failures, hardware data path failures, etc., but neither will cover "structure" errors which I believe is what happened to BG's snap chain.

If the method used to determine incremental/differential changes to be taken has been compromised (for whatever reason), then the chain or baseline/differential backup set will be compromised. In the case of AX64, all it takes is a tracking file hiccup, or in the case of Perfect Disk, an undetected system modification to wreck the chain from that point on. In the case of Macrium, all you need is an error in your file structure at the time of imaging and its backups become toast... this will be the same with AX64. From what I can tell, Macrium actually compares the in place structure (how much of it, I don't know) to the last backed up in place file structure to determine its next data move. There may be file structure errors at this time but you may never know it 'til you try and use a portion of your system that's housed in the file structure error... then BOOM.

All the methods seem to have some flaws in what they can catch and what they can't, but the one that seems to have the greatest capability to catch most of these problems is an actual CheckDsk of the taken image. Sure, it takes time to do but it'll catch file structure anomalies usually without problem. Most hardware data path errors that accompany images will also cause possible file structure problems which should be caught by ChkDsk. The only thing these checks will miss is real changed non-file structure data (file content rather than file structure). Sure, hardware data errors just may not affect file structure part of the image, and in that case, ChkDsk will not find the error... but CheckSum or full data compare features should.

I don't know the answer... but from previous experience I find a reasonable CheckSum feature good enough for almost all data type errors (it's pretty hard to get random data failures to beat a long enough CheckSum), and if I really care about the integrity of my image chain at some point in time, I mount it and run the system's ChkDsk feature on the mounted chain just to be sure...

...and then I play Pete's video for that final "feel good" of the day (and NO, it's not porn... it's a FROG video)