Oh Gods of Truecrypt, please hear my prayer! (technical knowledge required)

Dear All,

I had the misfortune over summer of having some nefarious character(s) break into both my house and my office (they found the office keys at home) and steal everything of value that wasn't bolted down.

Fortunately I had hidden (quite well) three external drives used to back up my entire life, and I have spent a large block of time over the last month restoring my life to normal, but here is the problem.

All the drives are encrypted (truecrypt) but unfortunately one of the drives will not mount in TC due to a large number of bad sectors. (Yes, I have tried it on different machines, including linux based TC with option not to mount file system)

I have been very careful with the drive and read both this forum, the truecrypt forums, and various data recovery forums extensively, but I would appreciate some (any) advice how best to approach the recovery, as the data is very important to me (as it is with many people).

Dantz, I have a licensed copy of Winhex and followed your instructions in the following post:

https://www.wilderssecurity.com/showthread.php?t=336671

It worked perfectly, and I was able to mount the test file and see various file headers meaning that decryption was ok. I also have a backup copy of the volume header on one of the other working disks.

My dilemma is where to proceed from here, which is partly data recovery, and partly understanding about truecrypt. So first the situation:

The disk in question is a 2TB ext USB western digital drive with a single RAW partition (starting from decimal offset 1048576 [sector2048 of 3907024896]), the disk also shows TC like data on the final sectors (up to 3907024895 - assuming first sector is 0) meaning (I think) that partition continues until the end of the disk (not sure) [NB: there is no hidden TC container/partition]. The disk is currently showing in Win7 disk manager as healthy. (Gsmart is reporting reading faults due to CRC check failure with certain sectors)

See screenshot attachement 165k below

I narrowed down the failing sectors to a specific area on the disk by trial and error (reducing by halves) using WinHex in read only mode. Please note that WinHex also hangs when trying to read the faulty sectors (there may be some issues with caching of adjacent sectors, but I am not sure with WinHex).

They are Sectors/LBAs 1641532328 to 1641677529

I take this to mean that 144,212(inclusive) sectors are lost or about 74MB (@512k blocks)of the 2TB(approx), meaning that if karma is on my side it may be possible to recover the rest. (Just for the record I know the drive is pretty much full so it does contain a $**tload of data that is important to me.)

I assume that truecrypt is hanging when I try to mount the drive because some important part of the NTFS file system (default allocation size - 512) such as MFT$ must be contained within those addresses, however I am not sure if there is any other operation(s) prior to accessing the NTFS related files that truecrypt undertakes

I note that if the drive is physically removed by unplugging the USB, that truecrypt resumes responding and says that the drive was dismounted, so I assume at least some part of the mounting process is successful (and that the lack of response is more related to the drive than to truecrypt-same with WinHex), even though it does not show up as being mounted in the TC main window.

I am unsure as to which of the following scenario's will give me the best option of mounting/decrypting and recovering the data and this is where I would appreciate some feedback.

Options:

Create an image or a sector by sector copy or take chunks of the file (maybe all - but very large @2TB) and create a virtual 'test file' using WinHex for decrypting at a later stage(as per earlier referenced post).

I am unsure if it would be easier to explore recovery options by creating an image (of either the disk or the partition) using Clonezilla '-rescue' option or perhaps ddrescue (any thoughts appreciated) and mounting it, or if creating a direct clone of the disk (using similar/same tools) and attempting to mount the partition is better.

If I create an image, will truecrypt (attempting to mount from a mounted image) find that things are not in the right place because most imaging software ignores blank/bad sectors and may change the physical/logical locations?

I am not sure how truecrypt would deal with such an issue or even how the cloning/imaging software would deal with it for that matter. I make the assumption (perhaps wrongly) that TC volume headers/setup information contains information that relates specifically the location/offset of various critical items on the disk (even if these are subject to drive based [read firmware] virtual re-allocations - I assume that these should be ok) as it obviously does for the volume headers and backups (correct me if I am wrong).

If I create a sector by sector clone and even if the tools (same as above) write zero's or something else ('BADSECTOR') to the logical addresses (in the clone/image) of the bad sectors, will I not possibly/probably face the same problem with hanging because TC can't find the right NTFS files, OR will it likely mount and state the file system is corrupted giving me the opportunity to perform file recovery, or perhaps even the chance to permanently decrypt the disk for future recovery attempts?

My next TC related question is do I attempt (assuming i could achieve one of the above) is to permanently decrypt the partition after cloning/imaging and then attempt file recovery (if so what is the best way) or do I stick with the mounting of image/partition, and then try and recover from there? Any advice here will be greatly appreciated.

If any of these options was to work I would simply recover the files that were possible and copy them off to another disk.

Perhaps it is not critical to the task at hand but the disk in question contain a lot of work(most of it replaceable over time), but more importantly, the backups of all the digital footage of my children since they were born (the oldest being 9 the second being 7) and everything else has been lost since computers and copies have been lost from both sites due to these a$$hole thieves, so if you can help I will be forever in your debt.

Some further notes:

I am unable to run standard Clonezilla ISO directly as it seems to have compatibility issues with my new hardware, but I have been able to run the version included with the Parted Magic ISO. Also the disk I want to use to create the backup image/clone is a GPT based 3TB seagate (I cannot seem to get it to operate in legacy mode with the current BIOS - Gigabyte Z77X-UD3H - American Megatrends-F18[2012/10/24]) if anyone has any thoughts on this I would be happy to hear them.

I have tried a similarly sized 2TB seagate but it failed to clone due size issues. I also have two identical Western Digital external drives (haven't tried yet unitl I can afford a few more drives to backup them again), and if I cant find a way (to get the image/clone to the 3TB drive) I may be able to backup one of the 2TB WD's to the 3TB and restore any image/clone to the one of the 2TB drives. Of course all this takes time given the volume of data and the delays due to bad sectors, so again any thoughts would be appreciated.

Any further questions, please ask. Any errors, I'm sorry, please point them out, took me a long time to get to this...

Thanks again in advance.

In return for everyone's help I will assist in creating a definitive and well documented guide of how to recover wholly encrypted non-system TC disks/partitions when they are affected by bad sectors for future use by the community.

If you got to the end of this and read it all, then whomever you are thank you for taking the time to read about my issues, I know it is almost an essay, so five stars. May good karma revisit you ten times over when it is needed.

Cheers,

Mark

P.S. Given the complications, I understand if it takes a while for people to think about this and respond, double the above karma if you manage to get round to it

P.P.S. If you have read the post here (and the many others in the forum) then the above few paragraphs will make even more sense if you are careful - think before you act

 

Your timing is unfortunate, as I am about to leave for two weeks and probably won't be able to be participate in this thread until I return. But here are my comments to date:

I was unaware that Clonezilla had a -rescue switch. I generally recommend using ddrescue for this type of situation (and I suggest you create the logfile). And don't worry, imaging software doesn't actually "remove" bad sectors that it comes across (unless it is capable of doing so and you ask it to). And since you will be operating in sector-by-sector mode, I don't believe there will even be an option for excluding sectors from the output file. If certain sectors can't be read after several attempts then that portion of the image will merely contain filler, which you can set to whatever you want. The overall size of the drive will be the same. And you're right, if this were not the case then TrueCrypt would have problems decrypting the data.

It's far safer to image the drive and then restore the image onto another drive, but of course this will require two large drives. If you can't swing that then I would suggest using ddrescue to make a clone instead.

TrueCrypt doesn't care what the contents are or whether the file system is intact, it merely mounts the volume and makes it available (via on-the-fly decryption) to Windows or whatever other program will be used to access it. You can mount a partition that contains total garbage and no filesystem whatsoever, and it will mount just as quickly as a formatted partition. Under normal circumstances TrueCrypt will either quickly mount a volume or it will present the "incorrect password" error (if for example the header is corrupted). If TrueCrypt seems to 'hang' while mounting a volume then there is usually a hardware error involved. The fact that WinHex also hangs when trying to read certain sectors confirms that the problem is almost certainly hardware related.

Once you make your image and then restore it to a healthy drive then TrueCrypt should be able to mount the volume normally, even if the file system is bad. If Windows won't allow you to browse the mounted volume due to excessive file system damage then at that point you might have to use various types of data-recovery software to recover what you can, or attempt to repair the file system using tools like TestDisk. It's safer to recover your data first, but if you have an image (not merely a clone) of the drive then you can safely try to repair the file system first, as you can always restore the backup image and try again if you screw things up. The idea, of course, is to perform all of your data recovery and/or file system repairs on the restored image.

I can't help you much here, I'm afraid. However, at this point it seems likely that your failing hardware is creating obstacles for whatever software/hardware combination you attempt to use. I think your best shot is to try using ddrescue, although other software might work. I've heard of other cases where Clonezilla wouldn't run but ddrescue would. But if Clonezilla (with -rescue) seems to run normally and doesn't hang when it gets to the bad portions then it's probably just the hardware compatibility issues that you mentioned above. I leave it to you to sort that part out.

I apologize if the above contains errors. I don't have time to fact-check or do anything else right now, as I have to get going. I'll check back in a couple of weeks to see how you're doing.

 

Thankyou so much Dantz for clarifying the TC issues and your thoughts regarding ddrescue, I will look into it. Enjoy whatever you are up to over the next couple of weeks. I will report back on the progress in due course.

 

Try this: http://www.grc.com/spinrite.htm

 

I second this recommendation by axle00.

 

Thanks guys for your input. I tried ddrescue and clonezilla without success, both reporting recovery transfer rates of between 15-30kB/s, which at my calculation would take somewhere in the vicinity of 2000+ years. Of course the data probably won't be much use to me then, assuming the drive could hang on that long I did have some moderate success tranferring chunks of the data about (100GB) manually with winhex, but wherever it runs into an ECC error presumably due to a bad sector it just hangs. I guess one of the problems with recovering a wholly encrypted truecrypt device is that you don't know where your relevant data is...

I ran up spinrite as suggested, and that too failed halting with an unrecoverable error at 0.03%.

My next step although it is not without its risks, as P-list and various other settings may be different, is to replace the circuit board on the drive. (I puchased three identical drives at the same time) Then to try and recover some more data, and perhaps manually parse it all back into an image file filling the gaps with 0's and trying to recover from there. I will have to back up the other drives first though, which could take some time. Will report back on progress at a later stage.

Cheers,

Mark

 

When you say that SpinRite halted with an unrecoverable error at 0.03% are you sure that it "halted"? A lot of people report that SpinRite will run into a problematic sector and try repeatedly to recover the data and take a long time doing so. SpinRite will even fail to recover some of the data, mark it as unrecoverable and move on. Also, are you sure that you ran SpinRite on the correct level? Just trying to help.

 

I've used HDD Regenerator in the past to recover bad sectors and to identify delays on external HDD. It's a commercial program but worth a look.

 

@dallen

Yes I am sure it halted, including messages of the failed assembly commands and registers to be sent back to Gibson Research. I did only try it the once (Level 2) and was a little reluctant after it failed so early to persevere as I would like to be conservative rather than rush in. Also the drive is quite new (purchased 2012) and Spinrite was written quite a while before (2004 - I think) and I thought it might not be compatible with either the drive or my 64bit architecture. If you have experience using it on newer drives and architectures I would be happy to hear about it, and may give it a re-run.

@pajenn

Thanks, I will look into your suggestion.

 

The year the drive is made and/or architecture does not matter to Spinrite, because it works at such a low level (I'm pretty sure Steve Gibson has said this many times before). You can also email him to find out for sure. I'm quite surprised that Spinrite failed, you should definitely email Steve with the error messages. I'm positive he can help you out and let you know if the information on the drive is recoverable at all. He's a guru on this subject and definitely knows as much or more than anyone else about this stuff.