Fully encrypted Truecrypt drive accidentally formatted

Hi,

I searched a lot on this and other forums to find a solution to this problem in vain, and I decided to post on this forum because it's the one that helped me the most, any help will be immensely appreciated.

Here is the problem: I have all of my life's data on a 2TB external Truecrypt Hard Drive (Device-hosted, entirely encrypted, with no partitions, and yesterday I accidentally formatted this drive instead of the new one I just bought yesterday which is the same brand and model.

Unfortunately the backup drive I had broke a few weeks ago and now I have no other copy of my data except for the current hard drive.

I tried to use the Truecrypt option "Restore Volume Header" to no avail, and I don't know if the procedure I found on this forum used to restore the header on a partition-hosted encrypted drive would work in my case.

I only quickly formatted the Hard Drive (NTFS, using Linux Debian), so I assume that the backup header that should be located at the end of the drive it's still intact

Is there a way to use that header to be able to access again all my data?

Thank you for any help you may give me.

 

According to this page http://www.truecrypt.org/docs/volume-format-specification the backup volume header is located at S-131072 where S is the size of the volume in bytes, which is the same size dantz described in another thread regarding partition encryption, but i'm not sure how to use that information to restore the header since i did not have a partition.

I have already downloaded and installed an evaluation copy of WinHex on my Windows machine, and installed GHex on Debian. I also installed TestDisk but I'm not sure if I can use it to restore the header

I should also mention that when I used the "Restore Volume Header" option on Truecrypt the formatted drive was partitioned, so i thought this may prevent truecrypt from using the backup header. I tried unmounting the partition in the OS, however Truecrypt was still displaying it in the window where it lists all the drives and partitions available to mount and when I tried again to restore the header it failed.

If I delete the partition leaving only unallocated space and try again restoring the backup header with the option in Truecrypt is there any chance of success? Or do I risk to do more harm than good?

 

According to the docs (and further docs) you must try to mount the volume with several attempts in a row.
Also, make sure to choose the correct partition ("\Device\Harddisk?\Partition0" represents the entire disk).

 

Thank You for your answer, racoon_tc, I did try what you suggested, I tried a couple hundred times, of course selecting the entire drive and not just the partition, I even tried unmounting the partition in the OS and tried to mount the drive again but with no results.

I tried to open the hard drive with GHex and noticed that the last couple of MegaBytes as well as the first couple are completely zeroed, I fear Linux did that when I formatted, even though I used the quick format option. Is there any chance of getting the data back if both the header and the backup header are lost?

 

Unfortunately, if both the standard header and the backup header are lost, you cannot mount the volume any longer. The erasure of the last sectors sounds plausible even if you have chosen quick-format.

 

Is there any way to check if both the standard header and the backup header are truly lost?

 

In my experience a quick-format doesn't overwrite the embedded backup header of a device-encrypted disk, although I haven't tried this using Linux Debian.

If the header is still good then it should work when you use TrueCrypt to attempt to mount the disk (not the partition) and select the "use backup header embedded in volume if available" mount option. TrueCrypt automatically looks in the correct place when you do that, and if the header is intact then it should work.

However, if you want to go in and look for it manually, just to see if it looks ok and has not been overwritten by zeros or something, then try this WinHex procedure (or adapt this to another hex editor if you prefer):

Open WinHex
Tools: Open Disk
select the correct Physical Media
Click once in the offset column to switch to Decimal offsets
Navigation : Go to Offset: 131071 Bytes (decimal)
specify "Relative to end (back from)"
click OK

You should now be at the very beginning of the embedded backup header. The next 512 bytes are the crucial portion of the header that must remain unaltered and intact. It should look totally random, so if you see any repetitive patterns in there (such as 00 00 00 00 00 or any other non-random patterns) then that header has likely been overwritten.

Note: If WinHex can't display Decimal offsets at the end of your disk because your disk is too large then you can use Hexadecimal offsets instead. In that case enter 1FFFF (hexadecimal) instead of 131071 (decimal).

Also, before you give up, ask yourself if you are absolutely certain that the disk was encrypted using entire-device encryption. Are you sure you didn't merely encrypt an existing partition? If you had encrypted a partition then what you're seeing at this point would be normal, and the recovery process would be different.

Another approach that I would try: Confirm the overall scenario and the assumed outcome by performing the quick-format again using a different disk (hopefully you have a spare disk that you can mess around with), then check to see whether or not the end of the disk gets overwritten. It shouldn't be too hard to prepare the disk in advance so you can observe the changes. Fill that portion of the disk with non-zero characters before you quick-format it, for example. Or use TC to device-encrypt the entire disk, and then see if the quick-format breaks it again.

Afterthought: Of course you have a spare disk, you just bought one. However, before you go ahead and mess it up, I suggest you look at it very carefully to make sure it's not the wrong disk. Maybe you have swapped them and are mistaken about which is which.

 

THANK YOU for your answer dantz, sorry that it took me so long to answer.

Yes, I did check using the offset you suggested, and to my amazement I discovered there was random data from then on for at least 512 Bytes!!!

I am absolutely sure I used device-hosted encryption, and I did try to mount the device (not the partition) repeatedly hoping for it to mount to no avail.

The option you suggested ("use backup header embedded in volume if available") I cannot find on Debian, I will try on Windows as soon as I can.

Now that I found the backup header could you please give advice on how to restore it and finally be able to access my disk again?


UPDATE:

I tried to mount the entire Hard Drive using TrueCrypt on a Windows machine with the option "use backup header embedded in volume if available" ticked,
but unfortunately it said "Incorrect password or not a TrueCrypt volume", I tried selecting the partition too but with the same result.

2nd UPDATE:
Something very strange happened: when i plugged in the formatted hard drive on the Windows machine I noticed strange characters instead of "New Volume", and when I opened the hard drive I could see folders named with random ascii characters and files that I am sure I never had because are too big. I am not sure what all this means.

 

I don't have time right now, give me a day or two to get back to you.

 

Did the area of random data extend from there to the very end of the drive? It should. In fact, the whole frickin' drive should be full of totally random data, except any portions that were overwritten during the accidental format.

TrueCrypt ought to be able to find the embedded backup header all by itself when you select the entire device (the disk, not a partition) in the Select Device screen. It merely starts at the end of the disk, looks backwards the specified number of bytes, and tries to process whatever it finds there. If the header is there, and intact, and you supply the correct PW, then it should work. Sorry it's not.

You could also use WinHex to block that area (starting at the offset I mentioned earlier and going forward for at least 20KB (or to the end of the disk if you like, it should work either way) and save it as a file, and then try to mount the file using TrueCrypt. If the file mounts then you've managed to locate a working copy of the embedded backup header, but I wouldn't get too hopeful, because TC itself usually has no problem finding the embedded backup header of a fully-encrypted device. It never moves, because there aren't any partition boundaries to get screwed up. TC merely goes to the end of the physical disk and then looks backwards the specified distance, so it can't really get lost doing this, unless for some weird reason your disk itself has changed its size (I don't know how).

Not good! Of course, the drive would have been fully random if it hadn't gotten messed up, and Windows would have even less to say about it in that case. Let's just hope this is a case of a screwed-up (but unimportant for our purposes) format rather than some sort of hardware error.

I'm still not sure of how else to proceed! Personally, I would take a spare (empty or unimportant) disk, delete any partitions, use TC to encrypt the entire disk, and then try to carefully repeat the entire sequence of events (accidental format, etc.) to see what happens and whether or not the TC volume can be mounted afterwards using the embedded backup header. Make sure it works normally before you intentionally screw up the disk, and then try it again afterwards. Look with WinHex too, of course.

The question we're trying to answer is, would Linux Debian have overwritten that portion of the disk (the location of the embedded backup header) during a quick NTFS format? If not then maybe something else is going on. Something that needs to be understood.

 

There's a portion of the drive at the end which is full of zeroes, I don't know why.

I tried to save a dump of data from the offset for 20 KB of data but it would not mount with Truecrypt, also I should add that I noticed an option in the Offset search, where I can select either "Offset" or "Sector", and the default is Sector, which I used previously, I don't know exactly what this means and I couldn't find a manual or information about it.

If I try to search for an offset from the end of the disk with the option "Sector" it displays random data for at least 512 KB, if instead I select the option Offset it displays random data. I will try with a version of WinHex on Windows as soon as I can.

I did the whole procedure with a USB drive (not a hard disk, I am trying now with it), if I format with Windows AND I try to mount in Windows it will mount (it won't mount on Debian), if instead I format with Debian I cannot mount with either Windows or Debian.

I will try to see if I can mount a hard drive after formatting it with truecrypt as soon as I can and i will see what WinHex will display.

But, let's assume that for some unknown reason the backup header and the standard header are lost forever, Is there really no way of retrieving the data?

UPDATE:
I installed WinHex on Windows and when I searched for the offset (Go to Offset > 1FFFF Bytes (hexadecimal) > end (back from)) I discovered that from exactly that point to the end of the disk there's only zeroes, when I searched on Debian I must have searched for Sectors. The rest of the disk is random data except for the first few MegaBytes (from 4B to A1FA400) which again are zeroes. Is everything lost?

 

Not unless you have a backup header stored somewhere, or an image of the disk that might contain that information. However, we still don't quite understand what happened, and I hate to make any firm statements without considering the "unknowns" of the situation.

I had originally assumed that the block of zeros that you found near the end of the disk was related to the formatting that occurred under Debian.

However, now that you've described the exact location of the zeros, my thinking has changed. The only program I know of that would write data to that specific location, while ignoring everything before that location, would be TrueCrypt for Windows. (I don't know about the Linux version). But TC for Windows would not write a big block of zeros there, it would write 131,072 bytes of random data (consisting of the two embedded backup headers).

Maybe the Linux version of TC writes zeros there (or just skips over that area) when you create a disk-based volume? I don't know why this would be the case (although I could make some guesses), but it wouldn't be that hard for you to check. You already have all the tools, just try it and see. Use your test disk, zero it out, encrypt the entire raw drive, DON'T format it, then have a look at the end of the disk using WinHex. You could also post to the TrueCrypt forums, in the Linux subforum, and see if anyone there knows.

Are the zeros there on both disks, including the disk that you're using for testing? Sorry I don't know more about the Linux version of TC.

But yes, at this point the situation is looking fairly bleak.

 

I forgot to mention that I encrypted every hard drive with Truecrypt for Windows, never once I encrypted a device using Linux, I only use Linux to mount the drives.

I encrypted the other drive as I did before on Windows and there's random data at the offset at the end of the drive. If then I format it on Linux the last 1FFFF Bytes get zeroed, confirming the theory that a quick format on Linux also deletes the last 1FFFF Bytes of data.

Considering that I have no other backup of any part of the disk or the headers and the last 1FFFF Bytes and the first A1FA400 Bytes of the disk are zeroes is there anything else that I can do before going into despair?

 

What is the condition of this other broke drive? How did it fail? Do you still have it?

 

Actually, that description is short by one byte. I only used 1FFFF to get you into the correct position. I had to subtract one, since you were counting backwards from the end of the disk and your cursor was already on the last byte. The actual size of the complete block (and the two embedded backup headers) is 20000 (hexadecimal), or 131,072 bytes (decimal).

But what a strange coincidence! Why would a Linux quick format write zeros over the exact location of TrueCrypt's embedded backup headers on a fully-encrypted device? If the overwrite had just begun one sector later then the crucial portion of your embedded backup header would still be present. Very strange.

Now let's just hold on a second, before you go into despair. You stated earlier that your drive was fully encrypted, device-hosted, with no partitions. This may in fact be true, but let's make sure. It's been my experience that a large number of TrueCrypt users believe that's what they've done, when they've actually encrypted a pre-existing partition that was already present on the drive. It's amazing how many users think that they have fully-encrypted their drives when they haven't. Let's just check:

1. If you encrypted an entire, RAW drive then that drive would appear in Windows' Disk Management window, but no partitions would be listed. Windows would (at various times) prompt you to initialize (but not format) the drive. You would never see the "Do you want to format" prompt for the unmounted drive. Do you confirm this?

2. If you tried to encrypted an entire device and it already contained partitions and data then TrueCrypt would stop you and would steer you towards partition encryption instead. If you followed the steps, you would then have the option of encrypting your data "in-place". (If you did this then you encrypted a partition, of course.)

3. If the drive contained partitions and you insisted on encrypting the entire device then you would first have to exit TrueCrypt and then clear the drive (delete all the partitions) before you could return to TrueCrypt and encrypt the entire device. Next you would need to copy your data back into the mounted volume. (There would not be any partitions on the drive at this point). Confirmed?

4. Whenever you used TrueCrypt (for Windows) to mount a fully-encrypted device (manually, not by pressing "AutoMount Devices"), you would click on "Select Device" and then select the correct disk such as Harddisk 1, or Removable Disk 2. You would never select anything with the word Partition in it. However, in the main TrueCrypt screen, the selection that you made would resolve to something like \Device\Harddisk1\Partition0". It would always say Partition0, never Partition1 or higher. Correct?

5. If, as you stated, you mistakenly formatted your fully-encrypted drive, (as you said, "I only quick-formatted the hard drive"), then you would have to begin by initializing it (at least, in Windows), and then you would need to partition it, and only then you would be able to format the newly-created partition. Is that the process you followed?

I'm sorry to be so annoying. I'm just making extra sure. Because if you happen to be mistaken, and if you merely lost an encrypted partition, then the whole recovery process would be different and there might still be a chance. We haven't checked the right areas if we're looking for a lost partition. (This is my last ditch attempt. I don't give up easily!)

But yeah, if none of the above pans out then I guess we're basically done. But try not to be too hard on yourself. Lots of people have lost all of their data (yes, all!!), but the truth is, there's still a lot of stuff left after it's gone. Let's list a few: body, health, friends, loved ones, good brain, future, and a lot more, I'm sure.

 

Well, it has been broken into little pieces, and thrown away, so I doubt there's any chance retrieving the data from it.

1. Yes, Everytime I plugged the disk on Windows it would ask me if I want to initialize it.

2. Before encrypting the device I had to delete all the partitions

3. Confirmed

4. Yes, Partition0. Always

5. I had to format the MBR, and then create a new partition (I did all this on Linux)

I guess I can go into despair now. Thank you all for your help, especially dantz as you were the most helpful.

 

You're welcome. Sorry we couldn't solve it.