Restoring TrueCrypt partition from DriveimageXML

I asked about this on the TrueCrypt forum but didn't get any help. By Googling for information I discovered that this forum seems to have some real TrueCrypt experts, so hopefully you guys can help me.

I have a 500gb drive that was formatted by TrueCrypt as a non-system partition. I thought I didn't need the data on it, so I formatted the partition with NTFS and overwrote about 1/3 of it. Then I realized there was stuff on it that wasn't backed up, so I tried mounting the partition with TC to at least restore some files, and sure enough it mounted with the backup header.

No files were visible of course, because the MFT was gone. Before trying to recover files, I backed up the whole partition with DriveimageXML set to raw mode.

But then I mounted the partition again and stupidly tried to repair the header. Since 1/3 the drive was overwritten and Windows still had it mounted as an NTFS partition, I guess Windows didn't let TrueCrypt do that. Then TrueCrypt returned an "access denied error" I believe.

I restored the backup from DriveimageXML, but TrueCrypt couldn't mount the partition. I tried formatting it, and that actually prevented me from restoring the image again because the partition was too small... I figured out that Windows 7 (my current OS) wastes some space, but formatting the partition with XP allowed me to restore the backup again.

TrueCrypt still can't mount it. I thought maybe the backup header wasn't in exactly the right place due to the backup/restore process, but the partition and the image are both the same number of sectors (976,768,002 with 63 offset). Then again, maybe there was some issue during the imaging process that put the backup header in the wrong place.

Any ideas?

Thanks in advance

 

I just had an idea. Is it possible to manually extract the header, decrypt the key from it using the password, and attempt to mount the drive using that? That would let me try potential headers starting at various distances from the end of the partition.

 

I agree that one option is to 'test around' until you can find the embedded backup header, but if you are able to restore your raw image without changing its size then the partition endpoint should already be correct and the header will be in its usual location 131,072 bytes back from the partition's end.

Are you familiar with the testing procedure? Here's how I use WinHex to grab and test the (potential) embedded backup header:

1) Go to starting point of embedded backup header:
Position; Go To Offset; 1FFFF (bytes hexadecimal); end (back from); OK.
(note: It's 1FFFF instead of 20000 (hex) because you're already on the first byte, so we go back 131,071 more bytes.)

2) Select header plus some additional data (so resulting file will exceed TC's minimum volume size of around 20KB):
Edit; Define Block; Beginning [current position]; End [end of file]; OK. (note: instead of choosing [end of file] you could add approximately 20KB onto the end of the block, if you want to calculate that out, but since we're already fairly near the end of the partition, I just extended the block all the way to the end because it was more convenient).

3) Save selection as a file:
Edit: Copy Block; Into New File; enter a filename such as "HeaderTest-1.tc".

4) Test the file to see if the header works:
Open TC, select the test file and try to mount it as though it were a normal container file. If your password is accepted then you've located the embedded backup header.

Be aware that you will also need the partition's starting point to be exactly correct or the data will not decrypt, even if you manage to get the embedded backup header working, so it's crucial to restore your image without altering its size in the slightest. If possible I would focus on doing that rather than starting out with the embedded backup header out of position and then trying to figure out where it's hiding.

PS: If you restore your image into a target partition that is too large then perhaps you can visually spot the endpoint of the restored partition. Make sure the target partition is zeroed out, then restore the image, then look with WinHex and try to spot the end of your encrypted data where it transitions to zeros. Then try my header testing method (above), but modify it to match the endpoint of the encrypted data.

 

Thanks, Dantz.

Yesterday I opened the DriveimageXML image file in WinHex (actually just the last piece since I have it segmented into 4gb parts), and it matches the end of the partition exactly. No zeroes at the end.
To recap what originally happened:
1) Drive had one partition either made at the Maxtor factory or by WinXP (I don't remember if I ever used it unencrypted)
2) Years ago formatted the partition with TC.
3) Filled it up with data
4) A few months ago formatted NTFS using Windows 7
5) Wrote 160GB to partition
6) Tried to mount with TC -- it worked
7) Backed up partition using DXML
Mounted again and tried to restore header but process failed and TC then said that it's not a TC volume
9) Restored image from DXML but mounting still didn't work
10) Deleted partition and created a new one with Win7
11) Tried to restore DXML image to new partition, but partition was too small
12) Deleted partition and created new NTFS one using old Windows XP computer
13) Successfully restored DXML image again
14) Partition still won't mount




So I need to select from the end of the file back to and including the backup header, plus some additional bytes to get to at least 20KB?

Then if that doesn't work, do the same process but exclude the last sector? Then try excluding the last 2 sectors? Etc etc?

Or are you saying to export small files with the backup header positioned at the beginning?


Also, could you please clarify what you mean about the starting point of the partition? Do you mean where it begins on the physical drive? What I know is that the partition and the image seem to be exactly the same number of sectors. But does that necessarily mean that it matches the original state of the hard drive? DriveImageXML can only image a partition, rather than the entire physical drive. In other words, I imaged "Partition1" rather than "Partition0". But I'm pretty sure that's what I always selected in TC anyway--Partition1.

So in my mind, DXML and TrueCrypt both operate on partitions, and the partially overwritten partition that I managed to mount with TC before is the same one that DXML copied.

 

Yes, that's basically it, but rather than excluding various sectors at the end, just move your starting point around. And you'd probably want to look forward, not back. However, you said that the end of the restored partition is identical to the end of the DriveImageXML file, in which case you shouldn't have to search around for the header at all. It should be in the expected location and it should already be working. Unless, that is, you suspect that the image file somehow missed copying some data from the end of the original partition. Did you alter the partition boundaries before making the image, or something like that? If that somehow happened then you will have a pretty huge job testing all nearby sectors. I would suggest using an automated approach.

Be aware that this whole approach is kind of a long shot. If your image is accurate then you shouldn't have to hunt at all. And I still don't understand why the embedded backup header stopped working. Was there a software crash in the middle of the header restore process? But that shouldn't matter, because you made the image prior to attempting to restore the header, right? Or is your step-by-step description inaccurate? I suggest you go to the expected location of the header and examine the first sector to see if it looks non-random in any way. The first 512-byte sector is the only portion that matters.

Yes, that's a part of the testing procedure that I described earlier.

I wouldn't worry about that right now. You need to get the header working before this becomes an issue, and you've probably already imaged the entire partition, including the beginning. It doesn't matter where the partition starts on the physical drive as long as its size is identical to the original partition.

I suggest you practice the test procedure on a small TrueCrypt container file. It works the same way.

 

Thanks. Yeah, I was thinking about trying it with a test container.

My steps are accurate. I imaged the partition, then I mounted it with TC and tried to repair the header. It had me move my mouse around to create the random number, then tried to restore but failed and gave some error. I'm guessing it was unable to write to the beginning of the partition. I don't understand why that would also mess up the backup header though. You would TC would be set up so that if repairing the header failed, you wouldn't lose all your data.

Yes, it shouldn't matter because the image is from before I messed it up. I think the only explanation is that DXML didn't copy the image properly. But it's exactly the same number of sectors as the partition that I made with WinXP. If the image was bigger than the partition, DXML would refuse to copy. If the image was smaller, the bytes at the end of the partition wouldn't match the image. That should indicate that nothing is missing. I don't really get it.

 

I must be going insane...

I accidentally went to 1FFFFF offset from the end.

The bytes there are exactly the same as the ones at 1FFFF. I'm scrolling down and down and not seeing any difference. Why would the data at 1FFFF from the end be repeated at 1FFFFF?? And 1FFFFFF is the same, and 1DFF, and 1DFFF, etc. It seems like the header, if that's really what it is, is repeated many many times and so is the sector right before it.

Screenshots:
http://i.imgur.com/UUUfB.jpg
http://i.imgur.com/o6Sr9.jpg

Edit: I copied 64 kilobytes out of the drive, starting with that header, and TC mounted it. Here's the catch... the password I use for outer volumes is the one that worked. It shows up at 51gb in TC though (It's a 500gb drive). I guess this means I had the partition set up as hidden and forgot about it. BUT that password doesn't work on the drive itself (why not?).

Sure enough, the password also works on the header starting at 1DFFFF from the end


Edit again: I can mount the last DriveimageXML file using the same outer volume password. It just doesn't seem to work on the drive.

Well I guess I've found the outer volume header. The hidden volume header, according to the TC website, should be at the very end of the drive, right after the outer volume header. .

Edit yet again: I went to offset FFFF (65545) which I think is where the hidden header should be. It's exactly the same as 1FFFF etc. What does this mean? That DriveimageXML didn't image the drive properly? Why would it copy the header, of all things, to so many locations?
http://i.imgur.com/7a5sh.jpg

 

I'm guessing this means the data is lost... Unless the hidden volume header is somewhere other than the very end of the partition.

One more thing I'll try is the Passware demo. I have a hiberfile.sys from a year ago when I upgraded my hard drive and didn't format the old one. The odds are extremely low that I had the drive mounted when that hiberfile.sys was created, but might as well try.

 

Your results are difficult to explain. My first guess is that you're messing with me. Another possibility would be some sort of WinHex-related problem. Perhaps you've been copying and pasting within the open file by mistake. WinHex defaults to read-only mode (in Options). Which edit mode are you in?

 

I promise I'm not messing with you!

When I started it up, I selected read only mode with the forensic interface.

Well actually the first time I ran WinHex, I did it straight from the installer folder without actually installing it. For some reason I figured it was a portable app. I think I selected read-only that time too but I'm not sure. The program seemed to work but it showed a lot of zeroes at the end of the partition. After I installed it and ran again, it showed the whole drive properly (I assume).

 

Maybe you should try another hex editor, just in case. HxD is quite good and is freeware.

 

OK, I'll try that one. WinHex just finished imaging the drive (I made an image to try PassWare on). It copied 972,768,000 sectors instead of 972,768,002 that DriveimageXML found...

 

I made a new temp copy of the 500MB final chunk from the DXML image (in case WinHEX had messed up the one I was working with), and it's the same deal. That AC 68 42 2B... hex string is at 1FFFF, FFFF, 1FFFFF, etc etc. The whole of that 500MB chunk is that header repeated. I zipped the file and it compresses down to 3.26MB.

I can understand DXML messing up and copying the same thing over and over, but it seems strange that it would happen to be the outer volume header

 

Very strange indeed. I'll play around a bit to see how hard it would be to accidentally copy the selected blocks in WinHex, as I can't really come up with much else in the way of explanation. It's extremely unlikely that a DriveImageXML error would focus on that specific data, as the headers are indistinguishable from the rest of the encrypted partition. More likely it happened afterwards while you were focusing your attention on those areas of the image file. Bummer if you overwrote some of your data, though.

Have you compared your image file to the image that you restored to disk?

 

I checked 1FFF, 1FFFF, etc on both the disk and the image and they're the same. I didn't check every single byte though.

I didn't mess up the backup. Before I opened it in WinHex I copied the chunk over from the external drive to my system drive. To make sure I didn't cause the error with WinHex, I copied it again from the external drive to a different folder and checked that one with HxD instead. So it's definitely the image that has the error.

I also have a second copy of the image on another external... I wasn't taking any chances!


I'm looking at the second to last chunk right now, and it's the same way. That header repeated over and over. I know the data at the beginning of the image is good, because I can access it fine on the drive. So it seems like at some point DXML started filling the files with the header. I really don't get why that would happen.

I also don't get why Truecrypt doesn't mount the drive using the outer header password. It should recognize the header like it does with the files.

 

There are a lot of strange things happening here, so be extra careful not to mess things up.

FYI, you can paste the first 512 bytes of your embedded backup header onto the very front of the volume, and it should work just like a regular volume header. Be careful not to increase the file or partition size, though. Suggest you practice on a copy if possible.

Before you do this, mount a test file with your embedded backup header, pull up the "Volume Properties" screen in the TC interface and see if the volume size matches the size of the original partition. It should show the size of the partition minus the four 64KB headers that wrap around the volume, so add 262,144 bytes to the TC number to (hopefully) get the size of the original partition.

 

The size of the outer volume should be displayed as the full partition size right? In this case it should be 466GB, but TC says 54760570880 bytes (51GB)

http://i.imgur.com/LAr1b.jpg

 

We obviously don't quite have a grip on what's going on here. You may have found the hidden header, but it was apparently located where the outer header goes. Weird. But ok, the real question is, is this header capable of decrypting your outer volume's data? Copy a test file (and it should be large enough to include some user data, so make it large enough to exceed whatever overwrite occurred) from the very beginning of your image, then replace the first 512 bytes of the file with the first 512 bytes of the header. Mount the test file and examine the mounted volume for decrypted data. Scroll through it far enough to bypass the overwritten portion, then look for nonrandom data such as readable text, 00 00 00 00 or other obvious patterns.

Actually, since you apparently overwrote over a third of your disk, the above test isn't very practical. You'll probably need to perform it on the disk itself. At this point I don't think there's much to lose by overwriting the first sector, and if you want you can always save it and then copy it back afterwards.

edit: This probably won't work because the scope of encryption listed in the header is too small, but you get the idea.

 

It has to be the outer volume header, because the password that works is the one I use for my outers. The outer volume would only have had a few files-- the hidden was the main one that most of my data was on. There's no way the hidden was only 51GB.

I copied the first 512 to the start of the disk and it mounted in TC as the 51GB volume. Now I'm running GetDataBack. I doubt it will find anything though, because whatever files were in the outer volume would have been overwritten.

BTW why did I only have to copy the first 512 bytes of the header? According to the TrueCrypt spec, the header is 65536 bytes long http://www.truecrypt.org/docs/?s=volume-format-specification

 

You don't have to find actual files to see if it's decrypting, although GetDataBack might be limited to that. I would use a hex editor to search for a commonly-found string such as 00 00 00 00 00 (hex). This should exist in abundance even in areas of the partition that never contained any user data. In WinHex, "Search; Find Hex Values; 0000000000; Down; OK".

I honestly don't know what the rest of the header is used for. If you ever find out, please let me know. My guess is, nothing. As far as I am aware, the first 512 bytes contains all of the "good stuff".

 

GDB found no NTFS partition... but I just realized the outer partition would be FAT32 anyway. I only own the NTFS version.

WinHex found no 0000000000. I think that's to be expected since everything in the outer volume had to be overwritten.


Too bad I couldn't find any other free programs besides DXML when I was imaging the drive. All I could find was Acronis and other paid ones. Just now discovered Easus.

 

It's not physically possible to brute force the 256 bit AES key right? My understanding is it would take billions of years or something like that. I assume knowing the password doesn't help... Or does it?

 

Edit: I found the first occurrence of the header (AC 68 42 2B....) and it's right after one of the new files I put onto the drive--- a 100gb TC file container. So it seems like DXML imaged the drive, then for the last 13% of the drive filled the image with the old header. I have no idea why.

Edit: I was scrolling through some empty space filled with zeroes (no idea where those zeroes came from) and found a 512KB segment... so used it to overwrite the first 512KB of one of my header test files. TC managed to mount it with the same 8 char password and it shows as a 100gb volume. WTF? And it's repeated several more times among those zeroes.

Basically here is what my image consists of:
NTFS header --- a few gigs of photos --- 65gb TC container --- empty space filled with zeroes and a 512KB header that mounts with my 8 char password --- 100gb TC container --- a 512KB header that also mounts with the 8 char password repeated to the end of the file


I think much more of the drive was overwritten than I thought. I forgot about the photos. I had deleted them before I made the two big TC containers, but I guess Windows chose not to overwrite the same space??

 

That's quite a scramble you've described. There's no way DriveImageXML could have created this sort of mess. It has absolutely no way of identifying TrueCrypt headers and thus it would be extremely unlikely to somehow focus on the headers and start making multiple copies of them all over the drive.

A much more likely explanation is that you somehow did this to yourself. It's fairly easy to do this sort of thing using WinHex. Once you have selected a block and chosen Copy, all you have to do is press Ctrl+B (write clipboard data) to write it back to the drive at the location of your cursor, and depending on the edit mode you are in you might not see much of a prompt to confirm.

In the future I will always start my WinHex instructions with "Make sure you are in read-only mode (the default). If for some reason you've changed the edit mode then change it back to read-only."

FYI, a fully-encrypted partition looks like totally random data from start to finish. The zeros were placed there during the formatting process (although the extent of the zero-fill depends on whether you selected a 'quick' or 'full' format.)

About the various headers that you're finding: You initially stated that you had a fully-encrypted drive and later on you added that there was also a very large hidden volume. That's two headers (plus their backups). Either header, if used to mount a test file, would indicate (in TC's Volume Properties) a very large volume, either the size of the entire drive or one almost as large as the entire drive. This data is stored when the header is created and it does not change, even if the header is used to mount a much smaller volume for testing purposes. So the headers for the relatively small volumes that you're finding are most likely not going to help you at all, and they shouldn't even be in the locations that you described finding them in. Probably you've restored them to the wrong locations or something like that.

Encrypted data is basically incompressible. That final 500MB chunk of your image file apparently contains mostly non-encrypted data, which indicates that the overwrite progressed much farther than you thought. Apparently almost the entire drive was overwritten.

You couldn't have copied all of that data into a partially-formatted partition. You obviously formatted the whole thing. A 'quick' format would have spared the TC embedded backup headers and much of the old encrypted data. And then you apparently overwrote more of your previous encrypted data by copying in new data and creating some new and quite large TC file-hosted containers.

Getting back to your first post:

I performed the same steps on my virtual machine. After formatting (quick format) I had no trouble restoring the headers from the embedded backups. I also tried it with an interrupted quick format, with the same outcome. Note that a TC volume must be dismounted before you can restore its header. Perhaps that's the error message you saw.

All in all, considering how much of your data was overwritten, I doubt if there's much to recover, even if you do manage to find the correct headers. And you've apparently altered things so much that it's almost impossible to figure out what's where or why. All in all, I don't think there's much hope of unscrambling this egg.

 

Yes I agree, nothing seems to be recoverable.

But I definitely didn't mess all this up in WinHex. Remember, I copied that final DXML chunk again from my external drive backup and viewed it in WinHex, and it was exactly the same. I never opened the original image files with WH, only the drive itself and copies of the image chunks.