What happened exactly after formatting my USB stick

Hi, I found the answer to my question in my previous topic:

https://www.wilderssecurity.com/showthread.php?p=1695421

I put it to the test.

I expected one of three possibilities:

• Everything is recoverable.
• Nothing is recoverable.
• For every block of 4K there will be 2 or 4 bytes lost.

This is what I tried:

• I format my USB stick. (1003.5 MB free).
• I make a file of 1000 MB on my hard disk with random numbers of 4 bytes.
• I copy it to the stick.
• I format the stick again.
• I make a new file on the stick, open it, set its length to 1000 MB and close it.
• I copy that file to my hard disk.

Then I tried several times to choose a number somewhere in one file and search that number in the other file.
I tried that with both files.
Whatever number I chose, I didn't find it.
So if it's possible to find a number from one file in the other, then the chance is very small.
So after formatting my stick all data is probably completely unrecoverable.
But this isn't necessarily the same for all USB sticks.
That should depend on the controller chip.

 

Probably...
What Brand do you have?

 

The controller chip is an OnFlash 5188B.
The memory chip is a K9G8G08U0A.

 

I'm not sure what and how exactly you tested, especially
"I make a new file on the stick, open it, set its length to 1000 MB and close it."

If you don't overwrite the complete USB stick some data will always be recoverable.
You can easily verify this for example by writing a pattern to it with badblocks and then open the usb drive in hexedit. The first several sectors will be overwritten when you create a new FS, but after that everything is still there on the same place.
You could also put some plain text files on it, delete and/or reformat and use the search function of hexedit to look for phrases or grep through the strings output.
(use a Linux live CD if you are on Windows)

As noted in the other thread newer Windows zeros out the complete disk when doing a full format.

 

I was using a Macintosh with OS 9.2.2.
If I format the stick then it's FAT16.
On a Macintosh you can make a new file, open it, set its length and close it.
Then you have a file that contains the data that was already in those sectors.
Since the data were no zeroes, they haven't been overwritten with zeroes.
Since the data were random, they have been overwritten with random numbers.

 

OK, thanks for the explanation. Would be nice though if you had used an OS that hasn't been obsolete for such a long time as that makes it kind of difficult to reproduce

So to make sure I understand it correctly: you use a ~1GB usb stick and have two ~1 GB files completely filled with random data but they completely differ.
I'm not sure what that could prove and my first suspicion is that there's a flaw in your methodology or the software you use, for example:
Formating also overwrote the drive (OS X has that option in Disk Utility)
When copying the sparse file to the hdd it doesn't copy it sector by sector but creates a new sparse file which is obviously filled with free sectors of the hdd containing whatever data was on there. Comparing the hashsums should rule that out.
The way you parse the file doesn't take into account that everything is likely shifted by several sectors or it's decoding the data wrong for some reason, use an hexeditor with a good search function.
Or it has something to do with wear leveling and the way data is written to USB drives which would really surprise me (that's handled by the firmware/hardware and completely transparent to the OS, so sector 1 is alway sector 1 unless it's been mapped out as a bad one)

In any case, if you don't overwrite everything data will be recoverable and a skilled forensics expert will be able to reassemble even broken and split up files. Therefore encryption or full overwriting is a must if you worry about this risk.

 

Windows XP is 5 months older than Mac OS 9.2.2.

I make a 1000 MB file on my HD with random numbers of 4 bytes. I copy it to the stick. I format the stick. I create a new file on the stick. I open it. I set its length to 1000 MB. At that moment, all the free blocks on the stick are linked to one file that contains the data that was in those free sectors. I close the file. I copy it to my HD. I start searching sequences of 4 bytes from one file in the other. If I find no match, but only garbled data, then that proves that the stick has been overwritten with random numbers. Now the speed at which this happens proves that it's not the Mac OS 9.2.2 software that has done this. (Mac OS 9.2.2 is 100% compatible with FAT16, but it works at a speed of 1.5 MB/second.) So I think that when you ask to format the stick, then the controller chip on the stick overwrites the memory chip with random numbers.

No, I never saw a program that can recover files from a USB stick. Norton Utilities for Mac OS 9.2.2 doesn't work on a USB stick because it's not a ATA or SCSI device. For a hex editor it doesnt make a difference whether the file is on the stick or on the HD. I just put it on the HD because searching on the HD is faster.

 

Windows XP is obsolete too... You didn't say SP2 which is newer and even then I'd call XP obsolete too. But at least I could reproduce it as it's such a common OS and works on any x86 hardware.
Can't you use another computer or a PPC linux distro with common tools like dd?

What about the hash check or open the file directly on the usb drive?

You may have never seen one but I can assure you there are several that can do it, maybe not for OS 9...

 

Show me one that can do it on my XP computer. I would like to have that.

 

Try TestDisk/PhotoRec:
http://www.cgsecurity.org/wiki/Main_Page

 

I know this sort of programs.
They can recover from CompactFlash cards but not USB sticks.
CF cards have a miniature ATA-controller.
They are ATA-devices.
That's the reason why those recovery programs work for CF cards.
So they will advertize their program as "this can recover from removable media".
But it doesn't work on USB sticks because they are not an ATA-device.
If you want to be able to recover from removable media, then you better use a CF card.

Now, do you know:

1. a program for XP that can with certainty recover a file from a USB stick with FAT16, a file that was erased by normal deletion?
2. a program for XP that can with certainty recover a file from a USB stick with FAT16, a stick that was erased by formatting?

Is there a free program?

 

Cliff Huylebroeck
There is of course the chance that this indeed depends on the chip controller as you say, that's easy to check if you can get your hands on another usb stick and do the same tests.
However you are wrong about TestDisk/PhotoRec.

I tested it myself, here's how:
I first overwrote the thumbdrive with zeros so we don't have to deal with a tons of old data.
I formatted with FAT32 and put 4 files on it, then formatted again.
PhotoRec recovers all 4 files, the exe is corrupted, filenames are lost but the
image and documents are completely restored.

The fact that I used FAT32 and Windows 7 as opposed to Windows XP and FAT16 shouldn't change a thing. If you are convinced of the opposite, prove it.
A few more questions:
Did you finally compare the hash sums fo the two spare files?
Did you try with different USB sticks?
Have you actually ever use PhotoRec?

(the attachements got mixed up, the second one should be the last)

 

You chose the option Quick Format.
On Mac OS 9.2.2 this option is not available.
That could have made the difference.
I copied TestDisk/PhotoRec.
I'll try it on XP.

 

It recovered 464 files of 473 deleted files.
Only 23 MB of 217 MB was recovered from the empty space.
The recovered files were the files that I just deleted and 1 file that I deleted the day before.
I formatted the stick (about 700 files = 97 MB deleted).
It recovered 1757 files = 99 MB.
In other words: another 21 MB (= 97 + 23 - 99) was lost.
I formatted the stick.
I copied the 1000 MB file with random numbers to the stick.
I formatted the stick.
I tried to recover.
It recovered nothing.
http://img835.imageshack.us/img835/7601/nothingrecovered.gif

 

I can try a combination of the test in my first post and PhotoRec.
I write files to the stick, files that can be easily recovered, like jpegs.
I delete them.
I create the 1000 MB file by just setting its length.
I delete it.
I format the stick.
I try PhotoRec.

If everything is lost like in my first post, then I found a fast way to destroy the deleted files in the empty space.
(If I have to overwrite the empty space with zeroes on Mac OS 9.2.2 then that takes 23 minutes.)
If it can recover the jpegs, then I don't understand why ALL data was lost in the test in my first post.

 

Of your methods, this is the most reliable way to secure delete/wipe free space. And it also cross-platform compatible.

This is not reliable and it's not cross-platform compatible, and as you say, it's dependent on the controller chip and also on the software you use to create the file.

 

I tried what I described in my last post.
PhotoRec could recover about 90%.
Then I tried the first test again but with sequential numbers instead of random numbers.
My program recovered the entire 1000 MB file except the first 128 bytes.
All numbers were in order, except the last 128 bytes were garbage.
I still don't know what happened with the random numbers.

 

I wrote a program for Mac OS 9.2.2 that can erase unused space on a USB stick:

1. the unused part of the last block of every file. It scans the directory and for every file it looks at the difference between the logical and the physical length. Then it adds that number of zeroes. Then it sets the length to the original logical length.
2. the free space. It searches the block size and the number of free blocks. Then it makes temporary files that are a multiple of the block size, filled with zeroes. Then it deletes the temporary files.

The speed of part 1 is acceptable, but the speed of part 2 is inacceptable (Mac OS 9.2.2 supports no more than USB 1.1).

Part 1 was about 64 MB on a stick of which 105 MB were used, so it's really necessary to wipe this space.

I'll write a Windows program that makes a file with zeroes that I can copy to the stick with USB 2 to overwrite the empty space. Then I have a complete and fast solution.

 

Or you can use on of the ready made solutions like Heidi Eraser

 

I tried USB Flash Tools v2.10, a free program.
On XP with USB 2 it was slower than my program on OS 9 with USB 1.
The unused part of the last block of a file is called 'file slack' in this program.
Do you know a program for XP that can make a file of a specific length, for example a file with zeroes or a file with perfectly distributed random numbers?
Then all I have to do is erase the file slack, make a new file and copy it to the stick.
That will be fastest.

 

They call it a test vector generator.

http://en.wikipedia.org/wiki/Test_Vector_Generator
http://en.wikipedia.org/wiki/Automatic_test_pattern_generation

I installed Visual C++ but I forgot how it worked.
I created a dialog app with app wizard in no time and I don't know what next.

 

First I made it work on XP.
Then I installed Windows 98 in Virtual PC for Mac, then Visual C++.
It looks oldfashioned but it works.
Unfortunately, I can't type {}[] on this Mac azerty keyboard in Windows 98 in Virtual PC.
A Windows USB keyboard could work.