Gutmann’s 35-Pass Overwrite Technique

[Pleonasm]

I was recently looking at Peter Gutmann’s famous 1996 paper Secure Deletion of Data from Magnetic and Solid-State Memory (http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html) again, and noticed that it now contains an epilogue:

-{ Quote: " In the time since this paper was published, some people have treated the 35-pass overwrite technique described in it more as a kind of voodoo incantation to banish evil spirits than the result of a technical analysis of drive encoding techniques…. For any modern PRML/EPRML drive, a few passes of random scrubbing is the best you can do. As the paper says, "A good scrubbing with random data will do about as well as can be expected". This was true in 1996, and is still true now." }-
This practical advice from the “guru” of file erasing may be of interest to readers of this forum.

[HURST]

Yeah, that was discused some time ago in a tread i'm too lazy to look for right now.

If I remember well, in modern disks, 35 passes are overkill.


EDIT: I looked for it after all ;D: http://www.wilderssecurity.com/showthread.php?t=194601

[Carver]

I now use the 7 pass Schneier method

[LockBox]

That has actually been in more threads than we could probably count in one night. It's still good reading though.

[steve161]

While the shredders in the context menu of my 2 linux partitions use the 7 pass method, I only use 2 or 3 passes for free space wiping. I have yet to see a specific example of a file being recovered, even marginally intact, after a 3 pass wipe.

[n8chavez]

-{ Quote: "I now use the 7 pass Schneier method" }-

Can I ask what the advantages are to using as opposed to say a 5-6 pass of psuedo data? I have read that psuedo data is better because the wiping wouldn't be obvious.

How does the Schneier method differ from DoD?

[caspian]

-{ Quote: "Can I ask what the advantages are to using as opposed to say a 5-6 pass of psuedo data? I have read that psuedo data is better because the wiping wouldn't be obvious.

How does the Schneier method differ from DoD?" }-

Well if there is nothing recognizable there, wouldn't it be obvious that it was overwritten?

[LockBox]

Bruce Schneier (7 passes): The Bruce Schneier method offers a seven pass overwriting algorithm. The first with all ones, the second all with zeroes and then five times with a cryptographically secure pseudo-random sequence.

[HyperFlow]

i tried a 35 pass a few times wOw the time it takes :wacko: i just do 3 passes now ;D

[Dogbiscuit]

-{ Quote: "A single write is sufficient if the overwrite is truly random, even given an STM microscope with far greater powers than those in the references." }-
Can Intelligence Agencies Read Overwritten Data? (http://www.nber.org/sys-admin/overwritten-data-guttman.html)

[Pleonasm]

Dogbiscuit, the article you referenced (“Can Intelligence Agencies Read Overwritten Data? A response to Gutmann.”) is fascinating, and I recommend that all who visit this thread read it.

The article raises the following challenge: is there any empirical evidence that a governmental agency or data recovery company has ever been able to successfully retrieve the contents of a file from a modern hard disk drive that has been overwritten – even if only with a single pass?

I ask that forum members comment upon this challenge.

Thank you.

[Pleonasm]

As a matter of curiosity, I conducted a “live chat” with Ontrack Data Recovery (http://www.ontrackdatarecovery.com/), a leading and highly respected commercial service for the recovery of data from hard disk drives. They indicated that if a disk sector has been overwritten (even once), then they are not able to retrieve the prior contents of that sector. This information supports the thesis of the article “Can Intelligence Agencies Read Overwritten Data?” cited previously.

I encourage forum members to contact other competitive data recovery companies to see what they have to say on the issue, and post the insights in this thread.

[LockBox]

-{ Quote: "As a matter of curiosity, I conducted a “live chat” with Ontrack Data Recovery (http://www.ontrackdatarecovery.com/), a leading and highly respected commercial service for the recovery of data from hard disk drives. They indicated that if a disk sector has been overwritten (even once), then they are not able to retrieve the prior contents of that sector. This information supports the thesis of the article “Can Intelligence Agencies Read Overwritten Data?” cited previously.

I encourage forum members to contact other competitive data recovery companies to see what they have to say on the issue, and post the insights in this thread." }-

I think you are right. The only caveat being that certain intelligence agencies may (emphasis there on 'may') have highly classified methods of advanced data recovery. But this business of adding 35-wipes on erasing software is just ridiculous. One wipe and it's gone. Period. Unless you're wanted by the heaviest of agencies for purposes of national security. They wouldn't waste their time, money and information leakage on 'typical' crimes, no matter how heinous.

[ErikAlbert]

I zero my WD HDD with a tool provided by WD one time. After that my HDD looks like a new one, according WD. It takes 20m for a full zero (80gb), which I consider as very reasonable. I have also a very quick zero, which zeroes the first and last part of the HDD, probably 1m or so, don't remember.
The Gutmann's 35-pass overwrite is good for Nutmann or is it Nutmen ?

I don't use my zero tool for privacy reasons. I use it to get rid of all the malware like low level harddisk changes, corrupted partitions, Kid Of The Roots, ... ;)

[BlueZannetti]

-{ Quote: "I ask that forum members comment upon this challenge." }-There's really not much to comment upon - a legacy possibility existed in the past, but technology changed and the concern is paranoia run amok when discussing current HDD's. One overwrite is enough. There are plenty of real issues out there is one needs a topic to be concerned about.

Blue

[Pleonasm]

-{ Quote: "One wipe and it's gone. Period." }-
-{ Quote: "One overwrite is enough." }-
These statements are consistent with my own conversation with Ontrack Data Recovery, described in a prior post. However, it is nonetheless reasonable to ask: what sources can be cited as supporting evidence for the assertion?

Thank you.

[EASTER]

Well, if any of us could actually lay hold to an affordable micro magnetic-scope we might see other things but all things being equal we're all mostly constrained to use what's deemed reasonably effective in determining how well our HD Disk is been wiped.

Personally, i used to use D-Ban, and like EricAlbert sometimes i used the HD manufacturer's zero tool, but anymore i use White Canyon's Wipe Drive Pro with no more than a single pass of different methods, several to choose from algorithms.

Where before on a new install the yellow line on XP installer used to move across at seemingly timed intervals, after a wipe that same yellow line goes all the way to 100% on each XP new install i do now, and the full install time has really dropped off for me.

IMHO the Gutmann's 35 pass overwrite technique may be at least of some Psychological usefullness on individual files and folders although i use single pass pseudo and files/folders are changed dramatically.

The only real noticable difference i seem to gain on a daily basis is from using RESTORATION to Delete Completely as it says already deleted files, and i'm no expert so i can't explain in so much detail the WHY it works, only that it does enough that it's a basic part of my own daily routine to squeeze out every fraction of speed possible.

[BlueZannetti]

-{ Quote: "However, it is nonetheless reasonable to ask: what sources can be cited as supporting evidence for the assertion?" }-Sure: My own technical assessment of the current physical and logisitical capabilities of the various atomic scale imaging microscopies that would need to be used - I'm a techie in my day job and this specific technical area does fall under the scope of the R&D section that I head.
Recovering Unrecoverable Data: The Need for Drive-Independent Data Recovery (http://www.actionfront.com/whitepaper/Drive-Independent%20Data%20Recovery%20Ver14Alrs.pdf) - note : direct link to pdf file
Data Removal and Erasure from Hard Disk Drives (http://www.actionfront.com/ts_dataremoval.aspx)
These tend to be referenced in the various other discussions/summaries out there, so these are not new references. The thing is to read them, then understand the technology behind some of the approaches discussed to recover the overwritten data.

Blue

[RejZoR]

Performing regular 35 pass erasing on SSD devices is not really advised as it will wear memory extremelly fast. Regular HDD's are not affected as they have virtually unlimited reads/writes.

[LockBox]

-{ Quote: "Performing regular 35 pass erasing on SSD devices is not really advised as it will wear memory extremelly fast. Regular HDD's are not affected as they have virtually unlimited reads/writes." }-

Not anymore. For all practical purposes, SSD is also unlimited read/writes. The confusion came with two different standards in determining the life of these drives. The standard for traditional hard drives was based on MTBF (mean time between failures) while SSD was measured in number of read/write cycles. When the equivalent testing is done it shows superior lifetimes for SSD. Time and time again. In one DELL study, they found the following:

"Mean Time Between Failure (MTBF) calculations of 1.9M hours for SSD vs. 550K hours for standard HDD by Dell Labs using Telcordia methodology."

Google this and you'll find that we now know that the read/write cycles are even fewer over the life of the traditional magnetic hard drives versus SSD. The confusion came about by using two differing methodologies for determining the life cycle of these drives. Using the same standards - SSD blows away the magnetic drive in MTBF and read/write life cycles.

[caspian]

-{ Quote: " i use White Canyon's Wipe Drive Pro " }-

Is this product more effective than Eraser or R-Wipe? I looked at the website and the only difference that I see between the Wipedrive and Wipedrive Pro is the ability to use it on multiple computers. Do you think that there is any other benefit to using Pro? Or is Wipedrive just as good for a single user?

[EASTER]

-{ Quote: "Is this product more effective than Eraser or R-Wipe? I looked at the website and the only difference that I see between the Wipedrive and Wipedrive Pro is the ability to use it on multiple computers. Do you think that there is any other benefit to using Pro? Or is Wipedrive just as good for a single user?" }-

I really can't say in all honesty, only that i gave it a try and was satisfied with the results so much i went ahead and took the dive.

So far i haven't been disappointed, my installs are for me more swifter now after running it's wipe which is been my purpose for trying to find something, anything to fully wipe soundly enough to squeeze out some additional speed and White Canyon's disk seems to do what i been looking for.

With me it's not so much a matter of privacy as performance, and although that floppy disk eraser HDDErase also seems pretty potent, it locked me out of my drive once and i been a little leary of going that route again ever since, although who knows, maybe it does an even better job.

I'd like to read Blue's opinions of it as well as anyone else who is gone that route a few times with it if it's wipe is proportionally a bit better than some others or not.

EASTER

[Pleonasm]

BlueZannetti, thank you for providing links to references on the subject of data recovery (see post #18 ). I have had an opportunity to read these documents, and – indeed – they paint a picture in which overwriting data (even once) seems highly effective.

[Pleonasm]

Also of interest to readers of this thread...

-{ Quote: "An engineer at Maxtor, one of the world’s largest disk-drive vendors, recently told us that recovering overwritten data as something akin “to UFO experiences. I believe that it is probably possible… but it is not going to be something that is readily available to anyone outside the National Security Agency.”" }-Source: Remembrance of Data Passed: A Study of Disk Sanitization Practices (http://www.computer.org/portal/cms_docs_security/security/v1n1/garfinkel.pdf)

[Searching_ _ _]

Just a note:

Besides being one of the fastest wiping tools around, Secure Erase has a brother called Enhanced Secure Erase which is faster. The drive must support the feature.
But what does this mean? Where it may take an hour or so to wipe with Secure Erase, it will take, in some cases, "milliseconds" when using Enhanced Secure Erase.
So what drives out there support E.S.E.?
Hitachi for one.
http://www.hitachigst.com/hdd/support/bulk_faqs.htm
Q: I'm using a utility called HDDErase (version 3.2) (http://cmrr.ucsd.edu/people/Hughes/
SecureErase.shtml). Why doesn't your hard drive work with the enhanced security erase function?
A: The current version (v3.3) of HDDErase now works properly with the enhanced security erase feature. Please download and use this version.

Have fun

Fujitsu is another:
http://www.fujitsu.com/global/news/pr/archives/month/2008/20080421-01.html
Fujitsu to Launch World's First 320 GB 2.5" Hard Drive with AES 256-bit Encryption

This series is the first hard disk drive in the world to support the 256-bit Advanced Encryption Standard (AES)(3). The drive implements the AES hardware encryption directly into the processor chip of the hard disk drive, resulting in more robust security and faster system performance than software-based encryption.

All data stored on the hard disk drive can be erased instantly, in less than a second, using the advanced secure erase feature.

Built in encryption, wipe in less than a second. Yeah Boy! That's what I'm Talkin Bout!

[Pleonasm]

-{ Quote: "Built in encryption, wipe in less than a second" }-
Full Disk Encryption (FDE) is increasingly being adopted by hard disk drive manufacturers (e.g., Seagate and others). Since all data are always encrypted, there is no need to wipe the contents of the drive at all (assuming that a secure passphrase is utilized).

Traditionally, however, FDE hasn’t worked well in conjunction with image backup applications (e.g., Norton Ghost 14 by Symantec or ShadowProtect Desktop 3.2 by StorageCraft). Until that problem is solved in a realistic and practical way, then (for me at least) FDE isn’t a viable option.

[KookyMan]

TrueCrypt System Encryption seems to play mostly-nice with Acronis.

I have used Acronis to backup/restore a TC system encryption and it works just fine. The key to remember is you need to back up the cleartext files, not try to back up the encrypted data (sector by sector from outside the booted system).

Use Acronis password protection, which I believe uses AES to encrypt the image, or you can put the image on media otherwise secured. Why do a cleartext? 1) Backup location only needs to be the size of the data being imaged, not the size of the entire volume. 2) Encryption can't be compressed.

[Pleonasm]

-{ Quote: "I have used Acronis to backup/restore a TC system encryption and it works just fine." }-
KookyMan, that is very interesting. To clarify, are you saying that you created an image of a Windows system partition that is fully encrypted by TrueCrypt from within Windows – and successfully restored it using Acronis True Image?

My experience is that problems arise when you attempt to restore that image, since the pre-boot authentication is expecting the disk to be encrypted – but, the restore will deposit only the plaintext version of files on the volume.

Thank you.

[Kribe]

Interesting topic. I would love to read some technical information on this, as in-depth as possible would suit me well.

If memory serves me correct, the DoD used iLook (http://www.perlustro.com/) and dcfldd (http://dcfldd.sourceforge.net/) for a long while, and yes, they do claim that the tools can identify possible patterns of data wiped away from the disk magnetic surface although more complicated equipment is needed to actually recall it enough as readable -- e.g., through scanning of atomic and magnetic patterns and comparing the original magnetic strip to the now altered, (etc).

I am pretty sure there was a white paper on this but it has since been removed. Ibas (http://www.ibasuk.com/computer-forensics/procedures) based in UK published it around 2002. The link to the paper was part mentioned here: http://www.derkeiler.com/Newsgroups/alt.computer.security/2004-03/0354.html

EDIT - Found the paper: http://whitepapers.zdnet.co.uk/0,1000000651,260285576p,00.htm?r=7

Nearly all of the claims for that actually being possible rely solely on brute forced methods which do not have any empirical testing outside the scope of governmental institutes published, and a large chunk of them were referenced in Gutmann's original paper (http://wipe.sourceforge.net/secure_del.html). The problem with this call is, every discovery literally comes from these Top Class confidential scientific departments so chances are very high they can have something you are unaware of because they want you to remain unaware of it simply stated. I know the crime departments working in terrorist branches within the UK have access to some very advanced and little publically known techniques for data retrieval since I've come across them a few times but it really is way beyond my knowledge or position to understand or comment on it and I don't even know if the techniques work successfully, but only know they were being trialed at least in two branches.

However, that's just one side of the hip... I work within a sub-division of a government medical department and quite obviously data security is a profoundly major concern of ours especially since the last two years where corporational civilian, law and government hacking has now risen dramatically. To be totally honest, we [mainly] use these tools by Intelligent Computer Solutions for drive duplications, backing up, data sanitization and scrubbing: ImageMASSter 6007SAS (http://www.ics-iq.com/index.cfm/action/product.show/id_product/d122efeb-10e8-46ed-a645-82fe3be2b3ac/id_category/729f7ac2-9b88-46bd-9272-f6b86ec489f6?CFID=14987932&CFTOKEN=48972939) and IM WipeMasster (http://www.ics-iq.com/index.cfm/action/product.show/id_product/433e6b6c-3247-49ed-b6f9-16f356acc67a/id_category/8c53de87-f79f-419b-8f6a-9d889d5b97c4)

A rather out-dated short review of the IM WipeMasster is here: http://www.eweek.com/c/a/Storage/IM-WipeMasster-Gives-Data-the-Clean-Sweep/1/

We've used them for a long while now. Personally speaking, and for most of my colleagues... we do not believe you can retrieve any useful data once the above or any similar sector scrubbing technique has been implemented successfully. Hardly anyone in our industry does and that is exactly the reason why we use these methods daily, because it has assured us 100% data confidentiality, security and sanitization. Till this day, and with many attempts, there has not been a company of many acclaimed (http://www.google.com/Top/Computers/Hardware/Storage/Data_Recovery/) which we commissioned, who follow the Association of Chief Police Officers (http://www.acpo.police.uk/) and The National Hi Tech Crime Unit's (http://www.nhtcu.org/) guidelines for digital evidence recovery (http://www.acpo.police.uk/asp/policies/Data/gpg_computer_based_evidence_v3.pdf), that has been able to successfully retrieve data we wiped using the above method and we have issued internal testings many times to evaluate our security risks and area's of weakness, given our professional industrial nature. I do not believe it is possible to retrieve the data as claimed by some, no, and if there was, the above linked companies would very easily be out of business by a while now with security breaches happening daily (yes, attack to retrieve wiped data such as these are extremely common in corporations and within government departments as even the slightest data is highly valuable). I think it is purely a scare tactic, in other words, FUD by preying hard on our ignorance of the higher powers. I think firms pry on our ignorant paranoia to withdraw profit with services such as these (http://www.sitd.co.uk/data_removal_services.htm). That all said, I would love to see myself proven incorrect and enlightened.

Alina

[jrmhng]

-{ Quote: "Not anymore. For all practical purposes, SSD is also unlimited read/writes. The confusion came with two different standards in determining the life of these drives. The standard for traditional hard drives was based on MTBF (mean time between failures) while SSD was measured in number of read/write cycles. When the equivalent testing is done it shows superior lifetimes for SSD. Time and time again. In one DELL study, they found the following:

"Mean Time Between Failure (MTBF) calculations of 1.9M hours for SSD vs. 550K hours for standard HDD by Dell Labs using Telcordia methodology."

Google this and you'll find that we now know that the read/write cycles are even fewer over the life of the traditional magnetic hard drives versus SSD. The confusion came about by using two differing methodologies for determining the life cycle of these drives. Using the same standards - SSD blows away the magnetic drive in MTBF and read/write life cycles." }-

Is that even the issue with SSD drives? Do they still even need to be wiped?

[BlueZannetti]

-{ Quote: "Is that even the issue with SSD drives? Do they still even need to be wiped?" }-The same statements generally hold - file deletion is not really active removal of content, it's just a resetting of it's storage state until the space is reclaimed by some other/unrelated activity. If you want the content gone - a wipe of the contents is a prudent measure, although once is enough (IMHO, this is also true of current technology HDD's).

Blue

[jrmhng]

-{ Quote: "The same statements generally hold - file deletion is not really active removal of content, it's just a resetting of it's storage state until the space is reclaimed by some other/unrelated activity. If you want the content gone - a wipe of the contents is a prudent measure, although once is enough (IMHO, this is also true of current technology HDD's).

Blue" }-

They are totally different technologies. I can understand wiping it once as your operating system doesn't actually remove it. But once it is wiped it should be gone right?

[BlueZannetti]

-{ Quote: "But once it is wiped it should be gone right?" }-In an SSD the answer is an unambiguous yes since one can't appeal to mechanically based tracking variations as a source of residual ghost signatures that can, in theoretical principle, possibly be harvested. I was simply emphasizing that the same operationally holds for current generation classical HDD's as well.

The commotion on this whole discussion is really misplaced. If one is worried about information remaining on a drive for any reason, virtually any approach that overwrites that file space one or more times should yield the same result, and overwriting more than once is simply wasted effort.

What isn't always addressed by this step are all the locations on that HDD where information derived from that file, either in it's entirety or simply via the filename or metadata, may be transiently located and eliminated by a classical file deletion. Unless those locations are also actively wiped, information remains resident on the device and can be harvested. 35 passes or 4,206,309 passes isn't the real issue, one pass will do it. Unhandled residual copies of information (in free/slack/file system zones) is where the focus should be if you wish to have something to dwell upon.

Blue

[Pleonasm]

-{ Quote: "Unless those locations are also actively wiped, information remains resident on the device and can be harvested. 35 passes or 4,206,309 passes isn't the real issue." }-
BlueZannetti, thanks for adding some perspective to the discussion. I don’t think that anyone would argue that the completeness (i.e., comprehensiveness or thoroughness) of the scope of the erase operation is the more important facet. Stated different, ensuring that all privacy traces – wherever they may reside - are erased is more important than how they are erased (i.e., number of overwrite passes).

[PiCo]

Excuse me for asking, how long would US DoD 5220.22-M take on a 80gb hard disk?

-{ Quote: "US DoD 5220.22-M: The write head passes over each sector three times. The first time with zeros (0x00), second time with 0xFF and the third time with random characters. There is one final pass to verify random characters by reading." }-

I plan using it next month to bring my hard disk to a clean condition, as a new OS will be installed on it :)

[Pleonasm]

-{ Quote: "…how long would US DoD 5220.22-M take on a 80gb hard disk?" }-
PiCo, you don’t need to erase/wipe the hard disk drive in order to install a new operating system. Simply reformat the partition.

If, however, you still wish to erase the hard disk drive, the duration will depend on which software utility you are using and the speed of your PC. In my own case, my erase speed is about 3GB/minute (on a HP xw4600 Workstation with 15K RPM SAS drives), based upon a simple one-pass overwrite of free space with random bytes.

[PiCo]

-{ Quote: "PiCo, you don’t need to erase/wipe the hard disk drive in order to install a new operating system. Simply reformat the partition.

If, however, you still wish to erase the hard disk drive, the duration will depend on which software utility you are using and the speed of your PC. In my own case, my erase speed is about 3GB/minute (on a HP xw4600 Workstation with 15K RPM SAS drives), based upon a simple one-pass overwrite of free space with random bytes." }-
I have a messed-up MBR actually and the hard disk has a lot of corrupted data on it due to previous bad OS installs/uninstalls, so I thought what the heck?
I have the time I have the SoftWare (Active KillDisk), why not go on a full erase?

It's a 80gb SeaGate SATA drive, which I use to intall only OS. I think it will be a nice exerience :argh:

[Pleonasm]

-{ Quote: "… why not go on a full erase?" }-
I have read reports of hard disk drives experiencing a fatal thermal overload during a disk-intensive, multi-pass erase. Thus, it would be wise to first check that your disk is operating within its proper thermal parameters by using a utility such as SpeedFan (http://www.almico.com/speedfan.php). If your disk temperatures are running “high,” then improve the air flow in the case or reconsider the decision to run a full disk erase.

[EASTER]

I'm curious because software eraser is limited in scope AFAIK no matter the algorithms or passes for a full sanitized wipe of the metal platter, so whats anyone's thoughts on applying a mobile type of magnetic degaussing for home and office use if any exists, or even a garage project to fully eliminate any and all previously written data on a hard drive.

It's a simple matter to pull the lid off, and if you could degauss the platter safely, i wonder if thats something possible or not, or would that process simply destroy the usefulness of the drive ever again.

[PiCo]

-{ Quote: "I have read reports of hard disk drives experiencing a fatal thermal overload during a disk-intensive, multi-pass erase. Thus, it would be wise to first check that your disk is operating within its proper thermal parameters by using a utility such as SpeedFan (http://www.almico.com/speedfan.php). If your disk temperatures are running “high,” then improve the air flow in the case or reconsider the decision to run a full disk erase." }-Hmm I use PC Wizard 2008 and hard disk temperature never got above 30°C, cause I even have a fan cooling it.

DoD makes 3 passes so I guess it is pretty safe, but I wouldn't know about Gutmann’s 35-Pass Overwrite Technique. It would probably need 1-2 days to complete.

[jrmhng]

-{ Quote: "In an SSD the answer is an unambiguous yes since one can't appeal to mechanically based tracking variations as a source of residual ghost signatures that can, in theoretical principle, possibly be harvested. I was simply emphasizing that the same operationally holds for current generation classical HDD's as well.

The commotion on this whole discussion is really misplaced. If one is worried about information remaining on a drive for any reason, virtually any approach that overwrites that file space one or more times should yield the same result, and overwriting more than once is simply wasted effort.

What isn't always addressed by this step are all the locations on that HDD where information derived from that file, either in it's entirety or simply via the filename or metadata, may be transiently located and eliminated by a classical file deletion. Unless those locations are also actively wiped, information remains resident on the device and can be harvested. 35 passes or 4,206,309 passes isn't the real issue, one pass will do it. Unhandled residual copies of information (in free/slack/file system zones) is where the focus should be if you wish to have something to dwell upon.

Blue" }-

I understand. It seems to me that the best solution is really to use whole disk encryption. That way there won't be any redundant copies.