Following on from the Matousec discussion in the OA64bit thread, I thought I'd dig a bit deeper into the testing approach to take a look at why Matousec is such bad science. In Nov 2010 Matousec tried to justify the tests as relevant to real malware by analysing 20 malware samples. From these 20 it was found that 30 of the 148 test methods were used by real malware. Despite any evidence that the other 118 tests were relevant Matousec concluded "We are sure that if larger set of malware was used we would see even more techniques that are implemented in our tests". Bad bad science. No evidence at all. Just "we are sure". Yeah right. Anyway, take these 30 methods used by real malware and take a look at how one of the "Not recommended" apps performed. Let's pick Trend Micro, which scored a pitiful 9%: • Well, it passed Yalta and Leaktest which is used by 50% of the samples. That's good. • And it passed Autorun 3, the second most common method used by the malware • And it passed Autorun 1, the next most common method • Next up is Hostsblock which Trend passes also • Then Kernel 1, Kernel2, and Svckill but Matousec doesn't allow Trend to be tested against these because it doesn't pass his arbitrary "levels" testing approach. • Next is Jumper which Trend passes • Runner 1 & 2 next, along with Autorun7, Inject2, DNSTester, and a bunch of autoruns but again Matousec doesn't allow them to be tested. So far 100% for Trend. • Back to tests Trend is allowed to be tested against and Wallbreaker4 and AWFT1 are passed, but then one sample of malware uses the DNSTest technique and Trend has its first fail. • Every other test is not allowed to be tested against Trend. So by my reckoning we have 1 test failed out of the methods being used by malware in Matousecs own sample, yet Trend gets awarded 9% by Matousec and a "not recommended" status. And since 50% of the malware used Yalta or Leaktest, there's a good chance that Trend would have alerted to that piece of malware anyway. So I'm going to take artistic license and award it 100% against real malware rather than the 89% it got against the tests. Well done Trend!