ESS 5 fails !

[dorgane]

hi,

video :

part 1 (just software) : -http://www.youtube.com/watch?v=7XvJKu07ZxI
p2 : -http://www.youtube.com/watch?v=GDF_l6sq6VA
p3 : -http://www.youtube.com/watch?v=-Kikg0KPqh4
p4 : -http://www.youtube.com/watch?v=WLfK2lZDsKA
p5 : -http://www.youtube.com/watch?v=LCDPQDB2eLA
p6 end : -http://www.youtube.com/watch?v=ICwN5gqhGnU

i am not author of this videos.

but after see :

-> not detect 5xx of 6000 virus ( good but not perfect) in archive
-> cloud is very poor, file is unknow but not action of the cloud ( file is running )
-> HIPS lot lot lot notification and don't stop


idea :

cloud : green = allow ; yellow = ask (run, hips, firewall) ; red/unknow = block
in the cloud : can be terminate process and connection
in the cloud : auto-upload and notification unknow file send

---

hips : better stop notification
hips : block startup modifications in automatic


end ideas for the moment !


thank you for reply, ideas, comment...
sorry for my bad english :/

Arnaud

[Marcos]

1, no security solution detects and protects against 100% of malware
2, I saw corrupted Autoit application and some dubious Chinese soft in the video. How can one then be sure all those samples are valid for a test?
3, was the reasonably huge collection of samples saved to the disk with real-time protection enabled or was it at least scanned by the on-demand scanner prior to executing them?
4, I went through the videos quickly but didn't notice them testing an archive with 6000 samples. If it was there, couldn't it be that they used the very famous archive with 5917 samples from vx sites containing a lot of prehistoric DOS COM files, corrupted files and benign files? In that case, ESET detects everything that should be detected and the rest is just junk that is not subject to detection.

[dorgane]

hi

thank you for reply

but i don't talk about just detection...Marcos, i talk about improvement and bugs !

why cloud is not a cloud ? lol
i have always read on the web the cloud is real time analysis with server and communauty (okay i do esay explain my english is bad)
but in eset
, the cloud is more outdated !

when I see Norton ( Sonar), Kapersky ( KSN i believe), and eset ...i laught
in "my cloud" :

228037

sorry but i don't want bad with eset, i like lot software but when i see prevx in cloud in orange with 3 month...the cloud is not live analysis...

The real time cloud is very good for stop, how you say old virus, 0 day attack...if it is real time...sorry but eset cloud is nothing,If I run SysInspector it's the same as watching the cloud.


And i don't talk about HIPS allow unknow file to modify startup key :gack:

edit : don't be angry Marcos, i just want eset improve eset5 ;)

[vigen]

-{ Quote: "1, no security solution detects and protects against 100% of malware
2, I saw corrupted Autoit application and some dubious Chinese soft in the video. How can one then be sure all those samples are valid for a test?
3, was the reasonably huge collection of samples saved to the disk with real-time protection enabled or was it at least scanned by the on-demand scanner prior to executing them?
4, I went through the videos quickly but didn't notice them testing an archive with 6000 samples. If it was there, couldn't it be that they used the very famous archive with 5917 samples from vx sites containing a lot of prehistoric DOS COM files, corrupted files and benign files? In that case, ESET detects everything that should be detected and the rest is just junk that is not subject to detection." }-

im the video author..

1- its true..
2- yes all files is valid for the test and he was tried by hash (6950 samples)
3- Methodology: The folder of malware are present at the installation of the product, and i make a on-demand scanner on c:
4- Its not a malwares of vx site, this malwares was collecting on different url's and back-up for testing product and tried by hash. the malwares have different categories: virus, rogues, trojans, javascript etc etc...its not a "prehistoric files" ^^.
For example spysheriff contains the trojan Pskill and is was not detected by eset...eset don't detect everything same other product to..

Sorry for my bad english.

The vm was so infected, its not possible to make a sysinspector rapport, he bug at 64%... A alureon rootkit in a false xvid set-up infected the vm etc etc...

The very good evolution since the 4.2 version is a better detection of malware in memory, and better Rogues Blocking

[Marcos]

I noticed a sort of adware / riskware in the test and also that running an Autoit file resulted in an error message (ie. the script must have been corrupted).
In order for any test to be taken seriously, authors should adhere to the principles set by AMTSO (http://www.amtso.org/). One of the principles is that tested samples must be provided to the vendors for verification of the quality and objectivity of tests.

[vigen]

Hello Marcos, thanks for reply,

But i want testing the av product in "customers profile".

And my malwares folder, contains set of malware meet by the user daily.

Im not a "professionnal", its true, but i think eset don't make products only for professionnal.

If 30 malwares/500 is corrupted, 470 was not detected...the number is not imortant, its the danger family the important...

I remain at your disposal for further inquiry

[Marcos]

Given that these "tests" are not professional as you admitted, they should not be taken seriously as they are performed on a testbed consisting of corrupted samples and samples of dubious quality. The samples haven't been verified by the vendor which is required step before prestigious testers publish their test results If you wish, you can supply us with the missed samples for verification so that we can tell how many of them were actually supposed to be detected.

[toxinon12345]

Are you testing the beta or release candidate?

[vigen]

-{ Quote: "Are you testing the beta or release candidate?" }-


hello, I have testing the two versions... The Beta and the Release candidate.

The release candidate is more effective...

[vigen]

-{ Quote: "Given that these "tests" are not professional as you admitted, they should not be taken seriously as they are performed on a testbed consisting of corrupted samples and samples of dubious quality. The samples haven't been verified by the vendor which is required step before prestigious testers publish their test results If you wish, you can supply us with the missed samples for verification so that we can tell how many of them were actually supposed to be detected." }-

Most customers who use your products are not as professional.

The few samples which you think is a problem, can not justify such a gap detection.

Most files are verifiable on VT.

More like old programs are still not detected (spywarethis etc etc...) the collect date of this sample is 2010 (old programs)

if I take as an example antivir, it leaves 284 samples and Eset 550, been thought that this difference justifies the corrupted files

collection of these files and the fruit of my labor and time consuming, I do not provide a lab

Please see this link, the protection test,Eset was under industry average.. Particulary blocking malware on post execution.
http://www.av-test.org/reports/2011q1/avtest_report_eset_110941.pdf
Its more professionnal???

[toxinon12345]

Is easier for you to provide such samples to ESET

the fact another scanners detected the samples is insufficient, that only could show the reputation of a given file, in such cases can be False positives
testing is not an easy task, and could create confusion between customers, if the samples have poor quality

edit: prevalent samples are more important than zoo samples collected in VX sites or posted in sites as MDL

[vigen]

-{ Quote: "Is easier for you to provide such samples to ESET

the fact another scanners detected the samples is insufficient, that only could show the reputation of a given file, in such cases can be False positives
testing is not an easy task, and could create confusion between customers, if the samples have poor quality

edit: prevalent samples are more important than zoo samples collected in VX sites or posted in sites as MDL" }-

Is not easier:
1- i have a poor internet connection.
2- its a 2Gbytes malware folder ^^ Too long in upload.
3- I am not employed, im a customer, its not my job.

False positive is impossible in my malware folder...I spent many hours to verify that...

My samples are représentaifs that can meet the users on the Web.

If you want i make another video-test with samples of threatcenter:
http://threatcenter.crdf.fr/

I already try the result is the same...and you will see that the problem does not come from of the quality of my malware folder.

[toxinon12345]

Have you mentioned you tried with file hash?

[dmaasland]

-{ Quote: "Is not easier:
1- i have a poor internet connection.
2- its a 2Gbytes malware folder ^^ Too long in upload.
3- I am not employed, im a customer, its not my job.

False positive is impossible in my malware folder...I spent many hours to verify that...

My samples are représentaifs that can meet the users on the Web.

If you want i make another video-test with samples of threatcenter:
http://threatcenter.crdf.fr/

I already try the result is the same...and you will see that the problem does not come from of the quality of my malware folder." }-

The fact remains that you're just one guy, doing a test you designed, with malware you say is genuine. That makes it unreliable. Like Marcos said, if you're really serious about AV testing, take a look at AMTSO.

[patch]

-{ Quote: "The samples haven't been verified by the vendor which is required step before prestigious testers publish their test results If you wish, you can supply us with the missed samples for verification so that we can tell how many of them were actually supposed to be detected." }-
-{ Quote: "Is not easier:
1- i have a poor internet connection.
2- its a 2Gbytes malware folder ^^ Too long in upload.
" }-
Maybe you could burn a DVD and post it.
I am keen to see ESET improve their detection when ever possible.

[vigen]

-{ Quote: "The fact remains that you're just one guy, doing a test you designed, with malware you say is genuine. That makes it unreliable. Like Marcos said, if you're really serious about AV testing, take a look at AMTSO." }-
Its true im one guy, and one guy bypassed Eset with malware representative of threats present on the Web..

I don't need AMTSO...I need a browser and Threatcenter samples...as any users..

I do not see the interest of protecting against threats that the user has little chance to meet and pass the most common.

I think AVtest.org are not "one guy".

After everyone's opinion, my goal is to share with other users, not polemics

[trjam]

-{ Quote: "The fact remains that you're just one guy, doing a test you designed, with malware you say is genuine. That makes it unreliable. Like Marcos said, if you're really serious about AV testing, take a look at AMTSO." }-


The fact remains, he also could be exactly right with his findings. We sit here and ridicule because he isnt a professionnal tester, but when the professional testers post their results, we claim they are idiots. :gack:

In a situation like this there isnt a way to prove it for all of us, so in the end it really means nothing. Over time, if he is correct it will get validity from other testing sites, with Esets ability or lack of, to detect malware.

[dmaasland]

-{ Quote: "The fact remains, he also could be exactly right with his findings. " }-

I'm not saying he's wrong, I'm saying that there is no way to validate the results.

[Temp Member]

-{ Quote: "I'm not saying he's wrong, I'm saying that there is no way to validate the results." }-


But in 2 threads now you jump on someone for speaking out against ESET!

Everyone who pays for ESET is entitled to say their piece on it be it good or bad and ESS v5 has a lot of work to be as good as it can get.

In both threads you did not even present a good argument as the peeps were both in the right!

I personally think it is good someone who knows a bit more than the average Joe does some testing as it is the closest we can get to real user usage!

[trjam]

Arguing is pointless. This road never ends. Like I said he may be right, but I did some testing the other night and the RC was outstanding. Blocking all and stopping the one Panda let through that tried to wipe my hard drive.

So it just really doesnt matter, to each his own

[toxinon12345]

-{ Quote: "Its true im one guy, and one guy bypassed Eset with malware representative of threats present on the Web.." }-
i wonder who is that user wanting protection and they infect the system intentionally with samples posted in sites that 1% of users will visit

and not taking into account the samples are not randomly selected and some samples are corrupted



that does not make sense,

[trjam]

-{ Quote: "i and not taking into account the samples are not randomly selected and some samples are corrupted



that does not make sense," }-


Actually it does make sense and proves that this thread is pointless.

[Marcos]

As I always say - judge security products according to your personal experience. What's important is how well they protect you and not how many "samples" they detect in various serious or amateurish tests. If you get infected frequently you won't like the product no matter how high it ranks in tests.

[vigen]

-{ Quote: "The fact remains, he also could be exactly right with his findings. We sit here and ridicule because he isnt a professionnal tester, but when the professional testers post their results, we claim they are idiots. :gack:

In a situation like this there isnt a way to prove it for all of us, so in the end it really means nothing. Over time, if he is correct it will get validity from other testing sites, with Esets ability or lack of, to detect malware." }-


exceptional ... it's a bit much either?

[vigen]

-{ Quote: "i wonder who is that user wanting protection and they infect the system intentionally with samples posted in sites that 1% of users will visit

and not taking into account the samples are not randomly selected and some samples are corrupted



that does not make sense," }-

Of course but the problem is that some of these have malware was collected on sites so your argument of scarcity does not ..... but if I take the case of site keygenguru now correctly detected by Eset, there 'a few were not there. And even if it is clear that the behavior of the user is most important, we know that unfortunately this type of site is very popular, so now what you think you are free men.

[vigen]

-{ Quote: "As I always say - judge security products according to your personal experience. What's important is how well they protect you and not how many "samples" they detect in various serious or amateurish tests. If you get infected frequently you won't like the product no matter how high it ranks in tests." }-


I'm agree with that, I personally think that the time signatures and over and really starting time behavioral analysis.

[CogitoTesting]

-{ Quote: "The fact remains that you're just one guy, doing a test you designed, with malware you say is genuine. That makes it unreliable. Like Marcos said, if you're really serious about AV testing, take a look at AMTSO." }-

Really, what is in a word: "unreliable". I have seen his tests and I do understand French by the way and I can tell you that his tests are reliable and he has a good methodology. Marcos will always try to find all the excuses in a book. Youtube bring testing to the masses with regard to computer security. Moreover his tests results are in par with other testing organizations' results with regard to ESET.

Vigen you are doing an outstanding job and the only thing I can is that you keep doing what are you are doing. Job well done.

Merci pour tout ton effort et ne soit pas decourage par certains commentaires negatifs. A bientot. i.e Thanks for all your effort and do not be discouraged by some negative comments. See you soon.

Thanks. :thumb:

[ronjor]

-{ Quote: "Every security forum gets its share of posters who gob off about how "XYZ is the worst antivirus program in the world! PoopScan found 317 viruses it missed on my hard drive!"

A couple of things such posts almost always have in common are :

(1) the poster can never identify the allegedly missed viruses to other forum users because (a) it happened so long ago he forgot their names or (b) he didn't write down the names or (c) blah blah blah yakkety yak.

(2) the poster can never provide samples of those viruses for examination by the antivirus vendor he's been slagging off because (a) PoopScan deleted them or (b) his hard drive crashed or (c) he's not going to help XYZ improve their crappy program or (d) blah blah blah yakkety yak." }-http://www.eset.com.au/quick/snakeoil.html

[vigen]

-{ Quote: "Really, what is in a word: "unreliable". I have seen his tests and I do understand French by the way and I can tell you that his tests are reliable and he has a good methodology. Marcos will always try to find all the excuses in a book. Youtube bring testing to the masses with regard to computer security. Moreover his tests results are in par with other testing organizations' results with regard to ESET.

Vigen you are doing an outstanding job and the only thing I can is that you keep doing what are you are doing. Job well done.

Merci pour tout ton effort et ne soit pas decourage par certains commentaires negatifs. A bientot. i.e Thanks for all your effort and do not be discouraged by some negative comments. See you soon.

Thanks. :thumb:" }-

Merci pour ces encouragements :) thank you for the encouragement.

I try to be consistent.

[vigen]

-{ Quote: "http://www.eset.com.au/quick/snakeoil.html" }-

So in summary the best tests are performed by the editors of their own products??? ;D

[CogitoTesting]

-{ Quote: "i wonder who is that user wanting protection and they infect the system intentionally with samples posted in sites that 1% of users will visit
" }-

Well, well, well now it is official: 99% of users should go on the Internet naked without any protection whatsoever since it is only the other 1% of users that will visit those shady sites full with malware. Please security companies, whoever you are, do not advertise to me since I'm not part of that 1%. Moreover, I will only visit good looking, professional, and ethical websites, nothing shady. Furthermore, I know those websites will never be hijacked and they do not have any drive by download etc...

Man, you make life so simpler for me right now.

Thanks a million...;D ;D ;D ;D ;D ;D.

[Thankful]

Eset has always been reasonable regarding missed samples. They are always willing to take a look at the samples and add them if they are malware.
Eset has to balance these submitted samples against currently spreading malware.

[ellison64]

-{ Quote: "http://www.eset.com.au/quick/snakeoil.html" }-

Interesting link.It might have been written by a lawyer as it more or less covers every eventuality of someone/organization finding fault with eset ,which is then dismissed by one of those eset pearls of wisdom :dry:
ellison

[Cutting_Edgetech]

-{ Quote: "
Please see this link, the protection test,Eset was under industry average.. Particulary blocking malware on post execution.
http://www.av-test.org/reports/2011q1/avtest_report_eset_110941.pdf
Its more professionnal???" }-
Vigen, I have felt that this has been a weak area of Eset for a long time now. Sometimes its just too late if the malware has been allowed to execute to stop it from infecting your system even if it was detected after execution. Eset should really focus on improving in this area since I have noticed more than 1 of these test over the past year where they fell under the industry standard. Just dismissing your test is not going to make them improve the protection offered by their products. If you don't want to to submit your sample to Eset then you may be able to submit them to a third party for verification. If it were me then I would submit them to Eset, and a third party for verification if possible.

[ellison64]

-{ Quote: "Vigen, I have felt that this has been a weak area of Eset for a long time now. Sometimes its just too late if the malware has been allowed to execute to stop it from infecting your system even if it was detected after execution. Eset should really focus on improving in this area since I have noticed more than 1 of these test over the past year where they fell under the industry standard. Just dismissing your test is not going to make them improve their the protection offered by their products. If you don't want to to submit your sample to Eset then you may be able to submit them to a third party for verification that way. If it were me then I would submit them to Eset, and a third party for verification." }-

:thumb:
Nice to see a common sense and balanced response

[Cutting_Edgetech]

Thanks! Its always good to be fair, and balanced.

[Cudni]

The real service to users would be to allow an AV to detect malware that is claimed it does not. When there is will there is a way.

[Cutting_Edgetech]

I should also make clear NOD 32 is the only AV i'm currently using, and i always give none bias opinions. I also look at the facts when drawling any conclusions about anything in life. So i don't want to sound sided against Eset. Samples should be provided upon request if someone is going to publish their work. A request such as wanting to verify samples is totally reasonable. I can not make any comments about the testing methodology, and guidelines set by AMTSO since I have not looked into them yet. One thing to keep in mind is an innovative tester may not agree with the methodology or guidelines used by AMTSO, and if we all adhered to some set guidelines then we could find ourselves being nothing more than drones, and technology & science would be greatly limited in it's growth. Sometimes you have to think outside the box.

[toxinon12345]

-{ Quote: "99% of users should go on the Internet naked without any protection" }-
sorry, that is not what i mean
the important here is how your product protect you under real world conditions

i can assure you there is no other scanner that can guarant to you reliable detection when it comes to "in the wild" threats detection, which are real [threats with higher priority] than other threats because they are widespread

[dorgane]

i have write at the beta form about the cloud :


-{ Quote: "
Dear Customer,

we are continuously working on improving our product. So, we can say that RTM and final version should be both better than previous versions during the beta stage.
" }-

[toxinon12345]

-{ Quote: "i have write at the beta form about the cloud :" }-
URLs posted in sites as "Threatcenter", "malware domain list" and similars must be blocked

and the signatures can be added later to VSD, only when the prevalence is increasing or as soon as they can be added, not necessarily the date were discovered

[GrammatonCleric]

Everyone is arguing about his test methology and samples but no one addresses the 2nd issue and that is the absolesence of the cloud detection. The cloud needs massive improvement in order to compete with the likes of Sonar or Kav.

I am not being harsh I am just saying that if ESET 5 is going to introduce CLOUD detection then at least introduce a good one..

[trjam]

Eset is at a turning point right now. To which way it turns is about to be played out. I said when the beta came out, I was surprised this is all they came forth with. It seemed like a half-hearted effort, for which some have argued for years was their standard operating process.

Eset has also reaped great rewards for certain aspects of their product that others could not match. So, it is like, do we go left, or do we go right. They have never clearly defined as to which direction they intend to go.

As for the beta, the HIPS module blew me away. We have to look at things from a user stand point, not through our eyes, but that of the 99.8 percent of normal non-educated users. Not to deflame them but the reality is, the HIPS module will never work on a globally sold product. It is to confusing and in its current state, very frustrating. What were the internal beta testers thinking when going through this process. Now, it isnt finished and they know more then I, so maybe, just maybe, they have a final outcome for this module that will suffice for average users.

Back to going to the left or right. Product are designed for the masses. Most are continuing to try and become user friendly and easy to use, but very protective at the same time. I think, this is why Avast made the statement that its sandbox in the future may be set to be on by default. All it takes is one, just one infection for normal users to bail. They dont understand, this is just a reality.

You either have to find ways to offer stellar detection or you fold. Behavior blockers are nice but really take time to develope. I renewed my license till 2014 for Eset, I like them and their product. But they have better deliver before this RC is officially released, or it will be the version that is remembered in bad terms, like all security products have seen. I hope the best for them, but some better change and quick, or my license till 2014 may not even end up vailid till then.

[GrammatonCleric]

-{ Quote: "
Eset has also reaped great rewards for certain aspects of their product that others could not match. So, it is like, do we go left, or do we go right. They have never clearly defined as to which direction they intend to go.
" }-


Who knows if those rewards and "aspects" (mainly heuristics) will continue to be "great" considering that their main "hueuristic" expert jumped ship some time ago.

But alas, I have hope.

[agoretsky]

Hello,

The "main 'heuristic' expert" is still at ESET and was actually appointed CEO (http://www.eset.eu/press-2011/richard-marko-ceo) earlier this year.

Regards,

Aryeh Goretsky


-{ Quote: "Who knows if those rewards and "aspects" (mainly heuristics) will continue to be "great" considering that their main "hueuristic" expert jumped ship some time ago.

But alas, I have hope." }-

[trjam]

err, thats part of the problem Aryeh. The experts keep getting promoted and that leaves the cupboard bare for a period.:)

you folks will be fine, just get that HIPS module ready for everyday users and version 5 will be a total sucess.

[GrammatonCleric]

-{ Quote: "Hello,

The "main 'heuristic' expert" is still at ESET and was actually appointed CEO (http://www.eset.eu/press-2011/richard-marko-ceo) earlier this year.

Regards,

Aryeh Goretsky" }-


Ahh the great Power Point Reviewer position. :)

I DEMAND MORE POWER POINT ~ Snipped as per TOS (https://www.wilderssecurity.com/faq.php?faq=wilders_tos#faq_wilders_tos_1) ~!

[toxinon12345]

-{ Quote: "Hello,

The "main 'heuristic' expert" is still at ESET and was actually appointed CEO (http://www.eset.eu/press-2011/richard-marko-ceo) earlier this year.

Regards,

Aryeh Goretsky" }-
and maybe that heuristic pioneer is hearing their users very closer ;)