I want a test... (split from "Comodo vs AVC")

Discussion in 'other anti-virus software' started by Hungry Man, Nov 28, 2011.

Thread Status:
Not open for further replies.
  1. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    Re: Comodo vs AVC

    I want a test where someone is pulled off of a street and put on a windows computer. They have 100.exe's, 50 malicious and 50 safe - they have no idea which is which. They need to install the safe applications without infecting the computer based only on the product and their own wits.

    Let's see how well a HIPS does in that kind of test.
     
  2. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,833
    Re: Comodo vs AVC

    @ Hungry Man

    Well it sounds like you've just set yourself a challenge :D Go to Malware Domains etc & grab as many links with Malware as you can. Download & run them & see for yourself how your comp copes, or not :D Then post back, if it's still working that is :p & show us the results.

    Yes, seriously why not you ?

    A lot of us on here, including myself, have done so Many times over the years, with All sorts of nasties. Most of us are still here with no harm done. It'll be Very interesting to see how you do ;)
     
  3. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    Re: Comodo vs AVC

    Well... that's not exactly my test. The knowledge that each of those links is malicious will change how I treat them, of course. The idea is that the user believes there to be an equal chance that the program is malicious or suspicious. They want to install 50 legit apps but there are also 50 malicious apps mixed in. That's the point - they don't know which is which. Me choosing them from a list of 100% malware defeats the purpose =p

    And I'm currently running almost no security software (only EMET.) I wouldn't consider my setup very secure, certainly not against socially engineered malware.

    But I'd love to see how a default deny setup like your own works when you don't know which is which.
     
  4. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,833
    Re: Comodo vs AVC

    If i run anything new from some unknown or not so well known source, as often happens with for eg; ARK's etc & other Tools/Apps, or even known sources, i'll run them with ShadowDefender activated. That's for two reasons, 1st is just in case they "might" be dodgy, 2nd in case they are crap or don't work properly etc.

    But my HIPS = ProcessGuard & Zemana will alert me to various things they want to do, such as install a Kernel Hook, Driver, Reg startup entry etc etc.

    The thing you'll be most interested in, i presume is, If something needs a restart to install properly & therefore needs to be done Without SD, what then ?

    I'll upload the file to VT etc & see what they say. If showing clean it's then a matter of going with my intuition. Up till now i'm unscathed from the Hundreds + such decisions i've made over the years. Apart from one about 6 years ago on 98SE when i didn't upload it & i got some Adware installed as well :D No big deal to get rid of, & yes it could have been worse, but lesson learned :thumb: Even then i had an AntiExe etc.
     
  5. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    Re: Comodo vs AVC

    The idea is to up the ante a bit. 50% of the files we encounter aren't going to be malicious. The idea is to take a user and put them in a position where they have no clue what the origin of the files is but they know that half are 100% legit and half are 100% malicious.

    It might be more fun to say "Install them all without infection" and not tell them how many are legit so as to not sway with statistical analysis.

    I'd just be curious to see how certain setups would work in this situation. With an average user I'd say quite a few programs would flat out fail. With a wilders user who knows to use VT... maybe not.
     
  6. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,833
    Re: Comodo vs AVC

    Non Wilders etc peeps, i expect would be more than likely screwed. Especially with the nastier stuff that gets released these days :eek: You only have to look at cleanup www's to see how MANY people even with fully patched Vista.W7 get owned :D
     
  7. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    Re: Comodo vs AVC

    I would think so.
     
  8. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,468
    I would do it if i still had my VM AND if my PC was more powerful :D
    I stopped suing VM to play around with malware because i didn't have enough ram, although i added 2GB more like a month ago totaling 4GB as of now which would be somewhat enough. :D

    Also since now i'm working + studying i dont have enough time xD
     
  9. Baserk

    Baserk Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    1,317
    Location:
    AmstelodamUM
    Re: Comodo vs AVC

    With all due respect, I think the test would be meaningless.
    A bare HIPS isn't suitable for a 'pulled-of-the-street' guy/gal.
    How would average Joe know whether a 'dll abc requests access to xyz.whatever' message, means mayhem or not?
    The HIPS would score as good as the guy/gal gambles, unless it also offers cloud reputation/community rating etc.

    Heck, who wants to handle a bare HIPS install if you only have a neutrally named exe and no idea what kind of program it is?
    A HIPS isn't, imo, something to ascertain whether something is malware or not.
    If I'd get the message 'notepad.exe want to access the interwebs', I would have failed before.
     
  10. Hugger

    Hugger Registered Member

    Joined:
    Oct 27, 2007
    Posts:
    1,003
    Location:
    Hackensack, USA
    If you're going to go this far, then why not go the distance and get some poor soul to use a clean pc and ask them to install the programs, both clean and infected, without trying to change the settings of the different security suites being tested.
    It's true that many of the Wilder's members have more knowledge and skill when it comes to using security products. But we're preaching to the choir most of the time.
    Let's see how really effective the security products are out of the box.
    Hugger
     
  11. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    It wouldn't be a purely HIPS test. It would be a test that any product can go into and see how a regular user would use their product.

    I mean... if you have to be a security researcher or hwhatever for a setup to work it's probably not a great product, right? Or at least a very niche one...
     
  12. Sevens

    Sevens Guest

    Anybody with teenagers pulls this test daily.:D
     
  13. 1000db

    1000db Registered Member

    Joined:
    Jan 9, 2009
    Posts:
    718
    Location:
    Missouri
    In order for such a test to be useful you would need to consider the following:

    1. The subjects shouldn't know what they are testing as that could skew the results and prevent them from responding naturally.

    2. Other factors such as built in browser security should be considered.

    3. Testing only exe's will only simulate an install-type scenario, so the subjects would need to browse through a series of controlled sites with exploits, etc.(or at least browse through a pre-determined set of bookmarks or favorites that contain real malware).

    4. You would need several people testing each product in order to gather useful data.

    I'm sure there are other factors to consider also. This could be a beneficial test for vendors if done correctly.
     
  14. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    The hard part is choosing an "average" user and then having them go through hundreds of files with different products.

    I would only test executable files and not exploits since:
    1) Most exploits lead to a dropper file anyway
    2) It's easy to tell an exploit page from a regular website
    3) It's not so much about protecting against exploits as protecting against socially engineered malware - a situation where you don't know which files can be trusted.
     
  15. Ibrad

    Ibrad Registered Member

    Joined:
    Dec 8, 2009
    Posts:
    1,949
    In order for this test to work you would need to gather some avg. everyday people and have them browse the web through a specially designed web browser. This web browser would randomly have a "file would like to download and launch" alert which would be offering them either a good file or bad file. The files would be randomly named according to the site they are on. Such as if they are on Facebook a file Facebook.exe could be offered which could be a good program such as Skype or a bad program such as zbot. Then observe as the user answers the alerts and see what happens. A wide range of good programs would need to be offered to make sure not all of them are on the white list.
     
  16. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    Adding the extra step of browsing wouldn't change anything though.

    I mean if I'm going to go into details this is how I would do it:

    1) Gather a large group of people, maybe filter out the people who have no idea how to open a file or people who are lead programmers or something.

    2) Select 100 .exe's. Rename them to test 1, test 2, etc. Around half will be malicious, around half will be safe. Not necessarily a 50/50 split.

    3) Have the product already installed with default settings. Maybe have it be a week old in terms of definitions.

    4) Instruct the user that they can do anything to the software that they like except uninstall it (such as update it or change settings.) They can't install any other software except for what's in the test folder. The user can only run each .exe once.

    5) Record the average infection rate. Record average number of valid programs installed.

    A successful installation would mean that the program is able to open/ work properly. If it's installed but is broken it would not count.
     
  17. CJsDad

    CJsDad Registered Member

    Joined:
    Jan 22, 2006
    Posts:
    618
    :thumb:

    Couldn't agree more. Let a few teens have access to a computer and see how these tests hold up.
     
  18. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,833
    Good to see this thread spilt off ;) Quite why it's in here though ?

    :D

    However, i doubt they would give truly realistic results, as they are not representative of All the other regular users. I can imagine what the score might be already with RU's = Not good :p

    Making the OS etc a pain to use like Vista :thumbd: is one way, but even then it still gets blasted every day :eek: I don't see Any 100% solution to prevent RU's from getting infected, apart from user education. But MOST people i know, have known, hear/read about etc, can't be bothered etc to learn, sometimes even basics. So it's mainly their fault, & will continue to be.
     
  19. Zyrtec

    Zyrtec Registered Member

    Joined:
    Mar 4, 2008
    Posts:
    534
    Location:
    USA
    Re: Comodo vs AVC



    I strongly second that !
     
  20. qakbot

    qakbot Registered Member

    Joined:
    Aug 25, 2010
    Posts:
    380
    Re: Comodo vs AVC

    Kind of a pointless test since more major vendors that have even half a brain are constantly crawling or receive feeds from malwaredomains, etc. so you are likely to see everyone detect 100% or close to it.

    You want to really test malware, load up a P2P client like eMule etc., search for common search terms like serial, crack, xxx, video etc., download those exes and run them. Watch products like Trend and others that heavily depend on URLs, fall flat on their face.
     
  21. Boost

    Boost Registered Member

    Joined:
    Feb 2, 2007
    Posts:
    1,293
    Your idea,you do the test. I've played with malware for years,it's no fun anymore (boring). YOU wanna play the game,pony up and do it,otherwise hide behind the curtain.
     
  22. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    I don't have the motivation, time, or resources to do these tests.

    And maybe I'm just misreading but you seem a tidbit aggressive for no apparent reason.
     
  23. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,833
    You might think/hope so ! Unfortunately that isn't the case, at least not initially as they need to analyise them, & then provide DEF's for them. This can take days or even weeks :(

    And what's more, you might be surprised to know that there are Thousands + of older Malware that still go undetected by Lots of vendors, even after being out there for years :eek:

    I've tested hundreds of Malware from MDL etc etc, fresh & not so. Most AV's are "Late to the Plate" :thumbd:
     
  24. Boost

    Boost Registered Member

    Joined:
    Feb 2, 2007
    Posts:
    1,293
    Found some links that went undetected even after 8 years :D Yup,I lost faith in AV's a loooong time ago.
     
  25. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,148
    I have had the same experience. Many videos reviewing AM's will show similar results.
     
Loading...
Thread Status:
Not open for further replies.