AVLab: Advanced In-The-Wild Malware Test in November 2023

AVLab: Advanced In-The-Wild Malware Test in November 2023

21 December 2023

https://avlab.pl/en/windows-10-pro-under-attack-advanced-in-the-wild-malware-test-in-november-2023/

 

Thanks for the "heads up". I thought I tested it, but I guess I didn't. I've corrected the link in my initial post.

 

Once again a testing lab shows us why these labs and their tests are inept, unreliable, and to be taken with a ton of salt.

Where's the world's dominate anti-malware solution provider, Norton?
Where's McAfee - the 2nd most widely used solution?
BitDefender?
TrendMicro?

They don't even include Microsoft Defender!

Yet they claim this simulates the real world? Not even.

These tests don't help consumers. They mislead them.

 

I fully agree.

Advanced In-The-Wild Malware Test

 

The link in the first post works fine for me.

Everybody got 100% "combined protection"--no fun. So let's rag on Comodo aka with its long remediation time--wow I happen to dislike antivirus software in general and malicious programs are so sophisticated nowadays, if anything bizarrely slipped past my excellent defenses I would consider my system shot and reach for my Hasleo backup or USB with my creation tool on it. No piddling around trying to clean that.

But these lab "studies" are somewhat entertaining at least--I guess for those who cheer on their AV of choice.

 

Yeah, he said in post #3 he fixed it.

LOL - actually, that is why I mentioned Microsoft Defender - not because it is my AV of choice, but the report leaving it out meant all those MS haters out there couldn't rag on it!

 

"Tested solutions to install at home and in a small office:
...
Hidden name 1 - private test 1
Hidden name 2 - private test 2
..."

Could it be?

As with most tests I believe, the vendors have the right to choose to participate or not. As well as the choice to keep the results private if they do participate.

 

I agree, I actually like these smaller tests, because it gives a good overview of how good an AV is in protecting against newer malware samples, perhaps even zero days. So I don't see how these smaller tests mislead anyone. But anyway, from what I understood they all passed the test, but some took a bit longer to neutralize them? Actually, I just read that Emsisoft Business Security failed to block one sample.

 

At first, I was like: where? But then if you go to the link, the results are somewhat less uniform. Nice to point it out, Rasheed. Why weren't the two pages consolidated into one general outcome, I wonder? But maybe the answer is somewhere in the text.

https://avlab.pl/en/recent-results/

 

I saw those and wondered too. But I listed 5 not named in the test and those were only what I thought of off the top of my head. Sophos is another and it has a bigger market share than Avast, which was listed, for example.

So what good is a private test if the results are kept private? In fact, why even announce that there are private tests? Reminds me of a little bratty kid, bragging to their friends they he knows something they don't.

I don't believe this is true. I am aware that some developers tried to stop being included, but I believe they lost that battle in the courts. If a product is publicly available, it can be included, was the ruling. They do have the right to dispute findings. And if they opt-in (typically for a fee, BTW), they can see ALL the data and testing criteria that otherwise might be kept private.

It is the same with other products. Toyota cannot refuse to have their Camry tested and compared with the Honda Accord, for example. Nor can AMD refuse to allow its processors from being compared to Intel's. Same with Sony TVs and Samsung TVs, Apple phones and Android phones. Chrome and Firefox. Why should this be any different? The courts agreed.

That may be fine if the program being tested is one you are interested in. What if it isn't? What if you use Norton? Defender? Or BitDefender? Or another solution not in the test like Avira?

And truth be told, it is actually very hard to infect a properly maintained W10/W11 computer these days. It typically requires the user to open the door and invite the bad guy in by the user clicking on an unsolicited link while running with outdated security and Windows that is not up to date!

We need to be realistic. Most users sit behind a router - a very robust layer of protection. All Windows computers have a decent firewall enabled by default - another very robust layer of protection. All Windows computers have a decent anti-malware solution enabled by default, or they have a decent alternative installed. All W10/W11 computers stay current by default as do the security software running on those systems. All the popular browsers have built in security feature. Most email providers have spam blockers enabled by default.

Seriously, when was the last time your system was compromised? When was the last time your computer or security even yelled at you to say it just blocked a real threat? The one and only time one of the systems I was responsible for was infected was close to 30 years ago and that was via the sneakernet! A co-worker brought in an infected floppy and left it in one of our systems overnight. The next morning I failed to check the floppy drive and booted up the computer.

As of late, I have had my browser (Edge) block some http sites because they did not use a secure socket layer (SSL/https). And I have had Malwarebytes find a couple "wanted" and safe PUPs (potentially unwanted programs). That's it. And other than being pretty disciplined at avoiding being "click-happy" on unsolicited links, and I avoid illegal porn and gambling sites, I do nothing special to stay secure beyond keeping my OS and security current - and that is all done by default! And this behavior is pretty much how most the world behaves.

There's a very good reason bad guys in recent years have been focusing on corporate/organizational networks these days - it's easier pickings and more lucrative. But even then, most of those breaches happened because a user failed victim to a socially engineered threat, tricking them into clicking on an unsolicited link that then infected their network - a network the IT folks and senior managment failed to properly secure!!!!

 

The vendors pay to participate. It's their prerogative to remain hidden or not. Therefore, this to me is not a true scientific test as you're supposed to include all the subjects in outcome reporting.

If I cared about antivirus, I would object to that. But despite "real world" in titles, these are just snapshots in a continuum, probably designed to indirectly sell security products rather than providing scientific outcomes.

 

As far as the criticism that AVLab doesn't test other AV vendors other than those shown in the most recent test, it does do so but only on a periodic basis.

Windows Defender - https://avlab.pl/en/results-may-2023/
Sophos and TrendMicro - https://avlab.pl/en/results-september-2023/
Norton - https://avlab.pl/en/results-july-2023/

 

"Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level."
So, Sophos RT 10 seconds, Trend Micro RT 434 seconds. Quite a big gap. G-Data RT: now available, why?

 

Hi, thank you for the question. Why? Because all threats have been detected on the browser level protection by G Data extension in Firefox, so there was nothing to get to calculate Remediation Time. This is the best situation and in theory the RT is equal 0s. Maybe we have to change this uncelar describe and situation when threats are blocked at Pre-Lanuch level. Thank you.

 

What does this have to do with if a certain test is misleading or not? This test simply tried to infect systems in a way real life users would possibly come into contact with malware, depending on their computing practices of course. And in this particular test, most AV's managed to protect the system, nothing more, nothing less.

I have never been infected in the last 20 years AFAIK. But I also know that while AV's are pretty good in detecting malware, they will probably never be bulletproof, so that's why I also extra security tools, I do think it did save me a couple of times when I saw shady app behavior, which was blocked by my behavior blockers.

All true, but it's naive to think that home users don't get infected and AV's can block 100% of all malware. I have seen plenty of tests where AV's failed to block some samples, so if it's your unlucky day and get tricked into downloading these .exe files your system might still be toast. That's why companies spend billions on so called EDRs, which monitor the network in case AV fails to spot (advanced) malware, which are most of the time zero day samples.

 

I think there is a disconnect here.

First, if the sample malware the lab tested was against your anti-malware solution then great! That may be useful information - for you.

But it provides no information, useful or otherwise, for the 100s of millions of users of the other, even more popular, but untested antimalware solutions. And in this case, because it doesn't even test with the most popular anti-malware solutions, IMO, it is misleading.

Second, these tests are NOT scenarios real-life users encounter, regardless how much these labs claim they are. No "normal" "real-life" user, with his or her unique computer, sits around and intentionally exposes their computers to 10s of 1000s of malware that may or may not currently be circulating out "in the wild".

This is actually why Windows Defender, the anti-malware version with W10/11, not the old anti-spyware version for W7, initially scored poorly on many of these testing lab tests - despite the fact, Defender users were NOT getting infected. Microsoft intentionally coded (and updated) Defender based on the current threats currently circulating in the wild. It scored poorly on old threats, designed, for example, for XP even though those threats were no longer in circulation. However, the commercial (free and paid) did protect against those obsolete threats so scored better.

Let's not forget the companies that produce the commercial (free and paid) solutions depend on malware for their very existence. And they use those test results as marketing fodder. But if malware went away, they would go out of business! So one has to wonder, what incentive do they have to really stop malware at its source? They don't.

Microsoft, on the other hand, gets no revenue from Defender - which they include free in W10/W11. They do, however, have a very good incentive for malware to go away - they will stop getting falsely accused for the poor security situation we are in now.

Just something to think about.

I am NOT suggesting these labs are intentionally doing it wrong. I am simply noting that a "simulated" test in a laboratory setting is just that - a simulation in a laboratory and not a "real-world" scenario.

It is similar to pharmaceutical companies doing drug testing. They can spend $millions, or even $billions running tests and simulations on their new vaccines or drugs. But until they actually test the medicines on very large samples of real "unique" people, living in their own "unique" home/work/school environments, doing their own "unique" day-to-day" routines can they really see if the drug is (1) safe and (2) effective. And even then, only time will really tell if, some time down the road, these people start growing a 3rd arm out their back.

 

I fail to see why a test is misleading because it didn't test ALL major AV's. What would be misleading is if this test claimed that AV's are bullet proof because they stopped all 245 samples. Because in plenty of other tests I have seen Win Defender and Malwarebytes fail to block ALL samples. Does this mean that they are crap? No of course not, that's not what I'm saying.

I have to disagree. How do you know people weren't getting infected worldwide? Also, the reason why Win Defender was crap on Win 7 and 8, is because MS didn't give a damn about home users, but once they saw that there was money to be made in the IT security market, they started to take things more seriously and they beefed up security.

I understand what you're saying, but that's just the way testing work. And I disagree with that it's not real world, how do you think AV's for enterprises are tested? In the exact same way, they simulate how hackers would normally try to infect companies. You can obviously not predict whether users (corporate or home) will be tricked into running some malicious file in real life. You should take a look at SE Labs which simulates ransomware attacks on companies in one of their reports: https://selabs.uk

 

And for some decisions they just deserve it. E.g how can it be that I still have for every windows install activate "show filename extensions" by hand for the explorer? It can't be too hard to change that for them to make it the default at every install.

 

BTW, what sample was missed by Emsisoft? Was it ransomware or some infostealer? It would be nice if this type of information was disclosed.

 

Huh? Come on, Rasheed! Are you seriously going to tell us you don't see the irony of that question? How do you know users of Norton, McAfee or your favorite software have not been infected world wide? You say you have not been infected in 20 years, how do you know? You added, "AFAIK" and that is the exact same with everyone. Like you, they probably saw no evidence of infection. They scanned with secondary, alternative solutions and those scans came out clean - just like I assume you did.

Right, in the exact same "simulated" way. Why are you trying to suggest anyone is saying AVs for enterprises would be different? Simulated tests in labs are just that; simulated lab tests.

Okay. I've tried to explain in more ways than one - you still don't get it so I give up. Have a good day.

True. MS has been guilty of shooting themselves in the foot on many occasions. No denying that. And I'm all for bashing MS, when due.

But Microsoft is really tired of being blamed for much of the security mess we are in when it clearly was the actions of the bad guys who put us here, and the failure of Norton, McAfee, CA, TrendMicro and the other security providers who failed to stop those bad guys!

It is important to remember that Microsoft wanted to put AV code in XP but the reason they didn't was because Norton, McAfee, and the others whined and cried to Congress and the EU claiming Microsoft was trying to rule and monopolize the world. They were! But not the point.

The anti-malware industry claimed it was their job to thwart malware. So Congress and the EU threatened to breakup Microsoft Ma Bell style if they included anti-malware code in XP. So MS left it out.

But Norton, McAfee and the others failed miserably! As I noted, they had (and have) no financial incentive to defeat the bad guys. If they succeed, they go out of business.

But who got blamed for their failure the next 10, 15 and more years - and even up to today? Microsoft - even though it was the badguys who perpetrated the offenses and the anti-malware industry who failed to stop, or even hinder their proliferation.

Notice how now Congress, the EU and Norton have kept their mouths shut about Microsoft integrating anti-malware in W10 and W11? It is because they know they were wrong. Had MS been allowed to include anti-virus code in XP, it could have at least slowed the unhindered advances of the bad guys.

Even most of the biased IT media and MS bashers have, for the most part, kept quiet about MS including anti-malware code in W10/W11.

I am all for blaming Microsoft - when due. But not for things they have no control over. The bad guys put us here, not Microsoft.

But that is all water under the bridge now. We are here and that's that.

 

Since Defender politely steps out of the way and waits in the wings until needed whenever a 3rd-party security solution is installed, the concern that MS was going to monopolize the field and force all other players out of the arena has also receded into the background.

It's a different scenario than if MS forcibly activated Defender or equivalent on all W10/W11 installations, and blocked or disabled 3rd party solutions.

 

You misunderstand, what I meant is do you have any statistics to prove that MS Defender users didn't get infected? Or is it just your gut feeling? I personally don't have a clue how many people get their PC's infected every year, I'm sure it's not as bad as in the past, but I wouldn't be surprised it's still thousands of people, which is bad enough. It's widely known that AV's will sometimes fail to protect against zero day samples. And how come so many companies get hacked? Funny that they never disclose which security products they were using, they are probably not even allowed to.

Did you check out SE labs? Do you also think this isn't real world enough? Of course, we're now talking purely about the testing method, not about the fact they used only a couple of popular ransomware families and that it's probably a sponsored test, so of course CrowdStrike was not going to fail.

https://selabs.uk/reports/enterprise-advanced-security-ransomware-crowdstrike-falcon-2023/

I think they gave up because they came to the conclusion that they would simply lose the lawsuit. I mean macOS also integrates built-in security. And they did nothing when MS integrated Internet Explorer into Windows and killed Netscape in the process.

 

This is an important fact some don't understand. This happens because (1) most users just don't need for security reasons to run 2 anti-malware solutions at the same time. (2) Running two security solutions at the same time needlessly uses valuable resources and (3) in some cases, conflicts can occur (like two dogs guarding the same bone).

I will add this for Malwarebytes Premium (not the free version) users. Malwarebytes Premium and Microsoft Defender play very well together, with no conflicts while still being very light on resource utilization. So if you want to run both Malwarebytes Premium and Microsoft Defender at the same time, you just have to tell Malwarebytes Premium "not" to register itself in Windows Security Center. Fortunately, this is simple.

Open the Malwarebytes control panel. The click on the settings "gear" icon in the upper right. Then click on the Security tab and scroll down to Windows Security Center. Then slide the slider switch to "Off" (to the left). Exit Malwarebytes and reboot. Then see if all is working as you expect.

I did not misunderstand. You are not following. Now you are asking me to prove a negative - to prove that unicorns don't exist! That makes no sense.

I have the same proof you have that users of Norton, McAfee and your own solution didn't get infected!

What I do know is "IF" Defender (or Norton, McAfee or your solution) was allowing users to get infected at an "abnormal" rate, that fact would be highly publicized all over the Internet. This would be ESPECIALLY TRUE if Microsoft Defender was letting users get infected at an abnormal rate simply because there are so many Microsoft haters out there who love to bash Microsoft! So where are the reports? Not there. Why? Because, by far, the vast majority of users are NOT getting infected.

Of course there are still thousands of people. But there are over 1.6 billion (that is 1,600 million) Windows computers out there. Thousands is but a drop in the bucket. A couple million is just a couple drops.

HOWEVER - what we do know is that the vast majority of infections are NOT due to security software failures. No! The vast majority of infections are caused by "socially engineered" methods of malware distribution. That is, the bad guys trick users into opening the door and letting (inviting!) the bad guys in. And then those bad guys are exploiting known vulnerabilities that have patches available, but the user (or IT managers) failed to install!!!!

Users need to stop being "click happy" on unsolicited links!

You can have 3 deadbolts on your door. But if the bad guy is wearing a nice clean uniform, pretending to be from the gas company, and you open the door and let him in, you are going to get robbed!

No security software can protect us 100% of the time if we invite the bad guy to go around our security.

Zero day exploits are real, but very rare. And even more rare for the bad guys to discover before the "white hats" do.

The user is, always has been, and will continue to be the weakest link in security. That is why so many companies get hacked - that and negligent IT and C-Level executives who fail to do their jobs. (read up on the cause of the Equifax hack).

Equifax recap: It managers failed to apply patch they had for 6 months. Employee user clicked on unsolicited link in unsolicited email, let the bad guys in and 143 million users very sensitive information was compromised. And NO ONE was held accountable.

They didn't give up. They didn't even try. They tucked their tail between their legs and hid because they knew they failed to protect us, like they cried and whined to Congress and the EU that it was their job to do, not Microsoft's. And now, all this time later, it is too late. No way to stop the bad guys now.

Not true - at least not in Europe. The EU forced Microsoft to provide a version of XP without IE. In the US, Microsoft was forced to allow users to set a different browser as their default.

FTR, I was a diehard Netscape fan. The only reason I switched to IE5/6 was my company forced us to switch, or risk termination. But once I got used to IE, I realized it was a better browser. Netscape failed to keep up. That is why it went away.

 

LOL, so if you don't read about it in the news, it didn't happen, makes sense to me. I never said that millions of PC were getting infected in the past. But not per se because AV's were that good (or bad) in protecting people, but perhaps lots of people never encountered malware in the first place? Of those 20 years that I AFAIK didn't get infected, I didn't even use an AV for 12 years! I relied strictly on behavior blockers and VirusTotal, and tried to download software from trusted sources.

Why do you keep repeating this, did you see anyone claim something else? My point is that we both agree that you can take certain AV tests with a grain of salt, but not because they don't always include ALL major AV's players, but because they often don't test AV's against true zero days that may be used in (targeted) attacks. I believe this test from AVLab was trying to achieve this.

Yes correct, what else is new. I suggest you take a look at these threads. I think it's unlikely that hackers will go to so much trouble if they can't bypass built-in AV's like MS Defender and XProtect (macOS). One of them is about Google Search being abused to trick people into downloading legitimate software that are bundled with malware downloaders. The other one is about how a blockchain developer was tricked into downloading an infostealer on LinkedIn, which stole money from his crypto account. And this is just one story, there may be thousands of other that don't make it to the news.

https://www.wilderssecurity.com/thr...s-to-spread-malware-in-legit-software.449530/
https://www.wilderssecurity.com/thr...rypto-wallet-emptied-in-job-interview.453263/

What do you mean they failed to protect us? I thought that based on your gut feeling (and lack of news stories) people were barely getting infected in the first place, no matter what type of AV's they used? Or did they stop getting infected since Windows Defender was improved on Win 10?

Yes correct, IE was better at some point. But we all know that if tools are integrated in Windows there is a big chance users will use it, because of the convenience factor. That's why they have now sued MS again regarding its bundling practice of MS Teams, see link.

https://www.theverge.com/2023/7/27/...antitrust-investigation-office-bundling-slack