ESET Scanning of AVI's, MPEGS and VOB files

When scanning over 300 GB of AVI's, MPEGS and DVD VOB files, ESET Scans these files in less than 5 seconds. This issue has been reproduced on 2 other computers as well. Does anyone know if ESET is actually scanning these files? When scanning a folder of about 30GB of PHOTOS's on the same external HDD as the movies, the scan did run about 10minutes or so. The scan log does show how many files it has scanned. I dropped the EICar file in one of the directories so I know the scanner is working. Also unchecked LIVE GRID and disabled real-time.

 

So I assume that eicar was detected.

 

YES..it did pickup the TEST file.

 

I can confirm that a 12GB folder containing only video files - mpg, avi, mp4 etc, is scanned virtually instantaneously, in a fraction of a second, on my system too.

I think the explanation is that the files are scanned but only the first block (disk sector) of each file. The total scan time is orders of magnitude less than the time required to read the complete files from disk into memory. However, it is consistent with reading only the first disk sector of each file into memory. Therefore, the factor that determines scan time is the number of files, not their size. A large number of small files require reading and testing a large number of items. A small number of large files require reading and testing only a small number of items. Your 300GB of videos are a small number of files compared to the 30GB of photos.

I performed the following experiment, which you can reproduce.

1 Temporarily disable protection so you can complete the next steps.
2 Copy the eicar ascii test string of 68 bytes from the eicar.org site.
3 Take a video file, such as an mpg, and make two extra copies of it to a new folder, renaming the copies as for example, vid-eic-start and vid-eic-mid. Keep this new folder, complete with its altered files, so they can be re-used for further tests.
4 With one of the copies, paste the eicar string to the start of the file, using a hex editor.
5 With the second copy, paste the eicar string to the middle of the file, using a hex editor.
6 Repeat steps 3 to 5 with different video files. I did this with an mpg and an mp4 file.
7 Copy the edited files into the original large folder of videos, so it now contains all the original files plus the ones with eicar string.
8 Enable protection.
9 From the context menu, scan the large folder of videos with NOD32.

10 The scan detected the eicar string in the files which had it in the start but not in those which had it in the middle. The scan of the folder took 29 seconds rather than a fraction of a second as NOD32 spent time trying to clean and quarantine.

I then performed the following experiment.

11 Disable protection and delete any eicar files from the large folder if they had not actually been removed by NOD32.
12 In the folder of edited videos from step 6, make a zip file containing the videos with the eicar string plus a non-edited normal video file.
13 Copy the zip file into the large folder of videos.
14 Enable protection.
15 From the context menu, scan the large folder of videos with NOD32.

16 NOD32 detected eicar within the zip but only in those videos which had the eicar string at the start. The scan took 3 min 26 sec.

In order to detect within the zip, NOD32 had to read the complete zip file from the disk and expand it. Although the individual video files within the zip cannot all have been at the start of the zip file, eicar was detected. But eicar was only detected within the zip in the videos which had eicar at their start.

I also tried changing the extension of the videos to exe and repeated the above experiment. The results were the same. This confirms that NOD32 is not treating video files less strictly than exe files.

I draw the following conclusion.

If a file is not an archive (zip etc), only the first block of the file is read into memory and scanned for signature based threats.

If a file is an archive, the whole archive is read from disk and expanded and then only the first block of each component file is scanned for signature based threats.

This confirms the explanation that scanning speed for large files is so fast because only the first block is read and scanned by signature. Presumably, if a large file is actually executed, HIPS would detect and block suspicious activity which could have resulted from malicious code anywhere within a file.

 

Currently the "HIPS" doesnt have an file/process analyzer.

 

I was not implying that file scanning is related to HIPS. I need not have mentioned HIPS at all and I did not test anything regarding HIPS.

I demonstrated that scanning only detects signature based infection in the first block of the file. Hence a file could be malicious even if its infection does not result from the first block and such a file would not be detected in a scan.

I only mentioned HIPS to reassure myself that HIPS would detect malicious action caused by execution of such a file. In other words, HIPS has nothing to do with the scanning process but it would come into action if execution of a file caused behaviour which HIPS regarded as suspicious.

 

Eicar is not supposed to be detected in this case. The definition of the eicar test file is as follows:

Any anti-virus product that supports the EICAR test file should detect it in any file providing that the file starts with the following 68 characters, and is exactly 68 bytes long:
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
(http://eicar.org/86-0-Intended-use.html)

 

Most of Your technical assumptions on file scanning are correct.
I have made a similar experiment with on demand scans in a SFX-ZIP and the scanning of the EXE is performed, then scanning of the extracted content is performed. Superficial and then internal scanning

But the current HIPS is not intended for determination of good and bad behavior. It wont tell you it has detected a suspicious behavior.

 

OK, let us forget about HIPS. I wish I had never mentioned it. The question (and my reply) were not about HIPS.

I also was aware that the eicar string is only meant to be detected when it is at the start of a file and that such is stated in the extract that Marcos quoted.

However, the speed of scanning confirms that only the first sector of each file is scanned. The total scan time for the large files is orders of magnitude smaller than the time required to read the complete files from disk into memory. There is no way that 300GB could be read in 5 seconds, let alone examined. (Or, as in my own experiment, 12GB in a fraction of a second). Only the first sector of each file is read and scanned.

This leads to the following question.

Why is it safe to only scan the first sector of each file?

 

Not sure the necessity to scan avi mpg, vob files except to establish that they are indeed such files and then promptly ignore them.

 

The second part of my quote is also very important to understand what the eicar test file is:
the file starts with the following 68 characters, and is exactly 68 bytes long

Every AV scanner scans files optimally to detect malware without slowing down the computer to a crawl. It's not true that it's necessary to scan gigabytes of data in order to determine if a file is malware or infected with a virus.

 

Indeed. I was actually somewhat surprised that eicar was detected at all in video files.

It might be possible to determine from the first sector whether a file's content was consistent with its extension. And then non-executable files need not be scanned. However, it appears that NOD32 is actually examining the first sector of (non-executable) video files and detecting eicar when at the start of the file.

The scanning speed for folders of videos is consistent with only scanning the first sector of each file.

My test does not distinguish between the following possible explanations of why only the first sector is scanned.

1 Only the first sector of any (non-archive) file is ever scanned.
2 The first sector is scanned to determine if a file's content is consistent with its extension. If a file is consistent with its extension but not executable, then it is not scanned further.

 

Understood. But this means eicar is a rather weak test. It establishes only that the AV scanner detects a specially crafted test string at the start of a file. An AV could pass that test even if it could not detect any other infection. The scan certainly does detect eicar even within a file that is considerably longer than 68 bytes, provided that the string is at the start of the file.

I can see that malware or infection can be detected without scanning a complete file. However, it is much harder to see how a file can be proved to be safe by scanning only part of it. Of course, I understand that the scan must be optimised to prevent slowing the computer.

The OP was concerned about whether the speed of the scan meant that video files were being incorrectly ignored. I can see that it is perhaps not necessary to scan video files at all.

I suppose it would be possible to test scan speed on a folder consisting only of very large exe files but large exe files are hard to acquire.

 

Exactly. Eicar is detected by all AV vendors and is used to test program's functionality. It's no way meant to be used for testing detection capabilities.

 

Thanks Marcos. I guess that the purpose of the eicar file is to say "Show me what would happen if a known virus were found". It shows only that some scanning is taking place and shows what happens when the scan detects an infection.

It is clear that NOD32 is not scanning complete video files. It is also clear that this behaviour is by design and is not a fault.

 

Exactly, taking that into account I have no doubts ESET detect all "other" threats in different locations other than the first byte inside a file, and I think testing consistency with the File extension is unnecessary as the File type is defined by the header. Exceptions? Maybe...

 

you can scan them with other Ondemand scanner

 

Hello,

Some multimedia formats can have exploits embedded in them, such a a link to download a malicious codec, but the locations in which such things can be injected are fairly determinate, and those are the locations in the file in which the engine is going to look for those threats.

Regards,

Aryeh Goretsky

 

Thank you Aryeh. That is very clever, as I would expect from ESET! It explains why the scan can be so fast, by only looking at a few special logical blocks of each file, but still not miss infections.

My test with eicar clearly demonstrated that the first sector of each file is included in the scan, since eicar was detected when it was present there. But because eicar is not designed to be detected in the middle of a file, the test could not indicate which other sectors were scanned. The speed clearly indicates that only a few sectors could be scanned in each file.

Your answer puts the pieces of the mystery together, explaining the speed and giving us the confidence that threats would be detected. And it is extremely clever, because scanning every sector of 300GB would take a very long time.

I have always been somewhat awestruck by the depth of knowledge which developers of AV software require.