How to find malware in installation files?

How do I find malware in large installation files?

Lets say I have a 300MB installation file which I know has a rootkit hidden in the installation somewhere. Lets also assume the rootkit is obfuscated and no anti-malware scanner is detecting it.

Would I have to install the application in vmware and use some kind of cloud scanner to find it?

 

I use Prevx in my virtual machines for this purpose.

 

Install into a VM then use the below search utility to find all created files in last 5, 10 or 20 minutes.

*.* will find all files or if you want specific files then just use .exe, .sys or .dll etc.

Nirsoft Search My Files

If you find a suspected undetected rootkit you could grab and upload the sample to kernelmode and ask if one of the experts over there can take a look at it.

 

would some rootkits be able to recognize the virtualized environment and remain stealth?

 

There are a lot of installers that you can use something like 7-Zip and open as archive.

 

Yes, they could. But if you already have doubts of this kind on a installer then I would stay miles away from it anyway.

 

You could try to install the program into an empty Sandbox by running the installer using Sanboxie (it's a tiny program with a very free version available). If the installation is successful, scan the Sandbox with your preferred AV. Even if the installation fails partially, in the sense that the program won't run inside the Sandbox, it may still be possible to scan the files for viruses. I think there are specialized add-on programs available for Sandboxie to analyze installations, but they take some effort to learn.

You can also try Universal Extractor (free program) to try to extract the installer and then scan the files.

Installing the program inside a virtual machine is the best way, but VMs take a lot of space.

 

300MB is to much but with little installers you can try this:
http://camas.comodo.com/
http://www.sunbeltsecurity.com/sandbox/
http://www.joebox.org/ (very detailed)
http://anubis.iseclab.org/index.php (very detailed)

 

I usually extract installers with Universal Extractor or run them in a sandbox and grab/check the contents from there.