A learning model to detect maliciousness of portable executable using integrated feature set

itman · Jul 23, 2017

An interesting research paper for those interested in AI/Next Gen malware detection methods.

Summarized as follows:

Methodology and feature set

The proposed work is carried out in four main steps: (1) raw samples collection, (2) pre-processing (includes labeling), (3) feature extraction and 4) training & testing. Fig. 2 illustrates all four steps of the work. In the proposed work, raw malware and benign samples are collected from various sources (explained in sub-Section 4.1) and are stored separately for further processing. Raw samples must be processed to make use of machine learning training. The tasks in pre-processing step are file type identification, duplicate removal, and labeling.

~ Removed Copyrighted Image ~

Labeling is an important task and it can be performed either locally by installing one or more signature-based anti-virus engine or using cloud-based service where multiple anti-virus engine based scan could be performed in parallel. We labeled each sample in both malware and benign group by cloud based service Virustotal.

Labeled samples are then passed to feature extraction step where PE header field’s values are extracted to create raw and integrated feature set (explained in further Section 3.1 and 3.2). With raw and integrated feature set, different machine learning algorithms are trained and tested in last step (explained in Section 4 i.e. training and testing).

Feature generation is an important task of the proposed work, Algorithm 1 for that lays down the important steps performed to generate the raw and integrated feature set. All labeled malware and benign samples along with crafted header rules are input to the algorithm. Two loops, one for benign and other for malware samples run to bring sample one by one for processing. By calling FetchFieldsValue() procedure, all values are extracted from different fields of PE sample file. Raw feature set (i.e. RawF) is updated with all extracted values and appending class label. For integrated feature set (i.e IntF), CheckRules() procedure is called by passing extracted values (i.e. RawValue) and rules and it returned derived values which is used to update integrated feature set along with FileValue and class label. This CheckRules() procedure takes all raw values as input but rules are checked against only selected fields (explained in Section 3.2.2) and other field values are returned without any change. ExtractFileFeatures() procedure takes a sample as input and return other derived values such as file size, file entropy related to file properties (explained in Section 3.2.2).
Click to expand...

http://www.sciencedirect.com/science/article/pii/S1319157817300149

itman · Jul 23, 2017

Of note is that although signatures are not directly employed by the AI/Next Gen products, they are very much used in the "conditioning" phase of their respective detection engines.

This provokes the question of what happens if AV signatures and the products that use them were to no longer exist? Would the AI/Next Gen engines be as effective?

Rasheed187 · Jul 23, 2017

Perhaps Dan from VoodooShield will find this interesting, but it really looks too complex for me.

itman said: ↑

This provokes the question of what happens if AV signatures and the products that use them were to no longer exist? Would the AI/Next Gen engines be as effective?
Click to expand...

I think you will always need to have signatures, but AI will become more and more important. False positives should also be considered, I've read that this is the biggest problem with Cylance.

Log in or Sign up

A learning model to detect maliciousness of portable executable using integrated feature set

itman Registered Member

itman Registered Member

Rasheed187 Registered Member

Log in or Sign up

A learning model to detect maliciousness of portable executable using integrated feature set

itman Registered Member

itman Registered Member

Rasheed187 Registered Member

Useful Searches