PE-pipeline

This pipeline analyses binary files.

Tools run in the pipeline

Binwalk/PEframe/ClamAV/snowman-decompiler/python-extract-code/iocstrings/feature_extractor/ghidra/ssdeep/ssdc/osslsigncode

Pipeline workflow:

  1. Poll for new files in git repository

  2. Run jobs:

    • Run binwalk, clamAV, peframe, ssdeep, ssdec
    • Run decompile-binary to identify the compiler used in the binary sample, then run snowman-decompiler OR python-extract-code depending on the compiler identification

  3. Run ioc_strings for files extracted by binwalk

  4. Run feature_extractor for ioc_strings' output

The repositories

branch: master

  • Contains the scripts for the pipeline

  • The results will be written to tool specific folder under master branch

branch: sample-source

  • Files to be analysed

branch: configuration

  • Place to put API keys for the feature_extractor (and possibly other configuration files)

How to set up the pipeline

USING THE SCRIPT

You can set up the pipeline to the pilot environment with sudo ./setup-pipeline.sh 

sudo ./setup-pipeline.sh

[+] Cloning the pipelines.git

Available pipelines
1) document-pipeline
2) pdf-pipeline
3) pdf-pipeline Private registry version
4) pe-pipeline
5) Quit
Your choice:

Or you can give the desired pipeline as an argument: sudo ./setup-pipeline.sh pe-pipeline

The setup will copy the pilot environment's SSH keys to Gitlab using a private access token, and modify the pipeline's credentials to use resources with SSH.

MANUALLY

  1. Setup concourse (tutorial)

  2. Setup a git repository with branch:master, with the files included in the "scripts" folder.

  3. Setup branch:sample-source for the samples.

  4. Create branch:configuration with the files from the "conf" folder.

  5. Edit the credentials.yml with the details of your git and your ssh key.

  6. Login to concourse:

fly -t CONCOURSE_TARGET_NAME login -c http://127.0.0.1:8080 -u CONCOURSE_USERNAME -p CONCOURSE_PASSWORD

  1. Set up the pipeline:

fly -t CONCOURSE_TARGET_NAME sp -c pipeline.yml -p pe-pipeline -l credentials.yml

  1. Unpause the pipeline:

fly -t CONCOURSE_TARGET_NAME unpause-pipeline -p pe-pipeline

  1. Upload your samples to sample-source -branch