Virustotal Pipeline

This pipeline consumes pcap files from s3 resource compatibible storage, archives pcaps and analyzes the files with suricata and virustotal.

Note: A virustotal api key is required, the free public api key has 4 request/min limit, which will slow down the pipeline.

Tools run in the pipeline

Suricata / iocextract / Virustotal

How it works

The pipeline requires s3 storage with 3 buckets (Default names): 1. concourse-archive 2. concourse-input 3. concourse-output

Any .pcap file dropped into the concourse-input will be consumed (analyzed and archived). The analysis includes 3 tasks: 1. Suricata, which parses the pcap and runs the packets through thousands of rules and outputs multiple logs. The output logs are passed to the next task and stored in concourse-output bucket. 2. Iocextract, goes through all the suricata logs and extracts any md5, sha1 or sha256 hashes and logs them into newline delimited text file. 3. Virustotal checks if the hashes match any files their system has analyzed. The pure data from virustotal and a readable log file are stored in concourse-output bucket.

Running the pipeline

  1. Setup concourse, useful tutorial here.
  2. Setup s3 instance with concourse-archive, concourse-input and concourse-output buckets. Example docker-compose.yml with concourse and minio as s3 resource can be found here
  3. Rename credentials.yml.template and fill in the needed credentials
  4. Login:
    fly --target CONCOURSE_TARGET_NAME login -u CONCOURSE_USERNAME -p CONCOURSE_PASSWORD
  5. Set the pipeline:
    fly --target CONCOURSE_TARGET_NAME set-pipeline -p PIPELINE_NAME -c suricata-virustotal.yml -l credentials.yml
  6. Drop pcap files to concourse-output bucket and follow the progress with browser at 127.0.0.1:8080

For more secure way to handle credentials check out concourses official tutorial.

Further possibilities with tweaking

  1. Iocextract tool is capable of extracting urls and Virustotal can analyze the given urls.
    Iocextract can also list emails and ip addresses found in the files.
  2. Push logs and reports to git.
  3. Virustotal can analyze whole files through the API (Under 32Mb), suricata can extract unencrypted files found in the pcap, which could be a significant analysis -> Virustotal could be changed to any other analysis tool for malware files