pipeline status

Introduction

This repository contains the CinCan pilot environment that constructs from two main services Concourse CI and Gitlab CE. Concourse CI is used for creating, executing and managing the CinCan CI pipelines whereas Gitlab CE stores samples to be analyzed, found artifacts, results, logs etc. Every component of the service is run in a separate Docker container which makes the environment easy to deploy and destroy without affecting the host system.

The full list of components:

Prerequisites

  • Ubuntu 18.04 (others untested)
  • Ensure that your system has more than 6 GB memory
  • Ensure that your system has more than 4 GB free disk space
  • Docker >= 17.12.0
  • Docker Compose >= 1.18.0

1. Install docker & docker-compose

Read the wiki for help about installing/upgrading docker & docker-compose.

2. Deploy the pilot environment

# Clone the repository
git clone https://gitlab.com/CinCan/environment.git

# Change directory
cd ./environment

# Run the script (requires root privileges)
sudo bash build.sh -e BUILD_PATH=/path/to (default: /opt/cincan)

# Change directory
cd /path/to/build

# Build the environment using Docker Compose
docker-compose up -d db \
                     ci \
                     worker \
                     gitlab \
                     dns (optional) \
                     reg (optional)

The Gitlab will take some time to get up (~5 minutes). You can follow the starting process using following command:

docker logs gitlab.cincan.io --follow

* See an asciicast video of the environment setup *

3. Set up a pipeline

You can list all pipelines that have a setup script with sudo bash setup-pipeline.sh

sudo bash setup-pipeline.sh  

[+] Cloning the pipelines.git

Available pipelines
1) pdf-pipeline
2) pdf-pipeline Private registry version
3) document-pipeline
4) Quit

Currently, there are two pilot pipelines with quick setup: pdf-pipeline, and the more advanced document-pipeline

To setup a pipeline:

  1. Set up the pilot environment

  2. sudo bash setup-pipeline.sh

  3. Select the pipeline to set up

  4. Login to https://172.20.0.3 to see the pipeline work. (Or https://concourse.cincan.io if proxy is enabled)

* See an asciicast video of the pipeline setup *


How to stop and remove the whole pilot environment

cd /path/to/build
docker-compose down --volumes --rmi all && rm -rf /path/to

Steps to set up a pipeline in the private registry environment

  1. Deploy the pilot environment as described above, including the "reg" option

  2. Run sudo bash setup-private-registry.sh

sudo bash setup-private-registry.sh
  • Type in environment username and password when prompted.
  • Select the tools to use from the pop up dialog box. For example, if you are setting up the pdf-pipeline, check "jsunpack-n", "pdfid" and "peepdf".
  • Choose whether you want to download the tool images from the Docker hub, or to build them locally.
  1. Run sudo bash setup-pipeline.sh and choose a Private registry version pipeline

  2. Login to https://172.20.0.3 to see the pipeline work. (Or https://concourse.cincan.io if proxy is enabled)

Read more

Read more about the pilot environment installation options from the wiki
Read more about the pdf-pipeline here.
Read more about the document-pipeline here.