Customizing Pipelines

As mentioned in previously, Ocular is designed to be highly configurable through the use of container images. In the previous guide we configured the scanner in the profile to be a custom image, but Ocular can support much more than that.

This guide aims to show you how you can customize the other two parts of a pipeline: downloaders and uploaders

Downloaders

Downloaders allow the customizing of how a target is written to disk to be scanned. A downloader will be given two environment variables to specify the target:

  1. OCULAR_TARGET_IDENTIFIER - This should uniquely identify the target for this downloader
  2. OCULAR_TARGET_VERSION (Optional) - This (if specified) should contain the version of the target to download.

The downloader will be expected to write the target to disk, at the ’target’ directory which is set to the current working directory of the downloader when executed. (The path is also given by the environment variable $OCULAR_TARGET_DIR). The program should exit with code 0 upon success, and any other code when it was unable to download the target.

NOTE: It is up to the container to determine how the identifier and version are parsed and additionally how to

In the example below we will write a custom downloader that can be used to download the contents of any webpage to scan.

Creating a custom downloader

Lets assume we want to scan webpages that we find on the internet. In order to conform with the standards for a downloader, we need to have a container image that will be given a web URL and download the contents of the file to disk. We can do this will a simple curl command:

curl -fsSL "$OCULAR_TARGET_IDENTIFIER" -o ./target

This command will use the target identifier as the web URL and output the contents to the file ./target. This example doesn’t use a target version since its not required. We need to run this on an image, luckily we can use alpine/curl as our image since it comes with curl and a shell.

We can define a downloader using YAML (NOTE: API supports both JSON and YAML for encoding):

# my-downloader.yaml
image: alpine/curl:latest
imagePullPolicy: IfNotPresent
# We are setting the command as /bin/sh
# so that the environment variables can be resolved
command: ['/bin/sh', '-c']
args: ['curl -fsSL "$OCULAR_TARGET_IDENTIFIER" -o ./target']

Using this YAML (the file my-downloader.yaml) as a payload we can define it in the Ocular API and give it the name webpage.

curl -fsSL "${OCULAR_API_HOST}/api/v1/downloaders/webpage" \
	-X POST \
	-H "Authorization: Bearer ${OCULAR_API_TOKEN}" \
	-H "Accept: application/yaml" -H "Content-Type: application/x-yaml" \
	-d @my-downloader.yaml

Triggering a Pipeline with your downloader

We can now trigger a pipeline using our uploader. It is as simple as setting the value target.downloader to your new downloader in the pipeline request.

In this example, we will use our webpage downloader to download https://ocularproject.io/index.html:

curl -fsSL "${OCULAR_API_HOST}/api/v1/pipelines" \
	-X POST \
	-H "Authorization: Bearer ${OCULAR_API_TOKEN}" \
	-H "Accept: application/json" -H "Content-Type: application/json" \
	-d '{ "target": { "identifier": "https://ocularproject.io/index.html", "downloader": "webpage" }, "profileName": "example"}'

NOTE: this example assumes that the profile example exists and is able to scan the file ./target

Uploaders

Uploaders allow the customizing of how a artifacts are sent to 3rd party services. An uploader will be given the paths to all files that should be uploaded. The CLI arguments to the container will be appended with each file path, prefixed by -- (to differentiate from any CLI arguments the user may add). For example, if the artifacts file1.txt, file2.yaml and file3.go are being uploaded and the uploader definitions has args set to ["--arg1", "--arg2"] - then the resulting args for the container would be ["--arg1", "--arg2", "--", "file1.txt", "file2.yaml", "file3.go"].

NOTE: It is up to the container to determine how and where to upload the files.

In the example below we will write a custom uploader to trigger a lambda with the contents of each file

Creating a custom uploader

Lets assume we want to trigger a “processor” lambda, that will do something with the results. In order to conform with the standards for an uploader, we need to have a container image that will be given file paths via the CLI and trigger a lambda for each, with the payload being the contents of the file.

We can use the AWS CLI for this (and a bit of clever bash). The command aws lambda invoke --function-name $OCULAR_PARAM_LAMBDA_NAME --payload "file://$file" will invoke the lambda $OCULAR_PARAM_LAMBDA_NAME and make the payload $file.

NOTE: This would require authentication as well, but we’re going to skip over that for now.

Uploaders can be defined with parameters, so that separate profiles can use the same uploader but invoke it in different ways. For this example we will make the lambda function name the parameter and will use the name LAMBDA_NAME. Ocular will then assign the value given when the uploader is invoked to an environment variable with the same name, prefixed by OCULAR_PARAM (You can see that in the command above). We’ll use the amazon/aws-cli image since it comes with both bash and the AWS CLI.

We can define our uploader using YAML (NOTE: API supports both JSON and YAML for encoding):

# my-uploader.yaml
image: amazon/aws-cli:latest
imagePullPolicy: IfNotPresent
# This next section involves some clever
# bash, TLDR we will loop through each file
# calling the invoke function with the payload as the file name
command: ['/bin/bash', '-c']
args:
  - |
	for file in "${@:1}"; do
		aws lambda invoke --function-name $OCULAR_PARAM_LAMBDA_NAME --payload "file://$file"
	done

# This script works because the file paths
# will be appended to 'args' and when using bash -c
# any args after the script will be set at $@,
# we just need to remove the first element ('--')
# and we can iterate through all files

Using this YAML (the file my-uploader.yaml) as a payload we can define it in the Ocular API and give it the name lambda.

curl -fsSL "${OCULAR_API_HOST}/api/v1/uploaders/lambda" \
	-X POST \
	-H "Authorization: Bearer ${OCULAR_API_TOKEN}" \
	-H "Accept: application/yaml" -H "Content-Type: application/x-yaml" \
	-d @my-uploader.yaml

Specifying the Uploader in a Profile

Once the uploader is specified we need to configure a profile to use it. Lets assume we two scanner images ocular/scanner-a:latest and ocular/scanner-b:latest. The will produce the outputs a.json and b.json, respectfully. We can use our lambda uploader to trigger our lambda My-Custom-Lambda with both results.

In the profile below we define the two scanners, their artifacts, and configure the uploader using the LAMBDA_NAME parameter.

scanners:
	- image: ocular/scanner-a:latest
	  command: ["/bin/sh", -c]
	  args: ["scanner-a --output $OCULAR_RESULTS_DIR/a.json"]
	- image: ocular/scanner-b:latest
	  command: ["/bin/sh", -c]
	  args: ["scanner-b --output $OCULAR_RESULTS_DIR/b.json"]

artifacts:
	- a.json
	- b.json

uploaders:
	- name: lambda
	  parameters:
		  LAMBDA_NAME: "My-Custom-Function"

You can then use this profile to scan and upload using your custom lambda image!

Summary

In this guide, you’ve learned how to:

  1. Create a custom downloader for targets
  2. Create a custom uploader to integrate with a third party service

In the next guide, we will configure searches to automatically crawl and scan targets.