Customizing Pipelines

As mentioned in previously, Ocular is designed to be highly configurable through the use of container images. In the previous guide we configured the scanner in the profile to be a custom image, but Ocular can support much more than that.

This guide aims to show you how you can customize the other two parts of a pipeline: downloaders and uploaders

The section below assumes you created the example-profile from the previous section “Quick Start”.

Downloaders

Downloaders allow the customizing of how a target is written to disk to be scanned. A downloader will be given two environment variables to specify the target:

  1. OCULAR_TARGET_IDENTIFIER - This should uniquely identify the target for this downloader
  2. OCULAR_TARGET_VERSION (Optional) - This (if specified) should contain the version of the target to download.

The downloader will be expected to write the target to disk, at the ’target’ directory which is set to the current working directory of the downloader when executed. (The path is also given by the environment variable $OCULAR_TARGET_DIR). The program should exit with code 0 upon success, and any other code when it was unable to download the target.

NOTE: It is up to the container to determine how the identifier and version are parsed and additionally how to

In the example below we will write a custom downloader that can be used to download the contents of any webpage to scan.

Creating a custom downloader

Lets assume we want to scan webpages that we find on the internet. In order to conform with the standards for a downloader, we need to have a container image that will be given a web URL and download the contents of the file to disk. We can do this will a simple wget command:

wget "$OCULAR_TARGET_IDENTIFIER"

This command will use the target identifier as the web URL and download the file to disk. This example doesn’t use a target version since its not required. We need to run this on an image, luckily we can use busybox since we only need wget to be installed.

# curl-downloader.yaml
apiVersion: ocular.crashoverride.run/v1beta1
kind: Downloader
metadata:
	name: wget-downloader
spec:
	container:
		name: static-download
		image: busybox:latest
		imagePullPolicy: IfNotPresent
		# We are setting the command as /bin/sh
		# so that the environment variables can be resolved
		command: ['/bin/sh', '-c']
		args: ['wget "$OCULAR_TARGET_IDENTIFIER"']

Using this YAML (the file wget-downloader.yaml) as a payload we can define it in the Ocular API and give it the name webpage.

kubectl apply -f ./wget-downloader.yaml

Triggering a Pipeline with your downloader

We can now trigger a pipeline using our uploader. It is as simple as setting the value target.downloader to your new downloader in the pipeline request.

In this example, we will use our wget downloader to download a random script from the Linux kernel https://raw.githubusercontent.com/torvalds/linux/refs/heads/master/scripts/gdb/linux/stackdepot.py

cat <<EOF | kubectl create -f -
apiVersion: ocular.crashoverride.run/v1beta1
kind: Pipeline
metadata:
	generate-name: my-pipeline-
spec:
	downloaderRef:
		name: curl-downloader
	profileRef:
		name: example-profile
	target:
		name: "https://raw.githubusercontent.com/torvalds/linux/refs/heads/master/scripts/gdb/linux/stackdepot.py"
EOF

NOTE: this example assumes that the profile example-profile was created in the previous step

Viewing the logs of the scanner pods created should show that they scan the file ‘stackdepot.py’

Uploaders

Uploaders allow the customizing of how artifacts are sent to 3rd party services. An uploader will be given the paths to all files that should be uploaded. The CLI arguments to the container will be appended with each file path, prefixed by -- (to differentiate from any CLI arguments the user may add). For example, if the artifacts file1.txt, file2.yaml and file3.go are being uploaded and the uploader definitions has args set to ["--arg1", "--arg2"] - then the resulting args for the container would be ["--arg1", "--arg2", "--", "file1.txt", "file2.yaml", "file3.go"].

NOTE: It is up to the container to determine how and where to upload the files.

In the example below we will write a custom uploader to trigger a lambda for each file with the body set to the contents of the file.

Creating a custom uploader

Lets assume we want to trigger a “processor” lambda, that will do something with the results. In order to conform with the standards for an uploader, we need to have a container image that will be given file paths via the CLI and trigger a lambda for each, with the payload being the contents of the file.

We can use the AWS CLI for this (and a bit of clever bash). The command aws lambda invoke --function-name $OCULAR_PARAM_LAMBDA_NAME --payload "file://$file" will invoke the lambda ‘$OCULAR_PARAM_LAMBDA_NAME’ and make the payload ‘$file’.

NOTE: This would require authentication as well, but we’re going to skip over that for now and assume the serivce account lambda-uploader-sa exists in the cluster and is tied to an AWS role that has the ability to trigger the lambda. Later in the docs will we cover how to configure this.

Uploaders can be defined with parameters, so that separate profiles can use the same uploader but invoke it in different ways. For this example we will make the lambda function name the parameter and will use the name LAMBDA_NAME. Ocular will then assign the value given when the uploader is invoked to an environment variable with the same name, prefixed by OCULAR_PARAM (You can see that in the command above). We’ll use the amazon/aws-cli image since it comes with both bash and the AWS CLI.

# lambda-uploader.yaml
apiVersion: v1beta1
kind: Uploader
metadata:
	name: lambda-uploader
spec:
	container:
		name: lambda
		image: amazon/aws-cli:latest
		imagePullPolicy: IfNotPresent
		# This next section involves some clever
		# bash, TLDR we will loop through each file
		# calling the invoke function with the payload as the file name
		command: ['/bin/bash', '-c']
		args:
			- |
				for file in "${@:1}"; do
					aws lambda invoke --function-name $OCULAR_PARAM_LAMBDA_NAME --payload "file://$file"
				done
				# This script works because the file paths
				# will be appended to 'args' and when using bash -c
				# any args after the script will be set at $@,
				# we just need to remove the first element ('--')
				# and we can iterate through all files
	parameters:
		- name: LAMBDA_NAME
		  required: true

Using this YAML (the file lambda-uploader.yaml) as a payload we can create the resource in Kubernetes

kubectl apply -f lambda-uploader.yaml

Specifying the Uploader in a Profile

Once the uploader is specified we need to configure a profile to use it. Lets assume we two scanner images ocular/scanner-a:latest and ocular/scanner-b:latest. The will produce the outputs a.json and b.json, respectfully. We can use our lambda uploader to trigger our lambda My-Custom-Lambda with both results.

In the profile below we define the two scanners, their artifacts, and configure the uploader using the LAMBDA_NAME parameter.

apiVersion: v1beta1
kind: Profile
metadata:
	name: example-profile-with-uploader
spec:
	scanners:
		- image: ocular/scanner-a:latest
		  name: scanner-a
		  command: ["/bin/sh", -c]
		  args: ["scanner-a --output $OCULAR_RESULTS_DIR/a.json"]
	    - image: ocular/scanner-b:latest
		  name: scanner-b
		  command: ["/bin/sh", -c]
		  args: ["scanner-b --output $OCULAR_RESULTS_DIR/b.json"]

	artifacts:
		- a.json
		- b.json

	uploaderRefs:
		- name: lambda
		  parameters:
		    - name: "LAMBDA_NAME"
			  value: "My-Custom-Function"

Then triggering a pipeline with this profile will invoke our uploader with the resulting a.json and b.json as arugments. The following is the updated pipeline invocation:

cat <<EOF | kubectl create -f -
apiVersion: ocular.crashoverride.run/v1beta1
kind: Pipeline
metadata:
	generate-name: my-pipeline-
spec:
	downloaderRef:
		name: curl-downloader
	profileRef:
		name: example-profile-with-uploader
	target:
		name: "https://github.com/my-org/my-repository"
	# We assume the service account 'lambda-uploader-sa'
	# exists in the same namespace and is tied to an IAM
	# role that can invoke 'My-Custom-Function'
	uploaderServiceAccountName: "lambda-uploader-sa"
EOF

Summary

In this guide, you’ve learned how to:

  1. Create a custom downloader for targets
  2. Create a custom uploader to integrate with a third party service

In the next guide, we will configure searches to automatically crawl and scan targets.