How to get started developing for the Pipelines-as-Code project #

Please read the Code of conduct #

It’s important: https://github.com/openshift-pipelines/pipelines-as-code/blob/main/code-of-conduct.md

Use the all in one install on kind to develop #

It uses kind under docker. You start it with:

make dev

When it finishes, you will have the following installed in your kind cluster:

Kind Cluster deployment
Internal registry to push to from ko
An ingress controller with nginx for routing.
Tekton and Dashboard installed with an ingress route.
Pipelines as code deployed from your repo with ko.
Gitea service running locally so you can run the E2E tests against it (Gitea has the most comprehensive set of tests).

By default, it will try to install from $GOPATH/src/github.com/openshift-pipelines/pipelines-as-code. To override it, set the PAC_DIRS environment variable.

It will deploy under the nip.io domain reflector, the URL will be:
- http://controller.paac-127-0-0-1.nip.io
- http://dashboard.paac-127-0-0-1.nip.io
You will need to create the secret yourself. If you have the pass cli installed, you can point to a folder that contains: github-application-id, github-private-key, webhook.secret as configured from your GitHub application. Set the PAC_PASS_SECRET_FOLDER environment variable to point to it. For example:
```
pass insert github-app/github-application-id
pass insert github-app/webhook.secret
pass insert -m github-app/github-private-key
```

If you need to redeploy your pac install (and only pac), you can do:

./hack/dev/kind/install.sh -p

make rdev

or you can do this directly with ko:

env KO_DOCKER_REPO=localhost:5000 ko apply -f ${1:-"config"} -B

more flags: -b to only do the kind creation+nginx+docker image, -r to install from the latest stable release (override with the env variable PAC_RELEASE) instead of ko. -c will only do the pac configuration (i.e., creation of secrets/ingress, etc..)
see the install.sh -h for all flags

Gitea #

Gitea is “unofficially” supported. You just need to configure Gitea the same way you do for other webhook methods with a token.

Here is an example of a Gitea NS/CRD/Secret (set to empty):

---
apiVersion: v1
kind: Namespace
metadata:
  name: gitea

---
apiVersion: "pipelinesascode.tekton.dev/v1alpha1"
kind: Repository
metadata:
  name: gitea
  namespace: gitea
spec:
  url: "https://gitea.my.com/owner/repo"
  git_provider:
    user: "git"
    url: "Your gitea installation URL, i.e: https://gitea.my.com/"
    secret:
      name: "secret"
      key: token
    webhook_secret:
      name: "secret"
      key: "webhook"
---
apiVersion: v1
kind: Secret
metadata:
  name: gitea-home-chmouel
  namespace: gitea
type: Opaque
stringData:
  token: "your token has generated in gitea"
  webhook: "" # make sure it's empty when you set this up on the interface and here

There are some gotchas with the webhook validation secret. Pipelines-as-Code detects a Gitea install and lets the user set an empty webhook secret (by default it’s enforced).

The install.sh script will by default spin up a new instance of GITEA to play with and run the Gitea E2E tests.

You will need to create a Hook URL generated from https://hook.pipelinesascode.com/new into the environment variable TEST_GITEA_SMEEURL.

The defaults are:

URL: https://localhost:3000/
Admin Username: pac
Admin Password: pac

The E2E tests will automatically create a repo using the admin username for each test.

Debugging E2E #

As long as you have the secrets set up, you should be able to run the e2e tests properly. Gitea is the easiest to run (since they are self-contained). For the rest, you will need to set up some environment variables.

See the e2e on kind workflow for all the variables set by the provider.

By default, the E2E tests clean up after themselves. If you want to keep the PR/MR open and the namespace where the test has been created, you can set the TEST_NOCLEANUP environment variable to true.

Debugging controller #

Create a hook URL and point your app/webhook to it. Use gosmee to forward the requests from GitHub to your locally installed controller (this can be either run on your debugger or inside kind).

An option of gosmee is to save the replay to a directory with --saveDir /tmp/save. If you go to that directory, a shell script will be created to replay the request that was sent directly to the controller without having to go through another push.

Use snazy to watch the logs. It supports pac by adding some context like which GitHub provider.

snazy screenshot

Using the Makefile targets #

Several targets are available in the Makefile if you want to run them manually. You can list all the makefile targets with:

make help

For example, to test and lint the go files:

make test lint-go

If you add a CLI command with help, you will need to regenerate the golden files:

make update-golden

Configuring the Pre Push Git checks #

We are using several tools to verify that pipelines-as-code is up to a good coding and documentation standard. We use pre-commit tools to ensure before you send your PR that the commit is valid.

First, you need to install pre-commit:

https://pre-commit.com/

It should be available as a package on Fedora and Brew or install it with pip.

Once installed, add the hook to your repo by doing:

pre-commit install

This will run several hooks on the files that have been changed before you push to your remote branch. If you need to skip the verification (for whatever reason), you can do:

git push --no-verify

or you can disable an individual hook with the SKIP variable:

SKIP=lint-md git push

If you want to manually run on everything:

make pre-commit

Developing the Documentation #

Documentation is important to us. Most of the time, new features or changes in behavior need to include documentation as part of the Pull Request.

We use hugo. If you want to preview the changes you made locally while developing, you can run this command:

make dev-docs

This will download a version of Hugo that is the same as what we use on Cloudflare Pages (where pipelinesascode.com is generated) and start the Hugo server with a live preview of the docs on:

https://localhost:1313

When we push the release, the docs get rebuilt automatically by CloudFare pages.

By default, the website https://pipelinesascode.com only contains the “stable” documentation. If you want to preview the dev documentation as from main, you need to go to this URL:

https://main.pipelines-as-code.pages.dev

There is a drop-down at the bottom of the page to let you change the older major version.

Documentation when we are doing the Release Process #

See here release-process

How to update all dependencies in Pipelines-as-Code #

Go Modules #

Unless that’s not possible, we try to update all dependencies to the latest version as long as it’s compatible with the Pipeline version as shipped by OpenShift Pipelines Operator (which should be conservative).

Every time you update the Go modules, check if you can remove the replace clause which pins a dependency to a specific version/commit or match the replace to the tektoncd/pipeline version.

Update all go modules:
```
go get -u ./...
make vendor
```
Go to https://github.com/google/go-github and note the latest go version, for example: v59
Open a file that uses the go-github library (i.e., pkg/provider/github/detect.go) and check the old version, for example: v56

Run this sed command:

find -name '*.go'|xargs sed -i 's,github.com/google/go-github/v56,github.com/google/go-github/v59,'

This will update everything. Sometimes the library ghinstallation is not updated with the new version, so you will need to keep the old version kept in there. For example, you will get this kind of error:

pkg/provider/github/parse_payload.go:56:33: cannot use &github.InstallationTokenOptions{…} (value of type *"github.com/google/go-github/v59/github".InstallationTokenOptions) as *"github.com/google/go-github/v57/github".InstallationTokenOptions value in assignment

Check that everything compiles and tests are passing with this command:
```
make allbinaries test lint
```
Some structs need to be updated. Some of them are going to fail as deprecated, so you will need to figure out how to update them. Don’t be lazy and avoid the update with a nolint or a pin to a dep. You only delay the inevitable until the problem comes back and hits you harder.

Go version #

Check that the go version is updated to the latest RHEL version:
```
docker pull golang
docker run golang go version
```
If this is not the same as what we have in go.mod, then you need to update the go.mod version. Then you need to update, for example, here 1.20:
```
go mod tidy -go=1.20
```
Grep for the image go-toolset everywhere with:
```
git grep golang:
```
and change the old version to the new version

Update the pre-commit rules #

pre-commit autoupdate

Update the vale rules #

vale sync
make lint-md

Tools that are useful #

Several tools are used in CI and pre-commit. The non-exhaustive list you need to have on your system:

golangci-lint - For golang lint
yamllint - For YAML lint
shellcheck - For shell scripts linting
ruff - Python code formatter check
vale - For grammar check
markdownlint - For markdown lint
codespell - For code spelling
gitlint - For git commit messages lint
hugo - For documentation
ko - To rebuild and push change to kube cluster.
kind - For local devs
snazy - To parse json logs nicely
pre-commit - For checking commits before sending it to the outer loop.
pass - For getting/storing secrets
gosmee - For replaying webhooks

Target architecture #

We target arm64 and amd64. The dogfooding is on arm64, so we need to ensure that all jobs and docker images used in the .tekton PipelineRuns are built for arm64.
A GitHub action is using ko to build the amd64 and arm64 images whenever there is a push to a branch or for a release.