Stainless AI dev-flow

This specification describes the continuous integration and delivery pipeline for Stainless AI’s projects. This proposal builds on existing work done at Stainless in this regard. This pipeline is ideal for small teams. The pipeline relies on the following tools/resources:

Goals

The pipeline should achieve the following goals:

  • Automate repetitive build tasks wherever possible.

  • Enforce versioning standards using tools.

  • Minimize the number of manual steps taken for any given operation:

  • Where it makes sense (for non-code repositories like DEV configuration environments, e.g.), a deployment can be automated

by a simple commit to a repository. * To release code, a single Jenkins job should be run * To promote code, a single Jenkins job should be run, etc. * Support different “team levels,” i.e., optimize for a single developer, a small team, and larger teams, by requiring differing levels of enforcement of lockdowns. * Write as little custom logic as possible to support this scheme, relying mostly on existing software, plugins and tools.

Team Levels (work-in-progress)

We introduce the concept of “team levels,” or “TL” to indicate levels of enforcement to support different teams in this structure. A TL is a code with an associated set of policy requirements intended to optimize the experience of that team using this framework. For example, if only a single developer is working on a project, using pull requests and locked branches may be more of a hindrance than an advantage (or maybe not, depends) so a project using “TL0” would not have any branch enforcement. Note that a single developer may elect to use TL3 policies, it’s up to them.

On larger teams, the level may change, so each level should be “backward-compatible,” i.e., all levels should share a compatible structure. Branches may change between levels (for example a shared “develop” branch may be introduced on moving to TL3 from TL2).

There are 3 team levels defined:

Team Levels

Team Level

Description

TL0

A single-developer project

TL1

2-3 developers/collaborators working on a project

TL2

3 or more developers working on a project

The policy enforcement increases with team level.

TL0: No policy enforcement.

  • No locked branches

  • All remote branches are read-write

  • Pull requests are not required for any operation

  • No required team code reviews

  • No formal approval workflow

  • Optional shared develop branch (or single develop - feature structure)

TL1: Small team policies

  • Master is read-only

  • Developers can push directly to release branches

  • Pull requests are not required for any operation

TL2: Full team policies

  • Master and release branches are read-only

  • Release branches are locked prior to release

  • Any commits that must go into a release branch must be merged by pull request

  • All commits must be merged into shared develop branch prior to moving to any release branch

  • All PRs must be approved

Versioning

This pipeline uses semantic versioning. Read about it at https://semver.org.

This pipeline uses pre-release semantics based on a -SNAPSHOT build suffix. Note that the word “SNAPSHOT” is arbitrary and this could be “alpha” or “candidate” or “rc” also.

Meta Build Data

For tags on the develop or feature branches, we append a build number to the standard semantic versioning scheme as build meta. So the full build identifier is: $MAJOR.$MINOR.$PATCH(-$BRANCH)-$BUILD_NUMBER-SNAPSHOT where BUILD_NUMBER is an increasing number, probably the Jenkins build number, although this is not strictly the required source of the number.

The BUILD_NUMBER is the only field that can be automatically bumped by Jenkins.

The $BRANCH field is only present on feature branches and the develop branch.

Git Repository Structure

This pipeline is based on the Gitflow structure.

To summarize, projects are structured as follows:

The master branch contains the latest published release (HEAD points to the latest version of the latest deployed release). Humans shouldn’t do anything on the master branch, only bots.

The develop branch contains the latest stable build that will become a future release. Artifacts published from develop have a -SNAPSHOT build meta tag.

The release branches contains all commits that are being staged for a particular release, and patch versions of the release tree moving forward.

The hotfix branch contains emergency release commits that must be merged to master in-between regular releases.

Specific features are developed in specific feature branches. Any artifacts deployed directly from feature branches must contain the branch name meta, e.g., myproject-0.2.5-myfeaturebranchname-$BUILD_NUMBER-SNAPSHOT.ext

Why develop and release branches?

There may be commits on develop that you want to deploy 2 patch releases out, or that haven’t been tested or committed in error. The release branch is a staging area where commits that are certain to be deployed go prior to the release artifact being produced.

_images/flow.jpg

Branch and Tag Names

The following branch names are reserved:

  • master

  • develop

  • v$MAJOR.$MINOR.x ← release branch, e.g., v0.1.x (contains patches only)

  • hotfixes

Any other valid branch name can be used to describe a feature branch.

Tags must follow the format:

$PREFIX@$SEMVER

Where $PREFIX is a valid (acceptable by Jenkins, build process and source repository) prefix string followed by the at sign “@” (omitted if no prefix is supplied), followed by the semantic version string.

For example, for a single-project repo, valid tags are of the form:

0.1.2

v0.1.2

and if multiple projects exist in a single repo, tags may exist of the form:

project1@0.2.5-00-SNAPSHOT

project1@0.2.5-myfeaturebranch-00-SNAPSHOT

project2@0.1.1

Development Stories

The following stories outline developer interaction with the process and describe the tasks that should be performed for each story. Actual implementation details are provided in a subsequent section.

Creating an New Project

A “project” as defined here is anything that maintains its own version history. It could live in its own repository or in a subdirectory of another repository. When creating a new project, the starting semantic version of the project is 0.0.1. If no tags or other versioning data is present, this default should be enforced by the build system. The initial branches are master, develop. An initial release can also be created at this time called v0.0.1.

Pushing Code

Any push to develop should result in a SNAPSHOT artifact being created and pushed to the artifact repository. The artifact will have the version $MAJOR.$LAST_MINOR+1.0, the build number and the SNAPSHOT suffix, e.g., the first snapshot from the repository will be 0.1.0-1-SNAPSHOT.

Creating a new Release

Releasing a version requires applying a tag to the repository commit that represents the revision you want to release. In order for the release artifact to be built, the build system should enforce that the tagged commit is on the master branch. If the tagged commit is not on master, the release version may be reserved but not released. It’s the developer’s responsibility to appropriately version the release. IOW, if it’s a breaking change, don’t make it a patch release, etc.

MAJOR and MINOR Releases

The develop branch should automatically build a SNAPSHOT release having version $MAJOR.$LAST_MINOR+1.0-SNAPSHOT. All release branches must end with .0, e.g., v1.0.0 or v1.1.0. Patch release versions should be computed from the tags in the repository. To bump MAJOR or MINOR, apply the appropriate tag to the commit that contains the desired release revision.

PATCH Releases

To bump patch, apply the appropriate tag to the commit that contains the desired release version. There should be a release branch labeled vMAJOR.MINOR.0 that tracks only the commits relevant to that MAJOR/MINOR revision. Builds on a release branch should automatically increment the patch version from the last tagged version. For example, if the v1.1.0 tag exists in the repository, any build on a release branch should have the semantic version v1.1.2-$BUILD_NUM-SNAPSHOT. The SNAPSHOT tag should only be removed when a v1.1.2 tag is applied to the repository, and at that point, all future builds on the release branch will have the semantic version v1.1.3-SNAPSHOT. To actually release the patch version, apply the appropriate tag, merge into master and build.

Working with Feature Branches

It’s a good practice to develop specific features on dedicated branches. Depending on team level, pull requests may be required or a simple merge into develop may be sufficient. All features branches should flow featurebranch develop release

Hotfixing Releases

Hotfixes are branched off of master, commits are added and then the commits are merged and released as per usual.

Build Process Implementation

Each project may implement a build toolchain of its choice. These notes describe a setup based on

  • Jenkins as the centralized, coordinating build system

  • Multibranch pipeline projects for automating artifact builds

  • GitHub webhooks for notifications

  • GitHub branches and tags for recording version history information. Jenkins will use the naming conventions described

herein to calculate version labels.

  • Each project should have an embedded Jenkinsfile (some repositories may have more than one) that contains the build

pipeline for that project (or subproject). Jenkins must be able to run the project’s toolchain in the build environment.

Ideally, the master branch should be locked down so that only a bot account can write to it. You’ll have to manually kick off any merges for these operations from Jenkins, where stainlessbot will be allowed to make the changes. All releases should come from the release branch only, developers will have to push commits there and then promote the release commit in a build job.

Jenkins is used to automate artifact creation. Jenkins will scan for, or be notified of, events that occur during the development cycle and take appropriate action. These events are detailed below.

Git commit to develop (including merges)

During regular development, developers will write code and commit to the develop branch, or a feature branch which is merged into develop. For each commit pushed, Jenkins will run the build as defined in each project’s Jenkinsfile(s). The build process for each project will determine if the build is up-to-date. If the build is up-to-date, the commit didn’t contain anything material to the change of the software deliverable (e.g., maybe a doc update or .gitignore file update, etc).

If the build changed, the build should produce a new artifact and automatically increment the BUILD_NUMBER of the artifact based on the last valid tag in the repository, then deploy that artifact to the configured repository, and tag the commit in GitHub with the new version.

Automatically Increment Version Numbers

This can cause confusion among developers, it should be discussed.

Building off any branch except master in Jenkins will result in an automatically incremented version number (based on the last version tag), not just a new build number. If you’re expecting 1.1.1 in a feature build then you should continue working off of 1.1.0 until ready to tag. Tagging as 1.1.1 and then building your feature branch will result in 1.1.2 builds. Once development is complete, add the patch version tag and then merge to master to release the desired version.

Project Requirements

Projects must adhere to certain requirements in order to support this build process.

Project build scripts should accept version information from Jenkins. The close release job will set the following environment:

VERSION_MAJOR, VERSION_MINOR, VERSION_PATCH, VERSION_PRERELEASE, BUILD_NUMBER

Tag Annotations

The annotation will serve as a record of all the changes that are going into a release. The text of the file should be human readable and should be written in laypeople’s terms. An average user of the app should be able to understand 90% of the changelog (occasionally a bug fix or code change will require jargon). Previous examples of this file can be found under the releases section of the GitHub repo.

Example Annotation

aws-cfn-configuration v1.7.0
   - Add new "Foo" API
   - Add new "Bar" section to left of Calendar
      - Add new "Bar List" panel
   - Add new "Bar" popup
   - Redesign "Status" screen
   - Minor changes and bugfixes:
      - Change type of number property in Event API
      - Correct typo in Logger GET API
   - Fix bug that prevented Settings screen from loading
   - Issues: HS-3 HS-23 HS-345 HS-555
      HS-434 HS-445 HS-554 HS-123

Indices and tables