In Part 1 of this guide, the concepts of GoCD Server, GoCD Agents, pipelines, materials, stages, jobs and tasks were introduced. You were also able to get a simple pipeline with a single stage, job and task each working.
In Part 2, you learnt about using pipelines as materials (dependency materials), artifacts and custom tabs.
Let us continue, in this part, by learning about the Value Stream Map (VSM) and about fan-in and fan-out.
The Value Stream Map is the full end-to-end (from commits to production) view across pipelines, and maps to the concept of a "Continuous Delivery Pipeline" or "Deployment Pipeline", in the book, "Continuous Delivery" by Jez Humble and David Farley. In their book, a couple of different views of a Deployment Pipeline are presented:
A corresponding representation of a Value Stream Map in GoCD looks like this:
There are some unfortunate differences in naming between GoCD and the book. What is termed as a "Stage of a Deployment Pipeline" in the book is called a "Pipeline" in GoCD. The "Deployment Pipeline" (end-to-end) is called the "Value Stream Map". In the rest of this guide, the term "Pipeline" will refer to a GoCD Pipeline and the term "Stage" will refer to a "Stage of a GoCD pipeline".
Interestingly, Jez Humble was the first product manager of GoCD, and was around while these naming decisions were made. Rumour has it that he regrets the choice of the overloaded term, "Pipeline". :)
The previously defined concepts of a Pipeline, Artifact and Pipeline Dependency (chaining of pipelines) come together to enable a true end-to-end Deployment Pipeline, which is known as a Value Stream Map (VSM) in GoCD. The ability to do this, and so this concept is important because:
To use a Value Stream Map in GoCD, you don't really need to do anything. If you've pipelines which use pipeline dependencies, then you can always get a VSM at the click of a button. Here is a dashboard view of pipelines which map to the example shown in the concept. Clicking on the label (marked in red boxes) of any of the pipelines shows the VSM from the perspective of that pipeline:
Clicking on the label of the "Production" pipeline, takes you to its Value Stream Map, in which all of its upstream pipelines and their source control materials are shown.
Clicking on the label of the "Build" pipeline, either from the dashboard or the VSM view above, takes you to a similar view with the perspective of the "Build" pipeline instance you clicked on. It is shown below. You can see that the source material of "CapacityTesting" pipeline is not shown, because it is not directly downstream of the "Build" pipeline.
Now, that we know about the VSM, let's delve a little deeper into powerful concepts called fan-in and fan-out, which the VSM visualizes.
Let's start with the basic concepts and then use an example to understand this further.
A pipeline "X" can be said to fan-out to downstream pipelines, when there are multiple downstream pipelines for which pipeline "X" is a dependency. It looks like this:
Similarly, a source material can be said to fan-out to downstream pipelines:
This brings in the advantages of parallelization and quicker feedback, which is a key goal of Continuous Delivery. As long as there are enough agents available, all three downstream dependencies will trigger together and can run together.
Opposite of fan-out. When multiple dependencies are used by a single pipeline, then the dependencies are said to fan-in to the pipeline. It looks like this:
In this example, there is a fan-in at the Production pipeline, from its two upstream pipelines and the git material. Similarly, there is a fan-in at the CapacityTesting pipeline. The important aspect of fan-in is not that there are multiple dependencies upstream, but the way that the pipeline is triggered, by ensuring that all of its dependencies are consistent in version, when the pipeline runs. This ensures correctness of the build, and quicker feedback through elimination of spurious builds.
Let's bring back an earlier VSM view, from the Production pipeline perspective, with the links leading into the Production pipeline emphasized a bit, by pinning it:
In this setup, the Build pipeline is said to fan-out to the UserAcceptance and CapacityTesting pipelines, and the git material, UserAcceptance pipeline, and CapacityTesting pipeline are said to fan-in to the Production pipeline. From a terminology perspective, that might make sense. But what does it really mean? Why is it important?
To understand that, let's consider what should happen when a commit (say C1) is made to the git material shown. Even though the Production pipeline has a direct dependency on the git material, it should not be triggered automatically. The correct order of triggering of pipelines is:
This ordering and consistency of versioning is what GoCD with fan-out and fan-in will ensure. There are two main reasons for this to be considered very important:
Correctness: Without fan-in, if the CapacityTesting pipeline was slow and UserAcceptance pipeline was fast, then the Production pipeline could have been triggered with an old version of CapacityTesting pipeline and a new version of UserAcceptance pipeline. Imagine deploying to Production without really knowing that all upstream dependent pipelines have not finished successfully!
A very important part of fan-out and fan-in is that a downstream pipeline will always figure out the correct upstream dependency versions, including repository materials, unless it is manually forced by a user not to.
Similarly, any artifacts fetched from any upstream pipeline, however many levels up, is guaranteed to be consistent and correct. This means that the right revision of those artifacts will be fetched, irrespective of any other runs of that pipeline that may have run after.
This has been a bit of a whirlwind tour of some of GoCD's concepts and ideas. To learn more about all the other features and how to use them, head on over to the GoCD documentation page.