DevOps Series — Continuous Integration Systems (Y.2020)

For any aspiring DevOps engineers, one of the must-know topics is continuous integration. It is a well-discussed topic and there’re a plethora of resources available on the Internet (a simple Google search “continuous integration tools” results in more than 242,000,000 results returned). Unlike most posts opt to dive into feature comparisons, in this post, I’ll draw from my own experiences of using some of the integration tools over the years and assesses each one.

Disclaimer: the assessment below is my 2 cents and not in any way comprehensive or “correct”, and it is highly opinionated.

Market analysis

A snippet of available continuous integration tools available on the market as of writing:

• Jenkins
• Travis
• CircleCI
• TeamCity
• GoCD
• Concourse CI
• Bamboo
• Spinnaker
• GitLab CI
• etc.

Consideration Criteria

There are so many continuous integration tools out there that can easily overwhelm newcomers and it is hard for experienced DevOps Engineers to keep track of them all as well. Each tool features a set of capabilities and they all seem useful at first glance, but soon it becomes apparent that the “marketing brochure” is not useful in any way and sometimes can deviate the users from making decisions on what truly matters. So instead of looking at what existing tools can do, why not take a step back and ask: what do we want the build system to do so we can have a good continuous integration experience?

Here are the requirements I think matters in terms of importance:

• *** Baseline: Build/compile source code into artifacts and deliver it to storage continuously
• *** Performance: How fast can the build environment be created to run the actual build, and how fast can each build finish (the second requirement highly depends on how good the pipeline is written, in here I’m focused on the systematic factors)
• *** Scalability: How good is the system to support tens of thousands or even millions jobs concurrently
• *** Pipeline as Code: Can the pipeline configuration be stored externally as code so the pipeline can be recreated idempotently and how easy is it to understand and write the configuration code. Is pipeline treated as a first-class citizen or an afterthought
• ** Programmability: this one is more important to CI system administrators for controllability and administrative overhead
• * Deployment capability: Can the CI system be used for deployment as well
• * Visualization: Does the CI system provides a way to visualize different build stages

I’ll discuss what constitutes a good continuous integration pipeline in a future post.

User Experiences

I’ve used Jenkins, Concourse CI, Travis CI, Bamboo, TeamCity, GoCD and Spinnaker over the past 3 years, below is a breakdown from my perspective (some tools I used so it might be changed/improved during that time)

Jenkins:

Jenkins is probably the most popular and almost synonymous with Continuous Integration. It has a large developer community, and plenty of documentation and questions answered all over the Internet. If you encounter an issue, it is almost guaranteed someone already had the same issue and highly likely there’s already a solution (either fix or known bug) verified.

Jenkins works as a plugin system, itself provides the core platform features such as managing builds, artifacts lifecycle and scheduling, while “outsource” other capabilities to plugins, currently, there are five categories of plugins can be found in Jenkins Plugins Index, Platforms, User Interface, Administration, Source code management and Build management. Jenkins supports various build environments and has a wide range of platform support. In recent years, Jenkins also released Jenkins X to adapt to the cloud-native trend.

Albeit its popularity, adopting Jenkins is far from a no-brainer. Some aspects of the Jenkins ecosystem to consider:

• Stability: Its plugin system is unstable and vetting standard varies. I’ve gone through multiple times once multiple plugins are updated to the latest version, the system crashes and I was deep into the weeds to find out the problematic plugin and in turn revert the system to a stable state. The confidence to upgrade degraded over time.
• Size: Memory consumption is pretty large, thus maintenance cost.
• Configuration as Code: In older versions of Jenkins build code is written in UI and if you have multiple similar or repeating jobs then it becomes a copy/paste/small modification exercise quickly. In Jenkins 2 it finally introduced pipeline as code which uses Groovy as pipeline language.
• Visualization: For normal job stages, parallelism is not properly displayed, although it has a bit better pipeline flow visualization in Blue Ocean.

Concourse CI:

Coming from Pivotal Cloud Foundry, Concourse CI is a newcomer to the CI space. It treats pipeline as the first-class citizen and this philosophy is baked into the product design from the very beginning. Concourse uses YAML manifest to define everything in the pipeline, from how artifacts flow through the pipeline to how the visualization looks like, and its visualization is very appealing. However, it does have a pretty steep learning curve at first and once you understand the principle behind why the developers design the software in this way it’s pretty easy to tailor it to your liking. Similar to Jenkins, it uses “resources” to extend functionalities and you can write your own “resource type” relatively easily. It is primarily used in Cloud Foundry ecosystem and I haven’t seen wide adoption in other scenarios.

Travis CI:

Travis CI focuses on Continuous Integration capabilities, thus it doesn’t provide native deployment features. However, it can be customized to perform deployment though I’d recommend not due to the hoops to jump through and maintenance cost.

It gains popularity due to its native integration with GitHub at first and over the years, slowly moving into Enterprise environments. It’s probably the most popular CI tool used in open source projects hosted in GitHub. It does provide the ability to configure the pipeline based on a “.travis.yml” file under the root directory of the project repository, though it can improve on how to integrate/manage simple scripts so the configuration file doesn’t “bloat”.

GoCD:

End to end pipeline visualization is a strong suit of GoCD, however, the configuration and administration overhead are not easy for newbies.

TeamCity:

TC is yet another CI system that didn’t leave too much impression on me.

Bamboo:

Yet another CI system with normal features, since it comes from Atlassian, Bamboo has native integration with other Atlassian products such as JIRA, Confluence, BitBucket and so on.

Spinnaker:

Spinnaker comes from Netflix and features on cloud deployment, I barely used it and don’t have a solid opinion. Maybe I’ll write another post once I get more hands-on experience with it.

Conclusion

What continuous integration system to use in your project highly depends on the specific situation and requirements, both functional and non-functional. That means even though Jenkins is the predominant/de-facto tool on the market today, it might not be the tool for you. Each tool has its strengths and its quirks, I encourage everyone to do a bit prototyping and get a sense of hands-on experience for a more informed decision.