Of all the buzzwords in vogue these days, the CI/CD pipeline seems to always be at the forefront. As engineering stakeholders, indeed to many engineers themselves, the term is sometimes amorphous and seems to mean different things to different people at different times. In this article, I’ll attempt to frame this and some related terms and ideas in relevant contexts while trying to pin down their meaning and importance to non-technical stakeholders.
The Mystery of DevOps
We can’t really understand the CI/CD pipeline without also understanding the overarching philosophy and practice that is DevOps. Short for Development Operations, DevOps is the combining of software engineering and IT operations into a single discipline rooted in the philosophy of Agile software development. It is important to remember that DevOps is not just a practice, it is also a philosophy! You don’t just do philosophies, you live them, and as a decision-making stakeholder for your organization, if you are going to say you embrace DevOps, then you must be the one to help enable it in engineering, product, systems, and executive leadership teams (ELTs).
At its base, DevOps as a practice is simply the aligning of software engineering and systems/IT teams into a cohesive functional unit. It relies heavily on an infrastructure-as-code paradigm enabled by modern cloud services (AWS, Azure, etc.) and uses tools like automation at multiple levels to enable fast development, testing, deployment, scalability, security, and recovery when necessary.
DevOps may-or-may-not mean these roles are on the same team but it does mean that there is a significant overlap of communication and responsibility when it comes to developing and shipping a product or feature. Each role in a DevOps organization is no longer siloed like software development used to be a decade ago. From start to finish, a DevOps-enabled team has engineering and systems people in continual cyclical collaboration to get work done.
How does CI/CD Fit In?
CI/CD stands for continuous integration and continuous development. These often go together but they don’t have to. There are many situations when organizations may have one or the other and that’s perfectly fine. In my opinion, the biggest limiting factor in full CI/CD integration is the amount of initial setup required to get an organization’s codebase, infrastructure, and hearts & minds ready for complete integration.
Continuous integration is where engineers quickly, easily, and frequently integrate their development code into shared branches. In practice, this is largely an exercise in automation. Relying heavily on automated tests and repository/pipeline hooks, CI removes most human blockers and enables engineering teams to move quickly through development cycles.
From a stakeholder perspective, there’s not much that needs to be done or understood here other than simply respecting the process as this will largely be out of sight and out of mind for most people outside of engineering. While most CI work is around automation, the one big manual step that can not be automated is code reviews. As a stakeholder, you should always be encouraging Sr Engineers and Engineering Managers to reinforce high code review standards at all phases of development. This not only helps ensure a quality product but also fosters team growth, cohesion, and engineer investment.
Continuous deployment is the continual and engineer-driven process of releasing code to staging or production environments without relying on manager approvals. The central idea is that if automation can help ensure quality, then features and fixes can and should be released continually as soon as they are ready and not held back for less frequent and subsequently larger releases.
As with CI, this process is heavily dependent on testing and automated processes to ensure quality releases. Heavy automation also enables releases to be done by any engineer in the organization, including Jr Engineers, who might otherwise never kickoff a production release. The main benefits of a CD process are that releases are small and thus generally less error-prone, they happen more frequently so development feels faster, and they are easier to roll back should a bug get introduced.
With CD, the big things that stakeholders have to manage are letting go and trusting the process. It can be hard for stakeholders and especially ELTs in smaller organizations to not maintain their grip on release cycles and customer expectations. Of course, some things like large features or revamps are things that typically get released with customer forewarning and on a schedule but for everything else, stakeholders need to let go of the notion of a “release” and understand that the vast majority of code could be continuously deployed at any time and this is generally a good thing.
Typical Barriers to Introducing CI/CD
As a stakeholder interested in fostering a DevOps mentality and a CI/CD pipeline in your organization, these are some of the major barriers to going fully into the process. Keep in mind that these assume you already have a product out in the wild.
This is the most fundamental prerequisite for a CI/CD pipeline and is by far the most challenging to implement if your codebase does not already have test coverage. While there are many types of tests, unit and end-to-end are generally the most important here (optionally, visual diffs depending on your application and resources). Unit tests cover smaller functional units of code like functions or components while end-to-end tests cover entire workflows and user paths through your system.
If you already have significant test coverage, you’re very ahead of the game. Your focus here should be continuing to work with your engineering managers and team leads to ensure as much test coverage as possible makes its way into the code repos. This is an ongoing process that requires time allocated to stories for engineers to write the tests and for QA engineers to validate and run them (if that’s your life cycle flow…otherwise, the engineers handle it all). One hundred percent test coverage is ideal but it’s also rarely practical. I would argue that anything above 90% is sufficient.
If, however, your codebase does not have test coverage then you and your engineers will need to formulate a plan to implement coverage. In general, I would recommend a two-pronged approach:
- Build effort into story point estimates and include acceptance criteria that account for an increase in QA that covers the writing of unit and end-to-end tests.
- Keep test writing stories to cover existing features without test coverage in the “Ready for Dev” column of your kanban board or prioritized queue so that they can be picked up during lulls in development. There will likely be a massive backlog of features that need tests written for them and most of the time, you won’t be able to stop all active development to address them. So do them slowly, over time, but be proactive about it and prioritize them when appropriate.
I’ll be frank and say that a good number of software engineers don’t like writing tests. Tests are not particularly glamorous and can sometimes be very hard to write depending on the feature or application. Tests can sometimes feel like a waste of time as they take a good deal of effort that can’t necessarily be seen by anyone. There may also be generational and career path factors at play too. While I am generalizing a bit, in my experience, younger engineers don’t typically like writing unit tests, and getting them to do so as a habit can be a challenge.
Likewise, I have seen some differences in career path history as well. Self-taught and/or boot camp graduate engineers, in my experience, generally aren’t keen on test writing. I’m not downplaying their experience or education at all but when coming from these backgrounds (of which I am one; self-taught though later an MCS grad), the major impetus is gaining career knowledge through experience and practice and test writing is not typically part of that.
To help convince engineers to get serious about test writing, it can be helpful to frame the problem around issues of bugs and on-call support. Nobody likes dealing with bugs, engineers included. While this should be well known to most engineering teams, present them with the raw number of how many bugs come up in a particular development cycle and juxtapose that against the amount of time being spent on bug fixes. This is especially important if primary engineering teams are not the ones doing most bug fixes as is sometimes the case.
If your organization has on-call rotations, show them the on-call schedules and the raw metrics of how many times after-hours support is being called and how disruptive the calls can be. Engineers sometimes need to be convinced of the problem before they decide to take action. Commanding them to just do it isn’t an effective long-term strategy. Show the benefits of fewer after-hours interruptions, less bug-fixing time per sprint/cycle, and more confidence during deployments.
Testable Code Refactoring
Piggybacking off the previous two points, you can’t write good tests without the code being architected in a manner that lends itself to being tested. You generally get into this position by not adhering to like DRY, SOLID, and other mainstream software engineering best practices. This is definitely a potential pitfall and possibly a complete blocker. While end-to-end tests can still be written to cover specific features in this case, unit tests cannot. In some cases, this may be enough to move forward with CI/CD implementation but I would only recommend that for very small applications or applications with basic user flows and not many dependencies.
In my experience, I have typically seen this in applications written primarily by inexperienced engineers or by 3rd party agencies that don’t have much of a concern for long-term support (i.e they are being paid to deliver the app, not support it). If you find your application code in this state, your only real option is to embark on an incremental refactoring of your application under the guidance of an experienced software architect. This is a long-term project that involves incremental and intentional refactoring of aspects of your application. Ideally, you should start with the critical pieces first and work your way out. That way, you can get test coverage, and possible early CI/CD integration, over the most important pieces of your application sooner.
Keep in mind though, that in some rare case, you will find that if core application features are built in such a way that a refactor is not possible, you may be looking at the option of needing to completely rewrite your application into an entirely new version. As drastic as this sounds, you’d be surprised at how many applications may, for a variety of reasons, be in need of this kind of overhaul anyway. From a business perspective, this can sometimes be justified if there have been significant changes in the market, technology, or customer expectations since the launch of the application. Be open-minded about this possibility as it’s sometimes a more feasible approach.
A properly implemented CI/CD pipeline can bring significant advantages to any application’s development cycle. Quicker code integrations mean smaller, faster, and safer releases with fewer bugs and faster rollbacks if necessary. However, there are both practical and behavioral prerequisites that the code, engineers, and stakeholders must have in order for the transition to be successful. Code must be architected in a way that lends itself to being testable and those tests must be written properly to cover the use cases as necessary. Finally, there has to be buy-in from engineering and stakeholders to commit both time and effort into consistently writing proper unit and end-to-end tests. Once these prerequisites are in place, your organization can start reaping the benefits of a robust CI/CD pipeline.
Not sure how to get started?
Let our team of experienced professionals get you off the right foot. We can help with everything from initial consulting to refactoring plans to training to actual CI/CD pipeline building and integration. We have experience building and working within CI/CD workflows and can lend our years of experience to get your organization up to speed.