The Strategic Importance of Continuous Delivery

Abstract

For many software development teams, the first things that come to mind regarding Continuous Delivery (CD) are its operational aspects and competitive benefits. For us, it was much more: it was a survival technique. This article presents our experience applying CD in a Brazilian Government project for the development of a Collaborative Development Environment (CDE), sharing its unconventional challenges and the strategies used to overcome them. This report from the trenches of the Brazilian Federal Government can help practitioners to understand how important CD adoption can be to their projects.

Introduction and Context

We worked on a three-year-long Brazilian government project to evolve an existing platform that had technical issues and lacked political support. The evolution project started in a presidential election year and everyone involved were under pressure to show results. Even with the re-election of the Brazilian President in 2014, leaderships in governmental agencies end up changing. Each one of them had different political agendas which caused impacts on the requirements previously approved. Despite this scenario of instability, we managed to avoid delays in the delivery schedule and to prevent the abortion of the project.

Our team from the University of Brasília (UnB) and the University of São Paulo (USP) developed the new platform for the Brazilian Public Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br) from 2014 to 2016. The SPB Portal evolved to a Collaborative Development Environment [1] and this evolution brought about important benefits not just to the Brazilian government, but also to society as a whole. For the government, the bureaucracy on using the same software across governmental agencies, duplicate works, and costs all were reduced. The society gained a transparency and collaboration mechanism, since anyone can check the government expenses on software and contribute to the software communities. To achieve these goals, rather than writing everything from scratch, we decided to integrate several free software tools such as Gitlab (www.gitlab.com), Mailman (www.gnu.org/software/mailman), Noosfero (www.noosfero.org), and Colab (www.github.com/colab).

During the entire SPB Portal evolution project, we had to handle three distinct issues, usual in a software engineering scenario: reaching the goals which have guided the platform development, managing the diversity of team project members, and communicating effectively with clients (in this particular case, government agents). Managing the interaction of these elements was not easy and the unstable Brazilian political scenario only made things worse.

To achieve the SPB project goals, we had to overcome strong political bias tied with complicated technical issues as well as relatively low budget. Because it is open to the public, the government representatives saw the platform as a marketing opportunity and sometimes ignored the technical advice in favor of political decisions. Furthermore, integrating a number of distinct systems to work seamlessly was not an easy job. We had to learn how each system worked and to come up with ideas of how to integrate them as fast as possible, with a team of mostly inexperienced developers.

We also had to manage the diversity of the SPB team members. This team was composed of approximately 50 undergraduate students (not all simultaneously) together with professors, masters students and professional designers as well as senior developers from the Brazilian free software community. Undergraduate students received a fellowship and, for most of them, this R&D project was their first professional experience. Seniors developers and masters students had two important contributions to the project: transfer knowledge to undergraduate students and address hard tasks. Finally, professors were responsible for interacting with the Brazilian government and controlling the political pressures applied to the project.

The third point to be handled was the communication with two independent groups of government representatives: requirements analysts and deployment technicians. Requirements analysts were the real representatives of the Brazilian government in the project and their function was to test new features, to provide feedback to the development team, and to report for the government leaders. Deployment technicians were in charge of accessing the host machines wherein SPB platform was running. They were, theoretically, responsible for deploying the project. However, the new SPB Portal was composed of more than ten different software projects, generating a complex deployment process, for instance, working on seven servers.

To handle the interaction between these three elements, we realised we needed to take control over the deployment process. We used CD as a mean to fulfill the government expectations and to provide quick response to their requests, which were influenced most of the time by the uncertainties of project continuity. We believed we would keep the project alive, even in a politically unstable and technically complex scenario. For this reason, we worked hard to automate all deploy process; for instance, one of our senior developers created a Chef-Server (www.chef.io/chef) front-end tool called Chake (www.gitlab.com/terceiro/chake) to help us to manage the multiple hosts needed for the project. We also formed a specific team dedicated to the deployment process. This team was responsible for maturing our CD pipeline, giving us confidence and agility to comply with government requirements.

In this report, we describe our CD pipeline and how it improved our delivery performance. The CD made us adaptable for all requested changes and helped us to mitigate those aforementioned political challenges, besides the technical issues. Among CD’s known benefits [2], we also explain those which were the most important in the project scenario for us: improving customer satisfaction and making reliable releases. Both kept the project alive for more two years during the worst political crisis after the re-democratization in Brazil.

Our Continuous Delivery Pipeline

The figure represents our CD pipeline: it starts with new code arriving which is tested and improved during as necessary as it goes through each one of the steps until finally reaches the production environment. At this point we get back to the first stage to release more new code.

Automated tests

Each one of the Portal's software components had their own test suite. The development teams were required to maintain and expand them if possible. The communication of all these software was based on a plugin architecture. Testing the plugins were our integration tests. Finally, all tests ran in each component continuous integration environment.

This two-level testing provided us with the necessary speed and trust that guaranteed both component and platform stability. If any error was found, the pipeline stopped and the developers were notified. Only after all tests passed we moved to preparing the release.

Preparing a new release

Our release process was divided in two perspectives in terms of Git tags: the application and the SPB Portal. The application tag refers to the specific feature or bug fix and is a monotonically increasing. A new tag on any system yielded a new SPB Portal tag.

When all tests passed for a given component, we manually created a new application tag for it. As a consequence, that automatically created a new tag for the SPB Portal. Notice that we forked of the original software projects and, as consequence, we had different tag values.

Packaging

The platform is running on the CentOS 7 GNU/Linux distribution. Basically, packaging a software for that distribution has three steps: write the script for the specific environment (RPM), build the package, and upload it to a package repository.

We chose to create our own packages for each software component for several reasons:

• Not all software was packaged by the community and those that existed were outdated;
• Packaging makes it easy to manage the software on a given distribution;
• It simplifies the deployment;
• Packaging follows the distribution’s best practices and,
• Allows configurations and permissions control.

After creating a new tag for one component, the DevOps team was notified and the packaging process began. In the normal case, the three packaging steps aforementioned were fully automated by a set of scripts. With all these scripts running successfully, the new packages would be ready to use by our subsequent deployment scripts.

Validation Environment Deployment

The Validation Environment (VE) is a replica of the Production Environment (PE), with two exceptions: only the government officers and us had access to it as well as all the data is anonymised. To configure the environment, we used our configuration management tool: Chake (serverless configuration with Chef). That maintained environment consistency simplifying the deployment process. Additionally, the packages we built on the last step were readily available to use by the management tool.

The VE was used by the government agents to validate new features and required changes. Also, the VE was useful to verify the integrity of the entire portal as part of the next step in the pipeline.

Acceptance Tests

After we completely deploy, a new SPB Portal version in the VE, the government agents are responsible for checking features and bug fixes required by them. If the technicians identify a problem, they notified the developers via comments on an git issue related to the user story (features) already registered in our Gitlab at the SPB Portal. These problems were fixed and the pipeline restarted from scratch. If everything is validated, we moved forward.

Production Environment Deployment

After the government finished the VE check, it was cleared for deployment and we could finally began the deployment to Production Environment (PE). For this we also used our configuration management tool as in the VE as well with same scripts and package versions. After the deploy was completed, both VE and PE were running identical software. This was the point where new features and bug fixes were finally available to the end users.

Benefits

Research points out many advantages of CD usage in industry, such as: accelerated time to market, building the right product, productivity and efficiency improvements, reliable releases and better customer satisfaction. Working with the government, we noticed the following additional benefits.

Response to mistrust

The direct benefit from the CD pipeline was the fast response to the changes required by the government. That was vital for the project’s renewal over the years. We could manage the tension between the government and the development team better. Every meeting with the government leader was delicate and resulted on many new requirements, most of them motivated by political needs. For example, once it was demanded a completely layout change because another government leader suddenly decided to make a marketing campaign about the new SPB portal. They would use undelivered requirements as a means to justify the lack of financial resource already planned. We believed that if we took too long to attend their demands, the project would end. CD helped us to move fast on deploying to production, even of smaller parts of the requirements. That way, we always had something to show on the meetings, reducing their eagerness to end the project. For our team, it made the developers more confident the project would last a little longer and they would not go looking for another jobs.

Build client’s trust

After we established the CD, the government agents started to be more confident in our work. First, because they noticed that each new deploy made by us in the VE was stable and reliable. Second, they could see new features fast since we constantly updated the VE based on their feedback. This made our relation strong and in moments that needed quick action they would rather give us access to production.

Shared Responsibility

When the government technicians were responsible for deploying the project, the developers lost track of what happened after code was delivered. After adopting CD, they felt more responsible for what was getting into production. CD influenced developers on taking ownership of the project. In the end of the project, we noticed that the entire team was working to improve the CD pipeline since they want to their new features in production.

Interestingly, the CD pipeline also made the government requirements analysts feel more responsible for the project. They were an active part of the pipeline and that engaged them on the whole process. In the end, they were even actively creating issues and discussing them during the development process.

CD pipeline protocol between Government and Development

In the beginning of the CD pipeline use, a bottleneck arose at the acceptance tests step due to delays in reviewing new features and starting the next step. These delays occured because the government analysts, responsible for reviewing, were sometimes busy or didn't have schedule this work. Furthermore, after the acceptance of the new code, there was a bureaucracy in the government IT infrastructure that often made us to wait until 3 days to get the production environment access and then to start the deployment step. This problem was softened when we clarify our pipeline for these analysts and they organized their schedule to speed up reviews and production access requests.

Work in small batches developed trust in our relation with government analysts

In the first three releases, government requirements analysts were validating all features to be released only at the end of a delivery cycle which often took almost four months. However their superintendents requested monthly reports about the project progress and this brought pressure on them which in turn put it on us. Making continuous deliveries, we really delivered intermediate and candidate releases. Thus, the analysts could have validate a small set of features, making possible more accurate feedbacks and better reports for regular meetings with their superintendents. As a result, we gained their trust and they also gained trust of their chiefs about the SPB project management.

Challenges

We successfully built a CD pipeline. In the end, we took over the deployment process from the government. That allowed us to survive into an unstable political scenario. However, we recognize that many challenges still need to be addressed by the industry and academia together.

Build CD from scratch

Taking on CD responsibilities had a significant impact on the team. We did not have the know-how and had little time to come up with a working pipeline. The senior developers were crucial at this point. They came up with an initial solution to get us started. That already enabled us to automatize the deploy, even though the process was still rudimentary. We had to evolve our solution on-the-fly. We dedicated a few developers to this task.

Building a CD pipeline was hard in the beginning. We believe that more tools that provide out-of-the-box standardized CD pipelines would be of great help for inexperienced teams. Tools that track each step of the pipeline and organize logs in a human-manageable way are necessary too.

Handling inexperienced teams

After the developers learned how CD worked, it was difficult to pass the knowledge along to other teammates. We tried to mitigate this by encouraging a member's migration to the DevOps team. Further research on how to effectively spread knowledge across inexperienced developers in a high turnover scenario are needed.

Overcoming mistrust

In the project’s first half we struggled with deploy related problems in the government structure. We were in a paradoxical situation. The government demanded speedy deliveries but would not give access to their production infrastructure. Afterwards some interactions with government agents, they created the VE as an isolated replica of the PE in their own infrastructure. The government agents then realised that it could be beneficial for the project if they granted us access to part of the structure. We believe it is required more research on development protocols and policies to improve the relation between industry and government, specially regarding CD.

References

1. G. Booch, A. W. Brown, "Collaborative Development Environments", in Advances in Computers, vol. 59, 2003, pp. 1–27.
2. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54.
3. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", in Proceedings of the 13th International Symposium on Open Collaboration, 2017.