Continuous Delivery: the strategy to overcome challenges in Government IT

Abstract

For many software development teams, the first things that come to mind regarding Continuous Delivery (CD) are the operational aspects and the competitive benefits. In our experience, it was much more: it was a survival technique. This article presents how we applied CD in a Brazilian Government project for the development of a Collaborative Development Environment (CDE), sharing its unconventional challenges and the strategies used to overcome them. This report from the trench of the Brazilian Federal Government can help practitioners to understand how important CD adoption is to their projects.

Introduction and Context

We worked on a three-year-long Brazilian government project to evolve an existing platform that had technical issues and lacked political support. Our team from the University of Brasília (UnB) and the University of São Paulo (USP) developed the new platform for the Brazilian Public Software (SPB, Portuguese acronym) Portal from 2014 to 2016. The SPB Portal (www.softwarepublico.gov.br) evolved to a Collaborative Development Environment [1] and this evolution brought important benefits not just to the Brazilian government, but also to society as a whole. For the government, the bureaucracy of using the same software across governmental agencies and the cost of developing similar software projects were all reduced. The society gained a transparent and collaborative mechanism, since anyone can check the government expenses on software and contribute to project communities. To achieve these goals, rather than writing everything from scratch, we decided to integrate several free software tools such as Noosfero (www.noosfero.org), Gitlab (www.gitlab.com), Mailman (www.gnu.org/software/mailman), Mezuro (www.mezuro.org), and Colab (www.github.com/colab).

The project started in a presidential election year and everyone involved was under pressure to show results. Even with the re-election of the Brazilian President in 2014, leaderships in governmental agencies ended up changing. Each one of them had different political agendas which affected the project's requirements previously approved. Besides that scenario of instability, we overcame three distinct issues: (i) achieving the goals which have guided the platform development, (ii) managing the diversity of team members, and (iii) communicating effectively with government agents (our "clients"). Handling the interaction of these elements was challenging and the unstable Brazilian political scenario only made things worse.

To achieve the SPB project goals, we had to overcome strong political bias tied with complicated technical issues and relatively low budget. Because the project is open to the public, the government representatives saw the platform as a marketing opportunity and sometimes ignored technical advice in favor of political decisions. Furthermore, integrating a number of distinct systems (such as social and collaboration network, Git repository manager, mailing list, source code metric evaluation, and a systems integration platform) to work seamlessly was not an easy job. We had to learn how each system worked and to come up with ideas of how to integrate them as fast as possible, with a team of mostly inexperienced developers.

We also had to manage the diversity of our team members. The team was composed of approximately 50 undergraduate students (not all simultaneously) together with professors, master students, professional designers and senior developers from the Brazilian free software community. Undergraduate students received a scholarship and, for most of them, this R&D project was their first professional experience. Senior developers and master students had two important contributions to the project: transfer knowledge to undergraduate students and more complex tasks. Finally, professors were responsible for interacting with the Brazilian government and controlling political pressures applied to the project.

Moreover, we needed to communicate with two independent groups of government representatives. Requirement analysts were real representatives of the Brazilian government in the project and their role was to test new features, to provide feedback to the development team, and to report to government leaders. Deployment technicians were in charge of managing the host machines wherein SPB platform was running. They were, theoretically, responsible for deploying the project. However, the new SPB Portal was composed of more than ten different software projects, working on seven servers, generating a complex deployment process.

To handle the interaction between these three elements, we realised we needed to take control over the deployment process. We used CD as a mean to fulfill the government expectations and to provide quick response to their requests, which were influenced most of the time by the uncertainties of the project's continuity. We believed we would keep the project alive, even in a politically unstable and technically complex scenario. For this reason, we focused on automating automate the deploy process; for instance, one of our senior developers created a Chef-Server (www.chef.io/chef) front-end tool called Chake (www.gitlab.com/terceiro/chake) to help us manage the multiple hosts needed for the project. We also formed a specific team dedicated to the deployment process. This team was responsible for maturing our CD pipeline, giving us confidence and agility to comply with government requirements.

In this report, we describe our CD pipeline and how it improved our delivery performance. Continuous Delivery made us adaptable for all requested changes and helped us mitigate those aforementioned political challenges, besides the technical issues. Among CD’s known benefits [2] we also explain those which were the most important in our scenario: improving customer satisfaction and making reliable releases. Both kept the project alive for years during the worst political crisis after the re-democratization in Brazil.

Our Continuous Delivery Pipeline

Figure 1 represents our CD pipeline. The pipeline started when new code arrived. As it went through each step, it was tested and improved until it finally reached the production environment. At this point we got back to the first stage to release more new code.

Automated tests

The SPB portal consists of more than 10 integrated software projects and each of them, as well as the entire platform, had to be tested. These software components had their own test suite, maintained and expanded by the development team. Communication between all components is orchestrated by Colab, a systems integration platform for web applications based on a plugins architecture. Therefore, specific plugins have been developed for each portal software component, such as for Gitlab and Mailman. Each plugin had its own test suite and this set also worked as integration test.

Both unit and integration tests provided us the performace and security needed to guarantee the stability for components and the platform. If any test suite failed, by either a test error or coverage reduction below a certain threshold, the pipeline stopped. Only when all tests pass, the pipeline proceed to the step of preparing the release.

Preparing a new release

A SPB Portal release was composed of all its software components releases. Each software component release was a git tag that referred to a specific feature or bug fix. When all tests passed for a given component, we manually created a new tag for it. Therefore, a new tag on any software component yielded a new SPB Portal release. More precisely, SPB had a script that produced a single release for the entire system based on each component tag. At the end of this process, we started packaging.

Packaging

The platform is running on the CentOS 7 GNU/Linux distribution. Basically, packaging a software for that distribution has three steps: write the script for the specific environment (RPM), build the package, and upload it to a package repository.

We decided to create our own packages for each software component for the following five reasons:

• Not all software was packaged by the community and those that existed were outdated;
• Packaging makes it easy to manage the software on a given distribution;
• Packaging simplifies the deployment;
• Packaging follows the distribution's best practices and,
• Packaging allows configurations and permissions control.

After creating a new tag for one component, the developers informed the DevOps [3] team and the packaging process began. The three packaging steps aforementioned were fully automated by a set of scripts. With all these scripts running successfully, the new packages would be ready to be used by our subsequent deployment scripts.

Validation Environment Deployment

The Validation Environment (VE) is a replica of the Production Environment (PE), with two exceptions: only the government officers and project leaders had access to it and all the data was anonymised. To configure the environment, we used a configuration management tool named Chef with Chake support (serverless configuration for Chef). That tool maintained environment consistency simplifying the deployment process. Additionally, the packages we built on the last step were readily available to be used by the management tool.

Government agents used the VE to validate new features and required changes. Also, the VE was useful to verify the integrity of the entire portal as part of the next step in the pipeline.

Acceptance Tests

After we deployed a new SPB Portal version in the VE, the government agents were responsible for checking features and bug fixes required by them. If the requirement analysts identified a problem, they notified the developers via comments on the SPB Portal's issue tracker. The problems were fixed and the pipeline restarted from scratch. If everything was validated, we moved forward.

Production Environment Deployment

After the government finished the VE check and it was cleared for deployment, we could finally begin the deployment to the Production Environment (PE). For this we also used our configuration management tool, the same scripts and package versions as in the VE. After the deploy was completed, both VE and PE were running identical software. This was the point where new features and bug fixes were finally available to the end users.

Benefits

Research points out many advantages of CD usage in industry[2], such as: accelerated time to market, building the right product, productivity and efficiency improvements, reliable releases and better customer satisfaction. Working with the government, we noticed the following additional benefits.

Responsiveness to change

Responsiveness was one of the direct benefits of adopting the CD pipeline. The ability to react quickly to changes requested by the government was vital for the renewal of the project over the years. Every meeting with the government leader resulted in new requirements, most of them motivated by political needs. These constant changes in requirements and priorities caused discomfort between the government and the development team. For example, once it was demanded a complete layout change because another government leader suddenly decided to make a marketing campaign about the new SPB portal. They would use undelivered requirements as a means to justify the lack of financial support, which was already approved in the first place. We believed that if we took too long to attend their demands, the project would end. CD helped us keep the production environment up-to-date, even with partial versions of a feature. That way, we always had something to show on meetings, reducing anxiety to get the platform concluded. For our team, it made the developers more confident that the project would last a little longer and they would not go looking for other jobs.

Shared responsibility

Before the adoption of CD, the development team could not track what happened to the code after its delivery, since government technicians were the only responsible for deploying the project. The implementation of the referred approach influenced developers on taking ownership of the project because it made them feel equally responsible for what was getting into production.

Interestingly, the CD pipeline had the same effect on the team of requirement analysts. They were an active part of the pipeline and became more engaged on the whole process. After the incorporation of the pipeline into the work process, analysts became more active in opening and discussing issues during the platform evolution. Additionally, developers worked to improve the CD pipeline to speed up the process of making available, in the production environment, new features for the platform.

Synchronicity between government and development

Despite the positive impacts that the CD pipeline brought to the project, its implementation was not easy at first. The CD pipeline performance depended on the synchronicity between developers and government analysts, so that the latter were prepared to start a step as soon as the former concluded the previous step, and vice versa. Initially, this concern was not contemplated in the agenda of the governmental team, which generated delays in the validation of new features. This situation combined with governmental bureaucracy (up to 3 days) to release access to the production environment resulted in additional delays for the deployment step to begin. This problem was softened when the analysts realized the impact of these delays on the final product and decided to allocate the revisions in its work schedule and to request the access to production in time.

Strengthening trust in work relationship with the government

Continuous delivery was also a tool that helped to strengthen trust in the relationship between developers and government analysts, as well as between the analysts group and its superiors. Before using CD, analysts had access to the features developed only at the end of the release, usually every four months. However, this periodicity did not meet the requirements of their leaders, who demanded monthly reports on the progress of the project.

With the implementation of CD, intermediate and candidate versions became available, allowing analysts to perform small validations over time. As they validated functionalities and sent feedback to developers, patches were developed and new versions were packaged and deployed to the VE quickly, steadily, and reliably. The constant monitoring of the development work brought greater security to the governmental nucleus and improved the interactions with our development team.

Challenges

We successfully built a CD pipeline. In the end, we took over the deployment process from the government. That allowed us to survive into an unstable political scenario. However, we recognize that many challenges still need to be addressed by the industry and academia together.

Build CD from scratch

Taking on CD responsibilities had a significant impact on the team. We did not have the know-how and had little time to come up with a working pipeline. The senior developers were crucial at this point. They came up with an initial solution to get the team started. That already enabled us to automatize deployment, even though the process was still rudimentary. We had to evolve our solution on-the-fly. We dedicated a few developers to this task.

Building a CD pipeline was hard in the beginning. More tools that provide out-of-the-box standardized CD pipelines would be of great help for inexperienced teams. Tools that track each step of the pipeline and organize logs in a human-manageable way are necessary too.

Handling inexperienced teams

After the developers learned how CD worked, it was difficult to pass the knowledge along to other teammates. We tried to mitigate this problem by encouraging members to migrate to the DevOps team. We suggest further research on how to effectively spread knowledge across inexperienced developers in a high turnover scenario.

Overcoming mistrust

In the project's beginning we struggled with deployment issues in the government structure. We were in a paradoxical situation. The government demanded fast deliveries but would not give access to their production infrastructure. After some interactions with government agents, they created the VE as an isolated replica of the PE in their own infrastructure. The government agents then realised that it could be beneficial for the project if they granted us access to part of the structure. More research is required on development protocols and policies to improve the relation between industry and government, specially regarding CD.

References

1. G. Booch, A. W. Brown, "Collaborative Development Environments", in Advances in Computers, vol. 59, 2003, pp. 1–27.
2. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54.
3. Davis, Jennifer and Daniels, Katherine, Effective DevOps: building a culture of collaboration, affinity, and tooling at scale, 2016, " O'Reilly Media, Inc."