title: "Continuous Delivery: Building Trust in a Large-scale, complex government organization" papersize: a4
geometry: "left=1in,right=1.5in"
Authors
Rodrigo Siqueira is a masters student at IME - The Institute of Mathematics and Statistics of the Sao Paulo University. His research interests include software engineering, operating system and computer architecture. He worked for two years with the Brazilian federal government as a coach and developer. Contact him at siqueira@ime.usp.br.
Diego Camarinha is a masters student at IME - The Institute of Mathematics and Statistics of the Sao Paulo University. His research interests include software engineering, computer networks and source code metrics. He worked for two years with the Brazilian federal government as a senior developer. Contact him at diegoamc@ime.usp.br.
Melissa Wen is a software developer. She worked on SPB project as a senior developer and also served as professor of Computer Science at UFBA - The Federal University of Bahia. Her areas of interest include software engineering and open source software development. Contact her at melissa.srw@gmail.com .
Paulo Meirelles received his Ph.D. in Computer Science from the Institute of Mathematics and Statistics at the University of São Paulo. He is full-time Professor at the University of Brasilia, and coordinated the new SPB portal project. His research interest areas are: Free Software, Agile methods, Static analysis, and Source code metrics. Contact him at paulormm@ime.usp.br.
Fabio Kon is Full Professor of the Department of Computer Science of the University of São Paulo. (...) .His research interest areas are: (...). Contact him at fabio.kon@ime.usp.br.
Abstract
For many software development teams, the first aspects that come to mind regarding Continuous Delivery (CD) are the operational challenges and the competitive benefits. In our experience, CD was much more: it was a survival technique. This article presents how and why we applied CD in a Brazilian government project for the development of a Collaborative Development Environment (CDE), sharing the challenges we faced and the strategies used to overcome them.
Introduction
We worked on a thirty-month-long Brazilian government project to modernize the Brazilian Public Software (SPB) portal (www.softwarepublico.gov.br) [1]. This project, started in 2014, was a partnership between the Ministry of Planning, Budget, and Management and two public universities: University of Brasília and University of São Paulo.
With this partnership, the SPB portal evolved to a Collaborative Development Environment (CDE) [2] which brought significant benefits for the government and the society. The government could minimize bureaucracy and costs of software development, encouraging the use of the same set of applications across different government agencies. The society gained a mechanism of transparency, follow government expenses, and collaboration, contribute to project communities. .
In this article, we discuss the use of Continuous Delivery (CD) during our experience as the academic partner in this project. We focus on how we managed to implement CD in a large institution with traditional values and how CD helped to build trust between the government and the university development team. CD enabled us to show our progress and earned the government’s confidence that we could adequately fulfill their requests, becoming an essential aspect of our interaction with them. According to this experience, the use of CD as a tool to build such trust relationships is yet another of its benefits [3].
Context
SPB is a governmental program created to foster sharing and collaboration on Open Source Software (OSS) development for the Brazilian public administration. For their projects, the Ministry managed both software requirements and server infrastructure. However, its hierarchical and traditional processes made them unfamiliar with new software development techniques, such as CD. Any of our requests had to pass through layers of bureaucracy before being answered, accessing their infrastructure to perform a deploy was not different.
During its lifetime, the project suffered significant interference from the board of directors because the portal represents an interface between government and society. In light of political interests, directors continually imposed changes to the platform while ignoring our technical advice. In 2015, the board of directors was changed and, with it, the vision of the project. New directors had different political agendas which affected the project's requirements previously approved.
<!--- mudar em todas as ocorrências de agents por staff, mas checar com o Fabio. -->
In this context, we overcame three distinct challenges: (1) finding a system solution with which government and development team agree, (2) deconstructing the widespread belief among government staff that any project in partnership with a University is doomed to fail, and (3) dealing with bureaucracies involved in the deployment process by the Ministry.
To face the first issue, we designed the SPB portal as a CDE with additional social features. Due to the complexity of creating such a system from scratch, we decided to adapt and integrate existing OSS tools to build a system-of-systems [4]. We created a solution that orchestrates multiple components and allowed us to smoothly provide a unified interface for final users, including single sign-on and global searches [1]. On top of that, the new SPB portal was an unprecedented platform to the Brazilian government, with a complicated deployment process.
Regarding the second problem, our team was not from a typical company, consisting mainly of undergraduate students coordinated by two professors. In the first year, we had a group composed of 24 undergraduate students, one designer, and two senior developers. In 2015, our team grew to 36 students, two designers, eight senior developers. In the end, due to budget constraints, our team shrinked to 20 students, one designer, and two developers. On the government side, the SPB portal evolution was the first software development collaboration between university and government experienced by the Ministry staff involved in the project.
Finally, our team thought software deployment differently than the Ministry. On our side, we believe that frequent deliveries are better for the project’s success. However, the Ministry works with the idea of a single deployment at the end of the project. In other words, neither the bureaucratic structure of the Ministry nor its technical abilities were conducive to this style of work. Furthermore, there was little effort to deploy new versions of the system promptly. That ended up hampering the benefits of the tool and preventing us from showing off the fruits of the project to those responsible for evaluating it.
These challenges made our relationship with the Ministry staff tense, in particular during the first year, and alerted us to the fact that they could finish the project at any time. The deployment limitation was the substantial technical issue we could tackle in the short term. As a result, we worked to deploy one version of the project onto our infrastructure and showed it to the government evaluators. This strategy proved them we could efficiently deliver new features, fulfill their expectations regarding the delivery of the requirements, and incited them to demand that the latest version be deployed in the Ministry infrastructure. This generated more pressure on the IT department responsible for the deployment routines. With each CD cycle, we gradually built a new relationship among all parties and, by the end of the project, we became active participants in the deploy operations.
Our Continuous Delivery Pipeline
The SPB portal is a system-of-systems with five integrated software projects. Colab (www.github.com/colab), a systems integration platform for web applications based on a plugin architecture, orchestrates communication among them. For this, we had also to develop specific plugins for each portal software component.
<!--- Falta dados do Colab e a soma destes ao total ---> These software component have different levels of complexity and size. Colab has had X commits which represent Y lines of code, Gitlab, 64.446 commits and 502.597 lines, Noosfero, 14.175 commits and 607.490 lines, Mezuro, 9.007 commits and 217.828 lines, and, finally, Mailman with 19.101 commits and 180.871 lines. We can thus realize, with only the sum of the integrated projects, we have a platform that totals more than 106.729 commits and 1.508.786 lines of code.
Figure 1 presents our CD pipeline. It follows a typical deployment pipeline [3], adapted to the technical and organizational complexity of our project and the use of OSS best practices. The pipeline started when new code arrived. A new feature might require changes to more than one SPB integrated software project. Notice that each one of them could be modified independently. As the code went through each step, it was tested and improved until it finally reached the production environment. At this point, we would restart the pipeline to release more versions.
<!--- Comentário do Fábio: A partir daqui o texto já deveria mostrar o tamanho da plataforma e trazer dados que comprovem isso.</p> <p><a href="https://www.openhub.net/p/gitlab">https://www.openhub.net/p/gitlab</a> <a href="https://www.openhub.net/p/noosfero">https://www.openhub.net/p/noosfero</a> <a href="https://www.openhub.net/p/mezuro">https://www.openhub.net/p/mezuro</a> <a href="https://www.openhub.net/p/mailman">https://www.openhub.net/p/mailman</a></p> <p>-->
Automated Tests
As a software integration, the entire SPB platform, as well as each of its components need to be tested. Futhermore, the plugins developed to integrate a component has its own test suite, and this also work as integration tests.
Both unit and integration tests provided us the performance and security needed to guarantee the stability of components and the platform. If any test suite failed, by either a test error or coverage reduction below a certain threshold, the process stopped. Only when all tests passed, the pipeline proceeded to the step of release preparation.
Preparing a New Release
An SPB portal release was composed of all its software component releases. Each software component release had a Git tag that referred to a specific feature or bug fix. When all tests passed for a given component, we manually created a new tag for it. Therefore, a new tag on any software component yielded a new SPB portal release. More precisely, SPB had a script that produced a single release for the entire system based on each component tag. At the end of this process, we started packaging.
Packaging
The platform runs on the CentOS 7 GNU/Linux distribution. Packaging a software for that distribution involves three steps: writing the script for the specific environment (RPM), building the package, and uploading it to a package repository.
We decided to create separate packages for each software component since: Packaging makes easy to manage the software on a given distribution, simplifies the deployment, follows the distribution's best practices, and enables configurations and permissions control.
After creating a new tag for a component, the developers informed our DevOps [6] team, and the packaging process began. A set of scripts fully automated the three packaging steps aforementioned. When all them ran successfully, the new packages would be ready for our deployment scripts.
Validation Environment Deployment
The Validation Environment (VE) is a replica of the Production Environment (PE) with its data anonymised, as well as only Ministry staffs and our DevOps team had access to it. To configure the environment, we used a configuration management tool named Chef (www.chef.io) with Chake support (www.github.com/terceiro/chake) -- a serverless configuration tool created by our team. It maintained environment consistency simplifying the deployment process. Additionally, the packages we built on the last step were readily available to the management tool.
The Ministry staff used the VE to validate new features and required changes. The VE also was used to verify the integrity of the entire portal as part of the next step in the pipeline.
Acceptance Tests
After we deployed a new SPB portal version in the VE, the Ministry staffs were responsible for checking the features and bug fixes they required. If the Ministry staffs identified a problem, they would notify the developers via comments on the SPB portal's issue tracker. The development team fixed the problem and the pipeline restarted. If everything was validated, we moved forward.
Production Environment Deployment
When the Ministry staff finished the VE check, we could finally begin the deployment in production. We also used our configuration management tool, the same scripts and package versions as in the VE. After the deploy was completed, both VE and PE were identical. Here was the point where new features and bug fixes were finally available to end users.
Benefits
Research points out many CD advantages in the industry [2, 7], such as accelerated time to market, building the right product, productivity and efficiency improvements, stable releases, and better customer satisfaction. Working with the government, we noticed the following additional benefits.
Strengthening Trust in the Relationship with the Government
CD helped to strengthen trust in the relationship between developers and Ministry staffs. Before using CD, Ministry staff had access to the features developed only at the end of the release, usually every four months.
With the implementation of CD, intermediate and candidate versions became available, allowing Ministry staffs to perform small validations over time. Constant monitoring of the development work brought greater security to the Ministry leaders and improved the interactions with our development team.
Responsiveness to Change
Responsiveness was one of the direct benefits of adopting the CD pipeline. The ability to react quickly to changes requested by the Ministry staff was vital to the project’s survival for 30 months. Every meeting with the Ministry leaders resulted in requirements and priorities changes, several of them motivated by political needs. We observed that if we took too long to meet their demands, the Ministry would use undelivered requirements to justify cut in the financial support and cancel the project.
CD helped us keep the PE up-to-date, even with partial versions of a feature. That way, we always had something to show on meetings, reducing anxiety in getting the platform finished. For our team, it made the developers more confident that the project would last a little longer.
Shared Responsibility
According to the conventional Ministry process, the development team could not track what happened to the code after its delivery, since Ministry staff were the only ones responsible for deployment. The implementation of CD made our development team feel equally responsible for what was getting into production and take ownership of the project.
Interestingly, the CD pipeline had the same effect on the Ministry staff. They became more engaged in the whole process, opening and discussing issues during platform evolution. Additionally, developers worked to improve the CD pipeline to speed up the process of making new features available in the production environment for the Ministry staff's validation.
Synchronicity Between Government and Development
The CD pipeline performance depended on the synchronicity between our development team and the Ministry staffs so that the latter were prepared to start a step as soon as the former concluded the previous step and vice versa. Initially, the agenda of the Ministry staffs did not contemplate this concern, which generated delays in the validation of new features. This situation combined with governmental bureaucracy to release access to the production environment (up to 3 days) resulted in additional delays for the deployment step begin. This problem was softened when the Ministry staff realized the impact of these delays on the final product and decided to allocate the revisions in their work schedule.
<!--- Fabio sugeriu Lessons Learned, mas vamos mostrar exemplos da revista para ele olhar. -->
Lessons Learned
Due to the successful building of the CD pipeline, we improved the Ministry deployment process and kept the project alive. We map now lessons learned.
Build CD From Scratch
Taking on responsibilities for implementing CD impacted on the whole team. Mostly, our team members did not have know-how in this approach, and we had few working hours available to allocate for building the pipeline. The construction and maintenance of the CD process were possible by taking some decisions to mature the project:
<!--- pensar em generalizar/filosofar -->
_Select the most experienced senior developers and some advanced students of the project to work on a specific DevOps team._These senior developers used their experiences in OSS projects to craft an initial proposal for the deployment process. The solution enabled us to automate the deployment, even though the process was, initially, still rudimentary.
Interchange team members and encourage teammates to migrate to the DevOps team. The benefits of these movements were twofold: mitigating the difficulty to transmit the knowledge between DevOps developers and feature developers, and evolving the process on-the-fly.
Overcoming Mistrust
Taking an unfamiliar approach requires trust. In the Ministry, traditional software was the product delivered at the end of a development contract. They expected and were prepared to validate and deploy a single delivery. Because the SPB portal is a system-of-systems, the steady growth of its complexity made large deliveries unsustainable. The long time for homologation of developed features also gave the government room to change requirements and priorities. The CD approach was necessary, but how to build trust and gain autonomy to implement a process that was not yet part of the dynamics of the Ministry?
Demonstrate actual results, do not simply tell. Initially, we did not have access to the Ministry infrastructure, so we created our own validation environment. Thus, we were able to follow the CD pipeline until the stage of production deployment, when we faced two problems. First, our pace of intermediate deliveries to the government was faster than the deployment in production by the Ministry staff. Second, specific issues of the Ministry infrastructure made some validated features not work as expected in the PE. That situation gave us arguments to negotiate access to production.
Make project management transparent and collaborative for government staff. Allowing the Ministry staff to follow our process for version deliveries and bug fixes, we showed them we were fulfilling our commitments. They started to interact more actively in the generation of versions and became part of the process. After understanding the process, the Ministry staff helped us in negotiations with the Ministry leaders. Finally, they created a VE as an isolated replica of the PE and gave us access to it.
Gain the confidence of government staff. With the replica of the PE, we were able to run the entire pipeline and won the trust of the Ministry staff involved in the process. They saw the mobilization and responsiveness of our team to generate a new version package. They also recognized the quality of our packages and our deployment process. Finally, the Ministry staff then realized that it could be beneficial for the project if they granted us access to the infrastructure, both VE and PE.
<!--- Paulo: Acho que precisamos de algo ligado ao Ha-Ha-moment para fechar o texto aqui; ou fechar falando que tudo foi feito de forma aberta e colaborativa com as comunidades dos projetos envolvidos, estão todos os fontes disponíveis em <a href="https://softwarepublico.gov.br/gitlab/softwarepublico/">https://softwarepublico.gov.br/gitlab/softwarepublico/</a> -->
References
- P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", Proceedings of the 13th International Symposium on Open Collaboration. ACM, Article 16, 2017, 10 pages.
- G. Booch and A. Brown, A. W. "Collaborative Development Environments", Advances in Computers, vol. 59, 2003, pp. 1-27.
- L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, vol. 32, no. 2, 2015, pp. 50-54.
- C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, Article 18, 2015, 41 pages.
- J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley Professional, 2010.
- J. Davis and K. Daniels, "Effective Devops: Building a Culture of Collaboration, Affinity, and Tooling at Scale", O'Reilly Media, Inc., 2016.
- T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), Austin, TX, 2016, pp. 21-30.