diff --git a/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md b/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md index b83b0c2..3924c1c 100644 --- a/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md +++ b/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md @@ -1,261 +1,250 @@ --- title: | - | __Continuous Delivery__ - | Building Trust in a Large-scale, Complex Government Organization + | __Continuous Delivery:__ + | Building Trust in a Large-scale, Complex + | Government Organization papersize: a4 geometry: "left=1in,right=1.5in" --- ## Authors -__Rodrigo Siqueira__ is a masters student at IME - The Institute of Mathematics -and Statistics of the Sao Paulo University. His research interests include -software engineering, operating system and computer architecture. He worked for -two years with the Brazilian federal government as a coach and developer. -Contact him at siqueira@ime.usp.br. - -__Diego Camarinha__ is a masters student at IME - The Institute of Mathematics -and Statistics of the Sao Paulo University. His research interests include -software engineering, computer networks and source code metrics. He worked for -two years with the Brazilian federal government as a senior developer. Contact -him at diegoamc@ime.usp.br. - -__Melissa Wen__ is a software developer. She worked on SPB project as a senior -developer and also served as professor of Computer Science at UFBA - The Federal -University of Bahia. Her areas of interest include software engineering and open -source software development. Contact her at melissa.srw@gmail.com . - -__Paulo Meirelles__ received his Ph.D. in Computer Science from the Institute of -Mathematics and Statistics at the University of São Paulo. He is full-time -Professor at the University of Brasilia, and coordinated the new SPB portal -project. His research interest areas are: Free Software, Agile methods, Static -analysis, and Source code metrics. Contact him at paulormm@ime.usp.br. - -__Fabio Kon__ is Full Professor of the Department of Computer Science of the -University of São Paulo. (...) .His research interest areas are: (...). Contact -him at fabio.kon@ime.usp.br. - - +__Rodrigo Siqueira__ is a Computer Science MSc student at the Institute of +Mathematics and Statistics of the University of São Paulo (IME-USP). His +research interests include software engineering, operating system, and computer +architecture. He worked for two years with the Brazilian federal government as +a coach and developer. Contact him at siqueira@ime.usp.br. + +__Diego Camarinha__ is a CS MSc masters student at IME-USP. His research +interests include software engineering, computer networks, and source code +metrics. He worked for two years with the Brazilian federal government as a +senior developer. Contact him at diegoamc@ime.usp.br. + +__Melissa Wen__ is a software developer. She worked on the SPB project as a +senior developer and also served as a Computer Science lecturer at the Federal +University of Bahia. Her areas of interest include software engineering and +open source software development. Contact her at melissa.srw@gmail.com. + +__Paulo Meirelles__ received his Ph.D. in Computer Science from IME-USP. He is +a full-time Professor at the University of Brasilia and coordinated the new SPB +portal project. His research interests are: Free Software, Agile methods, +Static analysis, and Source code metrics. Contact him at paulormm@ime.usp.br. + +__Fabio Kon__ is a Full Professor of Computer Science at IME-USP and +Editor-in-Chief of the SpringerOpen Journal of Internet Services and +Applications. His research interests include Agile Software Development, Smart +Cities, and Distributed Systems. Contact him at fabio.kon@ime.usp.br. + ## Abstract For many software development teams, the first aspects that come to mind regarding Continuous Delivery (CD) are the operational challenges and the competitive benefits. In our experience, CD was much more: it was a survival -technique. This article presents how and why we applied CD in a large Brazilian -government project for the development of a Collaborative Development -Environment (CDE), sharing the challenges we faced and the strategies used to -overcome them. ( >> FECHAR COM FRASE DE IMPACTO <<<). +technique. This article presents how and why we applied CD in a large +governmental project for the development of a Collaborative Development +Environment. We share the challenges we faced and the strategies we used to +overcome them. The article concludes with a set of lessons learned that can be +valuable for readers in a variety of situations. ## Introduction -From 2014 to 2016, we were part of a team that worked on a thirty-month-long -Brazilian government project to modernize the Brazilian Public Software (SPB) -portal (www.softwarepublico.gov.br) [1]. This project was a partnership between -the Ministry of Planning, Budget, and Management and two public universities: +From 2014 to 2016, we worked on a thirty-month-long Brazilian government +project to modernize the Brazilian Public Software (SPB) portal +(www.softwarepublico.gov.br) [1]. This project was a partnership between the +_Ministry of Planning, Budget, and Management_ and two public universities: University of Brasília and University of São Paulo. During this time, the SPB portal evolved into a Collaborative Development -Environment (CDE) which brought significant benefits for the government and the -society: The government could minimize bureaucracy and software development +Environment (CDE), which brought significant benefits for the government and +the society. The government could minimize bureaucracy and software development costs, by reusing the same set of applications across different government -agencies; society could more transparently follow government expenses and +Agencies. Society could more transparently follow government expenses and contribute to project communities. In this article, we discuss the use of Continuous Delivery (CD) during our -experience as the academic partner in this project. We focus on how we managed -to implement CD in a large institution with traditional values and how CD -helped to build trust between the government and the university development -team. CD enabled us to show our progress and to earn the government’s -confidence that we could adequately fulfill their requests, becoming an -essential aspect of our interaction with them. According to this experience, -the use of CD as a tool to build such trust relationships is yet another of its -benefits [2,3]. +experience as the academic partner in this project. We show how we implemented +CD in a large institution with traditional, waterfall-like values and how CD +helped to build trust between the government and the university team. CD +enabled us to show our progress and to earn the government’s confidence that we +could adequately fulfill their requests, becoming an essential aspect of our +interaction. According to this experience, the use of CD as a tool to build +trust relationships is yet another of its benefits [2,3]. ## Context SPB is a governmental program created to foster sharing and collaboration on Open Source Software (OSS) development for the Brazilian public administration. -In their own projects, the Ministry managed both software requirements and -server infrastructure. However, its hierarchical and traditional processes made -them unfamiliar with new software development techniques, such as CD. Any of -our requests had to pass through layers of bureaucracy before being answered; -accessing their infrastructure to deploy updated software was not different. -The difficulties were aggravated because the new SPB portal is an unprecedented -platform in the Brazilian government, with a complicated deployment process [4]. - -The project suffered significant interference from the board of directors +The ministry managed both software requirements and server infrastructure. +However, its hierarchical and traditional processes made them unfamiliar with +new software development techniques, such as CD. All our requests had to pass +through layers of bureaucracy before being answered; accessing their +infrastructure to deploy software was not different. The difficulties were +aggravated because the new SPB portal is an unprecedented platform in the +Brazilian government, with a complicated deployment process [4]. + +The project suffered significant interference from the Board of Directors throughout time because the portal represents an interface between government and society. In light of political interests, directors continually imposed -changes to the platform while ignoring our technical advice. In 2015, the board -of directors was changed and, with it, the vision of the project. New directors -had different political agendas which affected the project's requirements +changes to the platform while ignoring our technical advice. In 2015, the Board +of Directors was changed and, with it, the vision of the project. New directors +had different political agendas, which affected the project's requirements previously approved. - - -In this context, we overcame two distinct challenges: (1) deconstructing -the widespread belief among government staff that any project in partnership -with a University is doomed to fail, and (2) dealing with bureaucracies -involved in the Ministry deployment process. +In this context, we overcame two distinct challenges: (1) deconstructing the +widespread belief among government staff that any project in partnership with a +University is doomed to fail, and (2) dealing with bureaucracies involved in +the government deployment process. -First, our development team was not from a typical company, consisting mainly -of undergraduate interns supported by senior developers and designers, all +First, our development team was not the typical one, consisting mainly of +undergraduate interns supported by senior developers and designers, all coordinated by two professors. Our unconventional team structure and -organization might be considered as "unprofessional" by Ministry standards with -regard to time and resource allocation, accountability, and team continuity . +organization was considered as "unprofessional" by ministry standards with +regard to time and resource allocation, accountability, and team continuity. On the government side, the SPB portal evolution was the first software -development collaboration involving universities and the Ministry staff, +development collaboration involving universities and the ministry staff, raising disbelief. Second, our team approached software deployment differently from the -Ministry. We believed frequent delivery is better for the project’s success. -In contrast, the Ministry is used to the idea of a single deployment at the end +government. We believed frequent delivery is better for the project’s success. +In contrast, the ministry is used to the idea of a single deployment at the end of the project, and neither their bureaucratic structure nor their technical -expertise was conducive to this style of work. That ended up hampering the -benefits of the tool and preventing us from showing off the fruits of the -project to those responsible for evaluating it. +expertise was conducive to this style of work. That was hampering the benefits +of the tool and preventing us from showing off the fruits of the project to +those responsible for evaluating it. -These challenges made our relationship with the Ministry staff tense, in -particular during the first year, and alerted us to the fact that they could -cancel the project at any time. The deployment limitation was the substantial -technical issue we could tackle in the short term. As a result, we worked to -deploy one version of the project onto our infrastructure and showed it to the -government evaluators. This strategy proved them we could efficiently provide -new features, fulfill their expectations regarding the delivery of the -requirements, and incited them to demand that the latest version be deployed in -the Ministry infrastructure. Our CD approach generated more pressure on the IT -department responsible for the deployment routines. With each CD cycle, we -gradually built a new relationship among all parties and, by the end of the -project, we became active participants in the deploy operations. +These challenges made our relationship with their staff strained, in particular +during the first year, and alerted us to the fact that they could cancel the +project at any time. The deployment limitation was the substantial technical +issue we could tackle in the short term. Thus, we worked to deploy the software +into our infrastructure and showed it to the government evaluators. This +strategy proved them we could efficiently provide new features, fulfill their +requirement delivery expectations, and incited them to demand that the latest +version be deployed in their infrastructure. Our CD approach generated more +pressure on the IT department responsible for the deployment routines. With +each CD cycle, we gradually built a new relationship among all parties and, by +the end of the project, we became active participants in the deploy operations +delivering quality software very frequently. ## Our Continuous Delivery Pipeline -SPB portal is a CDE with additional social features, having 83 software -communities and 6,460 users, mostly government employees. We built -system-of-systems adapting and integrating five existing OSS projects: Colab -(www.github.com/colab), Noosfero (www.noosfero.org), Gitlab (www.gitlab.com), -Mezuro (www.mezuro.org), and -Mailman (www.list.org). Colab orchestrates these multiple systems using a -plugin architecture and allows us to smoothly provide a unified interface for -final users, including single sign-on and global searches [1]. All these -integrated systems represents a total of 106.253 commits and 1.347.421 lines of -code. +The SPB portal is an open CDE with additional social features, having 83 +software communities and 6,460 user accounts, mostly from government employees. +We built a system-of-systems [5] adapting and integrating five existing OSS +projects: Colab (www.github.com/colab), Noosfero (www.noosfero.org), Gitlab +(www.gitlab.com), Mezuro (www.mezuro.org), and Mailman (www.list.org). Colab +orchestrates these multiple systems using a plugin architecture and allowed us +to smoothly provide a unified interface to final users, including SSO and +global searches [1]. All these integrated systems involve a total of 106,253 +commits and 1,347,421 lines of code. ![The SPB Deployment Pipeline.](figures/pipeline.png) -The portal deployment follows a typical CD pipeline [5], adapted to the -technical and organizational context of our project and the use of OSS best -practices, as presented in Figure 1. It started when new code arrived; and when -it reaches the production environment, we would restart the pipeline to release -new versions. +Portal deployment follows a typical CD pipeline [6], adapted to our technical +and organizational context and the use of OSS best practices. As depicted in +Figure 1, it begins when a new feature is ready for deployment and ends when it +reaches production. + ### Automated Tests -In practice, each integrated system is a Colab plugin and had to be tested with -its own test suite. However, this checking was not enough: we also had to test -the platform as a whole. Since the plugins have their own test suites, they -also assume a double role as both plugin tests and as integration tests. If any -test suite failed, by either a test error or coverage reduction below a certain -threshold, the process stopped. Only when all tests passed, the pipeline -proceeded to the step of release preparation. +Each integrated system is a Colab plugin and had to be tested with its own test +suite. In addition, we also had to test the platform as a whole. Since the +plugins have their own test suites, they also assume a double role as both +plugin tests and as integration tests. If any test suite failed, by either a +test error or coverage reduction below a certain threshold, the process +stopped. Only when all tests passed, the pipeline proceeded to the next step. ### Preparing a New Release -Each software component was hosted in a separate Git repository. A new component -release was tagged referencing a specific new feature or bug fix. SPB -had its own Git repository (www.softwarepublico.gov.br/gitlab/softwarepublico). -An SPB portal release was an aggregate of versions of all its components. -When a new component release passed all of the SPB integration tests, we -manually created a corresponding new tag in its repository. Therefore, a new -tag on any software component yielded a new SPB portal release. At the end of -this process, we started packaging. +A separate Git repository hosted each software component. New component +releases were tagged referencing a specific new feature or bug fix. SPB had its +own Git repository (www.softwarepublico.gov.br/gitlab/softwarepublico). An SPB +portal release was an aggregate of all its components. When a new component +release passed all of the SPB integration tests, we manually created a +corresponding new tag in its repository. At the end of this process, we started +packaging. ### Packaging -After creating a new tag for a system, our DevOps team starts the packaging -process. Packaging brings portability, simplifies deployment, and enables -configuration and permission control. Our packaging software approach involves -building separate packages for each system, in three steps: writing the script -for the specific environment, building the package, and uploading it to a -package repository. A set of scripts fully automated these steps. When all them -ran successfully, the new packages would be ready and available for our -deployment scripts. +After creating a new tag, our DevOps team started the packaging process. +Packaging brings portability, simplifies deployment, and enables configuration +and permission control. Our approach involved building separate packages for +each system, in three fully automated steps: generating scripts for the +specific environment, building the package, and uploading it to a package +repository. When all ran successfully, the new packages would be ready and +available for our deployment scripts. -### Validation Environment Deployment +### Validation Environment The Validation Environment (VE) is a replica of the Production Environment (PE) -with anonymised data, and access restricted to Ministry staff and our DevOps +with anonymised data, and access restricted to ministry staff and our DevOps team. To configure this environment, we used Chef (www.chef.io) and Chake, a -serverless configuration tool created by our team -(www.github.com/terceiro/chake). This tool maintained environment consistency, +serverless configuration tool created by our team +(www.github.com/terceiro/chake) to maintain environment consistency, simplifying the deployment process. ### Acceptance Tests -After a new SPB portal deployment in the VE, we used the environment to verify -the integrity of the entire portal. In this step, Ministry staff also checked -the new features, required changes, and bug fixes. If they identified a -problem, they would notify the developers via comments on the SPB portal issue -tracker, prompting the team to fix it and restart the pipeline. Otherwise, we -could move forward. +After a new SPB portal VE deployment, we used the environment to verify the +integrity of the entire portal. Government staff also checked the new features, +required changes, and bug fixes. If they identified a problem, they would +notify developers via comments on the SPB portal issue tracker, prompting the +team to fix it and restart the pipeline. Otherwise, we could move forward. ### Production Environment Deployment After the VE check, we could finally begin the deployment in production, with the same configuration management tool, scripts, and package versions as in the -VE. After the deploy was completed, both VE and PE were identical. At that -point, new features and bug fixes were finally available to end users. +VE. After the deploy was completed, both VE and PE were identical, making new +features and bug fixes available to end users. ## Benefits CD brings many advantages such as accelerated time to market, building the right product, productivity and efficiency improvements, stable releases, and better customer satisfaction [2,3]. The charts presented in Figure 2 show these -benefits of CD in our project and associates them with the creation of a DevOps -team. In the time of 30 months, we deployed a total of 84 versions. Over 64% of -these activities happened in the second half of 2015, with the DevOps team -creation. Besides these results, working with the government, we noticed the -following additional benefits. +benefits in our project and associates them with the creation of a DevOps team. +Over 30 months, we deployed 84 versions. Over 64% of the releases happened in +the last 12 months, after the creation of the DevOps team. Besides these +results, working with the government we noticed the following additional +benefits. -![The evolution of SPB releases and the team members distribution in the period.](figures/CDReleaseAndTeamEvolution.png) +![The evolution of SPB releases and the development team members distribution.](figures/CDReleaseAndTeamEvolution.png) ### Strengthening Trust in the Relationship with the Government -CD helped strengthen trust in the relationship between developers and the -Ministry staff. Before using CD, they could validate features developed only at -the end of the release cycle, usually every four months. With the -implementation of CD, intermediate and candidate versions became available, -allowing them to perform small validations over time. Constant monitoring of -the development work brought greater security to the Ministry leaders and -improved the interactions with our team. +CD helped strengthen trust between developers and the government staff. Before +using CD, they could validate features developed only at the end of the release +cycle, usually every four months. With the implementation of CD, intermediate +and candidate versions became available, allowing them to perform small +validations over time. Constant monitoring of the development work brought +greater assurance to the ministry leaders and improved the interactions with +our team. ### Responsiveness to Change Responsiveness was one of the direct benefits of adopting the CD pipeline. -Political interests may change requirements and priorities at any time. The -ability to react quickly to changes requested by the Ministry was vital to the -project’s survival for 30 months. We noticed that if we took too long to meet -their demands, they would threaten to reduce financial support and even cancel -the project. +Political forces may change requirements and priorities at any time. The +ability to react quickly to changes requested by the government was vital to +the project survival for 30 months. We noticed that if we took too long to meet +their demands, they could reduce financial support and even cancel the project. CD helped us keep the PE up-to-date, even with partial versions of a feature. -Therefore, we always had something to show on meetings, easing their concerns -about the final delivery of the platform. For our team, CD made developers more +Therefore, we always had new results to present on meetings, easing their +concerns About expected final delivery. For our team, CD made developers always confident that the project would last a little longer. ### Shared Responsibility -According to the conventional Ministry process, the development team could not -track what happened to the code after its delivery, since their staff was the -only ones responsible for deployment. The implementation of CD made our +According to the conventional ministry process, the development team could not +track what happened to the code after its delivery, since their employees were +the only ones responsible for deployment. The implementation of CD made our development team feel equally responsible for what was getting into production and take ownership of the project [4]. -Interestingly, the CD pipeline had the same effect on the Ministry staff. They +Interestingly, the CD pipeline had the same effect on the ministry staff. They became more engaged in the whole process, opening and discussing issues during the evolution of the platform. Additionally, developers worked to improve the CD pipeline and speed up the process of making new features available in the @@ -264,39 +253,33 @@ production environment. ### Synchronization Between Government and Development The CD pipeline performance depended on the synchronization between our -development team and the Ministry staff: each party had to be prepared to take -action as soon as the other concluded a given task. Initially, the Ministry +development team and the ministry staff: each party had to be prepared to take +action as soon as the other concluded a given task. Initially, the ministry staff did not contemplate this in their schedule which, combined with the bureaucracy in providing access to the PE (up to 3 days), resulted in significant delays in the validation of new features. The use of an explicit CD pipeline helped us to identify critical points of delay, and increase our productivity. - - ## Lessons Learned - -Due to the successful building of the CD pipeline, we improved the Ministry -deployment process and kept the project alive. We now look at the lessons -learned. +Due to the successful building of the CD pipeline, we not only overcame the +challenges we described above but also improved the ministry deployment process +and kept the project alive until its successful conclusion. We now look at the +lessons we learned, which can be leveraged by readers in other situations. ### Build CD From Scratch Taking on the responsibility for implementing CD impacted the whole team. Most of our team members did not have CD know-how, and we had few working hours -available to build the pipeline. The construction and maintenance of the CD -process were made possible by the key decisions to: +available to build the initial version of the pipeline. The construction and +maintenance of the CD process were made possible by the key decisions to: -1. _Select the most experienced senior developers and some advanced students of -the project to work on a specific DevOps team._ These senior developers used -their experience in OSS projects to craft an initial proposal for the -deployment process. The solution enabled us to automate the deployment, even -though the process was, initially, still rudimentary. +1. _Select the most experienced senior developers and some advanced students to +work on a specific DevOps team._These senior developers used their experience +in OSS projects to craft an initial proposal for the deployment process. The +solution enabled us to automate the deployment, even though the process was, +initially, still rudimentary. 2. _Interchange team members and encourage teammates to migrate to the DevOps team._ The benefits were twofold: mitigating the difficulty in sharing @@ -305,46 +288,45 @@ process on-the-fly. ### Overcoming Mistrust -Adopting an unfamiliar approach requires trust. Ministry staff, traditionally, -expected and were prepared to validate and deploy a single deliverable. However, -the steady growth of SPB complexity made large deliveries unsustainable. The CD -approach was necessary [4]. Therefore, we developed the following line of action to -make this new dynamics possible: - -1. _Demonstrate actual results, do not simply tell._ Initially, we did not have -access to the Ministry infrastructure, so we created our own validation -environment. Thus, we were able to follow the CD pipeline until production -deployment, when we faced two problems. First, our pace of intermediate -deliveries to the government was faster than the deployment to production by -the Ministry staff. Second, specific issues of the Ministry infrastructure made -some validated features not work as expected in the PE. That situation gave us -arguments to negotiate access to the PE. +Adopting an unfamiliar approach requires trust. ministry staff, traditionally, +expected and were prepared to validate and deploy a single deliverable. +However, the steady growth of SPB complexity made large deliveries +unsustainable. The CD approach was necessary [4]. Therefore, we developed the +following line of action to enable this new dynamics: + +1. _Demonstrate actual results, instead of simply reporting them._ Initially, +we did not have access to the ministry infrastructure, so we created our own +validation environment. Thus, we were able to follow the CD pipeline until +production deployment, when we faced two problems. First, our pace of +intermediate deliveries was faster than the deployment to production by the +government staff. Second, specific issues of the governmental infrastructure +made some validated features not work as expected in the PE. That situation +gave us arguments to negotiate access to the PE. 2. _Make project management transparent and collaborative for government -staff._ Allowing the Ministry staff to track our development process showed +staff._ Allowing the ministry staff to track our development process showed them we were fulfilling our commitments. They started to interact more actively in the generation of versions and became involved in the CD. After -understanding it, the Ministry staff helped us negotiate access to a VE with -the Ministry leaders, creating as an isolated replica of the PE. +understanding it, the ministry staff helped us negotiate access to a VE with +the ministry leaders, creating an isolated replica of the PE. 3. _Gain the confidence of government staff._ With the PE replica, we were able -to run the entire pipeline and win the trust of the Ministry staff involved in +to run the entire pipeline and win the trust of the ministry staff involved in the process. They saw the mobilization and responsiveness of our team to generate each new version package. They also recognized the quality of our work -and deployment process. In the end, the Ministry staff realized that it would +and deployment process. In the end, the ministry staff realized that it would be beneficial for the project if they granted us access to the infrastructure, both VE and PE. +In summary, we encourage the use of a well-thought CD pipeline as a means to +gain trust in software development projects with large organizations. + ## References 1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", OpenSym, 2017. 1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, 32 (2) 2015. 1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", ICSE, 2016. 1. M. Shahin, M. A. Babar, and L. Zhu. “The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives”. ESEM, 2016. +1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, 2015. 1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley, 2010. - - - -- libgit2 0.21.2