From 5f1b08cd57e1015532050626d75567e5472a7ab2 Mon Sep 17 00:00:00 2001 From: Paulo Meirelles Date: Wed, 15 Nov 2017 00:32:31 -0200 Subject: [PATCH] [ieeeSW] General review --- ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md | 412 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 file changed, 199 insertions(+), 213 deletions(-) diff --git a/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md b/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md index 324a7ca..2aca505 100644 --- a/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md +++ b/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md @@ -30,7 +30,7 @@ source software development. Contact her at melissa.srw@gmail.com . __Paulo Meirelles__ received his Ph.D. in Computer Science from the Institute of Mathematics and Statistics at the University of São Paulo. He is full-time -Professor at the University of Brasilia, and coordinated the new SPB Portal +Professor at the University of Brasilia, and coordinated the new SPB portal project. His research interest areas are: Free Software, Agile methods, Static analysis, and Source code metrics. Contact him at paulormm@ime.usp.br. @@ -42,9 +42,9 @@ him at fabio.kon@ime.usp.br. For many software development teams, the first aspects that come to mind regarding Continuous Delivery (CD) are the operational challenges and the -competitive benefits. In our experience, the CD was much more: it was a survival -technique. This article presents how and why we applied CD in a Brazilian -government project for the development of a Collaborative Development +competitive benefits. In our experience, the CD was much more: it was a +survival technique. This article presents how and why we applied CD in a +Brazilian government project for the development of a Collaborative Development Environment (CDE), sharing the challenges we faced and the strategies used to overcome them. @@ -56,29 +56,29 @@ Siqueira/Paulo: Ao meu ver, a introdução deixa bem claro o nosso "punch" e o r --> We worked on a three-year-long Brazilian government project to evolve an -existing platform that had technical issues and lacked political support. -This project, started in 2014, was a partnership between the Ministry of -Planning, Budget, and Management (MP) and two public universities: -University of Brasília (UnB) and University of São Paulo (USP), to -modernize Brazilian Public Software (SPB) portal. - -With this partnership, the SPB Portal (www.softwarepublico.gov.br) evolved to a -Collaborative Development Environment[1], and this evolution brought significant -benefits not just to the Brazilian government, but also to society as a whole. -The government could minimize bureaucracy and costs of software development, -encouraging the use of the same set of applications across different government -agencies. The society also gained a mechanism of transparency and collaboration, -since anyone can check the government expenses on software and contribute to -project communities. +existing platform that had technical issues and lacked political support. This +project, started in 2014, was a partnership between the Ministry of Planning, +Budget, and Management (MP) and two public universities: University of Brasília +(UnB) and University of São Paulo (USP), to modernize Brazilian Public Software +(SPB) portal. + +With this partnership, the SPB portal (www.softwarepublico.gov.br) evolved to a +Collaborative Development Environment[1], and this evolution brought +significant benefits not just to the Brazilian government, but also to society +as a whole. The government could minimize bureaucracy and costs of software +development, encouraging the use of the same set of applications across +different government agencies. The society also gained a mechanism of +transparency and collaboration, since anyone can check the government expenses +on software and contribute to project communities. In this article, we discuss the use of Continuous Delivery (CD) during our experience as the academic partner in this project. We focus on how we managed -to implement CD in a large institution with traditional values and how CD helped -to build trust between the government and the university development team. CD -enabled us to show our progress and earned the government’s confidence that we -could adequately fulfill their requests, becoming an essential aspect of our -interaction with them. According to this experience, the use of CD as a tool to -build such trust relationships is yet another of its benefits[2]. +to implement CD in a large institution with traditional values and how CD +helped to build trust between the government and the university development +team. CD enabled us to show our progress and earned the government’s confidence +that we could adequately fulfill their requests, becoming an essential aspect +of our interaction with them. According to this experience, the use of CD as a +tool to build such trust relationships is yet another of its benefits[2]. ## Context The SPB is a governmental program of the MP created to foster sharing and -collaboration on Open Source Software (OSS) development for the Brazilian public -administration. For their projects, the MP managed both software requirements -and server infrastructure. However, its hierarchical and traditional processes -made them unfamiliar with new software development techniques, such as CD. Any -of our requests had to pass through layers of bureaucracy before being answered. -Requesting access to their infrastructure to make the deploy was not different. +collaboration on Open Source Software (OSS) development for the Brazilian +public administration. For their projects, the MP managed both software +requirements and server infrastructure. However, its hierarchical and +traditional processes made them unfamiliar with new software development +techniques, such as CD. Any of our requests had to pass through layers of +bureaucracy before being answered. Requesting access to their infrastructure +to make the deploy was not different. During its lifetime, the project suffered significant interference from the -board of directors because the portal represents an interface between government -and society. In light of political interests, directors continually imposed -changes to the platform while ignoring our technical advice. In 2015, the board -of directors was changed and, with it, the vision of the project. New directors -had different political agendas which affected the project's requirements -previously approved. +board of directors because the portal represents an interface between +government and society. In light of political interests, directors continually +imposed changes to the platform while ignoring our technical advice. In 2015, +the board of directors was changed and, with it, the vision of the project. New +directors had different political agendas which affected the project's +requirements previously approved. -Firstly, to find a system solution wherein government and development team -agree, we designed the SPB Portal as a CDE with additional social features. Due +Firstly, to find a system solution wherein MP agents and development team +agree, we designed the SPB portal as a CDE with additional social features. Due to the complexity of creating such a system from scratch, we decided to adapt -and integrate existing open source tools to build a system-of-systems [3]. We -created a solution that orchestrates software and allowed us to smoothly provide -a unified interface for final users, including single sign-on and global -searches [4]. On top of that, the new SPB Portal was an unprecedented platform -to the Brazilian government, with a complicated deployment process. +and integrate existing OSS tools to build a system-of-systems [3]. We created a +solution that orchestrates software and allowed us to smoothly provide a +unified interface for final users, including single sign-on and global searches +[4]. On top of that, the new SPB portal was an unprecedented platform to the +Brazilian government, with a complicated deployment process. - - - - -Secondly, we had to face the widespread belief among government agents that any -project in partnership with a University is doomed to fail. Our team was not -from a typical company, consisting mainly of undergraduate students coordinated -by two professors. At the first year, we had a group composed of 24 -undergraduate students, one designer, and two senior developers. In 2015, our -team grew to 36 students, two designers, eight senior developers. At the end of -in project, due to budget constraint, we had 20 students, one designer, and two -developers. Also, on the government side, the SPB Portal evolution was the first -software development collaboration between university and government experienced -by the MP analysts involved in the project. - - +Secondly, we had to face the widespread belief among MP agents that any project +in partnership with a University is doomed to fail. Our team was not from a +typical company, consisting mainly of undergraduate students coordinated by two +professors. At the first year, we had a group composed of 24 undergraduate +students, one designer, and two senior developers. In 2015, our team grew to 36 +students, two designers, eight senior developers. At the end, due to budget +constraint, we had 20 students, one designer, and two developers. On the +government side, the SPB portal evolution was the first software development +collaboration between university and government experienced by the MP analysts +involved in the project. Lastly, our team thought software deployment differently than the MP. On our side, we believe that frequent deliveries are better for the project’s success. However, the MP works with the idea of a single deployment at the end of the project. In other words, neither the bureaucratic structure of the MP nor its -technical abilities were conducive to this style of work. Furthermore, there was -little effort to deploy new versions of the system promptly. That ended up -hampering the benefits of the tool and preventing us from showing off the fruits -of the project to those responsible for evaluating it. +technical abilities were conducive to this style of work. Furthermore, there +was little effort to deploy new versions of the system promptly. That ended up +hampering the benefits of the tool and preventing us from showing off the +fruits of the project to those responsible for evaluating it. -These challenges made our relationship with the MP tense, in particular at the -first year, and alerted us to the fact that they could finish the project at any -time. The deployment limitation was the substantial technical issue we could -tackle in the short term. As a result, we worked to deploy one version of the -project onto our infrastructure and showed it to the government evaluators. This -strategy proved them we could efficiently deliver new features, fulfill their -expectations regarding the delivery of the requirements, and incited them to -demand the last version working in the MP infrastructure. These results, in turn, -generated more pressure on the IT department responsible for the deployment -routines. With each CD cycle, we gradually built a new relationship among all -parties and, by the end of the project, we became active participants in the -deploy operations. +These challenges made our relationship with the MP agents tense, in particular +at the first year, and alerted us to the fact that they could finish the +project at any time. The deployment limitation was the substantial technical +issue we could tackle in the short term. As a result, we worked to deploy one +version of the project onto our infrastructure and showed it to the government +evaluators. This strategy proved them we could efficiently deliver new +features, fulfill their expectations regarding the delivery of the +requirements, and incited them to demand the last version working in the MP +infrastructure. These results, in turn, generated more pressure on the IT +department responsible for the deployment routines. With each CD cycle, we +gradually built a new relationship among all parties and, by the end of the +project, we became active participants in the deploy operations. ## Our Continuous Delivery Pipeline -Figure 1 represents our CD pipeline. A new feature might require changes on more -than one SPB integrated software project. Notice that each one of them could be -modified independently. The pipeline started when new code arrived. As it went -through each step, it was tested and improved until it finally reached the -production environment. At this point we would restart the pipeline to release -more new code. - ![Deployment Pipeline](figures/pipeline_3.png) +Figure 1 represents our CD pipeline. A new feature might require changes on +more than one SPB integrated software project. Notice that each one of them +could be modified independently. The pipeline started when new code arrived. As +it went through each step, it was tested and improved until it finally reached +the production environment. At this point we would restart the pipeline to +release more new code. + ### Automated Tests -The SPB portal is a system-of-systems with several integrated software projects. -Each of them, as well as the entire platform, had to be tested. These software -components have own test suite. Colab, a systems integration platform for web -applications based on a plugins architecture, orchestrate communication between -all of them. Therefore, we developed specific plugins for each portal software -component, such as Gitlab (www.gitlab.com) and Noosfero (www.noosfero.org). Each -plugin also has own test suite, and this set also worked as integration tests. + +The SPB portal is a system-of-systems with several integrated software +projects. Each of them, as well as the entire platform, had to be tested. +These software components have own test suite. Colab (www.github.com/colab), a +systems integration platform for web applications based on a plugins +architecture, orchestrate communication between all of them. Therefore, we +developed specific plugins for each portal software component, such as Gitlab +(www.gitlab.com) and Noosfero (www.noosfero.org). Each plugin also has own test +suite, and this set also worked as integration tests. Both unit and integration tests provided us the performance and security needed to guarantee the stability of components and the platform. If any test suite @@ -236,13 +224,13 @@ step of preparing the release. ### Preparing a New Release -An SPB Portal release was composed of all its software components releases. Each -software component release was a git tag that referred to a specific feature or -bug fix. When all tests passed for a given component, we manually created a new -tag for it. Therefore, a new tag on any software component yielded a new SPB -Portal release. More precisely, SPB had a script that produced a single release -for the entire system based on each component tag. At the end of this process, -we started packaging. +An SPB portal release was composed of all its software components releases. +Each software component release was a Git tag that referred to a specific +feature or bug fix. When all tests passed for a given component, we manually +created a new tag for it. Therefore, a new tag on any software component +yielded a new SPB portal release. More precisely, SPB had a script that +produced a single release for the entire system based on each component tag. At +the end of this process, we started packaging. ### Packaging @@ -254,47 +242,48 @@ repository. We decided to create own packages for each software component for the following five reasons: -* Not all software was packaged by the community and those that existed were outdated; -* Packaging makes it easy to manage the software on a given distribution; -* Packaging simplifies the deployment; -* Packaging follows the distribution's best practices and, +1. Not all software was packaged by the community and those that existed were outdated; +1. Packaging makes it easy to manage the software on a given distribution; +1. Packaging simplifies the deployment; +1. Packaging follows the distribution's best practices and, * Packaging allows configurations and permissions control. -After creating a new tag for one component, the developers informed the DevOps +After creating a new tag for one component, the developers informed our DevOps [5] team, and the packaging process began. A set of scripts fully automated the three packaging steps aforementioned. With all them running successfully, the new packages would be ready to be used by our deployment scripts. ### Validation Environment Deployment -The Validation Environment (VE) is a replica of the Production Environment (PE), -with two exceptions: only the government officers and project leaders had access -to it and all the data became anonymous. To configure the environment, we used a -configuration management tool named Chef with Chake support (serverless -configuration for Chef). It maintained environment consistency simplifying the -deployment process. Additionally, the packages we built on the last step were -readily available to be used by the management tool. - -Government agents used the VE to validate new features and required changes. Also, -the VE was used to verify the integrity of the entire portal as part of the next +The Validation Environment (VE) is a replica of the Production Environment +(PE), with two exceptions: only the government officers and project leaders had +access to it and all the data became anonymous. To configure the environment, +we used a configuration management tool named Chef with Chake support (a +serverless configuration for Chef tool created by our team). It maintained +environment consistency simplifying the deployment process. Additionally, the +packages we built on the last step were readily available to be used by the +management tool. + +The MP agents used the VE to validate new features and required changes. The VE +also was used to verify the integrity of the entire portal as part of the next step in the pipeline. ### Acceptance Tests -After we deployed a new SPB Portal version in the VE, the government -agents were responsible for checking features and bug fixes required by them. If -the requirement analysts identified a problem, they would notify the developers via -comments on the SPB Portal's issue tracker. The development team fixed the problem -and the pipeline restarted from scratch. If everything was validated, we moved forward. +After we deployed a new SPB portal version in the VE, the MP agents were +responsible for checking features and bug fixes required by them. If the +requirement analysts identified a problem, they would notify the developers via +comments on the SPB portal's issue tracker. The development team fixed the +problem and the pipeline restarted from scratch. If everything was validated, +we moved forward. ### Production Environment Deployment -When the government finished the VE check, we could finally begin the deployment -in the Production Environment (PE). For this we also used our configuration -management tool, the same scripts and package versions as in the VE. After the -deploy was completed, both VE and PE were running identical software. Here was -the point where new features and bug fixes were finally available to the end -users. +When the MP agents finished the VE check, we could finally begin the deployment +in the PE. For this we also used our configuration management tool, the same +scripts and package versions as in the VE. After the deploy was completed, both +VE and PE were running identical software. Here was the point where new +features and bug fixes were finally available to the end users. ## Benefits @@ -306,26 +295,24 @@ Working with the government, we noticed the following additional benefits. ### Strengthening Trust in Work Relationship with the Government Continuous delivery was also a tool that helped to strengthen trust in the -relationship between developers and government analysts, as well as between the -analysts group and its superiors. Before using CD, analysts had access to the -features developed only at the end of the release, usually every four months. -However, this periodicity did not meet the requirements of their leaders, who -demanded monthly reports on the progress of the project. +relationship between developers and MP agents. Before using CD, the MP analysts +had access to the features developed only at the end of the release, usually +every four months. With the implementation of CD, intermediate and candidate versions became -available, allowing analysts to perform small validations over time. The -constant monitoring of the development work brought greater security to the -governmental nucleus and improved the interactions with our development team. +available, allowing the MP analysts to perform small validations over time. The +constant monitoring of the development work brought greater security to the MP +leaders and improved the interactions with our development team. ### Responsiveness to Change Responsiveness was one of the direct benefits of adopting the CD pipeline. The -ability to react quickly to changes requested by the government was vital for -the renewal of the project over the years. Every meeting with the government -leader resulted in requirements and priorities changes, most of them motivated -by political needs. We believed that if we took too long to attend their -demands, the government would use undelivered requirements as a means to -justify the lack of financial support and the end of the project. +ability to react quickly to changes requested by the MP agents was vital for +the renewal of the project over the years. Every meeting with the MP leaders +resulted in requirements and priorities changes, several of them motivated by +political needs. We observed that if we took too long to attend their demands, +the MP would use undelivered requirements as a means to justify the lack of +financial support and the end of the project. CD helped us keep the production environment up-to-date, even with partial versions of a feature. That way, we always had something to show on meetings, @@ -336,61 +323,61 @@ would not go looking for other jobs. ### Shared Responsibility According to the MP process, the development team could not track what happened -to the code after its delivery, since government technicians were the only ones -responsible for deployment. The implementation of CD made developers feel -equally responsible for what was getting into production and take ownership of -the project. -% -Interestingly, the CD pipeline had the same effect on the team of requirement -analysts. They became more engaged in the whole process, opening and discussing -issues during the platform evolution. Additionally, developers worked to improve -the CD pipeline to speed up the process of making new features available in the -production environment for analysts’ validation. +to the code after its delivery, since, theoretically, the MP technicians were +the only ones responsible for deployment. The implementation of CD made our +development team feel equally responsible for what was getting into production +and take ownership of the project. + +Interestingly, the CD pipeline had the same effect on the MP analysts. They +became more engaged in the whole process, opening and discussing issues during +the platform evolution. Additionally, developers worked to improve the CD +pipeline to speed up the process of making new features available in the +production environment for the MP analysts' validation. ### Synchronicity Between Government and Development Despite the positive impacts that the CD pipeline brought to the project, its implementation was not smooth at first. The CD pipeline performance depended on -the synchronicity between developers and government analysts so that the +the synchronicity between our development team and the MP agents so that the latter were prepared to start a step as soon as the former concluded the -previous step and vice versa. Initially, the agenda of the government team -did not contemplate this concern, which generated delays in the validation -of new features. This situation combined with governmental bureaucracy -(up to 3 days) to release access to the production environment resulted in -additional delays for the deployment step to begin. This problem was softened -when the analysts realized the impact of these delays on the final product and -decided to allocate the revisions in its work schedule and to request the access -to production in time. +previous step and vice versa. Initially, the agenda of the MP agents did not +contemplate this concern, which generated delays in the validation of new +features. This situation combined with governmental bureaucracy (up to 3 days) +to release access to the production environment resulted in additional delays +for the deployment step to begin. This problem was softened when the MP +analysts realized the impact of these delays on the final product and decided +to allocate the revisions in its work schedule and to request the access to +production in time. ## Challenges -Due to the successful building of the CD pipeline, we improved the government -deployment process and kept the project alive in an unstable political scenario. -We map here lessons learned in this successful case, as well as issues that -future works between industry and academia still need to address. +Due to the successful building of the CD pipeline, we improved the MP +deployment process and kept the project alive. We map here lessons learned in +this successful case, as well as issues that future works between industry and +academia still need to address. ### Build CD From Scratch Taking on responsibilities for implementing CD impacted on the whole team. -Our team did not have know-how in this approach, and we had few working hours -available to allocate for the building of the pipeline. The construction and -maintenance of the CD process were possible by taking some decisions to mature -the project: +Mostly, our team members did not have know-how in this approach, and we had few +working hours available to allocate for the building of the pipeline. The +construction and maintenance of the CD process were possible by taking some +decisions to mature the project: -1. _Select the most experienced professionals and some developers of the -project to work on a specific team for DevOps._ These professionals used their -experiences in OSS projects to get an initial proposal for the deployment process. -The solution enabled us to automate the deployment, even though the process was -still rudimentary. +1. _Select the most experienced senior developers and some advanced software +engineering students of the project to work on a specific team for DevOps._ +These senior developers used their experiences in OSS projects to get an +initial proposal for the deployment process. The solution enabled us to +automate the deployment, even though the process was still rudimentary. 2. _Interchange team members and encouraging teammates to migrate to DevOps team._ The benefits of these movements were twofold: mitigating the difficulty to pass the knowledge between DevOps developers and features developers, and evolving the process on-the-fly. -Building a CD pipeline was hard in the beginning. We believe that more tools -to provide out-of-the-box standardized CD pipelines would be of great help for +Building a CD pipeline was hard in the beginning. We believe that more tools to +provide out-of-the-box standardized CD pipelines would be of great help for inexperienced teams. Tools that track each step of the pipeline and organize logs in a human-manageable way are necessary too. We also suggest further research on how to effectively spread knowledge across inexperienced developers @@ -398,47 +385,48 @@ in a high turnover scenario. ### Overcoming Mistrust -Taking an unfamiliar approach requires trust. In the Ministry of Planning, -software is the product delivered at the end of a development contract. They -expected and were prepared to validate and deploy a single delivery. Because -the SPB portal is a system of systems, the steady growth of its complexity made -large deliveries unsustainable. Also, the long time for homologation of -developed features gave the government room to change requirements and priorities. -The CD approach was necessary, but how to build trust and gain autonomy to -implement a process that was not yet part of the dynamics of the Ministry? +Taking an unfamiliar approach requires trust. In the MP, software is the +product delivered at the end of a development contract. They expected and were +prepared to validate and deploy a single delivery. Because the SPB portal is a +system-of-systems, the steady growth of its complexity made large deliveries +unsustainable. The long time for homologation of developed features also gave +the government room to change requirements and priorities. The CD approach was +necessary, but how to build trust and gain autonomy to implement a process that +was not yet part of the dynamics of the Ministry? 1. _Demonstrate actual results, do not simply tell._ Initially, we did not have -access to the government infrastructure, so we created our own validation -environment. Thus, we were able to follow the CD pipeline until the stage of -production deployment, when we faced two problems. Our pace of intermediate -deliveries to the government was faster than the deployment in production by their -technicians. Furthermore, specific issues of government infrastructure made some -validated features not work as expected in the production environment. That -situation gave us arguments to negotiate access to production. - -2. _Make our project management transparent and collaborative for analysts._ -Allowing the analysts to follow our process for version deliveries and bug -fixes, we showed them we were meeting our commitments. They started to interact -more actively in the generation of versions and became part of the -process. After understanding the process, analysts helped us in negotiations -with government agents. Finally, the agents created a VE as an isolated replica -of PE and gave us access to it. +access to the MP infrastructure, so we created our own validation environment. +Thus, we were able to follow the CD pipeline until the stage of production +deployment, when we faced two problems. Our pace of intermediate deliveries to +the government was faster than the deployment in production by the MP +technicians. Furthermore, specific issues of the MP infrastructure made some +validated features not work as expected in the PE. That situation gave us +arguments to negotiate access to PE. + +2. _Make our project management transparent and collaborative for the MP +agents._ Allowing the MP analysts to follow our process for version deliveries +and bug fixes, we showed them we were meeting our commitments. They started to +interact more actively in the generation of versions and became part of the +process. After understanding the process, the MP analysts helped us in +negotiations with the MP leaders. Finally, the MP technicians created a VE as +an isolated replica of PE and gave us access to it. 3. _Gain the confidence of government agents._ With the replica of PE, we were -able to run the entire pipeline and won the trust of the government analysts and +able to run the entire pipeline and won the trust of the MP analysts and technicians involved in the process. On the one hand, analysts saw the -mobilization and responsiveness of the team to generate a new version package. +mobilization and responsiveness of our team to generate a new version package. On the other hand, technicians recognized the quality of our packages and our -deployment process. Finally, the government agents then realized that it could -be beneficial for the project if they granted us access to part of the -infrastructure. +deployment process. Finally, the MP agents then realized that it could be +beneficial for the project if they granted us access to the infrastructure, +included the PE. More research is required on developing protocols and policies to improve the relationship between industry and government, especially regarding CD. ## Acknowledgements -We thank our colleagues, Nelson Lago, Lucas Kanashiro and Rafael Manzo, and this article's reviewers. +We thank our colleagues, Nelson Lago, Lucas Kanashiro and Rafael Manzo, and +this article's reviewers. ## References @@ -454,5 +442,3 @@ We thank our colleagues, Nelson Lago, Lucas Kanashiro and Rafael Manzo, and this 9. Chen, L., 2015, May. Towards architecting for continuous delivery. In Software Architecture (WICSA), 2015 12th Working IEEE/IFIP Conference on (pp. 131-134). IEEE. ---> - - -- libgit2 0.21.2