From 7ded138ccf0ef036733663024c490bf5f7219afc Mon Sep 17 00:00:00 2001 From: rodrigosiqueira Date: Fri, 17 Nov 2017 18:30:38 -0200 Subject: [PATCH] Revisão com o Nelson --- ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md | 283 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------------------------------------------------------------------------------------------------------------- 1 file changed, 146 insertions(+), 137 deletions(-) diff --git a/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md b/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md index 4f92835..e0a4343 100644 --- a/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md +++ b/ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md @@ -45,25 +45,25 @@ overcome them. ## Introduction -We worked on a thirty-month-long Brazilian government project to modernize the +From 2014 to 2016, we were part of a team that worked on a thirty-month-long Brazilian government project to modernize the Brazilian Public Software (SPB) portal (www.softwarepublico.gov.br) [1]. This -project, started in 2014, was a partnership between the Ministry of Planning, +project was a partnership between the Ministry of Planning, Budget, and Management and two public universities: University of Brasília and University of São Paulo. -With this partnership, the SPB portal evolved to a Collaborative Development +During this time, the SPB portal evolved into a Collaborative Development Environment (CDE) [2] which brought significant benefits for the government and -the society. The government could minimize bureaucracy and costs of software -development, encouraging the use of the same set of applications across -different government agencies. The society gained a mechanism of transparency, -follow government expenses, and collaboration, contribute to project -communities. . +the society: The government could minimize bureaucracy and software +development costs, by reusing the same set of applications across +different government agencies; society could more transparently +follow government expenses and contribute to project +communities. In this article, we discuss the use of Continuous Delivery (CD) during our experience as the academic partner in this project. We focus on how we managed to implement CD in a large institution with traditional values and how CD helped to build trust between the government and the university development -team. CD enabled us to show our progress and earned the government’s confidence +team. CD enabled us to show our progress and to earn the government’s confidence that we could adequately fulfill their requests, becoming an essential aspect of our interaction with them. According to this experience, the use of CD as a tool to build such trust relationships is yet another of its benefits [3]. @@ -72,14 +72,17 @@ tool to build such trust relationships is yet another of its benefits [3]. SPB is a governmental program created to foster sharing and collaboration on Open Source Software (OSS) development for the Brazilian public administration. -For their projects, the Ministry managed both software requirements and server +In their own projects, the Ministry managed both software requirements and server infrastructure. However, its hierarchical and traditional processes made them unfamiliar with new software development techniques, such as CD. Any of our -requests had to pass through layers of bureaucracy before being answered, -accessing their infrastructure to perform a deploy was not different. +requests had to pass through layers of bureaucracy before being answered; +accessing their infrastructure to deploy updated software was not different. +The difficulties were aggravated because the +new SPB portal is an unprecedented platform in the Brazilian government, with +a complicated deployment process. -During its lifetime, the project suffered significant interference from the -board of directors because the portal represents an interface between +The project suffered significant interference from the +board of directors throughout time because the portal represents an interface between government and society. In light of political interests, directors continually imposed changes to the platform while ignoring our technical advice. In 2015, the board of directors was changed and, with it, the vision of the project. New @@ -90,39 +93,41 @@ requirements previously approved. mudar em todas as ocorrências de agents por staff, mas checar com o Fabio. --> + + In this context, we overcame three distinct challenges: (1) finding a system solution with which government and development team agree, (2) deconstructing the widespread belief among government staff that any project in partnership with a University is doomed to fail, and (3) dealing with bureaucracies -involved in the deployment process by the Ministry. +involved in the deployment process. + To face the first issue, we designed the SPB portal as a CDE with additional social features. Due to the complexity of creating such a system from scratch, we decided to adapt and integrate existing OSS tools to build a system-of-systems [4]. We created a solution that orchestrates multiple -components and allowed us to smoothly provide a unified interface for final -users, including single sign-on and global searches [1]. On top of that, the -new SPB portal was an unprecedented platform to the Brazilian government, with -a complicated deployment process. - +components and allows us to smoothly provide a unified interface for final +users, including single sign-on and global searches [1]. Regarding the second problem, our team was not from a typical company, -consisting mainly of undergraduate students coordinated by two professors. In -the first year, we had a group composed of 24 undergraduate students, one -designer, and two senior developers. In 2015, our team grew to 36 students, two -designers, eight senior developers. In the end, due to budget constraints, our -team shrinked to 20 students, one designer, and two developers. On the +consisting mainly of undergraduate students coordinated by two professors. +Accordingly, time and resources allocation, accountability, and team +continuity might be construed as "unprofessional". On the government side, the SPB portal evolution was the first software development -collaboration between university and government experienced by the Ministry -staff involved in the project. - -Finally, our team thought software deployment differently than the Ministry. On -our side, we believe that frequent deliveries are better for the project’s -success. However, the Ministry works with the idea of a single deployment at -the end of the project. In other words, neither the bureaucratic structure of -the Ministry nor its technical abilities were conducive to this style of work. -Furthermore, there was little effort to deploy new versions of the system -promptly. That ended up hampering the benefits of the tool and preventing us +collaboration between universities and the Ministry +staff involved, raising disbelief. + +Finally, our team approached software deployment differently from the Ministry. +We believed frequent delivery is better for the project’s +success. In contrast, the Ministry is used to the idea of a single deployment at +the end of the project, and neither their bureaucratic structure +nor their technical expertise were conductive with this style of work. +That ended up hampering the benefits of the tool and preventing us from showing off the fruits of the project to those responsible for evaluating it. @@ -131,26 +136,31 @@ particular during the first year, and alerted us to the fact that they could finish the project at any time. The deployment limitation was the substantial technical issue we could tackle in the short term. As a result, we worked to deploy one version of the project onto our infrastructure and showed it to the -government evaluators. This strategy proved them we could efficiently deliver +government evaluators. This strategy proved them we could efficiently provide new features, fulfill their expectations regarding the delivery of the -requirements, and incited them to demand that the latest version be deployed in +requirements, and incited them to demand the latest version to be deployed in the Ministry infrastructure. This generated more pressure on the IT department responsible for the deployment routines. With each CD cycle, we gradually built a new relationship among all parties and, by the end of the project, we became active participants in the deploy operations. + ## Our Continuous Delivery Pipeline ![The SPB Deployment Pipeline](figures/pipeline_3.png) Figure 1 presents our CD pipeline. It follows a typical deployment pipeline [3], adapted to the technical and organizational context of our project and the -use of OSS best practices. The pipeline started when new code arrived. A new -feature might require changes to more than one SPB integrated software project. -Notice that each one of them could be modified independently. As the code went -through each step, it was tested and improved until it finally reached the +use of OSS best practices. The pipeline started when new code arrived. As the code went +through each step, it was tested and improved until finally reaching the production environment. At this point, we would restart the pipeline to release -more versions. +new versions. The SPB portal is a system-of-systems with 5 integrated software projects. Each -of them, as well as the entire platform, had to be tested. These software -components have their own test suite. Colab (www.github.com/colab), a systems -integration platform for web applications based on a plugin architecture, -orchestrates communication among them. Therefore, we developed specific plugins +of them had to be tested with its own test suite. +This was not enough, however: we also had to test the platform as a whole. To +do this, we leveraged our choice of Colab (www.github.com/colab) as the +orchestrator in the SPB. Colab is a systems +integration platform for web applications based on a plugin architecture. In +SPB, we developed specific plugins for each portal software component, such as Gitlab (www.gitlab.com) and -Noosfero (www.noosfero.org). Each plugin also has its own test suite, working -also as integration tests. +Noosfero (www.noosfero.org). Given that +the plugins also have their own test suites, these suites assume a double role as both +plugin tests and as integration tests. Both unit and integration tests provided us the performance and security needed to guarantee the stability of components and the platform. If any test suite @@ -187,41 +200,41 @@ step of release preparation. ### Preparing a New Release -An SPB portal release was composed of all its software component releases. -Each software component release had a Git tag that referred to a specific -feature or bug fix. When all tests passed for a given component, we manually -created a new tag for it. Therefore, a new tag on any software component -yielded a new SPB portal release. More precisely, SPB had a script that -produced a single release for the entire system based on each component tag. At +Each software component was hosted in a separate Git repository. A new release +of a component was tagged with a reference to a specific new +feature or bug fix. SPB, as an integration project, had its own Git repository. +An SPB portal release was an aggregate of releases of all of its components. +When a new component release passed all of the SPB +integration tests, we manually created a corresponding new tag in its repository. +Therefore, a new tag on any software component +yielded a new SPB portal release. At the end of this process, we started packaging. ### Packaging -The platform runs on the CentOS 7 GNU/Linux distribution. Packaging a software +The platform runs on the CentOS 7 GNU/Linux distribution. Packaging software for that distribution involves three steps: writing the script for the specific environment (RPM), building the package, and uploading it to a package repository. -We decided to create separate packages for each software component since: -Packaging makes easy to manage the software on a given distribution, -simplifies the deployment, follows the distribution's best practices, and -enables configurations and permissions control. +We decided to create separate packages for each software component. +Packaging makes it easy to manage software in a given distribution, +simplifies deployment, follows the distribution's best practices, and +enables configuration and permission control. After creating a new tag for a component, the developers informed our DevOps -[6] team, and the packaging process began. A set of scripts fully automated the -three packaging steps aforementioned. When all them ran successfully, the new -packages would be ready for our deployment scripts. +[6] team and the packaging process began. A set of scripts fully automated the +three packaging steps aforementioned. When all of them ran successfully, the new +packages would be ready and available for our deployment scripts. ### Validation Environment Deployment The Validation Environment (VE) is a replica of the Production Environment (PE) -with its data anonymised, as well as only Ministry staffs and our DevOps team -had access to it. To configure the environment, we used a configuration -management tool named Chef (www.chef.io) with Chake support -(www.github.com/terceiro/chake) -- a serverless configuration tool created by -our team. It maintained environment consistency simplifying the deployment -process. Additionally, the packages we built on the last step were readily -available to the management tool. +with anonymised data, and access restricted to Ministry staff and our DevOps team. +To configure this environment, we used +Chef (www.chef.io) and Chake, a serverless configuration tool created by +our team (www.github.com/terceiro/chake). This maintained environment consistency, simplifying the deployment +process. The Ministry staff used the VE to validate new features and required changes. The VE also was used to verify the integrity of the entire portal as part of @@ -229,19 +242,18 @@ the next step in the pipeline. ### Acceptance Tests -After we deployed a new SPB portal version in the VE, the Ministry staffs were -responsible for checking the features and bug fixes they required. If the -Ministry staffs identified a problem, they would notify the developers via -comments on the SPB portal's issue tracker. The development team fixed the -problem and the pipeline restarted. If everything was validated, we moved -forward. +After a new SPB portal deployment in the VE, the Ministry were +responsible for checking the required features and bug fixes. If they +identified a problem, they would notify the developers via +comments on the SPB portal's issue tracker, prompting the team to fix +it and restart the pipeline. Otherwise, we could move forward. ### Production Environment Deployment -When the Ministry staff finished the VE check, we could finally begin the -deployment in production. We also used our configuration management tool, the -same scripts and package versions as in the VE. After the deploy was completed, -both VE and PE were identical. Here was the point where new features and bug +After the VE check, we could finally begin the +deployment in the PE, with the same configuration management tool, +scripts, and package versions as in the VE. After the deploy was completed, +both VE and PE were identical. At that point, new features and bug fixes were finally available to end users. ## Benefits @@ -253,56 +265,54 @@ Working with the government, we noticed the following additional benefits. ### Strengthening Trust in the Relationship with the Government -CD helped to strengthen trust in the relationship between developers and -Ministry staffs. Before using CD, Ministry staff had access to the features -developed only at the end of the release, usually every four months. - +CD helped strengthen trust in the relationship between developers and +the Ministry staff. Before using CD, they had access to the features +developed only at the end of the release cycle, usually every four months. With the implementation of CD, intermediate and candidate versions became -available, allowing Ministry staffs to perform small validations over time. +available, allowing them to perform small validations over time. Constant monitoring of the development work brought greater security to the -Ministry leaders and improved the interactions with our development team. +Ministry leaders and improved the interactions with our team. ### Responsiveness to Change Responsiveness was one of the direct benefits of adopting the CD pipeline. The -ability to react quickly to changes requested by the Ministry staff was vital +ability to react quickly to changes requested by the Ministry was vital to the project’s survival for 30 months. Every meeting with the Ministry -leaders resulted in requirements and priorities changes, several of them -motivated by political needs. We observed that if we took too long to meet -their demands, the Ministry would use undelivered requirements to justify cut -in the financial support and cancel the project. +leaders led to changes in requirements and priorities, several of them +motivated by political interests. We noticed that if we took too long to meet +their demands, they would threaten to reduce +financial support and even cancel the project. CD helped us keep the PE up-to-date, even with partial versions of a feature. -That way, we always had something to show on meetings, reducing anxiety in -getting the platform finished. For our team, it made the developers more +Therefore, we always had something to show on meetings, easying their +concerns about the final delivery of the platform. +For our team, CD made developers more confident that the project would last a little longer. ### Shared Responsibility According to the conventional Ministry process, the development team could not -track what happened to the code after its delivery, since Ministry staff were +track what happened to the code after its delivery, since their staff were the only ones responsible for deployment. The implementation of CD made our development team feel equally responsible for what was getting into production and take ownership of the project. Interestingly, the CD pipeline had the same effect on the Ministry staff. They became more engaged in the whole process, opening and discussing issues during -platform evolution. Additionally, developers worked to improve the CD pipeline -to speed up the process of making new features available in the production -environment for the Ministry staff's validation. - - -### Synchronicity Between Government and Development - -The CD pipeline performance depended on the synchronicity between our -development team and the Ministry staffs so that the latter were prepared to -start a step as soon as the former concluded the previous step and vice versa. -Initially, the agenda of the Ministry staffs did not contemplate this concern, -which generated delays in the validation of new features. This situation -combined with governmental bureaucracy to release access to the production -environment (up to 3 days) resulted in additional delays for the deployment -step begin. This problem was softened when the Ministry staff realized the -impact of these delays on the final product and decided to allocate the +the evolution of the platform. Additionally, developers worked to improve the CD pipeline +and speed up the process of making new features available in the production +environment. + +### Synchronization Between Government and Development + +The CD pipeline performance depended on the synchronization between our +development team and the Ministry staff: each party had to be prepared to +take action as soon as the other concluded a given task. +Initially, the Ministry staff did not contemplate this in their schedule which, +combined with the bureaucracy in providing access to the PE +(up to 3 days), resulted in significant delays in the validation of new features. +This problem was softened when they realized the +impact of these delays on the final product and decided to allocate revisions in their work schedule.