Compare View
Commits (78)
-
To make sense with the introduction and context.
-
…articles into ieee_new
-
…articles into ieee_new
-
…articles into ieee_new
-
Signed-off-by: Paulo Meirelles <paulo@softwarelivre.org> Signed-off-by: Melissa wen <melissa.srw@gmail.com> Signed-off-by: Rodrigo Siqueira <rodrigosiqueiramelo@gmail.com>
-
…peline; fix some information conclicts
-
…articles into ieee_new
Showing
9 changed files
Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
1 | 1 | --- |
2 | -title: "Continuous Delivery: the strategy to overcome challenges in Government IT" | |
2 | +title: | | |
3 | + | __Continuous Delivery:__ | |
4 | + | Building Trust in a Large-scale, Complex | |
5 | + | Government Organization | |
3 | 6 | papersize: a4 |
4 | 7 | geometry: "left=1in,right=1.5in" |
5 | 8 | --- |
6 | 9 | |
7 | 10 | ## Authors |
8 | 11 | |
9 | -__Diego Camarinha__ is a masters student at IME - The Institute of Mathematics and | |
10 | -Statistics of the Sao Paulo University. His research interests include software | |
11 | -engineering, computer networks and source code metrics. He worked for two years | |
12 | -with the Brazilian federal government as a senior developer. Contact him at | |
13 | -diegoamc@ime.usp.br. | |
14 | - | |
15 | -__Rodrigo Siqueira__ is a masters student at IME - The Institute of Mathematics and | |
16 | -Statistics of the Sao Paulo University. His research interests include software | |
17 | -engineering, operating system and computer architecture. He worked for two years | |
18 | -with the Brazilian federal government as a coach and developer. Contact him at | |
19 | -siqueira@ime.usp.br. | |
20 | - | |
21 | -__Melissa Wen__ is a software developer. She worked on SPB project as a senior developer and also served as professor of Computer Science at UFBA - The Federal University of Bahia. Her areas of interest include software engineering and open source software development. Contact her at melissa.srw@gmail.com . | |
22 | - | |
23 | -__Paulo Meirelles__ received his Ph.D. in Computer Science from the Institute of Mathematics and Statistics at the University of São Paulo. He is a full-time Professor at the University of Brasilia, and coordinated the new SPB Portal project. His research interest areas are: Free Software, Agile methods, Static analysis, and Source code metrics. Contact him at paulormm@ime.usp.br. | |
12 | +__Rodrigo Siqueira__ is a Computer Science MSc student at the Institute of | |
13 | +Mathematics and Statistics of the University of São Paulo (IME-USP). His | |
14 | +research interests include Software Engineering, Operating System, and Computer | |
15 | +Architecture. He worked for two years with the SPB project as a coach and | |
16 | +developer. Contact him at siqueira@ime.usp.br. | |
17 | + | |
18 | +__Diego Camarinha__ is a Computer Science MSc masters student at IME-USP. His | |
19 | +research interests include Software Engineering, Computer Networks, and Source | |
20 | +Code Metrics. He worked for two years with the SPB project as a senior | |
21 | +developer. Contact him at diegoamc@ime.usp.br. | |
22 | + | |
23 | +__Melissa Wen__ is a Bachelor of Computer Science from Federal University of | |
24 | +Bahia (UFBA). She worked on the SPB project as a senior developer and was core | |
25 | +developer of Noosfero social networking platform at Colivre. Her areas of | |
26 | +interest include Software Engineering and Free Software Development. Contact | |
27 | +her at melissa.srw@gmail.com. | |
28 | + | |
29 | +__Paulo Meirelles__ is a Professor of Software Engineering at the | |
30 | +University of Brasilia (UnB) and researcher at the Free/Libre/Open Source Software | |
31 | +Competence Center (CCSL) at IME-USP. He was the head of the new SPB portal | |
32 | +development. His research interests include Free Software development, Agile | |
33 | +Software Development, and Source code metrics. Contact him at paulormm@unb.br. | |
34 | + | |
35 | +__Fabio Kon__ is a Full Professor of Computer Science at IME-USP, co-founder of | |
36 | +the CCSL at IME-USP, and Editor-in-Chief of the SpringerOpen Journal of | |
37 | +Internet Services and Applications. His research interests include Agile | |
38 | +Software Development, Smart Cities, and Distributed Systems. Contact him at | |
39 | +fabio.kon@ime.usp.br. | |
24 | 40 | |
25 | 41 | ## Abstract |
26 | 42 | |
27 | -For many software development teams, the first things that come to mind | |
28 | -regarding Continuous Delivery (CD) are the operational aspects and the | |
29 | -competitive benefits. In our experience, it was much more: it was a survival | |
30 | -technique. This article presents how we applied CD in a Brazilian Government | |
31 | -project for the development of a Collaborative Development Environment (CDE), | |
32 | -sharing its unconventional challenges and the strategies used to overcome them. | |
33 | -This report can help | |
34 | -practitioners to understand how important CD adoption is to their projects. | |
35 | - | |
36 | -## Introduction and Context | |
37 | - | |
38 | -We worked on a three-year-long Brazilian government project to evolve an | |
39 | -existing platform that had technical issues and lacked political support. Our | |
40 | -team from the University of Brasília (UnB) and the University of São Paulo | |
41 | -(USP) developed the new platform for the Brazilian Public Software (SPB, | |
42 | -Portuguese acronym) Portal from 2014 to 2016. The SPB Portal | |
43 | -(www.softwarepublico.gov.br) evolved to a Collaborative Development Environment | |
44 | -[1] and this evolution brought important benefits not just to the Brazilian | |
45 | -government, but also to society as a whole. The government could reduce both | |
46 | -the bureaucracy of using the same software in government agencies and the cost | |
47 | -of developing similar software projects. The society gained a mechanism of | |
48 | -transparency and collaboration, since anyone can check the government | |
49 | -expenses on software (from information available on the SPB Portal) and | |
50 | -contribute to project communities. To achieve these goals, rather than writing | |
51 | -everything from scratch, we decided to integrate several free software tools | |
52 | -such as Noosfero (www.noosfero.org), Gitlab (www.gitlab.com), Mailman | |
53 | -(www.gnu.org/software/mailman), Mezuro (www.mezuro.org), and Colab | |
54 | -(www.github.com/colab). | |
55 | - | |
56 | -The project started in a presidential election year and everyone involved was | |
57 | -under pressure to show results. Even with the re-election of the Brazilian | |
58 | -President in 2014, leaderships in governmental agencies ended up changing. Each | |
59 | -one of them had different political agendas which affected the project's | |
60 | -requirements previously approved. Besides that scenario of instability, we | |
61 | -overcame three distinct issues: (i) achieving the goals which have guided the | |
62 | -platform development, (ii) managing the diversity of team members, and (iii) | |
63 | -communicating effectively with government agents (the client). Handling the | |
64 | -interaction of these elements was challenging and the unstable Brazilian | |
65 | -political scenario only made things worse. | |
66 | - | |
67 | -To achieve the SPB project goals, we had to overcome strong political bias tied | |
68 | -with complicated technical issues and relatively low budget. Because the | |
69 | -project is open to the public, the government representatives saw the platform | |
70 | -as a marketing opportunity and sometimes ignored technical advice in favor of | |
71 | -political decisions. Furthermore, integrating a number of distinct systems | |
72 | -(such as social and collaboration network, Git repository manager, mailing | |
73 | -list, source code metric evaluation, and a systems integration platform) to | |
74 | -work seamlessly was not an easy job. We had to learn how each system worked and | |
75 | -to come up with ideas of how to integrate them as fast as possible, with a team | |
76 | -of mostly inexperienced developers. | |
77 | - | |
78 | -We also had to manage the diversity of our team members. The team was composed | |
79 | -of approximately 50 undergraduate students (not all simultaneously) together | |
80 | -with professors, master students, professional designers and senior developers | |
81 | -from the Brazilian free software community. Undergraduate students received a | |
82 | -scholarship and, for most of them, this R&D project was their first | |
83 | -professional experience. Senior developers and master students had two | |
84 | -important contributions to the project: to transfer knowledge to undergraduate | |
85 | -students and to deal with more complex tasks. Finally, professors were | |
86 | -responsible for interacting with the Brazilian government and controlling | |
87 | -political pressures applied to the project. | |
88 | - | |
89 | -Moreover, we needed to communicate with two independent groups of government | |
90 | -representatives. Requirement analysts were real representatives of the | |
91 | -Brazilian government in the project and their role was to test new features, to | |
92 | -provide feedback to the development team, and to report to government leaders. | |
93 | -Deployment technicians were in charge of managing the host machines wherein SPB | |
94 | -platform was running. They were, theoretically, responsible for deploying the | |
95 | -project. However, the new SPB Portal was composed of more than ten different | |
96 | -software projects, working on seven servers, generating a complex deployment | |
97 | -process. | |
98 | - | |
99 | -To handle the interaction between these three elements, we realised we needed | |
100 | -to take control over the deployment process. We used CD as a mean to fulfill | |
101 | -the government expectations and to provide quick response to their requests, | |
102 | -which were influenced most of the time by the uncertainties of the project's | |
103 | -continuity. We believed we would keep the project alive, even in a politically | |
104 | -unstable and technically complex scenario. For this reason, we focused on | |
105 | -automating the deploy process; for instance, one of our senior developers | |
106 | -created a tool called Chake (www.gitlab.com/terceiro/chake) to help us manage | |
107 | -multiple hosts without the need for a Chef-Server (www.chef.io/chef). We also | |
108 | -formed a specific team dedicated to the deployment process. This team was | |
109 | -responsible for maturing our CD pipeline, giving us confidence and agility to | |
110 | -comply with government requirements. | |
111 | - | |
112 | -In this report, we describe our CD pipeline and how it improved our delivery | |
113 | -performance. Continuous Delivery made us adaptable for all requested changes | |
114 | -and helped us mitigate those aforementioned political challenges, besides the | |
115 | -technical issues. Among CD’s known benefits [2] we also explain those which | |
116 | -were the most important in our scenario: improving customer satisfaction and | |
117 | -making reliable releases. Both kept the project alive for years during the | |
118 | -worst political crisis after the re-democratization in Brazil. | |
43 | +For many software development teams, the first aspects that come to mind | |
44 | +regarding Continuous Delivery (CD) are the operational challenges and the | |
45 | +competitive benefits. In our experience, CD was much more: it was a survival | |
46 | +technique. This article presents how and why we applied CD in a large | |
47 | +governmental project for the development of a Collaborative Development | |
48 | +Environment. We share the challenges we faced and the strategies we used to | |
49 | +overcome them. The article concludes with a set of lessons learned that can be | |
50 | +valuable for readers in a variety of situations. | |
51 | + | |
52 | +## Introduction | |
53 | + | |
54 | +From 2014 to 2016, we worked on a thirty-month-long Brazilian government | |
55 | +project to modernize the Brazilian Public Software (SPB) portal | |
56 | +(www.softwarepublico.gov.br) [1]. This project was a partnership between the | |
57 | +_Ministry of Planning, Budget, and Management (MPOG)_ and two public | |
58 | +universities: University of Brasília and University of São Paulo. | |
59 | + | |
60 | +During this time, the SPB portal evolved into a Collaborative Development | |
61 | +Environment (CDE), which brought significant benefits for the government and | |
62 | +the society. The government could minimize bureaucracy and software development | |
63 | +costs, by reusing the same set of applications across different government | |
64 | +Agencies. Society could more transparently follow government expenses and | |
65 | +contribute to software project communities. | |
66 | + | |
67 | +In this article, we discuss the use of Continuous Delivery (CD) during our | |
68 | +experience as the academic partner in this project. We show how we implemented | |
69 | +CD in a large institution with traditional, waterfall-like values and how CD | |
70 | +helped to build trust between the government and the university team. CD | |
71 | +enabled us to show our progress and to earn the government’s confidence that we | |
72 | +could adequately fulfill their requests, becoming an essential aspect of our | |
73 | +interaction. According to this experience, the use of CD as a tool to build | |
74 | +trust relationships is yet another of its benefits [2,3]. | |
75 | + | |
76 | +## Context | |
77 | + | |
78 | +SPB is a governmental program created to foster sharing and collaboration on | |
79 | +Open Source Software (OSS) development for the Brazilian public administration. | |
80 | +The government managed both software requirements and server infrastructure. | |
81 | +However, its hierarchical and traditional processes made them unfamiliar with | |
82 | +new software development techniques, such as CD. All our requests had to pass | |
83 | +through layers of bureaucracy before being answered; accessing their | |
84 | +infrastructure to deploy software was not different. The difficulties were | |
85 | +aggravated because the new SPB portal is an unprecedented platform in the | |
86 | +Brazilian government, with a complicated deployment process [4]. | |
87 | + | |
88 | +The project suffered significant interference from the Board of Directors | |
89 | +throughout time because the portal represents an interface between government | |
90 | +and society. In light of political interests, directors continually imposed | |
91 | +changes to the platform while ignoring our technical advice. In 2015, the Board | |
92 | +of Directors was changed and, with it, the vision of the project. New directors | |
93 | +had different political agendas, which affected the project's requirements | |
94 | +previously approved. | |
95 | + | |
96 | +In this context, we overcame two distinct challenges: (1) deconstructing the | |
97 | +widespread belief among government staff that any project in partnership with a | |
98 | +University is doomed to fail, and (2) dealing with bureaucracies involved in | |
99 | +the government deployment process. | |
100 | + | |
101 | +First, our development team was not the typical one, consisting mainly of | |
102 | +undergraduate interns supported by senior developers and designers, all | |
103 | +coordinated by two professors. Our unconventional team structure and | |
104 | +organization was considered as "unprofessional" by government standards with | |
105 | +regard to time and resource allocation, accountability, and team continuity. | |
106 | +On the government side, the SPB portal evolution was the first software | |
107 | +development collaboration involving universities and the MPOG staff, raising | |
108 | +disbelief. | |
109 | + | |
110 | +Second, our team approached software deployment differently from the | |
111 | +government. We believed frequent delivery is better for the project’s success. | |
112 | +In contrast, the MPOG is used to the idea of a single deployment at the end of | |
113 | +the project, and neither their bureaucratic structure nor their technical | |
114 | +expertise was conducive to this style of work. That was hampering the benefits | |
115 | +of the tool and preventing us from showing off the fruits of the project to | |
116 | +those responsible for evaluating it. | |
117 | + | |
118 | +<!-- AHA Moment!!!!! --> | |
119 | + | |
120 | +These challenges made our relationship with their staff strained, in particular | |
121 | +during the first year, and alerted us to the fact that they could cancel the | |
122 | +project at any time. The deployment limitation was the substantial technical | |
123 | +issue we could tackle in the short term. Thus, we worked to deploy the software | |
124 | +into our infrastructure and showed it to the government evaluators. This | |
125 | +strategy proved them we could efficiently provide new features, fulfill their | |
126 | +requirement delivery expectations, and incited them to demand that the latest | |
127 | +version be deployed in their infrastructure. Our CD approach generated more | |
128 | +pressure on the IT department responsible for the deployment routines. With | |
129 | +each CD cycle, we gradually built a new relationship among all parties and, by | |
130 | +the end of the project, we became active participants in the deploy operations | |
131 | +delivering quality software very frequently. | |
119 | 132 | |
120 | 133 | ## Our Continuous Delivery Pipeline |
121 | 134 | |
122 | -![Deployment Pipeline](figures/pipeline_2.png) | |
135 | +The SPB portal is an open CDE with additional social features, having 83 | |
136 | +software communities and 6,460 user accounts, mostly from government employees. | |
137 | +We built a system-of-systems [5] adapting and integrating five existing OSS | |
138 | +projects: Colab (www.github.com/colab), Noosfero (www.noosfero.org), Gitlab | |
139 | +(www.gitlab.com), Mezuro (www.mezuro.org), and Mailman (www.list.org). Colab | |
140 | +orchestrates these multiple systems using a plugin architecture and allowed us | |
141 | +to smoothly provide a unified interface to final users, including single | |
142 | +sign-on and global searches [1]. All these integrated systems involve a total | |
143 | +of 106,253 commits and 1,347,421 lines of code. | |
123 | 144 | |
124 | -Figure 1 represents our CD pipeline. A new feature might require changes on more | |
125 | -than one SPB integrated software project. Notice that each one of them could be | |
126 | -modified independently. The pipeline started when new code arrived. As it went | |
127 | -through each step, it was tested and improved until it finally reached the | |
128 | -production environment. At this point we would restart the pipeline to release | |
129 | -more new code. | |
145 | +![The SPB Deployment Pipeline.](figures/pipeline.png) | |
130 | 146 | |
131 | -### Automated Tests | |
147 | +Portal deployment follows a typical CD pipeline [6], adapted to our technical | |
148 | +and organizational context and the use of OSS best practices. As depicted in | |
149 | +Figure 1, it begins when a new feature is ready for deployment and ends when it | |
150 | +reaches production. | |
132 | 151 | |
133 | -The SPB portal consists of more than 10 integrated software projects and each | |
134 | -of them, as well as the entire platform, had to be tested. These software | |
135 | -components have their own test suite. Communication between all components is | |
136 | -orchestrated by Colab, a systems integration platform for web applications based | |
137 | -on a plugins architecture. Therefore, specific plugins were developed for | |
138 | -each portal software component, such as Gitlab and Mailman. Each plugin has its | |
139 | -own test suite and this set also worked as integration tests. | |
140 | 152 | |
141 | -Both unit and integration tests provided us the performance and security needed | |
142 | -to guarantee the stability for components and the platform. If any test suite | |
143 | -failed, by either a test error or coverage reduction below a certain threshold, | |
144 | -the pipeline stopped. Only when all tests passed, the pipeline proceeded to the | |
145 | -step of preparing the release. | |
153 | +### Automated Tests | |
154 | + | |
155 | +Each integrated system is a Colab plugin and had to be tested with its own test | |
156 | +suite. In addition, we also had to test the platform as a whole. Since the | |
157 | +plugins have their own test suites, they also assume a double role as both | |
158 | +plugin tests and as integration tests. If any test suite failed, by either a | |
159 | +test error or coverage reduction below a certain threshold, the process | |
160 | +stopped. Only when all tests passed, the pipeline proceeded to the next step. | |
146 | 161 | |
147 | 162 | ### Preparing a New Release |
148 | 163 | |
149 | -A SPB Portal release was composed of all its software components releases. Each | |
150 | -software component release was a git tag that referred to a specific feature or | |
151 | -bug fix. When all tests passed for a given component, we manually created a new | |
152 | -tag for it. Therefore, a new tag on any software component yielded a new SPB | |
153 | -Portal release. More precisely, SPB had a script that produced a single release | |
154 | -for the entire system based on each component tag. At the end of this process, | |
155 | -we started packaging. | |
164 | +A separate Git repository hosted each system. New software component releases | |
165 | +were tagged referencing a specific new feature or bug fix. SPB had its own Git | |
166 | +repository (www.softwarepublico.gov.br/gitlab/softwarepublico). An SPB portal | |
167 | +release was an aggregate of all its systems. When a new component release | |
168 | +passed all of the SPB integration tests, we manually created a corresponding | |
169 | +new tag in its repository. At the end of this process, we started packaging. | |
156 | 170 | |
157 | 171 | ### Packaging |
158 | 172 | |
159 | -The platform is running on the CentOS 7 GNU/Linux distribution. Basically, | |
160 | -packaging a software for that distribution has three steps: write the script | |
161 | -for the specific environment (RPM), build the package, and upload it to a | |
162 | -package repository. | |
163 | - | |
164 | -We decided to create our own packages for each software component for the following five | |
165 | -reasons: | |
166 | - | |
167 | -* Not all software was packaged by the community and those that existed were | |
168 | -outdated; | |
169 | -* Packaging makes it easy to manage the software on a given distribution; | |
170 | -* Packaging simplifies the deployment; | |
171 | -* Packaging follows the distribution's best practices and, | |
172 | -* Packaging allows configurations and permissions control. | |
173 | - | |
174 | -After creating a new tag for one component, the developers informed the | |
175 | -DevOps [3] team and the packaging process began. The three packaging steps | |
176 | -aforementioned were fully automated by a set of scripts. With all these scripts | |
177 | -running successfully, the new packages would be ready to be used by our | |
178 | -subsequent deployment scripts. | |
179 | - | |
180 | -### Validation Environment Deployment | |
181 | - | |
182 | -The Validation Environment (VE) is a replica of the Production Environment | |
183 | -(PE), with two exceptions: only the government officers and project leaders had | |
184 | -access to it and all the data was anonymised. To configure the environment, we | |
185 | -used a configuration management tool named Chef with Chake support (serverless | |
186 | -configuration for Chef). That tool maintained environment consistency | |
187 | -simplifying the deployment process. Additionally, the packages we built on the | |
188 | -last step were readily available to be used by the management tool. | |
189 | - | |
190 | -Government agents used the VE to validate new features and required | |
191 | -changes. Also, the VE was useful to verify the integrity of the entire portal | |
192 | -as part of the next step in the pipeline. | |
173 | +After creating a new tag, our DevOps team started the packaging process. | |
174 | +Packaging brings portability, simplifies deployment, and enables configuration | |
175 | +and permission control. Our approach involved building separate packages for | |
176 | +each system, in three fully automated steps: generating scripts for the | |
177 | +specific environment, building the package, and uploading it to a package | |
178 | +repository. When all ran successfully, the new packages would be ready and | |
179 | +available for our deployment scripts. | |
180 | + | |
181 | +### Validation Environment | |
182 | + | |
183 | +The Validation Environment (VE) is a replica of the Production Environment (PE) | |
184 | +with anonymised data, and access restricted to MPOG staff and our DevOps team. | |
185 | +To configure this environment, we used Chef (www.chef.io) and Chake, a | |
186 | +serverless configuration tool created by our team | |
187 | +(www.github.com/terceiro/chake) to maintain environment consistency, | |
188 | +simplifying the deployment process. | |
193 | 189 | |
194 | 190 | ### Acceptance Tests |
195 | 191 | |
196 | -After we deployed a new SPB Portal version in the VE, the government | |
197 | -agents were responsible for checking features and bug fixes required by them. If | |
198 | -the requirement analysts identified a problem, they would notify the developers via | |
199 | -comments on the SPB Portal's issue tracker. The problems were fixed and the | |
200 | -pipeline restarted from scratch. If everything was validated, we moved forward. | |
192 | +After a new SPB portal VE deployment, we used the environment to verify the | |
193 | +integrity of the entire portal. MPOG staff also checked the new features, | |
194 | +required changes, and bug fixes. If they identified a problem, they would | |
195 | +notify developers via comments on the SPB portal issue tracker, prompting the | |
196 | +team to fix it and restart the pipeline. Otherwise, we could move forward. | |
201 | 197 | |
202 | 198 | ### Production Environment Deployment |
203 | 199 | |
204 | -After the government finished the VE check and it was cleared for deployment, | |
205 | -we could finally begin the deployment to the Production Environment (PE). For this | |
206 | -we also used our configuration management tool, the same scripts and package | |
207 | -versions as in the VE. After the deploy was completed, both VE and PE | |
208 | -were running identical software. This was the point where new features and bug | |
209 | -fixes were finally available to the end users. | |
200 | +After the VE check, we could finally begin the deployment in production, with | |
201 | +the same configuration management tool, scripts, and package versions as in the | |
202 | +VE. After the deploy was completed, both VE and PE were identical, making new | |
203 | +features and bug fixes available to end users. | |
210 | 204 | |
211 | 205 | ## Benefits |
212 | 206 | |
213 | -Research points out many advantages of CD usage in industry[2], such as: | |
214 | -accelerated time to market, building the right product, productivity and | |
215 | -efficiency improvements, reliable releases and better customer satisfaction. | |
216 | -Working with the government, we noticed the following additional benefits. | |
207 | +![The evolution of SPB releases and the development team members distribution.](figures/CDReleaseAndTeamEvolution.png) | |
208 | + | |
209 | +CD brings many advantages such as accelerated time to market, building the | |
210 | +right product, productivity and efficiency improvements, stable releases, and | |
211 | +better customer satisfaction [2,3]. The charts presented in Figure 2 show these | |
212 | +benefits in our project and associates them with the creation of a DevOps team. | |
213 | +Over 30 months, we deployed 84 versions. Over 64% of the releases happened in | |
214 | +the last 12 months, after the creation of the DevOps team. Besides these | |
215 | +results, working with the government we noticed the following additional | |
216 | +benefits. | |
217 | + | |
218 | +### Strengthening Trust in the Relationship with the Government | |
219 | + | |
220 | +CD helped strengthen trust between developers and the MPOG staff. Before using | |
221 | +CD, they could validate features developed only at the end of the release | |
222 | +cycle, usually every four months. With the implementation of CD, intermediate | |
223 | +and candidate versions became available, allowing them to perform small | |
224 | +validations over time. Constant monitoring of the development work brought | |
225 | +greater assurance to the MPOG leaders and improved the interactions with our | |
226 | +team. | |
217 | 227 | |
218 | 228 | ### Responsiveness to Change |
219 | 229 | |
220 | -Responsiveness was one of the direct benefits of adopting the CD pipeline. The | |
221 | -ability to react quickly to changes requested by the government was vital for | |
222 | -the renewal of the project over the years. Every meeting with the government | |
223 | -leader resulted in new requirements, most of them motivated by political | |
224 | -needs. These constant changes in requirements and priorities caused discomfort | |
225 | -between the government and the development team. For | |
226 | -example, the government leader required a complete layout change because he suddenly decided to make a marketing campaign about the new | |
227 | -SPB portal. In future meetings, the government would use undelivered requirements as a means to justify the | |
228 | -lack of financial support, which was already approved in the first place. We believed that if we took too | |
229 | -long to attend their demands, the project would end. CD helped us keep the | |
230 | -production environment up-to-date, even with partial versions of a feature. That | |
231 | -way, we always had something to show on meetings, reducing anxiety to get the platform concluded. For our team, it made the developers more confident that the | |
232 | -project would last a little longer and they would not go looking for other | |
233 | -jobs. | |
230 | +Responsiveness was one of the direct benefits of adopting the CD pipeline. | |
231 | +Political forces may change requirements and priorities at any time. The | |
232 | +ability to react quickly to changes requested by the government was vital to | |
233 | +the project survival for 30 months. We noticed that if we took too long to meet | |
234 | +their demands, they could reduce financial support and even cancel the project. | |
235 | + | |
236 | +CD helped us keep the PE up-to-date, even with partial versions of a feature. | |
237 | +Therefore, we always had new results to present on meetings, easing their | |
238 | +concerns about expected final delivery. For our team, CD made developers always | |
239 | +confident that the project would last a little longer. | |
234 | 240 | |
235 | 241 | ### Shared Responsibility |
236 | 242 | |
237 | -Before the adoption of CD, the development team could not track what happened to the code | |
238 | -after its delivery, since government technicians were the only responsible | |
239 | -for deploying the project. The implementation of the referred | |
240 | -approach influenced developers on taking ownership of the project because it | |
241 | -made them feel equally responsible for what was getting into production. | |
242 | - | |
243 | -Interestingly, the CD pipeline had the same effect on the team of requirement analysts. | |
244 | -They were an active part of the pipeline and became more engaged on the whole process. | |
245 | -After the incorporation of the pipeline into the work process, analysts | |
246 | -became more active in opening and discussing issues during the platform evolution. | |
247 | -Additionally, developers worked to improve the CD pipeline | |
248 | -to speed up the process of making new features available in the production environment. | |
249 | - | |
250 | -### Synchronicity Between Government and Development | |
251 | - | |
252 | -Despite the positive impacts that the CD pipeline brought to the project, its | |
253 | -implementation was not easy at first. The CD pipeline performance | |
254 | -depended on the synchronicity between developers and government | |
255 | -analysts, so that the latter were prepared to start a step as soon as the | |
256 | -former concluded the previous step, and vice versa. Initially, this concern was not | |
257 | -contemplated in the agenda of the governmental team, which generated delays in | |
258 | -the validation of new features. This situation combined with | |
259 | -governmental bureaucracy (up to 3 days) to release access to the production | |
260 | -environment resulted in additional delays for the deployment step to begin. | |
261 | -This problem was softened when the analysts realized the impact of | |
262 | -these delays on the final product and decided to allocate the revisions in its | |
263 | -work schedule and to request the access to production in time. | |
264 | - | |
265 | -### Strengthening Trust in Work Relationship with the Government | |
266 | - | |
267 | -Continuous delivery was also a tool that helped to strengthen trust in the | |
268 | -relationship between developers and government analysts, as well as between the | |
269 | -analysts group and its superiors. Before using CD, analysts had access to the | |
270 | -features developed only at the end of the release, usually every four months. | |
271 | -However, this periodicity did not meet the requirements of their leaders, who | |
272 | -demanded monthly reports on the progress of the project. | |
273 | - | |
274 | -With the implementation of CD, intermediate and candidate versions became | |
275 | -available, allowing analysts to perform small validations over time. As they | |
276 | -validated functionalities and sent feedback to developers, patches were | |
277 | -developed and new versions were packaged and deployed to the VE quickly, | |
278 | -steadily, and reliably. The constant monitoring of the development work brought | |
279 | -greater security to the governmental nucleus and improved the interactions | |
280 | -with our development team. | |
281 | - | |
282 | -## Challenges | |
283 | - | |
284 | -We successfully built a CD pipeline. In the end, we took over the deployment | |
285 | -process from the government. That allowed us to survive into an unstable | |
286 | -political scenario. However, we recognize that many challenges still need to be | |
287 | -addressed by the industry and academia together. | |
243 | +According to the conventional MPOG process, the development team could not | |
244 | +track what happened to the code after its delivery, since their employees were | |
245 | +the only ones responsible for deployment. The implementation of CD made our | |
246 | +development team feel equally responsible for what was getting into production | |
247 | +and take ownership of the project [4]. | |
288 | 248 | |
289 | -### Build CD From Scratch | |
249 | +Interestingly, the CD pipeline had the same effect on the MPOG staff. They | |
250 | +became more engaged in the whole process, opening and discussing issues during | |
251 | +the evolution of the platform. Additionally, developers worked to improve the | |
252 | +CD pipeline and speed up the process of making new features available in the | |
253 | +production environment. | |
290 | 254 | |
291 | -Taking on CD responsibilities had a significant impact on the team. We did not | |
292 | -have the know-how and had little time to come up with a working pipeline. The | |
293 | -senior developers were crucial at this point. They came up with an initial | |
294 | -solution to get the team started. That already enabled us to automatize | |
295 | -deployment, even though the process was still rudimentary. We had to evolve our | |
296 | -solution on-the-fly. We dedicated a few developers to this task. | |
255 | +### Synchronization Between Government and Development | |
297 | 256 | |
298 | -Building a CD pipeline was hard in the beginning. More tools that provide | |
299 | -out-of-the-box standardized CD pipelines would be of great help for | |
300 | -inexperienced teams. Tools that track each step of the pipeline and organize | |
301 | -logs in a human-manageable way are necessary too. | |
257 | +The CD pipeline performance depended on the synchronization between our | |
258 | +development team and the MPOG staff: each party had to be prepared to take | |
259 | +action as soon as the other concluded a given task. Initially, the MPOG staff | |
260 | +did not contemplate this in their schedule which, combined with the bureaucracy | |
261 | +in providing access to the PE (up to 3 days), resulted in significant delays in | |
262 | +the validation of new features. The use of an explicit CD pipeline helped us to | |
263 | +identify critical points of delay, and increase our productivity. | |
302 | 264 | |
303 | -### Handling Inexperienced Teams | |
265 | +## Lessons Learned | |
304 | 266 | |
305 | -After the developers learned how CD worked, it was difficult to pass the | |
306 | -knowledge along to other teammates. We tried to mitigate this problem by | |
307 | -encouraging members to migrate to the DevOps team. We suggest further research | |
308 | -on how to effectively spread knowledge across inexperienced developers in a high | |
309 | -turnover scenario. | |
267 | +Due to the successful building of the CD pipeline, we not only overcame the | |
268 | +challenges we described above but also improved the MPOG deployment process and | |
269 | +kept the project alive until its successful conclusion. We now look at the | |
270 | +lessons we learned, which can be leveraged by readers in other situations. | |
310 | 271 | |
311 | -### Overcoming Mistrust | |
272 | +### Build CD From Scratch | |
273 | + | |
274 | +Taking on the responsibility for implementing CD impacted the whole team. Most | |
275 | +of our team members did not have CD know-how, and we had few working hours | |
276 | +available to build the initial version of the pipeline. The construction and | |
277 | +maintenance of the CD process were made possible by the key decisions to: | |
312 | 278 | |
313 | -In the project's beginning we struggled with deployment issues in the government | |
314 | -structure. We were in a paradoxical situation. The government demanded fast | |
315 | -deliveries but would not give access to their production infrastructure. After | |
316 | -some interactions with government agents, they created the VE as an isolated | |
317 | -replica of the PE in their own infrastructure. The government agents then | |
318 | -realised that it could be beneficial for the project if they granted us access | |
319 | -to part of the infrastructure. More research is required on development protocols and | |
320 | -policies to improve the relation between industry and government, specially | |
321 | -regarding CD. | |
279 | +1. _Select the most experienced senior developers and some advanced interns to | |
280 | +work on a specific DevOps team._ These senior developers used their experience | |
281 | +in OSS projects to craft an initial proposal for the deployment process. The | |
282 | +solution enabled us to automate the deployment, even though the process was, | |
283 | +initially, still rudimentary. | |
322 | 284 | |
323 | -## Acknowledgements | |
285 | +2. _Interchange team members and encourage teammates to migrate to the DevOps | |
286 | +team._ The benefits were twofold: mitigating the difficulty in sharing | |
287 | +knowledge between DevOps developers and feature developers, and evolving the | |
288 | +process on-the-fly. | |
289 | + | |
290 | +### Overcoming Mistrust | |
324 | 291 | |
325 | -We thank our colleagues, Lucas Kanashiro and Rafael Manzo, and this article's reviewers. | |
292 | +Adopting an unfamiliar approach requires trust. The government staff, | |
293 | +traditionally, expected and were prepared to validate and deploy a single | |
294 | +deliverable. However, the steady growth of SPB complexity made large | |
295 | +deliveries unsustainable. The CD approach was necessary [4]. Therefore, we | |
296 | +developed the following line of action to enable this new dynamics: | |
297 | + | |
298 | +1. _Demonstrate actual results, instead of simply reporting them._ Initially, | |
299 | +we did not have access to the government infrastructure, so we created our own | |
300 | +validation environment. Thus, we were able to follow the CD pipeline until | |
301 | +production deployment, when we faced two problems. First, our pace of | |
302 | +intermediate deliveries was faster than the deployment to production by the | |
303 | +MPOG staff. Second, specific issues of the governmental infrastructure made | |
304 | +some validated features not work as expected in the PE. That situation gave us | |
305 | +arguments to negotiate access to the PE. | |
306 | + | |
307 | +2. _Make project management transparent and collaborative for government | |
308 | +staff._ Allowing the MPOG staff to track our development process showed them we | |
309 | +were fulfilling our commitments. They started to interact more actively in the | |
310 | +generation of versions and became involved in the CD. After understanding it, | |
311 | +the government staff helped us negotiate access to a VE with the MPOG leaders, | |
312 | +creating an isolated replica of the PE. | |
313 | + | |
314 | +3. _Gain the confidence of government staff._ With the PE replica, we were able | |
315 | +to run the entire pipeline and win the trust of the MPOG staff involved in the | |
316 | +process. They saw the mobilization and responsiveness of our team to generate | |
317 | +each new version package. They also recognized the quality of our work and | |
318 | +deployment process. In the end, the government staff realized that it would be | |
319 | +beneficial for the project if they granted us access to the infrastructure, | |
320 | +both VE and PE. | |
321 | + | |
322 | +In summary, we encourage the use of a well-thought CD pipeline as a means to | |
323 | +gain trust in software development projects with government and large | |
324 | +organizations as well. | |
326 | 325 | |
327 | 326 | ## References |
328 | 327 | |
329 | -1. G. Booch, A. W. Brown, "Collaborative Development Environments", in Advances in Computers, vol. 59, 2003, pp. 1–27. | |
330 | -2. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54. | |
331 | -3. Davis, Jennifer and Daniels, Katherine, Effective DevOps: building a culture of collaboration, affinity, and tooling at scale, 2016, " O'Reilly Media, Inc." | |
328 | +1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", OpenSym, 2017. | |
329 | +1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, 32 (2) 2015. | |
330 | +1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", ICSE, 2016. | |
331 | +1. M. Shahin, M. A. Babar, and L. Zhu. “The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives”. ESEM, 2016. | |
332 | +1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, 2015. | |
333 | +1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley, 2010. | |
332 | 334 | ... | ... |
211 KB
ieeeSW/releaseEng3/figures/pipeCD.png
44 KB
ieeeSW/releaseEng3/figures/pipeline.dia
No preview for this file type
ieeeSW/releaseEng3/figures/pipeline.png