Commit 2c513948401f95606ccf4baa04969210e7905f28
1 parent
614bbeec
Exists in
master
and in
3 other branches
Review from all authors
Showing
1 changed file
with
190 additions
and
208 deletions
Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
1 | --- | 1 | --- |
2 | title: | | 2 | title: | |
3 | - | __Continuous Delivery__ | ||
4 | - | Building Trust in a Large-scale, Complex Government Organization | 3 | + | __Continuous Delivery:__ |
4 | + | Building Trust in a Large-scale, Complex | ||
5 | + | Government Organization | ||
5 | papersize: a4 | 6 | papersize: a4 |
6 | geometry: "left=1in,right=1.5in" | 7 | geometry: "left=1in,right=1.5in" |
7 | --- | 8 | --- |
8 | 9 | ||
9 | ## Authors | 10 | ## Authors |
10 | 11 | ||
11 | -__Rodrigo Siqueira__ is a masters student at IME - The Institute of Mathematics | ||
12 | -and Statistics of the Sao Paulo University. His research interests include | ||
13 | -software engineering, operating system and computer architecture. He worked for | ||
14 | -two years with the Brazilian federal government as a coach and developer. | ||
15 | -Contact him at siqueira@ime.usp.br. | ||
16 | - | ||
17 | -__Diego Camarinha__ is a masters student at IME - The Institute of Mathematics | ||
18 | -and Statistics of the Sao Paulo University. His research interests include | ||
19 | -software engineering, computer networks and source code metrics. He worked for | ||
20 | -two years with the Brazilian federal government as a senior developer. Contact | ||
21 | -him at diegoamc@ime.usp.br. | ||
22 | - | ||
23 | -__Melissa Wen__ is a software developer. She worked on SPB project as a senior | ||
24 | -developer and also served as professor of Computer Science at UFBA - The Federal | ||
25 | -University of Bahia. Her areas of interest include software engineering and open | ||
26 | -source software development. Contact her at melissa.srw@gmail.com . | ||
27 | - | ||
28 | -__Paulo Meirelles__ received his Ph.D. in Computer Science from the Institute of | ||
29 | -Mathematics and Statistics at the University of São Paulo. He is full-time | ||
30 | -Professor at the University of Brasilia, and coordinated the new SPB portal | ||
31 | -project. His research interest areas are: Free Software, Agile methods, Static | ||
32 | -analysis, and Source code metrics. Contact him at paulormm@ime.usp.br. | ||
33 | - | ||
34 | -__Fabio Kon__ is Full Professor of the Department of Computer Science of the | ||
35 | -University of São Paulo. (...) .His research interest areas are: (...). Contact | ||
36 | -him at fabio.kon@ime.usp.br. | ||
37 | - | ||
38 | -<!-- Fabio: Esse resumo está não usual porque ele não resume o que o artigo apresenta. Ele serve como uma motivação apenas. Mas talvez podemos deixar assim pra ver o que acontece --> | 12 | +__Rodrigo Siqueira__ is a Computer Science MSc student at the Institute of |
13 | +Mathematics and Statistics of the University of São Paulo (IME-USP). His | ||
14 | +research interests include software engineering, operating system, and computer | ||
15 | +architecture. He worked for two years with the Brazilian federal government as | ||
16 | +a coach and developer. Contact him at siqueira@ime.usp.br. | ||
17 | + | ||
18 | +__Diego Camarinha__ is a CS MSc masters student at IME-USP. His research | ||
19 | +interests include software engineering, computer networks, and source code | ||
20 | +metrics. He worked for two years with the Brazilian federal government as a | ||
21 | +senior developer. Contact him at diegoamc@ime.usp.br. | ||
22 | + | ||
23 | +__Melissa Wen__ is a software developer. She worked on the SPB project as a | ||
24 | +senior developer and also served as a Computer Science lecturer at the Federal | ||
25 | +University of Bahia. Her areas of interest include software engineering and | ||
26 | +open source software development. Contact her at melissa.srw@gmail.com. | ||
27 | + | ||
28 | +__Paulo Meirelles__ received his Ph.D. in Computer Science from IME-USP. He is | ||
29 | +a full-time Professor at the University of Brasilia and coordinated the new SPB | ||
30 | +portal project. His research interests are: Free Software, Agile methods, | ||
31 | +Static analysis, and Source code metrics. Contact him at paulormm@ime.usp.br. | ||
32 | + | ||
33 | +__Fabio Kon__ is a Full Professor of Computer Science at IME-USP and | ||
34 | +Editor-in-Chief of the SpringerOpen Journal of Internet Services and | ||
35 | +Applications. His research interests include Agile Software Development, Smart | ||
36 | +Cities, and Distributed Systems. Contact him at fabio.kon@ime.usp.br. | ||
37 | + | ||
39 | ## Abstract | 38 | ## Abstract |
40 | 39 | ||
41 | For many software development teams, the first aspects that come to mind | 40 | For many software development teams, the first aspects that come to mind |
42 | regarding Continuous Delivery (CD) are the operational challenges and the | 41 | regarding Continuous Delivery (CD) are the operational challenges and the |
43 | competitive benefits. In our experience, CD was much more: it was a survival | 42 | competitive benefits. In our experience, CD was much more: it was a survival |
44 | -technique. This article presents how and why we applied CD in a large Brazilian | ||
45 | -government project for the development of a Collaborative Development | ||
46 | -Environment (CDE), sharing the challenges we faced and the strategies used to | ||
47 | -overcome them. ( >> FECHAR COM FRASE DE IMPACTO <<<). | 43 | +technique. This article presents how and why we applied CD in a large |
44 | +governmental project for the development of a Collaborative Development | ||
45 | +Environment. We share the challenges we faced and the strategies we used to | ||
46 | +overcome them. The article concludes with a set of lessons learned that can be | ||
47 | +valuable for readers in a variety of situations. | ||
48 | 48 | ||
49 | ## Introduction | 49 | ## Introduction |
50 | 50 | ||
51 | -From 2014 to 2016, we were part of a team that worked on a thirty-month-long | ||
52 | -Brazilian government project to modernize the Brazilian Public Software (SPB) | ||
53 | -portal (www.softwarepublico.gov.br) [1]. This project was a partnership between | ||
54 | -the Ministry of Planning, Budget, and Management and two public universities: | 51 | +From 2014 to 2016, we worked on a thirty-month-long Brazilian government |
52 | +project to modernize the Brazilian Public Software (SPB) portal | ||
53 | +(www.softwarepublico.gov.br) [1]. This project was a partnership between the | ||
54 | +_Ministry of Planning, Budget, and Management_ and two public universities: | ||
55 | University of Brasília and University of São Paulo. | 55 | University of Brasília and University of São Paulo. |
56 | 56 | ||
57 | During this time, the SPB portal evolved into a Collaborative Development | 57 | During this time, the SPB portal evolved into a Collaborative Development |
58 | -Environment (CDE) which brought significant benefits for the government and the | ||
59 | -society: The government could minimize bureaucracy and software development | 58 | +Environment (CDE), which brought significant benefits for the government and |
59 | +the society. The government could minimize bureaucracy and software development | ||
60 | costs, by reusing the same set of applications across different government | 60 | costs, by reusing the same set of applications across different government |
61 | -agencies; society could more transparently follow government expenses and | 61 | +Agencies. Society could more transparently follow government expenses and |
62 | contribute to project communities. | 62 | contribute to project communities. |
63 | 63 | ||
64 | In this article, we discuss the use of Continuous Delivery (CD) during our | 64 | In this article, we discuss the use of Continuous Delivery (CD) during our |
65 | -experience as the academic partner in this project. We focus on how we managed | ||
66 | -to implement CD in a large institution with traditional values and how CD | ||
67 | -helped to build trust between the government and the university development | ||
68 | -team. CD enabled us to show our progress and to earn the government’s | ||
69 | -confidence that we could adequately fulfill their requests, becoming an | ||
70 | -essential aspect of our interaction with them. According to this experience, | ||
71 | -the use of CD as a tool to build such trust relationships is yet another of its | ||
72 | -benefits [2,3]. | 65 | +experience as the academic partner in this project. We show how we implemented |
66 | +CD in a large institution with traditional, waterfall-like values and how CD | ||
67 | +helped to build trust between the government and the university team. CD | ||
68 | +enabled us to show our progress and to earn the government’s confidence that we | ||
69 | +could adequately fulfill their requests, becoming an essential aspect of our | ||
70 | +interaction. According to this experience, the use of CD as a tool to build | ||
71 | +trust relationships is yet another of its benefits [2,3]. | ||
73 | 72 | ||
74 | ## Context | 73 | ## Context |
75 | 74 | ||
76 | SPB is a governmental program created to foster sharing and collaboration on | 75 | SPB is a governmental program created to foster sharing and collaboration on |
77 | Open Source Software (OSS) development for the Brazilian public administration. | 76 | Open Source Software (OSS) development for the Brazilian public administration. |
78 | -In their own projects, the Ministry managed both software requirements and | ||
79 | -server infrastructure. However, its hierarchical and traditional processes made | ||
80 | -them unfamiliar with new software development techniques, such as CD. Any of | ||
81 | -our requests had to pass through layers of bureaucracy before being answered; | ||
82 | -accessing their infrastructure to deploy updated software was not different. | ||
83 | -The difficulties were aggravated because the new SPB portal is an unprecedented | ||
84 | -platform in the Brazilian government, with a complicated deployment process [4]. | ||
85 | - | ||
86 | -The project suffered significant interference from the board of directors | 77 | +The ministry managed both software requirements and server infrastructure. |
78 | +However, its hierarchical and traditional processes made them unfamiliar with | ||
79 | +new software development techniques, such as CD. All our requests had to pass | ||
80 | +through layers of bureaucracy before being answered; accessing their | ||
81 | +infrastructure to deploy software was not different. The difficulties were | ||
82 | +aggravated because the new SPB portal is an unprecedented platform in the | ||
83 | +Brazilian government, with a complicated deployment process [4]. | ||
84 | + | ||
85 | +The project suffered significant interference from the Board of Directors | ||
87 | throughout time because the portal represents an interface between government | 86 | throughout time because the portal represents an interface between government |
88 | and society. In light of political interests, directors continually imposed | 87 | and society. In light of political interests, directors continually imposed |
89 | -changes to the platform while ignoring our technical advice. In 2015, the board | ||
90 | -of directors was changed and, with it, the vision of the project. New directors | ||
91 | -had different political agendas which affected the project's requirements | 88 | +changes to the platform while ignoring our technical advice. In 2015, the Board |
89 | +of Directors was changed and, with it, the vision of the project. New directors | ||
90 | +had different political agendas, which affected the project's requirements | ||
92 | previously approved. | 91 | previously approved. |
93 | 92 | ||
94 | -<!--- | ||
95 | -mudar em todas as ocorrências de agents por staff, mas checar com o Fabio. | ||
96 | ---> | ||
97 | - | ||
98 | -In this context, we overcame two distinct challenges: (1) deconstructing | ||
99 | -the widespread belief among government staff that any project in partnership | ||
100 | -with a University is doomed to fail, and (2) dealing with bureaucracies | ||
101 | -involved in the Ministry deployment process. | 93 | +In this context, we overcame two distinct challenges: (1) deconstructing the |
94 | +widespread belief among government staff that any project in partnership with a | ||
95 | +University is doomed to fail, and (2) dealing with bureaucracies involved in | ||
96 | +the government deployment process. | ||
102 | 97 | ||
103 | -First, our development team was not from a typical company, consisting mainly | ||
104 | -of undergraduate interns supported by senior developers and designers, all | 98 | +First, our development team was not the typical one, consisting mainly of |
99 | +undergraduate interns supported by senior developers and designers, all | ||
105 | coordinated by two professors. Our unconventional team structure and | 100 | coordinated by two professors. Our unconventional team structure and |
106 | -organization might be considered as "unprofessional" by Ministry standards with | ||
107 | -regard to time and resource allocation, accountability, and team continuity . | 101 | +organization was considered as "unprofessional" by ministry standards with |
102 | +regard to time and resource allocation, accountability, and team continuity. | ||
108 | On the government side, the SPB portal evolution was the first software | 103 | On the government side, the SPB portal evolution was the first software |
109 | -development collaboration involving universities and the Ministry staff, | 104 | +development collaboration involving universities and the ministry staff, |
110 | raising disbelief. | 105 | raising disbelief. |
111 | 106 | ||
112 | Second, our team approached software deployment differently from the | 107 | Second, our team approached software deployment differently from the |
113 | -Ministry. We believed frequent delivery is better for the project’s success. | ||
114 | -In contrast, the Ministry is used to the idea of a single deployment at the end | 108 | +government. We believed frequent delivery is better for the project’s success. |
109 | +In contrast, the ministry is used to the idea of a single deployment at the end | ||
115 | of the project, and neither their bureaucratic structure nor their technical | 110 | of the project, and neither their bureaucratic structure nor their technical |
116 | -expertise was conducive to this style of work. That ended up hampering the | ||
117 | -benefits of the tool and preventing us from showing off the fruits of the | ||
118 | -project to those responsible for evaluating it. | 111 | +expertise was conducive to this style of work. That was hampering the benefits |
112 | +of the tool and preventing us from showing off the fruits of the project to | ||
113 | +those responsible for evaluating it. | ||
119 | 114 | ||
120 | <!-- AHA Moment!!!!! --> | 115 | <!-- AHA Moment!!!!! --> |
121 | 116 | ||
122 | -These challenges made our relationship with the Ministry staff tense, in | ||
123 | -particular during the first year, and alerted us to the fact that they could | ||
124 | -cancel the project at any time. The deployment limitation was the substantial | ||
125 | -technical issue we could tackle in the short term. As a result, we worked to | ||
126 | -deploy one version of the project onto our infrastructure and showed it to the | ||
127 | -government evaluators. This strategy proved them we could efficiently provide | ||
128 | -new features, fulfill their expectations regarding the delivery of the | ||
129 | -requirements, and incited them to demand that the latest version be deployed in | ||
130 | -the Ministry infrastructure. Our CD approach generated more pressure on the IT | ||
131 | -department responsible for the deployment routines. With each CD cycle, we | ||
132 | -gradually built a new relationship among all parties and, by the end of the | ||
133 | -project, we became active participants in the deploy operations. | 117 | +These challenges made our relationship with their staff strained, in particular |
118 | +during the first year, and alerted us to the fact that they could cancel the | ||
119 | +project at any time. The deployment limitation was the substantial technical | ||
120 | +issue we could tackle in the short term. Thus, we worked to deploy the software | ||
121 | +into our infrastructure and showed it to the government evaluators. This | ||
122 | +strategy proved them we could efficiently provide new features, fulfill their | ||
123 | +requirement delivery expectations, and incited them to demand that the latest | ||
124 | +version be deployed in their infrastructure. Our CD approach generated more | ||
125 | +pressure on the IT department responsible for the deployment routines. With | ||
126 | +each CD cycle, we gradually built a new relationship among all parties and, by | ||
127 | +the end of the project, we became active participants in the deploy operations | ||
128 | +delivering quality software very frequently. | ||
134 | 129 | ||
135 | ## Our Continuous Delivery Pipeline | 130 | ## Our Continuous Delivery Pipeline |
136 | 131 | ||
137 | -SPB portal is a CDE with additional social features, having 83 software | ||
138 | -communities and 6,460 users, mostly government employees. We built | ||
139 | -system-of-systems adapting and integrating five existing OSS projects: Colab | ||
140 | -(www.github.com/colab), Noosfero (www.noosfero.org), Gitlab (www.gitlab.com), | ||
141 | -Mezuro (www.mezuro.org), and | ||
142 | -Mailman (www.list.org). Colab orchestrates these multiple systems using a | ||
143 | -plugin architecture and allows us to smoothly provide a unified interface for | ||
144 | -final users, including single sign-on and global searches [1]. All these | ||
145 | -integrated systems represents a total of 106.253 commits and 1.347.421 lines of | ||
146 | -code. | 132 | +The SPB portal is an open CDE with additional social features, having 83 |
133 | +software communities and 6,460 user accounts, mostly from government employees. | ||
134 | +We built a system-of-systems [5] adapting and integrating five existing OSS | ||
135 | +projects: Colab (www.github.com/colab), Noosfero (www.noosfero.org), Gitlab | ||
136 | +(www.gitlab.com), Mezuro (www.mezuro.org), and Mailman (www.list.org). Colab | ||
137 | +orchestrates these multiple systems using a plugin architecture and allowed us | ||
138 | +to smoothly provide a unified interface to final users, including SSO and | ||
139 | +global searches [1]. All these integrated systems involve a total of 106,253 | ||
140 | +commits and 1,347,421 lines of code. | ||
147 | 141 | ||
148 |  | 142 |  |
149 | 143 | ||
150 | -The portal deployment follows a typical CD pipeline [5], adapted to the | ||
151 | -technical and organizational context of our project and the use of OSS best | ||
152 | -practices, as presented in Figure 1. It started when new code arrived; and when | ||
153 | -it reaches the production environment, we would restart the pipeline to release | ||
154 | -new versions. | 144 | +Portal deployment follows a typical CD pipeline [6], adapted to our technical |
145 | +and organizational context and the use of OSS best practices. As depicted in | ||
146 | +Figure 1, it begins when a new feature is ready for deployment and ends when it | ||
147 | +reaches production. | ||
148 | + | ||
155 | 149 | ||
156 | ### Automated Tests | 150 | ### Automated Tests |
157 | 151 | ||
158 | -In practice, each integrated system is a Colab plugin and had to be tested with | ||
159 | -its own test suite. However, this checking was not enough: we also had to test | ||
160 | -the platform as a whole. Since the plugins have their own test suites, they | ||
161 | -also assume a double role as both plugin tests and as integration tests. If any | ||
162 | -test suite failed, by either a test error or coverage reduction below a certain | ||
163 | -threshold, the process stopped. Only when all tests passed, the pipeline | ||
164 | -proceeded to the step of release preparation. | 152 | +Each integrated system is a Colab plugin and had to be tested with its own test |
153 | +suite. In addition, we also had to test the platform as a whole. Since the | ||
154 | +plugins have their own test suites, they also assume a double role as both | ||
155 | +plugin tests and as integration tests. If any test suite failed, by either a | ||
156 | +test error or coverage reduction below a certain threshold, the process | ||
157 | +stopped. Only when all tests passed, the pipeline proceeded to the next step. | ||
165 | 158 | ||
166 | ### Preparing a New Release | 159 | ### Preparing a New Release |
167 | 160 | ||
168 | -Each software component was hosted in a separate Git repository. A new component | ||
169 | -release was tagged referencing a specific new feature or bug fix. SPB | ||
170 | -had its own Git repository (www.softwarepublico.gov.br/gitlab/softwarepublico). | ||
171 | -An SPB portal release was an aggregate of versions of all its components. | ||
172 | -When a new component release passed all of the SPB integration tests, we | ||
173 | -manually created a corresponding new tag in its repository. Therefore, a new | ||
174 | -tag on any software component yielded a new SPB portal release. At the end of | ||
175 | -this process, we started packaging. | 161 | +A separate Git repository hosted each software component. New component |
162 | +releases were tagged referencing a specific new feature or bug fix. SPB had its | ||
163 | +own Git repository (www.softwarepublico.gov.br/gitlab/softwarepublico). An SPB | ||
164 | +portal release was an aggregate of all its components. When a new component | ||
165 | +release passed all of the SPB integration tests, we manually created a | ||
166 | +corresponding new tag in its repository. At the end of this process, we started | ||
167 | +packaging. | ||
176 | 168 | ||
177 | ### Packaging | 169 | ### Packaging |
178 | 170 | ||
179 | -After creating a new tag for a system, our DevOps team starts the packaging | ||
180 | -process. Packaging brings portability, simplifies deployment, and enables | ||
181 | -configuration and permission control. Our packaging software approach involves | ||
182 | -building separate packages for each system, in three steps: writing the script | ||
183 | -for the specific environment, building the package, and uploading it to a | ||
184 | -package repository. A set of scripts fully automated these steps. When all them | ||
185 | -ran successfully, the new packages would be ready and available for our | ||
186 | -deployment scripts. | 171 | +After creating a new tag, our DevOps team started the packaging process. |
172 | +Packaging brings portability, simplifies deployment, and enables configuration | ||
173 | +and permission control. Our approach involved building separate packages for | ||
174 | +each system, in three fully automated steps: generating scripts for the | ||
175 | +specific environment, building the package, and uploading it to a package | ||
176 | +repository. When all ran successfully, the new packages would be ready and | ||
177 | +available for our deployment scripts. | ||
187 | 178 | ||
188 | -### Validation Environment Deployment | 179 | +### Validation Environment |
189 | 180 | ||
190 | The Validation Environment (VE) is a replica of the Production Environment (PE) | 181 | The Validation Environment (VE) is a replica of the Production Environment (PE) |
191 | -with anonymised data, and access restricted to Ministry staff and our DevOps | 182 | +with anonymised data, and access restricted to ministry staff and our DevOps |
192 | team. To configure this environment, we used Chef (www.chef.io) and Chake, a | 183 | team. To configure this environment, we used Chef (www.chef.io) and Chake, a |
193 | -serverless configuration tool created by our team | ||
194 | -(www.github.com/terceiro/chake). This tool maintained environment consistency, | 184 | +serverless configuration tool created by our team |
185 | +(www.github.com/terceiro/chake) to maintain environment consistency, | ||
195 | simplifying the deployment process. | 186 | simplifying the deployment process. |
196 | 187 | ||
197 | ### Acceptance Tests | 188 | ### Acceptance Tests |
198 | 189 | ||
199 | -After a new SPB portal deployment in the VE, we used the environment to verify | ||
200 | -the integrity of the entire portal. In this step, Ministry staff also checked | ||
201 | -the new features, required changes, and bug fixes. If they identified a | ||
202 | -problem, they would notify the developers via comments on the SPB portal issue | ||
203 | -tracker, prompting the team to fix it and restart the pipeline. Otherwise, we | ||
204 | -could move forward. | 190 | +After a new SPB portal VE deployment, we used the environment to verify the |
191 | +integrity of the entire portal. Government staff also checked the new features, | ||
192 | +required changes, and bug fixes. If they identified a problem, they would | ||
193 | +notify developers via comments on the SPB portal issue tracker, prompting the | ||
194 | +team to fix it and restart the pipeline. Otherwise, we could move forward. | ||
205 | 195 | ||
206 | ### Production Environment Deployment | 196 | ### Production Environment Deployment |
207 | 197 | ||
208 | After the VE check, we could finally begin the deployment in production, with | 198 | After the VE check, we could finally begin the deployment in production, with |
209 | the same configuration management tool, scripts, and package versions as in the | 199 | the same configuration management tool, scripts, and package versions as in the |
210 | -VE. After the deploy was completed, both VE and PE were identical. At that | ||
211 | -point, new features and bug fixes were finally available to end users. | 200 | +VE. After the deploy was completed, both VE and PE were identical, making new |
201 | +features and bug fixes available to end users. | ||
212 | 202 | ||
213 | ## Benefits | 203 | ## Benefits |
214 | 204 | ||
215 | CD brings many advantages such as accelerated time to market, building the | 205 | CD brings many advantages such as accelerated time to market, building the |
216 | right product, productivity and efficiency improvements, stable releases, and | 206 | right product, productivity and efficiency improvements, stable releases, and |
217 | better customer satisfaction [2,3]. The charts presented in Figure 2 show these | 207 | better customer satisfaction [2,3]. The charts presented in Figure 2 show these |
218 | -benefits of CD in our project and associates them with the creation of a DevOps | ||
219 | -team. In the time of 30 months, we deployed a total of 84 versions. Over 64% of | ||
220 | -these activities happened in the second half of 2015, with the DevOps team | ||
221 | -creation. Besides these results, working with the government, we noticed the | ||
222 | -following additional benefits. | 208 | +benefits in our project and associates them with the creation of a DevOps team. |
209 | +Over 30 months, we deployed 84 versions. Over 64% of the releases happened in | ||
210 | +the last 12 months, after the creation of the DevOps team. Besides these | ||
211 | +results, working with the government we noticed the following additional | ||
212 | +benefits. | ||
223 | 213 | ||
224 | - | 214 | + |
225 | 215 | ||
226 | ### Strengthening Trust in the Relationship with the Government | 216 | ### Strengthening Trust in the Relationship with the Government |
227 | 217 | ||
228 | -CD helped strengthen trust in the relationship between developers and the | ||
229 | -Ministry staff. Before using CD, they could validate features developed only at | ||
230 | -the end of the release cycle, usually every four months. With the | ||
231 | -implementation of CD, intermediate and candidate versions became available, | ||
232 | -allowing them to perform small validations over time. Constant monitoring of | ||
233 | -the development work brought greater security to the Ministry leaders and | ||
234 | -improved the interactions with our team. | 218 | +CD helped strengthen trust between developers and the government staff. Before |
219 | +using CD, they could validate features developed only at the end of the release | ||
220 | +cycle, usually every four months. With the implementation of CD, intermediate | ||
221 | +and candidate versions became available, allowing them to perform small | ||
222 | +validations over time. Constant monitoring of the development work brought | ||
223 | +greater assurance to the ministry leaders and improved the interactions with | ||
224 | +our team. | ||
235 | 225 | ||
236 | ### Responsiveness to Change | 226 | ### Responsiveness to Change |
237 | 227 | ||
238 | Responsiveness was one of the direct benefits of adopting the CD pipeline. | 228 | Responsiveness was one of the direct benefits of adopting the CD pipeline. |
239 | -Political interests may change requirements and priorities at any time. The | ||
240 | -ability to react quickly to changes requested by the Ministry was vital to the | ||
241 | -project’s survival for 30 months. We noticed that if we took too long to meet | ||
242 | -their demands, they would threaten to reduce financial support and even cancel | ||
243 | -the project. | 229 | +Political forces may change requirements and priorities at any time. The |
230 | +ability to react quickly to changes requested by the government was vital to | ||
231 | +the project survival for 30 months. We noticed that if we took too long to meet | ||
232 | +their demands, they could reduce financial support and even cancel the project. | ||
244 | 233 | ||
245 | CD helped us keep the PE up-to-date, even with partial versions of a feature. | 234 | CD helped us keep the PE up-to-date, even with partial versions of a feature. |
246 | -Therefore, we always had something to show on meetings, easing their concerns | ||
247 | -about the final delivery of the platform. For our team, CD made developers more | 235 | +Therefore, we always had new results to present on meetings, easing their |
236 | +concerns About expected final delivery. For our team, CD made developers always | ||
248 | confident that the project would last a little longer. | 237 | confident that the project would last a little longer. |
249 | 238 | ||
250 | ### Shared Responsibility | 239 | ### Shared Responsibility |
251 | 240 | ||
252 | -According to the conventional Ministry process, the development team could not | ||
253 | -track what happened to the code after its delivery, since their staff was the | ||
254 | -only ones responsible for deployment. The implementation of CD made our | 241 | +According to the conventional ministry process, the development team could not |
242 | +track what happened to the code after its delivery, since their employees were | ||
243 | +the only ones responsible for deployment. The implementation of CD made our | ||
255 | development team feel equally responsible for what was getting into production | 244 | development team feel equally responsible for what was getting into production |
256 | and take ownership of the project [4]. | 245 | and take ownership of the project [4]. |
257 | 246 | ||
258 | -Interestingly, the CD pipeline had the same effect on the Ministry staff. They | 247 | +Interestingly, the CD pipeline had the same effect on the ministry staff. They |
259 | became more engaged in the whole process, opening and discussing issues during | 248 | became more engaged in the whole process, opening and discussing issues during |
260 | the evolution of the platform. Additionally, developers worked to improve the | 249 | the evolution of the platform. Additionally, developers worked to improve the |
261 | CD pipeline and speed up the process of making new features available in the | 250 | CD pipeline and speed up the process of making new features available in the |
@@ -264,39 +253,33 @@ production environment. | @@ -264,39 +253,33 @@ production environment. | ||
264 | ### Synchronization Between Government and Development | 253 | ### Synchronization Between Government and Development |
265 | 254 | ||
266 | The CD pipeline performance depended on the synchronization between our | 255 | The CD pipeline performance depended on the synchronization between our |
267 | -development team and the Ministry staff: each party had to be prepared to take | ||
268 | -action as soon as the other concluded a given task. Initially, the Ministry | 256 | +development team and the ministry staff: each party had to be prepared to take |
257 | +action as soon as the other concluded a given task. Initially, the ministry | ||
269 | staff did not contemplate this in their schedule which, combined with the | 258 | staff did not contemplate this in their schedule which, combined with the |
270 | bureaucracy in providing access to the PE (up to 3 days), resulted in | 259 | bureaucracy in providing access to the PE (up to 3 days), resulted in |
271 | significant delays in the validation of new features. The use of an explicit CD | 260 | significant delays in the validation of new features. The use of an explicit CD |
272 | pipeline helped us to identify critical points of delay, and increase our | 261 | pipeline helped us to identify critical points of delay, and increase our |
273 | productivity. | 262 | productivity. |
274 | 263 | ||
275 | -<!--- | ||
276 | -Fabio sugeriu Lessons Learned, mas vamos mostrar exemplos da revista para ele olhar. | ||
277 | ---> | ||
278 | - | ||
279 | ## Lessons Learned | 264 | ## Lessons Learned |
280 | 265 | ||
281 | -<!--- | ||
282 | -(>>>FABIO FAZER RELAÇÃO COM CHALLENGES DA SEÇÃO CONTEXTO<<<). | ||
283 | ---> | ||
284 | -Due to the successful building of the CD pipeline, we improved the Ministry | ||
285 | -deployment process and kept the project alive. We now look at the lessons | ||
286 | -learned. | 266 | +Due to the successful building of the CD pipeline, we not only overcame the |
267 | +challenges we described above but also improved the ministry deployment process | ||
268 | +and kept the project alive until its successful conclusion. We now look at the | ||
269 | +lessons we learned, which can be leveraged by readers in other situations. | ||
287 | 270 | ||
288 | ### Build CD From Scratch | 271 | ### Build CD From Scratch |
289 | 272 | ||
290 | Taking on the responsibility for implementing CD impacted the whole team. Most | 273 | Taking on the responsibility for implementing CD impacted the whole team. Most |
291 | of our team members did not have CD know-how, and we had few working hours | 274 | of our team members did not have CD know-how, and we had few working hours |
292 | -available to build the pipeline. The construction and maintenance of the CD | ||
293 | -process were made possible by the key decisions to: | 275 | +available to build the initial version of the pipeline. The construction and |
276 | +maintenance of the CD process were made possible by the key decisions to: | ||
294 | 277 | ||
295 | -1. _Select the most experienced senior developers and some advanced students of | ||
296 | -the project to work on a specific DevOps team._ These senior developers used | ||
297 | -their experience in OSS projects to craft an initial proposal for the | ||
298 | -deployment process. The solution enabled us to automate the deployment, even | ||
299 | -though the process was, initially, still rudimentary. | 278 | +1. _Select the most experienced senior developers and some advanced students to |
279 | +work on a specific DevOps team._These senior developers used their experience | ||
280 | +in OSS projects to craft an initial proposal for the deployment process. The | ||
281 | +solution enabled us to automate the deployment, even though the process was, | ||
282 | +initially, still rudimentary. | ||
300 | 283 | ||
301 | 2. _Interchange team members and encourage teammates to migrate to the DevOps | 284 | 2. _Interchange team members and encourage teammates to migrate to the DevOps |
302 | team._ The benefits were twofold: mitigating the difficulty in sharing | 285 | team._ The benefits were twofold: mitigating the difficulty in sharing |
@@ -305,46 +288,45 @@ process on-the-fly. | @@ -305,46 +288,45 @@ process on-the-fly. | ||
305 | 288 | ||
306 | ### Overcoming Mistrust | 289 | ### Overcoming Mistrust |
307 | 290 | ||
308 | -Adopting an unfamiliar approach requires trust. Ministry staff, traditionally, | ||
309 | -expected and were prepared to validate and deploy a single deliverable. However, | ||
310 | -the steady growth of SPB complexity made large deliveries unsustainable. The CD | ||
311 | -approach was necessary [4]. Therefore, we developed the following line of action to | ||
312 | -make this new dynamics possible: | ||
313 | - | ||
314 | -1. _Demonstrate actual results, do not simply tell._ Initially, we did not have | ||
315 | -access to the Ministry infrastructure, so we created our own validation | ||
316 | -environment. Thus, we were able to follow the CD pipeline until production | ||
317 | -deployment, when we faced two problems. First, our pace of intermediate | ||
318 | -deliveries to the government was faster than the deployment to production by | ||
319 | -the Ministry staff. Second, specific issues of the Ministry infrastructure made | ||
320 | -some validated features not work as expected in the PE. That situation gave us | ||
321 | -arguments to negotiate access to the PE. | 291 | +Adopting an unfamiliar approach requires trust. ministry staff, traditionally, |
292 | +expected and were prepared to validate and deploy a single deliverable. | ||
293 | +However, the steady growth of SPB complexity made large deliveries | ||
294 | +unsustainable. The CD approach was necessary [4]. Therefore, we developed the | ||
295 | +following line of action to enable this new dynamics: | ||
296 | + | ||
297 | +1. _Demonstrate actual results, instead of simply reporting them._ Initially, | ||
298 | +we did not have access to the ministry infrastructure, so we created our own | ||
299 | +validation environment. Thus, we were able to follow the CD pipeline until | ||
300 | +production deployment, when we faced two problems. First, our pace of | ||
301 | +intermediate deliveries was faster than the deployment to production by the | ||
302 | +government staff. Second, specific issues of the governmental infrastructure | ||
303 | +made some validated features not work as expected in the PE. That situation | ||
304 | +gave us arguments to negotiate access to the PE. | ||
322 | 305 | ||
323 | 2. _Make project management transparent and collaborative for government | 306 | 2. _Make project management transparent and collaborative for government |
324 | -staff._ Allowing the Ministry staff to track our development process showed | 307 | +staff._ Allowing the ministry staff to track our development process showed |
325 | them we were fulfilling our commitments. They started to interact more actively | 308 | them we were fulfilling our commitments. They started to interact more actively |
326 | in the generation of versions and became involved in the CD. After | 309 | in the generation of versions and became involved in the CD. After |
327 | -understanding it, the Ministry staff helped us negotiate access to a VE with | ||
328 | -the Ministry leaders, creating as an isolated replica of the PE. | 310 | +understanding it, the ministry staff helped us negotiate access to a VE with |
311 | +the ministry leaders, creating an isolated replica of the PE. | ||
329 | 312 | ||
330 | 3. _Gain the confidence of government staff._ With the PE replica, we were able | 313 | 3. _Gain the confidence of government staff._ With the PE replica, we were able |
331 | -to run the entire pipeline and win the trust of the Ministry staff involved in | 314 | +to run the entire pipeline and win the trust of the ministry staff involved in |
332 | the process. They saw the mobilization and responsiveness of our team to | 315 | the process. They saw the mobilization and responsiveness of our team to |
333 | generate each new version package. They also recognized the quality of our work | 316 | generate each new version package. They also recognized the quality of our work |
334 | -and deployment process. In the end, the Ministry staff realized that it would | 317 | +and deployment process. In the end, the ministry staff realized that it would |
335 | be beneficial for the project if they granted us access to the infrastructure, | 318 | be beneficial for the project if they granted us access to the infrastructure, |
336 | both VE and PE. | 319 | both VE and PE. |
337 | 320 | ||
321 | +In summary, we encourage the use of a well-thought CD pipeline as a means to | ||
322 | +gain trust in software development projects with large organizations. | ||
323 | + | ||
338 | ## References | 324 | ## References |
339 | 325 | ||
340 | 1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", OpenSym, 2017. | 326 | 1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", OpenSym, 2017. |
341 | 1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, 32 (2) 2015. | 327 | 1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, 32 (2) 2015. |
342 | 1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", ICSE, 2016. | 328 | 1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", ICSE, 2016. |
343 | 1. M. Shahin, M. A. Babar, and L. Zhu. “The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives”. ESEM, 2016. | 329 | 1. M. Shahin, M. A. Babar, and L. Zhu. “The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives”. ESEM, 2016. |
330 | +1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, 2015. | ||
344 | 1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley, 2010. | 331 | 1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley, 2010. |
345 | 332 | ||
346 | -<!--- | ||
347 | -1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, Article 18, 2015, 41 pages. | ||
348 | ---> | ||
349 | - | ||
350 | - |