Download as
Browse Code »

Commit 2c513948401f95606ccf4baa04969210e7905f28

Authored by Paulo Meireles
1 parent 614bbeec
Exists in master and in 3 other branches oss_2018, oss_2018_siqueira, rework_case_study

Review from all authors

Showing 1 changed file with 190 additions and 208 deletions   Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
  Diff comments   View file @2c51394
1 1 ---
2 2 title: |
3   - | __Continuous Delivery__
4   - | Building Trust in a Large-scale, Complex Government Organization
  3 + | __Continuous Delivery:__
  4 + | Building Trust in a Large-scale, Complex
  5 + | Government Organization
5 6 papersize: a4
6 7 geometry: "left=1in,right=1.5in"
7 8 ---
8 9  
9 10 ## Authors
10 11  
11   -__Rodrigo Siqueira__ is a masters student at IME - The Institute of Mathematics
12   -and Statistics of the Sao Paulo University. His research interests include
13   -software engineering, operating system and computer architecture. He worked for
14   -two years with the Brazilian federal government as a coach and developer.
15   -Contact him at siqueira@ime.usp.br.
16   -
17   -__Diego Camarinha__ is a masters student at IME - The Institute of Mathematics
18   -and Statistics of the Sao Paulo University. His research interests include
19   -software engineering, computer networks and source code metrics. He worked for
20   -two years with the Brazilian federal government as a senior developer. Contact
21   -him at diegoamc@ime.usp.br.
22   -
23   -__Melissa Wen__ is a software developer. She worked on SPB project as a senior
24   -developer and also served as professor of Computer Science at UFBA - The Federal
25   -University of Bahia. Her areas of interest include software engineering and open
26   -source software development. Contact her at melissa.srw@gmail.com .
27   -
28   -__Paulo Meirelles__ received his Ph.D. in Computer Science from the Institute of
29   -Mathematics and Statistics at the University of São Paulo. He is full-time
30   -Professor at the University of Brasilia, and coordinated the new SPB portal
31   -project. His research interest areas are: Free Software, Agile methods, Static
32   -analysis, and Source code metrics. Contact him at paulormm@ime.usp.br.
33   -
34   -__Fabio Kon__ is Full Professor of the Department of Computer Science of the
35   -University of São Paulo. (...) .His research interest areas are: (...). Contact
36   -him at fabio.kon@ime.usp.br.
37   -
38   -<!-- Fabio: Esse resumo está não usual porque ele não resume o que o artigo apresenta. Ele serve como uma motivação apenas. Mas talvez podemos deixar assim pra ver o que acontece -->
  12 +__Rodrigo Siqueira__ is a Computer Science MSc student at the Institute of
  13 +Mathematics and Statistics of the University of São Paulo (IME-USP). His
  14 +research interests include software engineering, operating system, and computer
  15 +architecture. He worked for two years with the Brazilian federal government as
  16 +a coach and developer. Contact him at siqueira@ime.usp.br.
  17 +
  18 +__Diego Camarinha__ is a CS MSc masters student at IME-USP. His research
  19 +interests include software engineering, computer networks, and source code
  20 +metrics. He worked for two years with the Brazilian federal government as a
  21 +senior developer. Contact him at diegoamc@ime.usp.br.
  22 +
  23 +__Melissa Wen__ is a software developer. She worked on the SPB project as a
  24 +senior developer and also served as a Computer Science lecturer at the Federal
  25 +University of Bahia. Her areas of interest include software engineering and
  26 +open source software development. Contact her at melissa.srw@gmail.com.
  27 +
  28 +__Paulo Meirelles__ received his Ph.D. in Computer Science from IME-USP. He is
  29 +a full-time Professor at the University of Brasilia and coordinated the new SPB
  30 +portal project. His research interests are: Free Software, Agile methods,
  31 +Static analysis, and Source code metrics. Contact him at paulormm@ime.usp.br.
  32 +
  33 +__Fabio Kon__ is a Full Professor of Computer Science at IME-USP and
  34 +Editor-in-Chief of the SpringerOpen Journal of Internet Services and
  35 +Applications. His research interests include Agile Software Development, Smart
  36 +Cities, and Distributed Systems. Contact him at fabio.kon@ime.usp.br.
  37 +
39 38 ## Abstract
40 39  
41 40 For many software development teams, the first aspects that come to mind
42 41 regarding Continuous Delivery (CD) are the operational challenges and the
43 42 competitive benefits. In our experience, CD was much more: it was a survival
44   -technique. This article presents how and why we applied CD in a large Brazilian
45   -government project for the development of a Collaborative Development
46   -Environment (CDE), sharing the challenges we faced and the strategies used to
47   -overcome them. ( >> FECHAR COM FRASE DE IMPACTO <<<).
  43 +technique. This article presents how and why we applied CD in a large
  44 +governmental project for the development of a Collaborative Development
  45 +Environment. We share the challenges we faced and the strategies we used to
  46 +overcome them. The article concludes with a set of lessons learned that can be
  47 +valuable for readers in a variety of situations.
48 48  
49 49 ## Introduction
50 50  
51   -From 2014 to 2016, we were part of a team that worked on a thirty-month-long
52   -Brazilian government project to modernize the Brazilian Public Software (SPB)
53   -portal (www.softwarepublico.gov.br) [1]. This project was a partnership between
54   -the Ministry of Planning, Budget, and Management and two public universities:
  51 +From 2014 to 2016, we worked on a thirty-month-long Brazilian government
  52 +project to modernize the Brazilian Public Software (SPB) portal
  53 +(www.softwarepublico.gov.br) [1]. This project was a partnership between the
  54 +_Ministry of Planning, Budget, and Management_ and two public universities:
55 55 University of Brasília and University of São Paulo.
56 56  
57 57 During this time, the SPB portal evolved into a Collaborative Development
58   -Environment (CDE) which brought significant benefits for the government and the
59   -society: The government could minimize bureaucracy and software development
  58 +Environment (CDE), which brought significant benefits for the government and
  59 +the society. The government could minimize bureaucracy and software development
60 60 costs, by reusing the same set of applications across different government
61   -agencies; society could more transparently follow government expenses and
  61 +Agencies. Society could more transparently follow government expenses and
62 62 contribute to project communities.
63 63  
64 64 In this article, we discuss the use of Continuous Delivery (CD) during our
65   -experience as the academic partner in this project. We focus on how we managed
66   -to implement CD in a large institution with traditional values and how CD
67   -helped to build trust between the government and the university development
68   -team. CD enabled us to show our progress and to earn the government’s
69   -confidence that we could adequately fulfill their requests, becoming an
70   -essential aspect of our interaction with them. According to this experience,
71   -the use of CD as a tool to build such trust relationships is yet another of its
72   -benefits [2,3].
  65 +experience as the academic partner in this project. We show how we implemented
  66 +CD in a large institution with traditional, waterfall-like values and how CD
  67 +helped to build trust between the government and the university team. CD
  68 +enabled us to show our progress and to earn the government’s confidence that we
  69 +could adequately fulfill their requests, becoming an essential aspect of our
  70 +interaction. According to this experience, the use of CD as a tool to build
  71 +trust relationships is yet another of its benefits [2,3].
73 72  
74 73 ## Context
75 74  
76 75 SPB is a governmental program created to foster sharing and collaboration on
77 76 Open Source Software (OSS) development for the Brazilian public administration.
78   -In their own projects, the Ministry managed both software requirements and
79   -server infrastructure. However, its hierarchical and traditional processes made
80   -them unfamiliar with new software development techniques, such as CD. Any of
81   -our requests had to pass through layers of bureaucracy before being answered;
82   -accessing their infrastructure to deploy updated software was not different.
83   -The difficulties were aggravated because the new SPB portal is an unprecedented
84   -platform in the Brazilian government, with a complicated deployment process [4].
85   -
86   -The project suffered significant interference from the board of directors
  77 +The ministry managed both software requirements and server infrastructure.
  78 +However, its hierarchical and traditional processes made them unfamiliar with
  79 +new software development techniques, such as CD. All our requests had to pass
  80 +through layers of bureaucracy before being answered; accessing their
  81 +infrastructure to deploy software was not different. The difficulties were
  82 +aggravated because the new SPB portal is an unprecedented platform in the
  83 +Brazilian government, with a complicated deployment process [4].
  84 +
  85 +The project suffered significant interference from the Board of Directors
87 86 throughout time because the portal represents an interface between government
88 87 and society. In light of political interests, directors continually imposed
89   -changes to the platform while ignoring our technical advice. In 2015, the board
90   -of directors was changed and, with it, the vision of the project. New directors
91   -had different political agendas which affected the project's requirements
  88 +changes to the platform while ignoring our technical advice. In 2015, the Board
  89 +of Directors was changed and, with it, the vision of the project. New directors
  90 +had different political agendas, which affected the project's requirements
92 91 previously approved.
93 92  
94   -<!---
95   -mudar em todas as ocorrências de agents por staff, mas checar com o Fabio.
96   --->
97   -
98   -In this context, we overcame two distinct challenges: (1) deconstructing
99   -the widespread belief among government staff that any project in partnership
100   -with a University is doomed to fail, and (2) dealing with bureaucracies
101   -involved in the Ministry deployment process.
  93 +In this context, we overcame two distinct challenges: (1) deconstructing the
  94 +widespread belief among government staff that any project in partnership with a
  95 +University is doomed to fail, and (2) dealing with bureaucracies involved in
  96 +the government deployment process.
102 97  
103   -First, our development team was not from a typical company, consisting mainly
104   -of undergraduate interns supported by senior developers and designers, all
  98 +First, our development team was not the typical one, consisting mainly of
  99 +undergraduate interns supported by senior developers and designers, all
105 100 coordinated by two professors. Our unconventional team structure and
106   -organization might be considered as "unprofessional" by Ministry standards with
107   -regard to time and resource allocation, accountability, and team continuity .
  101 +organization was considered as "unprofessional" by ministry standards with
  102 +regard to time and resource allocation, accountability, and team continuity.
108 103 On the government side, the SPB portal evolution was the first software
109   -development collaboration involving universities and the Ministry staff,
  104 +development collaboration involving universities and the ministry staff,
110 105 raising disbelief.
111 106  
112 107 Second, our team approached software deployment differently from the
113   -Ministry. We believed frequent delivery is better for the project’s success.
114   -In contrast, the Ministry is used to the idea of a single deployment at the end
  108 +government. We believed frequent delivery is better for the project’s success.
  109 +In contrast, the ministry is used to the idea of a single deployment at the end
115 110 of the project, and neither their bureaucratic structure nor their technical
116   -expertise was conducive to this style of work. That ended up hampering the
117   -benefits of the tool and preventing us from showing off the fruits of the
118   -project to those responsible for evaluating it.
  111 +expertise was conducive to this style of work. That was hampering the benefits
  112 +of the tool and preventing us from showing off the fruits of the project to
  113 +those responsible for evaluating it.
119 114  
120 115 <!-- AHA Moment!!!!! -->
121 116  
122   -These challenges made our relationship with the Ministry staff tense, in
123   -particular during the first year, and alerted us to the fact that they could
124   -cancel the project at any time. The deployment limitation was the substantial
125   -technical issue we could tackle in the short term. As a result, we worked to
126   -deploy one version of the project onto our infrastructure and showed it to the
127   -government evaluators. This strategy proved them we could efficiently provide
128   -new features, fulfill their expectations regarding the delivery of the
129   -requirements, and incited them to demand that the latest version be deployed in
130   -the Ministry infrastructure. Our CD approach generated more pressure on the IT
131   -department responsible for the deployment routines. With each CD cycle, we
132   -gradually built a new relationship among all parties and, by the end of the
133   -project, we became active participants in the deploy operations.
  117 +These challenges made our relationship with their staff strained, in particular
  118 +during the first year, and alerted us to the fact that they could cancel the
  119 +project at any time. The deployment limitation was the substantial technical
  120 +issue we could tackle in the short term. Thus, we worked to deploy the software
  121 +into our infrastructure and showed it to the government evaluators. This
  122 +strategy proved them we could efficiently provide new features, fulfill their
  123 +requirement delivery expectations, and incited them to demand that the latest
  124 +version be deployed in their infrastructure. Our CD approach generated more
  125 +pressure on the IT department responsible for the deployment routines. With
  126 +each CD cycle, we gradually built a new relationship among all parties and, by
  127 +the end of the project, we became active participants in the deploy operations
  128 +delivering quality software very frequently.
134 129  
135 130 ## Our Continuous Delivery Pipeline
136 131  
137   -SPB portal is a CDE with additional social features, having 83 software
138   -communities and 6,460 users, mostly government employees. We built
139   -system-of-systems adapting and integrating five existing OSS projects: Colab
140   -(www.github.com/colab), Noosfero (www.noosfero.org), Gitlab (www.gitlab.com),
141   -Mezuro (www.mezuro.org), and
142   -Mailman (www.list.org). Colab orchestrates these multiple systems using a
143   -plugin architecture and allows us to smoothly provide a unified interface for
144   -final users, including single sign-on and global searches [1]. All these
145   -integrated systems represents a total of 106.253 commits and 1.347.421 lines of
146   -code.
  132 +The SPB portal is an open CDE with additional social features, having 83
  133 +software communities and 6,460 user accounts, mostly from government employees.
  134 +We built a system-of-systems [5] adapting and integrating five existing OSS
  135 +projects: Colab (www.github.com/colab), Noosfero (www.noosfero.org), Gitlab
  136 +(www.gitlab.com), Mezuro (www.mezuro.org), and Mailman (www.list.org). Colab
  137 +orchestrates these multiple systems using a plugin architecture and allowed us
  138 +to smoothly provide a unified interface to final users, including SSO and
  139 +global searches [1]. All these integrated systems involve a total of 106,253
  140 +commits and 1,347,421 lines of code.
147 141  
148 142 ![The SPB Deployment Pipeline.](figures/pipeline.png)
149 143  
150   -The portal deployment follows a typical CD pipeline [5], adapted to the
151   -technical and organizational context of our project and the use of OSS best
152   -practices, as presented in Figure 1. It started when new code arrived; and when
153   -it reaches the production environment, we would restart the pipeline to release
154   -new versions.
  144 +Portal deployment follows a typical CD pipeline [6], adapted to our technical
  145 +and organizational context and the use of OSS best practices. As depicted in
  146 +Figure 1, it begins when a new feature is ready for deployment and ends when it
  147 +reaches production.
  148 +
155 149  
156 150 ### Automated Tests
157 151  
158   -In practice, each integrated system is a Colab plugin and had to be tested with
159   -its own test suite. However, this checking was not enough: we also had to test
160   -the platform as a whole. Since the plugins have their own test suites, they
161   -also assume a double role as both plugin tests and as integration tests. If any
162   -test suite failed, by either a test error or coverage reduction below a certain
163   -threshold, the process stopped. Only when all tests passed, the pipeline
164   -proceeded to the step of release preparation.
  152 +Each integrated system is a Colab plugin and had to be tested with its own test
  153 +suite. In addition, we also had to test the platform as a whole. Since the
  154 +plugins have their own test suites, they also assume a double role as both
  155 +plugin tests and as integration tests. If any test suite failed, by either a
  156 +test error or coverage reduction below a certain threshold, the process
  157 +stopped. Only when all tests passed, the pipeline proceeded to the next step.
165 158  
166 159 ### Preparing a New Release
167 160  
168   -Each software component was hosted in a separate Git repository. A new component
169   -release was tagged referencing a specific new feature or bug fix. SPB
170   -had its own Git repository (www.softwarepublico.gov.br/gitlab/softwarepublico).
171   -An SPB portal release was an aggregate of versions of all its components.
172   -When a new component release passed all of the SPB integration tests, we
173   -manually created a corresponding new tag in its repository. Therefore, a new
174   -tag on any software component yielded a new SPB portal release. At the end of
175   -this process, we started packaging.
  161 +A separate Git repository hosted each software component. New component
  162 +releases were tagged referencing a specific new feature or bug fix. SPB had its
  163 +own Git repository (www.softwarepublico.gov.br/gitlab/softwarepublico). An SPB
  164 +portal release was an aggregate of all its components. When a new component
  165 +release passed all of the SPB integration tests, we manually created a
  166 +corresponding new tag in its repository. At the end of this process, we started
  167 +packaging.
176 168  
177 169 ### Packaging
178 170  
179   -After creating a new tag for a system, our DevOps team starts the packaging
180   -process. Packaging brings portability, simplifies deployment, and enables
181   -configuration and permission control. Our packaging software approach involves
182   -building separate packages for each system, in three steps: writing the script
183   -for the specific environment, building the package, and uploading it to a
184   -package repository. A set of scripts fully automated these steps. When all them
185   -ran successfully, the new packages would be ready and available for our
186   -deployment scripts.
  171 +After creating a new tag, our DevOps team started the packaging process.
  172 +Packaging brings portability, simplifies deployment, and enables configuration
  173 +and permission control. Our approach involved building separate packages for
  174 +each system, in three fully automated steps: generating scripts for the
  175 +specific environment, building the package, and uploading it to a package
  176 +repository. When all ran successfully, the new packages would be ready and
  177 +available for our deployment scripts.
187 178  
188   -### Validation Environment Deployment
  179 +### Validation Environment
189 180  
190 181 The Validation Environment (VE) is a replica of the Production Environment (PE)
191   -with anonymised data, and access restricted to Ministry staff and our DevOps
  182 +with anonymised data, and access restricted to ministry staff and our DevOps
192 183 team. To configure this environment, we used Chef (www.chef.io) and Chake, a
193   -serverless configuration tool created by our team
194   -(www.github.com/terceiro/chake). This tool maintained environment consistency,
  184 +serverless configuration tool created by our team
  185 +(www.github.com/terceiro/chake) to maintain environment consistency,
195 186 simplifying the deployment process.
196 187  
197 188 ### Acceptance Tests
198 189  
199   -After a new SPB portal deployment in the VE, we used the environment to verify
200   -the integrity of the entire portal. In this step, Ministry staff also checked
201   -the new features, required changes, and bug fixes. If they identified a
202   -problem, they would notify the developers via comments on the SPB portal issue
203   -tracker, prompting the team to fix it and restart the pipeline. Otherwise, we
204   -could move forward.
  190 +After a new SPB portal VE deployment, we used the environment to verify the
  191 +integrity of the entire portal. Government staff also checked the new features,
  192 +required changes, and bug fixes. If they identified a problem, they would
  193 +notify developers via comments on the SPB portal issue tracker, prompting the
  194 +team to fix it and restart the pipeline. Otherwise, we could move forward.
205 195  
206 196 ### Production Environment Deployment
207 197  
208 198 After the VE check, we could finally begin the deployment in production, with
209 199 the same configuration management tool, scripts, and package versions as in the
210   -VE. After the deploy was completed, both VE and PE were identical. At that
211   -point, new features and bug fixes were finally available to end users.
  200 +VE. After the deploy was completed, both VE and PE were identical, making new
  201 +features and bug fixes available to end users.
212 202  
213 203 ## Benefits
214 204  
215 205 CD brings many advantages such as accelerated time to market, building the
216 206 right product, productivity and efficiency improvements, stable releases, and
217 207 better customer satisfaction [2,3]. The charts presented in Figure 2 show these
218   -benefits of CD in our project and associates them with the creation of a DevOps
219   -team. In the time of 30 months, we deployed a total of 84 versions. Over 64% of
220   -these activities happened in the second half of 2015, with the DevOps team
221   -creation. Besides these results, working with the government, we noticed the
222   -following additional benefits.
  208 +benefits in our project and associates them with the creation of a DevOps team.
  209 +Over 30 months, we deployed 84 versions. Over 64% of the releases happened in
  210 +the last 12 months, after the creation of the DevOps team. Besides these
  211 +results, working with the government we noticed the following additional
  212 +benefits.
223 213  
224   -![The evolution of SPB releases and the team members distribution in the period.](figures/CDReleaseAndTeamEvolution.png)
  214 +![The evolution of SPB releases and the development team members distribution.](figures/CDReleaseAndTeamEvolution.png)
225 215  
226 216 ### Strengthening Trust in the Relationship with the Government
227 217  
228   -CD helped strengthen trust in the relationship between developers and the
229   -Ministry staff. Before using CD, they could validate features developed only at
230   -the end of the release cycle, usually every four months. With the
231   -implementation of CD, intermediate and candidate versions became available,
232   -allowing them to perform small validations over time. Constant monitoring of
233   -the development work brought greater security to the Ministry leaders and
234   -improved the interactions with our team.
  218 +CD helped strengthen trust between developers and the government staff. Before
  219 +using CD, they could validate features developed only at the end of the release
  220 +cycle, usually every four months. With the implementation of CD, intermediate
  221 +and candidate versions became available, allowing them to perform small
  222 +validations over time. Constant monitoring of the development work brought
  223 +greater assurance to the ministry leaders and improved the interactions with
  224 +our team.
235 225  
236 226 ### Responsiveness to Change
237 227  
238 228 Responsiveness was one of the direct benefits of adopting the CD pipeline.
239   -Political interests may change requirements and priorities at any time. The
240   -ability to react quickly to changes requested by the Ministry was vital to the
241   -project’s survival for 30 months. We noticed that if we took too long to meet
242   -their demands, they would threaten to reduce financial support and even cancel
243   -the project.
  229 +Political forces may change requirements and priorities at any time. The
  230 +ability to react quickly to changes requested by the government was vital to
  231 +the project survival for 30 months. We noticed that if we took too long to meet
  232 +their demands, they could reduce financial support and even cancel the project.
244 233  
245 234 CD helped us keep the PE up-to-date, even with partial versions of a feature.
246   -Therefore, we always had something to show on meetings, easing their concerns
247   -about the final delivery of the platform. For our team, CD made developers more
  235 +Therefore, we always had new results to present on meetings, easing their
  236 +concerns About expected final delivery. For our team, CD made developers always
248 237 confident that the project would last a little longer.
249 238  
250 239 ### Shared Responsibility
251 240  
252   -According to the conventional Ministry process, the development team could not
253   -track what happened to the code after its delivery, since their staff was the
254   -only ones responsible for deployment. The implementation of CD made our
  241 +According to the conventional ministry process, the development team could not
  242 +track what happened to the code after its delivery, since their employees were
  243 +the only ones responsible for deployment. The implementation of CD made our
255 244 development team feel equally responsible for what was getting into production
256 245 and take ownership of the project [4].
257 246  
258   -Interestingly, the CD pipeline had the same effect on the Ministry staff. They
  247 +Interestingly, the CD pipeline had the same effect on the ministry staff. They
259 248 became more engaged in the whole process, opening and discussing issues during
260 249 the evolution of the platform. Additionally, developers worked to improve the
261 250 CD pipeline and speed up the process of making new features available in the
... ... @@ -264,39 +253,33 @@ production environment.
264 253 ### Synchronization Between Government and Development
265 254  
266 255 The CD pipeline performance depended on the synchronization between our
267   -development team and the Ministry staff: each party had to be prepared to take
268   -action as soon as the other concluded a given task. Initially, the Ministry
  256 +development team and the ministry staff: each party had to be prepared to take
  257 +action as soon as the other concluded a given task. Initially, the ministry
269 258 staff did not contemplate this in their schedule which, combined with the
270 259 bureaucracy in providing access to the PE (up to 3 days), resulted in
271 260 significant delays in the validation of new features. The use of an explicit CD
272 261 pipeline helped us to identify critical points of delay, and increase our
273 262 productivity.
274 263  
275   -<!---
276   -Fabio sugeriu Lessons Learned, mas vamos mostrar exemplos da revista para ele olhar.
277   --->
278   -
279 264 ## Lessons Learned
280 265  
281   -<!---
282   -(>>>FABIO FAZER RELAÇÃO COM CHALLENGES DA SEÇÃO CONTEXTO<<<).
283   --->
284   -Due to the successful building of the CD pipeline, we improved the Ministry
285   -deployment process and kept the project alive. We now look at the lessons
286   -learned.
  266 +Due to the successful building of the CD pipeline, we not only overcame the
  267 +challenges we described above but also improved the ministry deployment process
  268 +and kept the project alive until its successful conclusion. We now look at the
  269 +lessons we learned, which can be leveraged by readers in other situations.
287 270  
288 271 ### Build CD From Scratch
289 272  
290 273 Taking on the responsibility for implementing CD impacted the whole team. Most
291 274 of our team members did not have CD know-how, and we had few working hours
292   -available to build the pipeline. The construction and maintenance of the CD
293   -process were made possible by the key decisions to:
  275 +available to build the initial version of the pipeline. The construction and
  276 +maintenance of the CD process were made possible by the key decisions to:
294 277  
295   -1. _Select the most experienced senior developers and some advanced students of
296   -the project to work on a specific DevOps team._ These senior developers used
297   -their experience in OSS projects to craft an initial proposal for the
298   -deployment process. The solution enabled us to automate the deployment, even
299   -though the process was, initially, still rudimentary.
  278 +1. _Select the most experienced senior developers and some advanced students to
  279 +work on a specific DevOps team._These senior developers used their experience
  280 +in OSS projects to craft an initial proposal for the deployment process. The
  281 +solution enabled us to automate the deployment, even though the process was,
  282 +initially, still rudimentary.
300 283  
301 284 2. _Interchange team members and encourage teammates to migrate to the DevOps
302 285 team._ The benefits were twofold: mitigating the difficulty in sharing
... ... @@ -305,46 +288,45 @@ process on-the-fly.
305 288  
306 289 ### Overcoming Mistrust
307 290  
308   -Adopting an unfamiliar approach requires trust. Ministry staff, traditionally,
309   -expected and were prepared to validate and deploy a single deliverable. However,
310   -the steady growth of SPB complexity made large deliveries unsustainable. The CD
311   -approach was necessary [4]. Therefore, we developed the following line of action to
312   -make this new dynamics possible:
313   -
314   -1. _Demonstrate actual results, do not simply tell._ Initially, we did not have
315   -access to the Ministry infrastructure, so we created our own validation
316   -environment. Thus, we were able to follow the CD pipeline until production
317   -deployment, when we faced two problems. First, our pace of intermediate
318   -deliveries to the government was faster than the deployment to production by
319   -the Ministry staff. Second, specific issues of the Ministry infrastructure made
320   -some validated features not work as expected in the PE. That situation gave us
321   -arguments to negotiate access to the PE.
  291 +Adopting an unfamiliar approach requires trust. ministry staff, traditionally,
  292 +expected and were prepared to validate and deploy a single deliverable.
  293 +However, the steady growth of SPB complexity made large deliveries
  294 +unsustainable. The CD approach was necessary [4]. Therefore, we developed the
  295 +following line of action to enable this new dynamics:
  296 +
  297 +1. _Demonstrate actual results, instead of simply reporting them._ Initially,
  298 +we did not have access to the ministry infrastructure, so we created our own
  299 +validation environment. Thus, we were able to follow the CD pipeline until
  300 +production deployment, when we faced two problems. First, our pace of
  301 +intermediate deliveries was faster than the deployment to production by the
  302 +government staff. Second, specific issues of the governmental infrastructure
  303 +made some validated features not work as expected in the PE. That situation
  304 +gave us arguments to negotiate access to the PE.
322 305  
323 306 2. _Make project management transparent and collaborative for government
324   -staff._ Allowing the Ministry staff to track our development process showed
  307 +staff._ Allowing the ministry staff to track our development process showed
325 308 them we were fulfilling our commitments. They started to interact more actively
326 309 in the generation of versions and became involved in the CD. After
327   -understanding it, the Ministry staff helped us negotiate access to a VE with
328   -the Ministry leaders, creating as an isolated replica of the PE.
  310 +understanding it, the ministry staff helped us negotiate access to a VE with
  311 +the ministry leaders, creating an isolated replica of the PE.
329 312  
330 313 3. _Gain the confidence of government staff._ With the PE replica, we were able
331   -to run the entire pipeline and win the trust of the Ministry staff involved in
  314 +to run the entire pipeline and win the trust of the ministry staff involved in
332 315 the process. They saw the mobilization and responsiveness of our team to
333 316 generate each new version package. They also recognized the quality of our work
334   -and deployment process. In the end, the Ministry staff realized that it would
  317 +and deployment process. In the end, the ministry staff realized that it would
335 318 be beneficial for the project if they granted us access to the infrastructure,
336 319 both VE and PE.
337 320  
  321 +In summary, we encourage the use of a well-thought CD pipeline as a means to
  322 +gain trust in software development projects with large organizations.
  323 +
338 324 ## References
339 325  
340 326 1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", OpenSym, 2017.
341 327 1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, 32 (2) 2015.
342 328 1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", ICSE, 2016.
343 329 1. M. Shahin, M. A. Babar, and L. Zhu. “The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives”. ESEM, 2016.
  330 +1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, 2015.
344 331 1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley, 2010.
345 332  
346   -<!---
347   -1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, Article 18, 2015, 41 pages.
348   --->
349   -
350   -
... ...