Commit c2056b9dfc16f510095742fdf5190d44bb502e0e

Authored by Rodrigo Siqueira de Melo
2 parents f30af53d bab08ca3

Merge branch 'ieee_new'

1 1 ---
2   -title: "Continuous Delivery: the strategy to overcome challenges in Government IT"
  2 +title: |
  3 + | __Continuous Delivery:__
  4 + | Building Trust in a Large-scale, Complex
  5 + | Government Organization
3 6 papersize: a4
4 7 geometry: "left=1in,right=1.5in"
5 8 ---
6 9  
7 10 ## Authors
8 11  
9   -__Diego Camarinha__ is a masters student at IME - The Institute of Mathematics and
10   -Statistics of the Sao Paulo University. His research interests include software
11   -engineering, computer networks and source code metrics. He worked for two years
12   -with the Brazilian federal government as a senior developer. Contact him at
14   -
15   -__Rodrigo Siqueira__ is a masters student at IME - The Institute of Mathematics and
16   -Statistics of the Sao Paulo University. His research interests include software
17   -engineering, operating system and computer architecture. He worked for two years
18   -with the Brazilian federal government as a coach and developer. Contact him at
20   -
21   -__Melissa Wen__ is a software developer. She worked on SPB project as a senior developer and also served as professor of Computer Science at UFBA - The Federal University of Bahia. Her areas of interest include software engineering and open source software development. Contact her at .
22   -
23   -__Paulo Meirelles__ received his Ph.D. in Computer Science from the Institute of Mathematics and Statistics at the University of São Paulo. He is a full-time Professor at the University of Brasilia, and coordinated the new SPB Portal project. His research interest areas are: Free Software, Agile methods, Static analysis, and Source code metrics. Contact him at
  12 +__Rodrigo Siqueira__ is a Computer Science MSc student at the Institute of
  13 +Mathematics and Statistics of the University of São Paulo (IME-USP). His
  14 +research interests include Software Engineering, Operating System, and Computer
  15 +Architecture. He worked for two years with the SPB project as a coach and
  16 +developer. Contact him at
  17 +
  18 +__Diego Camarinha__ is a Computer Science MSc masters student at IME-USP. His
  19 +research interests include Software Engineering, Computer Networks, and Source
  20 +Code Metrics. He worked for two years with the SPB project as a senior
  21 +developer. Contact him at
  22 +
  23 +__Melissa Wen__ is a Bachelor of Computer Science from Federal University of
  24 +Bahia (UFBA). She worked on the SPB project as a senior developer and was core
  25 +developer of Noosfero social networking platform at Colivre. Her areas of
  26 +interest include Software Engineering and Free Software Development. Contact
  27 +her at
  28 +
  29 +__Paulo Meirelles__ is a Professor of Software Engineering at the
  30 +University of Brasilia (UnB) and researcher at the Free/Libre/Open Source Software
  31 +Competence Center (CCSL) at IME-USP. He was the head of the new SPB portal
  32 +development. His research interests include Free Software development, Agile
  33 +Software Development, and Source code metrics. Contact him at
  34 +
  35 +__Fabio Kon__ is a Full Professor of Computer Science at IME-USP, co-founder of
  36 +the CCSL at IME-USP, and Editor-in-Chief of the SpringerOpen Journal of
  37 +Internet Services and Applications. His research interests include Agile
  38 +Software Development, Smart Cities, and Distributed Systems. Contact him at
24 40  
25 41 ## Abstract
26 42  
27   -For many software development teams, the first things that come to mind
28   -regarding Continuous Delivery (CD) are the operational aspects and the
29   -competitive benefits. In our experience, it was much more: it was a survival
30   -technique. This article presents how we applied CD in a Brazilian Government
31   -project for the development of a Collaborative Development Environment (CDE),
32   -sharing its unconventional challenges and the strategies used to overcome them.
33   -This report can help
34   -practitioners to understand how important CD adoption is to their projects.
35   -
36   -## Introduction and Context
37   -
38   -We worked on a three-year-long Brazilian government project to evolve an
39   -existing platform that had technical issues and lacked political support. Our
40   -team from the University of Brasília (UnB) and the University of São Paulo
41   -(USP) developed the new platform for the Brazilian Public Software (SPB,
42   -Portuguese acronym) Portal from 2014 to 2016. The SPB Portal
43   -( evolved to a Collaborative Development Environment
44   -[1] and this evolution brought important benefits not just to the Brazilian
45   -government, but also to society as a whole. The government could reduce both
46   -the bureaucracy of using the same software in government agencies and the cost
47   -of developing similar software projects. The society gained a mechanism of
48   -transparency and collaboration, since anyone can check the government
49   -expenses on software (from information available on the SPB Portal) and
50   -contribute to project communities. To achieve these goals, rather than writing
51   -everything from scratch, we decided to integrate several free software tools
52   -such as Noosfero (, Gitlab (, Mailman
53   -(, Mezuro (, and Colab
54   -(
55   -
56   -The project started in a presidential election year and everyone involved was
57   -under pressure to show results. Even with the re-election of the Brazilian
58   -President in 2014, leaderships in governmental agencies ended up changing. Each
59   -one of them had different political agendas which affected the project's
60   -requirements previously approved. Besides that scenario of instability, we
61   -overcame three distinct issues: (i) achieving the goals which have guided the
62   -platform development, (ii) managing the diversity of team members, and (iii)
63   -communicating effectively with government agents (the client). Handling the
64   -interaction of these elements was challenging and the unstable Brazilian
65   -political scenario only made things worse.
66   -
67   -To achieve the SPB project goals, we had to overcome strong political bias tied
68   -with complicated technical issues and relatively low budget. Because the
69   -project is open to the public, the government representatives saw the platform
70   -as a marketing opportunity and sometimes ignored technical advice in favor of
71   -political decisions. Furthermore, integrating a number of distinct systems
72   -(such as social and collaboration network, Git repository manager, mailing
73   -list, source code metric evaluation, and a systems integration platform) to
74   -work seamlessly was not an easy job. We had to learn how each system worked and
75   -to come up with ideas of how to integrate them as fast as possible, with a team
76   -of mostly inexperienced developers.
77   -
78   -We also had to manage the diversity of our team members. The team was composed
79   -of approximately 50 undergraduate students (not all simultaneously) together
80   -with professors, master students, professional designers and senior developers
81   -from the Brazilian free software community. Undergraduate students received a
82   -scholarship and, for most of them, this R&D project was their first
83   -professional experience. Senior developers and master students had two
84   -important contributions to the project: to transfer knowledge to undergraduate
85   -students and to deal with more complex tasks. Finally, professors were
86   -responsible for interacting with the Brazilian government and controlling
87   -political pressures applied to the project.
88   -
89   -Moreover, we needed to communicate with two independent groups of government
90   -representatives. Requirement analysts were real representatives of the
91   -Brazilian government in the project and their role was to test new features, to
92   -provide feedback to the development team, and to report to government leaders.
93   -Deployment technicians were in charge of managing the host machines wherein SPB
94   -platform was running. They were, theoretically, responsible for deploying the
95   -project. However, the new SPB Portal was composed of more than ten different
96   -software projects, working on seven servers, generating a complex deployment
97   -process.
98   -
99   -To handle the interaction between these three elements, we realised we needed
100   -to take control over the deployment process. We used CD as a mean to fulfill
101   -the government expectations and to provide quick response to their requests,
102   -which were influenced most of the time by the uncertainties of the project's
103   -continuity. We believed we would keep the project alive, even in a politically
104   -unstable and technically complex scenario. For this reason, we focused on
105   -automating the deploy process; for instance, one of our senior developers
106   -created a tool called Chake ( to help us manage
107   -multiple hosts without the need for a Chef-Server ( We also
108   -formed a specific team dedicated to the deployment process. This team was
109   -responsible for maturing our CD pipeline, giving us confidence and agility to
110   -comply with government requirements.
111   -
112   -In this report, we describe our CD pipeline and how it improved our delivery
113   -performance. Continuous Delivery made us adaptable for all requested changes
114   -and helped us mitigate those aforementioned political challenges, besides the
115   -technical issues. Among CD’s known benefits [2] we also explain those which
116   -were the most important in our scenario: improving customer satisfaction and
117   -making reliable releases. Both kept the project alive for years during the
118   -worst political crisis after the re-democratization in Brazil.
  43 +For many software development teams, the first aspects that come to mind
  44 +regarding Continuous Delivery (CD) are the operational challenges and the
  45 +competitive benefits. In our experience, CD was much more: it was a survival
  46 +technique. This article presents how and why we applied CD in a large
  47 +governmental project for the development of a Collaborative Development
  48 +Environment. We share the challenges we faced and the strategies we used to
  49 +overcome them. The article concludes with a set of lessons learned that can be
  50 +valuable for readers in a variety of situations.
  51 +
  52 +## Introduction
  53 +
  54 +From 2014 to 2016, we worked on a thirty-month-long Brazilian government
  55 +project to modernize the Brazilian Public Software (SPB) portal
  56 +( [1]. This project was a partnership between the
  57 +_Ministry of Planning, Budget, and Management (MPOG)_ and two public
  58 +universities: University of Brasília and University of São Paulo.
  59 +
  60 +During this time, the SPB portal evolved into a Collaborative Development
  61 +Environment (CDE), which brought significant benefits for the government and
  62 +the society. The government could minimize bureaucracy and software development
  63 +costs, by reusing the same set of applications across different government
  64 +Agencies. Society could more transparently follow government expenses and
  65 +contribute to software project communities.
  66 +
  67 +In this article, we discuss the use of Continuous Delivery (CD) during our
  68 +experience as the academic partner in this project. We show how we implemented
  69 +CD in a large institution with traditional, waterfall-like values and how CD
  70 +helped to build trust between the government and the university team. CD
  71 +enabled us to show our progress and to earn the government’s confidence that we
  72 +could adequately fulfill their requests, becoming an essential aspect of our
  73 +interaction. According to this experience, the use of CD as a tool to build
  74 +trust relationships is yet another of its benefits [2,3].
  75 +
  76 +## Context
  77 +
  78 +SPB is a governmental program created to foster sharing and collaboration on
  79 +Open Source Software (OSS) development for the Brazilian public administration.
  80 +The government managed both software requirements and server infrastructure.
  81 +However, its hierarchical and traditional processes made them unfamiliar with
  82 +new software development techniques, such as CD. All our requests had to pass
  83 +through layers of bureaucracy before being answered; accessing their
  84 +infrastructure to deploy software was not different. The difficulties were
  85 +aggravated because the new SPB portal is an unprecedented platform in the
  86 +Brazilian government, with a complicated deployment process [4].
  87 +
  88 +The project suffered significant interference from the Board of Directors
  89 +throughout time because the portal represents an interface between government
  90 +and society. In light of political interests, directors continually imposed
  91 +changes to the platform while ignoring our technical advice. In 2015, the Board
  92 +of Directors was changed and, with it, the vision of the project. New directors
  93 +had different political agendas, which affected the project's requirements
  94 +previously approved.
  95 +
  96 +In this context, we overcame two distinct challenges: (1) deconstructing the
  97 +widespread belief among government staff that any project in partnership with a
  98 +University is doomed to fail, and (2) dealing with bureaucracies involved in
  99 +the government deployment process.
  100 +
  101 +First, our development team was not the typical one, consisting mainly of
  102 +undergraduate interns supported by senior developers and designers, all
  103 +coordinated by two professors. Our unconventional team structure and
  104 +organization was considered as "unprofessional" by government standards with
  105 +regard to time and resource allocation, accountability, and team continuity.
  106 +On the government side, the SPB portal evolution was the first software
  107 +development collaboration involving universities and the MPOG staff, raising
  108 +disbelief.
  109 +
  110 +Second, our team approached software deployment differently from the
  111 +government. We believed frequent delivery is better for the project’s success.
  112 +In contrast, the MPOG is used to the idea of a single deployment at the end of
  113 +the project, and neither their bureaucratic structure nor their technical
  114 +expertise was conducive to this style of work. That was hampering the benefits
  115 +of the tool and preventing us from showing off the fruits of the project to
  116 +those responsible for evaluating it.
  117 +
  118 +<!-- AHA Moment!!!!! -->
  119 +
  120 +These challenges made our relationship with their staff strained, in particular
  121 +during the first year, and alerted us to the fact that they could cancel the
  122 +project at any time. The deployment limitation was the substantial technical
  123 +issue we could tackle in the short term. Thus, we worked to deploy the software
  124 +into our infrastructure and showed it to the government evaluators. This
  125 +strategy proved them we could efficiently provide new features, fulfill their
  126 +requirement delivery expectations, and incited them to demand that the latest
  127 +version be deployed in their infrastructure. Our CD approach generated more
  128 +pressure on the IT department responsible for the deployment routines. With
  129 +each CD cycle, we gradually built a new relationship among all parties and, by
  130 +the end of the project, we became active participants in the deploy operations
  131 +delivering quality software very frequently.
119 132  
120 133 ## Our Continuous Delivery Pipeline
121 134  
122   -![Deployment Pipeline](figures/pipeline_2.png)
  135 +The SPB portal is an open CDE with additional social features, having 83
  136 +software communities and 6,460 user accounts, mostly from government employees.
  137 +We built a system-of-systems [5] adapting and integrating five existing OSS
  138 +projects: Colab (, Noosfero (, Gitlab
  139 +(, Mezuro (, and Mailman ( Colab
  140 +orchestrates these multiple systems using a plugin architecture and allowed us
  141 +to smoothly provide a unified interface to final users, including single
  142 +sign-on and global searches [1]. All these integrated systems involve a total
  143 +of 106,253 commits and 1,347,421 lines of code.
123 144  
124   -Figure 1 represents our CD pipeline. A new feature might require changes on more
125   -than one SPB integrated software project. Notice that each one of them could be
126   -modified independently. The pipeline started when new code arrived. As it went
127   -through each step, it was tested and improved until it finally reached the
128   -production environment. At this point we would restart the pipeline to release
129   -more new code.
  145 +![The SPB Deployment Pipeline.](figures/pipeline.png)
130 146  
131   -### Automated Tests
  147 +Portal deployment follows a typical CD pipeline [6], adapted to our technical
  148 +and organizational context and the use of OSS best practices. As depicted in
  149 +Figure 1, it begins when a new feature is ready for deployment and ends when it
  150 +reaches production.
132 151  
133   -The SPB portal consists of more than 10 integrated software projects and each
134   -of them, as well as the entire platform, had to be tested. These software
135   -components have their own test suite. Communication between all components is
136   -orchestrated by Colab, a systems integration platform for web applications based
137   -on a plugins architecture. Therefore, specific plugins were developed for
138   -each portal software component, such as Gitlab and Mailman. Each plugin has its
139   -own test suite and this set also worked as integration tests.
140 152  
141   -Both unit and integration tests provided us the performance and security needed
142   -to guarantee the stability for components and the platform. If any test suite
143   -failed, by either a test error or coverage reduction below a certain threshold,
144   -the pipeline stopped. Only when all tests passed, the pipeline proceeded to the
145   -step of preparing the release.
  153 +### Automated Tests
  154 +
  155 +Each integrated system is a Colab plugin and had to be tested with its own test
  156 +suite. In addition, we also had to test the platform as a whole. Since the
  157 +plugins have their own test suites, they also assume a double role as both
  158 +plugin tests and as integration tests. If any test suite failed, by either a
  159 +test error or coverage reduction below a certain threshold, the process
  160 +stopped. Only when all tests passed, the pipeline proceeded to the next step.
146 161  
147 162 ### Preparing a New Release
148 163  
149   -A SPB Portal release was composed of all its software components releases. Each
150   -software component release was a git tag that referred to a specific feature or
151   -bug fix. When all tests passed for a given component, we manually created a new
152   -tag for it. Therefore, a new tag on any software component yielded a new SPB
153   -Portal release. More precisely, SPB had a script that produced a single release
154   -for the entire system based on each component tag. At the end of this process,
155   -we started packaging.
  164 +A separate Git repository hosted each system. New software component releases
  165 +were tagged referencing a specific new feature or bug fix. SPB had its own Git
  166 +repository ( An SPB portal
  167 +release was an aggregate of all its systems. When a new component release
  168 +passed all of the SPB integration tests, we manually created a corresponding
  169 +new tag in its repository. At the end of this process, we started packaging.
156 170  
157 171 ### Packaging
158 172  
159   -The platform is running on the CentOS 7 GNU/Linux distribution. Basically,
160   -packaging a software for that distribution has three steps: write the script
161   -for the specific environment (RPM), build the package, and upload it to a
162   -package repository.
163   -
164   -We decided to create our own packages for each software component for the following five
165   -reasons:
166   -
167   -* Not all software was packaged by the community and those that existed were
168   -outdated;
169   -* Packaging makes it easy to manage the software on a given distribution;
170   -* Packaging simplifies the deployment;
171   -* Packaging follows the distribution's best practices and,
172   -* Packaging allows configurations and permissions control.
173   -
174   -After creating a new tag for one component, the developers informed the
175   -DevOps [3] team and the packaging process began. The three packaging steps
176   -aforementioned were fully automated by a set of scripts. With all these scripts
177   -running successfully, the new packages would be ready to be used by our
178   -subsequent deployment scripts.
179   -
180   -### Validation Environment Deployment
181   -
182   -The Validation Environment (VE) is a replica of the Production Environment
183   -(PE), with two exceptions: only the government officers and project leaders had
184   -access to it and all the data was anonymised. To configure the environment, we
185   -used a configuration management tool named Chef with Chake support (serverless
186   -configuration for Chef). That tool maintained environment consistency
187   -simplifying the deployment process. Additionally, the packages we built on the
188   -last step were readily available to be used by the management tool.
189   -
190   -Government agents used the VE to validate new features and required
191   -changes. Also, the VE was useful to verify the integrity of the entire portal
192   -as part of the next step in the pipeline.
  173 +After creating a new tag, our DevOps team started the packaging process.
  174 +Packaging brings portability, simplifies deployment, and enables configuration
  175 +and permission control. Our approach involved building separate packages for
  176 +each system, in three fully automated steps: generating scripts for the
  177 +specific environment, building the package, and uploading it to a package
  178 +repository. When all ran successfully, the new packages would be ready and
  179 +available for our deployment scripts.
  180 +
  181 +### Validation Environment
  182 +
  183 +The Validation Environment (VE) is a replica of the Production Environment (PE)
  184 +with anonymised data, and access restricted to MPOG staff and our DevOps team.
  185 +To configure this environment, we used Chef ( and Chake, a
  186 +serverless configuration tool created by our team
  187 +( to maintain environment consistency,
  188 +simplifying the deployment process.
193 189  
194 190 ### Acceptance Tests
195 191  
196   -After we deployed a new SPB Portal version in the VE, the government
197   -agents were responsible for checking features and bug fixes required by them. If
198   -the requirement analysts identified a problem, they would notify the developers via
199   -comments on the SPB Portal's issue tracker. The problems were fixed and the
200   -pipeline restarted from scratch. If everything was validated, we moved forward.
  192 +After a new SPB portal VE deployment, we used the environment to verify the
  193 +integrity of the entire portal. MPOG staff also checked the new features,
  194 +required changes, and bug fixes. If they identified a problem, they would
  195 +notify developers via comments on the SPB portal issue tracker, prompting the
  196 +team to fix it and restart the pipeline. Otherwise, we could move forward.
201 197  
202 198 ### Production Environment Deployment
203 199  
204   -After the government finished the VE check and it was cleared for deployment,
205   -we could finally begin the deployment to the Production Environment (PE). For this
206   -we also used our configuration management tool, the same scripts and package
207   -versions as in the VE. After the deploy was completed, both VE and PE
208   -were running identical software. This was the point where new features and bug
209   -fixes were finally available to the end users.
  200 +After the VE check, we could finally begin the deployment in production, with
  201 +the same configuration management tool, scripts, and package versions as in the
  202 +VE. After the deploy was completed, both VE and PE were identical, making new
  203 +features and bug fixes available to end users.
210 204  
211 205 ## Benefits
212 206  
213   -Research points out many advantages of CD usage in industry[2], such as:
214   -accelerated time to market, building the right product, productivity and
215   -efficiency improvements, reliable releases and better customer satisfaction.
216   -Working with the government, we noticed the following additional benefits.
  207 +![The evolution of SPB releases and the development team members distribution.](figures/CDReleaseAndTeamEvolution.png)
  208 +
  209 +CD brings many advantages such as accelerated time to market, building the
  210 +right product, productivity and efficiency improvements, stable releases, and
  211 +better customer satisfaction [2,3]. The charts presented in Figure 2 show these
  212 +benefits in our project and associates them with the creation of a DevOps team.
  213 +Over 30 months, we deployed 84 versions. Over 64% of the releases happened in
  214 +the last 12 months, after the creation of the DevOps team. Besides these
  215 +results, working with the government we noticed the following additional
  216 +benefits.
  217 +
  218 +### Strengthening Trust in the Relationship with the Government
  219 +
  220 +CD helped strengthen trust between developers and the MPOG staff. Before using
  221 +CD, they could validate features developed only at the end of the release
  222 +cycle, usually every four months. With the implementation of CD, intermediate
  223 +and candidate versions became available, allowing them to perform small
  224 +validations over time. Constant monitoring of the development work brought
  225 +greater assurance to the MPOG leaders and improved the interactions with our
  226 +team.
217 227  
218 228 ### Responsiveness to Change
219 229  
220   -Responsiveness was one of the direct benefits of adopting the CD pipeline. The
221   -ability to react quickly to changes requested by the government was vital for
222   -the renewal of the project over the years. Every meeting with the government
223   -leader resulted in new requirements, most of them motivated by political
224   -needs. These constant changes in requirements and priorities caused discomfort
225   -between the government and the development team. For
226   -example, the government leader required a complete layout change because he suddenly decided to make a marketing campaign about the new
227   -SPB portal. In future meetings, the government would use undelivered requirements as a means to justify the
228   -lack of financial support, which was already approved in the first place. We believed that if we took too
229   -long to attend their demands, the project would end. CD helped us keep the
230   -production environment up-to-date, even with partial versions of a feature. That
231   -way, we always had something to show on meetings, reducing anxiety to get the platform concluded. For our team, it made the developers more confident that the
232   -project would last a little longer and they would not go looking for other
233   -jobs.
  230 +Responsiveness was one of the direct benefits of adopting the CD pipeline.
  231 +Political forces may change requirements and priorities at any time. The
  232 +ability to react quickly to changes requested by the government was vital to
  233 +the project survival for 30 months. We noticed that if we took too long to meet
  234 +their demands, they could reduce financial support and even cancel the project.
  235 +
  236 +CD helped us keep the PE up-to-date, even with partial versions of a feature.
  237 +Therefore, we always had new results to present on meetings, easing their
  238 +concerns about expected final delivery. For our team, CD made developers always
  239 +confident that the project would last a little longer.
234 240  
235 241 ### Shared Responsibility
236 242  
237   -Before the adoption of CD, the development team could not track what happened to the code
238   -after its delivery, since government technicians were the only responsible
239   -for deploying the project. The implementation of the referred
240   -approach influenced developers on taking ownership of the project because it
241   -made them feel equally responsible for what was getting into production.
242   -
243   -Interestingly, the CD pipeline had the same effect on the team of requirement analysts.
244   -They were an active part of the pipeline and became more engaged on the whole process.
245   -After the incorporation of the pipeline into the work process, analysts
246   -became more active in opening and discussing issues during the platform evolution.
247   -Additionally, developers worked to improve the CD pipeline
248   -to speed up the process of making new features available in the production environment.
249   -
250   -### Synchronicity Between Government and Development
251   -
252   -Despite the positive impacts that the CD pipeline brought to the project, its
253   -implementation was not easy at first. The CD pipeline performance
254   -depended on the synchronicity between developers and government
255   -analysts, so that the latter were prepared to start a step as soon as the
256   -former concluded the previous step, and vice versa. Initially, this concern was not
257   -contemplated in the agenda of the governmental team, which generated delays in
258   -the validation of new features. This situation combined with
259   -governmental bureaucracy (up to 3 days) to release access to the production
260   -environment resulted in additional delays for the deployment step to begin.
261   -This problem was softened when the analysts realized the impact of
262   -these delays on the final product and decided to allocate the revisions in its
263   -work schedule and to request the access to production in time.
264   -
265   -### Strengthening Trust in Work Relationship with the Government
266   -
267   -Continuous delivery was also a tool that helped to strengthen trust in the
268   -relationship between developers and government analysts, as well as between the
269   -analysts group and its superiors. Before using CD, analysts had access to the
270   -features developed only at the end of the release, usually every four months.
271   -However, this periodicity did not meet the requirements of their leaders, who
272   -demanded monthly reports on the progress of the project.
273   -
274   -With the implementation of CD, intermediate and candidate versions became
275   -available, allowing analysts to perform small validations over time. As they
276   -validated functionalities and sent feedback to developers, patches were
277   -developed and new versions were packaged and deployed to the VE quickly,
278   -steadily, and reliably. The constant monitoring of the development work brought
279   -greater security to the governmental nucleus and improved the interactions
280   -with our development team.
281   -
282   -## Challenges
283   -
284   -We successfully built a CD pipeline. In the end, we took over the deployment
285   -process from the government. That allowed us to survive into an unstable
286   -political scenario. However, we recognize that many challenges still need to be
287   -addressed by the industry and academia together.
  243 +According to the conventional MPOG process, the development team could not
  244 +track what happened to the code after its delivery, since their employees were
  245 +the only ones responsible for deployment. The implementation of CD made our
  246 +development team feel equally responsible for what was getting into production
  247 +and take ownership of the project [4].
288 248  
289   -### Build CD From Scratch
  249 +Interestingly, the CD pipeline had the same effect on the MPOG staff. They
  250 +became more engaged in the whole process, opening and discussing issues during
  251 +the evolution of the platform. Additionally, developers worked to improve the
  252 +CD pipeline and speed up the process of making new features available in the
  253 +production environment.
290 254  
291   -Taking on CD responsibilities had a significant impact on the team. We did not
292   -have the know-how and had little time to come up with a working pipeline. The
293   -senior developers were crucial at this point. They came up with an initial
294   -solution to get the team started. That already enabled us to automatize
295   -deployment, even though the process was still rudimentary. We had to evolve our
296   -solution on-the-fly. We dedicated a few developers to this task.
  255 +### Synchronization Between Government and Development
297 256  
298   -Building a CD pipeline was hard in the beginning. More tools that provide
299   -out-of-the-box standardized CD pipelines would be of great help for
300   -inexperienced teams. Tools that track each step of the pipeline and organize
301   -logs in a human-manageable way are necessary too.
  257 +The CD pipeline performance depended on the synchronization between our
  258 +development team and the MPOG staff: each party had to be prepared to take
  259 +action as soon as the other concluded a given task. Initially, the MPOG staff
  260 +did not contemplate this in their schedule which, combined with the bureaucracy
  261 +in providing access to the PE (up to 3 days), resulted in significant delays in
  262 +the validation of new features. The use of an explicit CD pipeline helped us to
  263 +identify critical points of delay, and increase our productivity.
302 264  
303   -### Handling Inexperienced Teams
  265 +## Lessons Learned
304 266  
305   -After the developers learned how CD worked, it was difficult to pass the
306   -knowledge along to other teammates. We tried to mitigate this problem by
307   -encouraging members to migrate to the DevOps team. We suggest further research
308   -on how to effectively spread knowledge across inexperienced developers in a high
309   -turnover scenario.
  267 +Due to the successful building of the CD pipeline, we not only overcame the
  268 +challenges we described above but also improved the MPOG deployment process and
  269 +kept the project alive until its successful conclusion. We now look at the
  270 +lessons we learned, which can be leveraged by readers in other situations.
310 271  
311   -### Overcoming Mistrust
  272 +### Build CD From Scratch
  273 +
  274 +Taking on the responsibility for implementing CD impacted the whole team. Most
  275 +of our team members did not have CD know-how, and we had few working hours
  276 +available to build the initial version of the pipeline. The construction and
  277 +maintenance of the CD process were made possible by the key decisions to:
312 278  
313   -In the project's beginning we struggled with deployment issues in the government
314   -structure. We were in a paradoxical situation. The government demanded fast
315   -deliveries but would not give access to their production infrastructure. After
316   -some interactions with government agents, they created the VE as an isolated
317   -replica of the PE in their own infrastructure. The government agents then
318   -realised that it could be beneficial for the project if they granted us access
319   -to part of the infrastructure. More research is required on development protocols and
320   -policies to improve the relation between industry and government, specially
321   -regarding CD.
  279 +1. _Select the most experienced senior developers and some advanced interns to
  280 +work on a specific DevOps team._ These senior developers used their experience
  281 +in OSS projects to craft an initial proposal for the deployment process. The
  282 +solution enabled us to automate the deployment, even though the process was,
  283 +initially, still rudimentary.
322 284  
323   -## Acknowledgements
  285 +2. _Interchange team members and encourage teammates to migrate to the DevOps
  286 +team._ The benefits were twofold: mitigating the difficulty in sharing
  287 +knowledge between DevOps developers and feature developers, and evolving the
  288 +process on-the-fly.
  289 +
  290 +### Overcoming Mistrust
324 291  
325   -We thank our colleagues, Lucas Kanashiro and Rafael Manzo, and this article's reviewers.
  292 +Adopting an unfamiliar approach requires trust. The government staff,
  293 +traditionally, expected and were prepared to validate and deploy a single
  294 +deliverable. However, the steady growth of SPB complexity made large
  295 +deliveries unsustainable. The CD approach was necessary [4]. Therefore, we
  296 +developed the following line of action to enable this new dynamics:
  297 +
  298 +1. _Demonstrate actual results, instead of simply reporting them._ Initially,
  299 +we did not have access to the government infrastructure, so we created our own
  300 +validation environment. Thus, we were able to follow the CD pipeline until
  301 +production deployment, when we faced two problems. First, our pace of
  302 +intermediate deliveries was faster than the deployment to production by the
  303 +MPOG staff. Second, specific issues of the governmental infrastructure made
  304 +some validated features not work as expected in the PE. That situation gave us
  305 +arguments to negotiate access to the PE.
  306 +
  307 +2. _Make project management transparent and collaborative for government
  308 +staff._ Allowing the MPOG staff to track our development process showed them we
  309 +were fulfilling our commitments. They started to interact more actively in the
  310 +generation of versions and became involved in the CD. After understanding it,
  311 +the government staff helped us negotiate access to a VE with the MPOG leaders,
  312 +creating an isolated replica of the PE.
  313 +
  314 +3. _Gain the confidence of government staff._ With the PE replica, we were able
  315 +to run the entire pipeline and win the trust of the MPOG staff involved in the
  316 +process. They saw the mobilization and responsiveness of our team to generate
  317 +each new version package. They also recognized the quality of our work and
  318 +deployment process. In the end, the government staff realized that it would be
  319 +beneficial for the project if they granted us access to the infrastructure,
  320 +both VE and PE.
  321 +
  322 +In summary, we encourage the use of a well-thought CD pipeline as a means to
  323 +gain trust in software development projects with government and large
  324 +organizations as well.
326 325  
327 326 ## References
328 327  
329   -1. G. Booch, A. W. Brown, "Collaborative Development Environments", in Advances in Computers, vol. 59, 2003, pp. 1–27.
330   -2. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54.
331   -3. Davis, Jennifer and Daniels, Katherine, Effective DevOps: building a culture of collaboration, affinity, and tooling at scale, 2016, " O'Reilly Media, Inc."
  328 +1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", OpenSym, 2017.
  329 +1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, 32 (2) 2015.
  330 +1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", ICSE, 2016.
  331 +1. M. Shahin, M. A. Babar, and L. Zhu. “The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives”. ESEM, 2016.
  332 +1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, 2015.
  333 +1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley, 2010.
332 334  
... ...
ieeeSW/releaseEng3/figures/CDReleaseAndTeamEvolution.png 0 → 100644

211 KB


44 KB

No preview for this file type