Commit 2b94e087a4fca0ee5f0c2709e9cf05025e8dc9c5
1 parent
ff8bdf06
Exists in
master
and in
3 other branches
Create pipeline image
Showing
3 changed files
with
283 additions
and
278 deletions
Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
| 1 | -# The Strategic Importance of Continuous Delivery | ||
| 2 | - | ||
| 3 | -## Abstract | ||
| 4 | - | ||
| 5 | -Many software development teams think of the operational aspects of Continuous | ||
| 6 | -Delivery (CD) and the competitive benefits that come along with it. For us, it | ||
| 7 | -was much more: it was a survival technique. This article presents the | ||
| 8 | -experience applying CD in a Brazilian Government project for the development of | ||
| 9 | -a Collaborative Development Environment (CDE), sharing its unconventional | ||
| 10 | -challenges and our strategies used to overcome them. This report from the | ||
| 11 | -trenches of the Brazilian Federal Government can help practitioners to | ||
| 12 | -understand how important the CD adoption is to their projects. | ||
| 13 | - | ||
| 14 | -## Introduction and Context | ||
| 15 | - | ||
| 16 | -We have worked on a Brazilian government three-year-long project to redesign an | ||
| 17 | -existing platform that had technical issues and a lack of political support. | ||
| 18 | -This evolution project started in a presidential election year and everyone | ||
| 19 | -involved were under Presidential re-election campaign pressure to show results. | ||
| 20 | -Even with the re-election of the Brazilian President, leadership in government | ||
| 21 | -agencies suddenly changed that reflected on the project’s requirements: each | ||
| 22 | -new leader wanted to fulfill their political agenda. In this scenario, delivery | ||
| 23 | -delays could have sunk the project into oblivion. | ||
| 24 | - | ||
| 25 | -From 2013 to 2016, our team developed the new platform for the Brazilian Public | ||
| 26 | -Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br). The SPB | ||
| 27 | -Portal has evolved to a CDE [1] and this evolution has brought important | ||
| 28 | -benefits not just to the Brazilian government, but also to society as a whole. | ||
| 29 | -For the government, the bureaucracy on using the same software across | ||
| 30 | -government agencies, duplicate works and costs all are reduced. The society | ||
| 31 | -gains a transparency and collaboration mechanism, since anyone can check the | ||
| 32 | -government expenses on software and contribute to the software communities. To | ||
| 33 | -achieve these goals, rather than writing everything from scratch, we have | ||
| 34 | -chosen to integrate free software tools such as Gitlab (www.gitlab.com), | ||
| 35 | -Mailman (www.gnu.org/software/mailman), Noosfero (www.noosfero.org), and | ||
| 36 | -Colab (www.github.com/colab). | ||
| 37 | - | ||
| 38 | -During the entire SPB Portal evolution project, we had to handle three distinct | ||
| 39 | -issues, usual in a software engineering scenario: reaching the goals which have | ||
| 40 | -guided the platform development, managing the diversity of team project | ||
| 41 | -members, and communicating effectively with clients (in this particular case, | ||
| 42 | -government agents). Managing the interaction of these elements was not easy and | ||
| 43 | -the unstable Brazilian political scenario only made things worse. | ||
| 44 | - | ||
| 45 | -To reach the SPB project goals, we had to overcome strong political bias | ||
| 46 | -tied with complicated technical issues and relatively low budget. Because it is | ||
| 47 | -open to the public, the government representatives have seen the platform as a | ||
| 48 | -marketing opportunity and have often ignored the technical advice in favor | ||
| 49 | -of political decisions. Furthermore, integrating a number of distinct systems | ||
| 50 | -to work seamlessly was not an easy job. We had to learn how each system worked | ||
| 51 | -and come up with ideas of how to integrate them as fast as possible, with a | ||
| 52 | -team of mostly inexperienced developers. | ||
| 53 | - | ||
| 54 | -We also had to manage the diversity of the SPB team members. This team was | ||
| 55 | -composed of approximately 50 undergraduate students (not all simultaneously) | ||
| 56 | -together with professors, masters students and professional designers as well | ||
| 57 | -as senior developers from the Brazilian free software community. Undergraduate | ||
| 58 | -students have received a fellowship and, for most of them, this R&D project was | ||
| 59 | -their first professional experience. Seniors developers and masters students | ||
| 60 | -had two important contributions to the project: transfer knowledge to | ||
| 61 | -undergraduate students and address hard tasks. Finally, professors were | ||
| 62 | -responsible for interacting with the Brazilian government and control the | ||
| 63 | -political pressures applied to the project. | ||
| 64 | - | ||
| 65 | -Our third point to be handled was the communication with the two independent | ||
| 66 | -groups of government representatives: requirements analysts and deployment | ||
| 67 | -technicians. Requirements analysts were the real representatives of the | ||
| 68 | -Brazilian government in the project. They usually tested new features, provided | ||
| 69 | -feedback, and reported for directors. The deployment technicians only had the | ||
| 70 | -access to the host machines wherein SPB platform was running. They were, | ||
| 71 | -theoretically, responsible for deploying the project. However, the new SPB | ||
| 72 | -Portal was composited from more than ten different software projects, | ||
| 73 | -generating a complex deploy process working on seven servers. | ||
| 74 | - | ||
| 75 | -Therefore, we realised that we needed to take control over the deployment | ||
| 76 | -process. We would use CD as a mean to keep the government satisfied and provide | ||
| 77 | -quick response times to their requests which were, most of the time, influenced | ||
| 78 | -by the uncertainties of project continuity. We believed that would keep the | ||
| 79 | -project alive even in an politically unstable and complex technical scenario. | ||
| 80 | -For that, during months we have worked to automate all the deploy process. For | ||
| 81 | -instance, we have created a tool called Shak (Self Hosting Applications Kit) | ||
| 82 | -that offers an application installation with only instruction. It aims to make | ||
| 83 | -the deploy process easier for users without technical knowledge, ensuring | ||
| 84 | -privacy and security for their Internet applications. Thereby, we created a | ||
| 85 | -specific team dedicated to the deployment process to mature our CD pipeline | ||
| 86 | -that would give us confidence to meet the government’s requirements faster and | ||
| 87 | -faster. | ||
| 88 | - | ||
| 89 | -We describe our CD pipeline and how it speeded up our software delivery time. | ||
| 90 | -The CD made us adaptable for all requested changes and helped us to mitigate | ||
| 91 | -those aforementioned political challenges besides the technical issues. Among | ||
| 92 | -CD’s known benefits[2], we also explain those that were the most important in | ||
| 93 | -the project scenario for us: improving customer satisfaction and making | ||
| 94 | -reliable releases. Both kept the project alive for more two years during the | ||
| 95 | -worst political crisis after the re-democratization in Brazil. | ||
| 96 | - | ||
| 97 | -## Our Continuous Delivery Pipeline | ||
| 98 | - | ||
| 99 | -[//]: # (TODO - Fazer a imagem. Sem esquentar muito a cabeça, uma vez que a revista deve refazer tal imagem) | ||
| 100 | - | ||
| 101 | -[//]: # (conjunto de features -> testes automatizados → criação de tag → empacotamento →deploy em homologação → testes de aceitação → deploy em produção) | ||
| 102 | - | ||
| 103 | -[//]: # (TODO - Rever essa frase - é uma legenda da imagem?) | ||
| 104 | - | ||
| 105 | -Figure X represents our CD pipeline. After each release, we have defined a set | ||
| 106 | -of features we would develop to comply with government’s requirements and then | ||
| 107 | -we have followed this pipeline. | ||
| 108 | - | ||
| 109 | -### Automated tests | ||
| 110 | - | ||
| 111 | -[//]: # (TODO - Rever a escrita desta ideia com mais calma) | ||
| 112 | -[//]: # (TODO - Adicionar que as mudanças no Colab quebravam todos os plugins.) | ||
| 113 | -[//]: # (https://www.linux.com/blog/learn/chapter/dev-ops/2017/7/devops-fundamentals-part-3-continuous-delivery-and-deployment) | ||
| 114 | - | ||
| 115 | -The SPB portal is composed of a set of FOSS systems. All of them have their own | ||
| 116 | -automated test suite. So, we encouraged the teams to keep unit tests coverage | ||
| 117 | -high for each system. We implemented the systems integration using a plugin | ||
| 118 | -architecture and wrote integration tests to check if they were working | ||
| 119 | -properly. The plugin architecture made the PSPB’s components decoupled. That | ||
| 120 | -allowed us to be confident that changes on one system would not affect others. | ||
| 121 | -This characteristic allowed the test execution of only the component with | ||
| 122 | -changes instead of the entire system. | ||
| 123 | - | ||
| 124 | -Our CD’s first step was to execute each system’s automated test suite. If any | ||
| 125 | -error was found, the pipeline stopped and the developers were notified. Only | ||
| 126 | -after all tests passed we move to preparing the release. Preparing a new | ||
| 127 | -release | ||
| 128 | - | ||
| 129 | -Our release system was divided into two perspectives: the application and the | ||
| 130 | -PSPB. The application tag refers to the specific feature or bug fix and is a | ||
| 131 | -monotonically increasing. A new tag on any system yielded a new PSPB tag. | ||
| 132 | - | ||
| 133 | -When all tests passed for a given component, we manually created a new | ||
| 134 | -application tag for it. As a consequence, that automatically created a new tag | ||
| 135 | -for the PSPB. Notice that we have forks of the original softwares and, as | ||
| 136 | -consequence, we had different tag values. Packaging | ||
| 137 | - | ||
| 138 | -The PSPB platform is running under the CentOS GNU/Linux distribution. | ||
| 139 | -Basically, packaging a software for that distribution has three steps: (1) | ||
| 140 | -write the script for the specific environment (RPM), (2) build the package, and | ||
| 141 | -(3) upload it to a package repository. We chose to package our components for | ||
| 142 | -several reasons: | ||
| 143 | -* Not all software was packaged by the community; | ||
| 144 | -* And those that existed were outdated; | ||
| 145 | -* Packaging makes it easy to manage the software on a given distribution; | ||
| 146 | -* It simplifies the deployment; | ||
| 147 | -* Packaging follows the distribution’s best practices and, | ||
| 148 | -* Allows configurations and permissions control. | ||
| 149 | - | ||
| 150 | -After creating a new tag for one component, the DevOps team was notified and | ||
| 151 | -packaging process began. In the normal case, the three packaging steps | ||
| 152 | -aforementioned are fully automated by a set of scripts. However, if the team | ||
| 153 | -reports to DevOps any eventual dependency change, the first step has to be | ||
| 154 | -manual. For instance, one system could start requiring another system to be | ||
| 155 | -initialized and that made necessary for someone to manually update the package | ||
| 156 | -script. | ||
| 157 | - | ||
| 158 | -After all these scripts have run successfully, the new packages would be ready | ||
| 159 | -to use by our subsequent deployment scripts. | ||
| 160 | - | ||
| 161 | -### Validation Environment | ||
| 162 | - | ||
| 163 | -[//]: # (TODO - Mencionar que a ferramenta era baseada em Chef - Dá um peso importante.) | ||
| 164 | - | ||
| 165 | -The Validation Environment (VE) is a replica of the Production Environment | ||
| 166 | -(PE), with two exceptions: only the government officers and us had access to it | ||
| 167 | -and all the data is anonymised. To configure the environment, we use a | ||
| 168 | -configuration management tool. That maintained environment consistency which | ||
| 169 | -makes the deployment process simpler. Additionally, the packages we built on | ||
| 170 | -the last step were readily available to use by the management tool. | ||
| 171 | - | ||
| 172 | -The VE was used by the government agents to validate new features and required | ||
| 173 | -changes. Also, the VE was useful to verify the integrity of the entire portal | ||
| 174 | -as part of the next step in the pipeline. Acceptance Tests | ||
| 175 | - | ||
| 176 | -After we completely deploy a new PSPB version in the VE, the government agents | ||
| 177 | -are responsible for checking features and/or bug fixes required by them. If the | ||
| 178 | -technicians identify a problem, they notify the developers, the problems are | ||
| 179 | -fixed and the pipeline restarts from scratch. If everything is validated, we | ||
| 180 | -move to production deployment. Production Deployment | ||
| 181 | - | ||
| 182 | -After the government authorizes the VE, we can finally begin the production | ||
| 183 | -deployment. We use the same configuration management tool with the same | ||
| 184 | -scripts. After the deploy is completed, both VE and PE are on the same state. | ||
| 185 | -The new features and bug fixes are finally available to end users. | ||
| 186 | - | ||
| 187 | - | ||
| 188 | -## Benefits | ||
| 189 | - | ||
| 190 | -We had to handle many tensions between development and political issues. Our CD | ||
| 191 | -pipeline gave us strong mechanisms to tackle most of the problems. As a result | ||
| 192 | -we came with some benefits from our decision to adopt CD. | ||
| 193 | - | ||
| 194 | -[//]: # (TODO - Melhorar título - Ideias: Response to mistrust) | ||
| 195 | - | ||
| 196 | -### Response to tensions | ||
| 197 | - | ||
| 198 | -[//]: # (TODO - Decisão do secretario acima do diretor) | ||
| 199 | - | ||
| 200 | -The direct benefit from the CD pipeline was the fast response to the changes | ||
| 201 | -required by the government. That was vital for the project’s renewal over the | ||
| 202 | -years. We could manage the tension between the government and the development | ||
| 203 | -team better. Every meeting with the government leader was delicate and resulted | ||
| 204 | -on many new requirements, most of them motivated by political needs. For | ||
| 205 | -example, once it was demanded a completely layout change because one director | ||
| 206 | -suddenly decided to make a marketing campaign about the portal. They would use | ||
| 207 | -undelivered requirements as a means to suggest the project’s cancellation. We | ||
| 208 | -believed that if we took too long to attend their demands, the project would | ||
| 209 | -end. CD helped us to move fast on deploying to production, even of smaller | ||
| 210 | -parts of the requirements. That way, we always had something to show on the | ||
| 211 | -meetings, reducing their eagerness to end the project. For our team, it made | ||
| 212 | -the developers more confident the project would last a little longer and they | ||
| 213 | -would not go looking for another jobs. | ||
| 214 | - | ||
| 215 | -### Build client’s trust | ||
| 216 | - | ||
| 217 | -After we established the CD, the government agents started to be more confident | ||
| 218 | -in our work. First, because they noticed that each new deploy made by us in the | ||
| 219 | -VE was stable and reliable. Second, they could see new features fast since we | ||
| 220 | -constantly updated the VE based on their feedback. This made our relation | ||
| 221 | -strong and in moments that needed quick action they would rather give us access | ||
| 222 | -to production. | ||
| 223 | - | ||
| 224 | - | ||
| 225 | -## Challenges | ||
| 226 | - | ||
| 227 | -We successfully built a functional CD pipeline. In the end, we took over the | ||
| 228 | -deployment process from the government. That allowed us to survive into an | ||
| 229 | -unstable political scenario. However, we recognized that many challenges still | ||
| 230 | -need to be addressed by the industry and academia together. Build CD from | ||
| 231 | -scratch | ||
| 232 | - | ||
| 233 | -Taking on CD responsibilities had a significant impact on the team. We did not | ||
| 234 | -have the know-how and had little time to come up with a working pipeline. To | ||
| 235 | -make things worse, we were not aware of how companies normally organized their | ||
| 236 | -teams to make CD feasible. | ||
| 237 | - | ||
| 238 | -[//]: # (TODO - Deixar uma abertura para uma eventual pesquisa) | ||
| 239 | - | ||
| 240 | -The seniors were crucial at this point. They came up with an initial solution | ||
| 241 | -to get us started. That already enabled us to automatize the deploy, even | ||
| 242 | -though the process was still rudimentary. We had to evolve our solution | ||
| 243 | -on-the-fly. We dedicated a few developers to this task. | ||
| 244 | - | ||
| 245 | -### Handling inexperienced teams | ||
| 246 | - | ||
| 247 | -After the developers learned how CD worked, it was difficult to pass the | ||
| 248 | -knowledge along to other teammates. We tried to mitigate this by encouraging a | ||
| 249 | -member's migration to the DevOps team. Further research on how to effectively | ||
| 250 | -spread knowledge across inexperienced developers in a scenario with a high | ||
| 251 | -turnover are needed. | ||
| 252 | - | ||
| 253 | -[//]: # (TODO - Refazer título abaixo - ideias: overcoming the challenges / overcoming the mistrust) | ||
| 254 | - | ||
| 255 | -### Building trust | ||
| 256 | - | ||
| 257 | -[//]: # (TODO - Deixar claro que government = os técnicos , análista ou agentes do governo, conforme for padronizado no texto) | ||
| 258 | - | ||
| 259 | -In the project’s first half we struggled with deploy related problems in the | ||
| 260 | -government structure. We were in a paradoxical situation. The government | ||
| 261 | -demanded speedy deliveries but would not give access to their production | ||
| 262 | -infrastructure. As an example, only in a very specific situation the government | ||
| 263 | -allowed us to access the PE. After some interactions with the government we | ||
| 264 | -convinced them to create the VE as an isolated replica of the PE in their own | ||
| 265 | -infrastructure. The government agents then realized that it could be good for | ||
| 266 | -the project if they granted us access to part of the structure since we could | ||
| 267 | -deliver new features to them faster. We believe it is required more research on | ||
| 268 | -development protocols and policies to improve the relation between industry and | ||
| 269 | -government, specially regarding CD. | ||
| 270 | - | ||
| 271 | - | ||
| 272 | -## References | ||
| 273 | - | ||
| 274 | -1. G. Booch, A. W. Brown, "Collaborative Development Environments, in Advances in Computers", vol. 59, 2003, pp. 1–27. | ||
| 275 | -1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54. | ||
| 276 | -1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", in Proceedings of the 13th International Symposium on Open Collaboration, 2017. | ||
| 277 | - | ||
| 278 | - | 1 | +# The Strategic Importance of Continuous Delivery |
| 2 | + | ||
| 3 | +## Abstract | ||
| 4 | + | ||
| 5 | +Many software development teams think of the operational aspects of Continuous | ||
| 6 | +Delivery (CD) and the competitive benefits that come along with it. For us, it | ||
| 7 | +was much more: it was a survival technique. This article presents the | ||
| 8 | +experience applying CD in a Brazilian Government project for the development of | ||
| 9 | +a Collaborative Development Environment (CDE), sharing its unconventional | ||
| 10 | +challenges and our strategies used to overcome them. This report from the | ||
| 11 | +trenches of the Brazilian Federal Government can help practitioners to | ||
| 12 | +understand how important the CD adoption is to their projects. | ||
| 13 | + | ||
| 14 | +## Introduction and Context | ||
| 15 | + | ||
| 16 | +We have worked on a Brazilian government three-year-long project to evolve an | ||
| 17 | +existing platform that had technical issues and a lack of political support. | ||
| 18 | +The evolution project started in a presidential election year and everyone | ||
| 19 | +involved were under Presidential re-election campaign pressure to show results. | ||
| 20 | +Even with the re-election of the Brazilian President in 2014, leadership in | ||
| 21 | +government agencies suddenly changed, what was worsened by corruption scandals | ||
| 22 | +revealed by the Brazilian Federal police, in particular the "Car-Wash" | ||
| 23 | +investigation. That reflected on the project’s requirements: each new leader | ||
| 24 | +wanted to fulfill their political agenda. In this scenario, delivery delays | ||
| 25 | +could have sunk the project into oblivion. | ||
| 26 | + | ||
| 27 | +From 2014 to 2016, our team developed the new platform for the Brazilian Public | ||
| 28 | +Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br) funded | ||
| 29 | +by a grant of 2,619,965.00 BRL (about 1,000,000.00 USD in January 2014) from | ||
| 30 | +the Federal Government. The SPB Portal has evolved to a CDE [1] and this | ||
| 31 | +evolution has brought important benefits not just to the Brazilian government, | ||
| 32 | +but also to society as a whole. For the government, the bureaucracy on using | ||
| 33 | +the same software across government agencies, duplicate works and costs all are | ||
| 34 | +reduced. The society gains a transparency and collaboration mechanism, since | ||
| 35 | +anyone can check the government expenses on software and contribute to the | ||
| 36 | +software communities. To achieve these goals, rather than writing everything | ||
| 37 | +from scratch, we have chosen to integrate free software tools such as Gitlab | ||
| 38 | +(www.gitlab.com), Mailman (www.gnu.org/software/mailman), Noosfero | ||
| 39 | +(www.noosfero.org), and Colab (www.github.com/colab). | ||
| 40 | + | ||
| 41 | +During the entire SPB Portal evolution project, we had to handle three distinct | ||
| 42 | +issues, usual in a software engineering scenario: reaching the goals which have | ||
| 43 | +guided the platform development, managing the diversity of team project | ||
| 44 | +members, and communicating effectively with clients (in this particular case, | ||
| 45 | +government agents). Managing the interaction of these elements was not easy and | ||
| 46 | +the unstable Brazilian political scenario only made things worse. | ||
| 47 | + | ||
| 48 | +To reach the SPB project goals, we had to overcome strong political bias | ||
| 49 | +tied with complicated technical issues and relatively low budget. Because it is | ||
| 50 | +open to the public, the government representatives have seen the platform as a | ||
| 51 | +marketing opportunity and have often ignored the technical advice in favor | ||
| 52 | +of political decisions. Furthermore, integrating a number of distinct systems | ||
| 53 | +to work seamlessly was not an easy job. We had to learn how each system worked | ||
| 54 | +and come up with ideas of how to integrate them as fast as possible, with a | ||
| 55 | +team of mostly inexperienced developers. | ||
| 56 | + | ||
| 57 | +We also had to manage the diversity of the SPB team members. This team was | ||
| 58 | +composed of approximately 50 undergraduate students (not all simultaneously) | ||
| 59 | +together with professors, masters students and professional designers as well | ||
| 60 | +as senior developers from the Brazilian free software community. Undergraduate | ||
| 61 | +students have received a fellowship and, for most of them, this R&D project was | ||
| 62 | +their first professional experience. Seniors developers and masters students | ||
| 63 | +had two important contributions to the project: transfer knowledge to | ||
| 64 | +undergraduate students and address hard tasks. Finally, professors were | ||
| 65 | +responsible for interacting with the Brazilian government and control the | ||
| 66 | +political pressures applied to the project. | ||
| 67 | + | ||
| 68 | +Our third point to be handled was the communication with the two independent | ||
| 69 | +groups of government representatives: requirements analysts and deployment | ||
| 70 | +technicians. Requirements analysts were the real representatives of the | ||
| 71 | +Brazilian government in the project. They usually tested new features, provided | ||
| 72 | +feedback, and reported for directors. The deployment technicians only had the | ||
| 73 | +access to the host machines wherein SPB platform was running. They were, | ||
| 74 | +theoretically, responsible for deploying the project. However, the new SPB | ||
| 75 | +Portal was composited from more than ten different software projects, | ||
| 76 | +generating a complex deploy process working on seven servers. | ||
| 77 | + | ||
| 78 | +Therefore, we realised that we needed to take control over the deployment | ||
| 79 | +process. We would use CD as a mean to keep the government satisfied and provide | ||
| 80 | +quick response times to their requests which were, most of the time, influenced | ||
| 81 | +by the uncertainties of project continuity. We believed that would keep the | ||
| 82 | +project alive even in a politically unstable and technically complex scenario. | ||
| 83 | +For that, during months we have worked to automate all the deploy process. For | ||
| 84 | +instance, we have created a tool based on Chef called Shak (Self Hosting | ||
| 85 | +Applications Kit) that offers an application installation interface based only | ||
| 86 | + on instruction. It aims to make the deployment process easier for users | ||
| 87 | + without technical knowledge, ensuring privacy and security for their Internet | ||
| 88 | + applications. Thereby, we created a specific team dedicated to the deployment | ||
| 89 | + process to mature our CD pipeline that would give us confidence to meet the | ||
| 90 | + government’s requirements faster and faster. | ||
| 91 | + | ||
| 92 | +We describe our CD pipeline and how it speeded up our software delivery time. | ||
| 93 | +The CD made us adaptable for all requested changes and helped us to mitigate | ||
| 94 | +those aforementioned political challenges besides the technical issues. Among | ||
| 95 | +CD’s known benefits[2], we also explain those that were the most important in | ||
| 96 | +the project scenario for us: improving customer satisfaction and making | ||
| 97 | +reliable releases. Both kept the project alive for more two years during the | ||
| 98 | +worst political crisis after the re-democratization in Brazil. | ||
| 99 | + | ||
| 100 | +## Our Continuous Delivery Pipeline | ||
| 101 | + | ||
| 102 | +[//]: # (TODO - Fazer a imagem. Sem esquentar muito a cabeça, uma vez que a revista deve refazer tal imagem) | ||
| 103 | + | ||
| 104 | +[//]: # (conjunto de features -> testes automatizados → criação de tag → empacotamento →deploy em homologação → testes de aceitação → deploy em produção) | ||
| 105 | + | ||
| 106 | +[//]: # (TODO - Rever essa frase - é uma legenda da imagem?) | ||
| 107 | + | ||
| 108 | + | ||
| 109 | + | ||
| 110 | +Figure X represents our CD pipeline. After each release, we have defined a set | ||
| 111 | +of features we would develop to comply with government’s requirements and then | ||
| 112 | +we have followed this pipeline. | ||
| 113 | + | ||
| 114 | +### Automated tests | ||
| 115 | + | ||
| 116 | +[//]: # (TODO - Rever a escrita desta ideia com mais calma) | ||
| 117 | +[//]: # (TODO - Adicionar que as mudanças no Colab quebravam todos os plugins.) | ||
| 118 | +[//]: # (https://www.linux.com/blog/learn/chapter/dev-ops/2017/7/devops-fundamentals-part-3-continuous-delivery-and-deployment) | ||
| 119 | + | ||
| 120 | +The SPB portal is composed of a set of FOSS systems. All of them have their own | ||
| 121 | +automated test suite. So, we encouraged the teams to keep unit tests coverage | ||
| 122 | +high for each system. We implemented the systems integration using a plugin | ||
| 123 | +architecture and wrote integration tests to check if they were working | ||
| 124 | +properly. The plugin architecture made the PSPB’s components decoupled. That | ||
| 125 | +allowed us to be confident that changes on one system would not affect others. | ||
| 126 | +This characteristic allowed the test execution of only the component with | ||
| 127 | +changes instead of the entire system. | ||
| 128 | + | ||
| 129 | +Our CD’s first step was to execute each system’s automated test suite. If any | ||
| 130 | +error was found, the pipeline stopped and the developers were notified. Only | ||
| 131 | +after all tests passed we move to preparing the release. Preparing a new | ||
| 132 | +release | ||
| 133 | + | ||
| 134 | +Our release system was divided into two perspectives: the application and the | ||
| 135 | +PSPB. The application tag refers to the specific feature or bug fix and is a | ||
| 136 | +monotonically increasing. A new tag on any system yielded a new PSPB tag. | ||
| 137 | + | ||
| 138 | +When all tests passed for a given component, we manually created a new | ||
| 139 | +application tag for it. As a consequence, that automatically created a new tag | ||
| 140 | +for the PSPB. Notice that we have forks of the original softwares and, as | ||
| 141 | +consequence, we had different tag values. Packaging | ||
| 142 | + | ||
| 143 | +The PSPB platform is running under the CentOS GNU/Linux distribution. | ||
| 144 | +Basically, packaging a software for that distribution has three steps: (1) | ||
| 145 | +write the script for the specific environment (RPM), (2) build the package, and | ||
| 146 | +(3) upload it to a package repository. We chose to package our components for | ||
| 147 | +several reasons: | ||
| 148 | +* Not all software was packaged by the community; | ||
| 149 | +* And those that existed were outdated; | ||
| 150 | +* Packaging makes it easy to manage the software on a given distribution; | ||
| 151 | +* It simplifies the deployment; | ||
| 152 | +* Packaging follows the distribution’s best practices and, | ||
| 153 | +* Allows configurations and permissions control. | ||
| 154 | + | ||
| 155 | +After creating a new tag for one component, the DevOps team was notified and | ||
| 156 | +packaging process began. In the normal case, the three packaging steps | ||
| 157 | +aforementioned are fully automated by a set of scripts. However, if the team | ||
| 158 | +reports to DevOps any eventual dependency change, the first step has to be | ||
| 159 | +manual. For instance, one system could start requiring another system to be | ||
| 160 | +initialized and that made necessary for someone to manually update the package | ||
| 161 | +script. | ||
| 162 | + | ||
| 163 | +After all these scripts have run successfully, the new packages would be ready | ||
| 164 | +to use by our subsequent deployment scripts. | ||
| 165 | + | ||
| 166 | +### Validation Environment | ||
| 167 | + | ||
| 168 | +[//]: # (TODO - Mencionar que a ferramenta era baseada em Chef - Dá um peso importante.) | ||
| 169 | + | ||
| 170 | +The Validation Environment (VE) is a replica of the Production Environment | ||
| 171 | +(PE), with two exceptions: only the government officers and us had access to it | ||
| 172 | +and all the data is anonymised. To configure the environment, we use a | ||
| 173 | +configuration management tool. That maintained environment consistency which | ||
| 174 | +makes the deployment process simpler. Additionally, the packages we built on | ||
| 175 | +the last step were readily available to use by the management tool. | ||
| 176 | + | ||
| 177 | +The VE was used by the government agents to validate new features and required | ||
| 178 | +changes. Also, the VE was useful to verify the integrity of the entire portal | ||
| 179 | +as part of the next step in the pipeline. Acceptance Tests | ||
| 180 | + | ||
| 181 | +After we completely deploy a new PSPB version in the VE, the government agents | ||
| 182 | +are responsible for checking features and/or bug fixes required by them. If the | ||
| 183 | +technicians identify a problem, they notify the developers, the problems are | ||
| 184 | +fixed and the pipeline restarts from scratch. If everything is validated, we | ||
| 185 | +move to production deployment. Production Deployment | ||
| 186 | + | ||
| 187 | +After the government authorizes the VE, we can finally begin the production | ||
| 188 | +deployment. We use the same configuration management tool with the same | ||
| 189 | +scripts. After the deploy is completed, both VE and PE are on the same state. | ||
| 190 | +The new features and bug fixes are finally available to end users. | ||
| 191 | + | ||
| 192 | + | ||
| 193 | +## Benefits | ||
| 194 | + | ||
| 195 | +We had to handle many tensions between development and political issues. Our CD | ||
| 196 | +pipeline gave us strong mechanisms to tackle most of the problems. As a result | ||
| 197 | +we came with some benefits from our decision to adopt CD. | ||
| 198 | + | ||
| 199 | +[//]: # (TODO - Melhorar título - Ideias: Response to mistrust) | ||
| 200 | + | ||
| 201 | +### Response to tensions | ||
| 202 | + | ||
| 203 | +[//]: # (TODO - Decisão do secretario acima do diretor) | ||
| 204 | + | ||
| 205 | +The direct benefit from the CD pipeline was the fast response to the changes | ||
| 206 | +required by the government. That was vital for the project’s renewal over the | ||
| 207 | +years. We could manage the tension between the government and the development | ||
| 208 | +team better. Every meeting with the government leader was delicate and resulted | ||
| 209 | +on many new requirements, most of them motivated by political needs. For | ||
| 210 | +example, once it was demanded a completely layout change because one director | ||
| 211 | +suddenly decided to make a marketing campaign about the portal. They would use | ||
| 212 | +undelivered requirements as a means to suggest the project’s cancellation. We | ||
| 213 | +believed that if we took too long to attend their demands, the project would | ||
| 214 | +end. CD helped us to move fast on deploying to production, even of smaller | ||
| 215 | +parts of the requirements. That way, we always had something to show on the | ||
| 216 | +meetings, reducing their eagerness to end the project. For our team, it made | ||
| 217 | +the developers more confident the project would last a little longer and they | ||
| 218 | +would not go looking for another jobs. | ||
| 219 | + | ||
| 220 | +### Build client’s trust | ||
| 221 | + | ||
| 222 | +After we established the CD, the government agents started to be more confident | ||
| 223 | +in our work. First, because they noticed that each new deploy made by us in the | ||
| 224 | +VE was stable and reliable. Second, they could see new features fast since we | ||
| 225 | +constantly updated the VE based on their feedback. This made our relation | ||
| 226 | +strong and in moments that needed quick action they would rather give us access | ||
| 227 | +to production. | ||
| 228 | + | ||
| 229 | + | ||
| 230 | +## Challenges | ||
| 231 | + | ||
| 232 | +We successfully built a functional CD pipeline. In the end, we took over the | ||
| 233 | +deployment process from the government. That allowed us to survive into an | ||
| 234 | +unstable political scenario. However, we recognized that many challenges still | ||
| 235 | +need to be addressed by the industry and academia together. Build CD from | ||
| 236 | +scratch | ||
| 237 | + | ||
| 238 | +Taking on CD responsibilities had a significant impact on the team. We did not | ||
| 239 | +have the know-how and had little time to come up with a working pipeline. To | ||
| 240 | +make things worse, we were not aware of how companies normally organized their | ||
| 241 | +teams to make CD feasible. | ||
| 242 | + | ||
| 243 | +[//]: # (TODO - Deixar uma abertura para uma eventual pesquisa) | ||
| 244 | + | ||
| 245 | +The seniors were crucial at this point. They came up with an initial solution | ||
| 246 | +to get us started. That already enabled us to automatize the deploy, even | ||
| 247 | +though the process was still rudimentary. We had to evolve our solution | ||
| 248 | +on-the-fly. We dedicated a few developers to this task. | ||
| 249 | + | ||
| 250 | +### Handling inexperienced teams | ||
| 251 | + | ||
| 252 | +After the developers learned how CD worked, it was difficult to pass the | ||
| 253 | +knowledge along to other teammates. We tried to mitigate this by encouraging a | ||
| 254 | +member's migration to the DevOps team. Further research on how to effectively | ||
| 255 | +spread knowledge across inexperienced developers in a scenario with a high | ||
| 256 | +turnover are needed. | ||
| 257 | + | ||
| 258 | +[//]: # (TODO - Refazer título abaixo - ideias: overcoming the challenges / overcoming the mistrust) | ||
| 259 | + | ||
| 260 | +### Building trust | ||
| 261 | + | ||
| 262 | +[//]: # (TODO - Deixar claro que government = os técnicos , análista ou agentes do governo, conforme for padronizado no texto) | ||
| 263 | + | ||
| 264 | +In the project’s first half we struggled with deploy related problems in the | ||
| 265 | +government structure. We were in a paradoxical situation. The government | ||
| 266 | +demanded speedy deliveries but would not give access to their production | ||
| 267 | +infrastructure. As an example, only in a very specific situation the government | ||
| 268 | +allowed us to access the PE. After some interactions with the government we | ||
| 269 | +convinced them to create the VE as an isolated replica of the PE in their own | ||
| 270 | +infrastructure. The government agents then realized that it could be good for | ||
| 271 | +the project if they granted us access to part of the structure since we could | ||
| 272 | +deliver new features to them faster. We believe it is required more research on | ||
| 273 | +development protocols and policies to improve the relation between industry and | ||
| 274 | +government, specially regarding CD. | ||
| 275 | + | ||
| 276 | + | ||
| 277 | +## References | ||
| 278 | + | ||
| 279 | +1. G. Booch, A. W. Brown, "Collaborative Development Environments, in Advances in Computers", vol. 59, 2003, pp. 1–27. | ||
| 280 | +1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54. | ||
| 281 | +1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", in Proceedings of the 13th International Symposium on Open Collaboration, 2017. | ||
| 282 | + | ||
| 283 | + |
No preview for this file type
63.5 KB