Commit 2b94e087a4fca0ee5f0c2709e9cf05025e8dc9c5
1 parent
ff8bdf06
Exists in
master
and in
3 other branches
Create pipeline image
Showing
3 changed files
with
283 additions
and
278 deletions
Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
| 1 | -# The Strategic Importance of Continuous Delivery | |
| 2 | - | |
| 3 | -## Abstract | |
| 4 | - | |
| 5 | -Many software development teams think of the operational aspects of Continuous | |
| 6 | -Delivery (CD) and the competitive benefits that come along with it. For us, it | |
| 7 | -was much more: it was a survival technique. This article presents the | |
| 8 | -experience applying CD in a Brazilian Government project for the development of | |
| 9 | -a Collaborative Development Environment (CDE), sharing its unconventional | |
| 10 | -challenges and our strategies used to overcome them. This report from the | |
| 11 | -trenches of the Brazilian Federal Government can help practitioners to | |
| 12 | -understand how important the CD adoption is to their projects. | |
| 13 | - | |
| 14 | -## Introduction and Context | |
| 15 | - | |
| 16 | -We have worked on a Brazilian government three-year-long project to redesign an | |
| 17 | -existing platform that had technical issues and a lack of political support. | |
| 18 | -This evolution project started in a presidential election year and everyone | |
| 19 | -involved were under Presidential re-election campaign pressure to show results. | |
| 20 | -Even with the re-election of the Brazilian President, leadership in government | |
| 21 | -agencies suddenly changed that reflected on the project’s requirements: each | |
| 22 | -new leader wanted to fulfill their political agenda. In this scenario, delivery | |
| 23 | -delays could have sunk the project into oblivion. | |
| 24 | - | |
| 25 | -From 2013 to 2016, our team developed the new platform for the Brazilian Public | |
| 26 | -Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br). The SPB | |
| 27 | -Portal has evolved to a CDE [1] and this evolution has brought important | |
| 28 | -benefits not just to the Brazilian government, but also to society as a whole. | |
| 29 | -For the government, the bureaucracy on using the same software across | |
| 30 | -government agencies, duplicate works and costs all are reduced. The society | |
| 31 | -gains a transparency and collaboration mechanism, since anyone can check the | |
| 32 | -government expenses on software and contribute to the software communities. To | |
| 33 | -achieve these goals, rather than writing everything from scratch, we have | |
| 34 | -chosen to integrate free software tools such as Gitlab (www.gitlab.com), | |
| 35 | -Mailman (www.gnu.org/software/mailman), Noosfero (www.noosfero.org), and | |
| 36 | -Colab (www.github.com/colab). | |
| 37 | - | |
| 38 | -During the entire SPB Portal evolution project, we had to handle three distinct | |
| 39 | -issues, usual in a software engineering scenario: reaching the goals which have | |
| 40 | -guided the platform development, managing the diversity of team project | |
| 41 | -members, and communicating effectively with clients (in this particular case, | |
| 42 | -government agents). Managing the interaction of these elements was not easy and | |
| 43 | -the unstable Brazilian political scenario only made things worse. | |
| 44 | - | |
| 45 | -To reach the SPB project goals, we had to overcome strong political bias | |
| 46 | -tied with complicated technical issues and relatively low budget. Because it is | |
| 47 | -open to the public, the government representatives have seen the platform as a | |
| 48 | -marketing opportunity and have often ignored the technical advice in favor | |
| 49 | -of political decisions. Furthermore, integrating a number of distinct systems | |
| 50 | -to work seamlessly was not an easy job. We had to learn how each system worked | |
| 51 | -and come up with ideas of how to integrate them as fast as possible, with a | |
| 52 | -team of mostly inexperienced developers. | |
| 53 | - | |
| 54 | -We also had to manage the diversity of the SPB team members. This team was | |
| 55 | -composed of approximately 50 undergraduate students (not all simultaneously) | |
| 56 | -together with professors, masters students and professional designers as well | |
| 57 | -as senior developers from the Brazilian free software community. Undergraduate | |
| 58 | -students have received a fellowship and, for most of them, this R&D project was | |
| 59 | -their first professional experience. Seniors developers and masters students | |
| 60 | -had two important contributions to the project: transfer knowledge to | |
| 61 | -undergraduate students and address hard tasks. Finally, professors were | |
| 62 | -responsible for interacting with the Brazilian government and control the | |
| 63 | -political pressures applied to the project. | |
| 64 | - | |
| 65 | -Our third point to be handled was the communication with the two independent | |
| 66 | -groups of government representatives: requirements analysts and deployment | |
| 67 | -technicians. Requirements analysts were the real representatives of the | |
| 68 | -Brazilian government in the project. They usually tested new features, provided | |
| 69 | -feedback, and reported for directors. The deployment technicians only had the | |
| 70 | -access to the host machines wherein SPB platform was running. They were, | |
| 71 | -theoretically, responsible for deploying the project. However, the new SPB | |
| 72 | -Portal was composited from more than ten different software projects, | |
| 73 | -generating a complex deploy process working on seven servers. | |
| 74 | - | |
| 75 | -Therefore, we realised that we needed to take control over the deployment | |
| 76 | -process. We would use CD as a mean to keep the government satisfied and provide | |
| 77 | -quick response times to their requests which were, most of the time, influenced | |
| 78 | -by the uncertainties of project continuity. We believed that would keep the | |
| 79 | -project alive even in an politically unstable and complex technical scenario. | |
| 80 | -For that, during months we have worked to automate all the deploy process. For | |
| 81 | -instance, we have created a tool called Shak (Self Hosting Applications Kit) | |
| 82 | -that offers an application installation with only instruction. It aims to make | |
| 83 | -the deploy process easier for users without technical knowledge, ensuring | |
| 84 | -privacy and security for their Internet applications. Thereby, we created a | |
| 85 | -specific team dedicated to the deployment process to mature our CD pipeline | |
| 86 | -that would give us confidence to meet the government’s requirements faster and | |
| 87 | -faster. | |
| 88 | - | |
| 89 | -We describe our CD pipeline and how it speeded up our software delivery time. | |
| 90 | -The CD made us adaptable for all requested changes and helped us to mitigate | |
| 91 | -those aforementioned political challenges besides the technical issues. Among | |
| 92 | -CD’s known benefits[2], we also explain those that were the most important in | |
| 93 | -the project scenario for us: improving customer satisfaction and making | |
| 94 | -reliable releases. Both kept the project alive for more two years during the | |
| 95 | -worst political crisis after the re-democratization in Brazil. | |
| 96 | - | |
| 97 | -## Our Continuous Delivery Pipeline | |
| 98 | - | |
| 99 | -[//]: # (TODO - Fazer a imagem. Sem esquentar muito a cabeça, uma vez que a revista deve refazer tal imagem) | |
| 100 | - | |
| 101 | -[//]: # (conjunto de features -> testes automatizados → criação de tag → empacotamento →deploy em homologação → testes de aceitação → deploy em produção) | |
| 102 | - | |
| 103 | -[//]: # (TODO - Rever essa frase - é uma legenda da imagem?) | |
| 104 | - | |
| 105 | -Figure X represents our CD pipeline. After each release, we have defined a set | |
| 106 | -of features we would develop to comply with government’s requirements and then | |
| 107 | -we have followed this pipeline. | |
| 108 | - | |
| 109 | -### Automated tests | |
| 110 | - | |
| 111 | -[//]: # (TODO - Rever a escrita desta ideia com mais calma) | |
| 112 | -[//]: # (TODO - Adicionar que as mudanças no Colab quebravam todos os plugins.) | |
| 113 | -[//]: # (https://www.linux.com/blog/learn/chapter/dev-ops/2017/7/devops-fundamentals-part-3-continuous-delivery-and-deployment) | |
| 114 | - | |
| 115 | -The SPB portal is composed of a set of FOSS systems. All of them have their own | |
| 116 | -automated test suite. So, we encouraged the teams to keep unit tests coverage | |
| 117 | -high for each system. We implemented the systems integration using a plugin | |
| 118 | -architecture and wrote integration tests to check if they were working | |
| 119 | -properly. The plugin architecture made the PSPB’s components decoupled. That | |
| 120 | -allowed us to be confident that changes on one system would not affect others. | |
| 121 | -This characteristic allowed the test execution of only the component with | |
| 122 | -changes instead of the entire system. | |
| 123 | - | |
| 124 | -Our CD’s first step was to execute each system’s automated test suite. If any | |
| 125 | -error was found, the pipeline stopped and the developers were notified. Only | |
| 126 | -after all tests passed we move to preparing the release. Preparing a new | |
| 127 | -release | |
| 128 | - | |
| 129 | -Our release system was divided into two perspectives: the application and the | |
| 130 | -PSPB. The application tag refers to the specific feature or bug fix and is a | |
| 131 | -monotonically increasing. A new tag on any system yielded a new PSPB tag. | |
| 132 | - | |
| 133 | -When all tests passed for a given component, we manually created a new | |
| 134 | -application tag for it. As a consequence, that automatically created a new tag | |
| 135 | -for the PSPB. Notice that we have forks of the original softwares and, as | |
| 136 | -consequence, we had different tag values. Packaging | |
| 137 | - | |
| 138 | -The PSPB platform is running under the CentOS GNU/Linux distribution. | |
| 139 | -Basically, packaging a software for that distribution has three steps: (1) | |
| 140 | -write the script for the specific environment (RPM), (2) build the package, and | |
| 141 | -(3) upload it to a package repository. We chose to package our components for | |
| 142 | -several reasons: | |
| 143 | -* Not all software was packaged by the community; | |
| 144 | -* And those that existed were outdated; | |
| 145 | -* Packaging makes it easy to manage the software on a given distribution; | |
| 146 | -* It simplifies the deployment; | |
| 147 | -* Packaging follows the distribution’s best practices and, | |
| 148 | -* Allows configurations and permissions control. | |
| 149 | - | |
| 150 | -After creating a new tag for one component, the DevOps team was notified and | |
| 151 | -packaging process began. In the normal case, the three packaging steps | |
| 152 | -aforementioned are fully automated by a set of scripts. However, if the team | |
| 153 | -reports to DevOps any eventual dependency change, the first step has to be | |
| 154 | -manual. For instance, one system could start requiring another system to be | |
| 155 | -initialized and that made necessary for someone to manually update the package | |
| 156 | -script. | |
| 157 | - | |
| 158 | -After all these scripts have run successfully, the new packages would be ready | |
| 159 | -to use by our subsequent deployment scripts. | |
| 160 | - | |
| 161 | -### Validation Environment | |
| 162 | - | |
| 163 | -[//]: # (TODO - Mencionar que a ferramenta era baseada em Chef - Dá um peso importante.) | |
| 164 | - | |
| 165 | -The Validation Environment (VE) is a replica of the Production Environment | |
| 166 | -(PE), with two exceptions: only the government officers and us had access to it | |
| 167 | -and all the data is anonymised. To configure the environment, we use a | |
| 168 | -configuration management tool. That maintained environment consistency which | |
| 169 | -makes the deployment process simpler. Additionally, the packages we built on | |
| 170 | -the last step were readily available to use by the management tool. | |
| 171 | - | |
| 172 | -The VE was used by the government agents to validate new features and required | |
| 173 | -changes. Also, the VE was useful to verify the integrity of the entire portal | |
| 174 | -as part of the next step in the pipeline. Acceptance Tests | |
| 175 | - | |
| 176 | -After we completely deploy a new PSPB version in the VE, the government agents | |
| 177 | -are responsible for checking features and/or bug fixes required by them. If the | |
| 178 | -technicians identify a problem, they notify the developers, the problems are | |
| 179 | -fixed and the pipeline restarts from scratch. If everything is validated, we | |
| 180 | -move to production deployment. Production Deployment | |
| 181 | - | |
| 182 | -After the government authorizes the VE, we can finally begin the production | |
| 183 | -deployment. We use the same configuration management tool with the same | |
| 184 | -scripts. After the deploy is completed, both VE and PE are on the same state. | |
| 185 | -The new features and bug fixes are finally available to end users. | |
| 186 | - | |
| 187 | - | |
| 188 | -## Benefits | |
| 189 | - | |
| 190 | -We had to handle many tensions between development and political issues. Our CD | |
| 191 | -pipeline gave us strong mechanisms to tackle most of the problems. As a result | |
| 192 | -we came with some benefits from our decision to adopt CD. | |
| 193 | - | |
| 194 | -[//]: # (TODO - Melhorar título - Ideias: Response to mistrust) | |
| 195 | - | |
| 196 | -### Response to tensions | |
| 197 | - | |
| 198 | -[//]: # (TODO - Decisão do secretario acima do diretor) | |
| 199 | - | |
| 200 | -The direct benefit from the CD pipeline was the fast response to the changes | |
| 201 | -required by the government. That was vital for the project’s renewal over the | |
| 202 | -years. We could manage the tension between the government and the development | |
| 203 | -team better. Every meeting with the government leader was delicate and resulted | |
| 204 | -on many new requirements, most of them motivated by political needs. For | |
| 205 | -example, once it was demanded a completely layout change because one director | |
| 206 | -suddenly decided to make a marketing campaign about the portal. They would use | |
| 207 | -undelivered requirements as a means to suggest the project’s cancellation. We | |
| 208 | -believed that if we took too long to attend their demands, the project would | |
| 209 | -end. CD helped us to move fast on deploying to production, even of smaller | |
| 210 | -parts of the requirements. That way, we always had something to show on the | |
| 211 | -meetings, reducing their eagerness to end the project. For our team, it made | |
| 212 | -the developers more confident the project would last a little longer and they | |
| 213 | -would not go looking for another jobs. | |
| 214 | - | |
| 215 | -### Build client’s trust | |
| 216 | - | |
| 217 | -After we established the CD, the government agents started to be more confident | |
| 218 | -in our work. First, because they noticed that each new deploy made by us in the | |
| 219 | -VE was stable and reliable. Second, they could see new features fast since we | |
| 220 | -constantly updated the VE based on their feedback. This made our relation | |
| 221 | -strong and in moments that needed quick action they would rather give us access | |
| 222 | -to production. | |
| 223 | - | |
| 224 | - | |
| 225 | -## Challenges | |
| 226 | - | |
| 227 | -We successfully built a functional CD pipeline. In the end, we took over the | |
| 228 | -deployment process from the government. That allowed us to survive into an | |
| 229 | -unstable political scenario. However, we recognized that many challenges still | |
| 230 | -need to be addressed by the industry and academia together. Build CD from | |
| 231 | -scratch | |
| 232 | - | |
| 233 | -Taking on CD responsibilities had a significant impact on the team. We did not | |
| 234 | -have the know-how and had little time to come up with a working pipeline. To | |
| 235 | -make things worse, we were not aware of how companies normally organized their | |
| 236 | -teams to make CD feasible. | |
| 237 | - | |
| 238 | -[//]: # (TODO - Deixar uma abertura para uma eventual pesquisa) | |
| 239 | - | |
| 240 | -The seniors were crucial at this point. They came up with an initial solution | |
| 241 | -to get us started. That already enabled us to automatize the deploy, even | |
| 242 | -though the process was still rudimentary. We had to evolve our solution | |
| 243 | -on-the-fly. We dedicated a few developers to this task. | |
| 244 | - | |
| 245 | -### Handling inexperienced teams | |
| 246 | - | |
| 247 | -After the developers learned how CD worked, it was difficult to pass the | |
| 248 | -knowledge along to other teammates. We tried to mitigate this by encouraging a | |
| 249 | -member's migration to the DevOps team. Further research on how to effectively | |
| 250 | -spread knowledge across inexperienced developers in a scenario with a high | |
| 251 | -turnover are needed. | |
| 252 | - | |
| 253 | -[//]: # (TODO - Refazer título abaixo - ideias: overcoming the challenges / overcoming the mistrust) | |
| 254 | - | |
| 255 | -### Building trust | |
| 256 | - | |
| 257 | -[//]: # (TODO - Deixar claro que government = os técnicos , análista ou agentes do governo, conforme for padronizado no texto) | |
| 258 | - | |
| 259 | -In the project’s first half we struggled with deploy related problems in the | |
| 260 | -government structure. We were in a paradoxical situation. The government | |
| 261 | -demanded speedy deliveries but would not give access to their production | |
| 262 | -infrastructure. As an example, only in a very specific situation the government | |
| 263 | -allowed us to access the PE. After some interactions with the government we | |
| 264 | -convinced them to create the VE as an isolated replica of the PE in their own | |
| 265 | -infrastructure. The government agents then realized that it could be good for | |
| 266 | -the project if they granted us access to part of the structure since we could | |
| 267 | -deliver new features to them faster. We believe it is required more research on | |
| 268 | -development protocols and policies to improve the relation between industry and | |
| 269 | -government, specially regarding CD. | |
| 270 | - | |
| 271 | - | |
| 272 | -## References | |
| 273 | - | |
| 274 | -1. G. Booch, A. W. Brown, "Collaborative Development Environments, in Advances in Computers", vol. 59, 2003, pp. 1–27. | |
| 275 | -1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54. | |
| 276 | -1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", in Proceedings of the 13th International Symposium on Open Collaboration, 2017. | |
| 277 | - | |
| 278 | - | |
| 1 | +# The Strategic Importance of Continuous Delivery | |
| 2 | + | |
| 3 | +## Abstract | |
| 4 | + | |
| 5 | +Many software development teams think of the operational aspects of Continuous | |
| 6 | +Delivery (CD) and the competitive benefits that come along with it. For us, it | |
| 7 | +was much more: it was a survival technique. This article presents the | |
| 8 | +experience applying CD in a Brazilian Government project for the development of | |
| 9 | +a Collaborative Development Environment (CDE), sharing its unconventional | |
| 10 | +challenges and our strategies used to overcome them. This report from the | |
| 11 | +trenches of the Brazilian Federal Government can help practitioners to | |
| 12 | +understand how important the CD adoption is to their projects. | |
| 13 | + | |
| 14 | +## Introduction and Context | |
| 15 | + | |
| 16 | +We have worked on a Brazilian government three-year-long project to evolve an | |
| 17 | +existing platform that had technical issues and a lack of political support. | |
| 18 | +The evolution project started in a presidential election year and everyone | |
| 19 | +involved were under Presidential re-election campaign pressure to show results. | |
| 20 | +Even with the re-election of the Brazilian President in 2014, leadership in | |
| 21 | +government agencies suddenly changed, what was worsened by corruption scandals | |
| 22 | +revealed by the Brazilian Federal police, in particular the "Car-Wash" | |
| 23 | +investigation. That reflected on the project’s requirements: each new leader | |
| 24 | +wanted to fulfill their political agenda. In this scenario, delivery delays | |
| 25 | +could have sunk the project into oblivion. | |
| 26 | + | |
| 27 | +From 2014 to 2016, our team developed the new platform for the Brazilian Public | |
| 28 | +Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br) funded | |
| 29 | +by a grant of 2,619,965.00 BRL (about 1,000,000.00 USD in January 2014) from | |
| 30 | +the Federal Government. The SPB Portal has evolved to a CDE [1] and this | |
| 31 | +evolution has brought important benefits not just to the Brazilian government, | |
| 32 | +but also to society as a whole. For the government, the bureaucracy on using | |
| 33 | +the same software across government agencies, duplicate works and costs all are | |
| 34 | +reduced. The society gains a transparency and collaboration mechanism, since | |
| 35 | +anyone can check the government expenses on software and contribute to the | |
| 36 | +software communities. To achieve these goals, rather than writing everything | |
| 37 | +from scratch, we have chosen to integrate free software tools such as Gitlab | |
| 38 | +(www.gitlab.com), Mailman (www.gnu.org/software/mailman), Noosfero | |
| 39 | +(www.noosfero.org), and Colab (www.github.com/colab). | |
| 40 | + | |
| 41 | +During the entire SPB Portal evolution project, we had to handle three distinct | |
| 42 | +issues, usual in a software engineering scenario: reaching the goals which have | |
| 43 | +guided the platform development, managing the diversity of team project | |
| 44 | +members, and communicating effectively with clients (in this particular case, | |
| 45 | +government agents). Managing the interaction of these elements was not easy and | |
| 46 | +the unstable Brazilian political scenario only made things worse. | |
| 47 | + | |
| 48 | +To reach the SPB project goals, we had to overcome strong political bias | |
| 49 | +tied with complicated technical issues and relatively low budget. Because it is | |
| 50 | +open to the public, the government representatives have seen the platform as a | |
| 51 | +marketing opportunity and have often ignored the technical advice in favor | |
| 52 | +of political decisions. Furthermore, integrating a number of distinct systems | |
| 53 | +to work seamlessly was not an easy job. We had to learn how each system worked | |
| 54 | +and come up with ideas of how to integrate them as fast as possible, with a | |
| 55 | +team of mostly inexperienced developers. | |
| 56 | + | |
| 57 | +We also had to manage the diversity of the SPB team members. This team was | |
| 58 | +composed of approximately 50 undergraduate students (not all simultaneously) | |
| 59 | +together with professors, masters students and professional designers as well | |
| 60 | +as senior developers from the Brazilian free software community. Undergraduate | |
| 61 | +students have received a fellowship and, for most of them, this R&D project was | |
| 62 | +their first professional experience. Seniors developers and masters students | |
| 63 | +had two important contributions to the project: transfer knowledge to | |
| 64 | +undergraduate students and address hard tasks. Finally, professors were | |
| 65 | +responsible for interacting with the Brazilian government and control the | |
| 66 | +political pressures applied to the project. | |
| 67 | + | |
| 68 | +Our third point to be handled was the communication with the two independent | |
| 69 | +groups of government representatives: requirements analysts and deployment | |
| 70 | +technicians. Requirements analysts were the real representatives of the | |
| 71 | +Brazilian government in the project. They usually tested new features, provided | |
| 72 | +feedback, and reported for directors. The deployment technicians only had the | |
| 73 | +access to the host machines wherein SPB platform was running. They were, | |
| 74 | +theoretically, responsible for deploying the project. However, the new SPB | |
| 75 | +Portal was composited from more than ten different software projects, | |
| 76 | +generating a complex deploy process working on seven servers. | |
| 77 | + | |
| 78 | +Therefore, we realised that we needed to take control over the deployment | |
| 79 | +process. We would use CD as a mean to keep the government satisfied and provide | |
| 80 | +quick response times to their requests which were, most of the time, influenced | |
| 81 | +by the uncertainties of project continuity. We believed that would keep the | |
| 82 | +project alive even in a politically unstable and technically complex scenario. | |
| 83 | +For that, during months we have worked to automate all the deploy process. For | |
| 84 | +instance, we have created a tool based on Chef called Shak (Self Hosting | |
| 85 | +Applications Kit) that offers an application installation interface based only | |
| 86 | + on instruction. It aims to make the deployment process easier for users | |
| 87 | + without technical knowledge, ensuring privacy and security for their Internet | |
| 88 | + applications. Thereby, we created a specific team dedicated to the deployment | |
| 89 | + process to mature our CD pipeline that would give us confidence to meet the | |
| 90 | + government’s requirements faster and faster. | |
| 91 | + | |
| 92 | +We describe our CD pipeline and how it speeded up our software delivery time. | |
| 93 | +The CD made us adaptable for all requested changes and helped us to mitigate | |
| 94 | +those aforementioned political challenges besides the technical issues. Among | |
| 95 | +CD’s known benefits[2], we also explain those that were the most important in | |
| 96 | +the project scenario for us: improving customer satisfaction and making | |
| 97 | +reliable releases. Both kept the project alive for more two years during the | |
| 98 | +worst political crisis after the re-democratization in Brazil. | |
| 99 | + | |
| 100 | +## Our Continuous Delivery Pipeline | |
| 101 | + | |
| 102 | +[//]: # (TODO - Fazer a imagem. Sem esquentar muito a cabeça, uma vez que a revista deve refazer tal imagem) | |
| 103 | + | |
| 104 | +[//]: # (conjunto de features -> testes automatizados → criação de tag → empacotamento →deploy em homologação → testes de aceitação → deploy em produção) | |
| 105 | + | |
| 106 | +[//]: # (TODO - Rever essa frase - é uma legenda da imagem?) | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | +Figure X represents our CD pipeline. After each release, we have defined a set | |
| 111 | +of features we would develop to comply with government’s requirements and then | |
| 112 | +we have followed this pipeline. | |
| 113 | + | |
| 114 | +### Automated tests | |
| 115 | + | |
| 116 | +[//]: # (TODO - Rever a escrita desta ideia com mais calma) | |
| 117 | +[//]: # (TODO - Adicionar que as mudanças no Colab quebravam todos os plugins.) | |
| 118 | +[//]: # (https://www.linux.com/blog/learn/chapter/dev-ops/2017/7/devops-fundamentals-part-3-continuous-delivery-and-deployment) | |
| 119 | + | |
| 120 | +The SPB portal is composed of a set of FOSS systems. All of them have their own | |
| 121 | +automated test suite. So, we encouraged the teams to keep unit tests coverage | |
| 122 | +high for each system. We implemented the systems integration using a plugin | |
| 123 | +architecture and wrote integration tests to check if they were working | |
| 124 | +properly. The plugin architecture made the PSPB’s components decoupled. That | |
| 125 | +allowed us to be confident that changes on one system would not affect others. | |
| 126 | +This characteristic allowed the test execution of only the component with | |
| 127 | +changes instead of the entire system. | |
| 128 | + | |
| 129 | +Our CD’s first step was to execute each system’s automated test suite. If any | |
| 130 | +error was found, the pipeline stopped and the developers were notified. Only | |
| 131 | +after all tests passed we move to preparing the release. Preparing a new | |
| 132 | +release | |
| 133 | + | |
| 134 | +Our release system was divided into two perspectives: the application and the | |
| 135 | +PSPB. The application tag refers to the specific feature or bug fix and is a | |
| 136 | +monotonically increasing. A new tag on any system yielded a new PSPB tag. | |
| 137 | + | |
| 138 | +When all tests passed for a given component, we manually created a new | |
| 139 | +application tag for it. As a consequence, that automatically created a new tag | |
| 140 | +for the PSPB. Notice that we have forks of the original softwares and, as | |
| 141 | +consequence, we had different tag values. Packaging | |
| 142 | + | |
| 143 | +The PSPB platform is running under the CentOS GNU/Linux distribution. | |
| 144 | +Basically, packaging a software for that distribution has three steps: (1) | |
| 145 | +write the script for the specific environment (RPM), (2) build the package, and | |
| 146 | +(3) upload it to a package repository. We chose to package our components for | |
| 147 | +several reasons: | |
| 148 | +* Not all software was packaged by the community; | |
| 149 | +* And those that existed were outdated; | |
| 150 | +* Packaging makes it easy to manage the software on a given distribution; | |
| 151 | +* It simplifies the deployment; | |
| 152 | +* Packaging follows the distribution’s best practices and, | |
| 153 | +* Allows configurations and permissions control. | |
| 154 | + | |
| 155 | +After creating a new tag for one component, the DevOps team was notified and | |
| 156 | +packaging process began. In the normal case, the three packaging steps | |
| 157 | +aforementioned are fully automated by a set of scripts. However, if the team | |
| 158 | +reports to DevOps any eventual dependency change, the first step has to be | |
| 159 | +manual. For instance, one system could start requiring another system to be | |
| 160 | +initialized and that made necessary for someone to manually update the package | |
| 161 | +script. | |
| 162 | + | |
| 163 | +After all these scripts have run successfully, the new packages would be ready | |
| 164 | +to use by our subsequent deployment scripts. | |
| 165 | + | |
| 166 | +### Validation Environment | |
| 167 | + | |
| 168 | +[//]: # (TODO - Mencionar que a ferramenta era baseada em Chef - Dá um peso importante.) | |
| 169 | + | |
| 170 | +The Validation Environment (VE) is a replica of the Production Environment | |
| 171 | +(PE), with two exceptions: only the government officers and us had access to it | |
| 172 | +and all the data is anonymised. To configure the environment, we use a | |
| 173 | +configuration management tool. That maintained environment consistency which | |
| 174 | +makes the deployment process simpler. Additionally, the packages we built on | |
| 175 | +the last step were readily available to use by the management tool. | |
| 176 | + | |
| 177 | +The VE was used by the government agents to validate new features and required | |
| 178 | +changes. Also, the VE was useful to verify the integrity of the entire portal | |
| 179 | +as part of the next step in the pipeline. Acceptance Tests | |
| 180 | + | |
| 181 | +After we completely deploy a new PSPB version in the VE, the government agents | |
| 182 | +are responsible for checking features and/or bug fixes required by them. If the | |
| 183 | +technicians identify a problem, they notify the developers, the problems are | |
| 184 | +fixed and the pipeline restarts from scratch. If everything is validated, we | |
| 185 | +move to production deployment. Production Deployment | |
| 186 | + | |
| 187 | +After the government authorizes the VE, we can finally begin the production | |
| 188 | +deployment. We use the same configuration management tool with the same | |
| 189 | +scripts. After the deploy is completed, both VE and PE are on the same state. | |
| 190 | +The new features and bug fixes are finally available to end users. | |
| 191 | + | |
| 192 | + | |
| 193 | +## Benefits | |
| 194 | + | |
| 195 | +We had to handle many tensions between development and political issues. Our CD | |
| 196 | +pipeline gave us strong mechanisms to tackle most of the problems. As a result | |
| 197 | +we came with some benefits from our decision to adopt CD. | |
| 198 | + | |
| 199 | +[//]: # (TODO - Melhorar título - Ideias: Response to mistrust) | |
| 200 | + | |
| 201 | +### Response to tensions | |
| 202 | + | |
| 203 | +[//]: # (TODO - Decisão do secretario acima do diretor) | |
| 204 | + | |
| 205 | +The direct benefit from the CD pipeline was the fast response to the changes | |
| 206 | +required by the government. That was vital for the project’s renewal over the | |
| 207 | +years. We could manage the tension between the government and the development | |
| 208 | +team better. Every meeting with the government leader was delicate and resulted | |
| 209 | +on many new requirements, most of them motivated by political needs. For | |
| 210 | +example, once it was demanded a completely layout change because one director | |
| 211 | +suddenly decided to make a marketing campaign about the portal. They would use | |
| 212 | +undelivered requirements as a means to suggest the project’s cancellation. We | |
| 213 | +believed that if we took too long to attend their demands, the project would | |
| 214 | +end. CD helped us to move fast on deploying to production, even of smaller | |
| 215 | +parts of the requirements. That way, we always had something to show on the | |
| 216 | +meetings, reducing their eagerness to end the project. For our team, it made | |
| 217 | +the developers more confident the project would last a little longer and they | |
| 218 | +would not go looking for another jobs. | |
| 219 | + | |
| 220 | +### Build client’s trust | |
| 221 | + | |
| 222 | +After we established the CD, the government agents started to be more confident | |
| 223 | +in our work. First, because they noticed that each new deploy made by us in the | |
| 224 | +VE was stable and reliable. Second, they could see new features fast since we | |
| 225 | +constantly updated the VE based on their feedback. This made our relation | |
| 226 | +strong and in moments that needed quick action they would rather give us access | |
| 227 | +to production. | |
| 228 | + | |
| 229 | + | |
| 230 | +## Challenges | |
| 231 | + | |
| 232 | +We successfully built a functional CD pipeline. In the end, we took over the | |
| 233 | +deployment process from the government. That allowed us to survive into an | |
| 234 | +unstable political scenario. However, we recognized that many challenges still | |
| 235 | +need to be addressed by the industry and academia together. Build CD from | |
| 236 | +scratch | |
| 237 | + | |
| 238 | +Taking on CD responsibilities had a significant impact on the team. We did not | |
| 239 | +have the know-how and had little time to come up with a working pipeline. To | |
| 240 | +make things worse, we were not aware of how companies normally organized their | |
| 241 | +teams to make CD feasible. | |
| 242 | + | |
| 243 | +[//]: # (TODO - Deixar uma abertura para uma eventual pesquisa) | |
| 244 | + | |
| 245 | +The seniors were crucial at this point. They came up with an initial solution | |
| 246 | +to get us started. That already enabled us to automatize the deploy, even | |
| 247 | +though the process was still rudimentary. We had to evolve our solution | |
| 248 | +on-the-fly. We dedicated a few developers to this task. | |
| 249 | + | |
| 250 | +### Handling inexperienced teams | |
| 251 | + | |
| 252 | +After the developers learned how CD worked, it was difficult to pass the | |
| 253 | +knowledge along to other teammates. We tried to mitigate this by encouraging a | |
| 254 | +member's migration to the DevOps team. Further research on how to effectively | |
| 255 | +spread knowledge across inexperienced developers in a scenario with a high | |
| 256 | +turnover are needed. | |
| 257 | + | |
| 258 | +[//]: # (TODO - Refazer título abaixo - ideias: overcoming the challenges / overcoming the mistrust) | |
| 259 | + | |
| 260 | +### Building trust | |
| 261 | + | |
| 262 | +[//]: # (TODO - Deixar claro que government = os técnicos , análista ou agentes do governo, conforme for padronizado no texto) | |
| 263 | + | |
| 264 | +In the project’s first half we struggled with deploy related problems in the | |
| 265 | +government structure. We were in a paradoxical situation. The government | |
| 266 | +demanded speedy deliveries but would not give access to their production | |
| 267 | +infrastructure. As an example, only in a very specific situation the government | |
| 268 | +allowed us to access the PE. After some interactions with the government we | |
| 269 | +convinced them to create the VE as an isolated replica of the PE in their own | |
| 270 | +infrastructure. The government agents then realized that it could be good for | |
| 271 | +the project if they granted us access to part of the structure since we could | |
| 272 | +deliver new features to them faster. We believe it is required more research on | |
| 273 | +development protocols and policies to improve the relation between industry and | |
| 274 | +government, specially regarding CD. | |
| 275 | + | |
| 276 | + | |
| 277 | +## References | |
| 278 | + | |
| 279 | +1. G. Booch, A. W. Brown, "Collaborative Development Environments, in Advances in Computers", vol. 59, 2003, pp. 1–27. | |
| 280 | +1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54. | |
| 281 | +1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", in Proceedings of the 13th International Symposium on Open Collaboration, 2017. | |
| 282 | + | |
| 283 | + | ... | ... |
No preview for this file type
63.5 KB