Improving merged version and adding team compositions

Paulo Meireles
1 parent 7027cecd
Showing 1 changed file with 76 additions and 99 deletions Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
@@ -52,25 +52,24 @@ the Ministry of Planning, Budget, and Management and two public universities:
 University of Brasília and University of São Paulo.
 During this time, the SPB portal evolved into a Collaborative Development
-Environment (CDE) [2] which brought significant benefits for the government and
-the society: The government could minimize bureaucracy and software
-development costs, by reusing the same set of applications across
-different government agencies; society could more transparently
-follow government expenses and contribute to project
-communities.
+Environment (CDE) which brought significant benefits for the government and the
+society: The government could minimize bureaucracy and software development
+costs, by reusing the same set of applications across different government
+agencies; society could more transparently follow government expenses and
+contribute to project communities.
 In this article, we discuss the use of Continuous Delivery (CD) during our
 experience as the academic partner in this project. We focus on how we managed
 to implement CD in a large institution with traditional values and how CD
 helped to build trust between the government and the university development
-team. CD enabled us to show our progress and to earn the government’s confidence
-that we could adequately fulfill their requests, becoming an essential aspect
-of our interaction with them. According to this experience, the use of CD as a
-tool to build such trust relationships is yet another of its benefits [3].
+team. CD enabled us to show our progress and to earn the government’s
+confidence that we could adequately fulfill their requests, becoming an
+essential aspect of our interaction with them. According to this experience,
+the use of CD as a tool to build such trust relationships is yet another of its
+benefits [2].
 ## Context
-<!-- Avaliar se a descrição técnica sobre SPB não deveria vim aqui -->
 SPB is a governmental program created to foster sharing and collaboration on
 Open Source Software (OSS) development for the Brazilian public administration.
 In their own projects, the Ministry managed both software requirements and server
@@ -93,21 +92,13 @@ requirements previously approved.
 mudar em todas as ocorrências de agents por staff, mas checar com o Fabio.
 -->
-In this context, we overcame three distinct challenges: (1) deconstructing
+In this context, we overcame two distinct challenges: (1) deconstructing
 the widespread belief among government staff that any project in partnership
 with a University is doomed to fail, and (2) dealing with bureaucracies
 involved in the deployment process.
 <!-- Melissa: Acho que no segundo, seria bom falar: lidar com a aritmia que as burocracias do governo causavam ao nosso processo de deploy? -->
-<!-- TODO: 
-sugestão: Tira o primeiro challenge e acrescenta um texto no final desta
-seção dizendo "no final deu tudo certo: construímos uma ferramenta modular
-(inclui o conteúdo do parágrafo sobre o primeiro challenge) com uma equipe
-heterogênea (coloca a descrição das pessoas) e chegamos a um pacote com
-7 ferramentas blah blah blah
--->
-
-Firstly, our team was not from a typical company, consisting mainly of
+Firstly, our team was not from a typical company, consisting from
 undergraduate students coordinated by two professors. Accordingly, time and
 resources allocation, accountability, and team continuity might be construed
 as "unprofessional". On the government side, the SPB portal evolution was the
@@ -146,43 +137,30 @@ team shrinked to 20 students, one designer, and two developers.
 -->
 ## Our Continuous Delivery Pipeline
-SPB portal was designed as a CDE with additional social features. Due to the
-complexity of creating such a system from scratch, we decided to build
-system-of-systems, adapting and integrating five existing OSS projects - Colab,
-Noosfero, Gitlab, Mezuro, and Mailman. All integrated system represents a total
-of more than XXX.XXX commits and X.XXX.XXX lines of code. We created a solution
-that orchestrates multiple components and allowed us to smoothly provide a
-unified interface for final users, including single sign-on and global searches [1].
-
-<!--
-Melissa: avaliar se esse detalhamento é necessário para o texto
-These software component have different levels of complexity and size. Colab
-has had X commits which represent Y lines of code, Gitlab, 64.446 commits and
-502.597 lines, Noosfero, 14.175 commits and 607.490 lines, Mezuro, 9.007 commits
-and 217.828 lines, and, finally, Mailman with 19.101 commits and 180.871 lines.
-We can thus realize, with only the sum of the integrated projects, we have a
-platform that totals more than 106.729 commits and 1.508.786 lines of code.
--->
+SPB portal was designed as a CDE with additional social features. We build
+system-of-systems, adapting and integrating five existing OSS projects: Colab
+(www.github.com/colab), Noosfero (www.noosfero.org), Gitlab (www.gitlab.com),
+Mezuro (www.mezuro.org), and Mailman (www.list.org). All integrated system
+represents a total of more than XXX.XXX commits and X.XXX.XXX lines of code.
+That solution orchestrates these multiple components and allowed us to smoothly
+provide a unified interface for final users, including single sign-on and
+global searches [1]. 
 ![The SPB Deployment Pipeline](figures/pipeline_3.png)
-Figure 1 presents our CD pipeline. It follows a typical deployment pipeline [3],
-adapted to the technical and organizational context of our project and the use
-of OSS best practices. The pipeline started when new code arrived. As the code
-went through each step, it was tested and improved until finally reaching the
-production environment. At this point, we would restart the pipeline to release
-new versions.
-
-<!---
-Comentário do Fábio: A partir daqui o texto já deveria mostrar o tamanho da plataforma e trazer dados que comprovem isso.
--->
+The SPB portal deployment follows a typical CD pipeline [3], adapted to the
+technical and organizational context of our project and the use of OSS best
+practices, as presented in Figure 1. The pipeline started when new code
+arrived. As the code went through each step, it was tested and improved until
+finally reaching the production environment. At this point, we would restart
+the pipeline to release new versions.
 ### Automated Tests
 Each integrated system had to be tested with its own test suite. This was not
 enough, however: we also had to test the platform as a whole. Given that the
-plugins also have their own test suites and these suites assume a double role as
-both plugin tests and as integration tests. These tests provided us the
+plugins also have their own test suites and these suites assume a double role
+as both plugin tests and as integration tests. These tests provided us the
 performance and security needed to guarantee the stability of components and
 the platform. If any test suite failed, by either a test error or coverage
 reduction below a certain threshold, the process stopped. Only when all tests
@@ -191,38 +169,37 @@ passed, the pipeline proceeded to the step of release preparation.
 ### Preparing a New Release
 Each software component was hosted in a separate Git repository. A new release
-of a component was tagged with a reference to a specific new feature or bug fix.
-SPB, as an integration project, had its own Git repository. An SPB portal
-release was an aggregate of releases of all of its components. When a new
-component release passed all of the SPB integration tests, we manually created a
-corresponding new tag in its repository. Therefore, a new tag on any software
-component yielded a new SPB portal release. At the end of this process, we
-started packaging.
+of a component was tagged with a reference to a specific new feature or bug
+fix.  SPB, as an integration project, had its own Git repository
+(www.softwarepublico.gov.br/gitlab/softwarepublico). A SPB portal release was
+an aggregate of releases of all of its components. When a new component release
+passed all of the SPB integration tests, we manually created a corresponding
+new tag in its repository. Therefore, a new tag on any software component
+yielded a new SPB portal release. At the end of this process, we started
+packaging.
 ### Packaging
-The platform runs on the CentOS 7 GNU/Linux distribution.  Packaging software
-for that distribution involves three steps: writing the script for the specific
-environment (RPM), building the package, and uploading it to a package
-repository.
-
-We decided to create separate packages for each software component.
-Packaging makes it easy to manage software in a given distribution,
-simplifies deployment, follows the distribution's best practices, and
-enables configuration and permission control.
+Our packaging software approach involves three steps: writing the script for
+the specific environment, building the package, and uploading it to a package
+repository. We decided to create separate packages for each system.  Packaging
+makes it easy to manage software in a given distribution, simplifies
+deployment, follows the distribution's best practices, and enables
+configuration and permission control.
-After creating a new tag for a component, the developers informed our DevOps
-[6] team and the packaging process began. A set of scripts fully automated the
-three packaging steps aforementioned. When all of them ran successfully, the new
-packages would be ready and available for our deployment scripts.
+After creating a new tag for a system, the developers informed our DevOps [4]
+team and the packaging process began. A set of scripts fully automated the
+three packaging steps aforementioned. When all of them ran successfully, the
+new packages would be ready and available for our deployment scripts.
 ### Validation Environment Deployment
 The Validation Environment (VE) is a replica of the Production Environment (PE)
-with anonymised data, and access restricted to Ministry staff and our DevOps team.
-To configure this environment, we used Chef (www.chef.io) and Chake, a serverless
-configuration tool created by our team (www.github.com/terceiro/chake). This
-maintained environment consistency, simplifying the deployment process.
+with anonymised data, and access restricted to Ministry staff and our DevOps
+team.  To configure this environment, we used Chef (www.chef.io) and Chake, a
+serverless configuration tool created by our team
+(www.github.com/terceiro/chake). This maintained environment consistency,
+simplifying the deployment process.
 The Ministry staff used the VE to validate new features and required changes.
 The VE also was used to verify the integrity of the entire portal as part of
@@ -246,7 +223,7 @@ fixes were finally available to end users.
 ## Benefits
-Research points out many CD advantages in the industry [2, 7], such as
+Research points out many CD advantages in the industry [2, 5], such as
 accelerated time to market, building the right product, productivity and
 efficiency improvements, stable releases, and better customer satisfaction.
 Working with the government, we noticed the following additional benefits.
@@ -286,21 +263,20 @@ and take ownership of the project.
 Interestingly, the CD pipeline had the same effect on the Ministry staff. They
 became more engaged in the whole process, opening and discussing issues during
-the evolution of the platform. Additionally, developers worked to improve the CD pipeline
-and speed up the process of making new features available in the production
-environment.
+the evolution of the platform. Additionally, developers worked to improve the
+CD pipeline and speed up the process of making new features available in the
+production environment.
 ### Synchronization Between Government and Development
 The CD pipeline performance depended on the synchronization between our
-development team and the Ministry staff: each party had to be prepared to
-take action as soon as the other concluded a given task.
-Initially, the Ministry staff did not contemplate this in their schedule which,
-combined with the bureaucracy in providing access to the PE
-(up to 3 days), resulted in significant delays in the validation of new features.
-This problem was softened when they realized the
-impact of these delays on the final product and decided to allocate
-revisions in their work schedule.
+development team and the Ministry staff: each party had to be prepared to take
+action as soon as the other concluded a given task.  Initially, the Ministry
+staff did not contemplate this in their schedule which, combined with the
+bureaucracy in providing access to the PE (up to 3 days), resulted in
+significant delays in the validation of new features.  This problem was
+softened when they realized the impact of these delays on the final product and
+decided to allocate revisions in their work schedule.
 <!---
 Fabio sugeriu Lessons Learned, mas vamos mostrar exemplos da revista para ele olhar.
@@ -313,10 +289,16 @@ deployment process and kept the project alive. We now look at the lessons learne
 ### Build CD From Scratch
-Taking on the responsibility for implementing CD impacted the whole team.
-Most of our team members did not have CD know-how and we had few
-working hours available to build the pipeline. The construction
-and maintenance of the CD process were made possible by the key decisions to:
+We started with a team composed of 24 undergraduate students, one designer, and
+two senior developers. In 2015, our team grew to 36 students, two designers,
+eight senior developers. For 2016, due to budget constraints, our team shrinked
+to 20 students, one designer, and two developers; however, in the last 3
+months, only 6 students closed the project.
+
+Taking on the responsibility for implementing CD impacted the whole team.  Most
+of our team members did not have CD know-how and we had few working hours
+available to build the pipeline. The construction and maintenance of the CD
+process were made possible by the key decisions to:
 <!---
 pensar em generalizar/filosofar
@@ -370,20 +352,15 @@ work and deployment process. In the end, the Ministry staff realized
 that it would be beneficial for the project if they granted us access to the
 infrastructure, both VE and PE.
-<!---
-Paulo: Acho que precisamos de algo ligado ao Ha-Ha-moment para fechar o texto aqui; ou fechar falando que tudo foi feito de forma aberta e colaborativa com as comunidades dos projetos envolvidos, estão todos os fontes disponíveis em https://softwarepublico.gov.br/gitlab/softwarepublico/
-
-Melissa: Acho que não temos perna, mas podemos usar o gráfico como ah-ha 
--->
-
 ## References
 1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", Proceedings of the 13th International Symposium on Open Collaboration. ACM, Article 16, 2017, 10 pages.
-1. G. Booch and A. Brown, A. W. "Collaborative Development Environments", Advances in Computers, vol. 59, 2003, pp. 1-27.
 1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", IEEE Software, vol. 32, no. 2, 2015, pp. 50-54.
-1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, Article 18, 2015, 41 pages.
 1. J. Humble and D. Farley, "Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation", Addison-Wesley Professional, 2010.
 1. J. Davis and K. Daniels, "Effective Devops: Building a Culture of Collaboration, Affinity, and Tooling at Scale", O'Reilly Media, Inc., 2016.
 1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck and M. Stumm, "Continuous Deployment at Facebook and OANDA", 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), Austin, TX, 2016, pp. 21-30.
-
+<!---
+1. G. Booch and A. Brown, A. W. "Collaborative Development Environments", Advances in Computers, vol. 59, 2003, pp. 1-27.
+1. C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, "Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions", ACM Comput. Surv. 48, 2, Article 18, 2015, 41 pages.
+-->