[i3eSW] Re-organizing the references and the all source-text

Paulo Meireles
1 parent 139ada8a
Showing 1 changed file with 204 additions and 49 deletions Show diff stats
ieeeSW/releaseEng3/IEEE_ThemeIssue_ReleaseEng_CD.md
@@ -2,25 +2,100 @@
  
 ## Abstract
  
-Many software development teams think of the operational aspects of Continuous Delivery (CD) and the competitive benefits that come along with it. For us, it was much more: it was a survival technique. This article presents the experience applying CD in a Brazilian Government project for the development of a Collaborative Development Environment, sharing its unconventional challenges and our strategies used to overcome them. This report from the trenches of the Brazilian Federal Government can help practitioners to understand how important the CD adoption is to their projects.
+Many software development teams think of the operational aspects of Continuous
+Delivery (CD) and the competitive benefits that come along with it. For us, it
+was much more: it was a survival technique. This article presents the
+experience applying CD in a Brazilian Government project for the development of
+a Collaborative Development Environment (CDE), sharing its unconventional
+challenges and our strategies used to overcome them. This report from the
+trenches of the Brazilian Federal Government can help practitioners to
+understand how important the CD adoption is to their projects.
  
 ## Introduction and Context
  
-We have worked on a Brazilian government three-year-long project to evolve an existing platform that had technical issues and a lack of political support. The evolution project started in a presidential election year and everyone involved were under Presidential re-election campaign pressure to show results. Even with the re-election of the Brazilian President in 2014, leadership in government agencies suddenly changed, what was worsened by corruption scandals revealed by the Brazilian Federal police, in particular the Car-Wash investigation (FOOTNOTE). That reflected on the project’s requirements: each new leader wanted to fulfill their political agenda. In this scenario, delivery delays could have sunk the project into oblivion.
-
-From 2014 to 2016, our team developed the new platform for the Brazilian Public Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br) funded by a grant of 2,619,965.00 BRL (about 1.000,000.00 USD in January 2014) from the Federal Government. The SPB Portal has evolved to a Collaborative Development Environment (CDE)[1] and this evolution has brought important benefits not just to the Brazilian government , but also to society as a whole. For the government, the bureaucracy on using the same software across government agencies, duplicate works and costs all are reduced. The society gains a transparency and collaboration mechanism, since anyone can check the government expenses on software and contribute to the software communities. To achieve these goals, we have chosen to integrate more than free software tools (such as Gitlab (www.gitlab.com), Mailman (www.gnu.org/software/mailman), Noosfero (www.noosfero.org), and Colab (www.github.com/colab)) rather than write everything from scratch.
-
-During the entire SPB Portal evolution project, we had to handle three distinct issues, usual in a software engineering scenario: reaching the goals which have guided the platform development, managing the diversity of team project members, and communicating effectively with clients (in this particular case, government agents). Managing the interaction of these elements was not easy and the unstable Brazilian political scenario only made things worse.
-
-To reaching the SPB project goals, we had to overcome strong political bias tied with complicated technical issues and relatively low budget. Because it is open to the public, the government representatives have seen the platform as a opportunity of marketing and have often ignored the technical advice in favor of political decisions. Furthermore, integrating a number of distinct systems to work seamlessly was not an easy job. We had to learn how each system worked and come up with ideas of how to integrate them as fast as possible, with a team of mostly inexperienced developers.
-
-We also had to manage the diversity of the SPB team members. This team was composed of approximately 50 undergraduate students (not all simultaneously) together with professors, masters students and professional designers as well as senior developers from the Brazilian free software community. Undergraduate students have received a fellowship and, for most of them, this R&D project was their first professional experience. Seniors developers and masters students had two important contributions to the project: transfer knowledge to undergraduate students and address hard tasks. Finally, professors were responsible for interacting with the Brazilian government and control the political pressures applied to the project.
-
-Our third point to be handled was the communication with the two independent groups of government representatives: requirements analysts and deployment technicians. Requirements analysts were the real representatives of the Brazilian government in the project. They usually tested new features, provided feedback, and reported for directors. The deployment technicians only had the access to the host machines wherein SPB platform was running. They were, theoretically, responsible for deploying the project. However, the new SPB Portal was composited from more than ten different  software projects, generating a complex deploy process working on seven servers.
-
-Therefore, we realised that we needed to take control over the deployment process. We would use CD as a mean to keep the government satisfied and provide quick response times to their requests which were, most of the time, influenced by the uncertainties of project continuity. We believed that would keep the project alive even in an politically unstable and complex technical scenario. For that, during months we have worked to automate all the deploy process. For instance, we have created a tool called Shak (Self Hosting Applications Kit) that offers an application installation with only instruction. It aims to make the deploy process easier for users without technical knowledge, ensuring privacy and security for their Internet applications. Thereby, we created a specific team dedicated to the deployment process to mature our CD pipeline that would give us confidence to meet the government’s requirements faster and faster.
-
-We describe our CD pipeline and how it speeded up our software delivery time. The CD made us adaptable for all requested changes and helped us to mitigate those aforementioned political challenges besides the technical issues. Among CD’s known benefits[2], we also explain those that were the most important in the project scenario for us: improving customer satisfaction and making reliable releases. Both kept the project alive for more two years during the worst political crisis after the re-democratization in Brazil.
+We have worked on a Brazilian government three-year-long project to evolve an
+existing platform that had technical issues and a lack of political support.
+The evolution project started in a presidential election year and everyone
+involved were under Presidential re-election campaign pressure to show results.
+Even with the re-election of the Brazilian President in 2014, leadership in
+government agencies suddenly changed, what was worsened by corruption scandals
+revealed by the Brazilian Federal police, in particular the "Car-Wash"
+investigation. That reflected on the project’s requirements: each new leader
+wanted to fulfill their political agenda. In this scenario, delivery delays
+could have sunk the project into oblivion.
+
+From 2014 to 2016, our team developed the new platform for the Brazilian Public
+Software (SPB, Portuguese acronym) Portal (www.softwarepublico.gov.br) funded
+by a grant of 2,619,965.00 BRL (about 1.000,000.00 USD in January 2014) from
+the Federal Government. The SPB Portal has evolved to a CDE [1] and this
+evolution has brought important benefits not just to the Brazilian government ,
+but also to society as a whole. For the government, the bureaucracy on using
+the same software across government agencies, duplicate works and costs all are
+reduced. The society gains a transparency and collaboration mechanism, since
+anyone can check the government expenses on software and contribute to the
+software communities. To achieve these goals, we have chosen to integrate more
+than free software tools (such as Gitlab (www.gitlab.com), Mailman
+(www.gnu.org/software/mailman), Noosfero (www.noosfero.org), and Colab
+(www.github.com/colab)) rather than write everything from scratch.
+
+During the entire SPB Portal evolution project, we had to handle three distinct
+issues, usual in a software engineering scenario: reaching the goals which have
+guided the platform development, managing the diversity of team project
+members, and communicating effectively with clients (in this particular case,
+government agents). Managing the interaction of these elements was not easy and
+the unstable Brazilian political scenario only made things worse.
+
+To reaching the SPB project goals, we had to overcome strong political bias
+tied with complicated technical issues and relatively low budget. Because it is
+open to the public, the government representatives have seen the platform as a
+opportunity of marketing and have often ignored the technical advice in favor
+of political decisions. Furthermore, integrating a number of distinct systems
+to work seamlessly was not an easy job. We had to learn how each system worked
+and come up with ideas of how to integrate them as fast as possible, with a
+team of mostly inexperienced developers.
+
+We also had to manage the diversity of the SPB team members. This team was
+composed of approximately 50 undergraduate students (not all simultaneously)
+together with professors, masters students and professional designers as well
+as senior developers from the Brazilian free software community. Undergraduate
+students have received a fellowship and, for most of them, this R&D project was
+their first professional experience. Seniors developers and masters students
+had two important contributions to the project: transfer knowledge to
+undergraduate students and address hard tasks. Finally, professors were
+responsible for interacting with the Brazilian government and control the
+political pressures applied to the project.
+
+Our third point to be handled was the communication with the two independent
+groups of government representatives: requirements analysts and deployment
+technicians. Requirements analysts were the real representatives of the
+Brazilian government in the project. They usually tested new features, provided
+feedback, and reported for directors. The deployment technicians only had the
+access to the host machines wherein SPB platform was running. They were,
+theoretically, responsible for deploying the project. However, the new SPB
+Portal was composited from more than ten different  software projects,
+generating a complex deploy process working on seven servers.
+
+Therefore, we realised that we needed to take control over the deployment
+process. We would use CD as a mean to keep the government satisfied and provide
+quick response times to their requests which were, most of the time, influenced
+by the uncertainties of project continuity. We believed that would keep the
+project alive even in an politically unstable and complex technical scenario.
+For that, during months we have worked to automate all the deploy process. For
+instance, we have created a tool called Shak (Self Hosting Applications Kit)
+that offers an application installation with only instruction. It aims to make
+the deploy process easier for users without technical knowledge, ensuring
+privacy and security for their Internet applications. Thereby, we created a
+specific team dedicated to the deployment process to mature our CD pipeline
+that would give us confidence to meet the government’s requirements faster and
+faster.
+
+We describe our CD pipeline and how it speeded up our software delivery time.
+The CD made us adaptable for all requested changes and helped us to mitigate
+those aforementioned political challenges besides the technical issues. Among
+CD’s known benefits[2], we also explain those that were the most important in
+the project scenario for us: improving customer satisfaction and making
+reliable releases. Both kept the project alive for more two years during the
+worst political crisis after the re-democratization in Brazil.
  
 ## Our Continuous Delivery Pipeline
  
@@ -30,7 +105,9 @@ We describe our CD pipeline and how it speeded up our software delivery time. Th
  
 [//]: # (TODO - Rever essa frase - é uma legenda da imagem?)
  
-Figure X represents our CD pipeline. After each release, we have defined a set of features we would develop to comply with government’s requirements and then we have followed this pipeline.
+Figure X represents our CD pipeline. After each release, we have defined a set
+of features we would develop to comply with government’s requirements and then
+we have followed this pipeline.
  
 ### Automated tests
  
@@ -38,17 +115,34 @@ Figure X represents our CD pipeline. After each release, we have defined a set o
 [//]: # (TODO - Adicionar que as mudanças no Colab quebravam todos os plugins.)
 [//]: # (https://www.linux.com/blog/learn/chapter/dev-ops/2017/7/devops-fundamentals-part-3-continuous-delivery-and-deployment)
  
-The SPB portal is composed of a set of FOSS systems. All of them have their own automated test suite. So, we encouraged the teams to keep unit tests coverage high for each system. We implemented the systems integration using a plugin architecture and wrote integration tests to check if they were working properly. The plugin architecture made the PSPB’s components decoupled. That allowed us to be confident that changes on one system would not affect others. This characteristic allowed the test execution of only the component with changes instead of the entire system.
-
-Our CD’s first step was to execute each system’s automated test suite. If any error was found, the pipeline stopped and the developers were notified. Only after all tests passed we move to preparing the release.
-Preparing a new release
-
-Our release system was divided into two perspectives: the application and the PSPB. The application tag refers to the specific feature or bug fix and is a monotonically increasing. A new tag on any system yielded a new PSPB tag.
-
-When all tests passed for a given component, we manually created a new application tag for it. As a consequence, that automatically created a new tag for the PSPB. Notice that we have forks of the original softwares and, as consequence, we had different tag values.
-Packaging
-
-The PSPB platform is running under the CentOS GNU/Linux distribution. Basically, packaging a software for that distribution has three steps: (1) write the script for the specific environment (RPM), (2) build the package, and (3) upload it to a package repository. We chose to package our components for several reasons: 
+The SPB portal is composed of a set of FOSS systems. All of them have their own
+automated test suite. So, we encouraged the teams to keep unit tests coverage
+high for each system. We implemented the systems integration using a plugin
+architecture and wrote integration tests to check if they were working
+properly. The plugin architecture made the PSPB’s components decoupled. That
+allowed us to be confident that changes on one system would not affect others.
+This characteristic allowed the test execution of only the component with
+changes instead of the entire system.
+
+Our CD’s first step was to execute each system’s automated test suite. If any
+error was found, the pipeline stopped and the developers were notified. Only
+after all tests passed we move to preparing the release.  Preparing a new
+release
+
+Our release system was divided into two perspectives: the application and the
+PSPB. The application tag refers to the specific feature or bug fix and is a
+monotonically increasing. A new tag on any system yielded a new PSPB tag.
+
+When all tests passed for a given component, we manually created a new
+application tag for it. As a consequence, that automatically created a new tag
+for the PSPB. Notice that we have forks of the original softwares and, as
+consequence, we had different tag values.  Packaging
+
+The PSPB platform is running under the CentOS GNU/Linux distribution.
+Basically, packaging a software for that distribution has three steps: (1)
+write the script for the specific environment (RPM), (2) build the package, and
+(3) upload it to a package repository. We chose to package our components for
+several reasons: 
 * Not all software was packaged by the community;
 * And those that existed were outdated;
 * Packaging makes it easy to manage the software on a given distribution;
@@ -56,28 +150,49 @@ The PSPB platform is running under the CentOS GNU/Linux distribution. Basically,
 * Packaging follows the distribution’s best practices and,
 * Allows configurations and permissions control.
  
-After creating a new tag for one component, the DevOps team was notified and packaging process began. In the normal case, the three packaging steps aforementioned are fully automated by a set of scripts. However, if the team reports to DevOps any eventual dependency change, the first step has to be manual. For instance, one system could start requiring another system to be initialized and that made necessary for someone to manually update the package script.
+After creating a new tag for one component, the DevOps team was notified and
+packaging process began. In the normal case, the three packaging steps
+aforementioned are fully automated by a set of scripts. However, if the team
+reports to DevOps any eventual dependency change, the first step has to be
+manual. For instance, one system could start requiring another system to be
+initialized and that made necessary for someone to manually update the package
+script.
  
-After all these scripts have run successfully, the new packages would be ready to use by our subsequent deployment scripts.
+After all these scripts have run successfully, the new packages would be ready
+to use by our subsequent deployment scripts.
  
 ### Validation Environment
  
 [//]: # (TODO - Mencionar que a ferramenta era baseada em Chef - Dá um peso importante.)
  
-The Validation Environment (VE) is a replica of the Production Environment (PE), with two exceptions: only the government officers and us had access to it and all the data is anonymised. To configure the environment, we use a configuration management tool. That maintained environment consistency which makes the deployment process simpler. Additionally, the packages we built on the last step were readily available to use by the management tool.
+The Validation Environment (VE) is a replica of the Production Environment
+(PE), with two exceptions: only the government officers and us had access to it
+and all the data is anonymised. To configure the environment, we use a
+configuration management tool. That maintained environment consistency which
+makes the deployment process simpler. Additionally, the packages we built on
+the last step were readily available to use by the management tool.
  
-The VE was used by the government agents to validate new features and required changes. Also, the VE was useful to verify the integrity of the entire portal as part of the next step in the pipeline.
-Acceptance Tests
+The VE was used by the government agents to validate new features and required
+changes. Also, the VE was useful to verify the integrity of the entire portal
+as part of the next step in the pipeline.  Acceptance Tests
  
-After we completely deploy a new PSPB version in the VE, the government agents are responsible for checking features and/or bug fixes required by them. If the technicians identify a problem, they notify the developers, the problems are fixed and the pipeline restarts from scratch. If everything is validated, we move to production deployment.
-Production Deployment
+After we completely deploy a new PSPB version in the VE, the government agents
+are responsible for checking features and/or bug fixes required by them. If the
+technicians identify a problem, they notify the developers, the problems are
+fixed and the pipeline restarts from scratch. If everything is validated, we
+move to production deployment.  Production Deployment
  
-After the government authorizes the VE, we can finally begin the production deployment. We use the same configuration management tool with the same scripts. After the deploy is completed, both VE and PE are on the same state. The new features and bug fixes are finally available to end users.
+After the government authorizes the VE, we can finally begin the production
+deployment. We use the same configuration management tool with the same
+scripts. After the deploy is completed, both VE and PE are on the same state.
+The new features and bug fixes are finally available to end users.
  
  
 ## Benefits
  
-We had to handle many tensions between development and political issues. Our CD pipeline gave us strong mechanisms to tackle most of the problems. As a result we came with some benefits from our decision to adopt CD. 
+We had to handle many tensions between development and political issues. Our CD
+pipeline gave us strong mechanisms to tackle most of the problems. As a result
+we came with some benefits from our decision to adopt CD. 
  
 [//]: # (TODO - Melhorar título - Ideias: Response to mistrust)
  
@@ -85,27 +200,58 @@ We had to handle many tensions between development and political issues. Our CD 
  
 [//]: # (TODO - Decisão do secretario acima do diretor)
  
-The direct benefit from the CD pipeline was the fast response to the changes required by the government. That was vital for the project’s renewal over the years. We could manage the tension between the government and the development team better. Every meeting with the government leader was delicate and resulted on many new requirements, most of them motivated by political needs. For example, once it was demanded a completely layout change because one director suddenly decided to make a marketing campaign about the portal. They would use undelivered requirements as a means to suggest the project’s cancellation. We believed that if we took too long to attend their demands, the project would end. CD helped us to move fast on deploying to production, even of smaller parts of the requirements. That way, we always had something to show on the meetings, reducing their eagerness to end the project. For our team, it made the developers more confident the project would last a little longer and they would not go looking for another jobs.
+The direct benefit from the CD pipeline was the fast response to the changes
+required by the government. That was vital for the project’s renewal over the
+years. We could manage the tension between the government and the development
+team better. Every meeting with the government leader was delicate and resulted
+on many new requirements, most of them motivated by political needs. For
+example, once it was demanded a completely layout change because one director
+suddenly decided to make a marketing campaign about the portal. They would use
+undelivered requirements as a means to suggest the project’s cancellation. We
+believed that if we took too long to attend their demands, the project would
+end. CD helped us to move fast on deploying to production, even of smaller
+parts of the requirements. That way, we always had something to show on the
+meetings, reducing their eagerness to end the project. For our team, it made
+the developers more confident the project would last a little longer and they
+would not go looking for another jobs.
  
 ### Build client’s trust
  
-After we established the CD, the government agents started to be more confident in our work. First, because they noticed that each new deploy made by us in the VE was stable and reliable. Second, they could see new features fast since we constantly updated the VE based on their feedback. This made our relation strong and in moments that needed quick action they would rather give us access to production.
+After we established the CD, the government agents started to be more confident
+in our work. First, because they noticed that each new deploy made by us in the
+VE was stable and reliable. Second, they could see new features fast since we
+constantly updated the VE based on their feedback. This made our relation
+strong and in moments that needed quick action they would rather give us access
+to production.
  
  
 ## Challenges
  
-We successfully built a functional CD pipeline. In the end, we took over the deployment process from the government. That allowed us to survive into an unstable political scenario. However, we recognized that many challenges still need to be addressed by the industry and academia together.
-Build CD from scratch
+We successfully built a functional CD pipeline. In the end, we took over the
+deployment process from the government. That allowed us to survive into an
+unstable political scenario. However, we recognized that many challenges still
+need to be addressed by the industry and academia together.  Build CD from
+scratch
  
-Taking on CD responsibilities had a significant impact on the team. We did not have the know-how and had little time to come up with a working pipeline. To make things worse, we were not aware of how companies normally organized their teams to make CD feasible.
+Taking on CD responsibilities had a significant impact on the team. We did not
+have the know-how and had little time to come up with a working pipeline. To
+make things worse, we were not aware of how companies normally organized their
+teams to make CD feasible.
  
 [//]: # (TODO - Deixar uma abertura para uma eventual pesquisa)
  
-The seniors were crucial at this point. They came up with an initial solution to get us started. That already enabled us to automatize the deploy, even though the process was still rudimentary. We had to evolve our solution on-the-fly. We dedicated a few developers to this task.
+The seniors were crucial at this point. They came up with an initial solution
+to get us started. That already enabled us to automatize the deploy, even
+though the process was still rudimentary. We had to evolve our solution
+on-the-fly. We dedicated a few developers to this task.
  
 ### Handling inexperienced teams
  
-After the developers learned how CD worked, it was difficult to pass the knowledge along to other teammates. We tried to mitigate this by encouraging a member's migration to the DevOps team. Further research on how to effectively spread knowledge across inexperienced developers in a scenario with a high turnover are needed.
+After the developers learned how CD worked, it was difficult to pass the
+knowledge along to other teammates. We tried to mitigate this by encouraging a
+member's migration to the DevOps team. Further research on how to effectively
+spread knowledge across inexperienced developers in a scenario with a high
+turnover are needed.
  
 [//]: # (TODO - Refazer título abaixo - ideias: overcoming the challenges / overcoming the mistrust)
  
@@ -113,14 +259,23 @@ After the developers learned how CD worked, it was difficult to pass the knowled
  
 [//]: # (TODO - Deixar claro que government = os técnicos , análista ou agentes do governo, conforme for padronizado no texto)
  
-In the project’s first half we struggled with deploy related problems in the government structure. We were in a paradoxical situation. The government demanded speedy deliveries but would not give access to their production infrastructure. As an example, only in a very specific situation the government allowed us to access the PE. After some interactions with the government we convinced them to create the VE as an isolated replica of the PE in their own infrastructure. The government agents then realized that it could be good for the project if they granted us access to part of the structure since we could deliver new features to them faster. We believe it is required more research on development protocols and policies to improve the relation between industry and government, specially regarding CD.
+In the project’s first half we struggled with deploy related problems in the
+government structure. We were in a paradoxical situation. The government
+demanded speedy deliveries but would not give access to their production
+infrastructure. As an example, only in a very specific situation the government
+allowed us to access the PE. After some interactions with the government we
+convinced them to create the VE as an isolated replica of the PE in their own
+infrastructure. The government agents then realized that it could be good for
+the project if they granted us access to part of the structure since we could
+deliver new features to them faster. We believe it is required more research on
+development protocols and policies to improve the relation between industry and
+government, specially regarding CD.
  
  
 ## References
  
-1. Collaborative Development Environment (TO-DO REF)
-1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too," in IEEE Software, vol. 32, no. 2, pp. 50-54, Mar.-Apr. 2015.
-1. "Brazilian Public Software Portal: an integrated platform for collaborative development", in OpenSym ’17, August 23–25, 2017, Galway, Ireland.
-
+1. G. Booch, A. W. Brown, "Collaborative Development Environments, in Advances in Computers", vol. 59, 2003, pp. 1–27.
+1. L. Chen, "Continuous Delivery: Huge Benefits, but Challenges Too", in IEEE Software, vol. 32, no. 2, 2015, pp. 50-54.
+1. P. Meirelles, M. Wen, A. Terceiro, R. Siqueira, L. Kanashiro, and H. Neri, "Brazilian Public Software Portal: an integrated platform for collaborative development", in Proceedings of the 13th International Symposium on Open Collaboration, 2017.