Paulo Meireles · Paulo Meireles · Melissa Wen · Paulo Meireles · Paulo Meireles · Leonardo Leite
Showing 4 changed files Show diff stats
opensym2017/content/04-researchdesign.tex
opensym2017/content/05-requirements.tex
opensym2017/content/06-architecture.tex
opensym2017/content/08-process.tex
@@ -27,7 +27,7 @@ managers from the Brazilian Government.
 For the majority of the students, this was a first professional experience. 
 Even though, our development process defined a central role on students participation.
-textbf{Q3:} \textit{How to introduce collaborative and agile
+\textbf{Q3:} \textit{How to introduce collaborative and agile
 practices typical in FLOSS environments in the governmental development process?}
 %
 The software development on Brazilian government is based on a very traditional way,
 \section{Requirements}
 \label{sec:requirements}
+\begin{comment}
 In 2013, the SPB Portal had more than 600 thousand unique visitors, generating
 more than 16 million page views with about 50 million hits. By evaluating only
 the main projects, there were more than 15 thousand downloads and 4 thousand
 messages exchanged in their forums. These data illustrates the potential of the
 SPB Portal, even with several limitations in the past.
+\end{comment}
 By preparing the evolution project described in this paper, the Brazilian
 government promoted 3 events to collect the requirements, in particular from
@@ -15,32 +17,24 @@ SPB concepts and requirements with IT stakeholders from the Brazilian
 government and public organizations.
 After these 3 rounds discussing the new SPB platform, the Brazilian government
-listed about 145 requirements and developed a ``mind
-map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}}
-to guide the SPB portal evolution. In this scenario, the 10 most voted
-requirements were:
+listed about 145 requirements. The 10 most voted requirements were:
+(i) Source code repository with public access;
+(ii) Visit community pages without login;
+(iii) Distributed version control system;
+(iv) Scores of users and developers collaboration;
+(v) Search software by features;
+(vi) Integration with social networks;
+(vii) Repository for future ideas and requirements;
+(viii) Friendly URL to access a public software community page;
+(ix) User feedback about a public software;
+(x) Report of the experience about the use of a public software.
-\begin{enumerate}
-
-\item Source code repository with public access;
-\item Visit community pages without login;
-\item Distributed version control system;
-\item Scores of users and developers collaboration;
-\item Search software by features;
-\item Integration with social networks;
-\item Repository for future ideas and requirements;
-\item Friendly URL to access a public software community page;
-\item User feedback about a public software;
-\item Report of the experience about the use of a public software;
-
-\end{enumerate}
-
-\begin{figure}[hbt]
-  \centering
-    \includegraphics[width=\linewidth]{figures/technological-requirements.png}
-  \caption{Technological requirements.}
-  \label{fig:requirements}
-\end{figure}
+%\begin{figure}[hbt]
+%  \centering
+%    \includegraphics[width=\linewidth]{figures/technological-requirements.png}
+%  \caption{Technological requirements.}
+%  \label{fig:requirements}
+%\end{figure}
 There were other requirements based on the experience of the IT
@@ -54,23 +48,16 @@ user experience in the new platform.
 At the first moment, we desired to release an initial version that could
 replace the old SPB portal. For that, the first version should have
 features such as:
-
-\begin{enumerate}
-
-\item An organized public software catalog;
-\item Social network environment (profiles for users, software pages, and community pages);
-\item Content Management Systems (CMS) features;
-\item Web-based Git repository manager with wiki and issue tracking features;
-\item Mailing lists and discussion forums;
-
-\end{enumerate}
+(i) An organized public software catalog;
+(ii) Social network environment (profiles for users, software pages, and community pages);
+(iii) Content Management Systems (CMS) features;
+(iv) Web-based Git repository manager with wiki and issue tracking features;
+(v) Mailing lists and discussion forums;
 Other requirements were also planned during the conception phase of the
 SPB evolution project, such as an integrated search engine and a
 web-based source code static analysis monitor. By analyzing all of these
-requirements, we proposed the technological requirements
-illustrated in Figure \ref{fig:requirements} to guide the development of
-the new SPB platform. In other words, we have designed the SPB evolution
+requirements, we have designed the SPB evolution
 project based on existing FLOSS tools. However, the integration of
 several existing systems that were already implemented in different
 programming languages and frameworks, adding features such as a
@@ -25,46 +25,25 @@ specialized teams for work in the integration process. The teams learned
 how to develop for their assigned systems and contributed to the
 original communities, so that the version we used were not significantly
 different from the original.
-
+%
 At the end of the project, the SPB portal was composed of more than ten
-systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and
-Munin. The remainder of this section describes the most relevant of
-them, as well as how they were integrated into the platform.
-
-\subsection{Colab}
+systems, such as Colab, Noosfero, Gitlab, and Mezuro.
 Colab\footnote{\url{https://github.com/colab}} is a systems integration
 platform for web applications. One of its goals is allowing different
 applications to be combined in such a way that an user does not notice
 the change between applications. For that, Colab provides facilities
-for:
-
-\begin{itemize}
-  \item \textit{Centralized authentication}: Users are automatically 
-identified for each application without having to enter login
-credentials on each of them;
-  \item \textit{Visual consistency}: Colab is able to change the HTML
-code produced by the integrated applications, such as defining default
-template or reusing common HTML pages, in order to provide a unified
-interface;
-  \item \textit{Relaying of events between applications}: The
-integrated applications or Colab itself are able to trigger an action
-on another application;
-  \item \textit{Integrated search engine}: It is possible to set up
-exactly which data needs to be indexed from each application and how
-users are directed to this information on correspondent application;
-\end{itemize}
-
+for: (i) Centralized authentication, (ii) Visual consistency, (iii) Relaying of events between applications, and (iv) Integrated search engine.
+%
 Colab implements this integration by working as a reverse proxy for the
 integration applications, that is, all external requests pass through
 Colab before reaching them. 
-
+%
 Initially, Colab had support for a small set of applications (Trac, GNU
 Mailman, and Apache Lucene) and support for them was hard-coded. Our
 team evolved Colab so that it can now receive plugins that add support
 new applications without the need of changes to the Colab core.
-\subsection{Noosfero}
 Noosfero\footnote{\url{http://noosfero.org}} is a software for building
 social and collaboration networks. Besides the classical social
@@ -73,31 +52,29 @@ and a general-purpose CMS (Content Management System). Most of the user
 interactions with SPB is through Noosfero: user registration, project
 home pages and documentation, and contact forms.
-\subsection{Gitlab}
-GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository
-manager with wiki pages and issue tracking features. Gitlab is a FLOSS
+Gitlab\footnote{\url{http://gitlab.com}} is a web-based Git repository
+manager with wiki pages and issue tracking features. It is a FLOSS
 platform and focuses on delivering a holistic solution that will see
 developers from idea to production seamlessly and on a single platform.
-
+%
 Gitlab has several unique features, such as built-in continuous
 integration and continuous deployment, flexible permissions, tracking of
 Work-in-Progress work, moving issues between projects, group-level
 milestones, creating new branches from issues, issues board, and time
 tracking.
-\subsection{Mezuro}
 Mezuro\footnote{\url{http://mezuro.org/}} is a platform to
 collect source code metrics to monitor the internal quality of software written
 in C, C++, Java, Python, Ruby, and PHP.
-
+%
 In general, source code metrics tools also do not present a friendly way
 to interpret its results and, even more, do not follow a standardization
 between them. Mezuro collects and presents these results to the end
 user, specially, by analyzing source code metric history during its life
 cycle.
-
+%
 The Mezuro platform provides a single interface grouping available
 tools, allows selection and composition of metrics in a flexible manner,
 stores the metrics evolution history, presents results in a friendly
@@ -106,13 +83,6 @@ accordingly to their own context.
 \subsection{System unification and User eXperience evolution}
-\begin{figure}[hbt]
-  \centering
-    \includegraphics[width=\linewidth]{figures/arch.png}
-  \caption{SPB architecture overview.}
-  \label{fig:architecture}
-\end{figure}
-
 The conceptual architecture of the platform is presented in Figure
 \ref{fig:architecture}. Colab initially handles all user interaction,
 directing requests to one of the integrated applications. It
@@ -123,6 +93,13 @@ functionality of each application, a search o	n the SPB portal might
 return content from any of the applications, be it web pages, mailing
 list posts, or source code.
+\begin{figure}[hbt]
+  \centering
+    \includegraphics[width=.8\linewidth]{figures/arch.png}
+  \caption{SPB architecture overview.}
+  \label{fig:architecture}
+\end{figure}
+
 The integration of collaborative environments goes beyond functional aspects.
 Offering the population an unified experience across these environments has
 been the key to encourage the use of the platform as it reduces the perception
@@ -148,8 +125,6 @@ interface had to adapt for each individual scenario. In particular
 Noosfero did not yet have a responsive design; we engaged in its
 development and contributed towards that goal.
-\begin{comment}
-%FALTA DE ESPAÇO
 \subsection{Deploy}
 The SPB platform was deployed in 7 virtual machines with different functions,
@@ -157,7 +132,7 @@ as we can see in Figure \ref{fig:architecture2}.
 \begin{figure*}[hbt]
   \centering
-    \includegraphics[width=\linewidth]{figures/arch3.png}
+    \includegraphics[width=.8\linewidth]{figures/arch3.png}
   \caption{Instanciation view of the SPB architecture.}
   \label{fig:architecture2}
 \end{figure*}
@@ -187,4 +162,4 @@ static analysis tools on source code stored in repository and provide this data
 to Prezento. A social network and CMS (Content Manager System) is provided by
 Noosfero in \textit{social}, and the databases of all tools with a cache
 service are in \textit{database}.
-\end{comment}
+
@@ -2,113 +2,112 @@
 \label{sec:process}
 The SPB team was composed of a variety of professionals with different levels
-and skills, where most of them were undergraduate students with major in
-software engineering (from 4th semester or upper).  Since the students could
-not dedicate many hours per week to the project, they always had the
+and skills, where most of them were undergraduate students of
+software engineering. Since students could
+not dedicate many hours per week to the project, they had the
 flexibility to negotiate their work schedule during the semester in
-order to not harm their classes and coursework. Their daily work routine
+order to not harm their classes and coursework. Their work routine
 in the project included programming and DevOps tasks.
-The development of SPB project required a vast experience and background that
-usually undergraduate students do not have yet. For this reason, a few senior
-developers have joined to the project to help with the more difficult issues
+The project required a vast experience and background that
+usually undergraduate students do not have. For this reason, a few senior
+developers have joined the project to help with the more difficult issues
 and to transfer knowledge to the students. Their main task was to provide
-solutions for complex problems, in other words, they worked as developers. As
-these professionals are very skillful and the project could not fund a full
-time work for them, some of them worked partially on the project. In addition,
-they lived in a different states spread around Brazil which led much of the
+solutions for complex problems, working as developers. As
+these professionals are very skillful and the project could not fund full-time
+work for all of them, some of them worked partially on the project. In addition,
+they lived in different Brazilian states, which led much of the
 communication to be online.
 In short, our work process was based on open and collaborative software
-development practices. The development process was defined based on the
-adaptation of different agile and FLOSS communities practices, highlighting the
+development practices. The development process was based on the
+adaptation of different agile and FLOSS communities practices, with a
 high degree of automation resulting from DevOps practices. Thus, the work
 process was executed in a cadenced and continuous way.
 Finally, the last group of actors of this project was composed of employees
-formally working for the Brazilian government, in the Ministry of Planning,
+of the Brazilian Ministry of Planning,
 Development, and Management (MP is the Brazilian acronym). All the project
-decisions, validations, and scope definitions were made by them. In this way we
-developed software product incrementally, with releases aligned to business
-strategic objectives. As one can see, the project had many different
-sort of stakeholders that had to be organized and synchronized.
+decisions, validations, and scope definitions were made by them. In this way, we
+incrementally developed a software product with releases aligned to 
+strategic business objectives. As one can see, the project had a wide range
+of different stakeholders that had to be organized and synchronized.
 \subsection{Team organization}
-Approximately 70\% of the development teams were composed of software
+Approximately 70\% of the development team was composed of software
 engineering undergraduate students from UnB and they worked physically
 in the same laboratory. Each student had their own schedule based on
-their classes, what complicates the implementation of pair programming.
-The senior developers tried to synchronize their schedule with the
-students on their sub-team. To  cope with this environment, we had a few
-basic rules which guided the project organization:
+her classes, what complicates the implementation of pair programming.
+The senior developers tried to synchronize their schedule with 
+students schedules. To  cope with this scenario, we had a few
+basic rules guiding the project organization:
 \begin{enumerate}
   \item Classes have high priority for undergraduate students;
-  \item Work in pairs whenever possible (locally or remotely);
-  \item There must be one morning or afternoon per week when
-    \emph{everyone} should be together physically in the laboratory
-    (except of course the remote team members);
-  \item Every 3 to 4 months, the senior developers would fly in and work
+  \item Pairing whenever possible (locally or remotely);
+  \item We had one morning or afternoon per week when
+    \emph{everyone}, but the remote members, 
+    should be together physically in the laboratory;
+  \item Every 3 to 4 months the senior developers would travel to work
     alongside the students for a few days.
 \end{enumerate}
 With the aforementioned rules we divided all the project into four
-different teams: Colab, Noosfero, Design, and DevOps. Each team had one
-coach responsible for reducing the communication problem with the other
-teams and help the members to organize themselves in the best way for
-everyone (always respecting their work time). The coach was one of the
-students working in some of the teams with the extra duty of registering
-the current tasks developed in the sprint and with the responsibility to
-talk with other teams. One important thing to notice is the mutability
-of the team and the coach, during the project many students changed
-between the teams to try different areas.
+different teams: Colab, Noosfero, Design, and DevOps. One student of each team was the
+coach, responsible for reducing the communication problem with other
+teams and helping the members to organize themselves in the best way for
+everyone (always respecting their work time). The coach
+had also the extra duty of registering
+the current tasks developed in the sprint.
+One important thing to notice is the mutability
+of the team and the coach. During the project many students changed
+their teams to try different areas.
 One characteristic of the teams was the presence of (at least) one
 senior per team. This was essential, because hard decisions and complex
-problems were usually referred to them. This kept having to take
-complicated technical decisions from the coach tole, what encouraged
+problems were usually referred to them. So, it was not the coach role
+to deal with complicated technical decisions, what encouraged
 students to be coaches. Lastly, the senior developers worked directly
-with the students, and this was important to give the undergraduate the
-opportunity to interact with a savvy professional in his area and to
+with the students, and this was important to give to students the
+opportunity to interact with a savvy professional in their areas and to
 keep the knowledge flowing in the project.
-Finally, we had to add two last elements of the team organization, that
-was essential for the project harmony: the meta-coach and professors.
-The former was a software engineer recently graduated and which wanted
-to keep working on the project, the latter were professors that
+Finally, we had to add two more elements to the team organization that
+were essential for the project harmony: the meta-coach and professors.
+The former was a software engineer recently graduated that wanted
+to keep working on the project. The latter were professors that
 orchestrated all the interactions between all members of the project.
 The meta-coach usually worked in one specific team and had the extra
-task of knowing the current status of all teams. Professors and
-meta-coaches worked together to reduce the communication problem between
-all the teams. Lastly, all the paperwork tasks, such as reporting on the
-project progress to the Ministry, was handled by the professors.
+task of knowing the current status of all teams. Professors and the
+meta-coach worked together to reduce the communication problem among
+teams. Lastly, all the paperwork tasks, such as reporting on the
+project progress to the Ministry, was handled by professors.
 \subsection{Communication and management}
-Our team had many people working together, and most of the seniors worked in a
-different city remotely. Also, we tried to keep our work completely clear to
+Our team had many people working together, and most of the seniors worked in
+different cities remotely. Also, we tried to keep our work completely clear to
 the Brazilian government and citizens interested in following the project. To
-handle those cases, we used a set of tools to communication and other to manage
-the project.
+handle these cases, we used a set of communication and management tools.
-For communication between members in different places, we used: video
+For communication between members in different places we used: video
 conferencing with shared terminal tools, IRC, and mailing lists. For example,
 when one student had to work in pair with a senior, normally, they used video
 conferencing tool for talking and shared a terminal session (both typing and
-seeing the screen in real time). For questions and fast discussion, we used
+seeing each other screen in real time). For questions and fast discussion, we used
 IRC. For general notification, we used the mailing lists.
 For managing the project we used the SPB Portal itself; first to validate it by
 ourselves, and also because it had all the required tools. We basically created
 one wiki page per release in the SPB Gitlab instance with a mapping between
-strategical, tactical, and operational views. In a practical point of view, one
-milestone per user history (feature), and one or more issues for addressing
+strategical, tactical, and operational views. We had one
+milestone per user history (feature) and one or more issues for addressing
 each feature. With this approach we achieved two important goals: keeping all
-the management as close possible to to the source code, and tracking every
+the management as close as possible to the source code and tracking every
 feature developed during the project. Our decision to
-use Wiki initially was empirical, but later reinforced by a research conducted
-by Joseph Chao showing the advantage of using Wikis \cite{chao2007student,
+use the Wiki initially was empirical, but later reinforced by a research conducted
+by Joseph Chao showing the advantage of using Wikis~\cite{chao2007student,
 opensourcestyle}.
 \subsection{High-level project management and reporting}
@@ -117,14 +116,14 @@ The Brazilian government used to work with software development in a
 very traditional way. They would frequently focus on documents and not
 on what was, in our opinion, what really matters: working software. This
 dissonance caused us a communication noise with MP, because they would
-often question our work style.  It was especially hard to convince them
+often question our work style. It was especially hard to convince them
 to accept the idea of open scope and agile development, but after months
 of labor and showing results they stopped resisting.
 We defined some level of meeting granularity to avoid generating too
-much overhead to the developers. We had a strategical and validating
+much overhead to the developers. We had a strategical and a validating
 meeting with MP (the former once in a month and the latter each 15th
-day), release plaining with the entire team (one per month), and finally
+day), a release planning with the entire team (one per month), and finally
 a sprint planning (one each 15th day).  Figure \ref{fig:meeting} is a
 diagram that represents our meeting organization.
@@ -141,41 +140,42 @@ coach of each team, the meta-coach, and some employees of the MP would
 participate in this meeting. We usually discussed what the team already
 produced since our last meeting, and established the new features for
 the next release.  Notice that just part of the team would join this
-meeting to avoid generating unnecessary overhead to the developers, but
-all the students interested to participate were allowed to join (many
-students wanted this experience during the project).
+meeting to avoid generating unnecessary overhead to developers, but
+all the students interested to participate were allowed to join, since many
+students wanted this experience during the project.
 After the strategical meeting with Brazilian government agents, we had a
-planning turn with all teams together. In this part, each team worked together
+planning phase with all teams together. In this part, each team worked together
 to convert the MP wishes into smaller parts which were represented by the epics of
-the release. Each coach was responsible for conducting the planning, and after
-that record it on the project wiki (the wiki provided by Gitlab). With this
-epic, each 14th day the team have generated their sprint scheduler (with small
+the release. Each coach was responsible for conducting the planning and 
+recording it on the project wiki (the wiki provided by Gitlab). With this
+epic, each 14th day the team have documented their sprint schedule (with small
 achievements mapped to issues).
 To keep the Brazilian government always updated, we invited them to work
 with us to validate the new features in progress. Normally we had a
-meeting each 15th day. Basically, this was our work flow, we always kept
+meeting each 15th day. Basically, this was our work flow. We always kept
 everything extremely open to the MP (our way of working, and the one
 often used by open source projects) and to the team.
-To keep the track of all of those things we used the SPB itself,
+To keep the track of all of these things we used the SPB itself,
 especially Gitlab. Basically, we had:
 \begin{enumerate}
-  \item Project repository: We have one organization with many repositories;
-  \item Milestones: Each milestone was used to register a release;
-  \item Wiki: Each release has one page on wiki with the compilation of
+  \item Project repository: we have one organization with many repositories;
+  \item Milestones: each milestone was used to register a release;
+  \item Wiki: each release has one wiki page with the compilation of the
     strategical meeting;
-  \item Issues: Each sprint planning generated issues, which we associated to
-    the specific milestone and updated the wiki with the issue number related
-    with them. Finally each developer assigned the issue to itself.
+  \item Issues: each sprint planning generated issues, which we associated to
+    the related milestone and registered on the related wiki page.
+    Finally, each developer assigned the issue to herself.
 \end{enumerate}
 Notice that this workflow gave us and the Brazilian government agents full
 traceability from a high level view of each feature to the lowest level (code).
-It is important to highlight that we converge to this workflow after many
+It is important to highlight that we converged to this workflow after many
 experiments. For example, we used a tool named Redmine for organizing our tasks
-during the sprints, however, this tool revealed inefficient for our case since
+during some sprints. However, this tool revealed to be inefficient for our case since
 the government agents lost part of the project traceability. We realized that
-centralize all the work in the SPB portal is the best option for our case.
+centralize all the work in the SPB portal was the best option for our case.
+
	@@ -27,7 +27,7 @@ managers from the Brazilian Government.		@@ -27,7 +27,7 @@ managers from the Brazilian Government.
27	For the majority of the students, this was a first professional experience.	27	For the majority of the students, this was a first professional experience.
28	Even though, our development process defined a central role on students participation.	28	Even though, our development process defined a central role on students participation.
29		29
30	-textbf{Q3:} \textit{How to introduce collaborative and agile	30	+\textbf{Q3:} \textit{How to introduce collaborative and agile
31	practices typical in FLOSS environments in the governmental development process?}	31	practices typical in FLOSS environments in the governmental development process?}
32	%	32	%
33	The software development on Brazilian government is based on a very traditional way,	33	The software development on Brazilian government is based on a very traditional way,