re-organizing sections

Paulo Meireles
1 parent ae697ff7
Showing 9 changed files with 363 additions and 363 deletions Show diff stats
opensym2017/content/04-architecture.tex
opensym2017/content/04-process.tex
opensym2017/content/05-architecture.tex
opensym2017/content/05-features.tex
opensym2017/content/06-features.tex
opensym2017/content/06-ux.tex
opensym2017/content/07-process.tex
opensym2017/content/07-ux.tex
opensym2017/spb.tex
@@ -0,0 +1,159 @@
+\section{Architecture}
+\label{sec:architecture}
+
+Based on the great list of functional requirements desired by Brazilian Federal
+Government we decided to select some FOSS systems that already contemplate some
+of them and improve these systems. And bringing the idea of community, it is
+undoable build a platform to be used by communities, which is a complex
+scenario, using just one tool.
+
+At the point of view of the architecture, two main requirements was brought by
+the Brazilian Federal Government for the new platform:
+
+\begin{enumerate}
+  \item \textit{Integrate existing FOSS systems}, with minimal differences from
+    their original versions;
+  \item \textit{Provide a consistent user interface} across the different
+    systems, as well as centralized authentication.
+\end{enumerate}
+
+Adopting existing FOSS systems by the government could bring the benefit of
+improvements done by the upstream communities, and the maintenance effort. On
+the other hand, integrating different tools with distinct intent, it is not an
+easy task and it was important to have a consistent user interface which
+justifies the last requirement.
+
+For the first requirement, we identified four main systems that required
+specialized teams for work in the integration process. The teams learned how to
+develop for their assigned systems and contributed to the  original
+communities, so that the version we used were not significantly different from
+the original. Unfortunately, the deadlines at the end of the project forced us
+to use our own version before the features were fully reviewed and integrated
+into the upstream of the project.
+
+At the end of the project, SPB portal was composed of more than ten systems
+among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix,
+Munin, and so forth. The following sections explained a little bit of Colab,
+Noosfero, Mezuro and Gitlab (the main tools which we contributed). Later, we
+described how we integrated those tools and conclude with the deployment.
+
+\subsection{Colab}
+
+Colab integrates web applications as its main goal, the user of this composed
+system should not notice the change between the integrated applications. Colab
+was designed to use one plugin for each system under its domain, this is
+guaranteed by four levels of integration:
+
+\begin{itemize}
+  \item Authentication
+  \item Visual
+  \item Events
+  \item Data and search engine
+\end{itemize}
+
+The aforementioned integrations levels were possible, because Colab works as a
+reverse proxy, therefore all external requests pass through it.
+
+Single Sign-On (SSO) is used to login users throughout all integrated
+applications. REMOTE\_USER HTTP header is sent to the applications and we
+expect that they know how to handle it, since this is a common authentication
+mechanism. The integrated applications should be on a local network to avoid
+security issues. With this the user will be able to navigate through the
+platform applications and will not be asked for authentication credentials.
+
+As Colab is a reverse proxy, it makes some HTML transformations in the HTTP
+response of integrated applications to provide a unified interface. Allows one
+to define some default templates to be used by all applications and overwrite
+when needed in its plugin. This approach allowed us to reuse some HTML pages
+which facilitates maintenance.
+
+A publish-subscribe implementation was used to handle events in the platform.
+Thus, if some application want to trigger some action in case that other
+application do something is possible. A registration of the desired events and
+implementation of the handlers must be done in the plugin of each application.
+This bring us many options to innovate in these integrations.
+
+A integrated search engine is provided by Colab. Each plugin specify which data
+will be indexed and will became available for the users, just need an simple
+implementation of how should perform the collection of data.
+
+Initially, Colab had support for a small set of applications (Trac, GNU
+Mailman, Apache Lucene) and all of them was hard-coded; our team evolved Colab
+so that it can now receive plugins to add support for new applications with
+minimal changes to its existing core. We developed plugins to be used in the
+SPB platform: Noosfero, GitLab, and Mezuro.
+
+\subsection{Noosfero}
+
+Noosfero\footnote{\url{http://noosfero.org/}} is a software for building
+social and collaboration networks. Besides the classical social
+networking features, it also provides publication features such as blogs
+and a general-purpose CMS (Content Management System). Most of the user
+interactions with SPB is through Noosfero: user registration, project
+home pages and documentation, and contact forms.
+
+\subsection{Gitlab and Mezuro}
+
+GitLab\footnote{\url{http://gitlab.com/}} is a web-based Git repository
+manager with wiki pages and issue tracking features.
+Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code
+metric to monitor the internal quality of softwares written in C, C++,
+Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists.
+
+\subsection{System unification}
+
+\begin{figure}[hbt]
+  \centering
+    \includegraphics[width=\linewidth]{figures/arch.png}
+  \caption{SPB architecture overview.}
+  \label{fig:architecture}
+\end{figure}
+
+The conceptual architecture of the platform is presented in Figure
+\ref{fig:architecture}. Colab initially handles all user interaction,
+directing requests to one of the integrated applications. It
+post-processes responses from the applications to apply a consistent
+visual appearance, manages authentication, and provides a unified search
+functionality: instead of using the redundant restricted search
+functionality of each application, a search in the SPB portal might
+return content from any of the applications, be it web pages, mailing
+list posts, or source code.
+
+% Falar do devops
+\subsection{Deploy}
+
+The SPB platform was deployed in 7 virtual machines with different functions,
+as we can see in Figure \ref{fig:architecture2}.
+
+\begin{figure*}[hbt]
+  \centering
+    \includegraphics[width=.8\linewidth]{figures/arch2.png}
+  \caption{Instanciation view of the SPB architecture.}
+  \label{fig:architecture2}
+\end{figure*}
+
+The \textit{reverseproxy} handles the HTTP requests and redirects them to the
+\textit{integration}, the \textit{email} sends and receives e-mails on behalf
+of the platform and the \textit{monitor} keeps the entire environment tracked.
+These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and
+\textit{monitor} - are accessible via Internet and the other ones are only
+available in the local network created between them.
+
+\textit{Integration} works as a second layer of proxy beneath
+\textit{reverseproxy}, any request to the platform will be handled by it. The
+Colab service provides interface, authentication and search engine integration
+among all the services. When a request is received to a specific service,
+Colab authenticates the user in the target tool, sends the request and makes a
+visual transformation in the HTML page which is the content of the response.
+Another user-oriented feature is the integrated search engine, when the user
+want to find something in the platform Colab will perform the search in the
+whole databases. Colab itself provides a web interface for GNU Mailman and we
+have two others integrated tools in \textit{integration}: Gitlab and Prezento.
+Gitlab provides web interface for Git repositories and issues tracker, and
+Prezento is a front-end for source code static analysis.
+
+The source code static analysis is performed by \textit{mezuro}. It runs some
+static analysis tools on source code stored in repository and provide this data
+to Prezento. A social network and CMS (Content Manager System) is provided by
+Noosfero in \textit{social}, and the databases of all tools with a cache
+service are in \textit{database}.
@@ -1,144 +0,0 @@
-\section{Development Organization and Process}
-\label{sec:process}
-
-The SPB team was composed of a variety of professionals with different levels and skills, where most of them were undergraduate students with major in software engineering (from 4th semester or upper). 
-Since the students could not dedicate many hours per week to the project, they always had the flexibility to negotiate their work schedule during the semester in order not to cause any damage to their grades. Their daily work routine in the project included programming and devops tasks. 
-
-The development of SPB project required a vast experience and background that usually undergraduate students do not have yet. For this reason, some senior developers have joined to the project to help with hard issues and to transfer knowledge to the students. Their main task was to provide solutions for complex problems, in other words, they worked as a developer. As these professionals are very skillful and the project could not fund a full time work for them, some of them worked partially on the project. In addition, they lived in a different states spread around Brazil which led much of the communication to be made via Internet.
-
-Finally, the last group of actors of this project was composed of employees
-formally working for the Brazilian Government, in the Ministery of Planning,
-development and Management (MPOG is the Brazilian acronyms). All the project
-decisions, validations, and scope definitions were made by them. As can be
-seen, the project had many kinds of profiles that had to be organized and
-synchronized.
-
-\subsection{Teams organizations}
-
-More than X\% of the teams was composed of undergraduate students and they
-worked physically in the same laboratory in the opposite of the senior. Each
-student had their own scheduler based on their class, it made complicated to
-implement pair programming. Also, they had a different area of interests. To
-cope with those diversity, we had two basic rules which guided the project
-organization:
-
-\begin{enumerate}
-  \item Classes have to be the top priority for undergraduate students;
-  \item Always work in pair (locally or remotely).
-\end{enumerate}
-
-With the aforementioned rules we divided all the project into four different
-teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible
-for reducing the communication problem with the other teams and help the
-members to organize itself in the best way for everyone (always respecting the
-work time). The coach, was a normal student working in some of the teams with
-the extra duty of register the current tasks developed in the sprint and with
-the responsibility to talk with other teams. One important thing to notice is
-the mutability of the team and the coach, during the project many students
-changed between the teams to try different areas.
-
-One characteristic of the teams was the presence of (at least) one senior per
-team. This was essential, because hard decisions and complex problems were
-usually addressed to them, this relieved the coach duty to take a complicated
-technical decisions and encouraged students to be a coach. Lastly, the senior
-had to respect a rule number two and work with students, this was important to
-gave the undergraduate the opportunity to interact with a savvy professional in
-his area and keeping the knowledge flow in the project.
-
-Finally, we had to add two last elements of the team organization, that was
-essential for the project harmony: the meta-coach and professors. The former
-was a software engineer recently graduated and which wanted to keep working on
-the project, the latter were professors that orchestrated all the interactions
-between all members of the project. The meta-coach usually worked in one
-specific team and had the extra task of knowing the current status of all
-teams. Professors and meta-coaches worked together to reduce the communication
-problem between all the teams. Lastly, all the bureaucracy tasks was
-centralized in the professors.
-
-\subsection{Meetings}
-
-Brazilian government used to work with software development in a very
-traditional way, frequently they claim on documents and does not focus on what
-really matter (running software). This way of thinking caused to us a
-communication noise with MPOG, because they constantly tried to leverage on our
-work style. It was especially hard to convince them to accept the idea of open
-scope and agile development, but after months of labor and showing results they
-stopped resisting.
-
-We defined some level of meeting granularity to avoid to generate overheads to
-the developers. We had a strategical and validating meeting with MPOG (the
-former once in a month and the latter each 15th day), release plaining with the
-entire team (one per month), and finally a sprint planning (one each 15th day).
-Figure \ref{fig:meeting} is a diagram that represents our meeting organization.
-
-\begin{figure}[hbt]
-  \centering
-    \includegraphics[width=\linewidth]{figures/meeting_flows.png}
-  \caption{Meetings cycles}
-  \label{fig:meeting}
-\end{figure}
-
-In the strategical meeting we usually defined the priorities and new features
-with MPOG (we always had to negotiate next steps with them). Normally the
-professors, the coach of each team, the meta-coach, and some employees of MPOG
-join in this meeting. We usually discussed what the team already produced since
-our last meeting, and we establish the new features for the next release.
-Notice that just part of the team join in this meeting to avoid generating
-unnecessary overhead to the developers, but all the students interested to
-participate was allowed to join (many students wanted this experience during
-the project).
-
-After the strategical meeting with MPOG, we had a planning turn with all teams
-together. In this part, each team worked together to convert the MPOG wishes
-into small parts which was represented by the epics of the release. Each coach
-was responsible for conducting the planning, and after that register it on the
-project wiki (the wiki provided by Gitlab). With this epic, each 14th day the
-team have generated their sprint scheduler (with small achievements mapped in
-issues).
-
-To keep MPOG always updated, we invited them to work with us to validate the
-new features in progress. Normally we had a meeting each 15th day. Basically,
-this was our work flow, we always kept everything extremely open to the MPOG
-(our way of work and open source projects) and to the team.
-
-To keep the track of all of those things we used the SPB, especially the
-Gitlab. Basically, we had:
-
-\begin{enumerate}
-  \item Project repository: We have one organization with many repositories
-  \item Milestones: Each milestone is used to register a release
-  \item Wiki: Each release has one page on wiki with the compilation of
-    strategical meeting
-  \item Issues: Each sprint planning generated issues, which we associated to
-    the specific milestone and updated the wiki with the issue number related
-    with them. Finally each developer assigned the issue to itself.
-\end{enumerate}
-
-Notice that this workflow gave to us and to the MPOG a full traceability from
-high view of the feature to the low view (code). This provided a way to MPOG
-validated all worked done and proof the concept that work with open source
-project can give a proper view to them check.
-
-\subsection{Tools for communication and management}
-
-Our team had many people worked together, and most of the seniors worked in a
-different city remotely. Also, we tried to keep our work completely clear to
-the Brazilian government and citizens interested in follow the project. To
-handle those cases, we used a set of tools to communication and other to manage
-the project.
-
-For communication between member in different places, we used: google-talk with
-tmate, IRC, and mailing-list. When one student had to work in pair with a
-senior, normally, they used google-hangout for communication and they shared a
-session with tmate which allow them to share the same terminal. For questions
-and fast discussion, we used IRC. For general notification, we used the
-mailing-list.
-
-For managing the project we used the SPB Portal to validate it by ourselves and
-because it had all the required tools. We basically create one wiki page per
-release in Gitlab, one milestone per sprint, and one or more issues for address
-one user history. With this approach we achieve two important things: keep all
-the management close to the source code and tracked every feature developed by
-the project.
-
-% Ainda falta adicionar a parte da visita dos seniors e o turno sagrado
@@ -1,159 +0,0 @@
-\section{Architecture}
-\label{sec:architecture}
-
-Based on the great list of functional requirements desired by Brazilian Federal
-Government we decided to select some FOSS systems that already contemplate some
-of them and improve these systems. And bringing the idea of community, it is
-undoable build a platform to be used by communities, which is a complex
-scenario, using just one tool.
-
-At the point of view of the architecture, two main requirements was brought by
-the Brazilian Federal Government for the new platform:
-
-\begin{enumerate}
-  \item \textit{Integrate existing FOSS systems}, with minimal differences from
-    their original versions;
-  \item \textit{Provide a consistent user interface} across the different
-    systems, as well as centralized authentication.
-\end{enumerate}
-
-Adopting existing FOSS systems by the government could bring the benefit of
-improvements done by the upstream communities, and the maintenance effort. On
-the other hand, integrating different tools with distinct intent, it is not an
-easy task and it was important to have a consistent user interface which
-justifies the last requirement.
-
-For the first requirement, we identified four main systems that required
-specialized teams for work in the integration process. The teams learned how to
-develop for their assigned systems and contributed to the  original
-communities, so that the version we used were not significantly different from
-the original. Unfortunately, the deadlines at the end of the project forced us
-to use our own version before the features were fully reviewed and integrated
-into the upstream of the project.
-
-At the end of the project, SPB portal was composed of more than ten systems
-among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix,
-Munin, and so forth. The following sections explained a little bit of Colab,
-Noosfero, Mezuro and Gitlab (the main tools which we contributed). Later, we
-described how we integrated those tools and conclude with the deployment.
-
-\subsection{Colab}
-
-Colab integrates web applications as its main goal, the user of this composed
-system should not notice the change between the integrated applications. Colab
-was designed to use one plugin for each system under its domain, this is
-guaranteed by four levels of integration:
-
-\begin{itemize}
-  \item Authentication
-  \item Visual
-  \item Events
-  \item Data and search engine
-\end{itemize}
-
-The aforementioned integrations levels were possible, because Colab works as a
-reverse proxy, therefore all external requests pass through it.
-
-Single Sign-On (SSO) is used to login users throughout all integrated
-applications. REMOTE\_USER HTTP header is sent to the applications and we
-expect that they know how to handle it, since this is a common authentication
-mechanism. The integrated applications should be on a local network to avoid
-security issues. With this the user will be able to navigate through the
-platform applications and will not be asked for authentication credentials.
-
-As Colab is a reverse proxy, it makes some HTML transformations in the HTTP
-response of integrated applications to provide a unified interface. Allows one
-to define some default templates to be used by all applications and overwrite
-when needed in its plugin. This approach allowed us to reuse some HTML pages
-which facilitates maintenance.
-
-A publish-subscribe implementation was used to handle events in the platform.
-Thus, if some application want to trigger some action in case that other
-application do something is possible. A registration of the desired events and
-implementation of the handlers must be done in the plugin of each application.
-This bring us many options to innovate in these integrations.
-
-A integrated search engine is provided by Colab. Each plugin specify which data
-will be indexed and will became available for the users, just need an simple
-implementation of how should perform the collection of data.
-
-Initially, Colab had support for a small set of applications (Trac, GNU
-Mailman, Apache Lucene) and all of them was hard-coded; our team evolved Colab
-so that it can now receive plugins to add support for new applications with
-minimal changes to its existing core. We developed plugins to be used in the
-SPB platform: Noosfero, GitLab, and Mezuro.
-
-\subsection{Noosfero}
-
-Noosfero\footnote{\url{http://noosfero.org/}} is a software for building
-social and collaboration networks. Besides the classical social
-networking features, it also provides publication features such as blogs
-and a general-purpose CMS (Content Management System). Most of the user
-interactions with SPB is through Noosfero: user registration, project
-home pages and documentation, and contact forms.
-
-\subsection{Gitlab and Mezuro}
-
-GitLab\footnote{\url{http://gitlab.com/}} is a web-based Git repository
-manager with wiki pages and issue tracking features.
-Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code
-metric to monitor the internal quality of softwares written in C, C++,
-Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists.
-
-\subsection{System unification}
-
-\begin{figure}[hbt]
-  \centering
-    \includegraphics[width=\linewidth]{figures/arch.png}
-  \caption{SPB architecture overview.}
-  \label{fig:architecture}
-\end{figure}
-
-The conceptual architecture of the platform is presented in Figure
-\ref{fig:architecture}. Colab initially handles all user interaction,
-directing requests to one of the integrated applications. It
-post-processes responses from the applications to apply a consistent
-visual appearance, manages authentication, and provides a unified search
-functionality: instead of using the redundant restricted search
-functionality of each application, a search in the SPB portal might
-return content from any of the applications, be it web pages, mailing
-list posts, or source code.
-
-% Falar do devops
-\subsection{Deploy}
-
-The SPB platform was deployed in 7 virtual machines with different functions,
-as we can see in Figure \ref{fig:architecture2}.
-
-\begin{figure*}[hbt]
-  \centering
-    \includegraphics[width=.8\linewidth]{figures/arch2.png}
-  \caption{Instanciation view of the SPB architecture.}
-  \label{fig:architecture2}
-\end{figure*}
-
-The \textit{reverseproxy} handles the HTTP requests and redirects them to the
-\textit{integration}, the \textit{email} sends and receives e-mails on behalf
-of the platform and the \textit{monitor} keeps the entire environment tracked.
-These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and
-\textit{monitor} - are accessible via Internet and the other ones are only
-available in the local network created between them.
-
-\textit{Integration} works as a second layer of proxy beneath
-\textit{reverseproxy}, any request to the platform will be handled by it. The
-Colab service provides interface, authentication and search engine integration
-among all the services. When a request is received to a specific service,
-Colab authenticates the user in the target tool, sends the request and makes a
-visual transformation in the HTML page which is the content of the response.
-Another user-oriented feature is the integrated search engine, when the user
-want to find something in the platform Colab will perform the search in the
-whole databases. Colab itself provides a web interface for GNU Mailman and we
-have two others integrated tools in \textit{integration}: Gitlab and Prezento.
-Gitlab provides web interface for Git repositories and issues tracker, and
-Prezento is a front-end for source code static analysis.
-
-The source code static analysis is performed by \textit{mezuro}. It runs some
-static analysis tools on source code stored in repository and provide this data
-to Prezento. A social network and CMS (Content Manager System) is provided by
-Noosfero in \textit{social}, and the databases of all tools with a cache
-service are in \textit{database}.
@@ -0,0 +1,23 @@
+\section{Features}
+\label{sec:spb}
+
+%TODO: Paulo e Melissa
+
+The new generation of the SPB portal is made up of a combination of collaborative softwares features. Some of these features were expanded or limited  according to government requirements. Other required features, unavailable in these softwares, were developed and sent back to their official repository as contributions. As result, we have a platform which integrate and harmonize features like social networking, mailing list, version control system, content management and source code quality monitoring.
+
+Our aim was to develop features reusing functions available in the collaborative softwares which integrate the platform. In addition, we treated to remaining this integration transparent to end users.
+
+\subsection{Software, development tools and community}
+In the new SPB portal, each software has a standard set of pages and tools. Besides the pages with help content and user support, a software has a section of download and version control and several mechanisms to manager the software development within the platform. These mechanisms includes git repository, wiki pages and issues tracking. 
+
+Focusing on the collaborative development, each software has pages and tools to support the dialog and communication between developers, users and enthusiasts. The software has its own mailing list, where its privacy can be moderated according to the maturity of the software in this aspect. It also has a related community, where you can find its members, a blog or any other relevant content made by the community and a summary of recent activities in the spaces of interaction of this software. 
+
+In order to assist decision-making, the new SPB has acquired assessment and statistical tools. Now, the users can rate the software using a score and comment, so this information is displayed to anyone. Moreover, the software dispose a statistic section containing values which are calculated through data provided by user and system.
+
+The rule of administrator exist in the software and in its community. The administrator is responsible to moderate content to be published, members and comments from user rating. He is also the person who can make changes in the software homepage content.
+
+\subsection{Software Catalog and global search}
+
+Some available softwares on the platform are highlighted in the portal homepage, but the best way to index all softwares are by a searching in the Software Catalog. In this catalog, some search options like filters by kind of software or categories, sorting and score were developed to improve the navigation.
+
+Expanding the scope of search to cover other kinds of content, the SPB team developed the global search tool. This tool integrates the search mechanisms provided by the collaborative softwares that make up the platform. Any user can find a public content in the context of social network, mailing list and software development.
 \ No newline at end of file
@@ -1,23 +0,0 @@
-\section{Features}
-\label{sec:spb}
-
-%TODO: Paulo e Melissa
-
-The new generation of the SPB portal is made up of a combination of collaborative softwares features. Some of these features were expanded or limited  according to government requirements. Other required features, unavailable in these softwares, were developed and sent back to their official repository as contributions. As result, we have a platform which integrate and harmonize features like social networking, mailing list, version control system, content management and source code quality monitoring.
-
-Our aim was to develop features reusing functions available in the collaborative softwares which integrate the platform. In addition, we treated to remaining this integration transparent to end users.
-
-\subsection{Software, development tools and community}
-In the new SPB portal, each software has a standard set of pages and tools. Besides the pages with help content and user support, a software has a section of download and version control and several mechanisms to manager the software development within the platform. These mechanisms includes git repository, wiki pages and issues tracking. 
-
-Focusing on the collaborative development, each software has pages and tools to support the dialog and communication between developers, users and enthusiasts. The software has its own mailing list, where its privacy can be moderated according to the maturity of the software in this aspect. It also has a related community, where you can find its members, a blog or any other relevant content made by the community and a summary of recent activities in the spaces of interaction of this software. 
-
-In order to assist decision-making, the new SPB has acquired assessment and statistical tools. Now, the users can rate the software using a score and comment, so this information is displayed to anyone. Moreover, the software dispose a statistic section containing values which are calculated through data provided by user and system.
-
-The rule of administrator exist in the software and in its community. The administrator is responsible to moderate content to be published, members and comments from user rating. He is also the person who can make changes in the software homepage content.
-
-\subsection{Software Catalog and global search}
-
-Some available softwares on the platform are highlighted in the portal homepage, but the best way to index all softwares are by a searching in the Software Catalog. In this catalog, some search options like filters by kind of software or categories, sorting and score were developed to improve the navigation.
-
-Expanding the scope of search to cover other kinds of content, the SPB team developed the global search tool. This tool integrates the search mechanisms provided by the collaborative softwares that make up the platform. Any user can find a public content in the context of social network, mailing list and software development.
 \ No newline at end of file
@@ -0,0 +1,33 @@
+\section{User eXperience evolution}
+
+The integration of collaborative environments goes beyond functional aspects.
+Offering the population an unified experience across these environments has
+been the key to encourage the use of the platform as it reduces the perception
+of complexity. Thus, the SPB Portal information architecture was redesigned
+to provide a transparent navigation and to reach users with different profiles.
+A process of harmonization has been employed on the interaction models of each
+tool to reduce the learning curve. At the same time, a new visual style was
+created to unify the navigation experience and to comply with the guidelines of
+the digital communication identity standard established by the Federal
+Government.
+
+With the increase in system features and the addition of new tools, the
+visual style has steadily evolved to keep the navigation unified.  Moreover,
+tools from different backgrounds, which in many cases provide similar
+functionality, prompted the development of an unified interface. Some
+features, such as search and user profile editing were eliminated from
+the individual applications, and implemented centrally to ensure a
+consistent look and feel.
+
+Another challenge was responsive web design. The integrated applications
+had varying degrees of support for responsiveness, and the common
+interface had to adapt for each individual scenario. In particular
+Noosfero did not yet have a responsive design; we engaged in its
+development and contributed towards that goal.
+
+After the initial release of the new SPB Portal in 2014, several
+validations activities were implemented in 2015 and 2016. The aim was to
+provide the most wanted features by casual users (such as public
+servants interested in downloads and documentation) immediately, while
+allowing more experienced users (such as developers) to easily drill down
+to the details.
@@ -0,0 +1,144 @@
+\section{Development Organization and Process}
+\label{sec:process}
+
+The SPB team was composed of a variety of professionals with different levels and skills, where most of them were undergraduate students with major in software engineering (from 4th semester or upper). 
+Since the students could not dedicate many hours per week to the project, they always had the flexibility to negotiate their work schedule during the semester in order not to cause any damage to their grades. Their daily work routine in the project included programming and devops tasks. 
+
+The development of SPB project required a vast experience and background that usually undergraduate students do not have yet. For this reason, some senior developers have joined to the project to help with hard issues and to transfer knowledge to the students. Their main task was to provide solutions for complex problems, in other words, they worked as a developer. As these professionals are very skillful and the project could not fund a full time work for them, some of them worked partially on the project. In addition, they lived in a different states spread around Brazil which led much of the communication to be made via Internet.
+
+Finally, the last group of actors of this project was composed of employees
+formally working for the Brazilian Government, in the Ministery of Planning,
+development and Management (MPOG is the Brazilian acronyms). All the project
+decisions, validations, and scope definitions were made by them. As can be
+seen, the project had many kinds of profiles that had to be organized and
+synchronized.
+
+\subsection{Teams organizations}
+
+More than X\% of the teams was composed of undergraduate students and they
+worked physically in the same laboratory in the opposite of the senior. Each
+student had their own scheduler based on their class, it made complicated to
+implement pair programming. Also, they had a different area of interests. To
+cope with those diversity, we had two basic rules which guided the project
+organization:
+
+\begin{enumerate}
+  \item Classes have to be the top priority for undergraduate students;
+  \item Always work in pair (locally or remotely).
+\end{enumerate}
+
+With the aforementioned rules we divided all the project into four different
+teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible
+for reducing the communication problem with the other teams and help the
+members to organize itself in the best way for everyone (always respecting the
+work time). The coach, was a normal student working in some of the teams with
+the extra duty of register the current tasks developed in the sprint and with
+the responsibility to talk with other teams. One important thing to notice is
+the mutability of the team and the coach, during the project many students
+changed between the teams to try different areas.
+
+One characteristic of the teams was the presence of (at least) one senior per
+team. This was essential, because hard decisions and complex problems were
+usually addressed to them, this relieved the coach duty to take a complicated
+technical decisions and encouraged students to be a coach. Lastly, the senior
+had to respect a rule number two and work with students, this was important to
+gave the undergraduate the opportunity to interact with a savvy professional in
+his area and keeping the knowledge flow in the project.
+
+Finally, we had to add two last elements of the team organization, that was
+essential for the project harmony: the meta-coach and professors. The former
+was a software engineer recently graduated and which wanted to keep working on
+the project, the latter were professors that orchestrated all the interactions
+between all members of the project. The meta-coach usually worked in one
+specific team and had the extra task of knowing the current status of all
+teams. Professors and meta-coaches worked together to reduce the communication
+problem between all the teams. Lastly, all the bureaucracy tasks was
+centralized in the professors.
+
+\subsection{Meetings}
+
+Brazilian government used to work with software development in a very
+traditional way, frequently they claim on documents and does not focus on what
+really matter (running software). This way of thinking caused to us a
+communication noise with MPOG, because they constantly tried to leverage on our
+work style. It was especially hard to convince them to accept the idea of open
+scope and agile development, but after months of labor and showing results they
+stopped resisting.
+
+We defined some level of meeting granularity to avoid to generate overheads to
+the developers. We had a strategical and validating meeting with MPOG (the
+former once in a month and the latter each 15th day), release plaining with the
+entire team (one per month), and finally a sprint planning (one each 15th day).
+Figure \ref{fig:meeting} is a diagram that represents our meeting organization.
+
+\begin{figure}[hbt]
+  \centering
+    \includegraphics[width=\linewidth]{figures/meeting_flows.png}
+  \caption{Meetings cycles}
+  \label{fig:meeting}
+\end{figure}
+
+In the strategical meeting we usually defined the priorities and new features
+with MPOG (we always had to negotiate next steps with them). Normally the
+professors, the coach of each team, the meta-coach, and some employees of MPOG
+join in this meeting. We usually discussed what the team already produced since
+our last meeting, and we establish the new features for the next release.
+Notice that just part of the team join in this meeting to avoid generating
+unnecessary overhead to the developers, but all the students interested to
+participate was allowed to join (many students wanted this experience during
+the project).
+
+After the strategical meeting with MPOG, we had a planning turn with all teams
+together. In this part, each team worked together to convert the MPOG wishes
+into small parts which was represented by the epics of the release. Each coach
+was responsible for conducting the planning, and after that register it on the
+project wiki (the wiki provided by Gitlab). With this epic, each 14th day the
+team have generated their sprint scheduler (with small achievements mapped in
+issues).
+
+To keep MPOG always updated, we invited them to work with us to validate the
+new features in progress. Normally we had a meeting each 15th day. Basically,
+this was our work flow, we always kept everything extremely open to the MPOG
+(our way of work and open source projects) and to the team.
+
+To keep the track of all of those things we used the SPB, especially the
+Gitlab. Basically, we had:
+
+\begin{enumerate}
+  \item Project repository: We have one organization with many repositories
+  \item Milestones: Each milestone is used to register a release
+  \item Wiki: Each release has one page on wiki with the compilation of
+    strategical meeting
+  \item Issues: Each sprint planning generated issues, which we associated to
+    the specific milestone and updated the wiki with the issue number related
+    with them. Finally each developer assigned the issue to itself.
+\end{enumerate}
+
+Notice that this workflow gave to us and to the MPOG a full traceability from
+high view of the feature to the low view (code). This provided a way to MPOG
+validated all worked done and proof the concept that work with open source
+project can give a proper view to them check.
+
+\subsection{Tools for communication and management}
+
+Our team had many people worked together, and most of the seniors worked in a
+different city remotely. Also, we tried to keep our work completely clear to
+the Brazilian government and citizens interested in follow the project. To
+handle those cases, we used a set of tools to communication and other to manage
+the project.
+
+For communication between member in different places, we used: google-talk with
+tmate, IRC, and mailing-list. When one student had to work in pair with a
+senior, normally, they used google-hangout for communication and they shared a
+session with tmate which allow them to share the same terminal. For questions
+and fast discussion, we used IRC. For general notification, we used the
+mailing-list.
+
+For managing the project we used the SPB Portal to validate it by ourselves and
+because it had all the required tools. We basically create one wiki page per
+release in Gitlab, one milestone per sprint, and one or more issues for address
+one user history. With this approach we achieve two important things: keep all
+the management close to the source code and tracked every feature developed by
+the project.
+
+% Ainda falta adicionar a parte da visita dos seniors e o turno sagrado
@@ -1,33 +0,0 @@
-\section{User eXperience evolution}
-
-The integration of collaborative environments goes beyond functional aspects.
-Offering the population an unified experience across these environments has
-been the key to encourage the use of the platform as it reduces the perception
-of complexity. Thus, the SPB Portal information architecture was redesigned
-to provide a transparent navigation and to reach users with different profiles.
-A process of harmonization has been employed on the interaction models of each
-tool to reduce the learning curve. At the same time, a new visual style was
-created to unify the navigation experience and to comply with the guidelines of
-the digital communication identity standard established by the Federal
-Government.
-
-With the increase in system features and the addition of new tools, the
-visual style has steadily evolved to keep the navigation unified.  Moreover,
-tools from different backgrounds, which in many cases provide similar
-functionality, prompted the development of an unified interface. Some
-features, such as search and user profile editing were eliminated from
-the individual applications, and implemented centrally to ensure a
-consistent look and feel.
-
-Another challenge was responsive web design. The integrated applications
-had varying degrees of support for responsiveness, and the common
-interface had to adapt for each individual scenario. In particular
-Noosfero did not yet have a responsive design; we engaged in its
-development and contributed towards that goal.
-
-After the initial release of the new SPB Portal in 2014, several
-validations activities were implemented in 2015 and 2016. The aim was to
-provide the most wanted features by casual users (such as public
-servants interested in downloads and documentation) immediately, while
-allowing more experienced users (such as developers) to easily drill down
-to the details.
@@ -153,10 +153,10 @@
 \input{content/01-introduction}
 \input{content/02-spb}
 \input{content/03-requirements}
-\input{content/04-process}
-\input{content/05-architecture}
-\input{content/06-features}
-\input{content/07-ux}
+\input{content/04-architecture}
+\input{content/05-features}
+\input{content/06-ux}
+\input{content/07-process}
 \input{content/08-contributions}
 \input{content/09-lessons}
 \input{content/10-finals}
...	...	@@ -0,0 +1,159 @@
	1	+\section{Architecture}
	2	+\label{sec:architecture}
	3	+
	4	+Based on the great list of functional requirements desired by Brazilian Federal
	5	+Government we decided to select some FOSS systems that already contemplate some
	6	+of them and improve these systems. And bringing the idea of community, it is
	7	+undoable build a platform to be used by communities, which is a complex
	8	+scenario, using just one tool.
	9	+
	10	+At the point of view of the architecture, two main requirements was brought by
	11	+the Brazilian Federal Government for the new platform:
	12	+
	13	+\begin{enumerate}
	14	+ \item \textit{Integrate existing FOSS systems}, with minimal differences from
	15	+ their original versions;
	16	+ \item \textit{Provide a consistent user interface} across the different
	17	+ systems, as well as centralized authentication.
	18	+\end{enumerate}
	19	+
	20	+Adopting existing FOSS systems by the government could bring the benefit of
	21	+improvements done by the upstream communities, and the maintenance effort. On
	22	+the other hand, integrating different tools with distinct intent, it is not an
	23	+easy task and it was important to have a consistent user interface which
	24	+justifies the last requirement.
	25	+
	26	+For the first requirement, we identified four main systems that required
	27	+specialized teams for work in the integration process. The teams learned how to
	28	+develop for their assigned systems and contributed to the original
	29	+communities, so that the version we used were not significantly different from
	30	+the original. Unfortunately, the deadlines at the end of the project forced us
	31	+to use our own version before the features were fully reviewed and integrated
	32	+into the upstream of the project.
	33	+
	34	+At the end of the project, SPB portal was composed of more than ten systems
	35	+among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix,
	36	+Munin, and so forth. The following sections explained a little bit of Colab,
	37	+Noosfero, Mezuro and Gitlab (the main tools which we contributed). Later, we
	38	+described how we integrated those tools and conclude with the deployment.
	39	+
	40	+\subsection{Colab}
	41	+
	42	+Colab integrates web applications as its main goal, the user of this composed
	43	+system should not notice the change between the integrated applications. Colab
	44	+was designed to use one plugin for each system under its domain, this is
	45	+guaranteed by four levels of integration:
	46	+
	47	+\begin{itemize}
	48	+ \item Authentication
	49	+ \item Visual
	50	+ \item Events
	51	+ \item Data and search engine
	52	+\end{itemize}
	53	+
	54	+The aforementioned integrations levels were possible, because Colab works as a
	55	+reverse proxy, therefore all external requests pass through it.
	56	+
	57	+Single Sign-On (SSO) is used to login users throughout all integrated
	58	+applications. REMOTE\_USER HTTP header is sent to the applications and we
	59	+expect that they know how to handle it, since this is a common authentication
	60	+mechanism. The integrated applications should be on a local network to avoid
	61	+security issues. With this the user will be able to navigate through the
	62	+platform applications and will not be asked for authentication credentials.
	63	+
	64	+As Colab is a reverse proxy, it makes some HTML transformations in the HTTP
	65	+response of integrated applications to provide a unified interface. Allows one
	66	+to define some default templates to be used by all applications and overwrite
	67	+when needed in its plugin. This approach allowed us to reuse some HTML pages
	68	+which facilitates maintenance.
	69	+
	70	+A publish-subscribe implementation was used to handle events in the platform.
	71	+Thus, if some application want to trigger some action in case that other
	72	+application do something is possible. A registration of the desired events and
	73	+implementation of the handlers must be done in the plugin of each application.
	74	+This bring us many options to innovate in these integrations.
	75	+
	76	+A integrated search engine is provided by Colab. Each plugin specify which data
	77	+will be indexed and will became available for the users, just need an simple
	78	+implementation of how should perform the collection of data.
	79	+
	80	+Initially, Colab had support for a small set of applications (Trac, GNU
	81	+Mailman, Apache Lucene) and all of them was hard-coded; our team evolved Colab
	82	+so that it can now receive plugins to add support for new applications with
	83	+minimal changes to its existing core. We developed plugins to be used in the
	84	+SPB platform: Noosfero, GitLab, and Mezuro.
	85	+
	86	+\subsection{Noosfero}
	87	+
	88	+Noosfero\footnote{\url{http://noosfero.org/}} is a software for building
	89	+social and collaboration networks. Besides the classical social
	90	+networking features, it also provides publication features such as blogs
	91	+and a general-purpose CMS (Content Management System). Most of the user
	92	+interactions with SPB is through Noosfero: user registration, project
	93	+home pages and documentation, and contact forms.
	94	+
	95	+\subsection{Gitlab and Mezuro}
	96	+
	97	+GitLab\footnote{\url{http://gitlab.com/}} is a web-based Git repository
	98	+manager with wiki pages and issue tracking features.
	99	+Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code
	100	+metric to monitor the internal quality of softwares written in C, C++,
	101	+Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists.
	102	+
	103	+\subsection{System unification}
	104	+
	105	+\begin{figure}[hbt]
	106	+ \centering
	107	+ \includegraphics[width=\linewidth]{figures/arch.png}
	108	+ \caption{SPB architecture overview.}
	109	+ \label{fig:architecture}
	110	+\end{figure}
	111	+
	112	+The conceptual architecture of the platform is presented in Figure
	113	+\ref{fig:architecture}. Colab initially handles all user interaction,
	114	+directing requests to one of the integrated applications. It
	115	+post-processes responses from the applications to apply a consistent
	116	+visual appearance, manages authentication, and provides a unified search
	117	+functionality: instead of using the redundant restricted search
	118	+functionality of each application, a search in the SPB portal might
	119	+return content from any of the applications, be it web pages, mailing
	120	+list posts, or source code.
	121	+
	122	+% Falar do devops
	123	+\subsection{Deploy}
	124	+
	125	+The SPB platform was deployed in 7 virtual machines with different functions,
	126	+as we can see in Figure \ref{fig:architecture2}.
	127	+
	128	+\begin{figure*}[hbt]
	129	+ \centering
	130	+ \includegraphics[width=.8\linewidth]{figures/arch2.png}
	131	+ \caption{Instanciation view of the SPB architecture.}
	132	+ \label{fig:architecture2}
	133	+\end{figure*}
	134	+
	135	+The \textit{reverseproxy} handles the HTTP requests and redirects them to the
	136	+\textit{integration}, the \textit{email} sends and receives e-mails on behalf
	137	+of the platform and the \textit{monitor} keeps the entire environment tracked.
	138	+These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and
	139	+\textit{monitor} - are accessible via Internet and the other ones are only
	140	+available in the local network created between them.
	141	+
	142	+\textit{Integration} works as a second layer of proxy beneath
	143	+\textit{reverseproxy}, any request to the platform will be handled by it. The
	144	+Colab service provides interface, authentication and search engine integration
	145	+among all the services. When a request is received to a specific service,
	146	+Colab authenticates the user in the target tool, sends the request and makes a
	147	+visual transformation in the HTML page which is the content of the response.
	148	+Another user-oriented feature is the integrated search engine, when the user
	149	+want to find something in the platform Colab will perform the search in the
	150	+whole databases. Colab itself provides a web interface for GNU Mailman and we
	151	+have two others integrated tools in \textit{integration}: Gitlab and Prezento.
	152	+Gitlab provides web interface for Git repositories and issues tracker, and
	153	+Prezento is a front-end for source code static analysis.
	154	+
	155	+The source code static analysis is performed by \textit{mezuro}. It runs some
	156	+static analysis tools on source code stored in repository and provide this data
	157	+to Prezento. A social network and CMS (Content Manager System) is provided by
	158	+Noosfero in \textit{social}, and the databases of all tools with a cache
	159	+service are in \textit{database}.
...	...
...	...	@@ -1,144 +0,0 @@
1		-\section{Development Organization and Process}
2		-\label{sec:process}
3		-
4		-The SPB team was composed of a variety of professionals with different levels and skills, where most of them were undergraduate students with major in software engineering (from 4th semester or upper).
5		-Since the students could not dedicate many hours per week to the project, they always had the flexibility to negotiate their work schedule during the semester in order not to cause any damage to their grades. Their daily work routine in the project included programming and devops tasks.
6		-
7		-The development of SPB project required a vast experience and background that usually undergraduate students do not have yet. For this reason, some senior developers have joined to the project to help with hard issues and to transfer knowledge to the students. Their main task was to provide solutions for complex problems, in other words, they worked as a developer. As these professionals are very skillful and the project could not fund a full time work for them, some of them worked partially on the project. In addition, they lived in a different states spread around Brazil which led much of the communication to be made via Internet.
8		-
9		-Finally, the last group of actors of this project was composed of employees
10		-formally working for the Brazilian Government, in the Ministery of Planning,
11		-development and Management (MPOG is the Brazilian acronyms). All the project
12		-decisions, validations, and scope definitions were made by them. As can be
13		-seen, the project had many kinds of profiles that had to be organized and
14		-synchronized.
15		-
16		-\subsection{Teams organizations}
17		-
18		-More than X\% of the teams was composed of undergraduate students and they
19		-worked physically in the same laboratory in the opposite of the senior. Each
20		-student had their own scheduler based on their class, it made complicated to
21		-implement pair programming. Also, they had a different area of interests. To
22		-cope with those diversity, we had two basic rules which guided the project
23		-organization:
24		-
25		-\begin{enumerate}
26		- \item Classes have to be the top priority for undergraduate students;
27		- \item Always work in pair (locally or remotely).
28		-\end{enumerate}
29		-
30		-With the aforementioned rules we divided all the project into four different
31		-teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible
32		-for reducing the communication problem with the other teams and help the
33		-members to organize itself in the best way for everyone (always respecting the
34		-work time). The coach, was a normal student working in some of the teams with
35		-the extra duty of register the current tasks developed in the sprint and with
36		-the responsibility to talk with other teams. One important thing to notice is
37		-the mutability of the team and the coach, during the project many students
38		-changed between the teams to try different areas.
39		-
40		-One characteristic of the teams was the presence of (at least) one senior per
41		-team. This was essential, because hard decisions and complex problems were
42		-usually addressed to them, this relieved the coach duty to take a complicated
43		-technical decisions and encouraged students to be a coach. Lastly, the senior
44		-had to respect a rule number two and work with students, this was important to
45		-gave the undergraduate the opportunity to interact with a savvy professional in
46		-his area and keeping the knowledge flow in the project.
47		-
48		-Finally, we had to add two last elements of the team organization, that was
49		-essential for the project harmony: the meta-coach and professors. The former
50		-was a software engineer recently graduated and which wanted to keep working on
51		-the project, the latter were professors that orchestrated all the interactions
52		-between all members of the project. The meta-coach usually worked in one
53		-specific team and had the extra task of knowing the current status of all
54		-teams. Professors and meta-coaches worked together to reduce the communication
55		-problem between all the teams. Lastly, all the bureaucracy tasks was
56		-centralized in the professors.
57		-
58		-\subsection{Meetings}
59		-
60		-Brazilian government used to work with software development in a very
61		-traditional way, frequently they claim on documents and does not focus on what
62		-really matter (running software). This way of thinking caused to us a
63		-communication noise with MPOG, because they constantly tried to leverage on our
64		-work style. It was especially hard to convince them to accept the idea of open
65		-scope and agile development, but after months of labor and showing results they
66		-stopped resisting.
67		-
68		-We defined some level of meeting granularity to avoid to generate overheads to
69		-the developers. We had a strategical and validating meeting with MPOG (the
70		-former once in a month and the latter each 15th day), release plaining with the
71		-entire team (one per month), and finally a sprint planning (one each 15th day).
72		-Figure \ref{fig:meeting} is a diagram that represents our meeting organization.
73		-
74		-\begin{figure}[hbt]
75		- \centering
76		- \includegraphics[width=\linewidth]{figures/meeting_flows.png}
77		- \caption{Meetings cycles}
78		- \label{fig:meeting}
79		-\end{figure}
80		-
81		-In the strategical meeting we usually defined the priorities and new features
82		-with MPOG (we always had to negotiate next steps with them). Normally the
83		-professors, the coach of each team, the meta-coach, and some employees of MPOG
84		-join in this meeting. We usually discussed what the team already produced since
85		-our last meeting, and we establish the new features for the next release.
86		-Notice that just part of the team join in this meeting to avoid generating
87		-unnecessary overhead to the developers, but all the students interested to
88		-participate was allowed to join (many students wanted this experience during
89		-the project).
90		-
91		-After the strategical meeting with MPOG, we had a planning turn with all teams
92		-together. In this part, each team worked together to convert the MPOG wishes
93		-into small parts which was represented by the epics of the release. Each coach
94		-was responsible for conducting the planning, and after that register it on the
95		-project wiki (the wiki provided by Gitlab). With this epic, each 14th day the
96		-team have generated their sprint scheduler (with small achievements mapped in
97		-issues).
98		-
99		-To keep MPOG always updated, we invited them to work with us to validate the
100		-new features in progress. Normally we had a meeting each 15th day. Basically,
101		-this was our work flow, we always kept everything extremely open to the MPOG
102		-(our way of work and open source projects) and to the team.
103		-
104		-To keep the track of all of those things we used the SPB, especially the
105		-Gitlab. Basically, we had:
106		-
107		-\begin{enumerate}
108		- \item Project repository: We have one organization with many repositories
109		- \item Milestones: Each milestone is used to register a release
110		- \item Wiki: Each release has one page on wiki with the compilation of
111		- strategical meeting
112		- \item Issues: Each sprint planning generated issues, which we associated to
113		- the specific milestone and updated the wiki with the issue number related
114		- with them. Finally each developer assigned the issue to itself.
115		-\end{enumerate}
116		-
117		-Notice that this workflow gave to us and to the MPOG a full traceability from
118		-high view of the feature to the low view (code). This provided a way to MPOG
119		-validated all worked done and proof the concept that work with open source
120		-project can give a proper view to them check.
121		-
122		-\subsection{Tools for communication and management}
123		-
124		-Our team had many people worked together, and most of the seniors worked in a
125		-different city remotely. Also, we tried to keep our work completely clear to
126		-the Brazilian government and citizens interested in follow the project. To
127		-handle those cases, we used a set of tools to communication and other to manage
128		-the project.
129		-
130		-For communication between member in different places, we used: google-talk with
131		-tmate, IRC, and mailing-list. When one student had to work in pair with a
132		-senior, normally, they used google-hangout for communication and they shared a
133		-session with tmate which allow them to share the same terminal. For questions
134		-and fast discussion, we used IRC. For general notification, we used the
135		-mailing-list.
136		-
137		-For managing the project we used the SPB Portal to validate it by ourselves and
138		-because it had all the required tools. We basically create one wiki page per
139		-release in Gitlab, one milestone per sprint, and one or more issues for address
140		-one user history. With this approach we achieve two important things: keep all
141		-the management close to the source code and tracked every feature developed by
142		-the project.
143		-
144		-% Ainda falta adicionar a parte da visita dos seniors e o turno sagrado
...	...	@@ -1,159 +0,0 @@
1		-\section{Architecture}
2		-\label{sec:architecture}
3		-
4		-Based on the great list of functional requirements desired by Brazilian Federal
5		-Government we decided to select some FOSS systems that already contemplate some
6		-of them and improve these systems. And bringing the idea of community, it is
7		-undoable build a platform to be used by communities, which is a complex
8		-scenario, using just one tool.
9		-
10		-At the point of view of the architecture, two main requirements was brought by
11		-the Brazilian Federal Government for the new platform:
12		-
13		-\begin{enumerate}
14		- \item \textit{Integrate existing FOSS systems}, with minimal differences from
15		- their original versions;
16		- \item \textit{Provide a consistent user interface} across the different
17		- systems, as well as centralized authentication.
18		-\end{enumerate}
19		-
20		-Adopting existing FOSS systems by the government could bring the benefit of
21		-improvements done by the upstream communities, and the maintenance effort. On
22		-the other hand, integrating different tools with distinct intent, it is not an
23		-easy task and it was important to have a consistent user interface which
24		-justifies the last requirement.
25		-
26		-For the first requirement, we identified four main systems that required
27		-specialized teams for work in the integration process. The teams learned how to
28		-develop for their assigned systems and contributed to the original
29		-communities, so that the version we used were not significantly different from
30		-the original. Unfortunately, the deadlines at the end of the project forced us
31		-to use our own version before the features were fully reviewed and integrated
32		-into the upstream of the project.
33		-
34		-At the end of the project, SPB portal was composed of more than ten systems
35		-among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix,
36		-Munin, and so forth. The following sections explained a little bit of Colab,
37		-Noosfero, Mezuro and Gitlab (the main tools which we contributed). Later, we
38		-described how we integrated those tools and conclude with the deployment.
39		-
40		-\subsection{Colab}
41		-
42		-Colab integrates web applications as its main goal, the user of this composed
43		-system should not notice the change between the integrated applications. Colab
44		-was designed to use one plugin for each system under its domain, this is
45		-guaranteed by four levels of integration:
46		-
47		-\begin{itemize}
48		- \item Authentication
49		- \item Visual
50		- \item Events
51		- \item Data and search engine
52		-\end{itemize}
53		-
54		-The aforementioned integrations levels were possible, because Colab works as a
55		-reverse proxy, therefore all external requests pass through it.
56		-
57		-Single Sign-On (SSO) is used to login users throughout all integrated
58		-applications. REMOTE\_USER HTTP header is sent to the applications and we
59		-expect that they know how to handle it, since this is a common authentication
60		-mechanism. The integrated applications should be on a local network to avoid
61		-security issues. With this the user will be able to navigate through the
62		-platform applications and will not be asked for authentication credentials.
63		-
64		-As Colab is a reverse proxy, it makes some HTML transformations in the HTTP
65		-response of integrated applications to provide a unified interface. Allows one
66		-to define some default templates to be used by all applications and overwrite
67		-when needed in its plugin. This approach allowed us to reuse some HTML pages
68		-which facilitates maintenance.
69		-
70		-A publish-subscribe implementation was used to handle events in the platform.
71		-Thus, if some application want to trigger some action in case that other
72		-application do something is possible. A registration of the desired events and
73		-implementation of the handlers must be done in the plugin of each application.
74		-This bring us many options to innovate in these integrations.
75		-
76		-A integrated search engine is provided by Colab. Each plugin specify which data
77		-will be indexed and will became available for the users, just need an simple
78		-implementation of how should perform the collection of data.
79		-
80		-Initially, Colab had support for a small set of applications (Trac, GNU
81		-Mailman, Apache Lucene) and all of them was hard-coded; our team evolved Colab
82		-so that it can now receive plugins to add support for new applications with
83		-minimal changes to its existing core. We developed plugins to be used in the
84		-SPB platform: Noosfero, GitLab, and Mezuro.
85		-
86		-\subsection{Noosfero}
87		-
88		-Noosfero\footnote{\url{http://noosfero.org/}} is a software for building
89		-social and collaboration networks. Besides the classical social
90		-networking features, it also provides publication features such as blogs
91		-and a general-purpose CMS (Content Management System). Most of the user
92		-interactions with SPB is through Noosfero: user registration, project
93		-home pages and documentation, and contact forms.
94		-
95		-\subsection{Gitlab and Mezuro}
96		-
97		-GitLab\footnote{\url{http://gitlab.com/}} is a web-based Git repository
98		-manager with wiki pages and issue tracking features.
99		-Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code
100		-metric to monitor the internal quality of softwares written in C, C++,
101		-Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists.
102		-
103		-\subsection{System unification}
104		-
105		-\begin{figure}[hbt]
106		- \centering
107		- \includegraphics[width=\linewidth]{figures/arch.png}
108		- \caption{SPB architecture overview.}
109		- \label{fig:architecture}
110		-\end{figure}
111		-
112		-The conceptual architecture of the platform is presented in Figure
113		-\ref{fig:architecture}. Colab initially handles all user interaction,
114		-directing requests to one of the integrated applications. It
115		-post-processes responses from the applications to apply a consistent
116		-visual appearance, manages authentication, and provides a unified search
117		-functionality: instead of using the redundant restricted search
118		-functionality of each application, a search in the SPB portal might
119		-return content from any of the applications, be it web pages, mailing
120		-list posts, or source code.
121		-
122		-% Falar do devops
123		-\subsection{Deploy}
124		-
125		-The SPB platform was deployed in 7 virtual machines with different functions,
126		-as we can see in Figure \ref{fig:architecture2}.
127		-
128		-\begin{figure*}[hbt]
129		- \centering
130		- \includegraphics[width=.8\linewidth]{figures/arch2.png}
131		- \caption{Instanciation view of the SPB architecture.}
132		- \label{fig:architecture2}
133		-\end{figure*}
134		-
135		-The \textit{reverseproxy} handles the HTTP requests and redirects them to the
136		-\textit{integration}, the \textit{email} sends and receives e-mails on behalf
137		-of the platform and the \textit{monitor} keeps the entire environment tracked.
138		-These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and
139		-\textit{monitor} - are accessible via Internet and the other ones are only
140		-available in the local network created between them.
141		-
142		-\textit{Integration} works as a second layer of proxy beneath
143		-\textit{reverseproxy}, any request to the platform will be handled by it. The
144		-Colab service provides interface, authentication and search engine integration
145		-among all the services. When a request is received to a specific service,
146		-Colab authenticates the user in the target tool, sends the request and makes a
147		-visual transformation in the HTML page which is the content of the response.
148		-Another user-oriented feature is the integrated search engine, when the user
149		-want to find something in the platform Colab will perform the search in the
150		-whole databases. Colab itself provides a web interface for GNU Mailman and we
151		-have two others integrated tools in \textit{integration}: Gitlab and Prezento.
152		-Gitlab provides web interface for Git repositories and issues tracker, and
153		-Prezento is a front-end for source code static analysis.
154		-
155		-The source code static analysis is performed by \textit{mezuro}. It runs some
156		-static analysis tools on source code stored in repository and provide this data
157		-to Prezento. A social network and CMS (Content Manager System) is provided by
158		-Noosfero in \textit{social}, and the databases of all tools with a cache
159		-service are in \textit{database}.
...	...	@@ -0,0 +1,23 @@
	1	+\section{Features}
	2	+\label{sec:spb}
	3	+
	4	+%TODO: Paulo e Melissa
	5	+
	6	+The new generation of the SPB portal is made up of a combination of collaborative softwares features. Some of these features were expanded or limited according to government requirements. Other required features, unavailable in these softwares, were developed and sent back to their official repository as contributions. As result, we have a platform which integrate and harmonize features like social networking, mailing list, version control system, content management and source code quality monitoring.
	7	+
	8	+Our aim was to develop features reusing functions available in the collaborative softwares which integrate the platform. In addition, we treated to remaining this integration transparent to end users.
	9	+
	10	+\subsection{Software, development tools and community}
	11	+In the new SPB portal, each software has a standard set of pages and tools. Besides the pages with help content and user support, a software has a section of download and version control and several mechanisms to manager the software development within the platform. These mechanisms includes git repository, wiki pages and issues tracking.
	12	+
	13	+Focusing on the collaborative development, each software has pages and tools to support the dialog and communication between developers, users and enthusiasts. The software has its own mailing list, where its privacy can be moderated according to the maturity of the software in this aspect. It also has a related community, where you can find its members, a blog or any other relevant content made by the community and a summary of recent activities in the spaces of interaction of this software.
	14	+
	15	+In order to assist decision-making, the new SPB has acquired assessment and statistical tools. Now, the users can rate the software using a score and comment, so this information is displayed to anyone. Moreover, the software dispose a statistic section containing values which are calculated through data provided by user and system.
	16	+
	17	+The rule of administrator exist in the software and in its community. The administrator is responsible to moderate content to be published, members and comments from user rating. He is also the person who can make changes in the software homepage content.
	18	+
	19	+\subsection{Software Catalog and global search}
	20	+
	21	+Some available softwares on the platform are highlighted in the portal homepage, but the best way to index all softwares are by a searching in the Software Catalog. In this catalog, some search options like filters by kind of software or categories, sorting and score were developed to improve the navigation.
	22	+
	23	+Expanding the scope of search to cover other kinds of content, the SPB team developed the global search tool. This tool integrates the search mechanisms provided by the collaborative softwares that make up the platform. Any user can find a public content in the context of social network, mailing list and software development.
0	24	\ No newline at end of file
...	...
...	...	@@ -1,23 +0,0 @@
1		-\section{Features}
2		-\label{sec:spb}
3		-
4		-%TODO: Paulo e Melissa
5		-
6		-The new generation of the SPB portal is made up of a combination of collaborative softwares features. Some of these features were expanded or limited according to government requirements. Other required features, unavailable in these softwares, were developed and sent back to their official repository as contributions. As result, we have a platform which integrate and harmonize features like social networking, mailing list, version control system, content management and source code quality monitoring.
7		-
8		-Our aim was to develop features reusing functions available in the collaborative softwares which integrate the platform. In addition, we treated to remaining this integration transparent to end users.
9		-
10		-\subsection{Software, development tools and community}
11		-In the new SPB portal, each software has a standard set of pages and tools. Besides the pages with help content and user support, a software has a section of download and version control and several mechanisms to manager the software development within the platform. These mechanisms includes git repository, wiki pages and issues tracking.
12		-
13		-Focusing on the collaborative development, each software has pages and tools to support the dialog and communication between developers, users and enthusiasts. The software has its own mailing list, where its privacy can be moderated according to the maturity of the software in this aspect. It also has a related community, where you can find its members, a blog or any other relevant content made by the community and a summary of recent activities in the spaces of interaction of this software.
14		-
15		-In order to assist decision-making, the new SPB has acquired assessment and statistical tools. Now, the users can rate the software using a score and comment, so this information is displayed to anyone. Moreover, the software dispose a statistic section containing values which are calculated through data provided by user and system.
16		-
17		-The rule of administrator exist in the software and in its community. The administrator is responsible to moderate content to be published, members and comments from user rating. He is also the person who can make changes in the software homepage content.
18		-
19		-\subsection{Software Catalog and global search}
20		-
21		-Some available softwares on the platform are highlighted in the portal homepage, but the best way to index all softwares are by a searching in the Software Catalog. In this catalog, some search options like filters by kind of software or categories, sorting and score were developed to improve the navigation.
22		-
23		-Expanding the scope of search to cover other kinds of content, the SPB team developed the global search tool. This tool integrates the search mechanisms provided by the collaborative softwares that make up the platform. Any user can find a public content in the context of social network, mailing list and software development.
24	0	\ No newline at end of file
...	...	@@ -0,0 +1,33 @@
	1	+\section{User eXperience evolution}
	2	+
	3	+The integration of collaborative environments goes beyond functional aspects.
	4	+Offering the population an unified experience across these environments has
	5	+been the key to encourage the use of the platform as it reduces the perception
	6	+of complexity. Thus, the SPB Portal information architecture was redesigned
	7	+to provide a transparent navigation and to reach users with different profiles.
	8	+A process of harmonization has been employed on the interaction models of each
	9	+tool to reduce the learning curve. At the same time, a new visual style was
	10	+created to unify the navigation experience and to comply with the guidelines of
	11	+the digital communication identity standard established by the Federal
	12	+Government.
	13	+
	14	+With the increase in system features and the addition of new tools, the
	15	+visual style has steadily evolved to keep the navigation unified. Moreover,
	16	+tools from different backgrounds, which in many cases provide similar
	17	+functionality, prompted the development of an unified interface. Some
	18	+features, such as search and user profile editing were eliminated from
	19	+the individual applications, and implemented centrally to ensure a
	20	+consistent look and feel.
	21	+
	22	+Another challenge was responsive web design. The integrated applications
	23	+had varying degrees of support for responsiveness, and the common
	24	+interface had to adapt for each individual scenario. In particular
	25	+Noosfero did not yet have a responsive design; we engaged in its
	26	+development and contributed towards that goal.
	27	+
	28	+After the initial release of the new SPB Portal in 2014, several
	29	+validations activities were implemented in 2015 and 2016. The aim was to
	30	+provide the most wanted features by casual users (such as public
	31	+servants interested in downloads and documentation) immediately, while
	32	+allowing more experienced users (such as developers) to easily drill down
	33	+to the details.
...	...
...	...	@@ -0,0 +1,144 @@
	1	+\section{Development Organization and Process}
	2	+\label{sec:process}
	3	+
	4	+The SPB team was composed of a variety of professionals with different levels and skills, where most of them were undergraduate students with major in software engineering (from 4th semester or upper).
	5	+Since the students could not dedicate many hours per week to the project, they always had the flexibility to negotiate their work schedule during the semester in order not to cause any damage to their grades. Their daily work routine in the project included programming and devops tasks.
	6	+
	7	+The development of SPB project required a vast experience and background that usually undergraduate students do not have yet. For this reason, some senior developers have joined to the project to help with hard issues and to transfer knowledge to the students. Their main task was to provide solutions for complex problems, in other words, they worked as a developer. As these professionals are very skillful and the project could not fund a full time work for them, some of them worked partially on the project. In addition, they lived in a different states spread around Brazil which led much of the communication to be made via Internet.
	8	+
	9	+Finally, the last group of actors of this project was composed of employees
	10	+formally working for the Brazilian Government, in the Ministery of Planning,
	11	+development and Management (MPOG is the Brazilian acronyms). All the project
	12	+decisions, validations, and scope definitions were made by them. As can be
	13	+seen, the project had many kinds of profiles that had to be organized and
	14	+synchronized.
	15	+
	16	+\subsection{Teams organizations}
	17	+
	18	+More than X\% of the teams was composed of undergraduate students and they
	19	+worked physically in the same laboratory in the opposite of the senior. Each
	20	+student had their own scheduler based on their class, it made complicated to
	21	+implement pair programming. Also, they had a different area of interests. To
	22	+cope with those diversity, we had two basic rules which guided the project
	23	+organization:
	24	+
	25	+\begin{enumerate}
	26	+ \item Classes have to be the top priority for undergraduate students;
	27	+ \item Always work in pair (locally or remotely).
	28	+\end{enumerate}
	29	+
	30	+With the aforementioned rules we divided all the project into four different
	31	+teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible
	32	+for reducing the communication problem with the other teams and help the
	33	+members to organize itself in the best way for everyone (always respecting the
	34	+work time). The coach, was a normal student working in some of the teams with
	35	+the extra duty of register the current tasks developed in the sprint and with
	36	+the responsibility to talk with other teams. One important thing to notice is
	37	+the mutability of the team and the coach, during the project many students
	38	+changed between the teams to try different areas.
	39	+
	40	+One characteristic of the teams was the presence of (at least) one senior per
	41	+team. This was essential, because hard decisions and complex problems were
	42	+usually addressed to them, this relieved the coach duty to take a complicated
	43	+technical decisions and encouraged students to be a coach. Lastly, the senior
	44	+had to respect a rule number two and work with students, this was important to
	45	+gave the undergraduate the opportunity to interact with a savvy professional in
	46	+his area and keeping the knowledge flow in the project.
	47	+
	48	+Finally, we had to add two last elements of the team organization, that was
	49	+essential for the project harmony: the meta-coach and professors. The former
	50	+was a software engineer recently graduated and which wanted to keep working on
	51	+the project, the latter were professors that orchestrated all the interactions
	52	+between all members of the project. The meta-coach usually worked in one
	53	+specific team and had the extra task of knowing the current status of all
	54	+teams. Professors and meta-coaches worked together to reduce the communication
	55	+problem between all the teams. Lastly, all the bureaucracy tasks was
	56	+centralized in the professors.
	57	+
	58	+\subsection{Meetings}
	59	+
	60	+Brazilian government used to work with software development in a very
	61	+traditional way, frequently they claim on documents and does not focus on what
	62	+really matter (running software). This way of thinking caused to us a
	63	+communication noise with MPOG, because they constantly tried to leverage on our
	64	+work style. It was especially hard to convince them to accept the idea of open
	65	+scope and agile development, but after months of labor and showing results they
	66	+stopped resisting.
	67	+
	68	+We defined some level of meeting granularity to avoid to generate overheads to
	69	+the developers. We had a strategical and validating meeting with MPOG (the
	70	+former once in a month and the latter each 15th day), release plaining with the
	71	+entire team (one per month), and finally a sprint planning (one each 15th day).
	72	+Figure \ref{fig:meeting} is a diagram that represents our meeting organization.
	73	+
	74	+\begin{figure}[hbt]
	75	+ \centering
	76	+ \includegraphics[width=\linewidth]{figures/meeting_flows.png}
	77	+ \caption{Meetings cycles}
	78	+ \label{fig:meeting}
	79	+\end{figure}
	80	+
	81	+In the strategical meeting we usually defined the priorities and new features
	82	+with MPOG (we always had to negotiate next steps with them). Normally the
	83	+professors, the coach of each team, the meta-coach, and some employees of MPOG
	84	+join in this meeting. We usually discussed what the team already produced since
	85	+our last meeting, and we establish the new features for the next release.
	86	+Notice that just part of the team join in this meeting to avoid generating
	87	+unnecessary overhead to the developers, but all the students interested to
	88	+participate was allowed to join (many students wanted this experience during
	89	+the project).
	90	+
	91	+After the strategical meeting with MPOG, we had a planning turn with all teams
	92	+together. In this part, each team worked together to convert the MPOG wishes
	93	+into small parts which was represented by the epics of the release. Each coach
	94	+was responsible for conducting the planning, and after that register it on the
	95	+project wiki (the wiki provided by Gitlab). With this epic, each 14th day the
	96	+team have generated their sprint scheduler (with small achievements mapped in
	97	+issues).
	98	+
	99	+To keep MPOG always updated, we invited them to work with us to validate the
	100	+new features in progress. Normally we had a meeting each 15th day. Basically,
	101	+this was our work flow, we always kept everything extremely open to the MPOG
	102	+(our way of work and open source projects) and to the team.
	103	+
	104	+To keep the track of all of those things we used the SPB, especially the
	105	+Gitlab. Basically, we had:
	106	+
	107	+\begin{enumerate}
	108	+ \item Project repository: We have one organization with many repositories
	109	+ \item Milestones: Each milestone is used to register a release
	110	+ \item Wiki: Each release has one page on wiki with the compilation of
	111	+ strategical meeting
	112	+ \item Issues: Each sprint planning generated issues, which we associated to
	113	+ the specific milestone and updated the wiki with the issue number related
	114	+ with them. Finally each developer assigned the issue to itself.
	115	+\end{enumerate}
	116	+
	117	+Notice that this workflow gave to us and to the MPOG a full traceability from
	118	+high view of the feature to the low view (code). This provided a way to MPOG
	119	+validated all worked done and proof the concept that work with open source
	120	+project can give a proper view to them check.
	121	+
	122	+\subsection{Tools for communication and management}
	123	+
	124	+Our team had many people worked together, and most of the seniors worked in a
	125	+different city remotely. Also, we tried to keep our work completely clear to
	126	+the Brazilian government and citizens interested in follow the project. To
	127	+handle those cases, we used a set of tools to communication and other to manage
	128	+the project.
	129	+
	130	+For communication between member in different places, we used: google-talk with
	131	+tmate, IRC, and mailing-list. When one student had to work in pair with a
	132	+senior, normally, they used google-hangout for communication and they shared a
	133	+session with tmate which allow them to share the same terminal. For questions
	134	+and fast discussion, we used IRC. For general notification, we used the
	135	+mailing-list.
	136	+
	137	+For managing the project we used the SPB Portal to validate it by ourselves and
	138	+because it had all the required tools. We basically create one wiki page per
	139	+release in Gitlab, one milestone per sprint, and one or more issues for address
	140	+one user history. With this approach we achieve two important things: keep all
	141	+the management close to the source code and tracked every feature developed by
	142	+the project.
	143	+
	144	+% Ainda falta adicionar a parte da visita dos seniors e o turno sagrado
...	...
...	...	@@ -1,33 +0,0 @@
1		-\section{User eXperience evolution}
2		-
3		-The integration of collaborative environments goes beyond functional aspects.
4		-Offering the population an unified experience across these environments has
5		-been the key to encourage the use of the platform as it reduces the perception
6		-of complexity. Thus, the SPB Portal information architecture was redesigned
7		-to provide a transparent navigation and to reach users with different profiles.
8		-A process of harmonization has been employed on the interaction models of each
9		-tool to reduce the learning curve. At the same time, a new visual style was
10		-created to unify the navigation experience and to comply with the guidelines of
11		-the digital communication identity standard established by the Federal
12		-Government.
13		-
14		-With the increase in system features and the addition of new tools, the
15		-visual style has steadily evolved to keep the navigation unified. Moreover,
16		-tools from different backgrounds, which in many cases provide similar
17		-functionality, prompted the development of an unified interface. Some
18		-features, such as search and user profile editing were eliminated from
19		-the individual applications, and implemented centrally to ensure a
20		-consistent look and feel.
21		-
22		-Another challenge was responsive web design. The integrated applications
23		-had varying degrees of support for responsiveness, and the common
24		-interface had to adapt for each individual scenario. In particular
25		-Noosfero did not yet have a responsive design; we engaged in its
26		-development and contributed towards that goal.
27		-
28		-After the initial release of the new SPB Portal in 2014, several
29		-validations activities were implemented in 2015 and 2016. The aim was to
30		-provide the most wanted features by casual users (such as public
31		-servants interested in downloads and documentation) immediately, while
32		-allowing more experienced users (such as developers) to easily drill down
33		-to the details.
...	...	@@ -153,10 +153,10 @@
153	153	\input{content/01-introduction}
154	154	\input{content/02-spb}
155	155	\input{content/03-requirements}
156		-\input{content/04-process}
157		-\input{content/05-architecture}
158		-\input{content/06-features}
159		-\input{content/07-ux}
	156	+\input{content/04-architecture}
	157	+\input{content/05-features}
	158	+\input{content/06-ux}
	159	+\input{content/07-process}
160	160	\input{content/08-contributions}
161	161	\input{content/09-lessons}
162	162	\input{content/10-finals}
...	...