From f2e42ca6d92c79e79e657736fb974da5d28dfc33 Mon Sep 17 00:00:00 2001 From: Paulo Meirelles Date: Wed, 19 Jul 2017 13:36:56 -0300 Subject: [PATCH] Re-organizing sections and files --- opensym2017/content/01-introduction.tex | 15 +++++++++++++++ opensym2017/content/03-relatedworks.tex | 4 ++-- opensym2017/content/04-requirements.tex | 79 ------------------------------------------------------------------------------- opensym2017/content/04-researchdesign.tex | 14 ++++++++++++++ opensym2017/content/05-architecture.tex | 171 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- opensym2017/content/05-requirements.tex | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/content/06-architecture.tex | 171 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/content/06-features.tex | 76 ---------------------------------------------------------------------------- opensym2017/content/07-features.tex | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/content/07-ux.tex | 33 --------------------------------- opensym2017/content/08-process.tex | 181 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- opensym2017/content/08-ux.tex | 34 ++++++++++++++++++++++++++++++++++ opensym2017/content/09-contributions.tex | 51 --------------------------------------------------- opensym2017/content/09-process.tex | 181 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/content/10-contributions.tex | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/content/10-lessons.tex | 161 ----------------------------------------------------------------------------------------------------------------------------------------------------------------- opensym2017/content/11-conclusion.tex | 35 ----------------------------------- opensym2017/content/11-lessons.tex | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/content/12-conclusion.tex | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opensym2017/spb.tex | 17 +++++++++-------- 20 files changed, 828 insertions(+), 797 deletions(-) delete mode 100644 opensym2017/content/04-requirements.tex create mode 100644 opensym2017/content/04-researchdesign.tex delete mode 100644 opensym2017/content/05-architecture.tex create mode 100644 opensym2017/content/05-requirements.tex create mode 100644 opensym2017/content/06-architecture.tex delete mode 100644 opensym2017/content/06-features.tex create mode 100644 opensym2017/content/07-features.tex delete mode 100644 opensym2017/content/07-ux.tex delete mode 100644 opensym2017/content/08-process.tex create mode 100644 opensym2017/content/08-ux.tex delete mode 100644 opensym2017/content/09-contributions.tex create mode 100644 opensym2017/content/09-process.tex create mode 100644 opensym2017/content/10-contributions.tex delete mode 100644 opensym2017/content/10-lessons.tex delete mode 100644 opensym2017/content/11-conclusion.tex create mode 100644 opensym2017/content/11-lessons.tex create mode 100644 opensym2017/content/12-conclusion.tex diff --git a/opensym2017/content/01-introduction.tex b/opensym2017/content/01-introduction.tex index b39dad3..2aa2b53 100644 --- a/opensym2017/content/01-introduction.tex +++ b/opensym2017/content/01-introduction.tex @@ -88,3 +88,18 @@ the Brazilian government applying empirical software development methods. This case can help other projects to overcome similar software engineering challenges in the future, as well as to illustrate how universities can improve the real-world experience of their students by means of this kind of project. + + +The remainder of this work is organized as follows. +Section \ref{sec:spb}... +Section \ref{sec:relatedwork} enumerates a number of related works on the... +Section \ref{sec:researchdesign} presents the research design... +Section \ref{sec:requirements} reports ... +Section \ref{sec:architecture} ... +Section \ref{sec:features} ... +Section \ref{sec:ux} ... +Section \ref{sec:process} ... +Section \ref{sec:contributions} ... +Section \ref{sec:lessons} ... +Finally, Sections \ref{sec:conclusion} concludes the paper highlighting its main contributions and pointing paths to future works. + diff --git a/opensym2017/content/03-relatedworks.tex b/opensym2017/content/03-relatedworks.tex index 4f5a1c4..5215215 100644 --- a/opensym2017/content/03-relatedworks.tex +++ b/opensym2017/content/03-relatedworks.tex @@ -1,5 +1,5 @@ -\section{Related Works and Projects} -\label{sec:relatedworks} +\section{Related Work} +\label{sec:relatedwork} The new SPB platform presented in this paper is a fully integrated environment, as we can see in Figure \ref{fig:requirements}, being very diff --git a/opensym2017/content/04-requirements.tex b/opensym2017/content/04-requirements.tex deleted file mode 100644 index 4d0da46..0000000 --- a/opensym2017/content/04-requirements.tex +++ /dev/null @@ -1,79 +0,0 @@ -\section{Requirements} -\label{sec:requirements} - -In 2013, the SPB Portal had more than 600 thousand unique visitors, generating -more than 16 million page views with about 50 million hits. By evaluating only -the main projects, there were more than 15 thousand downloads and 4 thousand -messages exchanged in their forums. These data illustrates the potential of the -SPB Portal, even with several limitations in the past. - -By preparing the evolution project described in this paper, the Brazilian -government promoted 3 events to collect the requirements, in particular from -society point of view: (i) an online form to collect general ideas; (ii) a -face-to-face meeting with society in general; (iii) a workshop to review the -SPB concepts and requirements with IT stakeholders from the Brazilian -government and public organizations. - -After these 3 rounds discussing the new SPB platform, the Brazilian government -listed about 145 requirements and developed a ``mind -map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} -to guide the SPB portal evolution. In this scenario, the 10 most voted -requirements were: - -\begin{enumerate} - -\item Source code repository with public access; -\item Visit community pages without login; -\item Distributed version control system; -\item Scores of users and developers collaboration; -\item Search software by features; -\item Integration with social networks; -\item Repository for future ideas and requirements; -\item Friendly URL to access a public software community page; -\item User feedback about a public software; -\item Report of the experience about the use of a public software; - -\end{enumerate} - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/technological-requirements.png} - \caption{Technological requirements.} - \label{fig:requirements} -\end{figure} - - -There were other requirements based on the experience of the IT -stakeholders from the Brazilian government and from the Brazilian FOSS -community (that UnB and USP were representing too in this project). The -new platform would only work properly if there is a unique -authentication to use the provided tools. Additionally, a unified -interface was an important non-functional requirement to have a better -user experience in the new platform. - -At the first moment, we desired to release an initial version that could -replace the old SPB portal. For that, the first version should have -features such as: - -\begin{enumerate} - -\item An organized public software catalog; -\item Social network environment (profiles for users, software pages, and community pages); -\item Content Management Systems (CMS) features; -\item Web-based Git repository manager with wiki and issue tracking features; -\item Mailing lists and discussion forums; - -\end{enumerate} - -Other requirements were also planned during the conception phase of the -SPB evolution project, such as an integrated search engine and a -web-based source code static analysis monitor. By analyzing all of these -requirements, we proposed the technological requirements -illustrated in Figure \ref{fig:requirements} to guide the development of -the new SPB platform. In other words, we have designed the SPB evolution -project based on existing FOSS tools. However, the integration of -several existing systems that were already implemented in different -programming languages and frameworks, adding features such as a -centralized authentication, unified interface, and a search engine, as -well as, other back-end features, would require a non-trivial amount of -work. diff --git a/opensym2017/content/04-researchdesign.tex b/opensym2017/content/04-researchdesign.tex new file mode 100644 index 0000000..7c21c96 --- /dev/null +++ b/opensym2017/content/04-researchdesign.tex @@ -0,0 +1,14 @@ +\section{Research Design} +\label{sec:researchdesign} + +\begin{description} + +\item [RQ1:] \textit{Which strategy could be used to integrate several existing FOSS tools to promote the collaborative software development?} + + +\item [RQ2:] \textit{How to introduce the FOSS collaborative and agile practices to governmental development process?} + + +\item [RQ2:] \textit{How to involve undergraduate students in real-world projects, interacting with real customers from a Government?} + +\end{description} diff --git a/opensym2017/content/05-architecture.tex b/opensym2017/content/05-architecture.tex deleted file mode 100644 index 0df272a..0000000 --- a/opensym2017/content/05-architecture.tex +++ /dev/null @@ -1,171 +0,0 @@ -\section{Architecture} -\label{sec:architecture} - -Based on the extensive list of functional requirements defined by the -Brazilian Federal Government, we selected some FOSS systems to form our -solution. We looked for system that together could provide the largest -subset possible of the requirements, and were fully aware that we would -need to improve those systems in order to provide the rest. We were also -convinced that it would be impossible to provide all of those -requirements with a single tool. - -From the point of view of the architecture, two main requirements was -brought by the Brazilian Federal Government for the new platform: - -\begin{enumerate} - \item \textit{Integrate existing FOSS systems}, with minimal differences from - their original versions; - \item \textit{Provide a consistent user interface} across the different - systems, as well as centralized authentication. -\end{enumerate} - -Adopting existing FOSS systems and minimizing locally-made changes had -the objective of being able to upgrade to newer versions of the original -software, benefiting from improvements and maintenance done by the -existing project communities. Providing a consistent user interface on -top of those different tools was needed to make the transition between -the different systems seamless from the point of view of users, which -would be confused by jumping through completely different interfaces -while interacting with the portal. - -For the first requirement, we identified four main systems that required -specialized teams for work in the integration process. The teams learned -how to develop for their assigned systems and contributed to the -original communities, so that the version we used were not significantly -different from the original. - -At the end of the project, the SPB portal was composed of more than ten -systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and -Munin. The remainder of this section describes the most relevant of -them, as well as how they were integrated into the platform. - -\subsection{Colab} - -Colab\footnote{\url{https://github.com/colab}} is a systems integration -platform for web applications. One of its goals is allowing different -applications to be combined in such a way that an user does not notice -the change between applications. For that, Colab provides facilities -for: - -\begin{itemize} - \item \textit{Centralized authentication}: Users are automatically -identified for each application without having to enter login -credentials on each of them; - \item \textit{Visual consistency}: Colab is able to change the HTML -code produced by the integrated applications, such as defining default -template or reusing common HTML pages, in order to provide a unified -interface; - \item \textit{Relaying of events between applications}: The -integrated applications or Colab itself are able to trigger an action -on another application; - \item \textit{Integrated search engine}: It is possible to set up -exactly which data needs to be indexed from each application and how -users are directed to this information on correspondent application; -\end{itemize} - -Colab implements this integration by working as a reverse proxy for the -integration applications, that is, all external requests pass through -Colab before reaching them. - -Initially, Colab had support for a small set of applications (Trac, GNU -Mailman, and Apache Lucene) and support for them was hard-coded. Our -team evolved Colab so that it can now receive plugins that add support -new applications without the need of changes to the Colab core. - -\subsection{Noosfero} - -Noosfero\footnote{\url{http://noosfero.org}} is a software for building -social and collaboration networks. Besides the classical social -networking features, it also provides publication features such as blogs -and a general-purpose CMS (Content Management System). Most of the user -interactions with SPB is through Noosfero: user registration, project -home pages and documentation, and contact forms. - -\subsection{Gitlab} - -GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository -manager with wiki pages and issue tracking features. Gitlab is a FOSS -platform and focuses on delivering a holistic solution that will see -developers from idea to production seamlessly and on a single platform. - -Gitlab has several unique features, such as built-in continuous -integration and continuous deployment, flexible permissions, tracking of -Work-in-Progress work, moving issues between projects, group-level -milestones, creating new branches from issues, issues board, and time -tracking. - -\subsection{Mezuro} - -Mezuro\footnote{\url{http://mezuro.org/}} \cite{mezuro_oss} is a platform to -collect source code metrics to monitor the internal quality of software written -in C, C++, Java, Python, Ruby, and PHP. - -In general, source code metrics tools also do not present a friendly way -to interpret its results and, even more, do not follow a standardization -between them. Mezuro collects and presents these results to the end -user, specially, by analyzing source code metric history during its life -cycle. - -The Mezuro platform provides a single interface grouping available -tools, allows selection and composition of metrics in a flexible manner, -stores the metrics evolution history, presents results in a friendly -way, as well as, allows users to customize the given interpretation -accordingly to their own context. - -\subsection{System unification} - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/arch.png} - \caption{SPB architecture overview.} - \label{fig:architecture} -\end{figure} - -The conceptual architecture of the platform is presented in Figure -\ref{fig:architecture}. Colab initially handles all user interaction, -directing requests to one of the integrated applications. It -post-processes responses from the applications to apply a consistent -visual appearance, manages authentication, and provides a unified search -functionality: instead of using the redundant restricted search -functionality of each application, a search o n the SPB portal might -return content from any of the applications, be it web pages, mailing -list posts, or source code. - -% Falar do devops -\subsection{Deploy} - -The SPB platform was deployed in 7 virtual machines with different functions, -as we can see in Figure \ref{fig:architecture2}. - -\begin{figure*}[hbt] - \centering - \includegraphics[width=.8\linewidth]{figures/arch3.png} - \caption{Instanciation view of the SPB architecture.} - \label{fig:architecture2} -\end{figure*} - -The \textit{reverseproxy} handles the HTTP requests and redirects them to the -\textit{integration}, the \textit{email} sends and receives e-mails on behalf -of the platform and the \textit{monitor} keeps the entire environment tracked. -These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and -\textit{monitor} - are accessible via Internet and the other ones are only -available in the local network created between them. - -\textit{Integration} works as a second layer of proxy beneath -\textit{reverseproxy}, any request to the platform will be handled by it. The -Colab service provides interface, authentication and search engine integration -among all the services. When a request is received to a specific service, -Colab authenticates the user in the target tool, sends the request and makes a -visual transformation in the HTML page which is the content of the response. -Another user-oriented feature is the integrated search engine, when the user -want to find something in the platform Colab will perform the search in the -whole databases. Colab itself provides a web interface for GNU Mailman and we -have two others integrated tools in \textit{integration}: Gitlab and Prezento. -Gitlab provides web interface for Git repositories and issues tracker, and -Prezento is a front-end for source code static analysis. - -The source code static analysis is performed by \textit{mezuro}. It runs some -static analysis tools on source code stored in repository and provide this data -to Prezento. A social network and CMS (Content Manager System) is provided by -Noosfero in \textit{social}, and the databases of all tools with a cache -service are in \textit{database}. diff --git a/opensym2017/content/05-requirements.tex b/opensym2017/content/05-requirements.tex new file mode 100644 index 0000000..4d0da46 --- /dev/null +++ b/opensym2017/content/05-requirements.tex @@ -0,0 +1,79 @@ +\section{Requirements} +\label{sec:requirements} + +In 2013, the SPB Portal had more than 600 thousand unique visitors, generating +more than 16 million page views with about 50 million hits. By evaluating only +the main projects, there were more than 15 thousand downloads and 4 thousand +messages exchanged in their forums. These data illustrates the potential of the +SPB Portal, even with several limitations in the past. + +By preparing the evolution project described in this paper, the Brazilian +government promoted 3 events to collect the requirements, in particular from +society point of view: (i) an online form to collect general ideas; (ii) a +face-to-face meeting with society in general; (iii) a workshop to review the +SPB concepts and requirements with IT stakeholders from the Brazilian +government and public organizations. + +After these 3 rounds discussing the new SPB platform, the Brazilian government +listed about 145 requirements and developed a ``mind +map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} +to guide the SPB portal evolution. In this scenario, the 10 most voted +requirements were: + +\begin{enumerate} + +\item Source code repository with public access; +\item Visit community pages without login; +\item Distributed version control system; +\item Scores of users and developers collaboration; +\item Search software by features; +\item Integration with social networks; +\item Repository for future ideas and requirements; +\item Friendly URL to access a public software community page; +\item User feedback about a public software; +\item Report of the experience about the use of a public software; + +\end{enumerate} + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/technological-requirements.png} + \caption{Technological requirements.} + \label{fig:requirements} +\end{figure} + + +There were other requirements based on the experience of the IT +stakeholders from the Brazilian government and from the Brazilian FOSS +community (that UnB and USP were representing too in this project). The +new platform would only work properly if there is a unique +authentication to use the provided tools. Additionally, a unified +interface was an important non-functional requirement to have a better +user experience in the new platform. + +At the first moment, we desired to release an initial version that could +replace the old SPB portal. For that, the first version should have +features such as: + +\begin{enumerate} + +\item An organized public software catalog; +\item Social network environment (profiles for users, software pages, and community pages); +\item Content Management Systems (CMS) features; +\item Web-based Git repository manager with wiki and issue tracking features; +\item Mailing lists and discussion forums; + +\end{enumerate} + +Other requirements were also planned during the conception phase of the +SPB evolution project, such as an integrated search engine and a +web-based source code static analysis monitor. By analyzing all of these +requirements, we proposed the technological requirements +illustrated in Figure \ref{fig:requirements} to guide the development of +the new SPB platform. In other words, we have designed the SPB evolution +project based on existing FOSS tools. However, the integration of +several existing systems that were already implemented in different +programming languages and frameworks, adding features such as a +centralized authentication, unified interface, and a search engine, as +well as, other back-end features, would require a non-trivial amount of +work. diff --git a/opensym2017/content/06-architecture.tex b/opensym2017/content/06-architecture.tex new file mode 100644 index 0000000..0df272a --- /dev/null +++ b/opensym2017/content/06-architecture.tex @@ -0,0 +1,171 @@ +\section{Architecture} +\label{sec:architecture} + +Based on the extensive list of functional requirements defined by the +Brazilian Federal Government, we selected some FOSS systems to form our +solution. We looked for system that together could provide the largest +subset possible of the requirements, and were fully aware that we would +need to improve those systems in order to provide the rest. We were also +convinced that it would be impossible to provide all of those +requirements with a single tool. + +From the point of view of the architecture, two main requirements was +brought by the Brazilian Federal Government for the new platform: + +\begin{enumerate} + \item \textit{Integrate existing FOSS systems}, with minimal differences from + their original versions; + \item \textit{Provide a consistent user interface} across the different + systems, as well as centralized authentication. +\end{enumerate} + +Adopting existing FOSS systems and minimizing locally-made changes had +the objective of being able to upgrade to newer versions of the original +software, benefiting from improvements and maintenance done by the +existing project communities. Providing a consistent user interface on +top of those different tools was needed to make the transition between +the different systems seamless from the point of view of users, which +would be confused by jumping through completely different interfaces +while interacting with the portal. + +For the first requirement, we identified four main systems that required +specialized teams for work in the integration process. The teams learned +how to develop for their assigned systems and contributed to the +original communities, so that the version we used were not significantly +different from the original. + +At the end of the project, the SPB portal was composed of more than ten +systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and +Munin. The remainder of this section describes the most relevant of +them, as well as how they were integrated into the platform. + +\subsection{Colab} + +Colab\footnote{\url{https://github.com/colab}} is a systems integration +platform for web applications. One of its goals is allowing different +applications to be combined in such a way that an user does not notice +the change between applications. For that, Colab provides facilities +for: + +\begin{itemize} + \item \textit{Centralized authentication}: Users are automatically +identified for each application without having to enter login +credentials on each of them; + \item \textit{Visual consistency}: Colab is able to change the HTML +code produced by the integrated applications, such as defining default +template or reusing common HTML pages, in order to provide a unified +interface; + \item \textit{Relaying of events between applications}: The +integrated applications or Colab itself are able to trigger an action +on another application; + \item \textit{Integrated search engine}: It is possible to set up +exactly which data needs to be indexed from each application and how +users are directed to this information on correspondent application; +\end{itemize} + +Colab implements this integration by working as a reverse proxy for the +integration applications, that is, all external requests pass through +Colab before reaching them. + +Initially, Colab had support for a small set of applications (Trac, GNU +Mailman, and Apache Lucene) and support for them was hard-coded. Our +team evolved Colab so that it can now receive plugins that add support +new applications without the need of changes to the Colab core. + +\subsection{Noosfero} + +Noosfero\footnote{\url{http://noosfero.org}} is a software for building +social and collaboration networks. Besides the classical social +networking features, it also provides publication features such as blogs +and a general-purpose CMS (Content Management System). Most of the user +interactions with SPB is through Noosfero: user registration, project +home pages and documentation, and contact forms. + +\subsection{Gitlab} + +GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository +manager with wiki pages and issue tracking features. Gitlab is a FOSS +platform and focuses on delivering a holistic solution that will see +developers from idea to production seamlessly and on a single platform. + +Gitlab has several unique features, such as built-in continuous +integration and continuous deployment, flexible permissions, tracking of +Work-in-Progress work, moving issues between projects, group-level +milestones, creating new branches from issues, issues board, and time +tracking. + +\subsection{Mezuro} + +Mezuro\footnote{\url{http://mezuro.org/}} \cite{mezuro_oss} is a platform to +collect source code metrics to monitor the internal quality of software written +in C, C++, Java, Python, Ruby, and PHP. + +In general, source code metrics tools also do not present a friendly way +to interpret its results and, even more, do not follow a standardization +between them. Mezuro collects and presents these results to the end +user, specially, by analyzing source code metric history during its life +cycle. + +The Mezuro platform provides a single interface grouping available +tools, allows selection and composition of metrics in a flexible manner, +stores the metrics evolution history, presents results in a friendly +way, as well as, allows users to customize the given interpretation +accordingly to their own context. + +\subsection{System unification} + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/arch.png} + \caption{SPB architecture overview.} + \label{fig:architecture} +\end{figure} + +The conceptual architecture of the platform is presented in Figure +\ref{fig:architecture}. Colab initially handles all user interaction, +directing requests to one of the integrated applications. It +post-processes responses from the applications to apply a consistent +visual appearance, manages authentication, and provides a unified search +functionality: instead of using the redundant restricted search +functionality of each application, a search o n the SPB portal might +return content from any of the applications, be it web pages, mailing +list posts, or source code. + +% Falar do devops +\subsection{Deploy} + +The SPB platform was deployed in 7 virtual machines with different functions, +as we can see in Figure \ref{fig:architecture2}. + +\begin{figure*}[hbt] + \centering + \includegraphics[width=.8\linewidth]{figures/arch3.png} + \caption{Instanciation view of the SPB architecture.} + \label{fig:architecture2} +\end{figure*} + +The \textit{reverseproxy} handles the HTTP requests and redirects them to the +\textit{integration}, the \textit{email} sends and receives e-mails on behalf +of the platform and the \textit{monitor} keeps the entire environment tracked. +These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and +\textit{monitor} - are accessible via Internet and the other ones are only +available in the local network created between them. + +\textit{Integration} works as a second layer of proxy beneath +\textit{reverseproxy}, any request to the platform will be handled by it. The +Colab service provides interface, authentication and search engine integration +among all the services. When a request is received to a specific service, +Colab authenticates the user in the target tool, sends the request and makes a +visual transformation in the HTML page which is the content of the response. +Another user-oriented feature is the integrated search engine, when the user +want to find something in the platform Colab will perform the search in the +whole databases. Colab itself provides a web interface for GNU Mailman and we +have two others integrated tools in \textit{integration}: Gitlab and Prezento. +Gitlab provides web interface for Git repositories and issues tracker, and +Prezento is a front-end for source code static analysis. + +The source code static analysis is performed by \textit{mezuro}. It runs some +static analysis tools on source code stored in repository and provide this data +to Prezento. A social network and CMS (Content Manager System) is provided by +Noosfero in \textit{social}, and the databases of all tools with a cache +service are in \textit{database}. diff --git a/opensym2017/content/06-features.tex b/opensym2017/content/06-features.tex deleted file mode 100644 index 8291cc7..0000000 --- a/opensym2017/content/06-features.tex +++ /dev/null @@ -1,76 +0,0 @@ -\section{Features} -\label{sec:spb} - -The new generation of the SPB portal combines features adapted from existing collaborative software -and features developed by the SPB team. Whenever possible, new functions -(newly developed or partially modified) were contributed back to the official repositories. - -As a result, we have a platform that integrates and harmonizes different features such as -social networking, mailing list, version control system, content management and -source code quality monitoring. Our aim was to develop functionalities by reusing -functionality of collaborative software already integrated to the platform. In -addition, we tried to keep this integration transparent to end users. - -\subsection{Software and Software Community} - -In the new SPB portal, each software has a standard set of pages and tools. -Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download -different versions of the software and find several mechanisms of software development management. - -Focusing on the collaborative development, Mailman was integrated to the platform in order to allow -the dialogue and communication between developers, users and -enthusiasts of a determined software. The software has its own mailing list whose privacy -can be configured/set by administrators. - -The software has a social interface area ("software community") where users can find other users, blogs, -summary of recent activities, or any other relevant community-produced content. -Users logged to the platform can request membership to different software communities -and each community member can access and edit restricted content. For this purpose, -many Noosfero features related to social networking and content management were integrated to the portal. - -To assist decision-making, the new SPB has acquired assessment and statistical -tools. Now, users will be able to rate the software and make comments and all -information will be avaiable to other users. Moreover, the software has a section -containing its statistical data, where values are calculated based on data -provided by users and the system. - -The role of the administrator will be present in the software and in its community. The -administrator is responsible for moderating content, memberships and user -comments. The administrator is also the one who can make changes in the software homepage -content. - -\subsection{Software Catalog and global search} - -The platform also provides a search tool called Software Catalog, -which allows users to find softwares available in the portal. -In this catalog, some search options were developed to make the navigation easier, -such as filters (by type of software or category), sorting and score. - -In order to expand the searching scope and cover more types of content, the SPB team -developed the global search tool. This tool unifies search mechanisms -provided by the different collaborative software used on the SPB portal. Any user can -find a public content in the context of social networking, mailing list and -software development. - -\subsection{Software development tools} - -Usually, Collaborative Development Environments (CDE) demand a version control -system, trackers, build tools, knowledge centers, and communication tools -\cite{collaboration_tools}. -The new SPB also provides -tools to encourage developers to keep source code and its -development activity within the platform. Any created software has, by default, a -an associated git repository with wiki pages and issue tracking. These tools are -supplied by the integration of Gitlab into the platform. - -Developers can also evaluate the software source code to measure software -quality. With Mezuro, they can schedule the analysis of the source-code and follow its -metric results evolution over time. Results of each metric analysis are -public, which allows greater transparency between the developer and the -community that uses the software. Thereby, the maintainers can decide if the -given solution meets the source code quality requirements. - -Thus, the SPB became a platform to stimulate the openness of the source code; -dialogue between users and the development team; and also -maintenance and evolution of the software, which will provide more -transparency in government investments. diff --git a/opensym2017/content/07-features.tex b/opensym2017/content/07-features.tex new file mode 100644 index 0000000..0c37166 --- /dev/null +++ b/opensym2017/content/07-features.tex @@ -0,0 +1,76 @@ +\section{Features} +\label{sec:features} + +The new generation of the SPB portal combines features adapted from existing collaborative software +and features developed by the SPB team. Whenever possible, new functions +(newly developed or partially modified) were contributed back to the official repositories. + +As a result, we have a platform that integrates and harmonizes different features such as +social networking, mailing list, version control system, content management and +source code quality monitoring. Our aim was to develop functionalities by reusing +functionality of collaborative software already integrated to the platform. In +addition, we tried to keep this integration transparent to end users. + +\subsection{Software and Software Community} + +In the new SPB portal, each software has a standard set of pages and tools. +Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download +different versions of the software and find several mechanisms of software development management. + +Focusing on the collaborative development, Mailman was integrated to the platform in order to allow +the dialogue and communication between developers, users and +enthusiasts of a determined software. The software has its own mailing list whose privacy +can be configured/set by administrators. + +The software has a social interface area ("software community") where users can find other users, blogs, +summary of recent activities, or any other relevant community-produced content. +Users logged to the platform can request membership to different software communities +and each community member can access and edit restricted content. For this purpose, +many Noosfero features related to social networking and content management were integrated to the portal. + +To assist decision-making, the new SPB has acquired assessment and statistical +tools. Now, users will be able to rate the software and make comments and all +information will be avaiable to other users. Moreover, the software has a section +containing its statistical data, where values are calculated based on data +provided by users and the system. + +The role of the administrator will be present in the software and in its community. The +administrator is responsible for moderating content, memberships and user +comments. The administrator is also the one who can make changes in the software homepage +content. + +\subsection{Software Catalog and global search} + +The platform also provides a search tool called Software Catalog, +which allows users to find softwares available in the portal. +In this catalog, some search options were developed to make the navigation easier, +such as filters (by type of software or category), sorting and score. + +In order to expand the searching scope and cover more types of content, the SPB team +developed the global search tool. This tool unifies search mechanisms +provided by the different collaborative software used on the SPB portal. Any user can +find a public content in the context of social networking, mailing list and +software development. + +\subsection{Software development tools} + +Usually, Collaborative Development Environments (CDE) demand a version control +system, trackers, build tools, knowledge centers, and communication tools +\cite{collaboration_tools}. +The new SPB also provides +tools to encourage developers to keep source code and its +development activity within the platform. Any created software has, by default, a +an associated git repository with wiki pages and issue tracking. These tools are +supplied by the integration of Gitlab into the platform. + +Developers can also evaluate the software source code to measure software +quality. With Mezuro, they can schedule the analysis of the source-code and follow its +metric results evolution over time. Results of each metric analysis are +public, which allows greater transparency between the developer and the +community that uses the software. Thereby, the maintainers can decide if the +given solution meets the source code quality requirements. + +Thus, the SPB became a platform to stimulate the openness of the source code; +dialogue between users and the development team; and also +maintenance and evolution of the software, which will provide more +transparency in government investments. diff --git a/opensym2017/content/07-ux.tex b/opensym2017/content/07-ux.tex deleted file mode 100644 index 3dc36da..0000000 --- a/opensym2017/content/07-ux.tex +++ /dev/null @@ -1,33 +0,0 @@ -\section{User eXperience evolution} - -The integration of collaborative environments goes beyond functional aspects. -Offering the population an unified experience across these environments has -been the key to encourage the use of the platform as it reduces the perception -of complexity. Thus, the SPB Portal information architecture was redesigned -to provide a transparent navigation and to reach users with different profiles. -A process of harmonization has been employed on the interaction models of each -tool to reduce the learning curve. At the same time, a new visual style was -created to unify the navigation experience and to comply with the guidelines of -the digital communication identity standard established by the Federal -Government. - -With the increase in system features and the addition of new tools, the -visual style has steadily evolved to keep the navigation unified. Moreover, -tools from different backgrounds, which in many cases provide similar -functionality, prompted the development of an unified interface. Some -features, such as search and user profile editing were eliminated from -the individual applications, and implemented centrally to ensure a -consistent look and feel. - -Another challenge was responsive web design. The integrated applications -had varying degrees of support for responsiveness, and the common -interface had to adapt for each individual scenario. In particular -Noosfero did not yet have a responsive design; we engaged in its -development and contributed towards that goal. - -After the initial release of the new SPB Portal in 2014, several -validations activities were implemented in 2015 and 2016. The aim was to -provide the most wanted features by casual users (such as public -servants interested in downloads and documentation) immediately, while -allowing more experienced users (such as developers) to easily drill down -to the details. diff --git a/opensym2017/content/08-process.tex b/opensym2017/content/08-process.tex deleted file mode 100644 index 82781d2..0000000 --- a/opensym2017/content/08-process.tex +++ /dev/null @@ -1,181 +0,0 @@ -\section{Development Organization and Process} -\label{sec:process} - -The SPB team was composed of a variety of professionals with different levels -and skills, where most of them were undergraduate students with major in -software engineering (from 4th semester or upper). Since the students could -not dedicate many hours per week to the project, they always had the -flexibility to negotiate their work schedule during the semester in -order to not harm their classes and coursework. Their daily work routine -in the project included programming and DevOps tasks. - -The development of SPB project required a vast experience and background that -usually undergraduate students do not have yet. For this reason, a few senior -developers have joined to the project to help with the more difficult issues -and to transfer knowledge to the students. Their main task was to provide -solutions for complex problems, in other words, they worked as developers. As -these professionals are very skillful and the project could not fund a full -time work for them, some of them worked partially on the project. In addition, -they lived in a different states spread around Brazil which led much of the -communication to be online. - -In short, our work process was based on open and collaborative software -development practices. The development process was defined based on the -adaptation of different agile and FOSS communities practices, highlighting the -high degree of automation resulting from DevOps practices. Thus, the work -process was executed in a cadenced and continuous way. - -Finally, the last group of actors of this project was composed of employees -formally working for the Brazilian government, in the Ministry of Planning, -Development, and Management (MP is the Brazilian acronym). All the project -decisions, validations, and scope definitions were made by them. In this way we -developed software product incrementally, with releases aligned to business -strategic objectives. As one can see, the project had many different -sort of stakeholders that had to be organized and synchronized. - -\subsection{Team organization} - -Approximately 70\% of the development teams were composed of software -engineering undergraduate students from UnB and they worked physically -in the same laboratory. Each student had their own schedule based on -their classes, what complicates the implementation of pair programming. -The senior developers tried to synchronize their schedule with the -students on their sub-team. To cope with this environment, we had a few -basic rules which guided the project organization: - -\begin{enumerate} - \item Classes have high priority for undergraduate students; - \item Work in pairs whenever possible (locally or remotely); - \item There must be one morning or afternoon per week when - \emph{everyone} should be together physically in the laboratory - (except of course the remote team members); - \item Every 3 to 4 months, the senior developers would fly in and work - alongside the students for a few days. -\end{enumerate} - -With the aforementioned rules we divided all the project into four -different teams: Colab, Noosfero, Design, and DevOps. Each team had one -coach responsible for reducing the communication problem with the other -teams and help the members to organize themselves in the best way for -everyone (always respecting their work time). The coach was one of the -students working in some of the teams with the extra duty of registering -the current tasks developed in the sprint and with the responsibility to -talk with other teams. One important thing to notice is the mutability -of the team and the coach, during the project many students changed -between the teams to try different areas. - -One characteristic of the teams was the presence of (at least) one -senior per team. This was essential, because hard decisions and complex -problems were usually referred to them. This kept having to take -complicated technical decisions from the coach tole, what encouraged -students to be coaches. Lastly, the senior developers worked directly -with the students, and this was important to give the undergraduate the -opportunity to interact with a savvy professional in his area and to -keep the knowledge flowing in the project. - -Finally, we had to add two last elements of the team organization, that -was essential for the project harmony: the meta-coach and professors. -The former was a software engineer recently graduated and which wanted -to keep working on the project, the latter were professors that -orchestrated all the interactions between all members of the project. -The meta-coach usually worked in one specific team and had the extra -task of knowing the current status of all teams. Professors and -meta-coaches worked together to reduce the communication problem between -all the teams. Lastly, all the paperwork tasks, such as reporting on the -project progress to the Ministry, was handled by the professors. - -\subsection{Communication and management} - -Our team had many people working together, and most of the seniors worked in a -different city remotely. Also, we tried to keep our work completely clear to -the Brazilian government and citizens interested in following the project. To -handle those cases, we used a set of tools to communication and other to manage -the project. - -For communication between members in different places, we used: video -conferencing with shared terminal tools, IRC, and mailing lists. For example, -when one student had to work in pair with a senior, normally, they used video -conferencing tool for talking and shared a terminal session (both typing and -seeing the screen in real time). For questions and fast discussion, we used -IRC. For general notification, we used the mailing lists. - -For managing the project we used the SPB Portal itself; first to validate it by -ourselves, and also because it had all the required tools. We basically created -one wiki page per release in the SPB Gitlab instance with a mapping between -strategical, tactical, and operational views. In a practical point of view, one -milestone per user history (feature), and one or more issues for addressing -each feature. With this approach we achieved two important goals: keeping all -the management as close possible to to the source code, and tracking every -feature developed during the project. Our decision to -use Wiki initially was empirical, but later reinforced by a research conducted -by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, -opensourcestyle}. - -\subsection{High-level project management and reporting} - -The Brazilian government used to work with software development in a -very traditional way. They would frequently focus on documents and not -on what was, in our opinion, what really matters: working software. This -dissonance caused us a communication noise with MP, because they would -often question our work style. It was especially hard to convince them -to accept the idea of open scope and agile development, but after months -of labor and showing results they stopped resisting. - -We defined some level of meeting granularity to avoid generating too -much overhead to the developers. We had a strategical and validating -meeting with MP (the former once in a month and the latter each 15th -day), release plaining with the entire team (one per month), and finally -a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a -diagram that represents our meeting organization. - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/meeting_flows.png} - \caption{Meetings cycles.} - \label{fig:meeting} -\end{figure} - -In the strategical meeting we usually defined the priorities and new -features with the Brazilian government. Normally the professors, the -coach of each team, the meta-coach, and some employees of the MP would -participate in this meeting. We usually discussed what the team already -produced since our last meeting, and established the new features for -the next release. Notice that just part of the team would join this -meeting to avoid generating unnecessary overhead to the developers, but -all the students interested to participate were allowed to join (many -students wanted this experience during the project). - -After the strategical meeting with Brazilian government agents, we had a -planning turn with all teams together. In this part, each team worked together -to convert the MP wishes into smaller parts which were represented by the epics of -the release. Each coach was responsible for conducting the planning, and after -that record it on the project wiki (the wiki provided by Gitlab). With this -epic, each 14th day the team have generated their sprint scheduler (with small -achievements mapped to issues). - -To keep the Brazilian government always updated, we invited them to work -with us to validate the new features in progress. Normally we had a -meeting each 15th day. Basically, this was our work flow, we always kept -everything extremely open to the MP (our way of working, and the one -often used by open source projects) and to the team. - -To keep the track of all of those things we used the SPB itself, -especially Gitlab. Basically, we had: - -\begin{enumerate} - \item Project repository: We have one organization with many repositories; - \item Milestones: Each milestone was used to register a release; - \item Wiki: Each release has one page on wiki with the compilation of - strategical meeting; - \item Issues: Each sprint planning generated issues, which we associated to - the specific milestone and updated the wiki with the issue number related - with them. Finally each developer assigned the issue to itself. -\end{enumerate} - -Notice that this workflow gave us and the Brazilian government agents full -traceability from a high level view of each feature to the lowest level (code). -It is important to highlight that we converge to this workflow after many -experiments. For example, we used a tool named Redmine for organizing our tasks -during the sprints, however, this tool revealed inefficient for our case since -the government agents lost part of the project traceability. We realized that -centralize all the work in the SPB portal is the best option for our case. diff --git a/opensym2017/content/08-ux.tex b/opensym2017/content/08-ux.tex new file mode 100644 index 0000000..a5e539c --- /dev/null +++ b/opensym2017/content/08-ux.tex @@ -0,0 +1,34 @@ +\section{User eXperience evolution} +\label{sec:ux} + +The integration of collaborative environments goes beyond functional aspects. +Offering the population an unified experience across these environments has +been the key to encourage the use of the platform as it reduces the perception +of complexity. Thus, the SPB Portal information architecture was redesigned +to provide a transparent navigation and to reach users with different profiles. +A process of harmonization has been employed on the interaction models of each +tool to reduce the learning curve. At the same time, a new visual style was +created to unify the navigation experience and to comply with the guidelines of +the digital communication identity standard established by the Federal +Government. + +With the increase in system features and the addition of new tools, the +visual style has steadily evolved to keep the navigation unified. Moreover, +tools from different backgrounds, which in many cases provide similar +functionality, prompted the development of an unified interface. Some +features, such as search and user profile editing were eliminated from +the individual applications, and implemented centrally to ensure a +consistent look and feel. + +Another challenge was responsive web design. The integrated applications +had varying degrees of support for responsiveness, and the common +interface had to adapt for each individual scenario. In particular +Noosfero did not yet have a responsive design; we engaged in its +development and contributed towards that goal. + +After the initial release of the new SPB Portal in 2014, several +validations activities were implemented in 2015 and 2016. The aim was to +provide the most wanted features by casual users (such as public +servants interested in downloads and documentation) immediately, while +allowing more experienced users (such as developers) to easily drill down +to the details. diff --git a/opensym2017/content/09-contributions.tex b/opensym2017/content/09-contributions.tex deleted file mode 100644 index 2801e8d..0000000 --- a/opensym2017/content/09-contributions.tex +++ /dev/null @@ -1,51 +0,0 @@ -\section{Contributions to the upstream communities} -\label{sec:contributions} - -%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) -%* Colab -> RevProxy -%* Colab, atualização do python/django -%* Contribuições para o GitLab (autenticação) -%* Noosfero, atualização do Rails, preparação para federação, nova interface ... -%* Coper, empacotamentos (obs), omniauth - - -During the execution of this project we made several contributions to -the upstream communities we interacted with. This occurred due to our -development process aligned with those of the respective communities. -During development, we would explicitly discuss the features and bug -fixes that we were working on with the applicable upstream communities. -This contributed to improve the developers technical solutions with -expertise outside of our team, and make it easier for those changes to -be accepted in the original projects. Having changes accepted upstream -in turn makes our life easier as it minimizes the delta between our -codebase and upstream's allowing us to upgrade and benefit from -development work from others. - -In Colab, we helped upstream redesign the entirely architecture, -enabling the development of plugins to integrate new tools. We also -added a feature that allowed Colab to run asynchronous tasks, which was -a major improvement for us since we were developing a complex system. A -migration to the latest Django version was made (web framework used by -Colab). Moreover, we worked on RevProxy (the more important dependency -of Colab) to put it in a good shape, fixing many bugs. - -Gitlab was the tool that we made the least number of modifications. We -contributed with some improvements related with configuration files and -we developed a new plugin that enables user authentication in Gitlab -through the REMOTE\_USER HTTP header. This plugin was needed because -Colab uses this mechanism to manage the authentication. - -Noosfero was the tool that contemplated several functional requirements, -therefore we made a large number of contributions with upstream. We helped to -migrate to the latest Rails version (web framework used by Noosfero), enable -the federation implementation (federation with other social networks), and -decouple the interface and the back-end. - -We also made some contributions on the DevOps front. Some members of -them team became maintainers of some python libraries that were used by -our scripts to upload packages to OBS (Open Build Service). We developed -a tool called copr-status to keep track of the different stages of the -lifecycle of each of the individual packages we were working on. - -%TODO: Mezuro - diff --git a/opensym2017/content/09-process.tex b/opensym2017/content/09-process.tex new file mode 100644 index 0000000..82781d2 --- /dev/null +++ b/opensym2017/content/09-process.tex @@ -0,0 +1,181 @@ +\section{Development Organization and Process} +\label{sec:process} + +The SPB team was composed of a variety of professionals with different levels +and skills, where most of them were undergraduate students with major in +software engineering (from 4th semester or upper). Since the students could +not dedicate many hours per week to the project, they always had the +flexibility to negotiate their work schedule during the semester in +order to not harm their classes and coursework. Their daily work routine +in the project included programming and DevOps tasks. + +The development of SPB project required a vast experience and background that +usually undergraduate students do not have yet. For this reason, a few senior +developers have joined to the project to help with the more difficult issues +and to transfer knowledge to the students. Their main task was to provide +solutions for complex problems, in other words, they worked as developers. As +these professionals are very skillful and the project could not fund a full +time work for them, some of them worked partially on the project. In addition, +they lived in a different states spread around Brazil which led much of the +communication to be online. + +In short, our work process was based on open and collaborative software +development practices. The development process was defined based on the +adaptation of different agile and FOSS communities practices, highlighting the +high degree of automation resulting from DevOps practices. Thus, the work +process was executed in a cadenced and continuous way. + +Finally, the last group of actors of this project was composed of employees +formally working for the Brazilian government, in the Ministry of Planning, +Development, and Management (MP is the Brazilian acronym). All the project +decisions, validations, and scope definitions were made by them. In this way we +developed software product incrementally, with releases aligned to business +strategic objectives. As one can see, the project had many different +sort of stakeholders that had to be organized and synchronized. + +\subsection{Team organization} + +Approximately 70\% of the development teams were composed of software +engineering undergraduate students from UnB and they worked physically +in the same laboratory. Each student had their own schedule based on +their classes, what complicates the implementation of pair programming. +The senior developers tried to synchronize their schedule with the +students on their sub-team. To cope with this environment, we had a few +basic rules which guided the project organization: + +\begin{enumerate} + \item Classes have high priority for undergraduate students; + \item Work in pairs whenever possible (locally or remotely); + \item There must be one morning or afternoon per week when + \emph{everyone} should be together physically in the laboratory + (except of course the remote team members); + \item Every 3 to 4 months, the senior developers would fly in and work + alongside the students for a few days. +\end{enumerate} + +With the aforementioned rules we divided all the project into four +different teams: Colab, Noosfero, Design, and DevOps. Each team had one +coach responsible for reducing the communication problem with the other +teams and help the members to organize themselves in the best way for +everyone (always respecting their work time). The coach was one of the +students working in some of the teams with the extra duty of registering +the current tasks developed in the sprint and with the responsibility to +talk with other teams. One important thing to notice is the mutability +of the team and the coach, during the project many students changed +between the teams to try different areas. + +One characteristic of the teams was the presence of (at least) one +senior per team. This was essential, because hard decisions and complex +problems were usually referred to them. This kept having to take +complicated technical decisions from the coach tole, what encouraged +students to be coaches. Lastly, the senior developers worked directly +with the students, and this was important to give the undergraduate the +opportunity to interact with a savvy professional in his area and to +keep the knowledge flowing in the project. + +Finally, we had to add two last elements of the team organization, that +was essential for the project harmony: the meta-coach and professors. +The former was a software engineer recently graduated and which wanted +to keep working on the project, the latter were professors that +orchestrated all the interactions between all members of the project. +The meta-coach usually worked in one specific team and had the extra +task of knowing the current status of all teams. Professors and +meta-coaches worked together to reduce the communication problem between +all the teams. Lastly, all the paperwork tasks, such as reporting on the +project progress to the Ministry, was handled by the professors. + +\subsection{Communication and management} + +Our team had many people working together, and most of the seniors worked in a +different city remotely. Also, we tried to keep our work completely clear to +the Brazilian government and citizens interested in following the project. To +handle those cases, we used a set of tools to communication and other to manage +the project. + +For communication between members in different places, we used: video +conferencing with shared terminal tools, IRC, and mailing lists. For example, +when one student had to work in pair with a senior, normally, they used video +conferencing tool for talking and shared a terminal session (both typing and +seeing the screen in real time). For questions and fast discussion, we used +IRC. For general notification, we used the mailing lists. + +For managing the project we used the SPB Portal itself; first to validate it by +ourselves, and also because it had all the required tools. We basically created +one wiki page per release in the SPB Gitlab instance with a mapping between +strategical, tactical, and operational views. In a practical point of view, one +milestone per user history (feature), and one or more issues for addressing +each feature. With this approach we achieved two important goals: keeping all +the management as close possible to to the source code, and tracking every +feature developed during the project. Our decision to +use Wiki initially was empirical, but later reinforced by a research conducted +by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, +opensourcestyle}. + +\subsection{High-level project management and reporting} + +The Brazilian government used to work with software development in a +very traditional way. They would frequently focus on documents and not +on what was, in our opinion, what really matters: working software. This +dissonance caused us a communication noise with MP, because they would +often question our work style. It was especially hard to convince them +to accept the idea of open scope and agile development, but after months +of labor and showing results they stopped resisting. + +We defined some level of meeting granularity to avoid generating too +much overhead to the developers. We had a strategical and validating +meeting with MP (the former once in a month and the latter each 15th +day), release plaining with the entire team (one per month), and finally +a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a +diagram that represents our meeting organization. + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/meeting_flows.png} + \caption{Meetings cycles.} + \label{fig:meeting} +\end{figure} + +In the strategical meeting we usually defined the priorities and new +features with the Brazilian government. Normally the professors, the +coach of each team, the meta-coach, and some employees of the MP would +participate in this meeting. We usually discussed what the team already +produced since our last meeting, and established the new features for +the next release. Notice that just part of the team would join this +meeting to avoid generating unnecessary overhead to the developers, but +all the students interested to participate were allowed to join (many +students wanted this experience during the project). + +After the strategical meeting with Brazilian government agents, we had a +planning turn with all teams together. In this part, each team worked together +to convert the MP wishes into smaller parts which were represented by the epics of +the release. Each coach was responsible for conducting the planning, and after +that record it on the project wiki (the wiki provided by Gitlab). With this +epic, each 14th day the team have generated their sprint scheduler (with small +achievements mapped to issues). + +To keep the Brazilian government always updated, we invited them to work +with us to validate the new features in progress. Normally we had a +meeting each 15th day. Basically, this was our work flow, we always kept +everything extremely open to the MP (our way of working, and the one +often used by open source projects) and to the team. + +To keep the track of all of those things we used the SPB itself, +especially Gitlab. Basically, we had: + +\begin{enumerate} + \item Project repository: We have one organization with many repositories; + \item Milestones: Each milestone was used to register a release; + \item Wiki: Each release has one page on wiki with the compilation of + strategical meeting; + \item Issues: Each sprint planning generated issues, which we associated to + the specific milestone and updated the wiki with the issue number related + with them. Finally each developer assigned the issue to itself. +\end{enumerate} + +Notice that this workflow gave us and the Brazilian government agents full +traceability from a high level view of each feature to the lowest level (code). +It is important to highlight that we converge to this workflow after many +experiments. For example, we used a tool named Redmine for organizing our tasks +during the sprints, however, this tool revealed inefficient for our case since +the government agents lost part of the project traceability. We realized that +centralize all the work in the SPB portal is the best option for our case. diff --git a/opensym2017/content/10-contributions.tex b/opensym2017/content/10-contributions.tex new file mode 100644 index 0000000..2801e8d --- /dev/null +++ b/opensym2017/content/10-contributions.tex @@ -0,0 +1,51 @@ +\section{Contributions to the upstream communities} +\label{sec:contributions} + +%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) +%* Colab -> RevProxy +%* Colab, atualização do python/django +%* Contribuições para o GitLab (autenticação) +%* Noosfero, atualização do Rails, preparação para federação, nova interface ... +%* Coper, empacotamentos (obs), omniauth + + +During the execution of this project we made several contributions to +the upstream communities we interacted with. This occurred due to our +development process aligned with those of the respective communities. +During development, we would explicitly discuss the features and bug +fixes that we were working on with the applicable upstream communities. +This contributed to improve the developers technical solutions with +expertise outside of our team, and make it easier for those changes to +be accepted in the original projects. Having changes accepted upstream +in turn makes our life easier as it minimizes the delta between our +codebase and upstream's allowing us to upgrade and benefit from +development work from others. + +In Colab, we helped upstream redesign the entirely architecture, +enabling the development of plugins to integrate new tools. We also +added a feature that allowed Colab to run asynchronous tasks, which was +a major improvement for us since we were developing a complex system. A +migration to the latest Django version was made (web framework used by +Colab). Moreover, we worked on RevProxy (the more important dependency +of Colab) to put it in a good shape, fixing many bugs. + +Gitlab was the tool that we made the least number of modifications. We +contributed with some improvements related with configuration files and +we developed a new plugin that enables user authentication in Gitlab +through the REMOTE\_USER HTTP header. This plugin was needed because +Colab uses this mechanism to manage the authentication. + +Noosfero was the tool that contemplated several functional requirements, +therefore we made a large number of contributions with upstream. We helped to +migrate to the latest Rails version (web framework used by Noosfero), enable +the federation implementation (federation with other social networks), and +decouple the interface and the back-end. + +We also made some contributions on the DevOps front. Some members of +them team became maintainers of some python libraries that were used by +our scripts to upload packages to OBS (Open Build Service). We developed +a tool called copr-status to keep track of the different stages of the +lifecycle of each of the individual packages we were working on. + +%TODO: Mezuro + diff --git a/opensym2017/content/10-lessons.tex b/opensym2017/content/10-lessons.tex deleted file mode 100644 index 53b72ce..0000000 --- a/opensym2017/content/10-lessons.tex +++ /dev/null @@ -1,161 +0,0 @@ -\section{Lessons Learned} -\label{sec:lessons} - -\textbf{How to involve students real-world projects, interacting with -real customers.} -% -Our team was mainly composed of software engineering undergraduate -student, who had the opportunity to interact with senior developers and -designers inside the team, as well as with the team of technicians and -managers from the Brazilian Government, and the management team from -UnB. -% -They interacted with professionals that had diverse expertises, and were -able to participate in all levels of the software development process. -This contributed to a great learning opportunity, and for a majority of -them this was their first professional experience. - -\textbf{The participation of experienced professionals is crucial to -success of the project.} -One important factor for the students was the composition of the teams -with the participation of experienced professionals. -% -On the technical side, the senior developers and designers would handle -the more difficult technical decisions, creating a work environment -where the students could develop their skills in a didactic way without -pressure. -% -On the management side, the active participation of professors -- who -are, in the end, the ones responsible for the project -- is crucial to -make sure student participation is conducted in a healthy way, and is an -instance of leading by example. - -\textbf{A balanced relationship between academia and industry.} -The experience of the SPB project led UnB to develop a work style which -proved to be appropriate for an educational environment that brings -academia and industry together. -% -The highest priority from the university's point of view is the -students. Considering this, the activities of the project were never -prioritized to the detriment of classes and other pedagogical -activities. In summary, we had students working at different times, part -time, remotely or locally, always respecting their individual -conditions, but doing the work in a collective, collaborative and open -way. -% -And even under a potentially adverse environment, the project delivered -the desired solution with success. -% -At the end of the project, we noted that the skills developed by the -students were at the state of art of the software engineering practice. -After the project ended, we had team members successfully embracing -opportunities in public, private, national and international -organizations, in addition to those students that went into -entrepreneurship and opened their own companies. - -\textbf{Managing different organizational cultures.} -In the beginning of the project, the Brazilian Government stakeholders -had certain expectations about the development of project that, let's -say, didn't exactly match our work based on agile and FOSS practices. -% -We had to develop strategies that would support different these -organizational cultures. As reported in Section \ref{sec:process}, the -MP is organized in a functional hierarchical organizational structure, -typically, a traditional development paradigm. Therefore, we had to -create a translation process between our team and the MP managers who -managed the project on their side assuming a traditional, waterfall -process. - -\textbf{Manage higher-level and lower-level goals separately.} -During the initial phase of the project the MP team would often bring -strategic discussions to technical/operational meetings, where we were -supposed to discuss practical, technical decisions. -% -This produced a highly complex communication and management environment, -overloading the professors because they were supposed to be responsible -for maintaining the strategic alignment of the MP synchronized with -implementation plans of the development team, specially in light of the -aforementioned culture mismatch. Mixing both concerns in the same -discussions caused confusion on both sides. -% -Towards the middle and end of the project we were able to keep those -concerns separated, what eased the work of everyone involved. - -\textbf{Living with ill-advised political decisions.} -At the initial phases of the project, by political and personal -motivation, the main stakeholders from the Brazilian government imposed -the use of Colab to guide the development of the new SPB platform. Our -team was totally against the idea because we already knew that Colab was -a very experimental project and its adoption could dramatically increase -the project complexity. Even though we provided technical reasons to -not utilize Colab, MP was adamant and we had to manage this problem. As -explained in section \ref{sec:architecture} we did massive changes to -it, and at the end of the project we completely rewrote Colab and made -it stable. It is important to notice that the MP compelled us to accept -a technical issue based only on political interests, without considering -all the resources that would be spent due to that decision. At the end -of the project, we verified that Colab indeed consumed a vast amount of -the budget and increased the project complexity. In the end of the -project and after our analysis on the decision made by the MP, we -understand that MP's managers are capable of ignoring technical reasons -in favor to a political decision. - -\textbf{Consider sustainability from the beginning.} -In the process of deploying the SPB platform in the MP structure, we had -to interact with the MP technicians. We did several workshops, training -and a meticulous documentation describing all the required procedures to -update the platform, however, we realized that the MP technicians -constantly avoid the responsibility. After noticing the aforementioned -situation, we organized a specific DevOps that completely automated all -the deployment procedure. We simplified all the platform deployment to a -few steps: (1) initial configurations (just ssh configuration) and (2) -the execution of simple commands to completely update the platform. By -the end of the project, we observed that the MP technicians invariably -still depended on our support to update the platform even with all the -automation provided by us. We were sadly left with a feeling of -uncertainty about the future of the platform after the project ended. In -hindsight, we realize that the MP dedicated systems analysts and -managers to the project, but not operations technicians. The later -should have been more involved with the process so they could at least -comfortable with managing the platform infrastructure. - -% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados -%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa -%% afirmação, embora eu e Paulo consigamos perceber isso. - -%=========== -% Conclusion -%=========== - -\textbf{Final remarks.} -The portal is available at \url{softwarepublico.gov.br}. All -documentation, including detailed architecture and operation manuals are -also available\footnote{\url{https://softwarepublico.gov.br/doc/} -(in Portuguese only at the moment)}). -% -All the integrated tools are FOSS and our contributions were published -in open repositories, available on the SPB Portal itself. We also -contributed these features back to the respective communities: that -benefits those communities, as well as us since we can share future -development and maintenance effort with other organizations that -participate in their projects. - - -%* utilização do projeto para formação de recursos humanos (alunos) - -%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate - -%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) - -%* 69 projetos marcados como SPB, de 81 no total na plataforma. - -%* 47\% é desenvolvido em PHP. - -% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). - -% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. - -% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. - -%* Debater economia de recursos em orgão públicos - diff --git a/opensym2017/content/11-conclusion.tex b/opensym2017/content/11-conclusion.tex deleted file mode 100644 index 371960f..0000000 --- a/opensym2017/content/11-conclusion.tex +++ /dev/null @@ -1,35 +0,0 @@ -\section{Conclusion} -\label{sec:conclusion} - -In this paper we present and discuss issues experienced during a government- -funded project, in partnership with University of Brasilia and University of -São Paulo, to evolve the Brazilian Public Software portal. - -The contributions of this paper are twofold. First, we present how was -developed an unprecedent platform, delivered to Brazilian government. This -platform - developed by an heterogenous team of professors, master and -undergraduate students, IT professionals and governmental managers - provides -several modern features from the integration of more than 10 FLOSS systems. - -Second, the 30 months project which developed this platform results in an -important case that it is possible to mitigate issues seen as conflicting to IT -development environment and between industry and academy. We shown that, as -long as the institution can provide a healthy and challenging environment to -its students, its is possible to conciliate studies and professional training -in universities. After the end of the project, some students successfully -embraced opportunities in public and private sectos, within national borders -and abroad. Some others went further and started their own companies. -We also shown that, with some adaptations/"translation processes", was possible -to conciliate agile methodologies and FOSS practices to develop software to -governmental organizations with functional hierarchical structures that use -traditional development paradigm. - -In the SPB project it was employed many of our beliefs about FLOSS and Agile -practices. The team was engaged in creating a friendly environment for everyone -involved in the project and in showing to government agents another way to -interact with the FLOSS community and the university. With an open work style, -the project has seeked to be transparent for the whole society. We belive it is -still needed to analyze every data produced by the project and its impact on the -students. For future work, we would conduce a \textit{post-mortem} analyse in the -project, seeing that we have many open data to be explored and it was applied -approachs which still need extra validation. diff --git a/opensym2017/content/11-lessons.tex b/opensym2017/content/11-lessons.tex new file mode 100644 index 0000000..07b0b8f --- /dev/null +++ b/opensym2017/content/11-lessons.tex @@ -0,0 +1,121 @@ +\section{Lessons Learned} +\label{sec:lessons} + +\textbf{How to involve students real-world projects, interacting with +real customers.} +% +Our team was mainly composed of software engineering undergraduate +student, who had the opportunity to interact with senior developers and +designers inside the team, as well as with the team of technicians and +managers from the Brazilian Government, and the management team from +UnB. +% +They interacted with professionals that had diverse expertises, and were +able to participate in all levels of the software development process. +This contributed to a great learning opportunity, and for a majority of +them this was their first professional experience. + +\textbf{The participation of experienced professionals is crucial to +success of the project.} +One important factor for the students was the composition of the teams +with the participation of experienced professionals. +% +On the technical side, the senior developers and designers would handle +the more difficult technical decisions, creating a work environment +where the students could develop their skills in a didactic way without +pressure. +% +On the management side, the active participation of professors -- who +are, in the end, the ones responsible for the project -- is crucial to +make sure student participation is conducted in a healthy way, and is an +instance of leading by example. + +\textbf{A balanced relationship between academia and industry.} +The experience of the SPB project led UnB to develop a work style which +proved to be appropriate for an educational environment that brings +academia and industry together. +% +The highest priority from the university's point of view is the +students. Considering this, the activities of the project were never +prioritized to the detriment of classes and other pedagogical +activities. In summary, we had students working at different times, part +time, remotely or locally, always respecting their individual +conditions, but doing the work in a collective, collaborative and open +way. +% +And even under a potentially adverse environment, the project delivered +the desired solution with success. +% +At the end of the project, we noted that the skills developed by the +students were at the state of art of the software engineering practice. +After the project ended, we had team members successfully embracing +opportunities in public, private, national and international +organizations, in addition to those students that went into +entrepreneurship and opened their own companies. + +\textbf{Managing different organizational cultures.} +In the beginning of the project, the Brazilian Government stakeholders +had certain expectations about the development of project that, let's +say, didn't exactly match our work based on agile and FOSS practices. +% +We had to develop strategies that would support different these +organizational cultures. As reported in Section \ref{sec:process}, the +MP is organized in a functional hierarchical organizational structure, +typically, a traditional development paradigm. Therefore, we had to +create a translation process between our team and the MP managers who +managed the project on their side assuming a traditional, waterfall +process. + +\textbf{Manage higher-level and lower-level goals separately.} +During the initial phase of the project the MP team would often bring +strategic discussions to technical/operational meetings, where we were +supposed to discuss practical, technical decisions. +% +This produced a highly complex communication and management environment, +overloading the professors because they were supposed to be responsible +for maintaining the strategic alignment of the MP synchronized with +implementation plans of the development team, specially in light of the +aforementioned culture mismatch. Mixing both concerns in the same +discussions caused confusion on both sides. +% +Towards the middle and end of the project we were able to keep those +concerns separated, what eased the work of everyone involved. + +\textbf{Living with ill-advised political decisions.} +At the initial phases of the project, by political and personal +motivation, the main stakeholders from the Brazilian government imposed +the use of Colab to guide the development of the new SPB platform. Our +team was totally against the idea because we already knew that Colab was +a very experimental project and its adoption could dramatically increase +the project complexity. Even though we provided technical reasons to +not utilize Colab, MP was adamant and we had to manage this problem. As +explained in section \ref{sec:architecture} we did massive changes to +it, and at the end of the project we completely rewrote Colab and made +it stable. It is important to notice that the MP compelled us to accept +a technical issue based only on political interests, without considering +all the resources that would be spent due to that decision. At the end +of the project, we verified that Colab indeed consumed a vast amount of +the budget and increased the project complexity. In the end of the +project and after our analysis on the decision made by the MP, we +understand that MP's managers are capable of ignoring technical reasons +in favor to a political decision. + +\textbf{Consider sustainability from the beginning.} +In the process of deploying the SPB platform in the MP structure, we had +to interact with the MP technicians. We did several workshops, training +and a meticulous documentation describing all the required procedures to +update the platform, however, we realized that the MP technicians +constantly avoid the responsibility. After noticing the aforementioned +situation, we organized a specific DevOps that completely automated all +the deployment procedure. We simplified all the platform deployment to a +few steps: (1) initial configurations (just ssh configuration) and (2) +the execution of simple commands to completely update the platform. By +the end of the project, we observed that the MP technicians invariably +still depended on our support to update the platform even with all the +automation provided by us. We were sadly left with a feeling of +uncertainty about the future of the platform after the project ended. In +hindsight, we realize that the MP dedicated systems analysts and +managers to the project, but not operations technicians. The later +should have been more involved with the process so they could at least +comfortable with managing the platform infrastructure. + diff --git a/opensym2017/content/12-conclusion.tex b/opensym2017/content/12-conclusion.tex new file mode 100644 index 0000000..f6ee382 --- /dev/null +++ b/opensym2017/content/12-conclusion.tex @@ -0,0 +1,75 @@ +\section{Conclusion} +\label{sec:conclusion} + +In this paper we present and discuss issues experienced during a government- +funded project, in partnership with University of Brasilia and University of +São Paulo, to evolve the Brazilian Public Software portal. + +The contributions of this paper are twofold. First, we present how was +developed an unprecedent platform, delivered to Brazilian government. This +platform - developed by an heterogenous team of professors, master and +undergraduate students, IT professionals and governmental managers - provides +several modern features from the integration of more than 10 FLOSS systems. + +Second, the 30 months project which developed this platform results in an +important case that it is possible to mitigate issues seen as conflicting to IT +development environment and between industry and academy. We shown that, as +long as the institution can provide a healthy and challenging environment to +its students, its is possible to conciliate studies and professional training +in universities. After the end of the project, some students successfully +embraced opportunities in public and private sectos, within national borders +and abroad. Some others went further and started their own companies. +We also shown that, with some adaptations/"translation processes", was possible +to conciliate agile methodologies and FOSS practices to develop software to +governmental organizations with functional hierarchical structures that use +traditional development paradigm. + +In the SPB project it was employed many of our beliefs about FLOSS and Agile +practices. The team was engaged in creating a friendly environment for everyone +involved in the project and in showing to government agents another way to +interact with the FLOSS community and the university. With an open work style, +the project has seeked to be transparent for the whole society. We belive it is +still needed to analyze every data produced by the project and its impact on the +students. For future work, we would conduce a \textit{post-mortem} analyse in the +project, seeing that we have many open data to be explored and it was applied +approachs which still need extra validation. + +The portal is available at \url{softwarepublico.gov.br}. All +documentation, including detailed architecture and operation manuals are +also available\footnote{\url{https://softwarepublico.gov.br/doc/} +(in Portuguese only at the moment)}). +% +All the integrated tools are FOSS and our contributions were published +in open repositories, available on the SPB Portal itself. We also +contributed these features back to the respective communities: that +benefits those communities, as well as us since we can share future +development and maintenance effort with other organizations that +participate in their projects. + +%=========== +% Conclusion +%=========== + +% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados +%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa +%% afirmação, embora eu e Paulo consigamos perceber isso. + + +%* utilização do projeto para formação de recursos humanos (alunos) + +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate + +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) + +%* 69 projetos marcados como SPB, de 81 no total na plataforma. + +%* 47\% é desenvolvido em PHP. + +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). + +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. + +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. + +%* Debater economia de recursos em orgão públicos + diff --git a/opensym2017/spb.tex b/opensym2017/spb.tex index c09481c..a1a2ae2 100644 --- a/opensym2017/spb.tex +++ b/opensym2017/spb.tex @@ -171,14 +171,15 @@ \input{content/01-introduction} \input{content/02-spb} \input{content/03-relatedworks} -\input{content/04-requirements} -\input{content/05-architecture} -\input{content/06-features} -\input{content/07-ux} -\input{content/08-process} -\input{content/09-contributions} -\input{content/10-lessons} -\input{content/11-conclusion} +\input{content/04-researchdesign} +\input{content/05-requirements} +\input{content/06-architecture} +\input{content/07-features} +\input{content/08-ux} +\input{content/09-process} +\input{content/10-contributions} +\input{content/11-lessons} +\input{content/12-conclusion} %------------------------------------------------------------------------------ \bibliographystyle{SIGCHI-Reference-Format} -- libgit2 0.21.2