diff --git a/opensym2017/content/03-relatedworks.tex b/opensym2017/content/03-relatedworks.tex new file mode 100644 index 0000000..4f5a1c4 --- /dev/null +++ b/opensym2017/content/03-relatedworks.tex @@ -0,0 +1,61 @@ +\section{Related Works and Projects} +\label{sec:relatedworks} + +The new SPB platform presented in this paper is a fully integrated +environment, as we can see in Figure \ref{fig:requirements}, being very +advanced in comparison with related projects and initiatives. For example, +the USA government has a platform designed to improve access to the federal +government developed software\footnote{\url{https://code.gov}}. Code.gov +is an interface to organize the USA government projects and, in short, make +it easy for users and developers to obtain information and access their +source code repositories at GitHub. However, there are not social networking +and CMS features, as well as, other communication resources provided by that +platform. + +Additionally, there are two initiatives in Europe: +OSOR\footnote{\url{https://joinup.ec.europa.eu/community/osor}} and +OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) +is a community hosted in the JoinUp platform powered by the European +Commission. OSOR aims at exchanging information, experiences and best +practices around the use of FOSS in the public administration. It helps +to find a FOSS made available by other public administrations, providing +access to information such as news, events, studies and solutions +related to implementation of open source software. It also offers forum +discussions and community mailing lists, but it does not have an +integrated source code repository manager and for the each project there +is a link to its own external repository (or its tarball file). +% +OW2 is a FOSS community to promote the development of FOSS middleware, generic +business applications, cloud computing platforms and foster a community and +business ecosystem. In short, it aims to support the development, deployment +and management of distributed applications with a focus on FOSS middleware and +related development and management tools. +% +Moreover, from the European Commission in 2007 until 20011, there was the +QualiPSo project that aimed at providing FOSS users, developers, and consumers, +with quality resources and expertise on the various topics related to FOSS. The +QualiPSo project also had planned to develop a platform called QualiPSo +Factory but it was not fully completed. + +In Latin American there is an initiative based on the SPB project called ``Software +Publico Regional''\footnote{\url{http://softwarepublicoregionalbeta.net}}. From +a practical point of view, it provides a customized Gitlab instance to share +the source code and documentation of the project from the involved countries. +% +Like Brazil, Chile has its own portal also called ``Software +Publico''\footnote{\url{http://www.softwarepublico.gob.cl}}. Users can create +content in the communities (news items, documents, wiki pages), but +source code repositories are available at the Bitbucket +platform\footnote{\url{https://bitbucket.org/softwarepublico}}. + +The Brazilian government needed to evolve the SPB project that +existed since 2005. In 2013, when we started this project, the SPB Portal +had about 200 thousand registered users. We could not just contact these +users and ask them to register an account at Github as well. Moreover, +after the Edward Snowden case, the Brazilian government approved a +specific law decree (8.135/2013) to rule its communication services, +requiring the public administration to host its information systems to +be provided by itself, what rules out usage of private platforms, +specially ones provided by foreign companies. We thus developed our own +solution to cover all the requirements, producing a complete +governmental integrated platform for collaborative software development. diff --git a/opensym2017/content/03-requirements.tex b/opensym2017/content/03-requirements.tex deleted file mode 100644 index caa4e7f..0000000 --- a/opensym2017/content/03-requirements.tex +++ /dev/null @@ -1,138 +0,0 @@ -\section{Requirements and Related Projects} -\label{sec:requirements} - -In 2013, the SPB Portal had more than 600 thousand unique visitors, generating -more than 16 million page views with about 50 million hits. By evaluating only -the main projects, there were more than 15 thousand downloads and 4 thousand -messages exchanged in their forums. These data illustrates the potential of the -SPB Portal, even with several limitations in the past. - -By preparing the evolution project described in this paper, the Brazilian -government promoted 3 events to collect the requirements, in particular from -society point of view: (i) an online form to collect general ideas; (ii) a -face-to-face meeting with society in general; (iii) a workshop to review the -SPB concepts and requirements with IT stakeholders from the Brazilian -government and public organizations. - -After these 3 rounds discussing the new SPB platform, the Brazilian government -listed about 145 requirements and developed a ``mind -map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} -to guide the SPB portal evolution. In this scenario, the 10 most voted -requirements were, for example: - -\begin{enumerate} - -\item Source code repository with public access. -\item Visit community pages without login. -\item Distributed version control system. -\item Scores of users and developers collaboration. -\item Search software by features. -\item Integration with social networks. -\item Repository for future ideas and requirements. -\item Friendly URL to access a public software community page. -\item User feedback about a public software. -\item Report of the experience about the use of a public software. - -\end{enumerate} - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/technological-requirements.png} - \caption{Technological requirements overview.} - \label{fig:requirements} -\end{figure} - - -There were other requirements based on the experience of the IT -stakeholders from the Brazilian government and from the Brazilian FOSS -community (that UnB and USP were representing too in this project). The -new platform would only work properly if there is a unique -authentication to use the provided tools. Additionally, a unified -interface was an important non-functional requirement to have a better -user experience in the new platform. - -At the first moment, we desired to release an initial version that could -replace the old SPB portal. For that, the first version should have -features such as: - -\begin{enumerate} - -\item An organized public software catalog. -\item Social network environment (profiles for users, software pages, and community pages). -\item Content Management Systems (CMS) features. -\item Web-based Git repository manager with wiki and issue tracking features. -\item Mailing lists and discussion forums. - -\end{enumerate} - -Other requirements were also planned during the conception phase of the -SPB evolution project, such as an integrated search engine and a -web-based source code static analysis monitor. By analyzing all of these -requirements, we proposed the technological requirements overview -illustrated in Figure \ref{fig:requirements} to guide the development of -the new SPB platform. In other words, we have designed the SPB evolution -project based on existing FOSS tools. However, the integration of -several existing systems that were already implemented in different -programming languages and frameworks, adding features such as a -centralized authentication, unified interface, and a search engine, as -well as, other back-end features, would require a non-trivial amount of -work. - -The new SPB platform is a fully integrated environment, as we can see in -Figure \ref{fig:requirements}, being very advanced in comparison with -related projects and initiatives. For example, the USA government has a -platform designed to improve access to the federal government developed -software\footnote{\url{https://code.gov}}. Code.gov is an interface to -organize the USA government projects and, in short, make it easy for -users and developers to obtain information and access their source code -repositories at GitHub. However, there are not social networking and CMS -features, as well as, other communication resources provided by that -platform. - -Additionally, there are two initiatives in Europe: -OSOR\footnote{\url{https://joinup.ec.europa.eu/community/osor}} and -OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) -is a community hosted in the JoinUp platform powered by the European -Commission. OSOR aims at exchanging information, experiences and best -practices around the use of FOSS in the public administration. It helps -to find a FOSS made available by other public administrations, providing -access to information such as news, events, studies and solutions -related to implementation of open source software. It also offers forum -discussions and community mailing lists, but it does not have an -integrated source code repository manager and for the each project there -is a link to its own external repository (or its tarball file). -% -OW2 is a FOSS community to promote the development of FOSS middleware, generic -business applications, cloud computing platforms and foster a community and -business ecosystem. In short, it aims to support the development, deployment -and management of distributed applications with a focus on FOSS middleware and -related development and management tools. -% -Moreover, from the European Commission in 2007 until 20011, there was the -QualiPSo project that aimed at providing FOSS users, developers, and consumers, -with quality resources and expertise on the various topics related to FOSS. The -QualiPSo project also had planned to develop a platform called QualiPSo -Factory but it was not fully completed. - -In Latin American there is an initiative based on the SPB project called ``Software -Publico Regional''\footnote{\url{http://softwarepublicoregionalbeta.net}}. From -a practical point of view, it provides a customized Gitlab instance to share -the source code and documentation of the project from the involved countries. -% -Like Brazil, Chile has its own portal also called ``Software -Publico''\footnote{\url{http://www.softwarepublico.gob.cl}}. Users can create -content in the communities (news items, documents, wiki pages), but -source code repositories are available at the Bitbucket -platform\footnote{\url{https://bitbucket.org/softwarepublico}}. - -The Brazilian government needed to evolve the SPB project that -existed since 2005. In 2013, when we started this project, the SPB Portal -had about 200 thousand registered users. We could not just contact these -users and ask them to register an account at Github as well. Moreover, -after the Edward Snowden case, the Brazilian government approved a -specific law decree (8.135/2013) to rule its communication services, -requiring the public administration to host its information systems to -be provided by itself, what rules out usage of private platforms, -specially ones provided by foreign companies. We thus developed our own -solution to cover all the requirements, producing a complete -governmental integrated platform for collaborative software development. diff --git a/opensym2017/content/04-architecture.tex b/opensym2017/content/04-architecture.tex deleted file mode 100644 index ff74bfc..0000000 --- a/opensym2017/content/04-architecture.tex +++ /dev/null @@ -1,173 +0,0 @@ -\section{Architecture} -\label{sec:architecture} - -Based on the extensive list of functional requirements defined by the -Brazilian Federal Government, we selected some FOSS systems to form our -solution. We looked for system that together could provide the largest -subset possible of the requirements, and were fully aware that we would -need to improve those systems in order to provide the rest. We were also -convinced that it would be impossible to provide all of those -requirements with a single tool. - -From the point of view of the architecture, two main requirements was -brought by the Brazilian Federal Government for the new platform: - -\begin{enumerate} - \item \textit{Integrate existing FOSS systems}, with minimal differences from - their original versions; - \item \textit{Provide a consistent user interface} across the different - systems, as well as centralized authentication. -\end{enumerate} - -Adopting existing FOSS systems and minimizing locally-made changes had -the objective of being able to upgrade to newer versions of the original -software, benefiting from improvements and maintenance done by the -existing project communities. Providing a consistent user interface on -top of those different tools was needed to make the transition between -the different systems seamless from the point of view of users, which -would be confused by jumping through completely different interfaces -while interacting with the portal. - -For the first requirement, we identified four main systems that required -specialized teams for work in the integration process. The teams learned -how to develop for their assigned systems and contributed to the -original communities, so that the version we used were not significantly -different from the original. - -At the end of the project, the SPB portal was composed of more than ten -systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and -Munin. The remainder of this section describes the most relevant of -them, as well as how they were integrated into the platform. - -\subsection{Colab} - -Colab\footnote{\url{https://github.com/colab}} is a systems integration -platform for web applications. One of its goals is allowing different -applications to be combined in such a way that an user does not notice -the change between applications. For that, Colab provides facilities -for: - -\begin{itemize} - \item \textit{Centralized authentication}: Users are automatically -identified for each application without having to enter login -credentials on each of them. - \item \textit{Visual consistency}: Colab is able to change the HTML -code produced by the integrated applications, such as defining default -template or reusing common HTML pages, in order to provide a unified -interface. - \item \textit{Relaying of events between applications}: The -integrated applications or Colab itself are able to trigger an action -on another application. - \item \textit{Integrated search engine}: It is possible to set up -exactly which data needs to be indexed from each application and how -users are directed to this information on correspondent application. -\end{itemize} - -Colab implements this integration by working as a reverse proxy for the -integration applications, that is, all external requests pass through -Colab before reaching them. - -Initially, Colab had support for a small set of applications (Trac, GNU -Mailman, and Apache Lucene) and support for them was hard-coded. Our -team evolved Colab so that it can now receive plugins that add support -new applications without the need of changes to the Colab core. We also -developed plugins to be used in the SPB platform: Noosfero, GitLab, and -Mezuro. - -\subsection{Noosfero} - -Noosfero\footnote{\url{http://noosfero.org}} is a software for building -social and collaboration networks. Besides the classical social -networking features, it also provides publication features such as blogs -and a general-purpose CMS (Content Management System). Most of the user -interactions with SPB is through Noosfero: user registration, project -home pages and documentation, and contact forms. - -\subsection{Gitlab} - -GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository -manager with wiki pages and issue tracking features. Gitlab is a FOSS -platform and focuses on delivering a holistic solution that will see -developers from idea to production seamlessly and on a single platform. - -Gitlab has several unique features, such as built-in continuous -integration and continuous deployment, flexible permissions, tracking of -Work-in-Progress work, moving issues between projects, group-level -milestones, creating new branches from issues, issues board, and time -tracking. - -\subsection{Mezuro} - -Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code -metrics to monitor the internal quality of software written in C, C++, -Java, Python, Ruby, and PHP. - -In general, source code metrics tools also do not present a friendly way -to interpret its results and, even more, do not follow a standardization -between them. Mezuro collects and presents these results to the end -user, specially, by analyzing source code metric history during its life -cycle. - -The Mezuro platform provides a single interface grouping available -tools, allows selection and composition of metrics in a flexible manner, -stores the metrics evolution history, presents results in a friendly -way, as well as, allows users to customize the given interpretation -accordingly to their own context. - -\subsection{System unification} - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/arch.png} - \caption{SPB architecture overview.} - \label{fig:architecture} -\end{figure} - -The conceptual architecture of the platform is presented in Figure -\ref{fig:architecture}. Colab initially handles all user interaction, -directing requests to one of the integrated applications. It -post-processes responses from the applications to apply a consistent -visual appearance, manages authentication, and provides a unified search -functionality: instead of using the redundant restricted search -functionality of each application, a search in the SPB portal might -return content from any of the applications, be it web pages, mailing -list posts, or source code. - -% Falar do devops -\subsection{Deploy} - -The SPB platform was deployed in 7 virtual machines with different functions, -as we can see in Figure \ref{fig:architecture2}. - -\begin{figure*}[hbt] - \centering - \includegraphics[width=.8\linewidth]{figures/arch3.png} - \caption{Instanciation view of the SPB architecture.} - \label{fig:architecture2} -\end{figure*} - -The \textit{reverseproxy} handles the HTTP requests and redirects them to the -\textit{integration}, the \textit{email} sends and receives e-mails on behalf -of the platform and the \textit{monitor} keeps the entire environment tracked. -These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and -\textit{monitor} - are accessible via Internet and the other ones are only -available in the local network created between them. - -\textit{Integration} works as a second layer of proxy beneath -\textit{reverseproxy}, any request to the platform will be handled by it. The -Colab service provides interface, authentication and search engine integration -among all the services. When a request is received to a specific service, -Colab authenticates the user in the target tool, sends the request and makes a -visual transformation in the HTML page which is the content of the response. -Another user-oriented feature is the integrated search engine, when the user -want to find something in the platform Colab will perform the search in the -whole databases. Colab itself provides a web interface for GNU Mailman and we -have two others integrated tools in \textit{integration}: Gitlab and Prezento. -Gitlab provides web interface for Git repositories and issues tracker, and -Prezento is a front-end for source code static analysis. - -The source code static analysis is performed by \textit{mezuro}. It runs some -static analysis tools on source code stored in repository and provide this data -to Prezento. A social network and CMS (Content Manager System) is provided by -Noosfero in \textit{social}, and the databases of all tools with a cache -service are in \textit{database}. diff --git a/opensym2017/content/04-requirements.tex b/opensym2017/content/04-requirements.tex new file mode 100644 index 0000000..7cb5aa0 --- /dev/null +++ b/opensym2017/content/04-requirements.tex @@ -0,0 +1,79 @@ +\section{Requirements} +\label{sec:requirements} + +In 2013, the SPB Portal had more than 600 thousand unique visitors, generating +more than 16 million page views with about 50 million hits. By evaluating only +the main projects, there were more than 15 thousand downloads and 4 thousand +messages exchanged in their forums. These data illustrates the potential of the +SPB Portal, even with several limitations in the past. + +By preparing the evolution project described in this paper, the Brazilian +government promoted 3 events to collect the requirements, in particular from +society point of view: (i) an online form to collect general ideas; (ii) a +face-to-face meeting with society in general; (iii) a workshop to review the +SPB concepts and requirements with IT stakeholders from the Brazilian +government and public organizations. + +After these 3 rounds discussing the new SPB platform, the Brazilian government +listed about 145 requirements and developed a ``mind +map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} +to guide the SPB portal evolution. In this scenario, the 10 most voted +requirements were, for example: + +\begin{enumerate} + +\item Source code repository with public access. +\item Visit community pages without login. +\item Distributed version control system. +\item Scores of users and developers collaboration. +\item Search software by features. +\item Integration with social networks. +\item Repository for future ideas and requirements. +\item Friendly URL to access a public software community page. +\item User feedback about a public software. +\item Report of the experience about the use of a public software. + +\end{enumerate} + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/technological-requirements.png} + \caption{Technological requirements overview.} + \label{fig:requirements} +\end{figure} + + +There were other requirements based on the experience of the IT +stakeholders from the Brazilian government and from the Brazilian FOSS +community (that UnB and USP were representing too in this project). The +new platform would only work properly if there is a unique +authentication to use the provided tools. Additionally, a unified +interface was an important non-functional requirement to have a better +user experience in the new platform. + +At the first moment, we desired to release an initial version that could +replace the old SPB portal. For that, the first version should have +features such as: + +\begin{enumerate} + +\item An organized public software catalog. +\item Social network environment (profiles for users, software pages, and community pages). +\item Content Management Systems (CMS) features. +\item Web-based Git repository manager with wiki and issue tracking features. +\item Mailing lists and discussion forums. + +\end{enumerate} + +Other requirements were also planned during the conception phase of the +SPB evolution project, such as an integrated search engine and a +web-based source code static analysis monitor. By analyzing all of these +requirements, we proposed the technological requirements overview +illustrated in Figure \ref{fig:requirements} to guide the development of +the new SPB platform. In other words, we have designed the SPB evolution +project based on existing FOSS tools. However, the integration of +several existing systems that were already implemented in different +programming languages and frameworks, adding features such as a +centralized authentication, unified interface, and a search engine, as +well as, other back-end features, would require a non-trivial amount of +work. diff --git a/opensym2017/content/05-architecture.tex b/opensym2017/content/05-architecture.tex new file mode 100644 index 0000000..ff74bfc --- /dev/null +++ b/opensym2017/content/05-architecture.tex @@ -0,0 +1,173 @@ +\section{Architecture} +\label{sec:architecture} + +Based on the extensive list of functional requirements defined by the +Brazilian Federal Government, we selected some FOSS systems to form our +solution. We looked for system that together could provide the largest +subset possible of the requirements, and were fully aware that we would +need to improve those systems in order to provide the rest. We were also +convinced that it would be impossible to provide all of those +requirements with a single tool. + +From the point of view of the architecture, two main requirements was +brought by the Brazilian Federal Government for the new platform: + +\begin{enumerate} + \item \textit{Integrate existing FOSS systems}, with minimal differences from + their original versions; + \item \textit{Provide a consistent user interface} across the different + systems, as well as centralized authentication. +\end{enumerate} + +Adopting existing FOSS systems and minimizing locally-made changes had +the objective of being able to upgrade to newer versions of the original +software, benefiting from improvements and maintenance done by the +existing project communities. Providing a consistent user interface on +top of those different tools was needed to make the transition between +the different systems seamless from the point of view of users, which +would be confused by jumping through completely different interfaces +while interacting with the portal. + +For the first requirement, we identified four main systems that required +specialized teams for work in the integration process. The teams learned +how to develop for their assigned systems and contributed to the +original communities, so that the version we used were not significantly +different from the original. + +At the end of the project, the SPB portal was composed of more than ten +systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and +Munin. The remainder of this section describes the most relevant of +them, as well as how they were integrated into the platform. + +\subsection{Colab} + +Colab\footnote{\url{https://github.com/colab}} is a systems integration +platform for web applications. One of its goals is allowing different +applications to be combined in such a way that an user does not notice +the change between applications. For that, Colab provides facilities +for: + +\begin{itemize} + \item \textit{Centralized authentication}: Users are automatically +identified for each application without having to enter login +credentials on each of them. + \item \textit{Visual consistency}: Colab is able to change the HTML +code produced by the integrated applications, such as defining default +template or reusing common HTML pages, in order to provide a unified +interface. + \item \textit{Relaying of events between applications}: The +integrated applications or Colab itself are able to trigger an action +on another application. + \item \textit{Integrated search engine}: It is possible to set up +exactly which data needs to be indexed from each application and how +users are directed to this information on correspondent application. +\end{itemize} + +Colab implements this integration by working as a reverse proxy for the +integration applications, that is, all external requests pass through +Colab before reaching them. + +Initially, Colab had support for a small set of applications (Trac, GNU +Mailman, and Apache Lucene) and support for them was hard-coded. Our +team evolved Colab so that it can now receive plugins that add support +new applications without the need of changes to the Colab core. We also +developed plugins to be used in the SPB platform: Noosfero, GitLab, and +Mezuro. + +\subsection{Noosfero} + +Noosfero\footnote{\url{http://noosfero.org}} is a software for building +social and collaboration networks. Besides the classical social +networking features, it also provides publication features such as blogs +and a general-purpose CMS (Content Management System). Most of the user +interactions with SPB is through Noosfero: user registration, project +home pages and documentation, and contact forms. + +\subsection{Gitlab} + +GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository +manager with wiki pages and issue tracking features. Gitlab is a FOSS +platform and focuses on delivering a holistic solution that will see +developers from idea to production seamlessly and on a single platform. + +Gitlab has several unique features, such as built-in continuous +integration and continuous deployment, flexible permissions, tracking of +Work-in-Progress work, moving issues between projects, group-level +milestones, creating new branches from issues, issues board, and time +tracking. + +\subsection{Mezuro} + +Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code +metrics to monitor the internal quality of software written in C, C++, +Java, Python, Ruby, and PHP. + +In general, source code metrics tools also do not present a friendly way +to interpret its results and, even more, do not follow a standardization +between them. Mezuro collects and presents these results to the end +user, specially, by analyzing source code metric history during its life +cycle. + +The Mezuro platform provides a single interface grouping available +tools, allows selection and composition of metrics in a flexible manner, +stores the metrics evolution history, presents results in a friendly +way, as well as, allows users to customize the given interpretation +accordingly to their own context. + +\subsection{System unification} + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/arch.png} + \caption{SPB architecture overview.} + \label{fig:architecture} +\end{figure} + +The conceptual architecture of the platform is presented in Figure +\ref{fig:architecture}. Colab initially handles all user interaction, +directing requests to one of the integrated applications. It +post-processes responses from the applications to apply a consistent +visual appearance, manages authentication, and provides a unified search +functionality: instead of using the redundant restricted search +functionality of each application, a search in the SPB portal might +return content from any of the applications, be it web pages, mailing +list posts, or source code. + +% Falar do devops +\subsection{Deploy} + +The SPB platform was deployed in 7 virtual machines with different functions, +as we can see in Figure \ref{fig:architecture2}. + +\begin{figure*}[hbt] + \centering + \includegraphics[width=.8\linewidth]{figures/arch3.png} + \caption{Instanciation view of the SPB architecture.} + \label{fig:architecture2} +\end{figure*} + +The \textit{reverseproxy} handles the HTTP requests and redirects them to the +\textit{integration}, the \textit{email} sends and receives e-mails on behalf +of the platform and the \textit{monitor} keeps the entire environment tracked. +These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and +\textit{monitor} - are accessible via Internet and the other ones are only +available in the local network created between them. + +\textit{Integration} works as a second layer of proxy beneath +\textit{reverseproxy}, any request to the platform will be handled by it. The +Colab service provides interface, authentication and search engine integration +among all the services. When a request is received to a specific service, +Colab authenticates the user in the target tool, sends the request and makes a +visual transformation in the HTML page which is the content of the response. +Another user-oriented feature is the integrated search engine, when the user +want to find something in the platform Colab will perform the search in the +whole databases. Colab itself provides a web interface for GNU Mailman and we +have two others integrated tools in \textit{integration}: Gitlab and Prezento. +Gitlab provides web interface for Git repositories and issues tracker, and +Prezento is a front-end for source code static analysis. + +The source code static analysis is performed by \textit{mezuro}. It runs some +static analysis tools on source code stored in repository and provide this data +to Prezento. A social network and CMS (Content Manager System) is provided by +Noosfero in \textit{social}, and the databases of all tools with a cache +service are in \textit{database}. diff --git a/opensym2017/content/05-features.tex b/opensym2017/content/05-features.tex deleted file mode 100644 index 9767637..0000000 --- a/opensym2017/content/05-features.tex +++ /dev/null @@ -1,73 +0,0 @@ -\section{Features} -\label{sec:spb} - -The new generation of the SPB portal combines features adapted from existing collaborative software -and features developed by the SPB team. Whenever possible, new functions -(newly developed or partially modified) were contributed back to the official repositories. - -As a result, we have a platform that integrates and harmonizes different features such as -social networking, mailing list, version control system, content management and -source code quality monitoring. Our aim was to develop functionalities by reusing -functionality of collaborative software already integrated to the platform. In -addition, we tried to keep this integration transparent to end users. - -\subsection{Software and Software Community} - -In the new SPB portal, each software has a standard set of pages and tools. -Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download -different versions of the software and find several mechanisms of software development management. - -Focusing on the collaborative development, Mailman was integrated to the platform in order to allow -the dialogue and communication between developers, users and -enthusiasts of a determined software. The software has its own mailing list whose privacy -can be configured/set by administrators. - -The software has a social interface area ("software community") where users can find other users, blogs, -summary of recent activities, or any other relevant community-produced content. -Users logged to the platform can request membership to different software communities -and each community member can access and edit restricted content. For this purpose, -many Noosfero features related to social networking and content management were integrated to the portal. - -To assist decision-making, the new SPB has acquired assessment and statistical -tools. Now, users will be able to rate the software and make comments and all -information will be avaiable to other users. Moreover, the software has a section -containing its statistical data, where values are calculated based on data -provided by users and the system. - -The role of the administrator will be present in the software and in its community. The -administrator is responsible for moderating content, memberships and user -comments. The administrator is also the one who can make changes in the software homepage -content. - -\subsection{Software Catalog and global search} - -The platform also provides a search tool called Software Catalog, -which allows users to find softwares available in the portal. -In this catalog, some search options were developed to make the navigation easier, -such as filters (by type of software or category), sorting and score. - -In order to expand the searching scope and cover more types of content, the SPB team -developed the global search tool. This tool unifies search mechanisms -provided by the different collaborative software used in the SPB protal. Any user can -find a public content in the context of social networking, mailing list and -software development. - -\subsection{Software development tools} - -The new SPB also provides -tools to encourage developers to keep source code and its -development activity within the platform. Any created software has, by default, a -an associated git repository with wiki pages and issue tracking. These tools are -supplied by the integration of Gitlab into the platform. - -Developers can also evaluate the software source code to measure software -quality. With Mezuro, they can schedule the analysis of the source-code and follow its -metric results evolution over time. Results of each metric analysis are -public, which allows greater transparency between the developer and the -community that uses the software. Thereby, the maintainers can decide if the -given solution meets the source code quality requirements. - -Thus, the SPB became a platform to stimulate the openness of the source code; -dialogue between users and the development team; and also -maintenance and evolution of the software, which will provide more -transparency in government investments. diff --git a/opensym2017/content/06-features.tex b/opensym2017/content/06-features.tex new file mode 100644 index 0000000..9767637 --- /dev/null +++ b/opensym2017/content/06-features.tex @@ -0,0 +1,73 @@ +\section{Features} +\label{sec:spb} + +The new generation of the SPB portal combines features adapted from existing collaborative software +and features developed by the SPB team. Whenever possible, new functions +(newly developed or partially modified) were contributed back to the official repositories. + +As a result, we have a platform that integrates and harmonizes different features such as +social networking, mailing list, version control system, content management and +source code quality monitoring. Our aim was to develop functionalities by reusing +functionality of collaborative software already integrated to the platform. In +addition, we tried to keep this integration transparent to end users. + +\subsection{Software and Software Community} + +In the new SPB portal, each software has a standard set of pages and tools. +Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download +different versions of the software and find several mechanisms of software development management. + +Focusing on the collaborative development, Mailman was integrated to the platform in order to allow +the dialogue and communication between developers, users and +enthusiasts of a determined software. The software has its own mailing list whose privacy +can be configured/set by administrators. + +The software has a social interface area ("software community") where users can find other users, blogs, +summary of recent activities, or any other relevant community-produced content. +Users logged to the platform can request membership to different software communities +and each community member can access and edit restricted content. For this purpose, +many Noosfero features related to social networking and content management were integrated to the portal. + +To assist decision-making, the new SPB has acquired assessment and statistical +tools. Now, users will be able to rate the software and make comments and all +information will be avaiable to other users. Moreover, the software has a section +containing its statistical data, where values are calculated based on data +provided by users and the system. + +The role of the administrator will be present in the software and in its community. The +administrator is responsible for moderating content, memberships and user +comments. The administrator is also the one who can make changes in the software homepage +content. + +\subsection{Software Catalog and global search} + +The platform also provides a search tool called Software Catalog, +which allows users to find softwares available in the portal. +In this catalog, some search options were developed to make the navigation easier, +such as filters (by type of software or category), sorting and score. + +In order to expand the searching scope and cover more types of content, the SPB team +developed the global search tool. This tool unifies search mechanisms +provided by the different collaborative software used in the SPB protal. Any user can +find a public content in the context of social networking, mailing list and +software development. + +\subsection{Software development tools} + +The new SPB also provides +tools to encourage developers to keep source code and its +development activity within the platform. Any created software has, by default, a +an associated git repository with wiki pages and issue tracking. These tools are +supplied by the integration of Gitlab into the platform. + +Developers can also evaluate the software source code to measure software +quality. With Mezuro, they can schedule the analysis of the source-code and follow its +metric results evolution over time. Results of each metric analysis are +public, which allows greater transparency between the developer and the +community that uses the software. Thereby, the maintainers can decide if the +given solution meets the source code quality requirements. + +Thus, the SPB became a platform to stimulate the openness of the source code; +dialogue between users and the development team; and also +maintenance and evolution of the software, which will provide more +transparency in government investments. diff --git a/opensym2017/content/06-ux.tex b/opensym2017/content/06-ux.tex deleted file mode 100644 index 3dc36da..0000000 --- a/opensym2017/content/06-ux.tex +++ /dev/null @@ -1,33 +0,0 @@ -\section{User eXperience evolution} - -The integration of collaborative environments goes beyond functional aspects. -Offering the population an unified experience across these environments has -been the key to encourage the use of the platform as it reduces the perception -of complexity. Thus, the SPB Portal information architecture was redesigned -to provide a transparent navigation and to reach users with different profiles. -A process of harmonization has been employed on the interaction models of each -tool to reduce the learning curve. At the same time, a new visual style was -created to unify the navigation experience and to comply with the guidelines of -the digital communication identity standard established by the Federal -Government. - -With the increase in system features and the addition of new tools, the -visual style has steadily evolved to keep the navigation unified. Moreover, -tools from different backgrounds, which in many cases provide similar -functionality, prompted the development of an unified interface. Some -features, such as search and user profile editing were eliminated from -the individual applications, and implemented centrally to ensure a -consistent look and feel. - -Another challenge was responsive web design. The integrated applications -had varying degrees of support for responsiveness, and the common -interface had to adapt for each individual scenario. In particular -Noosfero did not yet have a responsive design; we engaged in its -development and contributed towards that goal. - -After the initial release of the new SPB Portal in 2014, several -validations activities were implemented in 2015 and 2016. The aim was to -provide the most wanted features by casual users (such as public -servants interested in downloads and documentation) immediately, while -allowing more experienced users (such as developers) to easily drill down -to the details. diff --git a/opensym2017/content/07-process.tex b/opensym2017/content/07-process.tex deleted file mode 100644 index 1a0871b..0000000 --- a/opensym2017/content/07-process.tex +++ /dev/null @@ -1,174 +0,0 @@ -\section{Development Organization and Process} -\label{sec:process} - -The SPB team was composed of a variety of professionals with different levels -and skills, where most of them were undergraduate students with major in -software engineering (from 4th semester or upper). Since the students could -not dedicate many hours per week to the project, they always had the -flexibility to negotiate their work schedule during the semester in -order to not harm their classes and coursework. Their daily work routine -in the project included programming and DevOps tasks. - -The development of SPB project required a vast experience and background that -usually undergraduate students do not have yet. For this reason, a few senior -developers have joined to the project to help with the more difficult issues -and to transfer knowledge to the students. Their main task was to provide -solutions for complex problems, in other words, they worked as developers. As -these professionals are very skillful and the project could not fund a full -time work for them, some of them worked partially on the project. In addition, -they lived in a different states spread around Brazil which led much of the -communication to be online. - -In short, our work process was based on open and collaborative software -development practices. The development process was defined based on the -adaptation of different agile and FOSS communities practices, highlighting the -high degree of automation resulting from DevOps practices. Thus, the work -process was executed in a cadenced and continuous way. - -Finally, the last group of actors of this project was composed of employees -formally working for the Brazilian government, in the Ministry of Planning, -Development, and Management (MP is the Brazilian acronym). All the project -decisions, validations, and scope definitions were made by them. In this way we -developed software product incrementally, with releases aligned to business -strategic objectives. As one can see, the project had many different -kinds of stakeholders that had to be organized and synchronized. - -\subsection{Team organization} - -Approximately 70\% of the development teams were composed of software -engineering undergraduate students from UnB and they worked physically -in the same laboratory. Each student had their own schedule based on -their classes, what complicates the implementation of pair programming. -The senior developers tried to synchronize their schedule with the -students on their sub-team. To cope with this environment, we had a few -basic rules which guided the project organization: - -\begin{enumerate} - \item Classes have high priority for undergraduate students; - \item Work in pairs whenever possible (locally or remotely). - \item There must be one morning or afternoon per week when - \emph{everyone} should be together physically in the laboratory - (except of course the remote team members). - \item Every 3 to 4 months, the senior developers would fly in and work - alongside the students for a few days. -\end{enumerate} - -With the aforementioned rules we divided all the project into four -different teams: Colab, Noosfero, Design, and DevOps. Each team had one -coach responsible for reducing the communication problem with the other -teams and help the members to organize themselves in the best way for -everyone (always respecting their work time). The coach was one of the -students working in some of the teams with the extra duty of registering -the current tasks developed in the sprint and with the responsibility to -talk with other teams. One important thing to notice is the mutability -of the team and the coach, during the project many students changed -between the teams to try different areas. - -One characteristic of the teams was the presence of (at least) one -senior per team. This was essential, because hard decisions and complex -problems were usually referred to them. This kept having to take -complicated technical decisions from the coach tole, what encouraged -students to be coaches. Lastly, the senior developers worked directly -with the students, and this was important to give the undergraduate the -opportunity to interact with a savvy professional in his area and to -keep the knowledge flowing in the project. - -Finally, we had to add two last elements of the team organization, that -was essential for the project harmony: the meta-coach and professors. -The former was a software engineer recently graduated and which wanted -to keep working on the project, the latter were professors that -orchestrated all the interactions between all members of the project. -The meta-coach usually worked in one specific team and had the extra -task of knowing the current status of all teams. Professors and -meta-coaches worked together to reduce the communication problem between -all the teams. Lastly, all the paperwork tasks, such as reporting on the -project progress to the Ministry, was handled by the professors. - -\subsection{Communication and management} - -Our team had many people working together, and most of the seniors worked in a -different city remotely. Also, we tried to keep our work completely clear to -the Brazilian government and citizens interested in following the project. To -handle those cases, we used a set of tools to communication and other to manage -the project. - -For communication between members in different places, we used: video -conferencing with shared terminal tools, IRC, and mailing lists. For example, -when one student had to work in pair with a senior, normally, they used video -conferencing tool for talking and shared a terminal session (both typing and -seeing the screen in real time). For questions and fast discussion, we used -IRC. For general notification, we used the mailing lists. - -For managing the project we used the SPB Portal itself; first to validate it by -ourselves, and also because it had all the required tools. We basically created -one wiki page per release in the SPB Gitlab instance with a mapping between -strategical, tactical, and operational views. In a practical point of view, one -milestone per user history (feature), and one or more issues for addressing -each feature. With this approach we achieved two important goals: keeping all -the management as close possible to to the source code, and tracking every -feature developed during the project. - -\subsection{High-level project management and reporting} - -The Brazilian government used to work with software development in a -very traditional way. They would frequently focus on documents and not -on what was, in our opinion, what really matters: working software. This -dissonance caused us a communication noise with MP, because they would -often question our work style. It was especially hard to convince them -to accept the idea of open scope and agile development, but after months -of labor and showing results they stopped resisting. - -We defined some level of meeting granularity to avoid generating too -much overhead to the developers. We had a strategical and validating -meeting with MP (the former once in a month and the latter each 15th -day), release plaining with the entire team (one per month), and finally -a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a -diagram that represents our meeting organization. - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/meeting_flows.png} - \caption{Meetings cycles} - \label{fig:meeting} -\end{figure} - -In the strategical meeting we usually defined the priorities and new -features with the Brazilian government. Normally the professors, the -coach of each team, the meta-coach, and some employees of the MP would -participate in this meeting. We usually discussed what the team already -produced since our last meeting, and established the new features for -the next release. Notice that just part of the team would join this -meeting to avoid generating unnecessary overhead to the developers, but -all the students interested to participate were allowed to join (many -students wanted this experience during the project). - -After the strategical meeting with Brazilian government agents, we had a -planning turn with all teams together. In this part, each team worked together -to convert the MP wishes into smaller parts which were represented by the epics of -the release. Each coach was responsible for conducting the planning, and after -that record it on the project wiki (the wiki provided by Gitlab). With this -epic, each 14th day the team have generated their sprint scheduler (with small -achievements mapped to issues). - -To keep the Brazilian government always updated, we invited them to work -with us to validate the new features in progress. Normally we had a -meeting each 15th day. Basically, this was our work flow, we always kept -everything extremely open to the MP (our way of working, and the one -often used by open source projects) and to the team. - -To keep the track of all of those things we used the SPB itself, -especially Gitlab. Basically, we had: - -\begin{enumerate} - \item Project repository: We have one organization with many repositories - \item Milestones: Each milestone was used to register a release - \item Wiki: Each release has one page on wiki with the compilation of - strategical meeting - \item Issues: Each sprint planning generated issues, which we associated to - the specific milestone and updated the wiki with the issue number related - with them. Finally each developer assigned the issue to itself. -\end{enumerate} - -Notice that this workflow gave us and the Brazilian government agents -full traceability from a high level view of each feature to the lowest -level (code). diff --git a/opensym2017/content/07-ux.tex b/opensym2017/content/07-ux.tex new file mode 100644 index 0000000..3dc36da --- /dev/null +++ b/opensym2017/content/07-ux.tex @@ -0,0 +1,33 @@ +\section{User eXperience evolution} + +The integration of collaborative environments goes beyond functional aspects. +Offering the population an unified experience across these environments has +been the key to encourage the use of the platform as it reduces the perception +of complexity. Thus, the SPB Portal information architecture was redesigned +to provide a transparent navigation and to reach users with different profiles. +A process of harmonization has been employed on the interaction models of each +tool to reduce the learning curve. At the same time, a new visual style was +created to unify the navigation experience and to comply with the guidelines of +the digital communication identity standard established by the Federal +Government. + +With the increase in system features and the addition of new tools, the +visual style has steadily evolved to keep the navigation unified. Moreover, +tools from different backgrounds, which in many cases provide similar +functionality, prompted the development of an unified interface. Some +features, such as search and user profile editing were eliminated from +the individual applications, and implemented centrally to ensure a +consistent look and feel. + +Another challenge was responsive web design. The integrated applications +had varying degrees of support for responsiveness, and the common +interface had to adapt for each individual scenario. In particular +Noosfero did not yet have a responsive design; we engaged in its +development and contributed towards that goal. + +After the initial release of the new SPB Portal in 2014, several +validations activities were implemented in 2015 and 2016. The aim was to +provide the most wanted features by casual users (such as public +servants interested in downloads and documentation) immediately, while +allowing more experienced users (such as developers) to easily drill down +to the details. diff --git a/opensym2017/content/08-contributions.tex b/opensym2017/content/08-contributions.tex deleted file mode 100644 index 2801e8d..0000000 --- a/opensym2017/content/08-contributions.tex +++ /dev/null @@ -1,51 +0,0 @@ -\section{Contributions to the upstream communities} -\label{sec:contributions} - -%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) -%* Colab -> RevProxy -%* Colab, atualização do python/django -%* Contribuições para o GitLab (autenticação) -%* Noosfero, atualização do Rails, preparação para federação, nova interface ... -%* Coper, empacotamentos (obs), omniauth - - -During the execution of this project we made several contributions to -the upstream communities we interacted with. This occurred due to our -development process aligned with those of the respective communities. -During development, we would explicitly discuss the features and bug -fixes that we were working on with the applicable upstream communities. -This contributed to improve the developers technical solutions with -expertise outside of our team, and make it easier for those changes to -be accepted in the original projects. Having changes accepted upstream -in turn makes our life easier as it minimizes the delta between our -codebase and upstream's allowing us to upgrade and benefit from -development work from others. - -In Colab, we helped upstream redesign the entirely architecture, -enabling the development of plugins to integrate new tools. We also -added a feature that allowed Colab to run asynchronous tasks, which was -a major improvement for us since we were developing a complex system. A -migration to the latest Django version was made (web framework used by -Colab). Moreover, we worked on RevProxy (the more important dependency -of Colab) to put it in a good shape, fixing many bugs. - -Gitlab was the tool that we made the least number of modifications. We -contributed with some improvements related with configuration files and -we developed a new plugin that enables user authentication in Gitlab -through the REMOTE\_USER HTTP header. This plugin was needed because -Colab uses this mechanism to manage the authentication. - -Noosfero was the tool that contemplated several functional requirements, -therefore we made a large number of contributions with upstream. We helped to -migrate to the latest Rails version (web framework used by Noosfero), enable -the federation implementation (federation with other social networks), and -decouple the interface and the back-end. - -We also made some contributions on the DevOps front. Some members of -them team became maintainers of some python libraries that were used by -our scripts to upload packages to OBS (Open Build Service). We developed -a tool called copr-status to keep track of the different stages of the -lifecycle of each of the individual packages we were working on. - -%TODO: Mezuro - diff --git a/opensym2017/content/08-process.tex b/opensym2017/content/08-process.tex new file mode 100644 index 0000000..1a0871b --- /dev/null +++ b/opensym2017/content/08-process.tex @@ -0,0 +1,174 @@ +\section{Development Organization and Process} +\label{sec:process} + +The SPB team was composed of a variety of professionals with different levels +and skills, where most of them were undergraduate students with major in +software engineering (from 4th semester or upper). Since the students could +not dedicate many hours per week to the project, they always had the +flexibility to negotiate their work schedule during the semester in +order to not harm their classes and coursework. Their daily work routine +in the project included programming and DevOps tasks. + +The development of SPB project required a vast experience and background that +usually undergraduate students do not have yet. For this reason, a few senior +developers have joined to the project to help with the more difficult issues +and to transfer knowledge to the students. Their main task was to provide +solutions for complex problems, in other words, they worked as developers. As +these professionals are very skillful and the project could not fund a full +time work for them, some of them worked partially on the project. In addition, +they lived in a different states spread around Brazil which led much of the +communication to be online. + +In short, our work process was based on open and collaborative software +development practices. The development process was defined based on the +adaptation of different agile and FOSS communities practices, highlighting the +high degree of automation resulting from DevOps practices. Thus, the work +process was executed in a cadenced and continuous way. + +Finally, the last group of actors of this project was composed of employees +formally working for the Brazilian government, in the Ministry of Planning, +Development, and Management (MP is the Brazilian acronym). All the project +decisions, validations, and scope definitions were made by them. In this way we +developed software product incrementally, with releases aligned to business +strategic objectives. As one can see, the project had many different +kinds of stakeholders that had to be organized and synchronized. + +\subsection{Team organization} + +Approximately 70\% of the development teams were composed of software +engineering undergraduate students from UnB and they worked physically +in the same laboratory. Each student had their own schedule based on +their classes, what complicates the implementation of pair programming. +The senior developers tried to synchronize their schedule with the +students on their sub-team. To cope with this environment, we had a few +basic rules which guided the project organization: + +\begin{enumerate} + \item Classes have high priority for undergraduate students; + \item Work in pairs whenever possible (locally or remotely). + \item There must be one morning or afternoon per week when + \emph{everyone} should be together physically in the laboratory + (except of course the remote team members). + \item Every 3 to 4 months, the senior developers would fly in and work + alongside the students for a few days. +\end{enumerate} + +With the aforementioned rules we divided all the project into four +different teams: Colab, Noosfero, Design, and DevOps. Each team had one +coach responsible for reducing the communication problem with the other +teams and help the members to organize themselves in the best way for +everyone (always respecting their work time). The coach was one of the +students working in some of the teams with the extra duty of registering +the current tasks developed in the sprint and with the responsibility to +talk with other teams. One important thing to notice is the mutability +of the team and the coach, during the project many students changed +between the teams to try different areas. + +One characteristic of the teams was the presence of (at least) one +senior per team. This was essential, because hard decisions and complex +problems were usually referred to them. This kept having to take +complicated technical decisions from the coach tole, what encouraged +students to be coaches. Lastly, the senior developers worked directly +with the students, and this was important to give the undergraduate the +opportunity to interact with a savvy professional in his area and to +keep the knowledge flowing in the project. + +Finally, we had to add two last elements of the team organization, that +was essential for the project harmony: the meta-coach and professors. +The former was a software engineer recently graduated and which wanted +to keep working on the project, the latter were professors that +orchestrated all the interactions between all members of the project. +The meta-coach usually worked in one specific team and had the extra +task of knowing the current status of all teams. Professors and +meta-coaches worked together to reduce the communication problem between +all the teams. Lastly, all the paperwork tasks, such as reporting on the +project progress to the Ministry, was handled by the professors. + +\subsection{Communication and management} + +Our team had many people working together, and most of the seniors worked in a +different city remotely. Also, we tried to keep our work completely clear to +the Brazilian government and citizens interested in following the project. To +handle those cases, we used a set of tools to communication and other to manage +the project. + +For communication between members in different places, we used: video +conferencing with shared terminal tools, IRC, and mailing lists. For example, +when one student had to work in pair with a senior, normally, they used video +conferencing tool for talking and shared a terminal session (both typing and +seeing the screen in real time). For questions and fast discussion, we used +IRC. For general notification, we used the mailing lists. + +For managing the project we used the SPB Portal itself; first to validate it by +ourselves, and also because it had all the required tools. We basically created +one wiki page per release in the SPB Gitlab instance with a mapping between +strategical, tactical, and operational views. In a practical point of view, one +milestone per user history (feature), and one or more issues for addressing +each feature. With this approach we achieved two important goals: keeping all +the management as close possible to to the source code, and tracking every +feature developed during the project. + +\subsection{High-level project management and reporting} + +The Brazilian government used to work with software development in a +very traditional way. They would frequently focus on documents and not +on what was, in our opinion, what really matters: working software. This +dissonance caused us a communication noise with MP, because they would +often question our work style. It was especially hard to convince them +to accept the idea of open scope and agile development, but after months +of labor and showing results they stopped resisting. + +We defined some level of meeting granularity to avoid generating too +much overhead to the developers. We had a strategical and validating +meeting with MP (the former once in a month and the latter each 15th +day), release plaining with the entire team (one per month), and finally +a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a +diagram that represents our meeting organization. + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/meeting_flows.png} + \caption{Meetings cycles} + \label{fig:meeting} +\end{figure} + +In the strategical meeting we usually defined the priorities and new +features with the Brazilian government. Normally the professors, the +coach of each team, the meta-coach, and some employees of the MP would +participate in this meeting. We usually discussed what the team already +produced since our last meeting, and established the new features for +the next release. Notice that just part of the team would join this +meeting to avoid generating unnecessary overhead to the developers, but +all the students interested to participate were allowed to join (many +students wanted this experience during the project). + +After the strategical meeting with Brazilian government agents, we had a +planning turn with all teams together. In this part, each team worked together +to convert the MP wishes into smaller parts which were represented by the epics of +the release. Each coach was responsible for conducting the planning, and after +that record it on the project wiki (the wiki provided by Gitlab). With this +epic, each 14th day the team have generated their sprint scheduler (with small +achievements mapped to issues). + +To keep the Brazilian government always updated, we invited them to work +with us to validate the new features in progress. Normally we had a +meeting each 15th day. Basically, this was our work flow, we always kept +everything extremely open to the MP (our way of working, and the one +often used by open source projects) and to the team. + +To keep the track of all of those things we used the SPB itself, +especially Gitlab. Basically, we had: + +\begin{enumerate} + \item Project repository: We have one organization with many repositories + \item Milestones: Each milestone was used to register a release + \item Wiki: Each release has one page on wiki with the compilation of + strategical meeting + \item Issues: Each sprint planning generated issues, which we associated to + the specific milestone and updated the wiki with the issue number related + with them. Finally each developer assigned the issue to itself. +\end{enumerate} + +Notice that this workflow gave us and the Brazilian government agents +full traceability from a high level view of each feature to the lowest +level (code). diff --git a/opensym2017/content/09-contributions.tex b/opensym2017/content/09-contributions.tex new file mode 100644 index 0000000..2801e8d --- /dev/null +++ b/opensym2017/content/09-contributions.tex @@ -0,0 +1,51 @@ +\section{Contributions to the upstream communities} +\label{sec:contributions} + +%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) +%* Colab -> RevProxy +%* Colab, atualização do python/django +%* Contribuições para o GitLab (autenticação) +%* Noosfero, atualização do Rails, preparação para federação, nova interface ... +%* Coper, empacotamentos (obs), omniauth + + +During the execution of this project we made several contributions to +the upstream communities we interacted with. This occurred due to our +development process aligned with those of the respective communities. +During development, we would explicitly discuss the features and bug +fixes that we were working on with the applicable upstream communities. +This contributed to improve the developers technical solutions with +expertise outside of our team, and make it easier for those changes to +be accepted in the original projects. Having changes accepted upstream +in turn makes our life easier as it minimizes the delta between our +codebase and upstream's allowing us to upgrade and benefit from +development work from others. + +In Colab, we helped upstream redesign the entirely architecture, +enabling the development of plugins to integrate new tools. We also +added a feature that allowed Colab to run asynchronous tasks, which was +a major improvement for us since we were developing a complex system. A +migration to the latest Django version was made (web framework used by +Colab). Moreover, we worked on RevProxy (the more important dependency +of Colab) to put it in a good shape, fixing many bugs. + +Gitlab was the tool that we made the least number of modifications. We +contributed with some improvements related with configuration files and +we developed a new plugin that enables user authentication in Gitlab +through the REMOTE\_USER HTTP header. This plugin was needed because +Colab uses this mechanism to manage the authentication. + +Noosfero was the tool that contemplated several functional requirements, +therefore we made a large number of contributions with upstream. We helped to +migrate to the latest Rails version (web framework used by Noosfero), enable +the federation implementation (federation with other social networks), and +decouple the interface and the back-end. + +We also made some contributions on the DevOps front. Some members of +them team became maintainers of some python libraries that were used by +our scripts to upload packages to OBS (Open Build Service). We developed +a tool called copr-status to keep track of the different stages of the +lifecycle of each of the individual packages we were working on. + +%TODO: Mezuro + diff --git a/opensym2017/content/09-lessons.tex b/opensym2017/content/09-lessons.tex deleted file mode 100644 index dd3839d..0000000 --- a/opensym2017/content/09-lessons.tex +++ /dev/null @@ -1,161 +0,0 @@ -\section{Lessons Learned} -\label{sec:lessons} - -\textbf{How to involve students real-world projects, interacting with -real customers.} -% -Our team was mainly composed of software engineering undergraduate -student, who had the opportunity to interact with senior developers and -designers inside the team, as well as with the team of technicians and -managers from the Brazilian Government, and the management team from -UnB. -% -They interacted with professionals that had diverse expertises, and were -able to participate in all levels of the software development process. -This contribted to a great learning opportunity, and for a majority of -them this was their first professional experience. - -\textbf{The participation of experienced professionals is crucial to -success of the project.} -One important factor for the students was the composition of the teams -with the participation of experienced professionals. -% -On the technical side, the senior developers and designers would handle -the more difficult technical decisions, creating a work environment -where the students could develop their skills in a didactic way without -pressure. -% -On the management side, the active participation of professors -- who -are, in the end, the ones responsible for the project -- is crucial to -make sure student participation is conducted in a healthy way, and is an -instance of leading by example. - -\textbf{A balanced relationship between academia and industry.} -The experience of the SPB project led UnB to develop a work style which -proved to be appropriate for an educational environment that brings -academia and industry together. -% -The highest priority from the university's point of view is the -students. Considering this, the activities of the project were never -prioritized to the detriment of classes and other pedagogical -activities. In summary, we had students working at different times, part -time, remotely or locally, always respecting their individual -conditions, but doing the work in a collective, collaborative and open -way. -% -And even under a potentially adverse environment, the project delivered -the desired solution with success. -% -At the end of the project, we noted that the skills developed by the -students were at the state of art of the software engineering practice. -After the project ended, we had team members successfully embracing -opportunities in public, private, national and international -organizations, in addition to those students that went into -entrepreneurship and opened their own companies. - -\textbf{Managing different organizational cultures.} -In the beginning of the project, the Brazilian Government stakeholders -had certain expectations about the development of project that, let's -say, didn't exactly match our work based on agile and FOSS practices. -% -We had to develop strategies that would support different these -organizational cultures. As reported in Section \ref{sec:process}, the -MP is organized in a functional hierarchical organizational structure, -typically, a traditional development paradigm. Therefore, we had to -create a translation process between our team and the MP managers who -managed the project on their side assuming a traditional, waterfall -process. - -\textbf{Manage higher-level and lower-level goals separately.} -During the initial phase of the project the MP team would often bring -strategic discussions to technical/operational meetings, where we were -supposed to discuss practical, technical decisions. -% -This produced a highly complex communication and management environment, -overloading the professors because they were supposed to be responsible -for maintaining the strategic alignment of the MP synchronized with -implementation plans of the development team, specially in light of the -aforementioned culture mismatch. Mixing both concerns in the same -discussions caused confusion on both sides. -% -Towards the middle and end of the project we were able to keep those -concerns separated, what eased the work of everyone involved. - -\textbf{Living with ill-advised political decisions.} -At the initial phases of the project, by political and personal -motivation, the main stakeholders from the Brazilian government imposed -the use of Colab to guide the development of the new SPB platform. Our -team was totally against the idea because we already knew that Colab was -a very experimental project and its adoption could dramatically increase -the project complexity. Even though we provided technical reasons to -not utilize Colab, MP was adamant and we had to manage this problem. As -explained in section \ref{sec:architecture} we did massive changes to -it, and at the end of the project we completely rewrote Colab and made -it stable. It is important to notice that the MP compelled us to accept -a technical issue based only on political interests, without considering -all the resources that would be spent due to that decision. At the end -of the project, we verified that Colab indeed consumed a vast amount of -the budget and increased the project complexity. In the end of the -project and after our analysis on the decision made by the MP, we -understand that MP's managers are capable of ignoring technical reasons -in favor to a political decision. - -\textbf{Consider sustainability from the beginning.} -In the process of deploying the SPB platform in the MP structure, we had -to interact with the MP technicians. We did several workshops, training -and a meticulous documentation describing all the required procedures to -update the platform, however, we realized that the MP technicians -constantly avoid the responsibility. After noticing the aforementioned -situation, we organized a specific DevOps that completely automated all -the deployment procedure. We simplified all the platform deployment to a -few steps: (1) initial configurations (just ssh configuration) and (2) -the execution of simple commands to completely update the platform. By -the end of the project, we observed that the MP technicians invariably -still depended on our support to update the platform even with all the -automation provided by us. We were sadly left with a feeling of -uncertainty about the future of the platform after the project ended. In -hindsight, we realize that the MP dedicated systems analysts and -managers to the project, but not operations technicians. The later -should have been more involved with the process so they could at least -comfortable with managing the platform infrastructure. - -% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados -%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa -%% afirmação, embora eu e Paulo consigamos perceber isso. - -%=========== -% Conclusion -%=========== - -\textbf{Final remarks.} -The portal is available at \url{softwarepublico.gov.br}. All -documentation, including detailed architecture and operation manuals are -also available\footnote{\url{https://softwarepublico.gov.br/doc/} -(in Portuguese only at the moment)}). -% -All the integrated tools are FOSS and our contributions were published -in open repositories, available on the SPB Portal itself. We also -contributed these features back to the respective communities: that -benefits those communities, as well as us since we can share future -development and maintenance effort with other organizations that -participate in their projects. - - -%* utilização do projeto para formação de recursos humanos (alunos) - -%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate - -%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) - -%* 69 projetos marcados como SPB, de 81 no total na plataforma. - -%* 47\% é desenvolvido em PHP. - -% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). - -% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. - -% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. - -%* Debater economia de recursos em orgão públicos - diff --git a/opensym2017/content/10-conclusion.tex b/opensym2017/content/10-conclusion.tex deleted file mode 100644 index fe65a0b..0000000 --- a/opensym2017/content/10-conclusion.tex +++ /dev/null @@ -1,26 +0,0 @@ -\section{Conclusion} -\label{sec:conclusion} - -In this paper we present and discuss issues experienced during a government- -funded project, in partnership with University of Brasilia and University of -São Paulo, to evolve the Brazilian Public Software portal. - -The contributions of this paper are twofold. First, we present how was -developed an unprecedent platform, delivered to Brazilian government. This -platform - developed by an heterogenous team of professors, master and -undergraduate students, IT professionals and governmental managers - provides -several modern features from the integration of more than 10 FLOSS systems. - -Second, the 30 months project which developed this platform, results in an -important case that it is possible to mitigate issues seen as conflicting to IT -development environment and between industry and academy. We shown that, as -long as the institution can provide a healthy and challenging environment to -its students, its is possible to conciliate studies and professional training -in universities. After the end of the project, some students successfully -embraced opportunities in public and private sectos, within national borders -and abroad. Some others went further and started their own companies. - -We also shown that, with some adaptations/"translation processes", was possible -to conciliate agile methodologies and FOSS practices to develop software to -governmental organizations with functional hierarchical structures that use -traditional development paradigm. diff --git a/opensym2017/content/10-lessons.tex b/opensym2017/content/10-lessons.tex new file mode 100644 index 0000000..dd3839d --- /dev/null +++ b/opensym2017/content/10-lessons.tex @@ -0,0 +1,161 @@ +\section{Lessons Learned} +\label{sec:lessons} + +\textbf{How to involve students real-world projects, interacting with +real customers.} +% +Our team was mainly composed of software engineering undergraduate +student, who had the opportunity to interact with senior developers and +designers inside the team, as well as with the team of technicians and +managers from the Brazilian Government, and the management team from +UnB. +% +They interacted with professionals that had diverse expertises, and were +able to participate in all levels of the software development process. +This contribted to a great learning opportunity, and for a majority of +them this was their first professional experience. + +\textbf{The participation of experienced professionals is crucial to +success of the project.} +One important factor for the students was the composition of the teams +with the participation of experienced professionals. +% +On the technical side, the senior developers and designers would handle +the more difficult technical decisions, creating a work environment +where the students could develop their skills in a didactic way without +pressure. +% +On the management side, the active participation of professors -- who +are, in the end, the ones responsible for the project -- is crucial to +make sure student participation is conducted in a healthy way, and is an +instance of leading by example. + +\textbf{A balanced relationship between academia and industry.} +The experience of the SPB project led UnB to develop a work style which +proved to be appropriate for an educational environment that brings +academia and industry together. +% +The highest priority from the university's point of view is the +students. Considering this, the activities of the project were never +prioritized to the detriment of classes and other pedagogical +activities. In summary, we had students working at different times, part +time, remotely or locally, always respecting their individual +conditions, but doing the work in a collective, collaborative and open +way. +% +And even under a potentially adverse environment, the project delivered +the desired solution with success. +% +At the end of the project, we noted that the skills developed by the +students were at the state of art of the software engineering practice. +After the project ended, we had team members successfully embracing +opportunities in public, private, national and international +organizations, in addition to those students that went into +entrepreneurship and opened their own companies. + +\textbf{Managing different organizational cultures.} +In the beginning of the project, the Brazilian Government stakeholders +had certain expectations about the development of project that, let's +say, didn't exactly match our work based on agile and FOSS practices. +% +We had to develop strategies that would support different these +organizational cultures. As reported in Section \ref{sec:process}, the +MP is organized in a functional hierarchical organizational structure, +typically, a traditional development paradigm. Therefore, we had to +create a translation process between our team and the MP managers who +managed the project on their side assuming a traditional, waterfall +process. + +\textbf{Manage higher-level and lower-level goals separately.} +During the initial phase of the project the MP team would often bring +strategic discussions to technical/operational meetings, where we were +supposed to discuss practical, technical decisions. +% +This produced a highly complex communication and management environment, +overloading the professors because they were supposed to be responsible +for maintaining the strategic alignment of the MP synchronized with +implementation plans of the development team, specially in light of the +aforementioned culture mismatch. Mixing both concerns in the same +discussions caused confusion on both sides. +% +Towards the middle and end of the project we were able to keep those +concerns separated, what eased the work of everyone involved. + +\textbf{Living with ill-advised political decisions.} +At the initial phases of the project, by political and personal +motivation, the main stakeholders from the Brazilian government imposed +the use of Colab to guide the development of the new SPB platform. Our +team was totally against the idea because we already knew that Colab was +a very experimental project and its adoption could dramatically increase +the project complexity. Even though we provided technical reasons to +not utilize Colab, MP was adamant and we had to manage this problem. As +explained in section \ref{sec:architecture} we did massive changes to +it, and at the end of the project we completely rewrote Colab and made +it stable. It is important to notice that the MP compelled us to accept +a technical issue based only on political interests, without considering +all the resources that would be spent due to that decision. At the end +of the project, we verified that Colab indeed consumed a vast amount of +the budget and increased the project complexity. In the end of the +project and after our analysis on the decision made by the MP, we +understand that MP's managers are capable of ignoring technical reasons +in favor to a political decision. + +\textbf{Consider sustainability from the beginning.} +In the process of deploying the SPB platform in the MP structure, we had +to interact with the MP technicians. We did several workshops, training +and a meticulous documentation describing all the required procedures to +update the platform, however, we realized that the MP technicians +constantly avoid the responsibility. After noticing the aforementioned +situation, we organized a specific DevOps that completely automated all +the deployment procedure. We simplified all the platform deployment to a +few steps: (1) initial configurations (just ssh configuration) and (2) +the execution of simple commands to completely update the platform. By +the end of the project, we observed that the MP technicians invariably +still depended on our support to update the platform even with all the +automation provided by us. We were sadly left with a feeling of +uncertainty about the future of the platform after the project ended. In +hindsight, we realize that the MP dedicated systems analysts and +managers to the project, but not operations technicians. The later +should have been more involved with the process so they could at least +comfortable with managing the platform infrastructure. + +% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados +%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa +%% afirmação, embora eu e Paulo consigamos perceber isso. + +%=========== +% Conclusion +%=========== + +\textbf{Final remarks.} +The portal is available at \url{softwarepublico.gov.br}. All +documentation, including detailed architecture and operation manuals are +also available\footnote{\url{https://softwarepublico.gov.br/doc/} +(in Portuguese only at the moment)}). +% +All the integrated tools are FOSS and our contributions were published +in open repositories, available on the SPB Portal itself. We also +contributed these features back to the respective communities: that +benefits those communities, as well as us since we can share future +development and maintenance effort with other organizations that +participate in their projects. + + +%* utilização do projeto para formação de recursos humanos (alunos) + +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate + +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) + +%* 69 projetos marcados como SPB, de 81 no total na plataforma. + +%* 47\% é desenvolvido em PHP. + +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). + +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. + +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. + +%* Debater economia de recursos em orgão públicos + diff --git a/opensym2017/content/11-conclusion.tex b/opensym2017/content/11-conclusion.tex new file mode 100644 index 0000000..fe65a0b --- /dev/null +++ b/opensym2017/content/11-conclusion.tex @@ -0,0 +1,26 @@ +\section{Conclusion} +\label{sec:conclusion} + +In this paper we present and discuss issues experienced during a government- +funded project, in partnership with University of Brasilia and University of +São Paulo, to evolve the Brazilian Public Software portal. + +The contributions of this paper are twofold. First, we present how was +developed an unprecedent platform, delivered to Brazilian government. This +platform - developed by an heterogenous team of professors, master and +undergraduate students, IT professionals and governmental managers - provides +several modern features from the integration of more than 10 FLOSS systems. + +Second, the 30 months project which developed this platform, results in an +important case that it is possible to mitigate issues seen as conflicting to IT +development environment and between industry and academy. We shown that, as +long as the institution can provide a healthy and challenging environment to +its students, its is possible to conciliate studies and professional training +in universities. After the end of the project, some students successfully +embraced opportunities in public and private sectos, within national borders +and abroad. Some others went further and started their own companies. + +We also shown that, with some adaptations/"translation processes", was possible +to conciliate agile methodologies and FOSS practices to develop software to +governmental organizations with functional hierarchical structures that use +traditional development paradigm. diff --git a/opensym2017/spb.tex b/opensym2017/spb.tex index 79fb3d0..18fb0d1 100644 --- a/opensym2017/spb.tex +++ b/opensym2017/spb.tex @@ -152,14 +152,15 @@ %------------------------------------------------------------------------------ \input{content/01-introduction} \input{content/02-spb} -\input{content/03-requirements} -\input{content/04-architecture} -\input{content/05-features} -\input{content/06-ux} -\input{content/07-process} -\input{content/08-contributions} -\input{content/09-lessons} -\input{content/10-conclusion} +\input{content/03-relatedworks} +\input{content/04-requirements} +\input{content/05-architecture} +\input{content/06-features} +\input{content/07-ux} +\input{content/08-process} +\input{content/09-contributions} +\input{content/10-lessons} +\input{content/11-conclusion} %------------------------------------------------------------------------------ \bibliographystyle{SIGCHI-Reference-Format} -- libgit2 0.21.2