diff --git a/sbqs2017/content/00-abstract.tex b/sbqs2017/content/00-abstract.tex index ba48bfa..ed6839d 100644 --- a/sbqs2017/content/00-abstract.tex +++ b/sbqs2017/content/00-abstract.tex @@ -3,7 +3,7 @@ The Brazilian Public Software (SPB) is a program by the Brazilian Federal Government to foster the sharing and collaboration on Free and Open Source Software (FOSS) solutions for the public administration. In the one hand, -Brazilian Public Softwares have some differences from FOSS projects, in +Brazilian Public Software have some differences from FOSS projects, in particular, the software is considered a public good and the Federal government assumes some responsibilities related to its use. In the other hand, the software development principles are the same: the trend towards @@ -22,8 +22,27 @@ the platform architecture, features, and the user experience efforts carried out, we also discuss our work process, based on agile and free software development practices, and the lessons learned in 30 months work on the SPB project. +\end{abstract} -\textbf{Keywords:} Brazilian Public Software, Free/Libre/Open Source Software, - Software Evolution, Integrated Platform +\begin{resumo} +O Software Público Brasileiro (SPB) é um programa do Governo Federal brasileiro +para promover o compartilhamento e a colaboração de projetos de Software Livre +para a administração pública. Por um lado, um software público brasileiro tem +algumas diferenças em relação a um software livre, em especial, o fato do +software público ser considerado com um bem público and o Governo Federal +assumir algumas responsabilidades relacionadas ao seu uso. Por outro lado, +teoricamente, os princípios de desenvolvimento de software são os mesmos: +tendência a descentralização das decisões sobre o projeto, compartilhamento de +informações e do desenvolvimento (código), e interação contínua com seus +usuários. +% +Neste contexto, projetamos uma plataforma baseada na evolução e integração de +ferramentas existentes que promovem a colaboração. Atualmente, o portal SPB +disponibiliza funcionalidaeds modernas para o desenvolvimento colaborativo de +software, ajudando a administração pública brasileira a compartilhar suas +solução. +% -\end{abstract} + + +\end{resumo} diff --git a/sbqs2017/content/01-introduction.tex b/sbqs2017/content/01-introduction.tex index f0b6c1b..13a8e30 100644 --- a/sbqs2017/content/01-introduction.tex +++ b/sbqs2017/content/01-introduction.tex @@ -1,50 +1,92 @@ \section{Introduction} \label{sec:intro} -During the last few decades, the Brazilian Federal Government has -improved its software adoption and development processes. In 2003, the -recommendation to adopt Free/Open Source Software (FOSS) become a public -policy. In 2007, the Brazilian Government released a portal called -Brazilian Public Software (\textit{Software Público Brasileiro} -- SPB, -in Portuguese), with the goal of sharing FOSS projects developed by, or -for, the Brazilian Government. - -The Brazilian legal instrument on software contracting -(\textit{Instrução Normativa} 04/2012) mandates that public management -must consult the SPB Portal to adopt a software solution. The +During the last few decades, the Brazilian Federal Government has been +trying to change its software adoption and development processes. For +instance, in 2003, the recommendation to adopt Free and Open Source +Software (FOSS) become a public policy. In 2007, the Brazilian +Government released a portal named Brazilian Public Software +(\textit{Software Público Brasileiro} -- SPB, in Portuguese), with the +goal of sharing FOSS projects developed by, or for, the Brazilian +Government. Additionally, the Brazilian legal instrument on software +contracting (known as IN 04/2012) mandates that public agents must give +priority to solutions available in the SPB Portal. In short, the acquisition of a proprietary solution must be explicitly justified by -demonstrating that there is no suitable option in the SPB Portal. - -Since 2009, however, the SPB Portal was having several technical issues. -The original codebase was not being developed anymore, and there was a -large amount of technical debt to overcome. The system was a modified -version of an existing FOSS platform that was not being developed -anymore, and the portal maintenance was becoming harder and harder. - -From January 2014 to June 2016, a new platform for the SPB Portal was -designed and developed by the University of Brasília (UnB) and the -University of São Paulo (USP) in a partnership with the Brazilian -Ministry of Budget, Planning, and Management(MP). This new Portal was -designed as an integrated platform for collaborative software -development. It includes functionality for social networking, mailing -lists, version control system, and source code quality monitoring. In -this paper, we present an overview of this new generation of the SPB -Portal. - -The project was developed by a team of 3 professors, 6 professionals, 2 -masters students, and approximately 40 undergraduate students (not all of -them at the same time, though -- graduations and other events triggered -changes in the team). +demonstrating that there is no suitable alternative in the SPB Portal. +In 2013, the Brazilian Federal Court issued a ruling document +(\textit{Acórdão 2314/2013}) about an audit survey regarding the use of +agile methodologies in software development contracts with the public +administration. + +Despite of that, in practice, FOSS or agile methodologies, that is, +collaborative and empirical software development methods are not widely +practiced and understood by the Brazilian government agents. Thus, the +hierarchical and traditional processes from the government and the lack +of expertise in real-world software development of its agents produces a +situation of inneficient software development contracts and +unjustifiable expending of taxpayers' money. + +% TODO: ^ references + +Since 2009, the SPB Portal was having several technical issues. The +original codebase was not being developed anymore, and there was a large +amount of technical debt to overcome. The system was a modified version +of an existing FOSS platform called +OpenACS\footnote{\url{http://openacs.org}}, and the old SPB portal was +not being updated anymore against the official OpenACS releases. In this +scenario, the portal maintenance was becoming harder and harder. + +After some events and meetings to collect requirements from the federal +government and from the society, a new platform for the SPB Portal was +developed, among January 2014 and June 2016, by the University of +Brasília (UnB) and the University of São Paulo (USP) in a partnership +with the Brazilian Ministry of Budget, Planning, and Management (MP). It +was designed as an integrated platform for collaborative software +development., and includes functionality for social networking, mailing +lists, version control system, and source code quality monitoring. To +coordinate and develop this project during 30 months, UnB received from +the Brazilian Federal Government a total of 2,619,965.00 BRL (about +750,000.00 USD in June 2016). \begin{figure*}[hbt] \centering - \includegraphics[width=.9\linewidth]{figures/home-SPB.png} + \includegraphics[width=.9\linewidth]{figures/home-SPB_2.png} \caption{The new SPB Portal.} \label{fig:spb} \end{figure*} +The project was developed by a team of 3 professors, 2 masters students, and +approximately 50 undergraduate students (not all of them at the same time, +though -- graduations and other events triggered changes in the team) together +with 2 professional designers and 6 senior developers from the FOSS +communities. The professors and all undergraduate student were from UnB, and +the master students were from USP. Regarding the designers and senior +developers, 7 of 8 they were living outside of Brasília: Curitiba/Brazil, São +Paulo/Brazil, Ribeirão Preto/Brazil, Salvador/Brazil, Punta Cana/Dominican +Republic, and Montreal/Canada. In other words, we had a team working in +distributed collaborative virtual environment. + Figure \ref{fig:spb} shows the home page of this integrated platform. -The development tried to be as faithful as possible to FOSS development. All development was done in the open, and the changes we needed in the -tools were contributed back to their communities. +FOSS tools were contributed back to their respective communities. Our +process was based on agile practices and FOSS communities interaction. +We defined development cycles and released 5 versions of the new SPB +Portal. The first release (beta) was in September 2014, only 9 months +from the beginning of the project. The old portal was shut down down in +September 2015. Finally, the last version illustrated in Figure 1 was +released in June 2016. + +In this paper, we present an overview of this new generation of the SPB +Portal. This experience report shares our methodology and process to +develop this project working with the Brazilian federal government to +comply with its requirements at the same time to be as faithful as +possible to FOSS development. Moreover, we discuss several lessons +learned to provide a distributed collaborative virtual environment +involving a large undergraduate student team and remote senior +developers. Lastly, we released an unprecedented platform for the +Brazilian government applying empirical software development methods. +This case can help other projects overcome similar software engineering +challenges in the future, as well as illustrate how universities can +improve the real-world experience of their students by means of this +kind of project. diff --git a/sbqs2017/content/02-spb.tex b/sbqs2017/content/02-spb.tex index 5c829ed..db4c120 100644 --- a/sbqs2017/content/02-spb.tex +++ b/sbqs2017/content/02-spb.tex @@ -1,60 +1,62 @@ -\section{Free/Open Source Software and Brazilian Public Software} +\section{Background} \label{sec:spb} FOSS is a phenomenon that has gained notoriety in recent years and has been -attarcting the interest of academia. However, since the beginning of computing +attracting the interest of academia. However, since the beginning of computing the majority of developers worked in the way that we now identify as free -software, that is, sharing code openly. This feature makes the code available +software, that is, sharing code openly. This openness makes the code available for inspection, modification, and use by any person or organization -\cite{kon2012}, \cite{hippel2003}. +\cite{hippel2003,kon2012}. The elements that distinguish FOSS from other types of software are the reasoning about the development process, the economic context, the relationship between developers and users, as well as the ethical and legal characteristics that relate to the software. In the context of FOSS, user freedom is promoted -and its development is based on open collaboration and development practices. -%TODO: Colocar referências sem ser nós mesmo e sem ser em PT-Br +and its development is based on open collaboration and development practices +\cite{meirelles2013}. From the economic point of view, unlike what happens with proprietary software, -FOSS promotes the establishment of several suppliers that compete with each +FOSS promotes the establishment of several suppliers that can compete with each other based on the same software. This stronger competition among suppliers brings benefits to users because it gives better assurances regarding the -evolution of the system and induces a reduction in prices. These freedoms and -assurances on software are guaranteed in Brazil by Law 9610/98 (copyright law). -Most of the time, this protection from the law complies with the terms -conferred by a contract related to certain software. This contract is called -``license''. A software license determines a list of rights that are +evolution of the system and induces a reduction in prices \cite{kon2012}. These +freedoms and assurances on software are guaranteed in Brazil by Law 9610/98 +(copyright law). Most of the time, this protection from the law complies with +the terms conferred by a contract related to certain software. This contract is +called ``license''. A software license determines a list of rights that are given to, and duties that are imposed on a user of the software. In particular, what differentiates FOSS from proprietary software is just the way they are -licensed\cite{sabino2009}. The FOSS licenses guarantee the right to execute, +licensed \cite{sabino2009}. The FOSS licenses guarantee the right to execute, study, adapt, and improve the software. Example of common FOSS licenses are the \textit{GPL (GNU General Public License)}, the Apache license, the MIT license, and the BSD license. -The SPB portal has been designed in 2005 and released in 2007. In a practical -view, it is a web system that has consolidated itself as a software project -sharing environment. It provides a space (community) for each software. -Therefore, the current platform for SPB was designed to include tools that -promote collaboration and interaction in communities (by managers, users, and -developers) of the projects, according to the practices used in FOSS -communities. This includes e-mail lists, discussion forums, issue trackers, -version control systems, and social networking environments. +The original incarnation of SPB portal has been designed in 2005 and +released in 2007. From a practical point of view, it is a web system +that has consolidated itself as an environment for sharing software +projects. It provides a space (community) for each software. +Therefore, it was designed to include tools that promote collaboration +and interaction in communities (by managers, users, and developers) of +the projects, according to the practices used in FOSS communities. This +includes mailing lists, discussion forums, issue trackers, version +control systems, and social networking environments. Initially, the purpose of the portal was only to share the software developed -in the Brazilian government, to reduce the costs of hiring software. However +in the Brazilian government, to reduce the costs of hiring software. However, it was observed that when softwares were released, their communities were formed around those software with several people collaborating and sharing the results obtained through the use of those solutions. In this way, some software development cooperatives and private companies have shown an interest in making their software available on the SPB platform. -The concept of Brazilian Public Software goes beyond FOSS. In addition to being -licensed under a FOSS license, a Brazilian Public Software needs to have -explicit guarantees that it is a public good, and that project must be -available in the SPB. Being a true public good assumes requirements that can -not be met solely by means of FOSS licensing. For example, there must be a -relaxed trademark usage policy by the original vendor that do not stop eventual -competitors from adversiting services for that same software. Inclusion in the -SPB also has extra requirements, such as having a public version control -system, installation manual, hardware requirements specification, etc. +The concept of Brazilian Public Software goes beyond FOSS. In addition +to being licensed under a FOSS license, a SPB needs to have explicit +guarantees that it is a public good, and that project must be available +in the SPB portal. Being a true public good assumes requirements that +can not be met solely by means of FOSS licensing. For example, there +must be a relaxed trademark usage policy by the original vendor that +does not stop eventual competitors from advertising services for that +same software. Inclusion in the SPB Portal also has extra requirements, +such as having a public version control system, installation manual, and +hardware requirements specification. diff --git a/sbqs2017/content/03-requirements.tex b/sbqs2017/content/03-requirements.tex index 9e6f55b..ae56e89 100644 --- a/sbqs2017/content/03-requirements.tex +++ b/sbqs2017/content/03-requirements.tex @@ -1,24 +1,24 @@ -\section{Requirements} +\section{Requirements and Related Projects} \label{sec:requirements} In 2013, the SPB Portal had more than 600 thousand unique visitors, generating more than 16 million page views with about 50 million hits. By evaluating only the main projects, there were more than 15 thousand downloads and 4 thousand -messages exchanged in their forums. This data illustrates the potential of the -SPB Portal, even with some limitations in the past. +messages exchanged in their forums. These data illustrates the potential of the +SPB Portal, even with several limitations in the past. By preparing the evolution project described in this paper, the Brazilian -government promote 3 events to collect the requirements, in particular from +government promoted 3 events to collect the requirements, in particular from society point of view: (i) an online form to collect general ideas; (ii) a face-to-face meeting with society in general; (iii) a workshop to review the SPB concepts and requirements with IT stakeholders from the Brazilian government and public organizations. -After these 3 rounds discussing the new SPB platform, the Brazilian government listed -about 145 requirements and developed a mind -model\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} +After these 3 rounds discussing the new SPB platform, the Brazilian government +listed about 145 requirements and developed a ``mind +map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} to guide the SPB portal evolution. In this scenario, the 10 most voted -requirements are, for example: +requirements were, for example: \begin{enumerate} @@ -37,26 +37,27 @@ requirements are, for example: \begin{figure}[hbt] \centering - \includegraphics[width=.6\linewidth]{figures/technological-requirements.png} + \includegraphics[width=\linewidth]{figures/technological-requirements.png} \caption{Technological requirements overview.} \label{fig:requirements} \end{figure} -Moreover, there were other requirements based on the experience of the IT +here were other requirements based on the experience of the IT stakeholders from the Brazilian government and from the Brazilian FOSS -community (that UnB and USP were representing too in this project). The new -platform just could work properly if there is a unique authentication to use -the provided tools. Additionally, a unified -interface was an important non-functional requirement to make easy the user -experience into the new platform. +community (that UnB and USP were representing too in this project). The +new platform would only work properly if there is a unique +authentication to use the provided tools. Additionally, a unified +interface was an important non-functional requirement to have a better +user experience in the new platform. -At the first moment, we wish to release an initial version to replace the old -SPB portal. For that, the first version must have some features such as: +At the first moment, we desired to release an initial version that could +replace the old SPB portal. For that, the first version should have +features such as: \begin{enumerate} -\item Organized public software catalog. +\item An organized public software catalog. \item Social network environment (profiles for users, software pages, and community pages). \item Content Management Systems (CMS) features. \item Web-based Git repository manager with wiki and issue tracking features. @@ -64,39 +65,42 @@ SPB portal. For that, the first version must have some features such as: \end{enumerate} -Other requirements also were planned during the conception phase of the SPB -evolution project such as an integrated search engine and a web-based source -code static analysis monitor. Therefore, by analyzing all of these -requirements, we propose the technological requirements overview, as -illustrated in Figure \ref{fig:requirements}, to guide the development of the -new SPB platform. In other words, we have designed the SPB evolution project -based on existing FOSS tools. However, the integration of several existing -systems that already was implemented in different programming language and -frameworks, adding features such as a unique authentication, a unified -interface, and a search engine, as well as, other back-end features, is not a -trivial work. - -The new SPB platform is fully an integrated environment, as we can see in -Figure \ref{fig:requirements}, being very advanced comparing to other related -projects and initiatives. For example, the USA government has a platform -designed to improve access to the federal government developed +Other requirements were also planned during the conception phase of the +SPB evolution project, such as an integrated search engine and a +web-based source code static analysis monitor. By analyzing all of these +requirements, we proposed the technological requirements overview +illustrated in Figure \ref{fig:requirements} to guide the development of +the new SPB platform. In other words, we have designed the SPB evolution +project based on existing FOSS tools. However, the integration of +several existing systems that were already implemented in different +programming languages and frameworks, adding features such as a +centralized authentication, unified interface, and a search engine, as +well as, other back-end features, would require a non-trivial amount of +work. + +The new SPB platform is a fully integrated environment, as we can see in +Figure \ref{fig:requirements}, being very advanced in comparison with +related projects and initiatives. For example, the USA government has a +platform designed to improve access to the federal government developed software\footnote{\url{https://code.gov}}. Code.gov is an interface to -organize the USA government projects and, in short, make easy that their users -and developers obtain some information and access their source code +organize the USA government projects and, in short, make it easy for +users and developers to obtain information and access their source code repositories at GitHub. However, there are not social networking and CMS -features, as well as, other communication resources provided by that platform. +features, as well as, other communication resources provided by that +platform. -Also, there are two initiatives in Europe: +Additionally, there are two initiatives in Europe: OSOR\footnote{\url{https://joinup.ec.europa.eu/community/osor}} and -OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) is a -community hosted in the JoinUp platform powereded by the European Commission. -OSOR aims exchanging information, experiences and best practices around FOSS -solutions for use in public administrations. Summarily, it helps to find a FOSS -made available by other public administrations, providing access to information -such as news, events, studies and solutions related to implementation of open -source software. It also offers forum discussions and community mailing lists, -but it does not have an integrated source code repository manager and for the -each project has a link to its own external repository (or its tarball file). +OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) +is a community hosted in the JoinUp platform powered by the European +Commission. OSOR aims at exchanging information, experiences and best +practices around the use of FOSS in the public administration. It helps +to find a FOSS made available by other public administrations, providing +access to information such as news, events, studies and solutions +related to implementation of open source software. It also offers forum +discussions and community mailing lists, but it does not have an +integrated source code repository manager and for the each project there +is a link to its own external repository (or its tarball file). % OW2 is a FOSS community to promote the development of FOSS middleware, generic business applications, cloud computing platforms and foster a community and @@ -104,33 +108,31 @@ business ecosystem. In short, it aims to support the development, deployment and management of distributed applications with a focus on FOSS middleware and related development and management tools. % -Moreover, from the European Commission in 2007 until 20011, there were the -QualiPSo project that aims to provide to FOSS users, developers, and consumers, -quality resources and expertise on the various topics related to FOSS. The +Moreover, from the European Commission in 2007 until 20011, there was the +QualiPSo project that aimed at providing FOSS users, developers, and consumers, +with quality resources and expertise on the various topics related to FOSS. The QualiPSo project also had planned to develop a platform called QualiPSo Factory but it was not fully completed. -In Latin American has an initiative based on the SPB project called Software -Publico Regional\footnote{\url{http://softwarepublicoregionalbeta.net}}. From -the practical point of view, it provides a customized Gitlab instance to share +In Latin American there is an initiative based on the SPB project called ``Software +Publico Regional''\footnote{\url{http://softwarepublicoregionalbeta.net}}. From +a practical point of view, it provides a customized Gitlab instance to share the source code and documentation of the project from the involved countries. % -Such as Brazil, Chile has its own portal also called Software -Publico\footnote{\url{http://www.softwarepublico.gob.cl}}. The user can create -content in the communities (news items, documents, wiki pages), but all -repository is available at the Bitbucket +Like Brazil, Chile has its own portal also called ``Software +Publico''\footnote{\url{http://www.softwarepublico.gob.cl}}. Users can create +content in the communities (news items, documents, wiki pages), but +source code repositories are available at the Bitbucket platform\footnote{\url{https://bitbucket.org/softwarepublico}}. -The Brazilian government needed to evolute the SPB project that exists since -2005. In 2013, when we started this project, the SPB Portal had about 200 -thousand registered users. We could not just contact these users and ask them -to register an account at Github as well. Moreover, after the Edward Snowden -case, the Brazilian government approved a specific law decree (8.135/2013) to -rule its communication service. Summarily, services like Gmail, Google Drive, -Dropbox, Live, Outlook, iCloud, as well, Google Groups, Github, etc do not -could be used by a Brazilian public agent as tool for your work. To use these -kinds of services the Brazilian government needs to provide them to itself. -Thus, we developed our own solution to cover all the requirements, producing a -complete governmental integrated platform for collaborative software -development. - +The Brazilian government needed to evolve the SPB project that +existedince 2005. In 2013, when we started this project, the SPB Portal +had about 200 thousand registered users. We could not just contact these +users and ask them to register an account at Github as well. Moreover, +after the Edward Snowden case, the Brazilian government approved a +specific law decree (8.135/2013) to rule its communication services, +requiring the public administration to host its information systems to +be provided by itself, what rules out usage of private platforms, +specially ones provided by foreign companies. We thus developed our own +solution to cover all the requirements, producing a complete +governmental integrated platform for collaborative software development. diff --git a/sbqs2017/content/04-architecture.tex b/sbqs2017/content/04-architecture.tex index f5f2678..85cfbc0 100644 --- a/sbqs2017/content/04-architecture.tex +++ b/sbqs2017/content/04-architecture.tex @@ -1,14 +1,16 @@ \section{Architecture} \label{sec:architecture} -Based on the great list of functional requirements desired by Brazilian Federal -Government we decided to select some FOSS systems that already contemplate some -of them and improve these systems. And bringing the idea of community, it is -undoable build a platform to be used by communities, which is a complex -scenario, using just one tool. +Based on the extensive list of functional requirements defined by the +Brazilian Federal Government, we selected some FOSS systems to form our +solution. We looked for system that together could provide the largest +subset possible of the requirements, and were fully aware that we would +need to improve those systems in order to provide the rest. We were also +convinced that it would be impossible to provide all of those +requirements with a single tool. -At the point of view of the architecture, two main requirements was brought by -the Brazilian Federal Government for the new platform: +From the point of view of the architecture, two main requirements was +brought by the Brazilian Federal Government for the new platform: \begin{enumerate} \item \textit{Integrate existing FOSS systems}, with minimal differences from @@ -17,69 +19,81 @@ the Brazilian Federal Government for the new platform: systems, as well as centralized authentication. \end{enumerate} -Adopting existing FOSS systems by the government could bring the benefit of -improvements done by the upstream communities, and the maintenance effort. On -the other hand, integrating different tools with distinct intent, it is not an -easy task and it was important to have a consistent user interface which -justifies the last requirement. +Adopting existing FOSS systems and minimizing locally-made changes had +the objective of being able to upgrade to newer versions of the original +software, benefiting from improvements and maintenance done by the +existing project communities. Providing a consistent user interface on +top of those different tools was needed to make the transition between +the different systems seamless from the point of view of users, which +would be confused by jumping through completely different interfaces +while interacting with the portal. For the first requirement, we identified four main systems that required -specialized teams for work in the integration process. The teams learned how to -develop for their assigned systems and contributed to the original -communities, so that the version we used were not significantly different from -the original. +specialized teams for work in the integration process. The teams learned +how to develop for their assigned systems and contributed to the +original communities, so that the version we used were not significantly +different from the original. -At the end of the project, SPB portal was composed of more than ten systems -among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, -Munin, and so forth. The following sections explained a little bit of Colab, -Noosfero, Mezuro, and Gitlab (the main tools which we contributed). Below, we -described how we integrated those tools and conclude with the deployment. +At the end of the project, the SPB portal was composed of more than ten +systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and +Munin. The remainder of this section describes the most relevant of +them, as well as how they were integrated into the platform. \subsection{Colab} -Colab integrates web applications as its main goal, the user of this composed -system should not notice the change between the integrated applications. Colab -was designed to use one plugin for each system under its domain, this is -guaranteed by four levels of integration: +Colab\footnote{\url{https://github.com/colab}} is a systems integration +platform for web applications. One of its goals is allowing different +applications to be combined in such a way that an user does not notice +the change between applications. For that, Colab provides facilities +for: \begin{itemize} - \item Authentication - \item Visual - \item Events - \item Data and search engine + \item Centralized authentication + \item Visual consistency + \item Relaying of events between applications + \item Integrated search engine \end{itemize} -The aforementioned integrations levels were possible, because Colab works as a -reverse proxy, therefore all external requests pass through it. - -Single Sign-On (SSO) is used to login users throughout all integrated -applications. REMOTE\_USER HTTP header is sent to the applications and we -expect that they know how to handle it, since this is a common authentication -mechanism. The integrated applications should be on a local network to avoid -security issues. With this the user will be able to navigate through the -platform applications and will not be asked for authentication credentials. - -As Colab\footnote{\url{https://github.com/colab}} is a reverse proxy, it makes -some HTML transformations in the HTTP response of integrated applications to -provide a unified interface. Allows one to define some default templates to be -used by all applications and overwrite when needed in its plugin. This approach -allowed us to reuse some HTML pages which facilitates maintenance. - -A publish-subscribe implementation was used to handle events in the platform. -Thus, if some application want to trigger some action in case that other -application do something is possible. A registration of the desired events and -implementation of the handlers must be done in the plugin of each application. -This bring us many options to innovate in these integrations. - -A integrated search engine is provided by Colab. Each plugin specify which data -will be indexed and will became available for the users, just need an simple -implementation of how should perform the collection of data. +Colab implements this integration by working as a reverse proxy for the +integration applications, that is, all external requests pass through +Colab before reaching them. + +Centralized authentication is handled by letting users authenticate to +Colab, which then sends a REMOTE\_USER HTTP header to applications, +which are expected to be pre-configured to accept that as an indication +of the current user (REMOTE\_USER is a standard authentication mechanism +for web applications). This allows users to be automatically identified +to each of the applications without having to enter credentials for each +of them. +% +Of course, this requires that the integrated applications are not +accessible on the public internet, otherwise malicious users could send +their own crafted REMOTE\_USER headers and impersonate any user. + +For the visual consistency, Colab is able to make transformations to the +HTML produced by the integrated applications, ir order to provide a +unified interface. It allows one to define default templates to be used +by all applications, as well as providing extra ones in a plugin. This +approach allowed us to reuse common HTML pages, what facilitates +maintenance. + +Colab also provides a publish-subscribe event system. This allows one of +the applications, or Colab itself, to trigger an action in another +application. For example, when a user registers with Colab, all +applications need to be notified in order to create their own internal +representation of that user account. + +Colab also provides an integrated search engine, which can be configured +to specify exactly what data needs to be indexed for each of the +applications, and how to direct the user to that piece of information on +the specific application. Initially, Colab had support for a small set of applications (Trac, GNU -Mailman, and Apache Lucene) and all of them was hard-coded. Our team evolved -Colab so that it can now receive plugins to add support for new applications -with minimal changes to its existing core. We developed plugins to be used in -the SPB platform: Noosfero, GitLab, and Mezuro. +Mailman, and Apache Lucene) and support for them was hard-coded. Our +team evolved Colab so that it can now receive plugins that add support +new applications without the need of changes to the Colab core. We also +developed plugins to be used in the SPB platform: Noosfero, GitLab, and +Mezuro. \subsection{Noosfero} @@ -92,44 +106,40 @@ home pages and documentation, and contact forms. \subsection{Gitlab} -GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository manager -with wiki pages and issue tracking features. On the one hand, Github is the -most famous and used Git repository manager. On the other hand, it is not a -FOSS. Gitlab is a FOSS platform and focuses on delivering a holistic solution -that will see developers from idea to production seamlessly and on a single -platform. Thus, we have worked on it to integrated to Colab. - -Moreover, Gitlab has several features working properly than Github that is -interesting in SPB context such as free for private projects, built-in -continuous integration and continuous deployment with best practices, more -control during downtime and upgrade, flexible permissions, Work-in-Progress -protection, move issues between projects, Group-level milestones, Create new -branches from issues, Issue board, Time tracking, as well new features and -improvements every month on the 22nd. +GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository +manager with wiki pages and issue tracking features. Gitlab is a FOSS +platform and focuses on delivering a holistic solution that will see +developers from idea to production seamlessly and on a single platform. + +Gitlab has several unique features, such as built-in continuous +integration and continuous deployment, flexible permissions, tracking of +Work-in-Progress work, moving issues between projects, group-level +milestones, creating new branches from issues, issues board, and time +tracking. \subsection{Mezuro} Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code -metric to monitor the internal quality of softwares written in C, C++, -Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists. +metrics to monitor the internal quality of software written in C, C++, +Java, Python, Ruby, and PHP. -In general, source code metric tools also do not present a friendly way to -interpret its results and, even more, do not follow a standardization between -them. Mezuro brings and presents these results to -the end user, specially, by analyzing source code metric history during its -life cycle. +In general, source code metrics tools also do not present a friendly way +to interpret its results and, even more, do not follow a standardization +between them. Mezuro collects and presents these results to the end +user, specially, by analyzing source code metric history during its life +cycle. -The Mezuro platform provides a single interface -grouping available tools, allows selection and composition of metrics in a -flexible manner, stores the metrics evolution history, presents -results in a friendly way, as well as, allows users to customize the given -interpretation accordingly to their own context. +The Mezuro platform provides a single interface grouping available +tools, allows selection and composition of metrics in a flexible manner, +stores the metrics evolution history, presents results in a friendly +way, as well as, allows users to customize the given interpretation +accordingly to their own context. \subsection{System unification} \begin{figure}[hbt] \centering - \includegraphics[width=.5\linewidth]{figures/arch.png} + \includegraphics[width=\linewidth]{figures/arch.png} \caption{SPB architecture overview.} \label{fig:architecture} \end{figure} @@ -143,42 +153,5 @@ functionality: instead of using the redundant restricted search functionality of each application, a search in the SPB portal might return content from any of the applications, be it web pages, mailing list posts, or source code. - -% Falar do devops -\subsection{Deploy} - -The SPB platform was deployed in 7 virtual machines with different functions, -as we can see in Figure \ref{fig:architecture2}. - -\begin{figure*}[hbt] - \centering - \includegraphics[width=.8\linewidth]{figures/arch2.png} - \caption{Instanciation view of the SPB architecture.} - \label{fig:architecture2} -\end{figure*} - -The \textit{reverseproxy} handles the HTTP requests and redirects them to the -\textit{integration}, the \textit{email} sends and receives e-mails on behalf -of the platform and the \textit{monitor} keeps the entire environment tracked. -These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and -\textit{monitor} - are accessible via Internet and the other ones are only -available in the local network created between them. - -\textit{Integration} works as a second layer of proxy beneath -\textit{reverseproxy}, any request to the platform will be handled by it. The -Colab service provides interface, authentication and search engine integration -among all the services. When a request is received to a specific service, -Colab authenticates the user in the target tool, sends the request and makes a -visual transformation in the HTML page which is the content of the response. -Another user-oriented feature is the integrated search engine, when the user -want to find something in the platform Colab will perform the search in the -whole databases. Colab itself provides a web interface for GNU Mailman and we -have two others integrated tools in \textit{integration}: Gitlab and Prezento. -Gitlab provides web interface for Git repositories and issues tracker, and -Prezento is a front-end for source code static analysis. - -The source code static analysis is performed by \textit{mezuro}. It runs some -static analysis tools on source code stored in repository and provide this data -to Prezento. A social network and CMS (Content Manager System) is provided by -Noosfero in \textit{social}, and the databases of all tools with a cache -service are in \textit{database}. +% +The SPB platform was deployed in 7 virtual machines with different functions. diff --git a/sbqs2017/content/05-features.tex b/sbqs2017/content/05-features.tex index cb0a490..9767637 100644 --- a/sbqs2017/content/05-features.tex +++ b/sbqs2017/content/05-features.tex @@ -1,14 +1,14 @@ \section{Features} \label{sec:spb} -The new generation of the SPB portal combines adapted features of existing collaborative softwares +The new generation of the SPB portal combines features adapted from existing collaborative software and features developed by the SPB team. Whenever possible, new functions -(newly developed or partially modified) were sent to official repositories, as a contribution. +(newly developed or partially modified) were contributed back to the official repositories. As a result, we have a platform that integrates and harmonizes different features such as social networking, mailing list, version control system, content management and source code quality monitoring. Our aim was to develop functionalities by reusing -functions of collaborative softwares already integrated to the platform. In +functionality of collaborative software already integrated to the platform. In addition, we tried to keep this integration transparent to end users. \subsection{Software and Software Community} @@ -17,47 +17,47 @@ In the new SPB portal, each software has a standard set of pages and tools. Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download different versions of the software and find several mechanisms of software development management. -Focusing on the collaborative development, the Mailman was integrated to the platform in order to allow +Focusing on the collaborative development, Mailman was integrated to the platform in order to allow the dialogue and communication between developers, users and enthusiasts of a determined software. The software has its own mailing list whose privacy can be configured/set by administrators. -The software has a social interface area (aka "software community") where users can find other users, blogs, +The software has a social interface area ("software community") where users can find other users, blogs, summary of recent activities, or any other relevant community-produced content. Users logged to the platform can request membership to different software communities and each community member can access and edit restricted content. For this purpose, -many Noosfero features related to social network and content management was integrated to the portal. +many Noosfero features related to social networking and content management were integrated to the portal. To assist decision-making, the new SPB has acquired assessment and statistical tools. Now, users will be able to rate the software and make comments and all information will be avaiable to other users. Moreover, the software has a section -containing its statistical data, where values are calculated through data +containing its statistical data, where values are calculated based on data provided by users and the system. The role of the administrator will be present in the software and in its community. The administrator is responsible for moderating content, memberships and user -comments. He is also the one who can make changes in the software homepage +comments. The administrator is also the one who can make changes in the software homepage content. \subsection{Software Catalog and global search} The platform also provides a search tool called Software Catalog, -which allows users to find softwares available in its directory. +which allows users to find softwares available in the portal. In this catalog, some search options were developed to make the navigation easier, such as filters (by type of software or category), sorting and score. In order to expand the searching scope and cover more types of content, the SPB team developed the global search tool. This tool unifies search mechanisms -provided by the mentioned collaborative softwares. Any user can -find a public content in the context of social network, mailing list and +provided by the different collaborative software used in the SPB protal. Any user can +find a public content in the context of social networking, mailing list and software development. \subsection{Software development tools} The new SPB also provides -tools to encourage developers to keep each source code and its -developments within the platform. Any created software has, by default, a -related git repository with wiki pages and issues tracking. These tools are +tools to encourage developers to keep source code and its +development activity within the platform. Any created software has, by default, a +an associated git repository with wiki pages and issue tracking. These tools are supplied by the integration of Gitlab into the platform. Developers can also evaluate the software source code to measure software @@ -67,7 +67,7 @@ public, which allows greater transparency between the developer and the community that uses the software. Thereby, the maintainers can decide if the given solution meets the source code quality requirements. -Thus, the SPB becomes a platform to stimulate the openness of the source code; +Thus, the SPB became a platform to stimulate the openness of the source code; dialogue between users and the development team; and also maintenance and evolution of the software, which will provide more transparency in government investments. diff --git a/sbqs2017/content/07-process.tex b/sbqs2017/content/07-process.tex index e6b0ecc..60b8e55 100644 --- a/sbqs2017/content/07-process.tex +++ b/sbqs2017/content/07-process.tex @@ -5,19 +5,19 @@ The SPB team was composed of a variety of professionals with different levels and skills, where most of them were undergraduate students with major in software engineering (from 4th semester or upper). Since the students could not dedicate many hours per week to the project, they always had the -flexibility to negotiate their work schedule during the semester in order not -to cause any damage to their grades. Their daily work routine in the project -included programming and devops tasks. +flexibility to negotiate their work schedule during the semester in +order to not harm their classes and coursework. Their daily work routine +in the project included programming and DevOps tasks. The development of SPB project required a vast experience and background that -usually undergraduate students do not have yet. For this reason, some senior -developers have joined to the project to help with hard issues and to transfer -knowledge to the students. Their main task was to provide solutions for complex -problems, in other words, they worked as a developer. As these professionals -are very skillful and the project could not fund a full time work for them, -some of them worked partially on the project. In addition, they lived in a -different states spread around Brazil which led much of the communication to be -made via Internet. +usually undergraduate students do not have yet. For this reason, a few senior +developers have joined to the project to help with the more difficult issues +and to transfer knowledge to the students. Their main task was to provide +solutions for complex problems, in other words, they worked as developers. As +these professionals are very skillful and the project could not fund a full +time work for them, some of them worked partially on the project. In addition, +they lived in a different states spread around Brazil which led much of the +communication to be online. In short, our work process was based on open and collaborative software development practices. The development process was defined based on the @@ -26,108 +26,143 @@ high degree of automation resulting from DevOps practices. Thus, the work process was executed in a cadenced and continuous way. Finally, the last group of actors of this project was composed of employees -formally working for the Brazilian government, in the Ministery of Planning, -Development, and Management (MP is the Brazilian acronyms). All the project +formally working for the Brazilian government, in the Ministry of Planning, +Development, and Management (MP is the Brazilian acronym). All the project decisions, validations, and scope definitions were made by them. In this way we -developed software product increments, releases, aligned with business -strategic objectives. As can be seen, the project had many kinds of profiles -that had to be organized and synchronized. +developed software product incrementally, with releases aligned to business +strategic objectives. As one can see, the project had many different +kinds of stakeholders that had to be organized and synchronized. -\subsection{Teams organizations} +\subsection{Team organization} Approximately 70\% of the development teams were composed of software -engineering undergraduate students from UnB and they worked physically in the -same laboratory in the opposite of the senior. Each student had their own -scheduler based on their class, it made complicated to implement pair -programming. Also, they had a different area of interests. To cope with those -diversity, we had two basic rules which guided the project organization: +engineering undergraduate students from UnB and they worked physically +in the same laboratory. Each student had their own schedule based on +their classes, what complicates the implementation of pair programming. +The senior developers tried to synchronize their schedule with the +students on their sub-team. To cope with this environment, we had a few +basic rules which guided the project organization: \begin{enumerate} - \item Classes have to be the high priority for undergraduate students; - \item Always work in pair (locally or remotely). + \item Classes have high priority for undergraduate students; + \item Work in pairs whenever possible (locally or remotely). + \item There must be one morning or afternoon per week when + \emph{everyone} should be together physically in the laboratory + (except of course the remote team members). + \item Every 3 to 4 months, the senior developers would fly in and work + alongside the students for a few days. \end{enumerate} -With the aforementioned rules we divided all the project into four different -teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible -for reducing the communication problem with the other teams and help the -members to organize itself in the best way for everyone (always respecting the -work time). The coach, was a normal student working in some of the teams with -the extra duty of register the current tasks developed in the sprint and with -the responsibility to talk with other teams. One important thing to notice is -the mutability of the team and the coach, during the project many students -changed between the teams to try different areas. - -One characteristic of the teams was the presence of (at least) one senior per -team. This was essential, because hard decisions and complex problems were -usually addressed to them, this relieved the coach duty to take a complicated -technical decisions and encouraged students to be a coach. Lastly, the senior -had to respect a rule number two and work with students, this was important to -gave the undergraduate the opportunity to interact with a savvy professional in -his area and keeping the knowledge flow in the project. - -Finally, we had to add two last elements of the team organization, that was -essential for the project harmony: the meta-coach and professors. The former -was a software engineer recently graduated and which wanted to keep working on -the project, the latter were professors that orchestrated all the interactions -between all members of the project. The meta-coach usually worked in one -specific team and had the extra task of knowing the current status of all -teams. Professors and meta-coaches worked together to reduce the communication -problem between all the teams. Lastly, all the bureaucracy tasks was -centralized in the professors. - -\subsection{Meetings} - -Brazilian government used to work with software development in a very -traditional way, frequently they claim on documents and does not focus on what -really matter (running software). This way of thinking caused to us a -communication noise with MP, because they constantly tried to leverage on our -work style. It was especially hard to convince them to accept the idea of open -scope and agile development, but after months of labor and showing results they -stopped resisting. - -We defined some level of meeting granularity to avoid to generate overheads to -the developers. We had a strategical and validating meeting with MP (the -former once in a month and the latter each 15th day), release plaining with the -entire team (one per month), and finally a sprint planning (one each 15th day). -Figure \ref{fig:meeting} is a diagram that represents our meeting organization. +With the aforementioned rules we divided all the project into four +different teams: Colab, Noosfero, Design, and DevOps. Each team had one +coach responsible for reducing the communication problem with the other +teams and help the members to organize themselves in the best way for +everyone (always respecting their work time). The coach was one of the +students working in some of the teams with the extra duty of registering +the current tasks developed in the sprint and with the responsibility to +talk with other teams. One important thing to notice is the mutability +of the team and the coach, during the project many students changed +between the teams to try different areas. + +One characteristic of the teams was the presence of (at least) one +senior per team. This was essential, because hard decisions and complex +problems were usually referred to them. This kept having to take +complicated technical decisions from the coach tole, what encouraged +students to be coaches. Lastly, the senior developers worked directly +with the students, and this was important to give the undergraduate the +opportunity to interact with a savvy professional in his area and to +keep the knowledge flowing in the project. + +Finally, we had to add two last elements of the team organization, that +was essential for the project harmony: the meta-coach and professors. +The former was a software engineer recently graduated and which wanted +to keep working on the project, the latter were professors that +orchestrated all the interactions between all members of the project. +The meta-coach usually worked in one specific team and had the extra +task of knowing the current status of all teams. Professors and +meta-coaches worked together to reduce the communication problem between +all the teams. Lastly, all the paperwork tasks, such as reporting on the +project progress to the Ministry, was handled by the professors. + +\subsection{Communication and management} + +Our team had many people working together, and most of the seniors worked in a +different city remotely. Also, we tried to keep our work completely clear to +the Brazilian government and citizens interested in following the project. To +handle those cases, we used a set of tools to communication and other to manage +the project. + +For communication between member in different places, we used: Google +handouts with tmate tool, IRC, and mailing lists. When one student had to +work in pair with a senior, normally, they used Google hangout for +communication and they shared a terminal session with tmate which allow +them to share the same terminal, with both typing and seeing the screen. +For questions and fast discussion, we used IRC. For general +notification, we used the mailing lists. + +For managing the project we used the SPB Portal itself; first to validate it by +ourselves, and also because it had all the required tools. We basically created +one wiki page per release in the SPB Gitlab instance with a mapping between +strategical, tactical, and operational views. In a practical point of view, one +milestone per user history (feature), and one or more issues for addressing +each feature. With this approach we achieved two important goals: keeping all +the management as close possible to to the source code, and tracking every +feature developed during the project. + +\subsection{High-level project management and reporting} + +The Brazilian government used to work with software development in a +very traditional way. They would frequently focus on documents and not +on what was, in our opinion, what really matters: working software. This +dissonance caused us a communication noise with MP, because they would +often question our work style. It was especially hard to convince them +to accept the idea of open scope and agile development, but after months +of labor and showing results they stopped resisting. + +We defined some level of meeting granularity to avoid generating too +much overhead to the developers. We had a strategical and validating +meeting with MP (the former once in a month and the latter each 15th +day), release plaining with the entire team (one per month), and finally +a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a +diagram that represents our meeting organization. \begin{figure}[hbt] \centering - \includegraphics[width=.75\linewidth]{figures/meeting_flows.png} + \includegraphics[width=\linewidth]{figures/meeting_flows.png} \caption{Meetings cycles} \label{fig:meeting} \end{figure} -In the strategical meeting we usually defined the priorities and new features -with the Brazilian government (we always had to negotiate next steps with -them). Normally the professors, the coach of each team, the meta-coach, and -some employees of the MP join in this meeting. We usually discussed what the -team already produced since our last meeting, and we establish the new features -for the next release. Notice that just part of the team join in this meeting -to avoid generating unnecessary overhead to the developers, but all the -students interested to participate was allowed to join (many students wanted -this experience during the project). +In the strategical meeting we usually defined the priorities and new +features with the Brazilian government. Normally the professors, the +coach of each team, the meta-coach, and some employees of the MP would +participate in this meeting. We usually discussed what the team already +produced since our last meeting, and established the new features for +the next release. Notice that just part of the team would join this +meeting to avoid generating unnecessary overhead to the developers, but +all the students interested to participate were allowed to join (many +students wanted this experience during the project). After the strategical meeting with Brazilian government agents, we had a planning turn with all teams together. In this part, each team worked together -to convert the MP wishes into small parts which was represented by the epics of +to convert the MP wishes into smaller parts which were represented by the epics of the release. Each coach was responsible for conducting the planning, and after -that register it on the project wiki (the wiki provided by Gitlab). With this +that record it on the project wiki (the wiki provided by Gitlab). With this epic, each 14th day the team have generated their sprint scheduler (with small -achievements mapped in issues). +achievements mapped to issues). -To keep the Brazilian government always updated, we invited them to work with -us to validate the new features in progress. Normally we had a meeting each -15th day. Basically, this was our work flow, we always kept everything -extremely open to the MP (our way of work and open source projects) and to the -team. +To keep the Brazilian government always updated, we invited them to work +with us to validate the new features in progress. Normally we had a +meeting each 15th day. Basically, this was our work flow, we always kept +everything extremely open to the MP (our way of working, and the one +often used by open source projects) and to the team. -To keep the track of all of those things we used the SPB, especially the -Gitlab. Basically, we had: +To keep the track of all of those things we used the SPB itself, +especially Gitlab. Basically, we had: \begin{enumerate} \item Project repository: We have one organization with many repositories - \item Milestones: Each milestone is used to register a release + \item Milestones: Each milestone was used to register a release \item Wiki: Each release has one page on wiki with the compilation of strategical meeting \item Issues: Each sprint planning generated issues, which we associated to @@ -135,32 +170,6 @@ Gitlab. Basically, we had: with them. Finally each developer assigned the issue to itself. \end{enumerate} -Notice that this workflow gave to us and to the Brazilian government agents a -full traceability from high view of the feature to the low view (code). This -provided to them a way to validate all worked done and proof the concept that -work with open source project can give a proper view to them check. - -\subsection{Tools for communication and management} - -Our team had many people worked together, and most of the seniors worked in a -different city remotely. Also, we tried to keep our work completely clear to -the Brazilian government and citizens interested in follow the project. To -handle those cases, we used a set of tools to communication and other to manage -the project. - -For communication between member in different places, we used: google-talk with -tmate, IRC, and mailing-list. When one student had to work in pair with a -senior, normally, they used google-hangout for communication and they shared a -session with tmate which allow them to share the same terminal. For questions -and fast discussion, we used IRC. For general notification, we used the -mailing-list. - -For managing the project we used the SPB Portal to validate it by ourselves and -because it had all the required tools. We basically create one wiki page per -release in Gitlab, one milestone per sprint, and one or more issues for address -one user history. With this approach we achieve two important things: keep all -the management close to the source code and tracked every feature developed by -the project. - -%TODO: Ainda falta adicionar a parte da visita dos seniors e o turno sagrado - +Notice that this workflow gave us and the Brazilian government agents +full traceability from a high level view of each feature to the lowest +level (code). diff --git a/sbqs2017/content/08-contributions.tex b/sbqs2017/content/08-contributions.tex index 1436ae9..286e18e 100644 --- a/sbqs2017/content/08-contributions.tex +++ b/sbqs2017/content/08-contributions.tex @@ -1,4 +1,4 @@ -\section{Contributing with Free Software Communities} +\section{Contributions to the upstream communities} \label{sec:contributions} %- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) @@ -9,24 +9,30 @@ %* Coper, empacotamentos (obs), omniauth -During the execution of this project we made several contributions from -different levels to the communities we interacted with. This occurred due to -our development process aligned with those of the respective communities. We -used to discuss with upstream the features and bug fixes that we was working -on, this kind of discussion improve the developers' technical solutions and -allowed upstream to accept our contribution more easily. - -In Colab we helped upstream to redesign the entirely architecture, enabling the -development of plugins to integrate new tools. We also added a feature that -allowed Colab to run asynchronous tasks, which was a major improvement for us -since we were developing a complex system. A migration to the latest Django -version was made (web framework used by Colab). Moreover, we worked on RevProxy -(the greatest Colab dependency) to put it in a good shape, fixing many bugs. +During the execution of this project we made several contributions to +the upstream communities we interacted with. This occurred due to our +development process aligned with those of the respective communities. +During development, we would explicitly discuss the features and bug +fixes that we were working on with the applicable upstream communities. +This contributed to improve the developers technical solutions with +expertise outside of our team, and make it easier for those changes to +be accepted in the original projects. Having changes accepted upstream +in turn makes our life easier as it minimizes the delta between our +codebase and upstream's allowing us to upgrade and benefit from +development work from others. + +In Colab, we helped upstream redesign the entirely architecture, +enabling the development of plugins to integrate new tools. We also +added a feature that allowed Colab to run asynchronous tasks, which was +a major improvement for us since we were developing a complex system. A +migration to the latest Django version was made (web framework used by +Colab). Moreover, we worked on RevProxy (the more important dependency +of Colab) to put it in a good shape, fixing many bugs. Gitlab was the tool that we made the least number of modifications. We -contributed with some improvements related with configuration files and we -developed a new omniauth plugin, which enables the user authentication in -Gitlab via REMOTE\_USER HTTP header. This omniauth plugin was needed because +contributed with some improvements related with configuration files and +we developed a new plugin that enables user authentication in Gitlab +through the REMOTE\_USER HTTP header. This plugin was needed because Colab uses this mechanism to manage the authentication. Noosfero was the tool that contemplated several functional requirements, @@ -35,15 +41,11 @@ migrate to the latest Rails version (web framework used by Noosfero), enable the federation implementation (federation with other social networks), decouple the interface and the back-end, and so forth. -We also contributed with some DevOps tools as well during the project. Some -member of our team took the maintenance of some python libraries that we used -to support our scripts to upload our packages to OBS (Open Build Service). -Since we were composed by many teams with large number of developers we had -some problems related with the tracking of our per team/software releases, the -DevOps team did not know when was the right time to package that software or -not. Thus we developed a tool called copr-status to keep tracked the version -packaged and the version finished by the developers, basically this is a web -interface that helps you to visualize the status of that package/software. +We also made some contributions on the DevOps front. Some members of +them team became maintainers of some python libraries that were used by +our scripts to upload packages to OBS (Open Build Service). We developed +a tool called copr-status to keep track of the different stages of the +lifecycle of each of the individual packages we were working on. %TODO: Mezuro diff --git a/sbqs2017/content/09-lessons.tex b/sbqs2017/content/09-lessons.tex index 1e2eec0..7d9c80e 100644 --- a/sbqs2017/content/09-lessons.tex +++ b/sbqs2017/content/09-lessons.tex @@ -1,77 +1,167 @@ \section{Lessons Learned} \label{sec:lessons} -The multidisciplinary composition of the development teams, mainly software -engineers and designers, is necessary for the development of good software -products, which naturally aim to meet the users needs. In the context of the -SPB project, there were also stakeholders from different areas who composed the -team of technicians and managers of the MP, as well as the administrative teams -of UnB. This interaction with different professionals brought a great learning -opportunity for the students, who had their first professional experience, even -during their graduation course. On the other hand, the different perceptions of -stakeholders generated high complexity of communications and expectations -management, burdening too much the professors who were responsible for project -management. - -The use of the Colab tool was a requirement required by the MP. They argueed -that this tool presented functionalities that would allow MP managers to -stimulate the participation of SPB service providers with gamefication -practices. As we said, in order for Colab to perform the expected behavior in -SPB its architecture had to be completely redefined and this caused in -practice, (i) the considerable increase in the architectural complexity of the -SPB and (ii) it was the subsystem that consumed the most effort and budget -during development. - -Due to the computational complexity related to the SPB deployment, associated -with the Brazilian government needs for product support, we made an effort to -provide complete automation of the entire deployment procedure, a result of -DevOps activities. In this way, we encapsulate all this complexity and enable -the deployment of new SPB releases through the execution of few commands, as -registered in the project documentation. Although we have provided a high -degree of automation, training workshops for the MP technical team, and a -meticulous description of the procedures in the documentation, we observed that -the MP technical staff invariably depended on our support to perform these -procedures. - -From the point of view of management and development processes, we had to -develop management strategies that would accommodate different organizational -cultures. As reported, the MP has a functional hierarchical organizational -culture, strongly supported by process management, typical of the traditional -development paradigm. The UnB teams use a process based on agile manifest -values and FOSS and agile community practices. So we created a process of -"translation" of work done to communicate with MP managers who manage their -portfolio based on the PMBoK processes. On the one hand, in the intermediary -and final project's phases we have matured this process, mainly due to the -perception of the results by MP, on the other hand, in the initial phase we had -a lot of intervention in our work style, which ended up focusing strategic -discussions for operational discussions. Again there was an overload in -professors, who were responsible for maintaining the strategic alignment of the -MP with the day to day development of the UnB team. - -Another importance factor for the students and maturing of the software -engineering practices used in the project was the composition of the teams with -the participation of experienced professionals from the FOSS communities. These -professionals together with the professors promoted a work environment where -the students could develop their skills in a didactic and practical way without -being transferred the pressures for them. In addition, these experienced -professionals were responsible for the most relevant technical decisions -related to the SPB software product. - +\textbf{How to involve students real-world projects, interacting with +real customers.} +% +Our team was mainly composed of software engineering undergraduate +student, who had the opportunity to interact with senior developers and +designers inside the team, as well as with the team of technicians and +managers from the Brazilian Government, and the management team from +UnB. +% +They interacted with professionals that had diverse expertises, and were +able to participate in all levels of the software development process. +This contribted to a great learning opportunity, and for a majority of +them this was their first professional experience. + +\textbf{The participation of experienced professionals is crucial to +success of the project.} +One important factor for the students was the composition of the teams +with the participation of experienced professionals. +% +On the technical side, the senior developers and designers would handle +the more difficult technical decisions, creating a work environment +where the students could develop their skills in a didactic way without +pressure. +% +On the management side, the active participation of professors -- who +are, in the end, the ones responsible for the project -- is crucial to +make sure student participation is conducted in a healthy way, and is an +instance of leading by example. + +\textbf{A balanced relationship between academia and industry.} +The experience of the SPB project led UnB to develop a work style which +proved to be appropriate for an educational environment that brings +academia and industry together. +% +The highest priority from the university's point of view is the +students. Considering this, the activities of the project were never +prioritized to the detriment of classes and other pedagogical +activities. In summary, we had students working at different times, part +time, remotely or locally, always respecting their individual +conditions, but doing the work in a collective, collaborative and open +way. +% +And even under a potentially adverse environment, the project delivered +the desired solution with success. +% +At the end of the project, we noted that the skills developed by the +students were at the state of art of the software engineering practice. +After the project ended, we had team members successfully embracing +opportunities in public, private, national and international +organizations, in addition to those students that went into +entrepreneurship and opened their own companies. + +\textbf{Managing different organizational cultures.} +In the beginning of the project, the Brazilian Government stakeholders +had certain expectations about the development of project that, let's +say, didn't exactly match our work based on agile and FOSS practices. +% +We had to develop strategies that would support different these +organizational cultures. As reported in Section \ref{sec:process}, the +MP is organized in a functional hierarchical organizational structure, +typically, a traditional development paradigm. Therefore, we had to +create a translation process between our team and the MP managers who +managed the project on their side assuming a traditional, waterfall +process. + +\textbf{Manage higher-level and lower-level goals separately.} +During the initial phase of the project the MP team would often bring +strategic discussions to technical/operational meetings, where we were +supposed to discuss practical, technical decisions. +% +This produced a highly complex communication and management environment, +overloading the professors because they were supposed to be responsible +for maintaining the strategic alignment of the MP synchronized with +implementation plans of the development team, specially in light of the +aforementioned culture mismatch. Mixing both concerns in the same +discussions caused confusion on both sides. +% +Towards the middle and end of the project we were able to keep those +concerns separated, what eased the work of everyone involved. + +\textbf{Living with ill-advised political decisions.} +At the initial phases of the project, by political and personal +motivation, the main stakeholders from the Brazilian government imposed +the use of Colab to guide the development of the new SPB platform. Our +team was totally against the idea because we already knew that Colab was +a very experimental project and its adoption could dramatically increase +the project complexity. Even though we provided technical reasons to +not utilize Colab, MP was adamant and we had to manage this problem. As +explained in section \ref{sec:architecture} we did massive changes to +it, and at the end of the project we completely rewrote Colab and made +it stable. It is important to notice that the MP compelled us to accept +a technical issue based only on political interests, without considering +all the resources that would be spent due to that decision. At the end +of the project, we verified that Colab indeed consumed a vast amount of +the budget and increased the project complexity. In the end of the +project and after our analysis on the decision made by the MP, we +understand that MP's managers are capable of ignoring technical reasons +in favor to a political decision. + +\textbf{Consider sustainability from the beginning.} +In the process of deploying the SPB platform in the MP structure, we had +to interact with the MP technicians. We did several workshops, training +and a meticulous documentation describing all the required procedures to +update the platform, however, we realized that the MP technicians +constantly avoid the responsibility. After noticing the aforementioned +situation, we organized a specific DevOps that completely automated all +the deployment procedure. We simplified all the platform deployment to a +few steps: (1) initial configurations (just ssh configuration) and (2) +the execution of simple commands to completely update the platform. By +the end of the project, we observed that the MP technicians invariably +still depended on our support to update the platform even with all the +automation provided by us. We were sadly left with a feeling of +uncertainty about the future of the platform after the project ended. In +hindsight, we realize that the MP dedicated systems analysts and +managers to the project, but not operations technicians. The later +should have been more involved with the process so they could at least +comfortable with managing the platform infrastructure. + % * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados %% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa %% afirmação, embora eu e Paulo consigamos perceber isso. -The experience of the SPB project led UnB to develop a model of team -composition and work style that proved to be appropriate to the cararteristics -of an education environment, bringing the academy closer to the industry. The -highest priority from the university's point of view is the students. -Considering this, the activities of the project were never prioritized to the -detriment of the classes and other didactic-pedagogical activities. In short we -had students working at different times, part time, remotely or presential, -always respecting their individual conditions, but doing the work in a -collective, collaborative and open way. At the end of the project we realized -that the skills developed by the students empowered them with the state in the -practice of software engineering. The members of the teams got opportunities to -work in public, private, national and international organizations, in addition -to those students they preferred entrepreneurship, opening their own companies. +%=========== +% Conclusion +%=========== + +\section{Final remarks} + +The portal is available at \url{softwarepublico.gov.br}. All +documentation, including detailed architecture and operation manuals are +also available\footnote{\url{https://softwarepublico.gov.br/doc/} +(in Portuguese only at the moment)}). +% +All the integrated tools are FOSS and our contributions were published +in open repositories, available on the SPB Portal itself. We also +contributed these features back to the respective communities: that +benefits those communities, as well as us since we can share future +development and maintenance effort with other organizations that +participate in their projects. + +More discussion ... + +... + +... + +%* utilização do projeto para formação de recursos humanos (alunos) + +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate + +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) + +%* 69 projetos marcados como SPB, de 81 no total na plataforma. + +%* 47\% é desenvolvido em PHP. + +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). + +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. + +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. + +%* Debater economia de recursos em orgão públicos diff --git a/sbqs2017/figures/arch3.png b/sbqs2017/figures/arch3.png new file mode 100644 index 0000000..570f167 Binary files /dev/null and b/sbqs2017/figures/arch3.png differ diff --git a/sbqs2017/figures/home-SPB_2.png b/sbqs2017/figures/home-SPB_2.png new file mode 100644 index 0000000..0390777 Binary files /dev/null and b/sbqs2017/figures/home-SPB_2.png differ diff --git a/sbqs2017/figures/technological-requirements.odg b/sbqs2017/figures/technological-requirements.odg index 815c544..9166cea 100644 Binary files a/sbqs2017/figures/technological-requirements.odg and b/sbqs2017/figures/technological-requirements.odg differ diff --git a/sbqs2017/figures/technological-requirements.png b/sbqs2017/figures/technological-requirements.png index 90fd00b..cbad494 100644 Binary files a/sbqs2017/figures/technological-requirements.png and b/sbqs2017/figures/technological-requirements.png differ diff --git a/sbqs2017/spb.tex b/sbqs2017/spb.tex index 557ee59..96561c6 100644 --- a/sbqs2017/spb.tex +++ b/sbqs2017/spb.tex @@ -12,12 +12,12 @@ \begin{document} \sloppy -\title{ Development experience report on the new \\ Brazilian Public Software Portal} +\title{Lições aprendidas ao longo do desenvolvimento do novo \\ Portal da Software Público Brasileiro} -\author{Paulo Meirelles\inst{1,3}, Antonio Terceiro\inst{2}, Hilmer Neri\inst{1},\\ - Melissa Wen\inst{2}, Rodrigo Siqueira\inst{3}, Lucas Kanashiro\inst{3}} +\author{Paulo Meirelles\inst{1,3}, Antonio Terceiro\inst{2}, Melissa Wen\inst{2}, + \\Rodrigo Siqueira\inst{3}, Hilmer Neri\inst{1}} -\address{Laboratory of Production, Research and Innovation in Software (LAPPIS)\\ +\address{Laboratório de Produção, Pesquisa e Inovação em Sofware(LAPPIS)\\ Faculdade UnB Gama (FGA) -- Universidade de Brasília (UnB)\\ Área Especial de Indústria Projeção A, Setor Leste -- Gama - DF -- Brasil \email{\{paulormm,hilmer\}@unb.br} @@ -26,10 +26,10 @@ Rua Marechal Floriano, 28, Canela, Salvador - BA -- Brasil \email{\{terceiro,melissa\}@colivre.coop.br} \nextinstitute - Free Libre Open Source Competence Center (CCSL)\\ + Centro de Competência em Software Livre (CCSL)\\ Instituto de Matemática e Estatística (IME) -- Universidade de São Paulo (USP)\\ Rua do Matão, 1010, Cidade Universitária -- São Paulo - SP -- Brasil - \email{\{siqueira,lkd\}@ime.usp.br} + \email{\{paulormm,siqueira\}@ime.usp.br} } \maketitle -- libgit2 0.21.2