diff --git a/opensym2017/content/01-introduction.tex b/opensym2017/content/01-introduction.tex index eec54f6..c553034 100644 --- a/opensym2017/content/01-introduction.tex +++ b/opensym2017/content/01-introduction.tex @@ -3,14 +3,14 @@ The Brazilian Government released in the year 2000 the Eletronic Government program (eGov) aiming at democratizing information access and improving the -public provision quality of service and information. In 2003, the Federal Government created a +public provision quality of service and information. In 2003, the Federal +Government created a committee\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/ DecretoComite}} for implementation of Free/Libre/Open Source Software (FLOSS\footnote{In this work, the acronym ``FLOSS'' is used as a representative for ``Free Software'', ``Open Source Software'' (OSS), and``Free/Open Source -Software'' (FOSS).}) and thereafter a -circular-letter was sent to all Ministries in which the recommendation to adopt FLOSS -became a public +Software'' (FOSS).}) and thereafter a circular-letter was sent to all +Ministries in which the recommendation to adopt FLOSS became a public policy\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/circulardoministro}}. In 2007, the Brazilian Public Software Portal (\textit{Portal do Software Público Brasileiro}, in Portuguese) was released with the goal of sharing FLOSS @@ -82,9 +82,11 @@ can help other projects to overcome similar software engineering challenges in the future, as well as to illustrate how universities can improve the real-world experience of their students. -The remainder of this work is organized as follows. Section \ref{sec:spb} -discusses the concepts of Brazilian Public Software and Free Software. Section -\ref{sec:related} enumerates a number of related projects from other countries. +\begin{comment} +%FALTA ESPAÇO +The remainder of this work is organized as follows. Section \ref{sec:spb} +discusses the concepts of Brazilian Public Software and Free Software. Section +\ref{sec:related} enumerates some related projects from other countries. Section \ref{sec:researchdesign} presents the open questions these guided this paper. Section \ref{sec:requirements} reports how the Brazilian Government stakeholders collected the theoretical requirements as well as how we define @@ -92,12 +94,9 @@ the technological requirements to release an initial version. Section \ref{sec:architecture} shares our decisions about the systems that together provided a wide subset of the requirements and our strategy to integrate them. Section \ref{sec:features} describes the main features of the new SPB Portal. -Section \ref{sec:ux} reports the user experience evolution during the -integration of the selected FLOSS tools. Section \ref{sec:process} discusses -our strategies to support the different organizational cultures and to involve -undergraduate students as protagonists of the development process. Section -\ref{sec:contributions} summarizes the contributions to the FLOSS upstream -communities we interacted with. Finally, Sections \ref{sec:conclusion} -concludes the paper highlighting its main contributions, sharing our lessons -learned, and pointing paths to future works. - +Section \ref{sec:process} discusses our strategies to support the different +organizational cultures and to involve undergraduate students as protagonists +of the development process. Finally, Sections \ref{sec:conclusion} concludes +the paper highlighting its main contributions, sharing our lessons learned, and +pointing paths to future works. +\end{comment} diff --git a/opensym2017/content/03-relatedworks.tex b/opensym2017/content/03-relatedworks.tex index e4676bf..0ea7bd4 100644 --- a/opensym2017/content/03-relatedworks.tex +++ b/opensym2017/content/03-relatedworks.tex @@ -11,10 +11,8 @@ making it easy for users and developers to obtain information and to access thei source code repositories at GitHub. However, it does not provide social networking and CMS features, as well as, other communication resources. -Additionally, there are two initiatives in Europe: -OSOR\footnote{\url{https://joinup.ec.europa.eu/community/osor}} and -OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) is a -community hosted in the JoinUp platform powered by the European Commission. +The European Commission supports the Open Source Observatory\footnote{\url{https://joinup.ec.europa.eu/community/osor}} (OSOR) that is a +community hosted in the JoinUp platform. OSOR intends to exchange information, experiences, and best practices around the use of FLOSS in the public administration. It supports the discovering of FLOSS projects made available by public agencies, providing information about these projects, such @@ -23,13 +21,6 @@ It also offers discussion forums and community mailing lists. But it does not have an integrated source code repository manager, so for each project there is a link to its own external repository. % -OW2 is an organization that promotes the development, deployment and management -of FLOSS middleware, generic -business applications, and cloud computing platforms, besides fostering a community and -business ecosystem around such projects. - -\leo{E aí, o que a OW2 tem a ver com o SPB?} - Moreover, in 2007 the European Comission published a study examining the impact that the development and distribution of FLOSS by public agencies have on eGovernment services and the economy~\cite{ghosh2004}. And, from 2007 to 2011, there @@ -61,6 +52,3 @@ avoiding the usage of private platforms, especially the ones provided by foreign companies. Therefore, we needed to develop our own solution to cover all the requirements, producing a complete governmental integrated platform for collaborative software development. - -\leo{Me parece que o decreto fala sobre sistemas de comunicação. Acho que o foco era e-mail. Não me parece que o SPB tenha a ver com isso. A menos que se considere algo pelos fóruns... para refletir.} - diff --git a/opensym2017/content/06-architecture.tex b/opensym2017/content/06-architecture.tex index ff7d066..4e89b20 100644 --- a/opensym2017/content/06-architecture.tex +++ b/opensym2017/content/06-architecture.tex @@ -104,7 +104,7 @@ stores the metrics evolution history, presents results in a friendly way, as well as, allows users to customize the given interpretation accordingly to their own context. -\subsection{System unification} +\subsection{System unification and User eXperience evolution} \begin{figure}[hbt] \centering @@ -123,7 +123,33 @@ functionality of each application, a search o n the SPB portal might return content from any of the applications, be it web pages, mailing list posts, or source code. -% Falar do devops +The integration of collaborative environments goes beyond functional aspects. +Offering the population an unified experience across these environments has +been the key to encourage the use of the platform as it reduces the perception +of complexity. Thus, the SPB Portal information architecture was redesigned +to provide a transparent navigation and to reach users with different profiles. +A process of harmonization has been employed on the interaction models of each +tool to reduce the learning curve. At the same time, a new visual style was +created to unify the navigation experience and to comply with the guidelines of +the digital communication identity standard established by the Federal +Government. + +With the increase in system features and the addition of new tools, the +visual style has steadily evolved to keep the navigation unified. Moreover, +tools from different backgrounds, which in many cases provide similar +functionality, prompted the development of an unified interface. Some +features, such as search and user profile editing were eliminated from +the individual applications, and implemented centrally to ensure a +consistent look and feel. + +Another challenge was responsive web design. The integrated applications +had varying degrees of support for responsiveness, and the common +interface had to adapt for each individual scenario. In particular +Noosfero did not yet have a responsive design; we engaged in its +development and contributed towards that goal. + +\begin{comment} +%FALTA DE ESPAÇO \subsection{Deploy} The SPB platform was deployed in 7 virtual machines with different functions, @@ -161,3 +187,4 @@ static analysis tools on source code stored in repository and provide this data to Prezento. A social network and CMS (Content Manager System) is provided by Noosfero in \textit{social}, and the databases of all tools with a cache service are in \textit{database}. +\end{comment} diff --git a/opensym2017/content/07-features.tex b/opensym2017/content/07-features.tex index 0c37166..e671eb7 100644 --- a/opensym2017/content/07-features.tex +++ b/opensym2017/content/07-features.tex @@ -74,3 +74,4 @@ Thus, the SPB became a platform to stimulate the openness of the source code; dialogue between users and the development team; and also maintenance and evolution of the software, which will provide more transparency in government investments. + diff --git a/opensym2017/content/08-process.tex b/opensym2017/content/08-process.tex new file mode 100644 index 0000000..43c02e5 --- /dev/null +++ b/opensym2017/content/08-process.tex @@ -0,0 +1,181 @@ +\section{Development Organization and Process} +\label{sec:process} + +The SPB team was composed of a variety of professionals with different levels +and skills, where most of them were undergraduate students with major in +software engineering (from 4th semester or upper). Since the students could +not dedicate many hours per week to the project, they always had the +flexibility to negotiate their work schedule during the semester in +order to not harm their classes and coursework. Their daily work routine +in the project included programming and DevOps tasks. + +The development of SPB project required a vast experience and background that +usually undergraduate students do not have yet. For this reason, a few senior +developers have joined to the project to help with the more difficult issues +and to transfer knowledge to the students. Their main task was to provide +solutions for complex problems, in other words, they worked as developers. As +these professionals are very skillful and the project could not fund a full +time work for them, some of them worked partially on the project. In addition, +they lived in a different states spread around Brazil which led much of the +communication to be online. + +In short, our work process was based on open and collaborative software +development practices. The development process was defined based on the +adaptation of different agile and FLOSS communities practices, highlighting the +high degree of automation resulting from DevOps practices. Thus, the work +process was executed in a cadenced and continuous way. + +Finally, the last group of actors of this project was composed of employees +formally working for the Brazilian government, in the Ministry of Planning, +Development, and Management (MP is the Brazilian acronym). All the project +decisions, validations, and scope definitions were made by them. In this way we +developed software product incrementally, with releases aligned to business +strategic objectives. As one can see, the project had many different +sort of stakeholders that had to be organized and synchronized. + +\subsection{Team organization} + +Approximately 70\% of the development teams were composed of software +engineering undergraduate students from UnB and they worked physically +in the same laboratory. Each student had their own schedule based on +their classes, what complicates the implementation of pair programming. +The senior developers tried to synchronize their schedule with the +students on their sub-team. To cope with this environment, we had a few +basic rules which guided the project organization: + +\begin{enumerate} + \item Classes have high priority for undergraduate students; + \item Work in pairs whenever possible (locally or remotely); + \item There must be one morning or afternoon per week when + \emph{everyone} should be together physically in the laboratory + (except of course the remote team members); + \item Every 3 to 4 months, the senior developers would fly in and work + alongside the students for a few days. +\end{enumerate} + +With the aforementioned rules we divided all the project into four +different teams: Colab, Noosfero, Design, and DevOps. Each team had one +coach responsible for reducing the communication problem with the other +teams and help the members to organize themselves in the best way for +everyone (always respecting their work time). The coach was one of the +students working in some of the teams with the extra duty of registering +the current tasks developed in the sprint and with the responsibility to +talk with other teams. One important thing to notice is the mutability +of the team and the coach, during the project many students changed +between the teams to try different areas. + +One characteristic of the teams was the presence of (at least) one +senior per team. This was essential, because hard decisions and complex +problems were usually referred to them. This kept having to take +complicated technical decisions from the coach tole, what encouraged +students to be coaches. Lastly, the senior developers worked directly +with the students, and this was important to give the undergraduate the +opportunity to interact with a savvy professional in his area and to +keep the knowledge flowing in the project. + +Finally, we had to add two last elements of the team organization, that +was essential for the project harmony: the meta-coach and professors. +The former was a software engineer recently graduated and which wanted +to keep working on the project, the latter were professors that +orchestrated all the interactions between all members of the project. +The meta-coach usually worked in one specific team and had the extra +task of knowing the current status of all teams. Professors and +meta-coaches worked together to reduce the communication problem between +all the teams. Lastly, all the paperwork tasks, such as reporting on the +project progress to the Ministry, was handled by the professors. + +\subsection{Communication and management} + +Our team had many people working together, and most of the seniors worked in a +different city remotely. Also, we tried to keep our work completely clear to +the Brazilian government and citizens interested in following the project. To +handle those cases, we used a set of tools to communication and other to manage +the project. + +For communication between members in different places, we used: video +conferencing with shared terminal tools, IRC, and mailing lists. For example, +when one student had to work in pair with a senior, normally, they used video +conferencing tool for talking and shared a terminal session (both typing and +seeing the screen in real time). For questions and fast discussion, we used +IRC. For general notification, we used the mailing lists. + +For managing the project we used the SPB Portal itself; first to validate it by +ourselves, and also because it had all the required tools. We basically created +one wiki page per release in the SPB Gitlab instance with a mapping between +strategical, tactical, and operational views. In a practical point of view, one +milestone per user history (feature), and one or more issues for addressing +each feature. With this approach we achieved two important goals: keeping all +the management as close possible to to the source code, and tracking every +feature developed during the project. Our decision to +use Wiki initially was empirical, but later reinforced by a research conducted +by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, +opensourcestyle}. + +\subsection{High-level project management and reporting} + +The Brazilian government used to work with software development in a +very traditional way. They would frequently focus on documents and not +on what was, in our opinion, what really matters: working software. This +dissonance caused us a communication noise with MP, because they would +often question our work style. It was especially hard to convince them +to accept the idea of open scope and agile development, but after months +of labor and showing results they stopped resisting. + +We defined some level of meeting granularity to avoid generating too +much overhead to the developers. We had a strategical and validating +meeting with MP (the former once in a month and the latter each 15th +day), release plaining with the entire team (one per month), and finally +a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a +diagram that represents our meeting organization. + +\begin{figure}[hbt] + \centering + \includegraphics[width=\linewidth]{figures/meeting_flows.png} + \caption{Meetings cycles.} + \label{fig:meeting} +\end{figure} + +In the strategical meeting we usually defined the priorities and new +features with the Brazilian government. Normally the professors, the +coach of each team, the meta-coach, and some employees of the MP would +participate in this meeting. We usually discussed what the team already +produced since our last meeting, and established the new features for +the next release. Notice that just part of the team would join this +meeting to avoid generating unnecessary overhead to the developers, but +all the students interested to participate were allowed to join (many +students wanted this experience during the project). + +After the strategical meeting with Brazilian government agents, we had a +planning turn with all teams together. In this part, each team worked together +to convert the MP wishes into smaller parts which were represented by the epics of +the release. Each coach was responsible for conducting the planning, and after +that record it on the project wiki (the wiki provided by Gitlab). With this +epic, each 14th day the team have generated their sprint scheduler (with small +achievements mapped to issues). + +To keep the Brazilian government always updated, we invited them to work +with us to validate the new features in progress. Normally we had a +meeting each 15th day. Basically, this was our work flow, we always kept +everything extremely open to the MP (our way of working, and the one +often used by open source projects) and to the team. + +To keep the track of all of those things we used the SPB itself, +especially Gitlab. Basically, we had: + +\begin{enumerate} + \item Project repository: We have one organization with many repositories; + \item Milestones: Each milestone was used to register a release; + \item Wiki: Each release has one page on wiki with the compilation of + strategical meeting; + \item Issues: Each sprint planning generated issues, which we associated to + the specific milestone and updated the wiki with the issue number related + with them. Finally each developer assigned the issue to itself. +\end{enumerate} + +Notice that this workflow gave us and the Brazilian government agents full +traceability from a high level view of each feature to the lowest level (code). +It is important to highlight that we converge to this workflow after many +experiments. For example, we used a tool named Redmine for organizing our tasks +during the sprints, however, this tool revealed inefficient for our case since +the government agents lost part of the project traceability. We realized that +centralize all the work in the SPB portal is the best option for our case. diff --git a/opensym2017/content/08-ux.tex b/opensym2017/content/08-ux.tex deleted file mode 100644 index a5e539c..0000000 --- a/opensym2017/content/08-ux.tex +++ /dev/null @@ -1,34 +0,0 @@ -\section{User eXperience evolution} -\label{sec:ux} - -The integration of collaborative environments goes beyond functional aspects. -Offering the population an unified experience across these environments has -been the key to encourage the use of the platform as it reduces the perception -of complexity. Thus, the SPB Portal information architecture was redesigned -to provide a transparent navigation and to reach users with different profiles. -A process of harmonization has been employed on the interaction models of each -tool to reduce the learning curve. At the same time, a new visual style was -created to unify the navigation experience and to comply with the guidelines of -the digital communication identity standard established by the Federal -Government. - -With the increase in system features and the addition of new tools, the -visual style has steadily evolved to keep the navigation unified. Moreover, -tools from different backgrounds, which in many cases provide similar -functionality, prompted the development of an unified interface. Some -features, such as search and user profile editing were eliminated from -the individual applications, and implemented centrally to ensure a -consistent look and feel. - -Another challenge was responsive web design. The integrated applications -had varying degrees of support for responsiveness, and the common -interface had to adapt for each individual scenario. In particular -Noosfero did not yet have a responsive design; we engaged in its -development and contributed towards that goal. - -After the initial release of the new SPB Portal in 2014, several -validations activities were implemented in 2015 and 2016. The aim was to -provide the most wanted features by casual users (such as public -servants interested in downloads and documentation) immediately, while -allowing more experienced users (such as developers) to easily drill down -to the details. diff --git a/opensym2017/content/09-conclusion.tex b/opensym2017/content/09-conclusion.tex new file mode 100644 index 0000000..c1f181c --- /dev/null +++ b/opensym2017/content/09-conclusion.tex @@ -0,0 +1,220 @@ +\section{Conclusion} +\label{sec:conclusion} + +In this work, we presented and discussed issues experienced during a government-funded +project, in partnership with the University of Brasilia and the University of +São Paulo, to evolve the Brazilian Public Software Portal. +Its contributions are twofold. First, we present the strategy used to develop +and to deliver an unprecedented platform to Brazilian government. Second, +based on the results of the SPB Portal project, we point out that it is +possible to mitigate conflicts experienced in the development environment +and to conciliate governmental and academy cultures. To summarize our main contributions, we answered in this section the three open questions those guided this paper. + + +\textbf{Q1 -- Which strategy could be used to integrate several existing +FLOSS tools to promote the collaborative software development?} +% +The SPB portal integrates more than 10 FLOSS tools and provides several features, +such as social network, mailing list, version control, content management and +source code quality monitoring. Concerned with the platform sustainability and +maintainability, the aforementioned 10 FLOSS tools were integrated with minimum +differences from their official versions and the new developed features were +sent upstream to ensure an alignment between the portal systems and their +respective official versions. In the integration process, the main softwares +were identified, specific teams were formed to work with each one of them +and each team was composed of students with different levels of skills and at +least one senior professional. + +\textbf{Q2 -- How to involve students in real-world projects interacting with +real customers?} +% +In terms of mitigating conflicts, we tried to show that, as long as the +university can provide a healthy and challenging environment to its students, +one may conciliate studies and professional training in universities. +% +The students interacted with professionals of diverse fields of expertise, and they were +able to participate in all levels of the software development process. +This contributed to a great learning opportunity. +% +In our work process, based on open and collaborative software development +practices, students could negotiate their work schedule as well as count on IT +professionals to solve development issues. +% +Among the students, we have defined coaches for each team and a meta-coach +(coach of whole project). All coaches, together with professors, have +intermediated the communication between client (Ministry of Planning of Brazil) +and the rest of the group. +After the end of the project, some students successfully +embraced opportunities in public and private sectors, within national borders +and abroad. Some other students went further and started their own companies. + +\textbf{Q3 -- How to introduce the FLOSS collaborative and agile +practices to governmental development process?} +With some adaptations, what we called the ``translation processes'', it is feasible +to conciliate agile methodologies and FLOSS practices to develop software to +governmental organizations with functional hierarchical structures that use +traditional development paradigm. +% +Aiming at reducing client questions about workconclusion, a DevOps front was +created to automate all deploy process and also to work in continuous +delivery. The government was brought to our work environment and interacted +with our management and communication tools. For the project success, we +focused on providing a friendly working environment as well as on showing to +governmental agents another way to interact with the FLOSS community and the +university. + +\subsection{Lessons Learned} +\label{sec:lessons} + +From the answers of our initial open questions, we can also highlighted six lessons learned to make easier to share our experience during the development of the new SPB Portal. + +\textbf{L1 -- The participation of experienced professionals is crucial to +the success of the project.} +One important factor for the students was the composition of the teams +with the participation of experienced professionals. +% +On the technical side, the senior developers and designers would handle +the more difficult technical decisions, creating a work environment +where the students could develop their skills in a didactic way without +pressure. +% +On the management side, the active participation of professors -- who +are, in the end, the ones responsible for the project -- is crucial to +make sure students participation is conducted in a healthy way, and it is an +instance of leading by example. + +\textbf{L2 -- A balanced relationship between academia and industry.} +The experience of the SPB project led UnB to develop a work style which +proved to be appropriate for an educational environment that brings +academia and industry together. +% +The highest priority from the university's point of view is the +students. Considering this, the activities of the project were never +prioritized to the detriment of classes and other pedagogical +activities. In summary, we had students working at different times, part +time, remotely or locally, always respecting their individual +conditions, but doing the work in a collective, collaborative and open +way. +% +And even under a potentially adverse environment, the project delivered +the desired solution with success. +% +At the end of the project, we noted that the skills developed by the +students were at the software engineering state of the art. +After the project ended, we had team members successfully embracing +opportunities in public, private, national, and international +organizations, in addition to those students that +opened their own companies. + +\textbf{L3 -- Managing different organizational cultures.} +In the beginning of the project, the Brazilian Government stakeholders +had certain expectations about the project development that, let's +say, didn't exactly match our work style based on agile and FLOSS practices. +% +We had to develop strategies that would support these different +organizational cultures. The +MP is organized in a functional hierarchical organizational structure, +typically adopting a traditional development paradigm. Therefore, we had to +create a translation process between our team and the MP managers who +managed the project on their side assuming a traditional waterfall +process. + +\textbf{L4 -- Managing higher-level and lower-level goals separately.} +During the initial phase of the project, the MP team has brought +strategic discussions to technical/operational meetings that +were supposed to be about practical technical decisions. +% +This produced a highly complex communication and management environment, +overloading the professors because they were supposed +for maintaining the MP strategy synchronized with the +implementation plans of the development team. This was hard, especially because the +aforementioned cultural mismatch. Mixing both concerns in the same +discussions caused confusion on both sides. +% +From the middle of the project we were able to keep those +concerns separated, what eased the work of everyone involved. + +\textbf{L5 -- Living with ill-advised political decisions.} +At the initial phases of the project, by political and personal +motivation, the main stakeholders from the Brazilian government imposed +the use of Colab to guide the development of the new SPB platform. Our +team was totally against the idea because we already knew that Colab was +a very experimental project and its adoption could dramatically increase +the project complexity. Even though we provided technical reasons to +not utilize Colab, the MP was adamant and we had to manage this problem. We did massive changes to +Colab, and at the end of the project we have completely rewritten it to make +it stable. It is important to notice that the MP compelled us to accept +a technical decision based only on political interests, without considering +all the resources that would be spent due to that decision. At the end +of the project, we verified that Colab indeed consumed a vast amount of +the budget and increased the project complexity. At the end of the +project and after our analysis on the decision made by the MP, we +understand that MP managers are capable of ignoring technical reasons +in favor of political decisions. + +\textbf{L6 -- Consider sustainability from the beginning.} +In the process of deploying the SPB platform in the MP infrastructure we had +to interact with the MP technicians. We did several workshops, training +and a meticulous documentation describing all the required procedures to +update the platform, however, we realized that the MP technicians +constantly avoid the responsibility. After noticing the aforementioned +situation, we organized a DevOps team that completely automated all +the deployment procedure. We simplified all the platform deployment to a +few steps: (1) initial configurations (just ssh configuration) and (2) +the execution of simple commands to completely update the platform. By +the end of the project, we observed that the MP technicians invariably +still depended on our support to update the platform even with all the +automation provided by us. We were sadly left with a feeling of +uncertainty about the future of the platform after the project ended. In +hindsight, we realize that the MP dedicated system analysts and +managers to the project, but not operations technicians. The later +should have been more involved with the process so they could at least be +comfortable in managing the platform infrastructure. + + +\textbf{Final remarks and future work} + +The portal is available at \url{softwarepublico.gov.br}. All +documentation, including detailed architecture and operation manuals are +also available\footnote{\url{https://softwarepublico.gov.br/doc/} +(in Portuguese only at the moment)}. +% +All the integrated tools are FLOSS and our contributions were published +in open repositories, available on the SPB Portal itself. We also +contributed these features back to the respective communities, which +benefits both those communities and us, since we can share future +development and maintenance effort with other organizations that +participate in these projects. + +Future work should use data produced by the project to validate and evaluate +how the used FLOSS and Agile practices have impacted the students and the +governmental development process. For this, we would conduce a \textit{postmortem} +analysis using the project open data and a survey targeting the involved actors. + +%=========== +% Conclusion +%=========== + +% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados +%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa +%% afirmação, embora eu e Paulo consigamos perceber isso. + + +%* utilização do projeto para formação de recursos humanos (alunos) + +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate + +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) + +%* 69 projetos marcados como SPB, de 81 no total na plataforma. + +%* 47\% é desenvolvido em PHP. + +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). + +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. + +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. + +%* Debater economia de recursos em orgão públicos + diff --git a/opensym2017/content/09-process.tex b/opensym2017/content/09-process.tex deleted file mode 100644 index 43c02e5..0000000 --- a/opensym2017/content/09-process.tex +++ /dev/null @@ -1,181 +0,0 @@ -\section{Development Organization and Process} -\label{sec:process} - -The SPB team was composed of a variety of professionals with different levels -and skills, where most of them were undergraduate students with major in -software engineering (from 4th semester or upper). Since the students could -not dedicate many hours per week to the project, they always had the -flexibility to negotiate their work schedule during the semester in -order to not harm their classes and coursework. Their daily work routine -in the project included programming and DevOps tasks. - -The development of SPB project required a vast experience and background that -usually undergraduate students do not have yet. For this reason, a few senior -developers have joined to the project to help with the more difficult issues -and to transfer knowledge to the students. Their main task was to provide -solutions for complex problems, in other words, they worked as developers. As -these professionals are very skillful and the project could not fund a full -time work for them, some of them worked partially on the project. In addition, -they lived in a different states spread around Brazil which led much of the -communication to be online. - -In short, our work process was based on open and collaborative software -development practices. The development process was defined based on the -adaptation of different agile and FLOSS communities practices, highlighting the -high degree of automation resulting from DevOps practices. Thus, the work -process was executed in a cadenced and continuous way. - -Finally, the last group of actors of this project was composed of employees -formally working for the Brazilian government, in the Ministry of Planning, -Development, and Management (MP is the Brazilian acronym). All the project -decisions, validations, and scope definitions were made by them. In this way we -developed software product incrementally, with releases aligned to business -strategic objectives. As one can see, the project had many different -sort of stakeholders that had to be organized and synchronized. - -\subsection{Team organization} - -Approximately 70\% of the development teams were composed of software -engineering undergraduate students from UnB and they worked physically -in the same laboratory. Each student had their own schedule based on -their classes, what complicates the implementation of pair programming. -The senior developers tried to synchronize their schedule with the -students on their sub-team. To cope with this environment, we had a few -basic rules which guided the project organization: - -\begin{enumerate} - \item Classes have high priority for undergraduate students; - \item Work in pairs whenever possible (locally or remotely); - \item There must be one morning or afternoon per week when - \emph{everyone} should be together physically in the laboratory - (except of course the remote team members); - \item Every 3 to 4 months, the senior developers would fly in and work - alongside the students for a few days. -\end{enumerate} - -With the aforementioned rules we divided all the project into four -different teams: Colab, Noosfero, Design, and DevOps. Each team had one -coach responsible for reducing the communication problem with the other -teams and help the members to organize themselves in the best way for -everyone (always respecting their work time). The coach was one of the -students working in some of the teams with the extra duty of registering -the current tasks developed in the sprint and with the responsibility to -talk with other teams. One important thing to notice is the mutability -of the team and the coach, during the project many students changed -between the teams to try different areas. - -One characteristic of the teams was the presence of (at least) one -senior per team. This was essential, because hard decisions and complex -problems were usually referred to them. This kept having to take -complicated technical decisions from the coach tole, what encouraged -students to be coaches. Lastly, the senior developers worked directly -with the students, and this was important to give the undergraduate the -opportunity to interact with a savvy professional in his area and to -keep the knowledge flowing in the project. - -Finally, we had to add two last elements of the team organization, that -was essential for the project harmony: the meta-coach and professors. -The former was a software engineer recently graduated and which wanted -to keep working on the project, the latter were professors that -orchestrated all the interactions between all members of the project. -The meta-coach usually worked in one specific team and had the extra -task of knowing the current status of all teams. Professors and -meta-coaches worked together to reduce the communication problem between -all the teams. Lastly, all the paperwork tasks, such as reporting on the -project progress to the Ministry, was handled by the professors. - -\subsection{Communication and management} - -Our team had many people working together, and most of the seniors worked in a -different city remotely. Also, we tried to keep our work completely clear to -the Brazilian government and citizens interested in following the project. To -handle those cases, we used a set of tools to communication and other to manage -the project. - -For communication between members in different places, we used: video -conferencing with shared terminal tools, IRC, and mailing lists. For example, -when one student had to work in pair with a senior, normally, they used video -conferencing tool for talking and shared a terminal session (both typing and -seeing the screen in real time). For questions and fast discussion, we used -IRC. For general notification, we used the mailing lists. - -For managing the project we used the SPB Portal itself; first to validate it by -ourselves, and also because it had all the required tools. We basically created -one wiki page per release in the SPB Gitlab instance with a mapping between -strategical, tactical, and operational views. In a practical point of view, one -milestone per user history (feature), and one or more issues for addressing -each feature. With this approach we achieved two important goals: keeping all -the management as close possible to to the source code, and tracking every -feature developed during the project. Our decision to -use Wiki initially was empirical, but later reinforced by a research conducted -by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, -opensourcestyle}. - -\subsection{High-level project management and reporting} - -The Brazilian government used to work with software development in a -very traditional way. They would frequently focus on documents and not -on what was, in our opinion, what really matters: working software. This -dissonance caused us a communication noise with MP, because they would -often question our work style. It was especially hard to convince them -to accept the idea of open scope and agile development, but after months -of labor and showing results they stopped resisting. - -We defined some level of meeting granularity to avoid generating too -much overhead to the developers. We had a strategical and validating -meeting with MP (the former once in a month and the latter each 15th -day), release plaining with the entire team (one per month), and finally -a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a -diagram that represents our meeting organization. - -\begin{figure}[hbt] - \centering - \includegraphics[width=\linewidth]{figures/meeting_flows.png} - \caption{Meetings cycles.} - \label{fig:meeting} -\end{figure} - -In the strategical meeting we usually defined the priorities and new -features with the Brazilian government. Normally the professors, the -coach of each team, the meta-coach, and some employees of the MP would -participate in this meeting. We usually discussed what the team already -produced since our last meeting, and established the new features for -the next release. Notice that just part of the team would join this -meeting to avoid generating unnecessary overhead to the developers, but -all the students interested to participate were allowed to join (many -students wanted this experience during the project). - -After the strategical meeting with Brazilian government agents, we had a -planning turn with all teams together. In this part, each team worked together -to convert the MP wishes into smaller parts which were represented by the epics of -the release. Each coach was responsible for conducting the planning, and after -that record it on the project wiki (the wiki provided by Gitlab). With this -epic, each 14th day the team have generated their sprint scheduler (with small -achievements mapped to issues). - -To keep the Brazilian government always updated, we invited them to work -with us to validate the new features in progress. Normally we had a -meeting each 15th day. Basically, this was our work flow, we always kept -everything extremely open to the MP (our way of working, and the one -often used by open source projects) and to the team. - -To keep the track of all of those things we used the SPB itself, -especially Gitlab. Basically, we had: - -\begin{enumerate} - \item Project repository: We have one organization with many repositories; - \item Milestones: Each milestone was used to register a release; - \item Wiki: Each release has one page on wiki with the compilation of - strategical meeting; - \item Issues: Each sprint planning generated issues, which we associated to - the specific milestone and updated the wiki with the issue number related - with them. Finally each developer assigned the issue to itself. -\end{enumerate} - -Notice that this workflow gave us and the Brazilian government agents full -traceability from a high level view of each feature to the lowest level (code). -It is important to highlight that we converge to this workflow after many -experiments. For example, we used a tool named Redmine for organizing our tasks -during the sprints, however, this tool revealed inefficient for our case since -the government agents lost part of the project traceability. We realized that -centralize all the work in the SPB portal is the best option for our case. diff --git a/opensym2017/content/10-contributions.tex b/opensym2017/content/10-contributions.tex deleted file mode 100644 index 91bf61d..0000000 --- a/opensym2017/content/10-contributions.tex +++ /dev/null @@ -1,58 +0,0 @@ -\section{Contributions to the upstream communities} -\label{sec:contributions} - -%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) -%* Colab -> RevProxy -%* Colab, atualização do python/django -%* Contribuições para o GitLab (autenticação) -%* Noosfero, atualização do Rails, preparação para federação, nova interface ... -%* Coper, empacotamentos (obs), omniauth - - -During the execution of this project we made several contributions to -the upstream communities we interacted with. This occurred due to our -development process aligned with those of the respective communities. -During development, we would explicitly discuss the features and bug -fixes that we were working on with the applicable upstream communities. -This contributed to improve the developers technical solutions with -expertise outside of our team, and make it easier for those changes to -be accepted in the original projects. Having changes accepted upstream -in turn makes our life easier as it minimizes the delta between our -codebase and upstream's allowing us to upgrade and benefit from -development work from others. - -In Colab, we helped upstream redesign the entirely architecture, -enabling the development of plugins to integrate new tools. We also -added a feature that allowed Colab to run asynchronous tasks, which was -a major improvement for us since we were developing a complex system. A -migration to the latest Django version was made (web framework used by -Colab). Moreover, we worked on RevProxy (the more important dependency -of Colab) to put it in a good shape, fixing many bugs. - -Gitlab was the tool that we made the least number of modifications. We -contributed with some improvements related with configuration files and -we developed a new plugin that enables user authentication in Gitlab -through the REMOTE\_USER HTTP header. This plugin was needed because -Colab uses this mechanism to manage the authentication. - -Noosfero was the tool that contemplated several functional requirements, -therefore we made a large number of contributions with upstream. We helped to -migrate to the latest Rails version (web framework used by Noosfero), enable -the federation implementation (federation with other social networks), and -decouple the interface and the back-end. - -Mezuro was completely rewritten and its architecture evolved adopting the -microservice architecture\cite{namiot2014micro}. This way, we minimize the -amount of code to maintain it, helping to test it and grant its code quality. -In practice, we modularize the Mezuro project in several independent services. -Currently, its computation and visualization modules use Kalibro and Prezento, -respectively. They were developed into the Mezuro project and evolved during -its integration to the new SPB Portal. - - -We also made some contributions on the DevOps front. Some members of -them team became maintainers of some Python libraries that were used by -our scripts to upload packages to OBS (Open Build Service). We developed -a tool called copr-status to keep track of the different stages of the -lifecycle of each of the individual packages we were working on. - diff --git a/opensym2017/content/11-conclusion.tex b/opensym2017/content/11-conclusion.tex deleted file mode 100644 index c1f181c..0000000 --- a/opensym2017/content/11-conclusion.tex +++ /dev/null @@ -1,220 +0,0 @@ -\section{Conclusion} -\label{sec:conclusion} - -In this work, we presented and discussed issues experienced during a government-funded -project, in partnership with the University of Brasilia and the University of -São Paulo, to evolve the Brazilian Public Software Portal. -Its contributions are twofold. First, we present the strategy used to develop -and to deliver an unprecedented platform to Brazilian government. Second, -based on the results of the SPB Portal project, we point out that it is -possible to mitigate conflicts experienced in the development environment -and to conciliate governmental and academy cultures. To summarize our main contributions, we answered in this section the three open questions those guided this paper. - - -\textbf{Q1 -- Which strategy could be used to integrate several existing -FLOSS tools to promote the collaborative software development?} -% -The SPB portal integrates more than 10 FLOSS tools and provides several features, -such as social network, mailing list, version control, content management and -source code quality monitoring. Concerned with the platform sustainability and -maintainability, the aforementioned 10 FLOSS tools were integrated with minimum -differences from their official versions and the new developed features were -sent upstream to ensure an alignment between the portal systems and their -respective official versions. In the integration process, the main softwares -were identified, specific teams were formed to work with each one of them -and each team was composed of students with different levels of skills and at -least one senior professional. - -\textbf{Q2 -- How to involve students in real-world projects interacting with -real customers?} -% -In terms of mitigating conflicts, we tried to show that, as long as the -university can provide a healthy and challenging environment to its students, -one may conciliate studies and professional training in universities. -% -The students interacted with professionals of diverse fields of expertise, and they were -able to participate in all levels of the software development process. -This contributed to a great learning opportunity. -% -In our work process, based on open and collaborative software development -practices, students could negotiate their work schedule as well as count on IT -professionals to solve development issues. -% -Among the students, we have defined coaches for each team and a meta-coach -(coach of whole project). All coaches, together with professors, have -intermediated the communication between client (Ministry of Planning of Brazil) -and the rest of the group. -After the end of the project, some students successfully -embraced opportunities in public and private sectors, within national borders -and abroad. Some other students went further and started their own companies. - -\textbf{Q3 -- How to introduce the FLOSS collaborative and agile -practices to governmental development process?} -With some adaptations, what we called the ``translation processes'', it is feasible -to conciliate agile methodologies and FLOSS practices to develop software to -governmental organizations with functional hierarchical structures that use -traditional development paradigm. -% -Aiming at reducing client questions about workconclusion, a DevOps front was -created to automate all deploy process and also to work in continuous -delivery. The government was brought to our work environment and interacted -with our management and communication tools. For the project success, we -focused on providing a friendly working environment as well as on showing to -governmental agents another way to interact with the FLOSS community and the -university. - -\subsection{Lessons Learned} -\label{sec:lessons} - -From the answers of our initial open questions, we can also highlighted six lessons learned to make easier to share our experience during the development of the new SPB Portal. - -\textbf{L1 -- The participation of experienced professionals is crucial to -the success of the project.} -One important factor for the students was the composition of the teams -with the participation of experienced professionals. -% -On the technical side, the senior developers and designers would handle -the more difficult technical decisions, creating a work environment -where the students could develop their skills in a didactic way without -pressure. -% -On the management side, the active participation of professors -- who -are, in the end, the ones responsible for the project -- is crucial to -make sure students participation is conducted in a healthy way, and it is an -instance of leading by example. - -\textbf{L2 -- A balanced relationship between academia and industry.} -The experience of the SPB project led UnB to develop a work style which -proved to be appropriate for an educational environment that brings -academia and industry together. -% -The highest priority from the university's point of view is the -students. Considering this, the activities of the project were never -prioritized to the detriment of classes and other pedagogical -activities. In summary, we had students working at different times, part -time, remotely or locally, always respecting their individual -conditions, but doing the work in a collective, collaborative and open -way. -% -And even under a potentially adverse environment, the project delivered -the desired solution with success. -% -At the end of the project, we noted that the skills developed by the -students were at the software engineering state of the art. -After the project ended, we had team members successfully embracing -opportunities in public, private, national, and international -organizations, in addition to those students that -opened their own companies. - -\textbf{L3 -- Managing different organizational cultures.} -In the beginning of the project, the Brazilian Government stakeholders -had certain expectations about the project development that, let's -say, didn't exactly match our work style based on agile and FLOSS practices. -% -We had to develop strategies that would support these different -organizational cultures. The -MP is organized in a functional hierarchical organizational structure, -typically adopting a traditional development paradigm. Therefore, we had to -create a translation process between our team and the MP managers who -managed the project on their side assuming a traditional waterfall -process. - -\textbf{L4 -- Managing higher-level and lower-level goals separately.} -During the initial phase of the project, the MP team has brought -strategic discussions to technical/operational meetings that -were supposed to be about practical technical decisions. -% -This produced a highly complex communication and management environment, -overloading the professors because they were supposed -for maintaining the MP strategy synchronized with the -implementation plans of the development team. This was hard, especially because the -aforementioned cultural mismatch. Mixing both concerns in the same -discussions caused confusion on both sides. -% -From the middle of the project we were able to keep those -concerns separated, what eased the work of everyone involved. - -\textbf{L5 -- Living with ill-advised political decisions.} -At the initial phases of the project, by political and personal -motivation, the main stakeholders from the Brazilian government imposed -the use of Colab to guide the development of the new SPB platform. Our -team was totally against the idea because we already knew that Colab was -a very experimental project and its adoption could dramatically increase -the project complexity. Even though we provided technical reasons to -not utilize Colab, the MP was adamant and we had to manage this problem. We did massive changes to -Colab, and at the end of the project we have completely rewritten it to make -it stable. It is important to notice that the MP compelled us to accept -a technical decision based only on political interests, without considering -all the resources that would be spent due to that decision. At the end -of the project, we verified that Colab indeed consumed a vast amount of -the budget and increased the project complexity. At the end of the -project and after our analysis on the decision made by the MP, we -understand that MP managers are capable of ignoring technical reasons -in favor of political decisions. - -\textbf{L6 -- Consider sustainability from the beginning.} -In the process of deploying the SPB platform in the MP infrastructure we had -to interact with the MP technicians. We did several workshops, training -and a meticulous documentation describing all the required procedures to -update the platform, however, we realized that the MP technicians -constantly avoid the responsibility. After noticing the aforementioned -situation, we organized a DevOps team that completely automated all -the deployment procedure. We simplified all the platform deployment to a -few steps: (1) initial configurations (just ssh configuration) and (2) -the execution of simple commands to completely update the platform. By -the end of the project, we observed that the MP technicians invariably -still depended on our support to update the platform even with all the -automation provided by us. We were sadly left with a feeling of -uncertainty about the future of the platform after the project ended. In -hindsight, we realize that the MP dedicated system analysts and -managers to the project, but not operations technicians. The later -should have been more involved with the process so they could at least be -comfortable in managing the platform infrastructure. - - -\textbf{Final remarks and future work} - -The portal is available at \url{softwarepublico.gov.br}. All -documentation, including detailed architecture and operation manuals are -also available\footnote{\url{https://softwarepublico.gov.br/doc/} -(in Portuguese only at the moment)}. -% -All the integrated tools are FLOSS and our contributions were published -in open repositories, available on the SPB Portal itself. We also -contributed these features back to the respective communities, which -benefits both those communities and us, since we can share future -development and maintenance effort with other organizations that -participate in these projects. - -Future work should use data produced by the project to validate and evaluate -how the used FLOSS and Agile practices have impacted the students and the -governmental development process. For this, we would conduce a \textit{postmortem} -analysis using the project open data and a survey targeting the involved actors. - -%=========== -% Conclusion -%=========== - -% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados -%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa -%% afirmação, embora eu e Paulo consigamos perceber isso. - - -%* utilização do projeto para formação de recursos humanos (alunos) - -%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate - -%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) - -%* 69 projetos marcados como SPB, de 81 no total na plataforma. - -%* 47\% é desenvolvido em PHP. - -% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). - -% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. - -% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. - -%* Debater economia de recursos em orgão públicos - diff --git a/opensym2017/content/out-contributions.tex b/opensym2017/content/out-contributions.tex new file mode 100644 index 0000000..91bf61d --- /dev/null +++ b/opensym2017/content/out-contributions.tex @@ -0,0 +1,58 @@ +\section{Contributions to the upstream communities} +\label{sec:contributions} + +%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) +%* Colab -> RevProxy +%* Colab, atualização do python/django +%* Contribuições para o GitLab (autenticação) +%* Noosfero, atualização do Rails, preparação para federação, nova interface ... +%* Coper, empacotamentos (obs), omniauth + + +During the execution of this project we made several contributions to +the upstream communities we interacted with. This occurred due to our +development process aligned with those of the respective communities. +During development, we would explicitly discuss the features and bug +fixes that we were working on with the applicable upstream communities. +This contributed to improve the developers technical solutions with +expertise outside of our team, and make it easier for those changes to +be accepted in the original projects. Having changes accepted upstream +in turn makes our life easier as it minimizes the delta between our +codebase and upstream's allowing us to upgrade and benefit from +development work from others. + +In Colab, we helped upstream redesign the entirely architecture, +enabling the development of plugins to integrate new tools. We also +added a feature that allowed Colab to run asynchronous tasks, which was +a major improvement for us since we were developing a complex system. A +migration to the latest Django version was made (web framework used by +Colab). Moreover, we worked on RevProxy (the more important dependency +of Colab) to put it in a good shape, fixing many bugs. + +Gitlab was the tool that we made the least number of modifications. We +contributed with some improvements related with configuration files and +we developed a new plugin that enables user authentication in Gitlab +through the REMOTE\_USER HTTP header. This plugin was needed because +Colab uses this mechanism to manage the authentication. + +Noosfero was the tool that contemplated several functional requirements, +therefore we made a large number of contributions with upstream. We helped to +migrate to the latest Rails version (web framework used by Noosfero), enable +the federation implementation (federation with other social networks), and +decouple the interface and the back-end. + +Mezuro was completely rewritten and its architecture evolved adopting the +microservice architecture\cite{namiot2014micro}. This way, we minimize the +amount of code to maintain it, helping to test it and grant its code quality. +In practice, we modularize the Mezuro project in several independent services. +Currently, its computation and visualization modules use Kalibro and Prezento, +respectively. They were developed into the Mezuro project and evolved during +its integration to the new SPB Portal. + + +We also made some contributions on the DevOps front. Some members of +them team became maintainers of some Python libraries that were used by +our scripts to upload packages to OBS (Open Build Service). We developed +a tool called copr-status to keep track of the different stages of the +lifecycle of each of the individual packages we were working on. + diff --git a/opensym2017/spb.tex b/opensym2017/spb.tex index 8dea98f..513729d 100644 --- a/opensym2017/spb.tex +++ b/opensym2017/spb.tex @@ -151,10 +151,8 @@ \input{content/05-requirements} \input{content/06-architecture} \input{content/07-features} -\input{content/08-ux} -\input{content/09-process} -\input{content/10-contributions} -\input{content/11-conclusion} +\input{content/08-process} +\input{content/09-conclusion} %------------------------------------------------------------------------------ \bibliographystyle{SIGCHI-Reference-Format} -- libgit2 0.21.2