Commit 5588ee4556b73c0b72ae04c3bfc235fbd5edece1
1 parent
ca3c632b
Exists in
master
and in
3 other branches
[opensym] Trying 10 pages
Showing
12 changed files
with
508 additions
and
529 deletions
Show diff stats
opensym2017/content/01-introduction.tex
| @@ -3,14 +3,14 @@ | @@ -3,14 +3,14 @@ | ||
| 3 | 3 | ||
| 4 | The Brazilian Government released in the year 2000 the Eletronic Government | 4 | The Brazilian Government released in the year 2000 the Eletronic Government |
| 5 | program (eGov) aiming at democratizing information access and improving the | 5 | program (eGov) aiming at democratizing information access and improving the |
| 6 | -public provision quality of service and information. In 2003, the Federal Government created a | 6 | +public provision quality of service and information. In 2003, the Federal |
| 7 | +Government created a | ||
| 7 | committee\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/ | 8 | committee\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/ |
| 8 | DecretoComite}} for implementation of Free/Libre/Open Source Software | 9 | DecretoComite}} for implementation of Free/Libre/Open Source Software |
| 9 | (FLOSS\footnote{In this work, the acronym ``FLOSS'' is used as a representative | 10 | (FLOSS\footnote{In this work, the acronym ``FLOSS'' is used as a representative |
| 10 | for ``Free Software'', ``Open Source Software'' (OSS), and``Free/Open Source | 11 | for ``Free Software'', ``Open Source Software'' (OSS), and``Free/Open Source |
| 11 | -Software'' (FOSS).}) and thereafter a | ||
| 12 | -circular-letter was sent to all Ministries in which the recommendation to adopt FLOSS | ||
| 13 | -became a public | 12 | +Software'' (FOSS).}) and thereafter a circular-letter was sent to all |
| 13 | +Ministries in which the recommendation to adopt FLOSS became a public | ||
| 14 | policy\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/circulardoministro}}. | 14 | policy\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/circulardoministro}}. |
| 15 | In 2007, the Brazilian Public Software Portal (\textit{Portal do Software | 15 | In 2007, the Brazilian Public Software Portal (\textit{Portal do Software |
| 16 | Público Brasileiro}, in Portuguese) was released with the goal of sharing FLOSS | 16 | Público Brasileiro}, in Portuguese) was released with the goal of sharing FLOSS |
| @@ -82,9 +82,11 @@ can help other projects to overcome similar software engineering challenges in | @@ -82,9 +82,11 @@ can help other projects to overcome similar software engineering challenges in | ||
| 82 | the future, as well as to illustrate how universities can improve the | 82 | the future, as well as to illustrate how universities can improve the |
| 83 | real-world experience of their students. | 83 | real-world experience of their students. |
| 84 | 84 | ||
| 85 | -The remainder of this work is organized as follows. Section \ref{sec:spb} | ||
| 86 | -discusses the concepts of Brazilian Public Software and Free Software. Section | ||
| 87 | -\ref{sec:related} enumerates a number of related projects from other countries. | 85 | +\begin{comment} |
| 86 | +%FALTA ESPAÇO | ||
| 87 | +The remainder of this work is organized as follows. Section \ref{sec:spb} | ||
| 88 | +discusses the concepts of Brazilian Public Software and Free Software. Section | ||
| 89 | +\ref{sec:related} enumerates some related projects from other countries. | ||
| 88 | Section \ref{sec:researchdesign} presents the open questions these guided this | 90 | Section \ref{sec:researchdesign} presents the open questions these guided this |
| 89 | paper. Section \ref{sec:requirements} reports how the Brazilian Government | 91 | paper. Section \ref{sec:requirements} reports how the Brazilian Government |
| 90 | stakeholders collected the theoretical requirements as well as how we define | 92 | stakeholders collected the theoretical requirements as well as how we define |
| @@ -92,12 +94,9 @@ the technological requirements to release an initial version. Section | @@ -92,12 +94,9 @@ the technological requirements to release an initial version. Section | ||
| 92 | \ref{sec:architecture} shares our decisions about the systems that together | 94 | \ref{sec:architecture} shares our decisions about the systems that together |
| 93 | provided a wide subset of the requirements and our strategy to integrate them. | 95 | provided a wide subset of the requirements and our strategy to integrate them. |
| 94 | Section \ref{sec:features} describes the main features of the new SPB Portal. | 96 | Section \ref{sec:features} describes the main features of the new SPB Portal. |
| 95 | -Section \ref{sec:ux} reports the user experience evolution during the | ||
| 96 | -integration of the selected FLOSS tools. Section \ref{sec:process} discusses | ||
| 97 | -our strategies to support the different organizational cultures and to involve | ||
| 98 | -undergraduate students as protagonists of the development process. Section | ||
| 99 | -\ref{sec:contributions} summarizes the contributions to the FLOSS upstream | ||
| 100 | -communities we interacted with. Finally, Sections \ref{sec:conclusion} | ||
| 101 | -concludes the paper highlighting its main contributions, sharing our lessons | ||
| 102 | -learned, and pointing paths to future works. | ||
| 103 | - | 97 | +Section \ref{sec:process} discusses our strategies to support the different |
| 98 | +organizational cultures and to involve undergraduate students as protagonists | ||
| 99 | +of the development process. Finally, Sections \ref{sec:conclusion} concludes | ||
| 100 | +the paper highlighting its main contributions, sharing our lessons learned, and | ||
| 101 | +pointing paths to future works. | ||
| 102 | +\end{comment} |
opensym2017/content/03-relatedworks.tex
| @@ -11,10 +11,8 @@ making it easy for users and developers to obtain information and to access thei | @@ -11,10 +11,8 @@ making it easy for users and developers to obtain information and to access thei | ||
| 11 | source code repositories at GitHub. However, it does not provide social networking | 11 | source code repositories at GitHub. However, it does not provide social networking |
| 12 | and CMS features, as well as, other communication resources. | 12 | and CMS features, as well as, other communication resources. |
| 13 | 13 | ||
| 14 | -Additionally, there are two initiatives in Europe: | ||
| 15 | -OSOR\footnote{\url{https://joinup.ec.europa.eu/community/osor}} and | ||
| 16 | -OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) is a | ||
| 17 | -community hosted in the JoinUp platform powered by the European Commission. | 14 | +The European Commission supports the Open Source Observatory\footnote{\url{https://joinup.ec.europa.eu/community/osor}} (OSOR) that is a |
| 15 | +community hosted in the JoinUp platform. | ||
| 18 | OSOR intends to exchange information, experiences, and best practices around the | 16 | OSOR intends to exchange information, experiences, and best practices around the |
| 19 | use of FLOSS in the public administration. It supports the discovering of FLOSS projects made | 17 | use of FLOSS in the public administration. It supports the discovering of FLOSS projects made |
| 20 | available by public agencies, providing information about these projects, such | 18 | available by public agencies, providing information about these projects, such |
| @@ -23,13 +21,6 @@ It also offers discussion forums and community mailing lists. But it | @@ -23,13 +21,6 @@ It also offers discussion forums and community mailing lists. But it | ||
| 23 | does not have an integrated source code repository manager, so for each | 21 | does not have an integrated source code repository manager, so for each |
| 24 | project there is a link to its own external repository. | 22 | project there is a link to its own external repository. |
| 25 | % | 23 | % |
| 26 | -OW2 is an organization that promotes the development, deployment and management | ||
| 27 | -of FLOSS middleware, generic | ||
| 28 | -business applications, and cloud computing platforms, besides fostering a community and | ||
| 29 | -business ecosystem around such projects. | ||
| 30 | - | ||
| 31 | -\leo{E aí, o que a OW2 tem a ver com o SPB?} | ||
| 32 | - | ||
| 33 | Moreover, in 2007 the European Comission published a study examining the impact | 24 | Moreover, in 2007 the European Comission published a study examining the impact |
| 34 | that the development and distribution of FLOSS by public agencies have on eGovernment | 25 | that the development and distribution of FLOSS by public agencies have on eGovernment |
| 35 | services and the economy~\cite{ghosh2004}. And, from 2007 to 2011, there | 26 | services and the economy~\cite{ghosh2004}. And, from 2007 to 2011, there |
| @@ -61,6 +52,3 @@ avoiding the usage of private platforms, especially the ones provided by foreign | @@ -61,6 +52,3 @@ avoiding the usage of private platforms, especially the ones provided by foreign | ||
| 61 | companies. Therefore, we needed to develop our own solution to cover all the | 52 | companies. Therefore, we needed to develop our own solution to cover all the |
| 62 | requirements, producing a complete governmental integrated platform for | 53 | requirements, producing a complete governmental integrated platform for |
| 63 | collaborative software development. | 54 | collaborative software development. |
| 64 | - | ||
| 65 | -\leo{Me parece que o decreto fala sobre sistemas de comunicação. Acho que o foco era e-mail. Não me parece que o SPB tenha a ver com isso. A menos que se considere algo pelos fóruns... para refletir.} | ||
| 66 | - |
opensym2017/content/06-architecture.tex
| @@ -104,7 +104,7 @@ stores the metrics evolution history, presents results in a friendly | @@ -104,7 +104,7 @@ stores the metrics evolution history, presents results in a friendly | ||
| 104 | way, as well as, allows users to customize the given interpretation | 104 | way, as well as, allows users to customize the given interpretation |
| 105 | accordingly to their own context. | 105 | accordingly to their own context. |
| 106 | 106 | ||
| 107 | -\subsection{System unification} | 107 | +\subsection{System unification and User eXperience evolution} |
| 108 | 108 | ||
| 109 | \begin{figure}[hbt] | 109 | \begin{figure}[hbt] |
| 110 | \centering | 110 | \centering |
| @@ -123,7 +123,33 @@ functionality of each application, a search o n the SPB portal might | @@ -123,7 +123,33 @@ functionality of each application, a search o n the SPB portal might | ||
| 123 | return content from any of the applications, be it web pages, mailing | 123 | return content from any of the applications, be it web pages, mailing |
| 124 | list posts, or source code. | 124 | list posts, or source code. |
| 125 | 125 | ||
| 126 | -% Falar do devops | 126 | +The integration of collaborative environments goes beyond functional aspects. |
| 127 | +Offering the population an unified experience across these environments has | ||
| 128 | +been the key to encourage the use of the platform as it reduces the perception | ||
| 129 | +of complexity. Thus, the SPB Portal information architecture was redesigned | ||
| 130 | +to provide a transparent navigation and to reach users with different profiles. | ||
| 131 | +A process of harmonization has been employed on the interaction models of each | ||
| 132 | +tool to reduce the learning curve. At the same time, a new visual style was | ||
| 133 | +created to unify the navigation experience and to comply with the guidelines of | ||
| 134 | +the digital communication identity standard established by the Federal | ||
| 135 | +Government. | ||
| 136 | + | ||
| 137 | +With the increase in system features and the addition of new tools, the | ||
| 138 | +visual style has steadily evolved to keep the navigation unified. Moreover, | ||
| 139 | +tools from different backgrounds, which in many cases provide similar | ||
| 140 | +functionality, prompted the development of an unified interface. Some | ||
| 141 | +features, such as search and user profile editing were eliminated from | ||
| 142 | +the individual applications, and implemented centrally to ensure a | ||
| 143 | +consistent look and feel. | ||
| 144 | + | ||
| 145 | +Another challenge was responsive web design. The integrated applications | ||
| 146 | +had varying degrees of support for responsiveness, and the common | ||
| 147 | +interface had to adapt for each individual scenario. In particular | ||
| 148 | +Noosfero did not yet have a responsive design; we engaged in its | ||
| 149 | +development and contributed towards that goal. | ||
| 150 | + | ||
| 151 | +\begin{comment} | ||
| 152 | +%FALTA DE ESPAÇO | ||
| 127 | \subsection{Deploy} | 153 | \subsection{Deploy} |
| 128 | 154 | ||
| 129 | The SPB platform was deployed in 7 virtual machines with different functions, | 155 | The SPB platform was deployed in 7 virtual machines with different functions, |
| @@ -161,3 +187,4 @@ static analysis tools on source code stored in repository and provide this data | @@ -161,3 +187,4 @@ static analysis tools on source code stored in repository and provide this data | ||
| 161 | to Prezento. A social network and CMS (Content Manager System) is provided by | 187 | to Prezento. A social network and CMS (Content Manager System) is provided by |
| 162 | Noosfero in \textit{social}, and the databases of all tools with a cache | 188 | Noosfero in \textit{social}, and the databases of all tools with a cache |
| 163 | service are in \textit{database}. | 189 | service are in \textit{database}. |
| 190 | +\end{comment} |
opensym2017/content/07-features.tex
| @@ -74,3 +74,4 @@ Thus, the SPB became a platform to stimulate the openness of the source code; | @@ -74,3 +74,4 @@ Thus, the SPB became a platform to stimulate the openness of the source code; | ||
| 74 | dialogue between users and the development team; and also | 74 | dialogue between users and the development team; and also |
| 75 | maintenance and evolution of the software, which will provide more | 75 | maintenance and evolution of the software, which will provide more |
| 76 | transparency in government investments. | 76 | transparency in government investments. |
| 77 | + |
| @@ -0,0 +1,181 @@ | @@ -0,0 +1,181 @@ | ||
| 1 | +\section{Development Organization and Process} | ||
| 2 | +\label{sec:process} | ||
| 3 | + | ||
| 4 | +The SPB team was composed of a variety of professionals with different levels | ||
| 5 | +and skills, where most of them were undergraduate students with major in | ||
| 6 | +software engineering (from 4th semester or upper). Since the students could | ||
| 7 | +not dedicate many hours per week to the project, they always had the | ||
| 8 | +flexibility to negotiate their work schedule during the semester in | ||
| 9 | +order to not harm their classes and coursework. Their daily work routine | ||
| 10 | +in the project included programming and DevOps tasks. | ||
| 11 | + | ||
| 12 | +The development of SPB project required a vast experience and background that | ||
| 13 | +usually undergraduate students do not have yet. For this reason, a few senior | ||
| 14 | +developers have joined to the project to help with the more difficult issues | ||
| 15 | +and to transfer knowledge to the students. Their main task was to provide | ||
| 16 | +solutions for complex problems, in other words, they worked as developers. As | ||
| 17 | +these professionals are very skillful and the project could not fund a full | ||
| 18 | +time work for them, some of them worked partially on the project. In addition, | ||
| 19 | +they lived in a different states spread around Brazil which led much of the | ||
| 20 | +communication to be online. | ||
| 21 | + | ||
| 22 | +In short, our work process was based on open and collaborative software | ||
| 23 | +development practices. The development process was defined based on the | ||
| 24 | +adaptation of different agile and FLOSS communities practices, highlighting the | ||
| 25 | +high degree of automation resulting from DevOps practices. Thus, the work | ||
| 26 | +process was executed in a cadenced and continuous way. | ||
| 27 | + | ||
| 28 | +Finally, the last group of actors of this project was composed of employees | ||
| 29 | +formally working for the Brazilian government, in the Ministry of Planning, | ||
| 30 | +Development, and Management (MP is the Brazilian acronym). All the project | ||
| 31 | +decisions, validations, and scope definitions were made by them. In this way we | ||
| 32 | +developed software product incrementally, with releases aligned to business | ||
| 33 | +strategic objectives. As one can see, the project had many different | ||
| 34 | +sort of stakeholders that had to be organized and synchronized. | ||
| 35 | + | ||
| 36 | +\subsection{Team organization} | ||
| 37 | + | ||
| 38 | +Approximately 70\% of the development teams were composed of software | ||
| 39 | +engineering undergraduate students from UnB and they worked physically | ||
| 40 | +in the same laboratory. Each student had their own schedule based on | ||
| 41 | +their classes, what complicates the implementation of pair programming. | ||
| 42 | +The senior developers tried to synchronize their schedule with the | ||
| 43 | +students on their sub-team. To cope with this environment, we had a few | ||
| 44 | +basic rules which guided the project organization: | ||
| 45 | + | ||
| 46 | +\begin{enumerate} | ||
| 47 | + \item Classes have high priority for undergraduate students; | ||
| 48 | + \item Work in pairs whenever possible (locally or remotely); | ||
| 49 | + \item There must be one morning or afternoon per week when | ||
| 50 | + \emph{everyone} should be together physically in the laboratory | ||
| 51 | + (except of course the remote team members); | ||
| 52 | + \item Every 3 to 4 months, the senior developers would fly in and work | ||
| 53 | + alongside the students for a few days. | ||
| 54 | +\end{enumerate} | ||
| 55 | + | ||
| 56 | +With the aforementioned rules we divided all the project into four | ||
| 57 | +different teams: Colab, Noosfero, Design, and DevOps. Each team had one | ||
| 58 | +coach responsible for reducing the communication problem with the other | ||
| 59 | +teams and help the members to organize themselves in the best way for | ||
| 60 | +everyone (always respecting their work time). The coach was one of the | ||
| 61 | +students working in some of the teams with the extra duty of registering | ||
| 62 | +the current tasks developed in the sprint and with the responsibility to | ||
| 63 | +talk with other teams. One important thing to notice is the mutability | ||
| 64 | +of the team and the coach, during the project many students changed | ||
| 65 | +between the teams to try different areas. | ||
| 66 | + | ||
| 67 | +One characteristic of the teams was the presence of (at least) one | ||
| 68 | +senior per team. This was essential, because hard decisions and complex | ||
| 69 | +problems were usually referred to them. This kept having to take | ||
| 70 | +complicated technical decisions from the coach tole, what encouraged | ||
| 71 | +students to be coaches. Lastly, the senior developers worked directly | ||
| 72 | +with the students, and this was important to give the undergraduate the | ||
| 73 | +opportunity to interact with a savvy professional in his area and to | ||
| 74 | +keep the knowledge flowing in the project. | ||
| 75 | + | ||
| 76 | +Finally, we had to add two last elements of the team organization, that | ||
| 77 | +was essential for the project harmony: the meta-coach and professors. | ||
| 78 | +The former was a software engineer recently graduated and which wanted | ||
| 79 | +to keep working on the project, the latter were professors that | ||
| 80 | +orchestrated all the interactions between all members of the project. | ||
| 81 | +The meta-coach usually worked in one specific team and had the extra | ||
| 82 | +task of knowing the current status of all teams. Professors and | ||
| 83 | +meta-coaches worked together to reduce the communication problem between | ||
| 84 | +all the teams. Lastly, all the paperwork tasks, such as reporting on the | ||
| 85 | +project progress to the Ministry, was handled by the professors. | ||
| 86 | + | ||
| 87 | +\subsection{Communication and management} | ||
| 88 | + | ||
| 89 | +Our team had many people working together, and most of the seniors worked in a | ||
| 90 | +different city remotely. Also, we tried to keep our work completely clear to | ||
| 91 | +the Brazilian government and citizens interested in following the project. To | ||
| 92 | +handle those cases, we used a set of tools to communication and other to manage | ||
| 93 | +the project. | ||
| 94 | + | ||
| 95 | +For communication between members in different places, we used: video | ||
| 96 | +conferencing with shared terminal tools, IRC, and mailing lists. For example, | ||
| 97 | +when one student had to work in pair with a senior, normally, they used video | ||
| 98 | +conferencing tool for talking and shared a terminal session (both typing and | ||
| 99 | +seeing the screen in real time). For questions and fast discussion, we used | ||
| 100 | +IRC. For general notification, we used the mailing lists. | ||
| 101 | + | ||
| 102 | +For managing the project we used the SPB Portal itself; first to validate it by | ||
| 103 | +ourselves, and also because it had all the required tools. We basically created | ||
| 104 | +one wiki page per release in the SPB Gitlab instance with a mapping between | ||
| 105 | +strategical, tactical, and operational views. In a practical point of view, one | ||
| 106 | +milestone per user history (feature), and one or more issues for addressing | ||
| 107 | +each feature. With this approach we achieved two important goals: keeping all | ||
| 108 | +the management as close possible to to the source code, and tracking every | ||
| 109 | +feature developed during the project. Our decision to | ||
| 110 | +use Wiki initially was empirical, but later reinforced by a research conducted | ||
| 111 | +by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, | ||
| 112 | +opensourcestyle}. | ||
| 113 | + | ||
| 114 | +\subsection{High-level project management and reporting} | ||
| 115 | + | ||
| 116 | +The Brazilian government used to work with software development in a | ||
| 117 | +very traditional way. They would frequently focus on documents and not | ||
| 118 | +on what was, in our opinion, what really matters: working software. This | ||
| 119 | +dissonance caused us a communication noise with MP, because they would | ||
| 120 | +often question our work style. It was especially hard to convince them | ||
| 121 | +to accept the idea of open scope and agile development, but after months | ||
| 122 | +of labor and showing results they stopped resisting. | ||
| 123 | + | ||
| 124 | +We defined some level of meeting granularity to avoid generating too | ||
| 125 | +much overhead to the developers. We had a strategical and validating | ||
| 126 | +meeting with MP (the former once in a month and the latter each 15th | ||
| 127 | +day), release plaining with the entire team (one per month), and finally | ||
| 128 | +a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a | ||
| 129 | +diagram that represents our meeting organization. | ||
| 130 | + | ||
| 131 | +\begin{figure}[hbt] | ||
| 132 | + \centering | ||
| 133 | + \includegraphics[width=\linewidth]{figures/meeting_flows.png} | ||
| 134 | + \caption{Meetings cycles.} | ||
| 135 | + \label{fig:meeting} | ||
| 136 | +\end{figure} | ||
| 137 | + | ||
| 138 | +In the strategical meeting we usually defined the priorities and new | ||
| 139 | +features with the Brazilian government. Normally the professors, the | ||
| 140 | +coach of each team, the meta-coach, and some employees of the MP would | ||
| 141 | +participate in this meeting. We usually discussed what the team already | ||
| 142 | +produced since our last meeting, and established the new features for | ||
| 143 | +the next release. Notice that just part of the team would join this | ||
| 144 | +meeting to avoid generating unnecessary overhead to the developers, but | ||
| 145 | +all the students interested to participate were allowed to join (many | ||
| 146 | +students wanted this experience during the project). | ||
| 147 | + | ||
| 148 | +After the strategical meeting with Brazilian government agents, we had a | ||
| 149 | +planning turn with all teams together. In this part, each team worked together | ||
| 150 | +to convert the MP wishes into smaller parts which were represented by the epics of | ||
| 151 | +the release. Each coach was responsible for conducting the planning, and after | ||
| 152 | +that record it on the project wiki (the wiki provided by Gitlab). With this | ||
| 153 | +epic, each 14th day the team have generated their sprint scheduler (with small | ||
| 154 | +achievements mapped to issues). | ||
| 155 | + | ||
| 156 | +To keep the Brazilian government always updated, we invited them to work | ||
| 157 | +with us to validate the new features in progress. Normally we had a | ||
| 158 | +meeting each 15th day. Basically, this was our work flow, we always kept | ||
| 159 | +everything extremely open to the MP (our way of working, and the one | ||
| 160 | +often used by open source projects) and to the team. | ||
| 161 | + | ||
| 162 | +To keep the track of all of those things we used the SPB itself, | ||
| 163 | +especially Gitlab. Basically, we had: | ||
| 164 | + | ||
| 165 | +\begin{enumerate} | ||
| 166 | + \item Project repository: We have one organization with many repositories; | ||
| 167 | + \item Milestones: Each milestone was used to register a release; | ||
| 168 | + \item Wiki: Each release has one page on wiki with the compilation of | ||
| 169 | + strategical meeting; | ||
| 170 | + \item Issues: Each sprint planning generated issues, which we associated to | ||
| 171 | + the specific milestone and updated the wiki with the issue number related | ||
| 172 | + with them. Finally each developer assigned the issue to itself. | ||
| 173 | +\end{enumerate} | ||
| 174 | + | ||
| 175 | +Notice that this workflow gave us and the Brazilian government agents full | ||
| 176 | +traceability from a high level view of each feature to the lowest level (code). | ||
| 177 | +It is important to highlight that we converge to this workflow after many | ||
| 178 | +experiments. For example, we used a tool named Redmine for organizing our tasks | ||
| 179 | +during the sprints, however, this tool revealed inefficient for our case since | ||
| 180 | +the government agents lost part of the project traceability. We realized that | ||
| 181 | +centralize all the work in the SPB portal is the best option for our case. |
opensym2017/content/08-ux.tex
| @@ -1,34 +0,0 @@ | @@ -1,34 +0,0 @@ | ||
| 1 | -\section{User eXperience evolution} | ||
| 2 | -\label{sec:ux} | ||
| 3 | - | ||
| 4 | -The integration of collaborative environments goes beyond functional aspects. | ||
| 5 | -Offering the population an unified experience across these environments has | ||
| 6 | -been the key to encourage the use of the platform as it reduces the perception | ||
| 7 | -of complexity. Thus, the SPB Portal information architecture was redesigned | ||
| 8 | -to provide a transparent navigation and to reach users with different profiles. | ||
| 9 | -A process of harmonization has been employed on the interaction models of each | ||
| 10 | -tool to reduce the learning curve. At the same time, a new visual style was | ||
| 11 | -created to unify the navigation experience and to comply with the guidelines of | ||
| 12 | -the digital communication identity standard established by the Federal | ||
| 13 | -Government. | ||
| 14 | - | ||
| 15 | -With the increase in system features and the addition of new tools, the | ||
| 16 | -visual style has steadily evolved to keep the navigation unified. Moreover, | ||
| 17 | -tools from different backgrounds, which in many cases provide similar | ||
| 18 | -functionality, prompted the development of an unified interface. Some | ||
| 19 | -features, such as search and user profile editing were eliminated from | ||
| 20 | -the individual applications, and implemented centrally to ensure a | ||
| 21 | -consistent look and feel. | ||
| 22 | - | ||
| 23 | -Another challenge was responsive web design. The integrated applications | ||
| 24 | -had varying degrees of support for responsiveness, and the common | ||
| 25 | -interface had to adapt for each individual scenario. In particular | ||
| 26 | -Noosfero did not yet have a responsive design; we engaged in its | ||
| 27 | -development and contributed towards that goal. | ||
| 28 | - | ||
| 29 | -After the initial release of the new SPB Portal in 2014, several | ||
| 30 | -validations activities were implemented in 2015 and 2016. The aim was to | ||
| 31 | -provide the most wanted features by casual users (such as public | ||
| 32 | -servants interested in downloads and documentation) immediately, while | ||
| 33 | -allowing more experienced users (such as developers) to easily drill down | ||
| 34 | -to the details. |
| @@ -0,0 +1,220 @@ | @@ -0,0 +1,220 @@ | ||
| 1 | +\section{Conclusion} | ||
| 2 | +\label{sec:conclusion} | ||
| 3 | + | ||
| 4 | +In this work, we presented and discussed issues experienced during a government-funded | ||
| 5 | +project, in partnership with the University of Brasilia and the University of | ||
| 6 | +São Paulo, to evolve the Brazilian Public Software Portal. | ||
| 7 | +Its contributions are twofold. First, we present the strategy used to develop | ||
| 8 | +and to deliver an unprecedented platform to Brazilian government. Second, | ||
| 9 | +based on the results of the SPB Portal project, we point out that it is | ||
| 10 | +possible to mitigate conflicts experienced in the development environment | ||
| 11 | +and to conciliate governmental and academy cultures. To summarize our main contributions, we answered in this section the three open questions those guided this paper. | ||
| 12 | + | ||
| 13 | + | ||
| 14 | +\textbf{Q1 -- Which strategy could be used to integrate several existing | ||
| 15 | +FLOSS tools to promote the collaborative software development?} | ||
| 16 | +% | ||
| 17 | +The SPB portal integrates more than 10 FLOSS tools and provides several features, | ||
| 18 | +such as social network, mailing list, version control, content management and | ||
| 19 | +source code quality monitoring. Concerned with the platform sustainability and | ||
| 20 | +maintainability, the aforementioned 10 FLOSS tools were integrated with minimum | ||
| 21 | +differences from their official versions and the new developed features were | ||
| 22 | +sent upstream to ensure an alignment between the portal systems and their | ||
| 23 | +respective official versions. In the integration process, the main softwares | ||
| 24 | +were identified, specific teams were formed to work with each one of them | ||
| 25 | +and each team was composed of students with different levels of skills and at | ||
| 26 | +least one senior professional. | ||
| 27 | + | ||
| 28 | +\textbf{Q2 -- How to involve students in real-world projects interacting with | ||
| 29 | +real customers?} | ||
| 30 | +% | ||
| 31 | +In terms of mitigating conflicts, we tried to show that, as long as the | ||
| 32 | +university can provide a healthy and challenging environment to its students, | ||
| 33 | +one may conciliate studies and professional training in universities. | ||
| 34 | +% | ||
| 35 | +The students interacted with professionals of diverse fields of expertise, and they were | ||
| 36 | +able to participate in all levels of the software development process. | ||
| 37 | +This contributed to a great learning opportunity. | ||
| 38 | +% | ||
| 39 | +In our work process, based on open and collaborative software development | ||
| 40 | +practices, students could negotiate their work schedule as well as count on IT | ||
| 41 | +professionals to solve development issues. | ||
| 42 | +% | ||
| 43 | +Among the students, we have defined coaches for each team and a meta-coach | ||
| 44 | +(coach of whole project). All coaches, together with professors, have | ||
| 45 | +intermediated the communication between client (Ministry of Planning of Brazil) | ||
| 46 | +and the rest of the group. | ||
| 47 | +After the end of the project, some students successfully | ||
| 48 | +embraced opportunities in public and private sectors, within national borders | ||
| 49 | +and abroad. Some other students went further and started their own companies. | ||
| 50 | + | ||
| 51 | +\textbf{Q3 -- How to introduce the FLOSS collaborative and agile | ||
| 52 | +practices to governmental development process?} | ||
| 53 | +With some adaptations, what we called the ``translation processes'', it is feasible | ||
| 54 | +to conciliate agile methodologies and FLOSS practices to develop software to | ||
| 55 | +governmental organizations with functional hierarchical structures that use | ||
| 56 | +traditional development paradigm. | ||
| 57 | +% | ||
| 58 | +Aiming at reducing client questions about workconclusion, a DevOps front was | ||
| 59 | +created to automate all deploy process and also to work in continuous | ||
| 60 | +delivery. The government was brought to our work environment and interacted | ||
| 61 | +with our management and communication tools. For the project success, we | ||
| 62 | +focused on providing a friendly working environment as well as on showing to | ||
| 63 | +governmental agents another way to interact with the FLOSS community and the | ||
| 64 | +university. | ||
| 65 | + | ||
| 66 | +\subsection{Lessons Learned} | ||
| 67 | +\label{sec:lessons} | ||
| 68 | + | ||
| 69 | +From the answers of our initial open questions, we can also highlighted six lessons learned to make easier to share our experience during the development of the new SPB Portal. | ||
| 70 | + | ||
| 71 | +\textbf{L1 -- The participation of experienced professionals is crucial to | ||
| 72 | +the success of the project.} | ||
| 73 | +One important factor for the students was the composition of the teams | ||
| 74 | +with the participation of experienced professionals. | ||
| 75 | +% | ||
| 76 | +On the technical side, the senior developers and designers would handle | ||
| 77 | +the more difficult technical decisions, creating a work environment | ||
| 78 | +where the students could develop their skills in a didactic way without | ||
| 79 | +pressure. | ||
| 80 | +% | ||
| 81 | +On the management side, the active participation of professors -- who | ||
| 82 | +are, in the end, the ones responsible for the project -- is crucial to | ||
| 83 | +make sure students participation is conducted in a healthy way, and it is an | ||
| 84 | +instance of leading by example. | ||
| 85 | + | ||
| 86 | +\textbf{L2 -- A balanced relationship between academia and industry.} | ||
| 87 | +The experience of the SPB project led UnB to develop a work style which | ||
| 88 | +proved to be appropriate for an educational environment that brings | ||
| 89 | +academia and industry together. | ||
| 90 | +% | ||
| 91 | +The highest priority from the university's point of view is the | ||
| 92 | +students. Considering this, the activities of the project were never | ||
| 93 | +prioritized to the detriment of classes and other pedagogical | ||
| 94 | +activities. In summary, we had students working at different times, part | ||
| 95 | +time, remotely or locally, always respecting their individual | ||
| 96 | +conditions, but doing the work in a collective, collaborative and open | ||
| 97 | +way. | ||
| 98 | +% | ||
| 99 | +And even under a potentially adverse environment, the project delivered | ||
| 100 | +the desired solution with success. | ||
| 101 | +% | ||
| 102 | +At the end of the project, we noted that the skills developed by the | ||
| 103 | +students were at the software engineering state of the art. | ||
| 104 | +After the project ended, we had team members successfully embracing | ||
| 105 | +opportunities in public, private, national, and international | ||
| 106 | +organizations, in addition to those students that | ||
| 107 | +opened their own companies. | ||
| 108 | + | ||
| 109 | +\textbf{L3 -- Managing different organizational cultures.} | ||
| 110 | +In the beginning of the project, the Brazilian Government stakeholders | ||
| 111 | +had certain expectations about the project development that, let's | ||
| 112 | +say, didn't exactly match our work style based on agile and FLOSS practices. | ||
| 113 | +% | ||
| 114 | +We had to develop strategies that would support these different | ||
| 115 | +organizational cultures. The | ||
| 116 | +MP is organized in a functional hierarchical organizational structure, | ||
| 117 | +typically adopting a traditional development paradigm. Therefore, we had to | ||
| 118 | +create a translation process between our team and the MP managers who | ||
| 119 | +managed the project on their side assuming a traditional waterfall | ||
| 120 | +process. | ||
| 121 | + | ||
| 122 | +\textbf{L4 -- Managing higher-level and lower-level goals separately.} | ||
| 123 | +During the initial phase of the project, the MP team has brought | ||
| 124 | +strategic discussions to technical/operational meetings that | ||
| 125 | +were supposed to be about practical technical decisions. | ||
| 126 | +% | ||
| 127 | +This produced a highly complex communication and management environment, | ||
| 128 | +overloading the professors because they were supposed | ||
| 129 | +for maintaining the MP strategy synchronized with the | ||
| 130 | +implementation plans of the development team. This was hard, especially because the | ||
| 131 | +aforementioned cultural mismatch. Mixing both concerns in the same | ||
| 132 | +discussions caused confusion on both sides. | ||
| 133 | +% | ||
| 134 | +From the middle of the project we were able to keep those | ||
| 135 | +concerns separated, what eased the work of everyone involved. | ||
| 136 | + | ||
| 137 | +\textbf{L5 -- Living with ill-advised political decisions.} | ||
| 138 | +At the initial phases of the project, by political and personal | ||
| 139 | +motivation, the main stakeholders from the Brazilian government imposed | ||
| 140 | +the use of Colab to guide the development of the new SPB platform. Our | ||
| 141 | +team was totally against the idea because we already knew that Colab was | ||
| 142 | +a very experimental project and its adoption could dramatically increase | ||
| 143 | +the project complexity. Even though we provided technical reasons to | ||
| 144 | +not utilize Colab, the MP was adamant and we had to manage this problem. We did massive changes to | ||
| 145 | +Colab, and at the end of the project we have completely rewritten it to make | ||
| 146 | +it stable. It is important to notice that the MP compelled us to accept | ||
| 147 | +a technical decision based only on political interests, without considering | ||
| 148 | +all the resources that would be spent due to that decision. At the end | ||
| 149 | +of the project, we verified that Colab indeed consumed a vast amount of | ||
| 150 | +the budget and increased the project complexity. At the end of the | ||
| 151 | +project and after our analysis on the decision made by the MP, we | ||
| 152 | +understand that MP managers are capable of ignoring technical reasons | ||
| 153 | +in favor of political decisions. | ||
| 154 | + | ||
| 155 | +\textbf{L6 -- Consider sustainability from the beginning.} | ||
| 156 | +In the process of deploying the SPB platform in the MP infrastructure we had | ||
| 157 | +to interact with the MP technicians. We did several workshops, training | ||
| 158 | +and a meticulous documentation describing all the required procedures to | ||
| 159 | +update the platform, however, we realized that the MP technicians | ||
| 160 | +constantly avoid the responsibility. After noticing the aforementioned | ||
| 161 | +situation, we organized a DevOps team that completely automated all | ||
| 162 | +the deployment procedure. We simplified all the platform deployment to a | ||
| 163 | +few steps: (1) initial configurations (just ssh configuration) and (2) | ||
| 164 | +the execution of simple commands to completely update the platform. By | ||
| 165 | +the end of the project, we observed that the MP technicians invariably | ||
| 166 | +still depended on our support to update the platform even with all the | ||
| 167 | +automation provided by us. We were sadly left with a feeling of | ||
| 168 | +uncertainty about the future of the platform after the project ended. In | ||
| 169 | +hindsight, we realize that the MP dedicated system analysts and | ||
| 170 | +managers to the project, but not operations technicians. The later | ||
| 171 | +should have been more involved with the process so they could at least be | ||
| 172 | +comfortable in managing the platform infrastructure. | ||
| 173 | + | ||
| 174 | + | ||
| 175 | +\textbf{Final remarks and future work} | ||
| 176 | + | ||
| 177 | +The portal is available at \url{softwarepublico.gov.br}. All | ||
| 178 | +documentation, including detailed architecture and operation manuals are | ||
| 179 | +also available\footnote{\url{https://softwarepublico.gov.br/doc/} | ||
| 180 | +(in Portuguese only at the moment)}. | ||
| 181 | +% | ||
| 182 | +All the integrated tools are FLOSS and our contributions were published | ||
| 183 | +in open repositories, available on the SPB Portal itself. We also | ||
| 184 | +contributed these features back to the respective communities, which | ||
| 185 | +benefits both those communities and us, since we can share future | ||
| 186 | +development and maintenance effort with other organizations that | ||
| 187 | +participate in these projects. | ||
| 188 | + | ||
| 189 | +Future work should use data produced by the project to validate and evaluate | ||
| 190 | +how the used FLOSS and Agile practices have impacted the students and the | ||
| 191 | +governmental development process. For this, we would conduce a \textit{postmortem} | ||
| 192 | +analysis using the project open data and a survey targeting the involved actors. | ||
| 193 | + | ||
| 194 | +%=========== | ||
| 195 | +% Conclusion | ||
| 196 | +%=========== | ||
| 197 | + | ||
| 198 | +% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados | ||
| 199 | +%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa | ||
| 200 | +%% afirmação, embora eu e Paulo consigamos perceber isso. | ||
| 201 | + | ||
| 202 | + | ||
| 203 | +%* utilização do projeto para formação de recursos humanos (alunos) | ||
| 204 | + | ||
| 205 | +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate | ||
| 206 | + | ||
| 207 | +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) | ||
| 208 | + | ||
| 209 | +%* 69 projetos marcados como SPB, de 81 no total na plataforma. | ||
| 210 | + | ||
| 211 | +%* 47\% é desenvolvido em PHP. | ||
| 212 | + | ||
| 213 | +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). | ||
| 214 | + | ||
| 215 | +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. | ||
| 216 | + | ||
| 217 | +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. | ||
| 218 | + | ||
| 219 | +%* Debater economia de recursos em orgão públicos | ||
| 220 | + |
opensym2017/content/09-process.tex
| @@ -1,181 +0,0 @@ | @@ -1,181 +0,0 @@ | ||
| 1 | -\section{Development Organization and Process} | ||
| 2 | -\label{sec:process} | ||
| 3 | - | ||
| 4 | -The SPB team was composed of a variety of professionals with different levels | ||
| 5 | -and skills, where most of them were undergraduate students with major in | ||
| 6 | -software engineering (from 4th semester or upper). Since the students could | ||
| 7 | -not dedicate many hours per week to the project, they always had the | ||
| 8 | -flexibility to negotiate their work schedule during the semester in | ||
| 9 | -order to not harm their classes and coursework. Their daily work routine | ||
| 10 | -in the project included programming and DevOps tasks. | ||
| 11 | - | ||
| 12 | -The development of SPB project required a vast experience and background that | ||
| 13 | -usually undergraduate students do not have yet. For this reason, a few senior | ||
| 14 | -developers have joined to the project to help with the more difficult issues | ||
| 15 | -and to transfer knowledge to the students. Their main task was to provide | ||
| 16 | -solutions for complex problems, in other words, they worked as developers. As | ||
| 17 | -these professionals are very skillful and the project could not fund a full | ||
| 18 | -time work for them, some of them worked partially on the project. In addition, | ||
| 19 | -they lived in a different states spread around Brazil which led much of the | ||
| 20 | -communication to be online. | ||
| 21 | - | ||
| 22 | -In short, our work process was based on open and collaborative software | ||
| 23 | -development practices. The development process was defined based on the | ||
| 24 | -adaptation of different agile and FLOSS communities practices, highlighting the | ||
| 25 | -high degree of automation resulting from DevOps practices. Thus, the work | ||
| 26 | -process was executed in a cadenced and continuous way. | ||
| 27 | - | ||
| 28 | -Finally, the last group of actors of this project was composed of employees | ||
| 29 | -formally working for the Brazilian government, in the Ministry of Planning, | ||
| 30 | -Development, and Management (MP is the Brazilian acronym). All the project | ||
| 31 | -decisions, validations, and scope definitions were made by them. In this way we | ||
| 32 | -developed software product incrementally, with releases aligned to business | ||
| 33 | -strategic objectives. As one can see, the project had many different | ||
| 34 | -sort of stakeholders that had to be organized and synchronized. | ||
| 35 | - | ||
| 36 | -\subsection{Team organization} | ||
| 37 | - | ||
| 38 | -Approximately 70\% of the development teams were composed of software | ||
| 39 | -engineering undergraduate students from UnB and they worked physically | ||
| 40 | -in the same laboratory. Each student had their own schedule based on | ||
| 41 | -their classes, what complicates the implementation of pair programming. | ||
| 42 | -The senior developers tried to synchronize their schedule with the | ||
| 43 | -students on their sub-team. To cope with this environment, we had a few | ||
| 44 | -basic rules which guided the project organization: | ||
| 45 | - | ||
| 46 | -\begin{enumerate} | ||
| 47 | - \item Classes have high priority for undergraduate students; | ||
| 48 | - \item Work in pairs whenever possible (locally or remotely); | ||
| 49 | - \item There must be one morning or afternoon per week when | ||
| 50 | - \emph{everyone} should be together physically in the laboratory | ||
| 51 | - (except of course the remote team members); | ||
| 52 | - \item Every 3 to 4 months, the senior developers would fly in and work | ||
| 53 | - alongside the students for a few days. | ||
| 54 | -\end{enumerate} | ||
| 55 | - | ||
| 56 | -With the aforementioned rules we divided all the project into four | ||
| 57 | -different teams: Colab, Noosfero, Design, and DevOps. Each team had one | ||
| 58 | -coach responsible for reducing the communication problem with the other | ||
| 59 | -teams and help the members to organize themselves in the best way for | ||
| 60 | -everyone (always respecting their work time). The coach was one of the | ||
| 61 | -students working in some of the teams with the extra duty of registering | ||
| 62 | -the current tasks developed in the sprint and with the responsibility to | ||
| 63 | -talk with other teams. One important thing to notice is the mutability | ||
| 64 | -of the team and the coach, during the project many students changed | ||
| 65 | -between the teams to try different areas. | ||
| 66 | - | ||
| 67 | -One characteristic of the teams was the presence of (at least) one | ||
| 68 | -senior per team. This was essential, because hard decisions and complex | ||
| 69 | -problems were usually referred to them. This kept having to take | ||
| 70 | -complicated technical decisions from the coach tole, what encouraged | ||
| 71 | -students to be coaches. Lastly, the senior developers worked directly | ||
| 72 | -with the students, and this was important to give the undergraduate the | ||
| 73 | -opportunity to interact with a savvy professional in his area and to | ||
| 74 | -keep the knowledge flowing in the project. | ||
| 75 | - | ||
| 76 | -Finally, we had to add two last elements of the team organization, that | ||
| 77 | -was essential for the project harmony: the meta-coach and professors. | ||
| 78 | -The former was a software engineer recently graduated and which wanted | ||
| 79 | -to keep working on the project, the latter were professors that | ||
| 80 | -orchestrated all the interactions between all members of the project. | ||
| 81 | -The meta-coach usually worked in one specific team and had the extra | ||
| 82 | -task of knowing the current status of all teams. Professors and | ||
| 83 | -meta-coaches worked together to reduce the communication problem between | ||
| 84 | -all the teams. Lastly, all the paperwork tasks, such as reporting on the | ||
| 85 | -project progress to the Ministry, was handled by the professors. | ||
| 86 | - | ||
| 87 | -\subsection{Communication and management} | ||
| 88 | - | ||
| 89 | -Our team had many people working together, and most of the seniors worked in a | ||
| 90 | -different city remotely. Also, we tried to keep our work completely clear to | ||
| 91 | -the Brazilian government and citizens interested in following the project. To | ||
| 92 | -handle those cases, we used a set of tools to communication and other to manage | ||
| 93 | -the project. | ||
| 94 | - | ||
| 95 | -For communication between members in different places, we used: video | ||
| 96 | -conferencing with shared terminal tools, IRC, and mailing lists. For example, | ||
| 97 | -when one student had to work in pair with a senior, normally, they used video | ||
| 98 | -conferencing tool for talking and shared a terminal session (both typing and | ||
| 99 | -seeing the screen in real time). For questions and fast discussion, we used | ||
| 100 | -IRC. For general notification, we used the mailing lists. | ||
| 101 | - | ||
| 102 | -For managing the project we used the SPB Portal itself; first to validate it by | ||
| 103 | -ourselves, and also because it had all the required tools. We basically created | ||
| 104 | -one wiki page per release in the SPB Gitlab instance with a mapping between | ||
| 105 | -strategical, tactical, and operational views. In a practical point of view, one | ||
| 106 | -milestone per user history (feature), and one or more issues for addressing | ||
| 107 | -each feature. With this approach we achieved two important goals: keeping all | ||
| 108 | -the management as close possible to to the source code, and tracking every | ||
| 109 | -feature developed during the project. Our decision to | ||
| 110 | -use Wiki initially was empirical, but later reinforced by a research conducted | ||
| 111 | -by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, | ||
| 112 | -opensourcestyle}. | ||
| 113 | - | ||
| 114 | -\subsection{High-level project management and reporting} | ||
| 115 | - | ||
| 116 | -The Brazilian government used to work with software development in a | ||
| 117 | -very traditional way. They would frequently focus on documents and not | ||
| 118 | -on what was, in our opinion, what really matters: working software. This | ||
| 119 | -dissonance caused us a communication noise with MP, because they would | ||
| 120 | -often question our work style. It was especially hard to convince them | ||
| 121 | -to accept the idea of open scope and agile development, but after months | ||
| 122 | -of labor and showing results they stopped resisting. | ||
| 123 | - | ||
| 124 | -We defined some level of meeting granularity to avoid generating too | ||
| 125 | -much overhead to the developers. We had a strategical and validating | ||
| 126 | -meeting with MP (the former once in a month and the latter each 15th | ||
| 127 | -day), release plaining with the entire team (one per month), and finally | ||
| 128 | -a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a | ||
| 129 | -diagram that represents our meeting organization. | ||
| 130 | - | ||
| 131 | -\begin{figure}[hbt] | ||
| 132 | - \centering | ||
| 133 | - \includegraphics[width=\linewidth]{figures/meeting_flows.png} | ||
| 134 | - \caption{Meetings cycles.} | ||
| 135 | - \label{fig:meeting} | ||
| 136 | -\end{figure} | ||
| 137 | - | ||
| 138 | -In the strategical meeting we usually defined the priorities and new | ||
| 139 | -features with the Brazilian government. Normally the professors, the | ||
| 140 | -coach of each team, the meta-coach, and some employees of the MP would | ||
| 141 | -participate in this meeting. We usually discussed what the team already | ||
| 142 | -produced since our last meeting, and established the new features for | ||
| 143 | -the next release. Notice that just part of the team would join this | ||
| 144 | -meeting to avoid generating unnecessary overhead to the developers, but | ||
| 145 | -all the students interested to participate were allowed to join (many | ||
| 146 | -students wanted this experience during the project). | ||
| 147 | - | ||
| 148 | -After the strategical meeting with Brazilian government agents, we had a | ||
| 149 | -planning turn with all teams together. In this part, each team worked together | ||
| 150 | -to convert the MP wishes into smaller parts which were represented by the epics of | ||
| 151 | -the release. Each coach was responsible for conducting the planning, and after | ||
| 152 | -that record it on the project wiki (the wiki provided by Gitlab). With this | ||
| 153 | -epic, each 14th day the team have generated their sprint scheduler (with small | ||
| 154 | -achievements mapped to issues). | ||
| 155 | - | ||
| 156 | -To keep the Brazilian government always updated, we invited them to work | ||
| 157 | -with us to validate the new features in progress. Normally we had a | ||
| 158 | -meeting each 15th day. Basically, this was our work flow, we always kept | ||
| 159 | -everything extremely open to the MP (our way of working, and the one | ||
| 160 | -often used by open source projects) and to the team. | ||
| 161 | - | ||
| 162 | -To keep the track of all of those things we used the SPB itself, | ||
| 163 | -especially Gitlab. Basically, we had: | ||
| 164 | - | ||
| 165 | -\begin{enumerate} | ||
| 166 | - \item Project repository: We have one organization with many repositories; | ||
| 167 | - \item Milestones: Each milestone was used to register a release; | ||
| 168 | - \item Wiki: Each release has one page on wiki with the compilation of | ||
| 169 | - strategical meeting; | ||
| 170 | - \item Issues: Each sprint planning generated issues, which we associated to | ||
| 171 | - the specific milestone and updated the wiki with the issue number related | ||
| 172 | - with them. Finally each developer assigned the issue to itself. | ||
| 173 | -\end{enumerate} | ||
| 174 | - | ||
| 175 | -Notice that this workflow gave us and the Brazilian government agents full | ||
| 176 | -traceability from a high level view of each feature to the lowest level (code). | ||
| 177 | -It is important to highlight that we converge to this workflow after many | ||
| 178 | -experiments. For example, we used a tool named Redmine for organizing our tasks | ||
| 179 | -during the sprints, however, this tool revealed inefficient for our case since | ||
| 180 | -the government agents lost part of the project traceability. We realized that | ||
| 181 | -centralize all the work in the SPB portal is the best option for our case. |
opensym2017/content/10-contributions.tex
| @@ -1,58 +0,0 @@ | @@ -1,58 +0,0 @@ | ||
| 1 | -\section{Contributions to the upstream communities} | ||
| 2 | -\label{sec:contributions} | ||
| 3 | - | ||
| 4 | -%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) | ||
| 5 | -%* Colab -> RevProxy | ||
| 6 | -%* Colab, atualização do python/django | ||
| 7 | -%* Contribuições para o GitLab (autenticação) | ||
| 8 | -%* Noosfero, atualização do Rails, preparação para federação, nova interface ... | ||
| 9 | -%* Coper, empacotamentos (obs), omniauth | ||
| 10 | - | ||
| 11 | - | ||
| 12 | -During the execution of this project we made several contributions to | ||
| 13 | -the upstream communities we interacted with. This occurred due to our | ||
| 14 | -development process aligned with those of the respective communities. | ||
| 15 | -During development, we would explicitly discuss the features and bug | ||
| 16 | -fixes that we were working on with the applicable upstream communities. | ||
| 17 | -This contributed to improve the developers technical solutions with | ||
| 18 | -expertise outside of our team, and make it easier for those changes to | ||
| 19 | -be accepted in the original projects. Having changes accepted upstream | ||
| 20 | -in turn makes our life easier as it minimizes the delta between our | ||
| 21 | -codebase and upstream's allowing us to upgrade and benefit from | ||
| 22 | -development work from others. | ||
| 23 | - | ||
| 24 | -In Colab, we helped upstream redesign the entirely architecture, | ||
| 25 | -enabling the development of plugins to integrate new tools. We also | ||
| 26 | -added a feature that allowed Colab to run asynchronous tasks, which was | ||
| 27 | -a major improvement for us since we were developing a complex system. A | ||
| 28 | -migration to the latest Django version was made (web framework used by | ||
| 29 | -Colab). Moreover, we worked on RevProxy (the more important dependency | ||
| 30 | -of Colab) to put it in a good shape, fixing many bugs. | ||
| 31 | - | ||
| 32 | -Gitlab was the tool that we made the least number of modifications. We | ||
| 33 | -contributed with some improvements related with configuration files and | ||
| 34 | -we developed a new plugin that enables user authentication in Gitlab | ||
| 35 | -through the REMOTE\_USER HTTP header. This plugin was needed because | ||
| 36 | -Colab uses this mechanism to manage the authentication. | ||
| 37 | - | ||
| 38 | -Noosfero was the tool that contemplated several functional requirements, | ||
| 39 | -therefore we made a large number of contributions with upstream. We helped to | ||
| 40 | -migrate to the latest Rails version (web framework used by Noosfero), enable | ||
| 41 | -the federation implementation (federation with other social networks), and | ||
| 42 | -decouple the interface and the back-end. | ||
| 43 | - | ||
| 44 | -Mezuro was completely rewritten and its architecture evolved adopting the | ||
| 45 | -microservice architecture\cite{namiot2014micro}. This way, we minimize the | ||
| 46 | -amount of code to maintain it, helping to test it and grant its code quality. | ||
| 47 | -In practice, we modularize the Mezuro project in several independent services. | ||
| 48 | -Currently, its computation and visualization modules use Kalibro and Prezento, | ||
| 49 | -respectively. They were developed into the Mezuro project and evolved during | ||
| 50 | -its integration to the new SPB Portal. | ||
| 51 | - | ||
| 52 | - | ||
| 53 | -We also made some contributions on the DevOps front. Some members of | ||
| 54 | -them team became maintainers of some Python libraries that were used by | ||
| 55 | -our scripts to upload packages to OBS (Open Build Service). We developed | ||
| 56 | -a tool called copr-status to keep track of the different stages of the | ||
| 57 | -lifecycle of each of the individual packages we were working on. | ||
| 58 | - |
opensym2017/content/11-conclusion.tex
| @@ -1,220 +0,0 @@ | @@ -1,220 +0,0 @@ | ||
| 1 | -\section{Conclusion} | ||
| 2 | -\label{sec:conclusion} | ||
| 3 | - | ||
| 4 | -In this work, we presented and discussed issues experienced during a government-funded | ||
| 5 | -project, in partnership with the University of Brasilia and the University of | ||
| 6 | -São Paulo, to evolve the Brazilian Public Software Portal. | ||
| 7 | -Its contributions are twofold. First, we present the strategy used to develop | ||
| 8 | -and to deliver an unprecedented platform to Brazilian government. Second, | ||
| 9 | -based on the results of the SPB Portal project, we point out that it is | ||
| 10 | -possible to mitigate conflicts experienced in the development environment | ||
| 11 | -and to conciliate governmental and academy cultures. To summarize our main contributions, we answered in this section the three open questions those guided this paper. | ||
| 12 | - | ||
| 13 | - | ||
| 14 | -\textbf{Q1 -- Which strategy could be used to integrate several existing | ||
| 15 | -FLOSS tools to promote the collaborative software development?} | ||
| 16 | -% | ||
| 17 | -The SPB portal integrates more than 10 FLOSS tools and provides several features, | ||
| 18 | -such as social network, mailing list, version control, content management and | ||
| 19 | -source code quality monitoring. Concerned with the platform sustainability and | ||
| 20 | -maintainability, the aforementioned 10 FLOSS tools were integrated with minimum | ||
| 21 | -differences from their official versions and the new developed features were | ||
| 22 | -sent upstream to ensure an alignment between the portal systems and their | ||
| 23 | -respective official versions. In the integration process, the main softwares | ||
| 24 | -were identified, specific teams were formed to work with each one of them | ||
| 25 | -and each team was composed of students with different levels of skills and at | ||
| 26 | -least one senior professional. | ||
| 27 | - | ||
| 28 | -\textbf{Q2 -- How to involve students in real-world projects interacting with | ||
| 29 | -real customers?} | ||
| 30 | -% | ||
| 31 | -In terms of mitigating conflicts, we tried to show that, as long as the | ||
| 32 | -university can provide a healthy and challenging environment to its students, | ||
| 33 | -one may conciliate studies and professional training in universities. | ||
| 34 | -% | ||
| 35 | -The students interacted with professionals of diverse fields of expertise, and they were | ||
| 36 | -able to participate in all levels of the software development process. | ||
| 37 | -This contributed to a great learning opportunity. | ||
| 38 | -% | ||
| 39 | -In our work process, based on open and collaborative software development | ||
| 40 | -practices, students could negotiate their work schedule as well as count on IT | ||
| 41 | -professionals to solve development issues. | ||
| 42 | -% | ||
| 43 | -Among the students, we have defined coaches for each team and a meta-coach | ||
| 44 | -(coach of whole project). All coaches, together with professors, have | ||
| 45 | -intermediated the communication between client (Ministry of Planning of Brazil) | ||
| 46 | -and the rest of the group. | ||
| 47 | -After the end of the project, some students successfully | ||
| 48 | -embraced opportunities in public and private sectors, within national borders | ||
| 49 | -and abroad. Some other students went further and started their own companies. | ||
| 50 | - | ||
| 51 | -\textbf{Q3 -- How to introduce the FLOSS collaborative and agile | ||
| 52 | -practices to governmental development process?} | ||
| 53 | -With some adaptations, what we called the ``translation processes'', it is feasible | ||
| 54 | -to conciliate agile methodologies and FLOSS practices to develop software to | ||
| 55 | -governmental organizations with functional hierarchical structures that use | ||
| 56 | -traditional development paradigm. | ||
| 57 | -% | ||
| 58 | -Aiming at reducing client questions about workconclusion, a DevOps front was | ||
| 59 | -created to automate all deploy process and also to work in continuous | ||
| 60 | -delivery. The government was brought to our work environment and interacted | ||
| 61 | -with our management and communication tools. For the project success, we | ||
| 62 | -focused on providing a friendly working environment as well as on showing to | ||
| 63 | -governmental agents another way to interact with the FLOSS community and the | ||
| 64 | -university. | ||
| 65 | - | ||
| 66 | -\subsection{Lessons Learned} | ||
| 67 | -\label{sec:lessons} | ||
| 68 | - | ||
| 69 | -From the answers of our initial open questions, we can also highlighted six lessons learned to make easier to share our experience during the development of the new SPB Portal. | ||
| 70 | - | ||
| 71 | -\textbf{L1 -- The participation of experienced professionals is crucial to | ||
| 72 | -the success of the project.} | ||
| 73 | -One important factor for the students was the composition of the teams | ||
| 74 | -with the participation of experienced professionals. | ||
| 75 | -% | ||
| 76 | -On the technical side, the senior developers and designers would handle | ||
| 77 | -the more difficult technical decisions, creating a work environment | ||
| 78 | -where the students could develop their skills in a didactic way without | ||
| 79 | -pressure. | ||
| 80 | -% | ||
| 81 | -On the management side, the active participation of professors -- who | ||
| 82 | -are, in the end, the ones responsible for the project -- is crucial to | ||
| 83 | -make sure students participation is conducted in a healthy way, and it is an | ||
| 84 | -instance of leading by example. | ||
| 85 | - | ||
| 86 | -\textbf{L2 -- A balanced relationship between academia and industry.} | ||
| 87 | -The experience of the SPB project led UnB to develop a work style which | ||
| 88 | -proved to be appropriate for an educational environment that brings | ||
| 89 | -academia and industry together. | ||
| 90 | -% | ||
| 91 | -The highest priority from the university's point of view is the | ||
| 92 | -students. Considering this, the activities of the project were never | ||
| 93 | -prioritized to the detriment of classes and other pedagogical | ||
| 94 | -activities. In summary, we had students working at different times, part | ||
| 95 | -time, remotely or locally, always respecting their individual | ||
| 96 | -conditions, but doing the work in a collective, collaborative and open | ||
| 97 | -way. | ||
| 98 | -% | ||
| 99 | -And even under a potentially adverse environment, the project delivered | ||
| 100 | -the desired solution with success. | ||
| 101 | -% | ||
| 102 | -At the end of the project, we noted that the skills developed by the | ||
| 103 | -students were at the software engineering state of the art. | ||
| 104 | -After the project ended, we had team members successfully embracing | ||
| 105 | -opportunities in public, private, national, and international | ||
| 106 | -organizations, in addition to those students that | ||
| 107 | -opened their own companies. | ||
| 108 | - | ||
| 109 | -\textbf{L3 -- Managing different organizational cultures.} | ||
| 110 | -In the beginning of the project, the Brazilian Government stakeholders | ||
| 111 | -had certain expectations about the project development that, let's | ||
| 112 | -say, didn't exactly match our work style based on agile and FLOSS practices. | ||
| 113 | -% | ||
| 114 | -We had to develop strategies that would support these different | ||
| 115 | -organizational cultures. The | ||
| 116 | -MP is organized in a functional hierarchical organizational structure, | ||
| 117 | -typically adopting a traditional development paradigm. Therefore, we had to | ||
| 118 | -create a translation process between our team and the MP managers who | ||
| 119 | -managed the project on their side assuming a traditional waterfall | ||
| 120 | -process. | ||
| 121 | - | ||
| 122 | -\textbf{L4 -- Managing higher-level and lower-level goals separately.} | ||
| 123 | -During the initial phase of the project, the MP team has brought | ||
| 124 | -strategic discussions to technical/operational meetings that | ||
| 125 | -were supposed to be about practical technical decisions. | ||
| 126 | -% | ||
| 127 | -This produced a highly complex communication and management environment, | ||
| 128 | -overloading the professors because they were supposed | ||
| 129 | -for maintaining the MP strategy synchronized with the | ||
| 130 | -implementation plans of the development team. This was hard, especially because the | ||
| 131 | -aforementioned cultural mismatch. Mixing both concerns in the same | ||
| 132 | -discussions caused confusion on both sides. | ||
| 133 | -% | ||
| 134 | -From the middle of the project we were able to keep those | ||
| 135 | -concerns separated, what eased the work of everyone involved. | ||
| 136 | - | ||
| 137 | -\textbf{L5 -- Living with ill-advised political decisions.} | ||
| 138 | -At the initial phases of the project, by political and personal | ||
| 139 | -motivation, the main stakeholders from the Brazilian government imposed | ||
| 140 | -the use of Colab to guide the development of the new SPB platform. Our | ||
| 141 | -team was totally against the idea because we already knew that Colab was | ||
| 142 | -a very experimental project and its adoption could dramatically increase | ||
| 143 | -the project complexity. Even though we provided technical reasons to | ||
| 144 | -not utilize Colab, the MP was adamant and we had to manage this problem. We did massive changes to | ||
| 145 | -Colab, and at the end of the project we have completely rewritten it to make | ||
| 146 | -it stable. It is important to notice that the MP compelled us to accept | ||
| 147 | -a technical decision based only on political interests, without considering | ||
| 148 | -all the resources that would be spent due to that decision. At the end | ||
| 149 | -of the project, we verified that Colab indeed consumed a vast amount of | ||
| 150 | -the budget and increased the project complexity. At the end of the | ||
| 151 | -project and after our analysis on the decision made by the MP, we | ||
| 152 | -understand that MP managers are capable of ignoring technical reasons | ||
| 153 | -in favor of political decisions. | ||
| 154 | - | ||
| 155 | -\textbf{L6 -- Consider sustainability from the beginning.} | ||
| 156 | -In the process of deploying the SPB platform in the MP infrastructure we had | ||
| 157 | -to interact with the MP technicians. We did several workshops, training | ||
| 158 | -and a meticulous documentation describing all the required procedures to | ||
| 159 | -update the platform, however, we realized that the MP technicians | ||
| 160 | -constantly avoid the responsibility. After noticing the aforementioned | ||
| 161 | -situation, we organized a DevOps team that completely automated all | ||
| 162 | -the deployment procedure. We simplified all the platform deployment to a | ||
| 163 | -few steps: (1) initial configurations (just ssh configuration) and (2) | ||
| 164 | -the execution of simple commands to completely update the platform. By | ||
| 165 | -the end of the project, we observed that the MP technicians invariably | ||
| 166 | -still depended on our support to update the platform even with all the | ||
| 167 | -automation provided by us. We were sadly left with a feeling of | ||
| 168 | -uncertainty about the future of the platform after the project ended. In | ||
| 169 | -hindsight, we realize that the MP dedicated system analysts and | ||
| 170 | -managers to the project, but not operations technicians. The later | ||
| 171 | -should have been more involved with the process so they could at least be | ||
| 172 | -comfortable in managing the platform infrastructure. | ||
| 173 | - | ||
| 174 | - | ||
| 175 | -\textbf{Final remarks and future work} | ||
| 176 | - | ||
| 177 | -The portal is available at \url{softwarepublico.gov.br}. All | ||
| 178 | -documentation, including detailed architecture and operation manuals are | ||
| 179 | -also available\footnote{\url{https://softwarepublico.gov.br/doc/} | ||
| 180 | -(in Portuguese only at the moment)}. | ||
| 181 | -% | ||
| 182 | -All the integrated tools are FLOSS and our contributions were published | ||
| 183 | -in open repositories, available on the SPB Portal itself. We also | ||
| 184 | -contributed these features back to the respective communities, which | ||
| 185 | -benefits both those communities and us, since we can share future | ||
| 186 | -development and maintenance effort with other organizations that | ||
| 187 | -participate in these projects. | ||
| 188 | - | ||
| 189 | -Future work should use data produced by the project to validate and evaluate | ||
| 190 | -how the used FLOSS and Agile practices have impacted the students and the | ||
| 191 | -governmental development process. For this, we would conduce a \textit{postmortem} | ||
| 192 | -analysis using the project open data and a survey targeting the involved actors. | ||
| 193 | - | ||
| 194 | -%=========== | ||
| 195 | -% Conclusion | ||
| 196 | -%=========== | ||
| 197 | - | ||
| 198 | -% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados | ||
| 199 | -%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa | ||
| 200 | -%% afirmação, embora eu e Paulo consigamos perceber isso. | ||
| 201 | - | ||
| 202 | - | ||
| 203 | -%* utilização do projeto para formação de recursos humanos (alunos) | ||
| 204 | - | ||
| 205 | -%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate | ||
| 206 | - | ||
| 207 | -%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) | ||
| 208 | - | ||
| 209 | -%* 69 projetos marcados como SPB, de 81 no total na plataforma. | ||
| 210 | - | ||
| 211 | -%* 47\% é desenvolvido em PHP. | ||
| 212 | - | ||
| 213 | -% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). | ||
| 214 | - | ||
| 215 | -% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. | ||
| 216 | - | ||
| 217 | -% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. | ||
| 218 | - | ||
| 219 | -%* Debater economia de recursos em orgão públicos | ||
| 220 | - |
| @@ -0,0 +1,58 @@ | @@ -0,0 +1,58 @@ | ||
| 1 | +\section{Contributions to the upstream communities} | ||
| 2 | +\label{sec:contributions} | ||
| 3 | + | ||
| 4 | +%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) | ||
| 5 | +%* Colab -> RevProxy | ||
| 6 | +%* Colab, atualização do python/django | ||
| 7 | +%* Contribuições para o GitLab (autenticação) | ||
| 8 | +%* Noosfero, atualização do Rails, preparação para federação, nova interface ... | ||
| 9 | +%* Coper, empacotamentos (obs), omniauth | ||
| 10 | + | ||
| 11 | + | ||
| 12 | +During the execution of this project we made several contributions to | ||
| 13 | +the upstream communities we interacted with. This occurred due to our | ||
| 14 | +development process aligned with those of the respective communities. | ||
| 15 | +During development, we would explicitly discuss the features and bug | ||
| 16 | +fixes that we were working on with the applicable upstream communities. | ||
| 17 | +This contributed to improve the developers technical solutions with | ||
| 18 | +expertise outside of our team, and make it easier for those changes to | ||
| 19 | +be accepted in the original projects. Having changes accepted upstream | ||
| 20 | +in turn makes our life easier as it minimizes the delta between our | ||
| 21 | +codebase and upstream's allowing us to upgrade and benefit from | ||
| 22 | +development work from others. | ||
| 23 | + | ||
| 24 | +In Colab, we helped upstream redesign the entirely architecture, | ||
| 25 | +enabling the development of plugins to integrate new tools. We also | ||
| 26 | +added a feature that allowed Colab to run asynchronous tasks, which was | ||
| 27 | +a major improvement for us since we were developing a complex system. A | ||
| 28 | +migration to the latest Django version was made (web framework used by | ||
| 29 | +Colab). Moreover, we worked on RevProxy (the more important dependency | ||
| 30 | +of Colab) to put it in a good shape, fixing many bugs. | ||
| 31 | + | ||
| 32 | +Gitlab was the tool that we made the least number of modifications. We | ||
| 33 | +contributed with some improvements related with configuration files and | ||
| 34 | +we developed a new plugin that enables user authentication in Gitlab | ||
| 35 | +through the REMOTE\_USER HTTP header. This plugin was needed because | ||
| 36 | +Colab uses this mechanism to manage the authentication. | ||
| 37 | + | ||
| 38 | +Noosfero was the tool that contemplated several functional requirements, | ||
| 39 | +therefore we made a large number of contributions with upstream. We helped to | ||
| 40 | +migrate to the latest Rails version (web framework used by Noosfero), enable | ||
| 41 | +the federation implementation (federation with other social networks), and | ||
| 42 | +decouple the interface and the back-end. | ||
| 43 | + | ||
| 44 | +Mezuro was completely rewritten and its architecture evolved adopting the | ||
| 45 | +microservice architecture\cite{namiot2014micro}. This way, we minimize the | ||
| 46 | +amount of code to maintain it, helping to test it and grant its code quality. | ||
| 47 | +In practice, we modularize the Mezuro project in several independent services. | ||
| 48 | +Currently, its computation and visualization modules use Kalibro and Prezento, | ||
| 49 | +respectively. They were developed into the Mezuro project and evolved during | ||
| 50 | +its integration to the new SPB Portal. | ||
| 51 | + | ||
| 52 | + | ||
| 53 | +We also made some contributions on the DevOps front. Some members of | ||
| 54 | +them team became maintainers of some Python libraries that were used by | ||
| 55 | +our scripts to upload packages to OBS (Open Build Service). We developed | ||
| 56 | +a tool called copr-status to keep track of the different stages of the | ||
| 57 | +lifecycle of each of the individual packages we were working on. | ||
| 58 | + |
opensym2017/spb.tex
| @@ -151,10 +151,8 @@ | @@ -151,10 +151,8 @@ | ||
| 151 | \input{content/05-requirements} | 151 | \input{content/05-requirements} |
| 152 | \input{content/06-architecture} | 152 | \input{content/06-architecture} |
| 153 | \input{content/07-features} | 153 | \input{content/07-features} |
| 154 | -\input{content/08-ux} | ||
| 155 | -\input{content/09-process} | ||
| 156 | -\input{content/10-contributions} | ||
| 157 | -\input{content/11-conclusion} | 154 | +\input{content/08-process} |
| 155 | +\input{content/09-conclusion} | ||
| 158 | 156 | ||
| 159 | %------------------------------------------------------------------------------ | 157 | %------------------------------------------------------------------------------ |
| 160 | \bibliographystyle{SIGCHI-Reference-Format} | 158 | \bibliographystyle{SIGCHI-Reference-Format} |