Commit f2e42ca6d92c79e79e657736fb974da5d28dfc33
1 parent
97a0eac5
Exists in
master
and in
3 other branches
Re-organizing sections and files
Showing
20 changed files
with
828 additions
and
797 deletions
Show diff stats
opensym2017/content/01-introduction.tex
@@ -88,3 +88,18 @@ the Brazilian government applying empirical software development methods. This | @@ -88,3 +88,18 @@ the Brazilian government applying empirical software development methods. This | ||
88 | case can help other projects to overcome similar software engineering challenges | 88 | case can help other projects to overcome similar software engineering challenges |
89 | in the future, as well as to illustrate how universities can improve the | 89 | in the future, as well as to illustrate how universities can improve the |
90 | real-world experience of their students by means of this kind of project. | 90 | real-world experience of their students by means of this kind of project. |
91 | + | ||
92 | + | ||
93 | +The remainder of this work is organized as follows. | ||
94 | +Section \ref{sec:spb}... | ||
95 | +Section \ref{sec:relatedwork} enumerates a number of related works on the... | ||
96 | +Section \ref{sec:researchdesign} presents the research design... | ||
97 | +Section \ref{sec:requirements} reports ... | ||
98 | +Section \ref{sec:architecture} ... | ||
99 | +Section \ref{sec:features} ... | ||
100 | +Section \ref{sec:ux} ... | ||
101 | +Section \ref{sec:process} ... | ||
102 | +Section \ref{sec:contributions} ... | ||
103 | +Section \ref{sec:lessons} ... | ||
104 | +Finally, Sections \ref{sec:conclusion} concludes the paper highlighting its main contributions and pointing paths to future works. | ||
105 | + |
opensym2017/content/03-relatedworks.tex
1 | -\section{Related Works and Projects} | ||
2 | -\label{sec:relatedworks} | 1 | +\section{Related Work} |
2 | +\label{sec:relatedwork} | ||
3 | 3 | ||
4 | The new SPB platform presented in this paper is a fully integrated | 4 | The new SPB platform presented in this paper is a fully integrated |
5 | environment, as we can see in Figure \ref{fig:requirements}, being very | 5 | environment, as we can see in Figure \ref{fig:requirements}, being very |
opensym2017/content/04-requirements.tex
@@ -1,79 +0,0 @@ | @@ -1,79 +0,0 @@ | ||
1 | -\section{Requirements} | ||
2 | -\label{sec:requirements} | ||
3 | - | ||
4 | -In 2013, the SPB Portal had more than 600 thousand unique visitors, generating | ||
5 | -more than 16 million page views with about 50 million hits. By evaluating only | ||
6 | -the main projects, there were more than 15 thousand downloads and 4 thousand | ||
7 | -messages exchanged in their forums. These data illustrates the potential of the | ||
8 | -SPB Portal, even with several limitations in the past. | ||
9 | - | ||
10 | -By preparing the evolution project described in this paper, the Brazilian | ||
11 | -government promoted 3 events to collect the requirements, in particular from | ||
12 | -society point of view: (i) an online form to collect general ideas; (ii) a | ||
13 | -face-to-face meeting with society in general; (iii) a workshop to review the | ||
14 | -SPB concepts and requirements with IT stakeholders from the Brazilian | ||
15 | -government and public organizations. | ||
16 | - | ||
17 | -After these 3 rounds discussing the new SPB platform, the Brazilian government | ||
18 | -listed about 145 requirements and developed a ``mind | ||
19 | -map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} | ||
20 | -to guide the SPB portal evolution. In this scenario, the 10 most voted | ||
21 | -requirements were: | ||
22 | - | ||
23 | -\begin{enumerate} | ||
24 | - | ||
25 | -\item Source code repository with public access; | ||
26 | -\item Visit community pages without login; | ||
27 | -\item Distributed version control system; | ||
28 | -\item Scores of users and developers collaboration; | ||
29 | -\item Search software by features; | ||
30 | -\item Integration with social networks; | ||
31 | -\item Repository for future ideas and requirements; | ||
32 | -\item Friendly URL to access a public software community page; | ||
33 | -\item User feedback about a public software; | ||
34 | -\item Report of the experience about the use of a public software; | ||
35 | - | ||
36 | -\end{enumerate} | ||
37 | - | ||
38 | -\begin{figure}[hbt] | ||
39 | - \centering | ||
40 | - \includegraphics[width=\linewidth]{figures/technological-requirements.png} | ||
41 | - \caption{Technological requirements.} | ||
42 | - \label{fig:requirements} | ||
43 | -\end{figure} | ||
44 | - | ||
45 | - | ||
46 | -There were other requirements based on the experience of the IT | ||
47 | -stakeholders from the Brazilian government and from the Brazilian FOSS | ||
48 | -community (that UnB and USP were representing too in this project). The | ||
49 | -new platform would only work properly if there is a unique | ||
50 | -authentication to use the provided tools. Additionally, a unified | ||
51 | -interface was an important non-functional requirement to have a better | ||
52 | -user experience in the new platform. | ||
53 | - | ||
54 | -At the first moment, we desired to release an initial version that could | ||
55 | -replace the old SPB portal. For that, the first version should have | ||
56 | -features such as: | ||
57 | - | ||
58 | -\begin{enumerate} | ||
59 | - | ||
60 | -\item An organized public software catalog; | ||
61 | -\item Social network environment (profiles for users, software pages, and community pages); | ||
62 | -\item Content Management Systems (CMS) features; | ||
63 | -\item Web-based Git repository manager with wiki and issue tracking features; | ||
64 | -\item Mailing lists and discussion forums; | ||
65 | - | ||
66 | -\end{enumerate} | ||
67 | - | ||
68 | -Other requirements were also planned during the conception phase of the | ||
69 | -SPB evolution project, such as an integrated search engine and a | ||
70 | -web-based source code static analysis monitor. By analyzing all of these | ||
71 | -requirements, we proposed the technological requirements | ||
72 | -illustrated in Figure \ref{fig:requirements} to guide the development of | ||
73 | -the new SPB platform. In other words, we have designed the SPB evolution | ||
74 | -project based on existing FOSS tools. However, the integration of | ||
75 | -several existing systems that were already implemented in different | ||
76 | -programming languages and frameworks, adding features such as a | ||
77 | -centralized authentication, unified interface, and a search engine, as | ||
78 | -well as, other back-end features, would require a non-trivial amount of | ||
79 | -work. |
@@ -0,0 +1,14 @@ | @@ -0,0 +1,14 @@ | ||
1 | +\section{Research Design} | ||
2 | +\label{sec:researchdesign} | ||
3 | + | ||
4 | +\begin{description} | ||
5 | + | ||
6 | +\item [RQ1:] \textit{Which strategy could be used to integrate several existing FOSS tools to promote the collaborative software development?} | ||
7 | + | ||
8 | + | ||
9 | +\item [RQ2:] \textit{How to introduce the FOSS collaborative and agile practices to governmental development process?} | ||
10 | + | ||
11 | + | ||
12 | +\item [RQ2:] \textit{How to involve undergraduate students in real-world projects, interacting with real customers from a Government?} | ||
13 | + | ||
14 | +\end{description} |
opensym2017/content/05-architecture.tex
@@ -1,171 +0,0 @@ | @@ -1,171 +0,0 @@ | ||
1 | -\section{Architecture} | ||
2 | -\label{sec:architecture} | ||
3 | - | ||
4 | -Based on the extensive list of functional requirements defined by the | ||
5 | -Brazilian Federal Government, we selected some FOSS systems to form our | ||
6 | -solution. We looked for system that together could provide the largest | ||
7 | -subset possible of the requirements, and were fully aware that we would | ||
8 | -need to improve those systems in order to provide the rest. We were also | ||
9 | -convinced that it would be impossible to provide all of those | ||
10 | -requirements with a single tool. | ||
11 | - | ||
12 | -From the point of view of the architecture, two main requirements was | ||
13 | -brought by the Brazilian Federal Government for the new platform: | ||
14 | - | ||
15 | -\begin{enumerate} | ||
16 | - \item \textit{Integrate existing FOSS systems}, with minimal differences from | ||
17 | - their original versions; | ||
18 | - \item \textit{Provide a consistent user interface} across the different | ||
19 | - systems, as well as centralized authentication. | ||
20 | -\end{enumerate} | ||
21 | - | ||
22 | -Adopting existing FOSS systems and minimizing locally-made changes had | ||
23 | -the objective of being able to upgrade to newer versions of the original | ||
24 | -software, benefiting from improvements and maintenance done by the | ||
25 | -existing project communities. Providing a consistent user interface on | ||
26 | -top of those different tools was needed to make the transition between | ||
27 | -the different systems seamless from the point of view of users, which | ||
28 | -would be confused by jumping through completely different interfaces | ||
29 | -while interacting with the portal. | ||
30 | - | ||
31 | -For the first requirement, we identified four main systems that required | ||
32 | -specialized teams for work in the integration process. The teams learned | ||
33 | -how to develop for their assigned systems and contributed to the | ||
34 | -original communities, so that the version we used were not significantly | ||
35 | -different from the original. | ||
36 | - | ||
37 | -At the end of the project, the SPB portal was composed of more than ten | ||
38 | -systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and | ||
39 | -Munin. The remainder of this section describes the most relevant of | ||
40 | -them, as well as how they were integrated into the platform. | ||
41 | - | ||
42 | -\subsection{Colab} | ||
43 | - | ||
44 | -Colab\footnote{\url{https://github.com/colab}} is a systems integration | ||
45 | -platform for web applications. One of its goals is allowing different | ||
46 | -applications to be combined in such a way that an user does not notice | ||
47 | -the change between applications. For that, Colab provides facilities | ||
48 | -for: | ||
49 | - | ||
50 | -\begin{itemize} | ||
51 | - \item \textit{Centralized authentication}: Users are automatically | ||
52 | -identified for each application without having to enter login | ||
53 | -credentials on each of them; | ||
54 | - \item \textit{Visual consistency}: Colab is able to change the HTML | ||
55 | -code produced by the integrated applications, such as defining default | ||
56 | -template or reusing common HTML pages, in order to provide a unified | ||
57 | -interface; | ||
58 | - \item \textit{Relaying of events between applications}: The | ||
59 | -integrated applications or Colab itself are able to trigger an action | ||
60 | -on another application; | ||
61 | - \item \textit{Integrated search engine}: It is possible to set up | ||
62 | -exactly which data needs to be indexed from each application and how | ||
63 | -users are directed to this information on correspondent application; | ||
64 | -\end{itemize} | ||
65 | - | ||
66 | -Colab implements this integration by working as a reverse proxy for the | ||
67 | -integration applications, that is, all external requests pass through | ||
68 | -Colab before reaching them. | ||
69 | - | ||
70 | -Initially, Colab had support for a small set of applications (Trac, GNU | ||
71 | -Mailman, and Apache Lucene) and support for them was hard-coded. Our | ||
72 | -team evolved Colab so that it can now receive plugins that add support | ||
73 | -new applications without the need of changes to the Colab core. | ||
74 | - | ||
75 | -\subsection{Noosfero} | ||
76 | - | ||
77 | -Noosfero\footnote{\url{http://noosfero.org}} is a software for building | ||
78 | -social and collaboration networks. Besides the classical social | ||
79 | -networking features, it also provides publication features such as blogs | ||
80 | -and a general-purpose CMS (Content Management System). Most of the user | ||
81 | -interactions with SPB is through Noosfero: user registration, project | ||
82 | -home pages and documentation, and contact forms. | ||
83 | - | ||
84 | -\subsection{Gitlab} | ||
85 | - | ||
86 | -GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository | ||
87 | -manager with wiki pages and issue tracking features. Gitlab is a FOSS | ||
88 | -platform and focuses on delivering a holistic solution that will see | ||
89 | -developers from idea to production seamlessly and on a single platform. | ||
90 | - | ||
91 | -Gitlab has several unique features, such as built-in continuous | ||
92 | -integration and continuous deployment, flexible permissions, tracking of | ||
93 | -Work-in-Progress work, moving issues between projects, group-level | ||
94 | -milestones, creating new branches from issues, issues board, and time | ||
95 | -tracking. | ||
96 | - | ||
97 | -\subsection{Mezuro} | ||
98 | - | ||
99 | -Mezuro\footnote{\url{http://mezuro.org/}} \cite{mezuro_oss} is a platform to | ||
100 | -collect source code metrics to monitor the internal quality of software written | ||
101 | -in C, C++, Java, Python, Ruby, and PHP. | ||
102 | - | ||
103 | -In general, source code metrics tools also do not present a friendly way | ||
104 | -to interpret its results and, even more, do not follow a standardization | ||
105 | -between them. Mezuro collects and presents these results to the end | ||
106 | -user, specially, by analyzing source code metric history during its life | ||
107 | -cycle. | ||
108 | - | ||
109 | -The Mezuro platform provides a single interface grouping available | ||
110 | -tools, allows selection and composition of metrics in a flexible manner, | ||
111 | -stores the metrics evolution history, presents results in a friendly | ||
112 | -way, as well as, allows users to customize the given interpretation | ||
113 | -accordingly to their own context. | ||
114 | - | ||
115 | -\subsection{System unification} | ||
116 | - | ||
117 | -\begin{figure}[hbt] | ||
118 | - \centering | ||
119 | - \includegraphics[width=\linewidth]{figures/arch.png} | ||
120 | - \caption{SPB architecture overview.} | ||
121 | - \label{fig:architecture} | ||
122 | -\end{figure} | ||
123 | - | ||
124 | -The conceptual architecture of the platform is presented in Figure | ||
125 | -\ref{fig:architecture}. Colab initially handles all user interaction, | ||
126 | -directing requests to one of the integrated applications. It | ||
127 | -post-processes responses from the applications to apply a consistent | ||
128 | -visual appearance, manages authentication, and provides a unified search | ||
129 | -functionality: instead of using the redundant restricted search | ||
130 | -functionality of each application, a search o n the SPB portal might | ||
131 | -return content from any of the applications, be it web pages, mailing | ||
132 | -list posts, or source code. | ||
133 | - | ||
134 | -% Falar do devops | ||
135 | -\subsection{Deploy} | ||
136 | - | ||
137 | -The SPB platform was deployed in 7 virtual machines with different functions, | ||
138 | -as we can see in Figure \ref{fig:architecture2}. | ||
139 | - | ||
140 | -\begin{figure*}[hbt] | ||
141 | - \centering | ||
142 | - \includegraphics[width=.8\linewidth]{figures/arch3.png} | ||
143 | - \caption{Instanciation view of the SPB architecture.} | ||
144 | - \label{fig:architecture2} | ||
145 | -\end{figure*} | ||
146 | - | ||
147 | -The \textit{reverseproxy} handles the HTTP requests and redirects them to the | ||
148 | -\textit{integration}, the \textit{email} sends and receives e-mails on behalf | ||
149 | -of the platform and the \textit{monitor} keeps the entire environment tracked. | ||
150 | -These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and | ||
151 | -\textit{monitor} - are accessible via Internet and the other ones are only | ||
152 | -available in the local network created between them. | ||
153 | - | ||
154 | -\textit{Integration} works as a second layer of proxy beneath | ||
155 | -\textit{reverseproxy}, any request to the platform will be handled by it. The | ||
156 | -Colab service provides interface, authentication and search engine integration | ||
157 | -among all the services. When a request is received to a specific service, | ||
158 | -Colab authenticates the user in the target tool, sends the request and makes a | ||
159 | -visual transformation in the HTML page which is the content of the response. | ||
160 | -Another user-oriented feature is the integrated search engine, when the user | ||
161 | -want to find something in the platform Colab will perform the search in the | ||
162 | -whole databases. Colab itself provides a web interface for GNU Mailman and we | ||
163 | -have two others integrated tools in \textit{integration}: Gitlab and Prezento. | ||
164 | -Gitlab provides web interface for Git repositories and issues tracker, and | ||
165 | -Prezento is a front-end for source code static analysis. | ||
166 | - | ||
167 | -The source code static analysis is performed by \textit{mezuro}. It runs some | ||
168 | -static analysis tools on source code stored in repository and provide this data | ||
169 | -to Prezento. A social network and CMS (Content Manager System) is provided by | ||
170 | -Noosfero in \textit{social}, and the databases of all tools with a cache | ||
171 | -service are in \textit{database}. |
@@ -0,0 +1,79 @@ | @@ -0,0 +1,79 @@ | ||
1 | +\section{Requirements} | ||
2 | +\label{sec:requirements} | ||
3 | + | ||
4 | +In 2013, the SPB Portal had more than 600 thousand unique visitors, generating | ||
5 | +more than 16 million page views with about 50 million hits. By evaluating only | ||
6 | +the main projects, there were more than 15 thousand downloads and 4 thousand | ||
7 | +messages exchanged in their forums. These data illustrates the potential of the | ||
8 | +SPB Portal, even with several limitations in the past. | ||
9 | + | ||
10 | +By preparing the evolution project described in this paper, the Brazilian | ||
11 | +government promoted 3 events to collect the requirements, in particular from | ||
12 | +society point of view: (i) an online form to collect general ideas; (ii) a | ||
13 | +face-to-face meeting with society in general; (iii) a workshop to review the | ||
14 | +SPB concepts and requirements with IT stakeholders from the Brazilian | ||
15 | +government and public organizations. | ||
16 | + | ||
17 | +After these 3 rounds discussing the new SPB platform, the Brazilian government | ||
18 | +listed about 145 requirements and developed a ``mind | ||
19 | +map''\footnote{\url{https://softwarepublico.gov.br/social/spb/gallery/mapaconceitual.png}} | ||
20 | +to guide the SPB portal evolution. In this scenario, the 10 most voted | ||
21 | +requirements were: | ||
22 | + | ||
23 | +\begin{enumerate} | ||
24 | + | ||
25 | +\item Source code repository with public access; | ||
26 | +\item Visit community pages without login; | ||
27 | +\item Distributed version control system; | ||
28 | +\item Scores of users and developers collaboration; | ||
29 | +\item Search software by features; | ||
30 | +\item Integration with social networks; | ||
31 | +\item Repository for future ideas and requirements; | ||
32 | +\item Friendly URL to access a public software community page; | ||
33 | +\item User feedback about a public software; | ||
34 | +\item Report of the experience about the use of a public software; | ||
35 | + | ||
36 | +\end{enumerate} | ||
37 | + | ||
38 | +\begin{figure}[hbt] | ||
39 | + \centering | ||
40 | + \includegraphics[width=\linewidth]{figures/technological-requirements.png} | ||
41 | + \caption{Technological requirements.} | ||
42 | + \label{fig:requirements} | ||
43 | +\end{figure} | ||
44 | + | ||
45 | + | ||
46 | +There were other requirements based on the experience of the IT | ||
47 | +stakeholders from the Brazilian government and from the Brazilian FOSS | ||
48 | +community (that UnB and USP were representing too in this project). The | ||
49 | +new platform would only work properly if there is a unique | ||
50 | +authentication to use the provided tools. Additionally, a unified | ||
51 | +interface was an important non-functional requirement to have a better | ||
52 | +user experience in the new platform. | ||
53 | + | ||
54 | +At the first moment, we desired to release an initial version that could | ||
55 | +replace the old SPB portal. For that, the first version should have | ||
56 | +features such as: | ||
57 | + | ||
58 | +\begin{enumerate} | ||
59 | + | ||
60 | +\item An organized public software catalog; | ||
61 | +\item Social network environment (profiles for users, software pages, and community pages); | ||
62 | +\item Content Management Systems (CMS) features; | ||
63 | +\item Web-based Git repository manager with wiki and issue tracking features; | ||
64 | +\item Mailing lists and discussion forums; | ||
65 | + | ||
66 | +\end{enumerate} | ||
67 | + | ||
68 | +Other requirements were also planned during the conception phase of the | ||
69 | +SPB evolution project, such as an integrated search engine and a | ||
70 | +web-based source code static analysis monitor. By analyzing all of these | ||
71 | +requirements, we proposed the technological requirements | ||
72 | +illustrated in Figure \ref{fig:requirements} to guide the development of | ||
73 | +the new SPB platform. In other words, we have designed the SPB evolution | ||
74 | +project based on existing FOSS tools. However, the integration of | ||
75 | +several existing systems that were already implemented in different | ||
76 | +programming languages and frameworks, adding features such as a | ||
77 | +centralized authentication, unified interface, and a search engine, as | ||
78 | +well as, other back-end features, would require a non-trivial amount of | ||
79 | +work. |
@@ -0,0 +1,171 @@ | @@ -0,0 +1,171 @@ | ||
1 | +\section{Architecture} | ||
2 | +\label{sec:architecture} | ||
3 | + | ||
4 | +Based on the extensive list of functional requirements defined by the | ||
5 | +Brazilian Federal Government, we selected some FOSS systems to form our | ||
6 | +solution. We looked for system that together could provide the largest | ||
7 | +subset possible of the requirements, and were fully aware that we would | ||
8 | +need to improve those systems in order to provide the rest. We were also | ||
9 | +convinced that it would be impossible to provide all of those | ||
10 | +requirements with a single tool. | ||
11 | + | ||
12 | +From the point of view of the architecture, two main requirements was | ||
13 | +brought by the Brazilian Federal Government for the new platform: | ||
14 | + | ||
15 | +\begin{enumerate} | ||
16 | + \item \textit{Integrate existing FOSS systems}, with minimal differences from | ||
17 | + their original versions; | ||
18 | + \item \textit{Provide a consistent user interface} across the different | ||
19 | + systems, as well as centralized authentication. | ||
20 | +\end{enumerate} | ||
21 | + | ||
22 | +Adopting existing FOSS systems and minimizing locally-made changes had | ||
23 | +the objective of being able to upgrade to newer versions of the original | ||
24 | +software, benefiting from improvements and maintenance done by the | ||
25 | +existing project communities. Providing a consistent user interface on | ||
26 | +top of those different tools was needed to make the transition between | ||
27 | +the different systems seamless from the point of view of users, which | ||
28 | +would be confused by jumping through completely different interfaces | ||
29 | +while interacting with the portal. | ||
30 | + | ||
31 | +For the first requirement, we identified four main systems that required | ||
32 | +specialized teams for work in the integration process. The teams learned | ||
33 | +how to develop for their assigned systems and contributed to the | ||
34 | +original communities, so that the version we used were not significantly | ||
35 | +different from the original. | ||
36 | + | ||
37 | +At the end of the project, the SPB portal was composed of more than ten | ||
38 | +systems, such as Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, and | ||
39 | +Munin. The remainder of this section describes the most relevant of | ||
40 | +them, as well as how they were integrated into the platform. | ||
41 | + | ||
42 | +\subsection{Colab} | ||
43 | + | ||
44 | +Colab\footnote{\url{https://github.com/colab}} is a systems integration | ||
45 | +platform for web applications. One of its goals is allowing different | ||
46 | +applications to be combined in such a way that an user does not notice | ||
47 | +the change between applications. For that, Colab provides facilities | ||
48 | +for: | ||
49 | + | ||
50 | +\begin{itemize} | ||
51 | + \item \textit{Centralized authentication}: Users are automatically | ||
52 | +identified for each application without having to enter login | ||
53 | +credentials on each of them; | ||
54 | + \item \textit{Visual consistency}: Colab is able to change the HTML | ||
55 | +code produced by the integrated applications, such as defining default | ||
56 | +template or reusing common HTML pages, in order to provide a unified | ||
57 | +interface; | ||
58 | + \item \textit{Relaying of events between applications}: The | ||
59 | +integrated applications or Colab itself are able to trigger an action | ||
60 | +on another application; | ||
61 | + \item \textit{Integrated search engine}: It is possible to set up | ||
62 | +exactly which data needs to be indexed from each application and how | ||
63 | +users are directed to this information on correspondent application; | ||
64 | +\end{itemize} | ||
65 | + | ||
66 | +Colab implements this integration by working as a reverse proxy for the | ||
67 | +integration applications, that is, all external requests pass through | ||
68 | +Colab before reaching them. | ||
69 | + | ||
70 | +Initially, Colab had support for a small set of applications (Trac, GNU | ||
71 | +Mailman, and Apache Lucene) and support for them was hard-coded. Our | ||
72 | +team evolved Colab so that it can now receive plugins that add support | ||
73 | +new applications without the need of changes to the Colab core. | ||
74 | + | ||
75 | +\subsection{Noosfero} | ||
76 | + | ||
77 | +Noosfero\footnote{\url{http://noosfero.org}} is a software for building | ||
78 | +social and collaboration networks. Besides the classical social | ||
79 | +networking features, it also provides publication features such as blogs | ||
80 | +and a general-purpose CMS (Content Management System). Most of the user | ||
81 | +interactions with SPB is through Noosfero: user registration, project | ||
82 | +home pages and documentation, and contact forms. | ||
83 | + | ||
84 | +\subsection{Gitlab} | ||
85 | + | ||
86 | +GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository | ||
87 | +manager with wiki pages and issue tracking features. Gitlab is a FOSS | ||
88 | +platform and focuses on delivering a holistic solution that will see | ||
89 | +developers from idea to production seamlessly and on a single platform. | ||
90 | + | ||
91 | +Gitlab has several unique features, such as built-in continuous | ||
92 | +integration and continuous deployment, flexible permissions, tracking of | ||
93 | +Work-in-Progress work, moving issues between projects, group-level | ||
94 | +milestones, creating new branches from issues, issues board, and time | ||
95 | +tracking. | ||
96 | + | ||
97 | +\subsection{Mezuro} | ||
98 | + | ||
99 | +Mezuro\footnote{\url{http://mezuro.org/}} \cite{mezuro_oss} is a platform to | ||
100 | +collect source code metrics to monitor the internal quality of software written | ||
101 | +in C, C++, Java, Python, Ruby, and PHP. | ||
102 | + | ||
103 | +In general, source code metrics tools also do not present a friendly way | ||
104 | +to interpret its results and, even more, do not follow a standardization | ||
105 | +between them. Mezuro collects and presents these results to the end | ||
106 | +user, specially, by analyzing source code metric history during its life | ||
107 | +cycle. | ||
108 | + | ||
109 | +The Mezuro platform provides a single interface grouping available | ||
110 | +tools, allows selection and composition of metrics in a flexible manner, | ||
111 | +stores the metrics evolution history, presents results in a friendly | ||
112 | +way, as well as, allows users to customize the given interpretation | ||
113 | +accordingly to their own context. | ||
114 | + | ||
115 | +\subsection{System unification} | ||
116 | + | ||
117 | +\begin{figure}[hbt] | ||
118 | + \centering | ||
119 | + \includegraphics[width=\linewidth]{figures/arch.png} | ||
120 | + \caption{SPB architecture overview.} | ||
121 | + \label{fig:architecture} | ||
122 | +\end{figure} | ||
123 | + | ||
124 | +The conceptual architecture of the platform is presented in Figure | ||
125 | +\ref{fig:architecture}. Colab initially handles all user interaction, | ||
126 | +directing requests to one of the integrated applications. It | ||
127 | +post-processes responses from the applications to apply a consistent | ||
128 | +visual appearance, manages authentication, and provides a unified search | ||
129 | +functionality: instead of using the redundant restricted search | ||
130 | +functionality of each application, a search o n the SPB portal might | ||
131 | +return content from any of the applications, be it web pages, mailing | ||
132 | +list posts, or source code. | ||
133 | + | ||
134 | +% Falar do devops | ||
135 | +\subsection{Deploy} | ||
136 | + | ||
137 | +The SPB platform was deployed in 7 virtual machines with different functions, | ||
138 | +as we can see in Figure \ref{fig:architecture2}. | ||
139 | + | ||
140 | +\begin{figure*}[hbt] | ||
141 | + \centering | ||
142 | + \includegraphics[width=.8\linewidth]{figures/arch3.png} | ||
143 | + \caption{Instanciation view of the SPB architecture.} | ||
144 | + \label{fig:architecture2} | ||
145 | +\end{figure*} | ||
146 | + | ||
147 | +The \textit{reverseproxy} handles the HTTP requests and redirects them to the | ||
148 | +\textit{integration}, the \textit{email} sends and receives e-mails on behalf | ||
149 | +of the platform and the \textit{monitor} keeps the entire environment tracked. | ||
150 | +These three \textit{VMs} mentioned - \textit{reverseproxy}, \textit{email} and | ||
151 | +\textit{monitor} - are accessible via Internet and the other ones are only | ||
152 | +available in the local network created between them. | ||
153 | + | ||
154 | +\textit{Integration} works as a second layer of proxy beneath | ||
155 | +\textit{reverseproxy}, any request to the platform will be handled by it. The | ||
156 | +Colab service provides interface, authentication and search engine integration | ||
157 | +among all the services. When a request is received to a specific service, | ||
158 | +Colab authenticates the user in the target tool, sends the request and makes a | ||
159 | +visual transformation in the HTML page which is the content of the response. | ||
160 | +Another user-oriented feature is the integrated search engine, when the user | ||
161 | +want to find something in the platform Colab will perform the search in the | ||
162 | +whole databases. Colab itself provides a web interface for GNU Mailman and we | ||
163 | +have two others integrated tools in \textit{integration}: Gitlab and Prezento. | ||
164 | +Gitlab provides web interface for Git repositories and issues tracker, and | ||
165 | +Prezento is a front-end for source code static analysis. | ||
166 | + | ||
167 | +The source code static analysis is performed by \textit{mezuro}. It runs some | ||
168 | +static analysis tools on source code stored in repository and provide this data | ||
169 | +to Prezento. A social network and CMS (Content Manager System) is provided by | ||
170 | +Noosfero in \textit{social}, and the databases of all tools with a cache | ||
171 | +service are in \textit{database}. |
opensym2017/content/06-features.tex
@@ -1,76 +0,0 @@ | @@ -1,76 +0,0 @@ | ||
1 | -\section{Features} | ||
2 | -\label{sec:spb} | ||
3 | - | ||
4 | -The new generation of the SPB portal combines features adapted from existing collaborative software | ||
5 | -and features developed by the SPB team. Whenever possible, new functions | ||
6 | -(newly developed or partially modified) were contributed back to the official repositories. | ||
7 | - | ||
8 | -As a result, we have a platform that integrates and harmonizes different features such as | ||
9 | -social networking, mailing list, version control system, content management and | ||
10 | -source code quality monitoring. Our aim was to develop functionalities by reusing | ||
11 | -functionality of collaborative software already integrated to the platform. In | ||
12 | -addition, we tried to keep this integration transparent to end users. | ||
13 | - | ||
14 | -\subsection{Software and Software Community} | ||
15 | - | ||
16 | -In the new SPB portal, each software has a standard set of pages and tools. | ||
17 | -Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download | ||
18 | -different versions of the software and find several mechanisms of software development management. | ||
19 | - | ||
20 | -Focusing on the collaborative development, Mailman was integrated to the platform in order to allow | ||
21 | -the dialogue and communication between developers, users and | ||
22 | -enthusiasts of a determined software. The software has its own mailing list whose privacy | ||
23 | -can be configured/set by administrators. | ||
24 | - | ||
25 | -The software has a social interface area ("software community") where users can find other users, blogs, | ||
26 | -summary of recent activities, or any other relevant community-produced content. | ||
27 | -Users logged to the platform can request membership to different software communities | ||
28 | -and each community member can access and edit restricted content. For this purpose, | ||
29 | -many Noosfero features related to social networking and content management were integrated to the portal. | ||
30 | - | ||
31 | -To assist decision-making, the new SPB has acquired assessment and statistical | ||
32 | -tools. Now, users will be able to rate the software and make comments and all | ||
33 | -information will be avaiable to other users. Moreover, the software has a section | ||
34 | -containing its statistical data, where values are calculated based on data | ||
35 | -provided by users and the system. | ||
36 | - | ||
37 | -The role of the administrator will be present in the software and in its community. The | ||
38 | -administrator is responsible for moderating content, memberships and user | ||
39 | -comments. The administrator is also the one who can make changes in the software homepage | ||
40 | -content. | ||
41 | - | ||
42 | -\subsection{Software Catalog and global search} | ||
43 | - | ||
44 | -The platform also provides a search tool called Software Catalog, | ||
45 | -which allows users to find softwares available in the portal. | ||
46 | -In this catalog, some search options were developed to make the navigation easier, | ||
47 | -such as filters (by type of software or category), sorting and score. | ||
48 | - | ||
49 | -In order to expand the searching scope and cover more types of content, the SPB team | ||
50 | -developed the global search tool. This tool unifies search mechanisms | ||
51 | -provided by the different collaborative software used on the SPB portal. Any user can | ||
52 | -find a public content in the context of social networking, mailing list and | ||
53 | -software development. | ||
54 | - | ||
55 | -\subsection{Software development tools} | ||
56 | - | ||
57 | -Usually, Collaborative Development Environments (CDE) demand a version control | ||
58 | -system, trackers, build tools, knowledge centers, and communication tools | ||
59 | -\cite{collaboration_tools}. | ||
60 | -The new SPB also provides | ||
61 | -tools to encourage developers to keep source code and its | ||
62 | -development activity within the platform. Any created software has, by default, a | ||
63 | -an associated git repository with wiki pages and issue tracking. These tools are | ||
64 | -supplied by the integration of Gitlab into the platform. | ||
65 | - | ||
66 | -Developers can also evaluate the software source code to measure software | ||
67 | -quality. With Mezuro, they can schedule the analysis of the source-code and follow its | ||
68 | -metric results evolution over time. Results of each metric analysis are | ||
69 | -public, which allows greater transparency between the developer and the | ||
70 | -community that uses the software. Thereby, the maintainers can decide if the | ||
71 | -given solution meets the source code quality requirements. | ||
72 | - | ||
73 | -Thus, the SPB became a platform to stimulate the openness of the source code; | ||
74 | -dialogue between users and the development team; and also | ||
75 | -maintenance and evolution of the software, which will provide more | ||
76 | -transparency in government investments. |
@@ -0,0 +1,76 @@ | @@ -0,0 +1,76 @@ | ||
1 | +\section{Features} | ||
2 | +\label{sec:features} | ||
3 | + | ||
4 | +The new generation of the SPB portal combines features adapted from existing collaborative software | ||
5 | +and features developed by the SPB team. Whenever possible, new functions | ||
6 | +(newly developed or partially modified) were contributed back to the official repositories. | ||
7 | + | ||
8 | +As a result, we have a platform that integrates and harmonizes different features such as | ||
9 | +social networking, mailing list, version control system, content management and | ||
10 | +source code quality monitoring. Our aim was to develop functionalities by reusing | ||
11 | +functionality of collaborative software already integrated to the platform. In | ||
12 | +addition, we tried to keep this integration transparent to end users. | ||
13 | + | ||
14 | +\subsection{Software and Software Community} | ||
15 | + | ||
16 | +In the new SPB portal, each software has a standard set of pages and tools. | ||
17 | +Besides accessing support pages (such as FAQ and installation guide) within the platform, users will be able to download | ||
18 | +different versions of the software and find several mechanisms of software development management. | ||
19 | + | ||
20 | +Focusing on the collaborative development, Mailman was integrated to the platform in order to allow | ||
21 | +the dialogue and communication between developers, users and | ||
22 | +enthusiasts of a determined software. The software has its own mailing list whose privacy | ||
23 | +can be configured/set by administrators. | ||
24 | + | ||
25 | +The software has a social interface area ("software community") where users can find other users, blogs, | ||
26 | +summary of recent activities, or any other relevant community-produced content. | ||
27 | +Users logged to the platform can request membership to different software communities | ||
28 | +and each community member can access and edit restricted content. For this purpose, | ||
29 | +many Noosfero features related to social networking and content management were integrated to the portal. | ||
30 | + | ||
31 | +To assist decision-making, the new SPB has acquired assessment and statistical | ||
32 | +tools. Now, users will be able to rate the software and make comments and all | ||
33 | +information will be avaiable to other users. Moreover, the software has a section | ||
34 | +containing its statistical data, where values are calculated based on data | ||
35 | +provided by users and the system. | ||
36 | + | ||
37 | +The role of the administrator will be present in the software and in its community. The | ||
38 | +administrator is responsible for moderating content, memberships and user | ||
39 | +comments. The administrator is also the one who can make changes in the software homepage | ||
40 | +content. | ||
41 | + | ||
42 | +\subsection{Software Catalog and global search} | ||
43 | + | ||
44 | +The platform also provides a search tool called Software Catalog, | ||
45 | +which allows users to find softwares available in the portal. | ||
46 | +In this catalog, some search options were developed to make the navigation easier, | ||
47 | +such as filters (by type of software or category), sorting and score. | ||
48 | + | ||
49 | +In order to expand the searching scope and cover more types of content, the SPB team | ||
50 | +developed the global search tool. This tool unifies search mechanisms | ||
51 | +provided by the different collaborative software used on the SPB portal. Any user can | ||
52 | +find a public content in the context of social networking, mailing list and | ||
53 | +software development. | ||
54 | + | ||
55 | +\subsection{Software development tools} | ||
56 | + | ||
57 | +Usually, Collaborative Development Environments (CDE) demand a version control | ||
58 | +system, trackers, build tools, knowledge centers, and communication tools | ||
59 | +\cite{collaboration_tools}. | ||
60 | +The new SPB also provides | ||
61 | +tools to encourage developers to keep source code and its | ||
62 | +development activity within the platform. Any created software has, by default, a | ||
63 | +an associated git repository with wiki pages and issue tracking. These tools are | ||
64 | +supplied by the integration of Gitlab into the platform. | ||
65 | + | ||
66 | +Developers can also evaluate the software source code to measure software | ||
67 | +quality. With Mezuro, they can schedule the analysis of the source-code and follow its | ||
68 | +metric results evolution over time. Results of each metric analysis are | ||
69 | +public, which allows greater transparency between the developer and the | ||
70 | +community that uses the software. Thereby, the maintainers can decide if the | ||
71 | +given solution meets the source code quality requirements. | ||
72 | + | ||
73 | +Thus, the SPB became a platform to stimulate the openness of the source code; | ||
74 | +dialogue between users and the development team; and also | ||
75 | +maintenance and evolution of the software, which will provide more | ||
76 | +transparency in government investments. |
opensym2017/content/07-ux.tex
@@ -1,33 +0,0 @@ | @@ -1,33 +0,0 @@ | ||
1 | -\section{User eXperience evolution} | ||
2 | - | ||
3 | -The integration of collaborative environments goes beyond functional aspects. | ||
4 | -Offering the population an unified experience across these environments has | ||
5 | -been the key to encourage the use of the platform as it reduces the perception | ||
6 | -of complexity. Thus, the SPB Portal information architecture was redesigned | ||
7 | -to provide a transparent navigation and to reach users with different profiles. | ||
8 | -A process of harmonization has been employed on the interaction models of each | ||
9 | -tool to reduce the learning curve. At the same time, a new visual style was | ||
10 | -created to unify the navigation experience and to comply with the guidelines of | ||
11 | -the digital communication identity standard established by the Federal | ||
12 | -Government. | ||
13 | - | ||
14 | -With the increase in system features and the addition of new tools, the | ||
15 | -visual style has steadily evolved to keep the navigation unified. Moreover, | ||
16 | -tools from different backgrounds, which in many cases provide similar | ||
17 | -functionality, prompted the development of an unified interface. Some | ||
18 | -features, such as search and user profile editing were eliminated from | ||
19 | -the individual applications, and implemented centrally to ensure a | ||
20 | -consistent look and feel. | ||
21 | - | ||
22 | -Another challenge was responsive web design. The integrated applications | ||
23 | -had varying degrees of support for responsiveness, and the common | ||
24 | -interface had to adapt for each individual scenario. In particular | ||
25 | -Noosfero did not yet have a responsive design; we engaged in its | ||
26 | -development and contributed towards that goal. | ||
27 | - | ||
28 | -After the initial release of the new SPB Portal in 2014, several | ||
29 | -validations activities were implemented in 2015 and 2016. The aim was to | ||
30 | -provide the most wanted features by casual users (such as public | ||
31 | -servants interested in downloads and documentation) immediately, while | ||
32 | -allowing more experienced users (such as developers) to easily drill down | ||
33 | -to the details. |
opensym2017/content/08-process.tex
@@ -1,181 +0,0 @@ | @@ -1,181 +0,0 @@ | ||
1 | -\section{Development Organization and Process} | ||
2 | -\label{sec:process} | ||
3 | - | ||
4 | -The SPB team was composed of a variety of professionals with different levels | ||
5 | -and skills, where most of them were undergraduate students with major in | ||
6 | -software engineering (from 4th semester or upper). Since the students could | ||
7 | -not dedicate many hours per week to the project, they always had the | ||
8 | -flexibility to negotiate their work schedule during the semester in | ||
9 | -order to not harm their classes and coursework. Their daily work routine | ||
10 | -in the project included programming and DevOps tasks. | ||
11 | - | ||
12 | -The development of SPB project required a vast experience and background that | ||
13 | -usually undergraduate students do not have yet. For this reason, a few senior | ||
14 | -developers have joined to the project to help with the more difficult issues | ||
15 | -and to transfer knowledge to the students. Their main task was to provide | ||
16 | -solutions for complex problems, in other words, they worked as developers. As | ||
17 | -these professionals are very skillful and the project could not fund a full | ||
18 | -time work for them, some of them worked partially on the project. In addition, | ||
19 | -they lived in a different states spread around Brazil which led much of the | ||
20 | -communication to be online. | ||
21 | - | ||
22 | -In short, our work process was based on open and collaborative software | ||
23 | -development practices. The development process was defined based on the | ||
24 | -adaptation of different agile and FOSS communities practices, highlighting the | ||
25 | -high degree of automation resulting from DevOps practices. Thus, the work | ||
26 | -process was executed in a cadenced and continuous way. | ||
27 | - | ||
28 | -Finally, the last group of actors of this project was composed of employees | ||
29 | -formally working for the Brazilian government, in the Ministry of Planning, | ||
30 | -Development, and Management (MP is the Brazilian acronym). All the project | ||
31 | -decisions, validations, and scope definitions were made by them. In this way we | ||
32 | -developed software product incrementally, with releases aligned to business | ||
33 | -strategic objectives. As one can see, the project had many different | ||
34 | -sort of stakeholders that had to be organized and synchronized. | ||
35 | - | ||
36 | -\subsection{Team organization} | ||
37 | - | ||
38 | -Approximately 70\% of the development teams were composed of software | ||
39 | -engineering undergraduate students from UnB and they worked physically | ||
40 | -in the same laboratory. Each student had their own schedule based on | ||
41 | -their classes, what complicates the implementation of pair programming. | ||
42 | -The senior developers tried to synchronize their schedule with the | ||
43 | -students on their sub-team. To cope with this environment, we had a few | ||
44 | -basic rules which guided the project organization: | ||
45 | - | ||
46 | -\begin{enumerate} | ||
47 | - \item Classes have high priority for undergraduate students; | ||
48 | - \item Work in pairs whenever possible (locally or remotely); | ||
49 | - \item There must be one morning or afternoon per week when | ||
50 | - \emph{everyone} should be together physically in the laboratory | ||
51 | - (except of course the remote team members); | ||
52 | - \item Every 3 to 4 months, the senior developers would fly in and work | ||
53 | - alongside the students for a few days. | ||
54 | -\end{enumerate} | ||
55 | - | ||
56 | -With the aforementioned rules we divided all the project into four | ||
57 | -different teams: Colab, Noosfero, Design, and DevOps. Each team had one | ||
58 | -coach responsible for reducing the communication problem with the other | ||
59 | -teams and help the members to organize themselves in the best way for | ||
60 | -everyone (always respecting their work time). The coach was one of the | ||
61 | -students working in some of the teams with the extra duty of registering | ||
62 | -the current tasks developed in the sprint and with the responsibility to | ||
63 | -talk with other teams. One important thing to notice is the mutability | ||
64 | -of the team and the coach, during the project many students changed | ||
65 | -between the teams to try different areas. | ||
66 | - | ||
67 | -One characteristic of the teams was the presence of (at least) one | ||
68 | -senior per team. This was essential, because hard decisions and complex | ||
69 | -problems were usually referred to them. This kept having to take | ||
70 | -complicated technical decisions from the coach tole, what encouraged | ||
71 | -students to be coaches. Lastly, the senior developers worked directly | ||
72 | -with the students, and this was important to give the undergraduate the | ||
73 | -opportunity to interact with a savvy professional in his area and to | ||
74 | -keep the knowledge flowing in the project. | ||
75 | - | ||
76 | -Finally, we had to add two last elements of the team organization, that | ||
77 | -was essential for the project harmony: the meta-coach and professors. | ||
78 | -The former was a software engineer recently graduated and which wanted | ||
79 | -to keep working on the project, the latter were professors that | ||
80 | -orchestrated all the interactions between all members of the project. | ||
81 | -The meta-coach usually worked in one specific team and had the extra | ||
82 | -task of knowing the current status of all teams. Professors and | ||
83 | -meta-coaches worked together to reduce the communication problem between | ||
84 | -all the teams. Lastly, all the paperwork tasks, such as reporting on the | ||
85 | -project progress to the Ministry, was handled by the professors. | ||
86 | - | ||
87 | -\subsection{Communication and management} | ||
88 | - | ||
89 | -Our team had many people working together, and most of the seniors worked in a | ||
90 | -different city remotely. Also, we tried to keep our work completely clear to | ||
91 | -the Brazilian government and citizens interested in following the project. To | ||
92 | -handle those cases, we used a set of tools to communication and other to manage | ||
93 | -the project. | ||
94 | - | ||
95 | -For communication between members in different places, we used: video | ||
96 | -conferencing with shared terminal tools, IRC, and mailing lists. For example, | ||
97 | -when one student had to work in pair with a senior, normally, they used video | ||
98 | -conferencing tool for talking and shared a terminal session (both typing and | ||
99 | -seeing the screen in real time). For questions and fast discussion, we used | ||
100 | -IRC. For general notification, we used the mailing lists. | ||
101 | - | ||
102 | -For managing the project we used the SPB Portal itself; first to validate it by | ||
103 | -ourselves, and also because it had all the required tools. We basically created | ||
104 | -one wiki page per release in the SPB Gitlab instance with a mapping between | ||
105 | -strategical, tactical, and operational views. In a practical point of view, one | ||
106 | -milestone per user history (feature), and one or more issues for addressing | ||
107 | -each feature. With this approach we achieved two important goals: keeping all | ||
108 | -the management as close possible to to the source code, and tracking every | ||
109 | -feature developed during the project. Our decision to | ||
110 | -use Wiki initially was empirical, but later reinforced by a research conducted | ||
111 | -by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, | ||
112 | -opensourcestyle}. | ||
113 | - | ||
114 | -\subsection{High-level project management and reporting} | ||
115 | - | ||
116 | -The Brazilian government used to work with software development in a | ||
117 | -very traditional way. They would frequently focus on documents and not | ||
118 | -on what was, in our opinion, what really matters: working software. This | ||
119 | -dissonance caused us a communication noise with MP, because they would | ||
120 | -often question our work style. It was especially hard to convince them | ||
121 | -to accept the idea of open scope and agile development, but after months | ||
122 | -of labor and showing results they stopped resisting. | ||
123 | - | ||
124 | -We defined some level of meeting granularity to avoid generating too | ||
125 | -much overhead to the developers. We had a strategical and validating | ||
126 | -meeting with MP (the former once in a month and the latter each 15th | ||
127 | -day), release plaining with the entire team (one per month), and finally | ||
128 | -a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a | ||
129 | -diagram that represents our meeting organization. | ||
130 | - | ||
131 | -\begin{figure}[hbt] | ||
132 | - \centering | ||
133 | - \includegraphics[width=\linewidth]{figures/meeting_flows.png} | ||
134 | - \caption{Meetings cycles.} | ||
135 | - \label{fig:meeting} | ||
136 | -\end{figure} | ||
137 | - | ||
138 | -In the strategical meeting we usually defined the priorities and new | ||
139 | -features with the Brazilian government. Normally the professors, the | ||
140 | -coach of each team, the meta-coach, and some employees of the MP would | ||
141 | -participate in this meeting. We usually discussed what the team already | ||
142 | -produced since our last meeting, and established the new features for | ||
143 | -the next release. Notice that just part of the team would join this | ||
144 | -meeting to avoid generating unnecessary overhead to the developers, but | ||
145 | -all the students interested to participate were allowed to join (many | ||
146 | -students wanted this experience during the project). | ||
147 | - | ||
148 | -After the strategical meeting with Brazilian government agents, we had a | ||
149 | -planning turn with all teams together. In this part, each team worked together | ||
150 | -to convert the MP wishes into smaller parts which were represented by the epics of | ||
151 | -the release. Each coach was responsible for conducting the planning, and after | ||
152 | -that record it on the project wiki (the wiki provided by Gitlab). With this | ||
153 | -epic, each 14th day the team have generated their sprint scheduler (with small | ||
154 | -achievements mapped to issues). | ||
155 | - | ||
156 | -To keep the Brazilian government always updated, we invited them to work | ||
157 | -with us to validate the new features in progress. Normally we had a | ||
158 | -meeting each 15th day. Basically, this was our work flow, we always kept | ||
159 | -everything extremely open to the MP (our way of working, and the one | ||
160 | -often used by open source projects) and to the team. | ||
161 | - | ||
162 | -To keep the track of all of those things we used the SPB itself, | ||
163 | -especially Gitlab. Basically, we had: | ||
164 | - | ||
165 | -\begin{enumerate} | ||
166 | - \item Project repository: We have one organization with many repositories; | ||
167 | - \item Milestones: Each milestone was used to register a release; | ||
168 | - \item Wiki: Each release has one page on wiki with the compilation of | ||
169 | - strategical meeting; | ||
170 | - \item Issues: Each sprint planning generated issues, which we associated to | ||
171 | - the specific milestone and updated the wiki with the issue number related | ||
172 | - with them. Finally each developer assigned the issue to itself. | ||
173 | -\end{enumerate} | ||
174 | - | ||
175 | -Notice that this workflow gave us and the Brazilian government agents full | ||
176 | -traceability from a high level view of each feature to the lowest level (code). | ||
177 | -It is important to highlight that we converge to this workflow after many | ||
178 | -experiments. For example, we used a tool named Redmine for organizing our tasks | ||
179 | -during the sprints, however, this tool revealed inefficient for our case since | ||
180 | -the government agents lost part of the project traceability. We realized that | ||
181 | -centralize all the work in the SPB portal is the best option for our case. |
@@ -0,0 +1,34 @@ | @@ -0,0 +1,34 @@ | ||
1 | +\section{User eXperience evolution} | ||
2 | +\label{sec:ux} | ||
3 | + | ||
4 | +The integration of collaborative environments goes beyond functional aspects. | ||
5 | +Offering the population an unified experience across these environments has | ||
6 | +been the key to encourage the use of the platform as it reduces the perception | ||
7 | +of complexity. Thus, the SPB Portal information architecture was redesigned | ||
8 | +to provide a transparent navigation and to reach users with different profiles. | ||
9 | +A process of harmonization has been employed on the interaction models of each | ||
10 | +tool to reduce the learning curve. At the same time, a new visual style was | ||
11 | +created to unify the navigation experience and to comply with the guidelines of | ||
12 | +the digital communication identity standard established by the Federal | ||
13 | +Government. | ||
14 | + | ||
15 | +With the increase in system features and the addition of new tools, the | ||
16 | +visual style has steadily evolved to keep the navigation unified. Moreover, | ||
17 | +tools from different backgrounds, which in many cases provide similar | ||
18 | +functionality, prompted the development of an unified interface. Some | ||
19 | +features, such as search and user profile editing were eliminated from | ||
20 | +the individual applications, and implemented centrally to ensure a | ||
21 | +consistent look and feel. | ||
22 | + | ||
23 | +Another challenge was responsive web design. The integrated applications | ||
24 | +had varying degrees of support for responsiveness, and the common | ||
25 | +interface had to adapt for each individual scenario. In particular | ||
26 | +Noosfero did not yet have a responsive design; we engaged in its | ||
27 | +development and contributed towards that goal. | ||
28 | + | ||
29 | +After the initial release of the new SPB Portal in 2014, several | ||
30 | +validations activities were implemented in 2015 and 2016. The aim was to | ||
31 | +provide the most wanted features by casual users (such as public | ||
32 | +servants interested in downloads and documentation) immediately, while | ||
33 | +allowing more experienced users (such as developers) to easily drill down | ||
34 | +to the details. |
opensym2017/content/09-contributions.tex
@@ -1,51 +0,0 @@ | @@ -1,51 +0,0 @@ | ||
1 | -\section{Contributions to the upstream communities} | ||
2 | -\label{sec:contributions} | ||
3 | - | ||
4 | -%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) | ||
5 | -%* Colab -> RevProxy | ||
6 | -%* Colab, atualização do python/django | ||
7 | -%* Contribuições para o GitLab (autenticação) | ||
8 | -%* Noosfero, atualização do Rails, preparação para federação, nova interface ... | ||
9 | -%* Coper, empacotamentos (obs), omniauth | ||
10 | - | ||
11 | - | ||
12 | -During the execution of this project we made several contributions to | ||
13 | -the upstream communities we interacted with. This occurred due to our | ||
14 | -development process aligned with those of the respective communities. | ||
15 | -During development, we would explicitly discuss the features and bug | ||
16 | -fixes that we were working on with the applicable upstream communities. | ||
17 | -This contributed to improve the developers technical solutions with | ||
18 | -expertise outside of our team, and make it easier for those changes to | ||
19 | -be accepted in the original projects. Having changes accepted upstream | ||
20 | -in turn makes our life easier as it minimizes the delta between our | ||
21 | -codebase and upstream's allowing us to upgrade and benefit from | ||
22 | -development work from others. | ||
23 | - | ||
24 | -In Colab, we helped upstream redesign the entirely architecture, | ||
25 | -enabling the development of plugins to integrate new tools. We also | ||
26 | -added a feature that allowed Colab to run asynchronous tasks, which was | ||
27 | -a major improvement for us since we were developing a complex system. A | ||
28 | -migration to the latest Django version was made (web framework used by | ||
29 | -Colab). Moreover, we worked on RevProxy (the more important dependency | ||
30 | -of Colab) to put it in a good shape, fixing many bugs. | ||
31 | - | ||
32 | -Gitlab was the tool that we made the least number of modifications. We | ||
33 | -contributed with some improvements related with configuration files and | ||
34 | -we developed a new plugin that enables user authentication in Gitlab | ||
35 | -through the REMOTE\_USER HTTP header. This plugin was needed because | ||
36 | -Colab uses this mechanism to manage the authentication. | ||
37 | - | ||
38 | -Noosfero was the tool that contemplated several functional requirements, | ||
39 | -therefore we made a large number of contributions with upstream. We helped to | ||
40 | -migrate to the latest Rails version (web framework used by Noosfero), enable | ||
41 | -the federation implementation (federation with other social networks), and | ||
42 | -decouple the interface and the back-end. | ||
43 | - | ||
44 | -We also made some contributions on the DevOps front. Some members of | ||
45 | -them team became maintainers of some python libraries that were used by | ||
46 | -our scripts to upload packages to OBS (Open Build Service). We developed | ||
47 | -a tool called copr-status to keep track of the different stages of the | ||
48 | -lifecycle of each of the individual packages we were working on. | ||
49 | - | ||
50 | -%TODO: Mezuro | ||
51 | - |
@@ -0,0 +1,181 @@ | @@ -0,0 +1,181 @@ | ||
1 | +\section{Development Organization and Process} | ||
2 | +\label{sec:process} | ||
3 | + | ||
4 | +The SPB team was composed of a variety of professionals with different levels | ||
5 | +and skills, where most of them were undergraduate students with major in | ||
6 | +software engineering (from 4th semester or upper). Since the students could | ||
7 | +not dedicate many hours per week to the project, they always had the | ||
8 | +flexibility to negotiate their work schedule during the semester in | ||
9 | +order to not harm their classes and coursework. Their daily work routine | ||
10 | +in the project included programming and DevOps tasks. | ||
11 | + | ||
12 | +The development of SPB project required a vast experience and background that | ||
13 | +usually undergraduate students do not have yet. For this reason, a few senior | ||
14 | +developers have joined to the project to help with the more difficult issues | ||
15 | +and to transfer knowledge to the students. Their main task was to provide | ||
16 | +solutions for complex problems, in other words, they worked as developers. As | ||
17 | +these professionals are very skillful and the project could not fund a full | ||
18 | +time work for them, some of them worked partially on the project. In addition, | ||
19 | +they lived in a different states spread around Brazil which led much of the | ||
20 | +communication to be online. | ||
21 | + | ||
22 | +In short, our work process was based on open and collaborative software | ||
23 | +development practices. The development process was defined based on the | ||
24 | +adaptation of different agile and FOSS communities practices, highlighting the | ||
25 | +high degree of automation resulting from DevOps practices. Thus, the work | ||
26 | +process was executed in a cadenced and continuous way. | ||
27 | + | ||
28 | +Finally, the last group of actors of this project was composed of employees | ||
29 | +formally working for the Brazilian government, in the Ministry of Planning, | ||
30 | +Development, and Management (MP is the Brazilian acronym). All the project | ||
31 | +decisions, validations, and scope definitions were made by them. In this way we | ||
32 | +developed software product incrementally, with releases aligned to business | ||
33 | +strategic objectives. As one can see, the project had many different | ||
34 | +sort of stakeholders that had to be organized and synchronized. | ||
35 | + | ||
36 | +\subsection{Team organization} | ||
37 | + | ||
38 | +Approximately 70\% of the development teams were composed of software | ||
39 | +engineering undergraduate students from UnB and they worked physically | ||
40 | +in the same laboratory. Each student had their own schedule based on | ||
41 | +their classes, what complicates the implementation of pair programming. | ||
42 | +The senior developers tried to synchronize their schedule with the | ||
43 | +students on their sub-team. To cope with this environment, we had a few | ||
44 | +basic rules which guided the project organization: | ||
45 | + | ||
46 | +\begin{enumerate} | ||
47 | + \item Classes have high priority for undergraduate students; | ||
48 | + \item Work in pairs whenever possible (locally or remotely); | ||
49 | + \item There must be one morning or afternoon per week when | ||
50 | + \emph{everyone} should be together physically in the laboratory | ||
51 | + (except of course the remote team members); | ||
52 | + \item Every 3 to 4 months, the senior developers would fly in and work | ||
53 | + alongside the students for a few days. | ||
54 | +\end{enumerate} | ||
55 | + | ||
56 | +With the aforementioned rules we divided all the project into four | ||
57 | +different teams: Colab, Noosfero, Design, and DevOps. Each team had one | ||
58 | +coach responsible for reducing the communication problem with the other | ||
59 | +teams and help the members to organize themselves in the best way for | ||
60 | +everyone (always respecting their work time). The coach was one of the | ||
61 | +students working in some of the teams with the extra duty of registering | ||
62 | +the current tasks developed in the sprint and with the responsibility to | ||
63 | +talk with other teams. One important thing to notice is the mutability | ||
64 | +of the team and the coach, during the project many students changed | ||
65 | +between the teams to try different areas. | ||
66 | + | ||
67 | +One characteristic of the teams was the presence of (at least) one | ||
68 | +senior per team. This was essential, because hard decisions and complex | ||
69 | +problems were usually referred to them. This kept having to take | ||
70 | +complicated technical decisions from the coach tole, what encouraged | ||
71 | +students to be coaches. Lastly, the senior developers worked directly | ||
72 | +with the students, and this was important to give the undergraduate the | ||
73 | +opportunity to interact with a savvy professional in his area and to | ||
74 | +keep the knowledge flowing in the project. | ||
75 | + | ||
76 | +Finally, we had to add two last elements of the team organization, that | ||
77 | +was essential for the project harmony: the meta-coach and professors. | ||
78 | +The former was a software engineer recently graduated and which wanted | ||
79 | +to keep working on the project, the latter were professors that | ||
80 | +orchestrated all the interactions between all members of the project. | ||
81 | +The meta-coach usually worked in one specific team and had the extra | ||
82 | +task of knowing the current status of all teams. Professors and | ||
83 | +meta-coaches worked together to reduce the communication problem between | ||
84 | +all the teams. Lastly, all the paperwork tasks, such as reporting on the | ||
85 | +project progress to the Ministry, was handled by the professors. | ||
86 | + | ||
87 | +\subsection{Communication and management} | ||
88 | + | ||
89 | +Our team had many people working together, and most of the seniors worked in a | ||
90 | +different city remotely. Also, we tried to keep our work completely clear to | ||
91 | +the Brazilian government and citizens interested in following the project. To | ||
92 | +handle those cases, we used a set of tools to communication and other to manage | ||
93 | +the project. | ||
94 | + | ||
95 | +For communication between members in different places, we used: video | ||
96 | +conferencing with shared terminal tools, IRC, and mailing lists. For example, | ||
97 | +when one student had to work in pair with a senior, normally, they used video | ||
98 | +conferencing tool for talking and shared a terminal session (both typing and | ||
99 | +seeing the screen in real time). For questions and fast discussion, we used | ||
100 | +IRC. For general notification, we used the mailing lists. | ||
101 | + | ||
102 | +For managing the project we used the SPB Portal itself; first to validate it by | ||
103 | +ourselves, and also because it had all the required tools. We basically created | ||
104 | +one wiki page per release in the SPB Gitlab instance with a mapping between | ||
105 | +strategical, tactical, and operational views. In a practical point of view, one | ||
106 | +milestone per user history (feature), and one or more issues for addressing | ||
107 | +each feature. With this approach we achieved two important goals: keeping all | ||
108 | +the management as close possible to to the source code, and tracking every | ||
109 | +feature developed during the project. Our decision to | ||
110 | +use Wiki initially was empirical, but later reinforced by a research conducted | ||
111 | +by Joseph Chao showing the advantage of using Wikis \cite{chao2007student, | ||
112 | +opensourcestyle}. | ||
113 | + | ||
114 | +\subsection{High-level project management and reporting} | ||
115 | + | ||
116 | +The Brazilian government used to work with software development in a | ||
117 | +very traditional way. They would frequently focus on documents and not | ||
118 | +on what was, in our opinion, what really matters: working software. This | ||
119 | +dissonance caused us a communication noise with MP, because they would | ||
120 | +often question our work style. It was especially hard to convince them | ||
121 | +to accept the idea of open scope and agile development, but after months | ||
122 | +of labor and showing results they stopped resisting. | ||
123 | + | ||
124 | +We defined some level of meeting granularity to avoid generating too | ||
125 | +much overhead to the developers. We had a strategical and validating | ||
126 | +meeting with MP (the former once in a month and the latter each 15th | ||
127 | +day), release plaining with the entire team (one per month), and finally | ||
128 | +a sprint planning (one each 15th day). Figure \ref{fig:meeting} is a | ||
129 | +diagram that represents our meeting organization. | ||
130 | + | ||
131 | +\begin{figure}[hbt] | ||
132 | + \centering | ||
133 | + \includegraphics[width=\linewidth]{figures/meeting_flows.png} | ||
134 | + \caption{Meetings cycles.} | ||
135 | + \label{fig:meeting} | ||
136 | +\end{figure} | ||
137 | + | ||
138 | +In the strategical meeting we usually defined the priorities and new | ||
139 | +features with the Brazilian government. Normally the professors, the | ||
140 | +coach of each team, the meta-coach, and some employees of the MP would | ||
141 | +participate in this meeting. We usually discussed what the team already | ||
142 | +produced since our last meeting, and established the new features for | ||
143 | +the next release. Notice that just part of the team would join this | ||
144 | +meeting to avoid generating unnecessary overhead to the developers, but | ||
145 | +all the students interested to participate were allowed to join (many | ||
146 | +students wanted this experience during the project). | ||
147 | + | ||
148 | +After the strategical meeting with Brazilian government agents, we had a | ||
149 | +planning turn with all teams together. In this part, each team worked together | ||
150 | +to convert the MP wishes into smaller parts which were represented by the epics of | ||
151 | +the release. Each coach was responsible for conducting the planning, and after | ||
152 | +that record it on the project wiki (the wiki provided by Gitlab). With this | ||
153 | +epic, each 14th day the team have generated their sprint scheduler (with small | ||
154 | +achievements mapped to issues). | ||
155 | + | ||
156 | +To keep the Brazilian government always updated, we invited them to work | ||
157 | +with us to validate the new features in progress. Normally we had a | ||
158 | +meeting each 15th day. Basically, this was our work flow, we always kept | ||
159 | +everything extremely open to the MP (our way of working, and the one | ||
160 | +often used by open source projects) and to the team. | ||
161 | + | ||
162 | +To keep the track of all of those things we used the SPB itself, | ||
163 | +especially Gitlab. Basically, we had: | ||
164 | + | ||
165 | +\begin{enumerate} | ||
166 | + \item Project repository: We have one organization with many repositories; | ||
167 | + \item Milestones: Each milestone was used to register a release; | ||
168 | + \item Wiki: Each release has one page on wiki with the compilation of | ||
169 | + strategical meeting; | ||
170 | + \item Issues: Each sprint planning generated issues, which we associated to | ||
171 | + the specific milestone and updated the wiki with the issue number related | ||
172 | + with them. Finally each developer assigned the issue to itself. | ||
173 | +\end{enumerate} | ||
174 | + | ||
175 | +Notice that this workflow gave us and the Brazilian government agents full | ||
176 | +traceability from a high level view of each feature to the lowest level (code). | ||
177 | +It is important to highlight that we converge to this workflow after many | ||
178 | +experiments. For example, we used a tool named Redmine for organizing our tasks | ||
179 | +during the sprints, however, this tool revealed inefficient for our case since | ||
180 | +the government agents lost part of the project traceability. We realized that | ||
181 | +centralize all the work in the SPB portal is the best option for our case. |
@@ -0,0 +1,51 @@ | @@ -0,0 +1,51 @@ | ||
1 | +\section{Contributions to the upstream communities} | ||
2 | +\label{sec:contributions} | ||
3 | + | ||
4 | +%- projeto feito do jeito certo com relação ao software livre (contribuições upstream etc) | ||
5 | +%* Colab -> RevProxy | ||
6 | +%* Colab, atualização do python/django | ||
7 | +%* Contribuições para o GitLab (autenticação) | ||
8 | +%* Noosfero, atualização do Rails, preparação para federação, nova interface ... | ||
9 | +%* Coper, empacotamentos (obs), omniauth | ||
10 | + | ||
11 | + | ||
12 | +During the execution of this project we made several contributions to | ||
13 | +the upstream communities we interacted with. This occurred due to our | ||
14 | +development process aligned with those of the respective communities. | ||
15 | +During development, we would explicitly discuss the features and bug | ||
16 | +fixes that we were working on with the applicable upstream communities. | ||
17 | +This contributed to improve the developers technical solutions with | ||
18 | +expertise outside of our team, and make it easier for those changes to | ||
19 | +be accepted in the original projects. Having changes accepted upstream | ||
20 | +in turn makes our life easier as it minimizes the delta between our | ||
21 | +codebase and upstream's allowing us to upgrade and benefit from | ||
22 | +development work from others. | ||
23 | + | ||
24 | +In Colab, we helped upstream redesign the entirely architecture, | ||
25 | +enabling the development of plugins to integrate new tools. We also | ||
26 | +added a feature that allowed Colab to run asynchronous tasks, which was | ||
27 | +a major improvement for us since we were developing a complex system. A | ||
28 | +migration to the latest Django version was made (web framework used by | ||
29 | +Colab). Moreover, we worked on RevProxy (the more important dependency | ||
30 | +of Colab) to put it in a good shape, fixing many bugs. | ||
31 | + | ||
32 | +Gitlab was the tool that we made the least number of modifications. We | ||
33 | +contributed with some improvements related with configuration files and | ||
34 | +we developed a new plugin that enables user authentication in Gitlab | ||
35 | +through the REMOTE\_USER HTTP header. This plugin was needed because | ||
36 | +Colab uses this mechanism to manage the authentication. | ||
37 | + | ||
38 | +Noosfero was the tool that contemplated several functional requirements, | ||
39 | +therefore we made a large number of contributions with upstream. We helped to | ||
40 | +migrate to the latest Rails version (web framework used by Noosfero), enable | ||
41 | +the federation implementation (federation with other social networks), and | ||
42 | +decouple the interface and the back-end. | ||
43 | + | ||
44 | +We also made some contributions on the DevOps front. Some members of | ||
45 | +them team became maintainers of some python libraries that were used by | ||
46 | +our scripts to upload packages to OBS (Open Build Service). We developed | ||
47 | +a tool called copr-status to keep track of the different stages of the | ||
48 | +lifecycle of each of the individual packages we were working on. | ||
49 | + | ||
50 | +%TODO: Mezuro | ||
51 | + |
opensym2017/content/10-lessons.tex
@@ -1,161 +0,0 @@ | @@ -1,161 +0,0 @@ | ||
1 | -\section{Lessons Learned} | ||
2 | -\label{sec:lessons} | ||
3 | - | ||
4 | -\textbf{How to involve students real-world projects, interacting with | ||
5 | -real customers.} | ||
6 | -% | ||
7 | -Our team was mainly composed of software engineering undergraduate | ||
8 | -student, who had the opportunity to interact with senior developers and | ||
9 | -designers inside the team, as well as with the team of technicians and | ||
10 | -managers from the Brazilian Government, and the management team from | ||
11 | -UnB. | ||
12 | -% | ||
13 | -They interacted with professionals that had diverse expertises, and were | ||
14 | -able to participate in all levels of the software development process. | ||
15 | -This contributed to a great learning opportunity, and for a majority of | ||
16 | -them this was their first professional experience. | ||
17 | - | ||
18 | -\textbf{The participation of experienced professionals is crucial to | ||
19 | -success of the project.} | ||
20 | -One important factor for the students was the composition of the teams | ||
21 | -with the participation of experienced professionals. | ||
22 | -% | ||
23 | -On the technical side, the senior developers and designers would handle | ||
24 | -the more difficult technical decisions, creating a work environment | ||
25 | -where the students could develop their skills in a didactic way without | ||
26 | -pressure. | ||
27 | -% | ||
28 | -On the management side, the active participation of professors -- who | ||
29 | -are, in the end, the ones responsible for the project -- is crucial to | ||
30 | -make sure student participation is conducted in a healthy way, and is an | ||
31 | -instance of leading by example. | ||
32 | - | ||
33 | -\textbf{A balanced relationship between academia and industry.} | ||
34 | -The experience of the SPB project led UnB to develop a work style which | ||
35 | -proved to be appropriate for an educational environment that brings | ||
36 | -academia and industry together. | ||
37 | -% | ||
38 | -The highest priority from the university's point of view is the | ||
39 | -students. Considering this, the activities of the project were never | ||
40 | -prioritized to the detriment of classes and other pedagogical | ||
41 | -activities. In summary, we had students working at different times, part | ||
42 | -time, remotely or locally, always respecting their individual | ||
43 | -conditions, but doing the work in a collective, collaborative and open | ||
44 | -way. | ||
45 | -% | ||
46 | -And even under a potentially adverse environment, the project delivered | ||
47 | -the desired solution with success. | ||
48 | -% | ||
49 | -At the end of the project, we noted that the skills developed by the | ||
50 | -students were at the state of art of the software engineering practice. | ||
51 | -After the project ended, we had team members successfully embracing | ||
52 | -opportunities in public, private, national and international | ||
53 | -organizations, in addition to those students that went into | ||
54 | -entrepreneurship and opened their own companies. | ||
55 | - | ||
56 | -\textbf{Managing different organizational cultures.} | ||
57 | -In the beginning of the project, the Brazilian Government stakeholders | ||
58 | -had certain expectations about the development of project that, let's | ||
59 | -say, didn't exactly match our work based on agile and FOSS practices. | ||
60 | -% | ||
61 | -We had to develop strategies that would support different these | ||
62 | -organizational cultures. As reported in Section \ref{sec:process}, the | ||
63 | -MP is organized in a functional hierarchical organizational structure, | ||
64 | -typically, a traditional development paradigm. Therefore, we had to | ||
65 | -create a translation process between our team and the MP managers who | ||
66 | -managed the project on their side assuming a traditional, waterfall | ||
67 | -process. | ||
68 | - | ||
69 | -\textbf{Manage higher-level and lower-level goals separately.} | ||
70 | -During the initial phase of the project the MP team would often bring | ||
71 | -strategic discussions to technical/operational meetings, where we were | ||
72 | -supposed to discuss practical, technical decisions. | ||
73 | -% | ||
74 | -This produced a highly complex communication and management environment, | ||
75 | -overloading the professors because they were supposed to be responsible | ||
76 | -for maintaining the strategic alignment of the MP synchronized with | ||
77 | -implementation plans of the development team, specially in light of the | ||
78 | -aforementioned culture mismatch. Mixing both concerns in the same | ||
79 | -discussions caused confusion on both sides. | ||
80 | -% | ||
81 | -Towards the middle and end of the project we were able to keep those | ||
82 | -concerns separated, what eased the work of everyone involved. | ||
83 | - | ||
84 | -\textbf{Living with ill-advised political decisions.} | ||
85 | -At the initial phases of the project, by political and personal | ||
86 | -motivation, the main stakeholders from the Brazilian government imposed | ||
87 | -the use of Colab to guide the development of the new SPB platform. Our | ||
88 | -team was totally against the idea because we already knew that Colab was | ||
89 | -a very experimental project and its adoption could dramatically increase | ||
90 | -the project complexity. Even though we provided technical reasons to | ||
91 | -not utilize Colab, MP was adamant and we had to manage this problem. As | ||
92 | -explained in section \ref{sec:architecture} we did massive changes to | ||
93 | -it, and at the end of the project we completely rewrote Colab and made | ||
94 | -it stable. It is important to notice that the MP compelled us to accept | ||
95 | -a technical issue based only on political interests, without considering | ||
96 | -all the resources that would be spent due to that decision. At the end | ||
97 | -of the project, we verified that Colab indeed consumed a vast amount of | ||
98 | -the budget and increased the project complexity. In the end of the | ||
99 | -project and after our analysis on the decision made by the MP, we | ||
100 | -understand that MP's managers are capable of ignoring technical reasons | ||
101 | -in favor to a political decision. | ||
102 | - | ||
103 | -\textbf{Consider sustainability from the beginning.} | ||
104 | -In the process of deploying the SPB platform in the MP structure, we had | ||
105 | -to interact with the MP technicians. We did several workshops, training | ||
106 | -and a meticulous documentation describing all the required procedures to | ||
107 | -update the platform, however, we realized that the MP technicians | ||
108 | -constantly avoid the responsibility. After noticing the aforementioned | ||
109 | -situation, we organized a specific DevOps that completely automated all | ||
110 | -the deployment procedure. We simplified all the platform deployment to a | ||
111 | -few steps: (1) initial configurations (just ssh configuration) and (2) | ||
112 | -the execution of simple commands to completely update the platform. By | ||
113 | -the end of the project, we observed that the MP technicians invariably | ||
114 | -still depended on our support to update the platform even with all the | ||
115 | -automation provided by us. We were sadly left with a feeling of | ||
116 | -uncertainty about the future of the platform after the project ended. In | ||
117 | -hindsight, we realize that the MP dedicated systems analysts and | ||
118 | -managers to the project, but not operations technicians. The later | ||
119 | -should have been more involved with the process so they could at least | ||
120 | -comfortable with managing the platform infrastructure. | ||
121 | - | ||
122 | -% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados | ||
123 | -%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa | ||
124 | -%% afirmação, embora eu e Paulo consigamos perceber isso. | ||
125 | - | ||
126 | -%=========== | ||
127 | -% Conclusion | ||
128 | -%=========== | ||
129 | - | ||
130 | -\textbf{Final remarks.} | ||
131 | -The portal is available at \url{softwarepublico.gov.br}. All | ||
132 | -documentation, including detailed architecture and operation manuals are | ||
133 | -also available\footnote{\url{https://softwarepublico.gov.br/doc/} | ||
134 | -(in Portuguese only at the moment)}). | ||
135 | -% | ||
136 | -All the integrated tools are FOSS and our contributions were published | ||
137 | -in open repositories, available on the SPB Portal itself. We also | ||
138 | -contributed these features back to the respective communities: that | ||
139 | -benefits those communities, as well as us since we can share future | ||
140 | -development and maintenance effort with other organizations that | ||
141 | -participate in their projects. | ||
142 | - | ||
143 | - | ||
144 | -%* utilização do projeto para formação de recursos humanos (alunos) | ||
145 | - | ||
146 | -%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate | ||
147 | - | ||
148 | -%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) | ||
149 | - | ||
150 | -%* 69 projetos marcados como SPB, de 81 no total na plataforma. | ||
151 | - | ||
152 | -%* 47\% é desenvolvido em PHP. | ||
153 | - | ||
154 | -% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). | ||
155 | - | ||
156 | -% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. | ||
157 | - | ||
158 | -% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. | ||
159 | - | ||
160 | -%* Debater economia de recursos em orgão públicos | ||
161 | - |
opensym2017/content/11-conclusion.tex
@@ -1,35 +0,0 @@ | @@ -1,35 +0,0 @@ | ||
1 | -\section{Conclusion} | ||
2 | -\label{sec:conclusion} | ||
3 | - | ||
4 | -In this paper we present and discuss issues experienced during a government- | ||
5 | -funded project, in partnership with University of Brasilia and University of | ||
6 | -São Paulo, to evolve the Brazilian Public Software portal. | ||
7 | - | ||
8 | -The contributions of this paper are twofold. First, we present how was | ||
9 | -developed an unprecedent platform, delivered to Brazilian government. This | ||
10 | -platform - developed by an heterogenous team of professors, master and | ||
11 | -undergraduate students, IT professionals and governmental managers - provides | ||
12 | -several modern features from the integration of more than 10 FLOSS systems. | ||
13 | - | ||
14 | -Second, the 30 months project which developed this platform results in an | ||
15 | -important case that it is possible to mitigate issues seen as conflicting to IT | ||
16 | -development environment and between industry and academy. We shown that, as | ||
17 | -long as the institution can provide a healthy and challenging environment to | ||
18 | -its students, its is possible to conciliate studies and professional training | ||
19 | -in universities. After the end of the project, some students successfully | ||
20 | -embraced opportunities in public and private sectos, within national borders | ||
21 | -and abroad. Some others went further and started their own companies. | ||
22 | -We also shown that, with some adaptations/"translation processes", was possible | ||
23 | -to conciliate agile methodologies and FOSS practices to develop software to | ||
24 | -governmental organizations with functional hierarchical structures that use | ||
25 | -traditional development paradigm. | ||
26 | - | ||
27 | -In the SPB project it was employed many of our beliefs about FLOSS and Agile | ||
28 | -practices. The team was engaged in creating a friendly environment for everyone | ||
29 | -involved in the project and in showing to government agents another way to | ||
30 | -interact with the FLOSS community and the university. With an open work style, | ||
31 | -the project has seeked to be transparent for the whole society. We belive it is | ||
32 | -still needed to analyze every data produced by the project and its impact on the | ||
33 | -students. For future work, we would conduce a \textit{post-mortem} analyse in the | ||
34 | -project, seeing that we have many open data to be explored and it was applied | ||
35 | -approachs which still need extra validation. |
@@ -0,0 +1,121 @@ | @@ -0,0 +1,121 @@ | ||
1 | +\section{Lessons Learned} | ||
2 | +\label{sec:lessons} | ||
3 | + | ||
4 | +\textbf{How to involve students real-world projects, interacting with | ||
5 | +real customers.} | ||
6 | +% | ||
7 | +Our team was mainly composed of software engineering undergraduate | ||
8 | +student, who had the opportunity to interact with senior developers and | ||
9 | +designers inside the team, as well as with the team of technicians and | ||
10 | +managers from the Brazilian Government, and the management team from | ||
11 | +UnB. | ||
12 | +% | ||
13 | +They interacted with professionals that had diverse expertises, and were | ||
14 | +able to participate in all levels of the software development process. | ||
15 | +This contributed to a great learning opportunity, and for a majority of | ||
16 | +them this was their first professional experience. | ||
17 | + | ||
18 | +\textbf{The participation of experienced professionals is crucial to | ||
19 | +success of the project.} | ||
20 | +One important factor for the students was the composition of the teams | ||
21 | +with the participation of experienced professionals. | ||
22 | +% | ||
23 | +On the technical side, the senior developers and designers would handle | ||
24 | +the more difficult technical decisions, creating a work environment | ||
25 | +where the students could develop their skills in a didactic way without | ||
26 | +pressure. | ||
27 | +% | ||
28 | +On the management side, the active participation of professors -- who | ||
29 | +are, in the end, the ones responsible for the project -- is crucial to | ||
30 | +make sure student participation is conducted in a healthy way, and is an | ||
31 | +instance of leading by example. | ||
32 | + | ||
33 | +\textbf{A balanced relationship between academia and industry.} | ||
34 | +The experience of the SPB project led UnB to develop a work style which | ||
35 | +proved to be appropriate for an educational environment that brings | ||
36 | +academia and industry together. | ||
37 | +% | ||
38 | +The highest priority from the university's point of view is the | ||
39 | +students. Considering this, the activities of the project were never | ||
40 | +prioritized to the detriment of classes and other pedagogical | ||
41 | +activities. In summary, we had students working at different times, part | ||
42 | +time, remotely or locally, always respecting their individual | ||
43 | +conditions, but doing the work in a collective, collaborative and open | ||
44 | +way. | ||
45 | +% | ||
46 | +And even under a potentially adverse environment, the project delivered | ||
47 | +the desired solution with success. | ||
48 | +% | ||
49 | +At the end of the project, we noted that the skills developed by the | ||
50 | +students were at the state of art of the software engineering practice. | ||
51 | +After the project ended, we had team members successfully embracing | ||
52 | +opportunities in public, private, national and international | ||
53 | +organizations, in addition to those students that went into | ||
54 | +entrepreneurship and opened their own companies. | ||
55 | + | ||
56 | +\textbf{Managing different organizational cultures.} | ||
57 | +In the beginning of the project, the Brazilian Government stakeholders | ||
58 | +had certain expectations about the development of project that, let's | ||
59 | +say, didn't exactly match our work based on agile and FOSS practices. | ||
60 | +% | ||
61 | +We had to develop strategies that would support different these | ||
62 | +organizational cultures. As reported in Section \ref{sec:process}, the | ||
63 | +MP is organized in a functional hierarchical organizational structure, | ||
64 | +typically, a traditional development paradigm. Therefore, we had to | ||
65 | +create a translation process between our team and the MP managers who | ||
66 | +managed the project on their side assuming a traditional, waterfall | ||
67 | +process. | ||
68 | + | ||
69 | +\textbf{Manage higher-level and lower-level goals separately.} | ||
70 | +During the initial phase of the project the MP team would often bring | ||
71 | +strategic discussions to technical/operational meetings, where we were | ||
72 | +supposed to discuss practical, technical decisions. | ||
73 | +% | ||
74 | +This produced a highly complex communication and management environment, | ||
75 | +overloading the professors because they were supposed to be responsible | ||
76 | +for maintaining the strategic alignment of the MP synchronized with | ||
77 | +implementation plans of the development team, specially in light of the | ||
78 | +aforementioned culture mismatch. Mixing both concerns in the same | ||
79 | +discussions caused confusion on both sides. | ||
80 | +% | ||
81 | +Towards the middle and end of the project we were able to keep those | ||
82 | +concerns separated, what eased the work of everyone involved. | ||
83 | + | ||
84 | +\textbf{Living with ill-advised political decisions.} | ||
85 | +At the initial phases of the project, by political and personal | ||
86 | +motivation, the main stakeholders from the Brazilian government imposed | ||
87 | +the use of Colab to guide the development of the new SPB platform. Our | ||
88 | +team was totally against the idea because we already knew that Colab was | ||
89 | +a very experimental project and its adoption could dramatically increase | ||
90 | +the project complexity. Even though we provided technical reasons to | ||
91 | +not utilize Colab, MP was adamant and we had to manage this problem. As | ||
92 | +explained in section \ref{sec:architecture} we did massive changes to | ||
93 | +it, and at the end of the project we completely rewrote Colab and made | ||
94 | +it stable. It is important to notice that the MP compelled us to accept | ||
95 | +a technical issue based only on political interests, without considering | ||
96 | +all the resources that would be spent due to that decision. At the end | ||
97 | +of the project, we verified that Colab indeed consumed a vast amount of | ||
98 | +the budget and increased the project complexity. In the end of the | ||
99 | +project and after our analysis on the decision made by the MP, we | ||
100 | +understand that MP's managers are capable of ignoring technical reasons | ||
101 | +in favor to a political decision. | ||
102 | + | ||
103 | +\textbf{Consider sustainability from the beginning.} | ||
104 | +In the process of deploying the SPB platform in the MP structure, we had | ||
105 | +to interact with the MP technicians. We did several workshops, training | ||
106 | +and a meticulous documentation describing all the required procedures to | ||
107 | +update the platform, however, we realized that the MP technicians | ||
108 | +constantly avoid the responsibility. After noticing the aforementioned | ||
109 | +situation, we organized a specific DevOps that completely automated all | ||
110 | +the deployment procedure. We simplified all the platform deployment to a | ||
111 | +few steps: (1) initial configurations (just ssh configuration) and (2) | ||
112 | +the execution of simple commands to completely update the platform. By | ||
113 | +the end of the project, we observed that the MP technicians invariably | ||
114 | +still depended on our support to update the platform even with all the | ||
115 | +automation provided by us. We were sadly left with a feeling of | ||
116 | +uncertainty about the future of the platform after the project ended. In | ||
117 | +hindsight, we realize that the MP dedicated systems analysts and | ||
118 | +managers to the project, but not operations technicians. The later | ||
119 | +should have been more involved with the process so they could at least | ||
120 | +comfortable with managing the platform infrastructure. | ||
121 | + |
@@ -0,0 +1,75 @@ | @@ -0,0 +1,75 @@ | ||
1 | +\section{Conclusion} | ||
2 | +\label{sec:conclusion} | ||
3 | + | ||
4 | +In this paper we present and discuss issues experienced during a government- | ||
5 | +funded project, in partnership with University of Brasilia and University of | ||
6 | +São Paulo, to evolve the Brazilian Public Software portal. | ||
7 | + | ||
8 | +The contributions of this paper are twofold. First, we present how was | ||
9 | +developed an unprecedent platform, delivered to Brazilian government. This | ||
10 | +platform - developed by an heterogenous team of professors, master and | ||
11 | +undergraduate students, IT professionals and governmental managers - provides | ||
12 | +several modern features from the integration of more than 10 FLOSS systems. | ||
13 | + | ||
14 | +Second, the 30 months project which developed this platform results in an | ||
15 | +important case that it is possible to mitigate issues seen as conflicting to IT | ||
16 | +development environment and between industry and academy. We shown that, as | ||
17 | +long as the institution can provide a healthy and challenging environment to | ||
18 | +its students, its is possible to conciliate studies and professional training | ||
19 | +in universities. After the end of the project, some students successfully | ||
20 | +embraced opportunities in public and private sectos, within national borders | ||
21 | +and abroad. Some others went further and started their own companies. | ||
22 | +We also shown that, with some adaptations/"translation processes", was possible | ||
23 | +to conciliate agile methodologies and FOSS practices to develop software to | ||
24 | +governmental organizations with functional hierarchical structures that use | ||
25 | +traditional development paradigm. | ||
26 | + | ||
27 | +In the SPB project it was employed many of our beliefs about FLOSS and Agile | ||
28 | +practices. The team was engaged in creating a friendly environment for everyone | ||
29 | +involved in the project and in showing to government agents another way to | ||
30 | +interact with the FLOSS community and the university. With an open work style, | ||
31 | +the project has seeked to be transparent for the whole society. We belive it is | ||
32 | +still needed to analyze every data produced by the project and its impact on the | ||
33 | +students. For future work, we would conduce a \textit{post-mortem} analyse in the | ||
34 | +project, seeing that we have many open data to be explored and it was applied | ||
35 | +approachs which still need extra validation. | ||
36 | + | ||
37 | +The portal is available at \url{softwarepublico.gov.br}. All | ||
38 | +documentation, including detailed architecture and operation manuals are | ||
39 | +also available\footnote{\url{https://softwarepublico.gov.br/doc/} | ||
40 | +(in Portuguese only at the moment)}). | ||
41 | +% | ||
42 | +All the integrated tools are FOSS and our contributions were published | ||
43 | +in open repositories, available on the SPB Portal itself. We also | ||
44 | +contributed these features back to the respective communities: that | ||
45 | +benefits those communities, as well as us since we can share future | ||
46 | +development and maintenance effort with other organizations that | ||
47 | +participate in their projects. | ||
48 | + | ||
49 | +%=========== | ||
50 | +% Conclusion | ||
51 | +%=========== | ||
52 | + | ||
53 | +% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados | ||
54 | +%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa | ||
55 | +%% afirmação, embora eu e Paulo consigamos perceber isso. | ||
56 | + | ||
57 | + | ||
58 | +%* utilização do projeto para formação de recursos humanos (alunos) | ||
59 | + | ||
60 | +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate | ||
61 | + | ||
62 | +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar) | ||
63 | + | ||
64 | +%* 69 projetos marcados como SPB, de 81 no total na plataforma. | ||
65 | + | ||
66 | +%* 47\% é desenvolvido em PHP. | ||
67 | + | ||
68 | +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket). | ||
69 | + | ||
70 | +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}. | ||
71 | + | ||
72 | +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos. | ||
73 | + | ||
74 | +%* Debater economia de recursos em orgão públicos | ||
75 | + |
opensym2017/spb.tex
@@ -171,14 +171,15 @@ | @@ -171,14 +171,15 @@ | ||
171 | \input{content/01-introduction} | 171 | \input{content/01-introduction} |
172 | \input{content/02-spb} | 172 | \input{content/02-spb} |
173 | \input{content/03-relatedworks} | 173 | \input{content/03-relatedworks} |
174 | -\input{content/04-requirements} | ||
175 | -\input{content/05-architecture} | ||
176 | -\input{content/06-features} | ||
177 | -\input{content/07-ux} | ||
178 | -\input{content/08-process} | ||
179 | -\input{content/09-contributions} | ||
180 | -\input{content/10-lessons} | ||
181 | -\input{content/11-conclusion} | 174 | +\input{content/04-researchdesign} |
175 | +\input{content/05-requirements} | ||
176 | +\input{content/06-architecture} | ||
177 | +\input{content/07-features} | ||
178 | +\input{content/08-ux} | ||
179 | +\input{content/09-process} | ||
180 | +\input{content/10-contributions} | ||
181 | +\input{content/11-lessons} | ||
182 | +\input{content/12-conclusion} | ||
182 | 183 | ||
183 | %------------------------------------------------------------------------------ | 184 | %------------------------------------------------------------------------------ |
184 | \bibliographystyle{SIGCHI-Reference-Format} | 185 | \bibliographystyle{SIGCHI-Reference-Format} |