Commit db4810a248c190845821d098c7e891a7418dd038

Authored by Paulo Meireles
1 parent 6dd0b976

[opensym] Reviewing the conclusion

opensym2017/content/01-introduction.tex
@@ -41,7 +41,7 @@ Approximately five years later from the first problems, a new platform for the @@ -41,7 +41,7 @@ Approximately five years later from the first problems, a new platform for the
41 SPB Portal was planned and funded. Between January 2014 and June 2016, the 41 SPB Portal was planned and funded. Between January 2014 and June 2016, the
42 University of Brasília (UnB) and the University of São Paulo (USP) in a 42 University of Brasília (UnB) and the University of São Paulo (USP) in a
43 partnership with the Brazilian Ministry of Planning, Budget, and Management 43 partnership with the Brazilian Ministry of Planning, Budget, and Management
44 -(MP) designed an integrated platform for collaborative software development 44 +designed an integrated platform for collaborative software development
45 \cite{bobr2003}, including social networking, mailing lists, version control 45 \cite{bobr2003}, including social networking, mailing lists, version control
46 system, and source code quality monitoring. To coordinate and develop this 46 system, and source code quality monitoring. To coordinate and develop this
47 project during 30 months, UnB was funded by a grant of 2,619,965.00 BRL (about 47 project during 30 months, UnB was funded by a grant of 2,619,965.00 BRL (about
opensym2017/content/04-researchdesign.tex
@@ -6,7 +6,7 @@ Portal by reporting the technical efforts carried out, our empirical work @@ -6,7 +6,7 @@ Portal by reporting the technical efforts carried out, our empirical work
6 process, and the lessons learned. The new SPB Portal project presented three 6 process, and the lessons learned. The new SPB Portal project presented three
7 main challenges, related to the open questions described below. 7 main challenges, related to the open questions described below.
8 8
9 -\textbf{Q1:} \textit{Which strategy could be used to integrate several existing 9 +\textbf{Q1.} \textit{Which strategy could be used to integrate several existing
10 FLOSS tools to promote a collaborative software development?} 10 FLOSS tools to promote a collaborative software development?}
11 % 11 %
12 Based on an extensive list of functional requirements defined by the Brazilian 12 Based on an extensive list of functional requirements defined by the Brazilian
@@ -17,7 +17,7 @@ were fully aware that we would need to improve those systems in order to @@ -17,7 +17,7 @@ were fully aware that we would need to improve those systems in order to
17 satisfy the remaining requirements. We were also convinced that it would be 17 satisfy the remaining requirements. We were also convinced that it would be
18 impossible to satisfy all of those requirements with a single tool. 18 impossible to satisfy all of those requirements with a single tool.
19 19
20 -\textbf{Q2:} \textit{How to involve students in real-world projects interacting 20 +\textbf{Q2.} \textit{How to involve students in real-world projects interacting
21 with real customers?} 21 with real customers?}
22 % 22 %
23 Our team was mainly composed of software engineering undergraduate students, 23 Our team was mainly composed of software engineering undergraduate students,
@@ -27,7 +27,7 @@ Government. For the majority of the students, this was a first professional @@ -27,7 +27,7 @@ Government. For the majority of the students, this was a first professional
27 experience. Even though, our development process defined a central role on 27 experience. Even though, our development process defined a central role on
28 students participation. 28 students participation.
29 29
30 -\textbf{Q3:} \textit{How to introduce typical FLOSS collaborative and agile practices in the governmental development process?} 30 +\textbf{Q3.} \textit{How to introduce typical FLOSS collaborative and agile practices in the governmental development process?}
31 % 31 %
32 The software development in Brazilian government is based on a very traditional 32 The software development in Brazilian government is based on a very traditional
33 way, frequently focusing documentation deliveries. We had to convince them to 33 way, frequently focusing documentation deliveries. We had to convince them to
opensym2017/content/06-architecture.tex
@@ -159,7 +159,7 @@ network created between them. @@ -159,7 +159,7 @@ network created between them.
159 159
160 \begin{figure*}[hbt] 160 \begin{figure*}[hbt]
161 \centering 161 \centering
162 - \includegraphics[width=\linewidth]{figures/arch3.png} 162 + \includegraphics[width=.85\linewidth]{figures/arch3.png}
163 \caption{Instanciation view of the SPB architecture.} 163 \caption{Instanciation view of the SPB architecture.}
164 \label{fig:architecture2} 164 \label{fig:architecture2}
165 \end{figure*} 165 \end{figure*}
opensym2017/content/08-process.tex
@@ -24,8 +24,8 @@ automation resulting from DevOps practices. Thus, the work process was executed @@ -24,8 +24,8 @@ automation resulting from DevOps practices. Thus, the work process was executed
24 in a cadenced and continuous way. 24 in a cadenced and continuous way.
25 25
26 Finally, the last group of actors of this project was composed of employees of 26 Finally, the last group of actors of this project was composed of employees of
27 -the Brazilian Ministry of Planning, Development, and Management (MP is the  
28 -Brazilian acronym). All the project decisions, validations, and scope 27 +the Brazilian Ministry of Planning, Development, and Management.
  28 +All the project decisions, validations, and scope
29 definitions were made by them. In this way, we incrementally developed a 29 definitions were made by them. In this way, we incrementally developed a
30 software product with releases aligned to strategic business objectives. As one 30 software product with releases aligned to strategic business objectives. As one
31 can see, the project had a wide range of different stakeholders that had to be 31 can see, the project had a wide range of different stakeholders that had to be
opensym2017/content/09-conclusion.tex
@@ -4,20 +4,20 @@ @@ -4,20 +4,20 @@
4 In this work, we presented and discussed issues experienced during a 4 In this work, we presented and discussed issues experienced during a
5 government-funded project, in partnership with the University of Brasilia and 5 government-funded project, in partnership with the University of Brasilia and
6 the University of São Paulo, to evolve the Brazilian Public Software Portal. 6 the University of São Paulo, to evolve the Brazilian Public Software Portal.
7 -Its contributions are twofold. First, we present the strategy used to develop 7 +Our contributions are twofold. First, we present the strategy used to develop
8 and to deliver an unprecedented platform to Brazilian government. Second, based 8 and to deliver an unprecedented platform to Brazilian government. Second, based
9 on the results of the SPB Portal project, we point out that it is possible to 9 on the results of the SPB Portal project, we point out that it is possible to
10 mitigate conflicts experienced in the development environment and to conciliate 10 mitigate conflicts experienced in the development environment and to conciliate
11 governmental and academy cultures. To summarize our main contributions, we 11 governmental and academy cultures. To summarize our main contributions, we
12 answered in this section the three open questions those guided this paper. 12 answered in this section the three open questions those guided this paper.
13 13
14 -\textbf{Which strategy could be used to integrate several existing  
15 -FLOSS tools to promote a collaborative software development?} 14 +\textbf{Which strategy could be used to integrate several existing FLOSS tools
  15 +to promote a collaborative software development?}
16 % 16 %
17 -The SPB portal integrates more than 10 FLOSS tools and provides several 17 +The SPB portal integrates more than ten FLOSS tools and provides several
18 features, such as social network, mailing list, version control, content 18 features, such as social network, mailing list, version control, content
19 management and source code quality monitoring. Concerned with the platform 19 management and source code quality monitoring. Concerned with the platform
20 -sustainability and maintainability, the aforementioned 10 FLOSS tools were 20 +sustainability and maintainability, the aforementioned FLOSS tools were
21 integrated with minimum differences from their official versions and the new 21 integrated with minimum differences from their official versions and the new
22 developed features were sent upstream to ensure an alignment between the portal 22 developed features were sent upstream to ensure an alignment between the portal
23 systems and their respective official versions. In the integration process, the 23 systems and their respective official versions. In the integration process, the
@@ -25,8 +25,8 @@ main softwares were identified, specific teams were formed to work with each @@ -25,8 +25,8 @@ main softwares were identified, specific teams were formed to work with each
25 one of them and each team was composed of students with different levels of 25 one of them and each team was composed of students with different levels of
26 skills and at least one senior professional. 26 skills and at least one senior professional.
27 27
28 -\textbf{How to involve students in real-world projects interacting with  
29 -real customers?} 28 +\textbf{How to involve students in real-world projects interacting with real
  29 +customers?}
30 % 30 %
31 In terms of mitigating conflicts, we tried to show that, as long as the 31 In terms of mitigating conflicts, we tried to show that, as long as the
32 university can provide a healthy and challenging environment to its students, 32 university can provide a healthy and challenging environment to its students,
@@ -42,29 +42,31 @@ professionals to solve development issues. @@ -42,29 +42,31 @@ professionals to solve development issues.
42 % 42 %
43 Among the students, we have defined coaches for each team and a meta-coach 43 Among the students, we have defined coaches for each team and a meta-coach
44 (coach of whole project). All coaches, together with professors, have 44 (coach of whole project). All coaches, together with professors, have
45 -intermediated the communication between client (Ministry of Planning of Brazil) 45 +intermediated the communication between client (the Brazilian Government)
46 and the rest of the group. 46 and the rest of the group.
47 After the end of the project, some students successfully 47 After the end of the project, some students successfully
48 embraced opportunities in public and private sectors, within national borders 48 embraced opportunities in public and private sectors, within national borders
49 and abroad. Some other students went further and started their own companies. 49 and abroad. Some other students went further and started their own companies.
50 50
51 -\textbf{How to introduce collaborative and agile  
52 -practices typical in FLOSS environments in the governmental development process?} 51 +\textbf{How to introduce typical FLOSS collaborative and agile practices in the
  52 +governmental development process?}
53 % 53 %
54 -With some adaptations, what we called the ``translation processes'', it is 54 +With some adaptations, it is
55 feasible to conciliate agile methodologies and FLOSS practices to develop 55 feasible to conciliate agile methodologies and FLOSS practices to develop
56 software to governmental organizations with functional hierarchical structures 56 software to governmental organizations with functional hierarchical structures
57 that use traditional development paradigm. 57 that use traditional development paradigm.
58 % 58 %
59 -Aiming at reducing client questions about workconclusion, a DevOps front was 59 +Aiming at reducing client questions about workconclusion, a DevOps team was
60 created to automate all deploy process and also to work in continuous 60 created to automate all deploy process and also to work in continuous
61 -delivery. The government was brought to our work environment and interacted 61 +delivery. The Government was brought to our work environment and interacted
62 with our management and communication tools. For the project success, we 62 with our management and communication tools. For the project success, we
63 focused on providing a friendly working environment as well as on showing to 63 focused on providing a friendly working environment as well as on showing to
64 governmental agents another way to interact with the FLOSS community and the 64 governmental agents another way to interact with the FLOSS community and the
65 university. 65 university.
66 66
67 -From the answers of our initial open questions, we can also highlighted six 67 +\subsection{Lessons Learned}
  68 +
  69 +From the answers of our initial open questions, we also can highlighted six
68 lessons learned to make easier to share our experience during the development 70 lessons learned to make easier to share our experience during the development
69 of the new SPB Portal. 71 of the new SPB Portal.
70 72
@@ -104,21 +106,22 @@ certain expectations about the project development that, let's say, did not @@ -104,21 +106,22 @@ certain expectations about the project development that, let's say, did not
104 exactly match our work style based on agile and FLOSS practices. 106 exactly match our work style based on agile and FLOSS practices.
105 % 107 %
106 We had to develop strategies that would support these different organizational 108 We had to develop strategies that would support these different organizational
107 -cultures. Therefore, we have created ``translation processes'' between our team  
108 -and the MP managers who managed the project on their side, assuming a 109 +cultures. Therefore, we have adapted the process between our team
  110 +and the Government managers who managed the project on their side, assuming a
109 traditional waterfall process. 111 traditional waterfall process.
110 112
111 \textbf{Managing higher-level and lower-level goals separately.} 113 \textbf{Managing higher-level and lower-level goals separately.}
112 % 114 %
113 -During the initial phase of the project, the MP team has brought strategic  
114 -discussions to technical/operational meetings that were supposed to be about  
115 -practical technical decisions. 115 +During the initial phase of the project, the Brazilian Government team has
  116 +brought strategic discussions to technical/operational meetings that were
  117 +supposed to be about practical technical decisions.
116 % 118 %
117 This produced a highly complex communication and management environment, 119 This produced a highly complex communication and management environment,
118 -overloading the professors because they were supposed for maintaining the MP  
119 -strategy synchronized with the implementation plans of the development team.  
120 -This was hard, especially because the aforementioned cultural mismatch. Mixing  
121 -both concerns in the same discussions caused confusion on both sides. 120 +overloading the professors because they were supposed for maintaining the
  121 +Government strategy synchronized with the implementation plans of the
  122 +development team. This was hard, especially because the aforementioned
  123 +cultural mismatch. Mixing both concerns in the same discussions caused
  124 +confusion on both sides.
122 % 125 %
123 From the middle of the project we were able to keep those concerns separated, 126 From the middle of the project we were able to keep those concerns separated,
124 what eased the work of everyone involved. 127 what eased the work of everyone involved.
@@ -126,40 +129,41 @@ what eased the work of everyone involved. @@ -126,40 +129,41 @@ what eased the work of everyone involved.
126 \textbf{Living with ill-advised political decisions.} 129 \textbf{Living with ill-advised political decisions.}
127 % 130 %
128 At the initial phases of the project, by political and personal motivation, the 131 At the initial phases of the project, by political and personal motivation, the
129 -main stakeholders from the Brazilian government imposed the use of Colab to 132 +main stakeholders from the Brazilian Government imposed the use of Colab to
130 guide the development of the new SPB platform. Our team was totally against the 133 guide the development of the new SPB platform. Our team was totally against the
131 idea because we already knew that Colab was a very experimental project and its 134 idea because we already knew that Colab was a very experimental project and its
132 -adoption could dramatically increase the project complexity. Even though we  
133 -provided technical reasons to not utilize Colab, the MP was adamant and we had  
134 -to manage this problem. We did massive changes to Colab, and at the end of the  
135 -project we have completely rewritten it to make it stable. It is important to  
136 -notice that the MP compelled us to accept a technical decision based only on  
137 -political interests, without considering all the resources that would be spent  
138 -due to that decision. At the end of the project, we verified that Colab indeed  
139 -consumed a vast amount of the budget and increased the project complexity. At  
140 -the end of the project and after our analysis on the decision made by the MP,  
141 -we understand that MP managers are capable of ignoring technical reasons in  
142 -favor of political decisions. 135 +adoption could dramatically increase the project complexity. Even though, we
  136 +provided technical reasons to not utilize Colab, the Government was adamant and
  137 +we had to manage this problem. We did massive changes to Colab, and at the end
  138 +of the project we have completely rewritten it to make it stable. It is
  139 +important to notice that the Government compelled us to accept a technical
  140 +decision based only on political interests, without considering all the
  141 +resources that would be spent due to that decision. At the end of the project,
  142 +we verified that Colab indeed consumed a vast amount of the budget and
  143 +increased the project complexity. After our analysis on the decision made by
  144 +the Government, we understand that some Brazilian Government managers are
  145 +capable of ignoring technical reasons in favor of political decisions.
143 146
144 \textbf{Consider sustainability from the beginning.} 147 \textbf{Consider sustainability from the beginning.}
145 % 148 %
146 -In the process of deploying the SPB platform in the MP infrastructure we had to  
147 -interact with the MP technicians. We did several workshops, training and a  
148 -meticulous documentation describing all the required procedures to update the  
149 -platform, however, we realized that the MP technicians constantly avoid the  
150 -responsibility. After noticing the aforementioned situation, we organized a  
151 -DevOps team that completely automated all the deployment procedure. We  
152 -simplified all the platform deployment to a few steps: (1) initial  
153 -configurations (just ssh configuration) and (2) the execution of simple  
154 -commands to completely update the platform. By the end of the project, we  
155 -observed that the MP technicians invariably still depended on our support to  
156 -update the platform even with all the automation provided by us. We were sadly  
157 -left with a feeling of uncertainty about the future of the platform after the  
158 -project ended. In hindsight, we realize that the MP dedicated system analysts  
159 -and managers to the project, but not operations technicians. The later should  
160 -have been more involved with the process so they could at least be comfortable  
161 -in managing the platform infrastructure.  
162 - 149 +In the process of deploying the SPB platform in the Brazilian Government
  150 +infrastructure we had to interact with the Government technicians. We did
  151 +several workshops, training and a meticulous documentation describing all the
  152 +required procedures to update the platform, however, we realized that they
  153 +constantly avoid the responsibility. After noticing the aforementioned
  154 +situation, we organized a DevOps team that completely automated all the
  155 +deployment procedure. We simplified all the platform deployment to a few steps:
  156 +(i) initial configurations (just ssh configuration) and (ii) the execution of
  157 +simple commands to completely update the platform. By the end of the project,
  158 +we observed that the Government technicians invariably still depended on our
  159 +support to update the platform even with all the automation provided by us. We
  160 +were sadly left with a feeling of uncertainty about the future of the platform
  161 +after the project ended. In hindsight, we realize that the Brazilian Government
  162 +dedicated system analysts and managers to the project, but not operations
  163 +technicians. The later should have been more involved with the process so they
  164 +could at least be comfortable in managing the platform infrastructure.
  165 +
  166 +\subsection{Final Remarks and Future Work}
163 167
164 Ultimately, the SPB portal is in 168 Ultimately, the SPB portal is in
165 production\footnote{\url{https://softwarepublico.gov.br}} and its full 169 production\footnote{\url{https://softwarepublico.gov.br}} and its full