[oss-2018] Results

Melissa Wen
1 parent 22ed8dc0
Showing 3 changed files with 253 additions and 12 deletions Show diff stats
icse2018/content/04-case.tex
icse2018/content/05-methods.tex
icse2018/content/06-results.tex
@@ -41,7 +41,7 @@ the project management and development process to harmonize the cultural and
 organizational differences of the institutions involved. Our analysis was guided
 by the following research questions:
-\textbf{Q1.} {How to combine different teams with different management processes
+\textbf{RQ1.} {How to combine different teams with different management processes
 in a government-academia collaboration project?}
 In this first moment, we describe what changes in the management model and the
@@ -58,7 +58,7 @@ approximate the mind-set and culture of teams but also to delimitate the
 interactions between different roles and responsibilities. Evaluating this
 synergy generates the second research question:
-\textbf{Q2.} \textit{Which boundaries should be established between government
+\textbf{RQ2.} \textit{Which boundaries should be established between government
 and academia teams in collaboration interactions?}
 We highlight positive and negative effects of boundaries created among project
@@ -2,8 +2,14 @@
 \label{sec:researchdesign}
 To answer the two research questions presented in the previous section, we
-conducted after-project surveys divided into three target groups of project
-participants:
+designed an interview and two questionnaires with quantitative and
+qualitative questions addressed to project members. We also collect data from
+tools that supported the project management activities.
+
+\subsection{Surveys}
+
+We conducted after-project surveys divided into three target groups of
+project participants:
 \begin{enumerate}
  \item \textit{MPOG Staff:} two government-side employees who have acted
@@ -13,8 +19,8 @@ of 2 hours with 28 open questions divided by subject: Professional profile;
 Organization, communication and development methodologies in the context of
 government and project; Satisfaction with the developed platform; Lessons
 learned.
- \item \textit{UnB students-developers:} 42 undergraduate students who
-participated in any time of the project receiving scholarship. A
+ \item \textit{UnB undegraduated students:} 42 undergraduate students who
+participated in any time of the project as developer and received scholarship. A
 questionnaire with 45 closed and six open questions was sent through emails using
 the Google Forms platform. The topics covered were: Organization, communication
 and development activities between the respondents and the different groups of
@@ -28,6 +34,8 @@ and distinct groups of the project; Development process; Experience with Free
 Software. All eight recipients answered the questions.
 \end{enumerate}
+\subsection{Data Collection}
+
 In a second round, we also collect post-mortem data from Redmine
 (outside the SPB portal), Gitlab and Mailman (inside the SPB portal) - tools
 used for management, communication and code versioning during the 30-month
@@ -40,9 +48,4 @@ interactions between the government and academia teams, and, in terms of
 development complexity, the platform size and quantity of software releases
 delivered.
-
-%Ambas possuem nível superior na área de TI e são funcionárias do governo a mais de 7 anos. Apenas uma delas segue atuando no mesmo ministério.
-%Para o segundo grupo, um formulário online foi enviado aos 42 alunos e ex-alunos do curso de engenharia de software da UnB que participaram do projeto como bolsistas. A idade média dos 37 respondentes é de 25 anos e 91,9\% deles são do sexo masculino. Atualmente, 35,1\% seguem na universidade como alunos de graduação ou pós-graduação, 18,9\% atuam como desenvolvedor em empresa de pequeno porte e 18,9\% em empresas de médio ou grande porte, 10,8\% são empreendedores, 8,1\% estão desempregados e os demais atuam como professor ou funcionário público.
-%O terceiro grupo é formado por 8 profissionais do mercado de TI ou alunos de mestrado (no período do projeto). Todos responderam ao formulário online enviado. As idades média é de 32 anos e 87,5\% são do sexo masculino. Eles possuem em média 11 anos de experiência no mercado de TI, e atualmente 25\% são estudantes de mestrado, 25\% empreendedores, 37,5\% são desenvolvedores freelancer e 62,5\% dos respondentes são funcionários de empresa. Já trabalharam em média em 5 empresas e participaram de 4 a 80 projetos. Eles participaram do projeto de colaboração estudado entre 7 a 24 meses.
-
 % And finally, we analized Colab code before and after the project to evaluate how much effort was spent to use this software as a component of the platform.
 \section{Results}
 \label{sec:results}
-In this section, we present the results obtained through interviews and questionnaries with project participants, analysis of the communication and project management tools, and analysis of the code developed during the project for the softwares that integrate the platform.
+
+\subsection{Respondents profile}
+
+\subsubsection{MPOG Staff}
+
+The two analysts interviewed are more than 30 years old and have been government
+employees for more than 7 years. Only one of them continues working in the same
+ministry. Both reported that the collaborative project studied was their first
+experience in collaborative projects between government and academia.
+
+\subsubsection{UnB undergraduated students}
+
+The average age of the 37 respondents is 25 years old and 91.9\% of them are male.
+Currently, 35.1\% continue at university as undergraduate or graduate students,
+18.9\% work as developer in a small company and 18.9\% in medium or large
+companies, 10.8\% are entrepreneurs, 8.1\% are unemployed and the others work as
+teachers or civil servants.
+
+\subsubsection{Senior Developers}
+The average age is 32 years old and 87.5\% are male. They have an average of 11
+years of experience in the IT market, and currently 62.5\% of respondents are
+company employees, 37.5\% are freelance developers, 25\% are master's degree
+students and 25\% entrepreneurs. They have worked on average in 5 companies and
+participated in 4 to 80 projects. They participated in the collaborative project
+studied between 7 to 24 months.
+
+\subsection{RQ1. Practices to well combine different teams with different
+management processes in a government-academia collaboration}
+
+The case study was analyzed and divided into two phases according to the project
+management model. In the second phase (after one year(?) of execution), several
+practices have been applied to harmonize the cultural and organizational
+divergences of the institutions involved. At the end of the project, an empirical
+model of management and development process was created by aligning experiences
+from the FLOSS universe, academic research and bureaucracies needed by the
+government. In this section, we present by context the practices adopted in this
+second phase and show the benefits generated by its deployment.
+
+\subsubsection{Project management and communication on the developing platform
+itself}
+
+\begin{itemize}
+  \item \textit{Migration from Redmine management to Gitlab (one of the
+integrated software):} Wiki feature was used for meeting logging, defining
+goals, sprint planning, and documentation of deployment processes and
+administration resources guide. Issue tracker was used for discussing
+requirements, monitoring the features under development, registrating changes,
+and validating functionalities delivered.
+  \item \textit{Using the project mailing list provided by Mailman's platform:}
+Scheduling meetings and defining schedule. Through the list, brainstorms and
+collaborative definition of requirements also happened.
+\end{itemize}
+
+Data from Gitlab shows 775 issues were opened and 4658 comments were made in the
+repository that versioned the platform (without considering the software
+repositories that integrated the platform) within the SPB platform. The issues
+were created by 59 different authors (8 MPOG representatives), and commented by
+64 different users (9 MPOG users). Of the 84 created issues with greater
+interaction (number of comments >= 10), ~43\% are authored by one of MPOG
+representatives. Students said the main communication ways used to interacted
+with software communities were: IRC (86.5\%); Mailing list (73\%); Issues
+(67.6\%). The main means of communication between senior developers and
+students was IRC (100\%), Mailing list (100\%), Videoconference (100\%) and
+Issue Tracker (62.5\%). Senior developers and MPOG staff interected mostly via
+Mailing List (87.5\%) and Issue tracker (50\%). One of the interviewed MPOG
+staff ratifies the main interaction spaces between teams: "We interacted a lot
+with mailing lists and chatting, sometimes with gitlab within the line of
+development."
+%
+\textbf{Benefits}
+
+\begin{itemize}
+\item \textit{Confidence in developed code} 
+\subitem One of the gov-side interviewees said "Everything was validated, we
+tested the features and the project was developed inside the platform, so that
+the feature was validated in the development of the software itself. Our team at
+MPOG, used the development and communication tools of the platform itself. From
+the moment we installed it, and began to use it for development, this validation
+was constant. We felt confident about the features"
+\item \textit{Transparency and efficiency in communication}
+\subitem Undegraduate student response: "We always had an open channel for
+conversations / interactions, whether it was the project issues, the IRC
+channel, via Whatsapp, etc."
+\subitem MPOG: "I think the communication was excellent, it was comprehensive,
+instantaneous and effective. We asked a question, a question, and very quickly
+it was answered. Communication goes far beyond that, you communicate to everyone
+in the project everything that was happening. We did not have issue related to
+emails, use the mailing list more and avoid e-mails, it helped a lot because it
+was all public and did not pollute our mailbox. You wanted to know something,
+could go there and look at what was happening. We also had instant chatting,
+where the team stayed online, working connected. We had a quick access to the
+team. This makes all the difference in a project."
+\item \textit{Monitoring of interactions among students, senior developers and
+public servants by coordinators}
+\subitem MPOG: "The leader himself informed us who was doing something, then we
+talked directly to that student [..] But this interaction was not very frequent,
+sometimes we would ask something on the list and the coordinator would answer
+first."
+\item \textit{Increase interaction between development team and contract
+management team}
+\subitem MPOG: "There was a lot of evolution, a lot of communication via Gitlab"
+\item \textit{Organically documentation and records generation}
+\subitem MPOG: "For me it was a lot of learning, there is a lot of things
+documented in the e-mails and also there in the portal itself of what happened
+in the project. At any moment we can go there and see how it worked, how the
+person did, and manages to salvage those good points."
+\end{itemize}
+
+\subsubsection{Continuos Delivery}
+
+\begin{itemize}
+ \item Creating DevOps Team
+ \item Defining continuous delivery pipeline
+ \item DevOps team periodically going to the ministry to help deploy each version
+\end{itemize}
+
+\textbf{Benefits}
+
+\begin{itemize}
+\item \textit{Increase government confidence for collaborative projects with the
+university}
+\subitem MPOG: "At first the government staff had a bias that universities did
+not deliver and we overcame that bias in the course of the project. We deliver a
+lot and with quality. Today, I think that if we had paid the same amount for a
+company, it would not have done what was delivered and with the quality that was
+delivered with the price that was paid."
+\item \textit{Motivate teams}
+\subitem {81.1\%} of students think new versions released in production motivated
+them during the project
+\subitem {75\%} of senior developers think new versions released in production
+motivated them during the project
+\subitem {81\%} of students think the presence of a specific DevOps team was
+necessary for the project
+\item \textit{Transfer of knowledge about DevOps and Continuous Deliveries from
+the academic team to the government infrastructure team}
+\subitem MPOG: "I only noticed positive aspects in the delivery. I think in the
+interaction, we had a lot of support to be able to deploy. From the time that
+the version was mature, which had already been tested in the UnB test
+environment and was ready to be put into production, we had a great agility to
+release in production. Then in the course of the project we realized that the
+infrastructure team [of MPOG] started to trust the UnB team a lot. Because, for
+you to put software in production in government, there is a whole process
+behind. The government has much of this security issue."
+\subitem MPOG: "If there was anything stopping the business from working, the
+software working inside, we would ask the seniors for support so we could
+investigate that, and the infrastructure team was also instructed to prioritize
+it. So when it came to an impasse, the teams were all together, both from within
+MPOG as well as senior developers and other UnB developers to unlock, to find
+the problem."
+\item \textit{Align the university and government teams pace in the execution of
+the activities}
+\subitem MPOG: "In the beginning, infrastructure personnel were not accustomed
+to deliveries so fast. They had to adapt to this pace. The portal of the SPB
+before the project was not there [in the MPOG infrastructure], it was in another
+place, they did not have that dynamics there. But what they asked for UnB (some
+configuration, installation manual, how to install everything inside) was
+requested and delivered."
+\item \textit{Improve translation from one development process to the other}
+\subitem MPOG: "We had an overview at the strategic level, but when we went down
+to the level of functionality we had this difficulty to do the planning of the
+release every four months. But in the end, I think this has not been a problem,
+because a project you are delivering, the results are going to production, the
+code is quality, the team is qualified/capable and the project is doing well, it
+does not impact as much in practice, because the result is being delivered.
+\end{itemize}
+
+\subsubsection{Organization of the project in teams for each front, with a
+undergraduate student as coach and at least one senior developer}
+
+\begin{itemize}
+  \item \textit{Four fronts: Colab, Noosfero, DevOps and Front-End/UX}
+  \item \textit{Definition of the role of team coaches and meta-coach, selected from undergraduate students group}
+  \item \textit{Hiring professionals from the IT market for face-to-face or remote work, specialists in the software components}
+\end{itemize}
+
+\textbf{Benefits}
+
+\begin{itemize}
+\item \textit{Help to conciliate development processes and decision-making}
+\subitem {62,5\%} of senior developers believe they have helped MPOG staff to more clearly express their requests
+\subitem {87,5\%} of seniors agreed with the project development process. For 37.5\% this process was little similar to their previous experiences, for the others there was a certain similarity.
+\subitem {62,5\%} of seniors did not understand MPOG's project management process. {50\%} of them believe their project productivity was affected by MPOG's project management process.
+\subitem Senior Dev: "I think my main contribution was to have balanced the relations between the MPOG staff and the UnB team"
+\subitem Senior Dev: "When I entered the project, the client had a disproportionate view of how to make explicit the requirements. They were still talking about use cases and were extremely concerned about validation processes and acceptance of these documents."
+\subitem MPOG: "You had the reviewers, who were the original developers of the software, that gave you confidence and confidence in the code."
+%
+\item \textit{Create support and reference points for students, senior developers, and government staff}
+\subitem {89.1\%} of students believe that the presence of the leader was essential to the running of Sprint
+\subitem {87.5\%} of seniors believe that the presence of team leaders was essential for their interaction with the team
+\subitem MPOG: "It interacted more with the project coordinator and team coaches (noosfero, colab, visual identity). Interacted with coaches by mailing list, hangouts The reason was usually to elucidate requirements, to ask questions about requirements, to understand some functionality. "
+\subitem MPOG: "There was interaction with the other [non-coaches] because they also participated in the bi-weekly meetings (sprints), but it was more with coaches."
+\subitem MPOG: "Access to coaches was faster, because we were in much more interaction with leaders than with senior developers. Sometimes the coaches brought the question to the senior developers."
+%
+\item \textit{Transfer of knowledge from industry and FLOSS community to both academia and government}
+\subitem {62.5\%} of senior developers believe that they have collaborated in the relationship between the management and development processes of the two institutions
+\subitem {100\%} of the students we interviewed believe that working with senior developers was important during the project
+\subitem {91.\%} of students also believe that working with seniors was important for learning
+\subitem {75\%} of senior developers believe that 'Working in pairs with a senior' and 62.5% who 'Participate in joint review tasks' were the tasks with the involvement of them that most contributed to the evolution of students in the project.
+\subitem {75\%} of senior developers believe that in guiding a student, this knowledge was widespread among the other students on the team.
+\subitem MPOG: "On the side of UnB, what we perceived so strongly was that the project took a very big leap when the original developers of the software (the official software development) were hired in the case of Noosfero and Colab [..] Because they had a guide on how to develop things in the best way and were able to solve non-trivial problems and quickly "
+\end{itemize}
+
+%* Filtrar a comunicação por níveis de maturidade/experiência e responsabilidades
+%MPOG: "Eu acho que esses pontos de conflito eram muito mais fáceis de lidar com a equipe do que com a própria coordenação. [..] Eu acho que tem uma diferença também de papel tem uma diferença de postura. Eu acho que a relação com a equipe, embora ela fosse saudável, eu acho que a equipe não tinha tanta autonomia quanto à coordenação tinha. Então talvez fosse mais difícil com a coordenação e não com a equipe, porque a equipe ela sabia o limite dela e a partir dali ela não agia mais, ela já convocava a coordenação para lidar (a gerência)."
+
+\subsubsection{Bringing the government staff directly responsible for the project together with development team}
+\begin{itemize}
+\item Biweekly meetings (planning and sprint review) in the development lab with the presence of government staff, team coaches and senior developers
+\item Discuss features under development directly on Gitlab Issue Tracker
+\item Only strategic decisions or bureaucratic issues involve the directors/secretaries
+\end{itemize}
+
+\textbf{Benefits}
+
+\begin{itemize}
+\item \textit{Reduce communication misunderstood}
+\subitem MPOG: "That's when the project started, people [MPOG staff] did not participate in anything. The communication process was horrible."; "The [MPOG] coordinator did not help, he would say something and UnB would talk to another at the meeting and it was the biggest mess." About the direct dialogue between the academic team and MPOG staff (without the involvement of coordinators and / or directors) , she said "That's where things really started to move, that the communication of the project began to improve."
+%
+\item \textit{Empathy between members on both sides}
+\subitem {72.9\%} of students believe that interacting with MPOG staff was important during the project
+\subitem Only 27\% of the students interviewed said they did not feel like attending meetings with MPOG employees
+\subitem MPOG: "You know people in person and it makes such a big difference because it causes empathy. You know what the person is going through on their side and she knows what we're going through on our side. So the next time you have a non-personal interaction (by mail, by list ...) I think it even facilitates, improves communication. You already know who that person is, it's not just a name. "
+%
+\item \textit{Develop requirements closer to the expectations of both sides}
+\subitem {81.1 \%} of students believe that the participation of MPOG staff in planning and closing sprints was important for the development of the project
+\subitem {75.6 \%} of students believe that writing the requirements together with the MPOG staff was very important
+\subitem Undergrad student: "Joint planning and timely meetings were very important for understanding the needs of MPOG, and the interaction via SPB tools helped validate the tool as a development platform"
+\subitem Undergrad student: "Often they did not know what they really wanted, and they caused some delays in the development of sprints"
+\subitem Undergrad student: "A relationship of constant attempt to balance and negotiate. The client does not always know the impacts of their requests"
+\subitem MPOG: "I believe it was very positive, we also liked to go there, to interact with the team. I think it brought more unity, more integration into the project, because we went there, where people were working and they show what was done. I think they also liked to receive our feedback about what had been done by them.This interaction did not just made with the coordinator. I found it very important and very positive it. "
+%
+\item \textit{Improve understanding of collaborative development by MPOG staff}
+\subitem Undergrad student: "In the beginning the demands of MPOG were very 'orders from above', but according to the progress of the project, they understood better our work philosophy and became more open"
+\subitem MPOG: "During development we realized that the team that was developing also felt like the owner of the project felt involved not only a mere executor of an order. It was not a client relationship, it was a partnership relationship, so there was a lot of team suggestions to be put into the project. Sometimes these were put in for us to decide and sometimes not."
+\subitem MPOG: "I think it was easy, I think the team was aligned. In addition to being aligned, these items that, for example, were not priorities and became priorities, were, in a sense, brought with some arguments from the team. So the team was able to argue and succeed in showing that it was important, that it needed to be prioritized, and I think the team was able to present the arguments well for some of the priorities that happened during the process."
+%
+\item \textit{Align the pace of both sides to execute activities}
+\subitem MPOG: "When we went there, I knew people and made that interaction more frequent, we also felt encouraged to validate faster and give faster feedback to the teams so they would not wait there. I knew they were waiting for our feedback and we were struggling to do it fast, because we ended a sprint and start another and not stop. We gave that feedback fast and they also gave quick feedback for any questions when they encountered a problem. That gave the project agility, things flowed faster and better. "
+\end{itemize}
	@@ -41,7 +41,7 @@ the project management and development process to harmonize the cultural and		@@ -41,7 +41,7 @@ the project management and development process to harmonize the cultural and
41	organizational differences of the institutions involved. Our analysis was guided	41	organizational differences of the institutions involved. Our analysis was guided
42	by the following research questions:	42	by the following research questions:
43		43
44	-\textbf{Q1.} {How to combine different teams with different management processes	44	+\textbf{RQ1.} {How to combine different teams with different management processes
45	in a government-academia collaboration project?}	45	in a government-academia collaboration project?}
46		46
47	In this first moment, we describe what changes in the management model and the	47	In this first moment, we describe what changes in the management model and the
	@@ -58,7 +58,7 @@ approximate the mind-set and culture of teams but also to delimitate the		@@ -58,7 +58,7 @@ approximate the mind-set and culture of teams but also to delimitate the
58	interactions between different roles and responsibilities. Evaluating this	58	interactions between different roles and responsibilities. Evaluating this
59	synergy generates the second research question:	59	synergy generates the second research question:
60		60
61	-\textbf{Q2.} \textit{Which boundaries should be established between government	61	+\textbf{RQ2.} \textit{Which boundaries should be established between government
62	and academia teams in collaboration interactions?}	62	and academia teams in collaboration interactions?}
63		63
64	We highlight positive and negative effects of boundaries created among project	64	We highlight positive and negative effects of boundaries created among project
1	\section{Results}	1	\section{Results}
2	\label{sec:results}	2	\label{sec:results}
3	-In this section, we present the results obtained through interviews and questionnaries with project participants, analysis of the communication and project management tools, and analysis of the code developed during the project for the softwares that integrate the platform.	3	+
		4	+\subsection{Respondents profile}
		5	+
		6	+\subsubsection{MPOG Staff}
		7	+
		8	+The two analysts interviewed are more than 30 years old and have been government
		9	+employees for more than 7 years. Only one of them continues working in the same
		10	+ministry. Both reported that the collaborative project studied was their first
		11	+experience in collaborative projects between government and academia.
		12	+
		13	+\subsubsection{UnB undergraduated students}
		14	+
		15	+The average age of the 37 respondents is 25 years old and 91.9\% of them are male.
		16	+Currently, 35.1\% continue at university as undergraduate or graduate students,
		17	+18.9\% work as developer in a small company and 18.9\% in medium or large
		18	+companies, 10.8\% are entrepreneurs, 8.1\% are unemployed and the others work as
		19	+teachers or civil servants.
		20	+
		21	+\subsubsection{Senior Developers}
		22	+The average age is 32 years old and 87.5\% are male. They have an average of 11
		23	+years of experience in the IT market, and currently 62.5\% of respondents are
		24	+company employees, 37.5\% are freelance developers, 25\% are master's degree
		25	+students and 25\% entrepreneurs. They have worked on average in 5 companies and
		26	+participated in 4 to 80 projects. They participated in the collaborative project
		27	+studied between 7 to 24 months.
		28	+
		29	+\subsection{RQ1. Practices to well combine different teams with different
		30	+management processes in a government-academia collaboration}
		31	+
		32	+The case study was analyzed and divided into two phases according to the project
		33	+management model. In the second phase (after one year(?) of execution), several
		34	+practices have been applied to harmonize the cultural and organizational
		35	+divergences of the institutions involved. At the end of the project, an empirical
		36	+model of management and development process was created by aligning experiences
		37	+from the FLOSS universe, academic research and bureaucracies needed by the
		38	+government. In this section, we present by context the practices adopted in this
		39	+second phase and show the benefits generated by its deployment.
		40	+
		41	+\subsubsection{Project management and communication on the developing platform
		42	+itself}
		43	+
		44	+\begin{itemize}
		45	+ \item \textit{Migration from Redmine management to Gitlab (one of the
		46	+integrated software):} Wiki feature was used for meeting logging, defining
		47	+goals, sprint planning, and documentation of deployment processes and
		48	+administration resources guide. Issue tracker was used for discussing
		49	+requirements, monitoring the features under development, registrating changes,
		50	+and validating functionalities delivered.
		51	+ \item \textit{Using the project mailing list provided by Mailman's platform:}
		52	+Scheduling meetings and defining schedule. Through the list, brainstorms and
		53	+collaborative definition of requirements also happened.
		54	+\end{itemize}
		55	+
		56	+Data from Gitlab shows 775 issues were opened and 4658 comments were made in the
		57	+repository that versioned the platform (without considering the software
		58	+repositories that integrated the platform) within the SPB platform. The issues
		59	+were created by 59 different authors (8 MPOG representatives), and commented by
		60	+64 different users (9 MPOG users). Of the 84 created issues with greater
		61	+interaction (number of comments >= 10), ~43\% are authored by one of MPOG
		62	+representatives. Students said the main communication ways used to interacted
		63	+with software communities were: IRC (86.5\%); Mailing list (73\%); Issues
		64	+(67.6\%). The main means of communication between senior developers and
		65	+students was IRC (100\%), Mailing list (100\%), Videoconference (100\%) and
		66	+Issue Tracker (62.5\%). Senior developers and MPOG staff interected mostly via
		67	+Mailing List (87.5\%) and Issue tracker (50\%). One of the interviewed MPOG
		68	+staff ratifies the main interaction spaces between teams: "We interacted a lot
		69	+with mailing lists and chatting, sometimes with gitlab within the line of
		70	+development."
		71	+%
		72	+\textbf{Benefits}
		73	+
		74	+\begin{itemize}
		75	+\item \textit{Confidence in developed code}
		76	+\subitem One of the gov-side interviewees said "Everything was validated, we
		77	+tested the features and the project was developed inside the platform, so that
		78	+the feature was validated in the development of the software itself. Our team at
		79	+MPOG, used the development and communication tools of the platform itself. From
		80	+the moment we installed it, and began to use it for development, this validation
		81	+was constant. We felt confident about the features"
		82	+\item \textit{Transparency and efficiency in communication}
		83	+\subitem Undegraduate student response: "We always had an open channel for
		84	+conversations / interactions, whether it was the project issues, the IRC
		85	+channel, via Whatsapp, etc."
		86	+\subitem MPOG: "I think the communication was excellent, it was comprehensive,
		87	+instantaneous and effective. We asked a question, a question, and very quickly
		88	+it was answered. Communication goes far beyond that, you communicate to everyone
		89	+in the project everything that was happening. We did not have issue related to
		90	+emails, use the mailing list more and avoid e-mails, it helped a lot because it
		91	+was all public and did not pollute our mailbox. You wanted to know something,
		92	+could go there and look at what was happening. We also had instant chatting,
		93	+where the team stayed online, working connected. We had a quick access to the
		94	+team. This makes all the difference in a project."
		95	+\item \textit{Monitoring of interactions among students, senior developers and
		96	+public servants by coordinators}
		97	+\subitem MPOG: "The leader himself informed us who was doing something, then we
		98	+talked directly to that student [..] But this interaction was not very frequent,
		99	+sometimes we would ask something on the list and the coordinator would answer
		100	+first."
		101	+\item \textit{Increase interaction between development team and contract
		102	+management team}
		103	+\subitem MPOG: "There was a lot of evolution, a lot of communication via Gitlab"
		104	+\item \textit{Organically documentation and records generation}
		105	+\subitem MPOG: "For me it was a lot of learning, there is a lot of things
		106	+documented in the e-mails and also there in the portal itself of what happened
		107	+in the project. At any moment we can go there and see how it worked, how the
		108	+person did, and manages to salvage those good points."
		109	+\end{itemize}
		110	+
		111	+\subsubsection{Continuos Delivery}
		112	+
		113	+\begin{itemize}
		114	+ \item Creating DevOps Team
		115	+ \item Defining continuous delivery pipeline
		116	+ \item DevOps team periodically going to the ministry to help deploy each version
		117	+\end{itemize}
		118	+
		119	+\textbf{Benefits}
		120	+
		121	+\begin{itemize}
		122	+\item \textit{Increase government confidence for collaborative projects with the
		123	+university}
		124	+\subitem MPOG: "At first the government staff had a bias that universities did
		125	+not deliver and we overcame that bias in the course of the project. We deliver a
		126	+lot and with quality. Today, I think that if we had paid the same amount for a
		127	+company, it would not have done what was delivered and with the quality that was
		128	+delivered with the price that was paid."
		129	+\item \textit{Motivate teams}
		130	+\subitem {81.1\%} of students think new versions released in production motivated
		131	+them during the project
		132	+\subitem {75\%} of senior developers think new versions released in production
		133	+motivated them during the project
		134	+\subitem {81\%} of students think the presence of a specific DevOps team was
		135	+necessary for the project
		136	+\item \textit{Transfer of knowledge about DevOps and Continuous Deliveries from
		137	+the academic team to the government infrastructure team}
		138	+\subitem MPOG: "I only noticed positive aspects in the delivery. I think in the
		139	+interaction, we had a lot of support to be able to deploy. From the time that
		140	+the version was mature, which had already been tested in the UnB test
		141	+environment and was ready to be put into production, we had a great agility to
		142	+release in production. Then in the course of the project we realized that the
		143	+infrastructure team [of MPOG] started to trust the UnB team a lot. Because, for
		144	+you to put software in production in government, there is a whole process
		145	+behind. The government has much of this security issue."
		146	+\subitem MPOG: "If there was anything stopping the business from working, the
		147	+software working inside, we would ask the seniors for support so we could
		148	+investigate that, and the infrastructure team was also instructed to prioritize
		149	+it. So when it came to an impasse, the teams were all together, both from within
		150	+MPOG as well as senior developers and other UnB developers to unlock, to find
		151	+the problem."
		152	+\item \textit{Align the university and government teams pace in the execution of
		153	+the activities}
		154	+\subitem MPOG: "In the beginning, infrastructure personnel were not accustomed
		155	+to deliveries so fast. They had to adapt to this pace. The portal of the SPB
		156	+before the project was not there [in the MPOG infrastructure], it was in another
		157	+place, they did not have that dynamics there. But what they asked for UnB (some
		158	+configuration, installation manual, how to install everything inside) was
		159	+requested and delivered."
		160	+\item \textit{Improve translation from one development process to the other}
		161	+\subitem MPOG: "We had an overview at the strategic level, but when we went down
		162	+to the level of functionality we had this difficulty to do the planning of the
		163	+release every four months. But in the end, I think this has not been a problem,
		164	+because a project you are delivering, the results are going to production, the
		165	+code is quality, the team is qualified/capable and the project is doing well, it
		166	+does not impact as much in practice, because the result is being delivered.
		167	+\end{itemize}
		168	+
		169	+\subsubsection{Organization of the project in teams for each front, with a
		170	+undergraduate student as coach and at least one senior developer}
		171	+
		172	+\begin{itemize}
		173	+ \item \textit{Four fronts: Colab, Noosfero, DevOps and Front-End/UX}
		174	+ \item \textit{Definition of the role of team coaches and meta-coach, selected from undergraduate students group}
		175	+ \item \textit{Hiring professionals from the IT market for face-to-face or remote work, specialists in the software components}
		176	+\end{itemize}
		177	+
		178	+\textbf{Benefits}
		179	+
		180	+\begin{itemize}
		181	+\item \textit{Help to conciliate development processes and decision-making}
		182	+\subitem {62,5\%} of senior developers believe they have helped MPOG staff to more clearly express their requests
		183	+\subitem {87,5\%} of seniors agreed with the project development process. For 37.5\% this process was little similar to their previous experiences, for the others there was a certain similarity.
		184	+\subitem {62,5\%} of seniors did not understand MPOG's project management process. {50\%} of them believe their project productivity was affected by MPOG's project management process.
		185	+\subitem Senior Dev: "I think my main contribution was to have balanced the relations between the MPOG staff and the UnB team"
		186	+\subitem Senior Dev: "When I entered the project, the client had a disproportionate view of how to make explicit the requirements. They were still talking about use cases and were extremely concerned about validation processes and acceptance of these documents."
		187	+\subitem MPOG: "You had the reviewers, who were the original developers of the software, that gave you confidence and confidence in the code."
		188	+%
		189	+\item \textit{Create support and reference points for students, senior developers, and government staff}
		190	+\subitem {89.1\%} of students believe that the presence of the leader was essential to the running of Sprint
		191	+\subitem {87.5\%} of seniors believe that the presence of team leaders was essential for their interaction with the team
		192	+\subitem MPOG: "It interacted more with the project coordinator and team coaches (noosfero, colab, visual identity). Interacted with coaches by mailing list, hangouts The reason was usually to elucidate requirements, to ask questions about requirements, to understand some functionality. "
		193	+\subitem MPOG: "There was interaction with the other [non-coaches] because they also participated in the bi-weekly meetings (sprints), but it was more with coaches."
		194	+\subitem MPOG: "Access to coaches was faster, because we were in much more interaction with leaders than with senior developers. Sometimes the coaches brought the question to the senior developers."
		195	+%
		196	+\item \textit{Transfer of knowledge from industry and FLOSS community to both academia and government}
		197	+\subitem {62.5\%} of senior developers believe that they have collaborated in the relationship between the management and development processes of the two institutions
		198	+\subitem {100\%} of the students we interviewed believe that working with senior developers was important during the project
		199	+\subitem {91.\%} of students also believe that working with seniors was important for learning
		200	+\subitem {75\%} of senior developers believe that 'Working in pairs with a senior' and 62.5% who 'Participate in joint review tasks' were the tasks with the involvement of them that most contributed to the evolution of students in the project.
		201	+\subitem {75\%} of senior developers believe that in guiding a student, this knowledge was widespread among the other students on the team.
		202	+\subitem MPOG: "On the side of UnB, what we perceived so strongly was that the project took a very big leap when the original developers of the software (the official software development) were hired in the case of Noosfero and Colab [..] Because they had a guide on how to develop things in the best way and were able to solve non-trivial problems and quickly "
		203	+\end{itemize}
		204	+
		205	+%* Filtrar a comunicação por níveis de maturidade/experiência e responsabilidades
		206	+%MPOG: "Eu acho que esses pontos de conflito eram muito mais fáceis de lidar com a equipe do que com a própria coordenação. [..] Eu acho que tem uma diferença também de papel tem uma diferença de postura. Eu acho que a relação com a equipe, embora ela fosse saudável, eu acho que a equipe não tinha tanta autonomia quanto à coordenação tinha. Então talvez fosse mais difícil com a coordenação e não com a equipe, porque a equipe ela sabia o limite dela e a partir dali ela não agia mais, ela já convocava a coordenação para lidar (a gerência)."
		207	+
		208	+\subsubsection{Bringing the government staff directly responsible for the project together with development team}
		209	+\begin{itemize}
		210	+\item Biweekly meetings (planning and sprint review) in the development lab with the presence of government staff, team coaches and senior developers
		211	+\item Discuss features under development directly on Gitlab Issue Tracker
		212	+\item Only strategic decisions or bureaucratic issues involve the directors/secretaries
		213	+\end{itemize}
		214	+
		215	+\textbf{Benefits}
		216	+
		217	+\begin{itemize}
		218	+\item \textit{Reduce communication misunderstood}
		219	+\subitem MPOG: "That's when the project started, people [MPOG staff] did not participate in anything. The communication process was horrible."; "The [MPOG] coordinator did not help, he would say something and UnB would talk to another at the meeting and it was the biggest mess." About the direct dialogue between the academic team and MPOG staff (without the involvement of coordinators and / or directors) , she said "That's where things really started to move, that the communication of the project began to improve."
		220	+%
		221	+\item \textit{Empathy between members on both sides}
		222	+\subitem {72.9\%} of students believe that interacting with MPOG staff was important during the project
		223	+\subitem Only 27\% of the students interviewed said they did not feel like attending meetings with MPOG employees
		224	+\subitem MPOG: "You know people in person and it makes such a big difference because it causes empathy. You know what the person is going through on their side and she knows what we're going through on our side. So the next time you have a non-personal interaction (by mail, by list ...) I think it even facilitates, improves communication. You already know who that person is, it's not just a name. "
		225	+%
		226	+\item \textit{Develop requirements closer to the expectations of both sides}
		227	+\subitem {81.1 \%} of students believe that the participation of MPOG staff in planning and closing sprints was important for the development of the project
		228	+\subitem {75.6 \%} of students believe that writing the requirements together with the MPOG staff was very important
		229	+\subitem Undergrad student: "Joint planning and timely meetings were very important for understanding the needs of MPOG, and the interaction via SPB tools helped validate the tool as a development platform"
		230	+\subitem Undergrad student: "Often they did not know what they really wanted, and they caused some delays in the development of sprints"
		231	+\subitem Undergrad student: "A relationship of constant attempt to balance and negotiate. The client does not always know the impacts of their requests"
		232	+\subitem MPOG: "I believe it was very positive, we also liked to go there, to interact with the team. I think it brought more unity, more integration into the project, because we went there, where people were working and they show what was done. I think they also liked to receive our feedback about what had been done by them.This interaction did not just made with the coordinator. I found it very important and very positive it. "
		233	+%
		234	+\item \textit{Improve understanding of collaborative development by MPOG staff}
		235	+\subitem Undergrad student: "In the beginning the demands of MPOG were very 'orders from above', but according to the progress of the project, they understood better our work philosophy and became more open"
		236	+\subitem MPOG: "During development we realized that the team that was developing also felt like the owner of the project felt involved not only a mere executor of an order. It was not a client relationship, it was a partnership relationship, so there was a lot of team suggestions to be put into the project. Sometimes these were put in for us to decide and sometimes not."
		237	+\subitem MPOG: "I think it was easy, I think the team was aligned. In addition to being aligned, these items that, for example, were not priorities and became priorities, were, in a sense, brought with some arguments from the team. So the team was able to argue and succeed in showing that it was important, that it needed to be prioritized, and I think the team was able to present the arguments well for some of the priorities that happened during the process."
		238	+%
		239	+\item \textit{Align the pace of both sides to execute activities}
		240	+\subitem MPOG: "When we went there, I knew people and made that interaction more frequent, we also felt encouraged to validate faster and give faster feedback to the teams so they would not wait there. I knew they were waiting for our feedback and we were struggling to do it fast, because we ended a sprint and start another and not stop. We gave that feedback fast and they also gave quick feedback for any questions when they encountered a problem. That gave the project agility, things flowed faster and better. "
		241	+\end{itemize}