Download as
Browse Code »

Commit 303f7df27769bc1a578e2d1dfa437b44d043a2d6

Authored by Rodrigo Siqueira de Melo
1 parent cfc06e6a
Exists in master and in 3 other branches oss_2018, oss_2018_siqueira, rework_case_study

Small fixes on process and updated architecture

Showing 2 changed files with 148 additions and 57 deletions   Show diff stats
opensym2017/content/04-process.tex
  Diff comments   View file @303f7df
1 1 \section{Development Organization and Process}
2 2 \label{sec:process}
3 3  
4   -The SPB members were composed of a variety of professionals with different levels and skills. Most of them were undergraduate students with major in software engineering (from 4th semester or upper), hence they could not dedicate many hours in the project per week. In their daily lives in the project, they had programming and devops tasks. It is important to highlight that the students always had the flexibility to negotiate their work hours during the semester, because we tried to not cause any damage to their grades.
5   -
6   -The development of SPB project required a vast experience and background that usually undergraduate students do not have yet, because of this, some senior developer joined to the project to help with hard issues and to transfer knowledge to the students. Their tasks were technical and provide solution to hard problems, in other words, they worked as a developer. All the senior lived in a different states spread around Brazil, and all the communication was made via internet. Finally, those professionals are very skilful and the project could not afford a full time work for them and as a result, some of them worked partially on the project.
7   -
8   -Finally, the last group of actors of this project was composed of employees formally work for the Brazilian Government, in the Ministery of Planning, development and Management (MPOG is the Brazilian acronyms). All the project decisions, validations, and scope definitions were made by them.
  4 +The SPB members were composed of a variety of professionals with different
  5 +levels and skills. Most of them were undergraduate students with major in
  6 +software engineering (from 4th semester or upper), hence they could not
  7 +dedicate many hours in the project per week. In their daily lives in the
  8 +project, they had programming and devops tasks. It is important to highlight
  9 +that the students always had the flexibility to negotiate their work hours
  10 +during the semester, because we tried to not cause any damage to their grades.
  11 +
  12 +The development of SPB project required a vast experience and background that
  13 +usually undergraduate students do not have yet, for this reason, some senior
  14 +developer joined to the project to help with hard issues and to transfer
  15 +knowledge to the students. Their main tasks provide solutions for hard
  16 +problems, in other words, they worked as a developer. All the senior lived in a
  17 +different states spread around Brazil, and all the communication was made via
  18 +internet. Finally, those professionals are very skilful and the project could
  19 +not afford a full time work for them, as a result, some of them worked
  20 +partially on the project.
  21 +
  22 +Finally, the last group of actors of this project was composed of employees
  23 +formally working for the Brazilian Government, in the Ministery of Planning,
  24 +development and Management (MPOG is the Brazilian acronyms). All the project
  25 +decisions, validations, and scope definitions were made by them. As can be
  26 +seen, the project had many kinds of profiles that had to be organized and
  27 +synchronized.
9 28  
10 29 \subsection{Teams organizations}
11 30  
12   -More than X\% of the teams was composed of undergraduate students and they worked physically in the same laboratory in opposite of the senior. Each student had their own scheduler based on their class, it made hard to implement pair programming. Also, they had a different area of interests. To cope with all this diversity we had two basic rules which guided the project organization:
  31 +More than X\% of the teams was composed of undergraduate students and they
  32 +worked physically in the same laboratory in the opposite of the senior. Each
  33 +student had their own scheduler based on their class, it made complicated to
  34 +implement pair programming. Also, they had a different area of interests. To
  35 +cope with those diversity, we had two basic rules which guided the project
  36 +organization:
13 37  
14 38 \begin{enumerate}
15 39 \item Classes have to be the top priority for undergraduate students;
16 40 \item Always work in pair (locally or remotely).
17 41 \end{enumerate}
18 42  
19   -With the aforementioned ideas we divided all the project into four different teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible for reducing the communication problem with the other teams and help the members to organize itself in the best way for everyone (always respecting the work time). One important thing to notice is the mutability of the team and the coach, during the project many students changed between the teams to try different areas.
20   -
21   -One important characteristic in the team organization was the presence of (at least) one senior per team. This was essential, once hard decisions and complex problems were usually addressed to them, this relieved the coach of the duty to take a complicated technical decisions (this encouraged students to be a coach). Lastly, the senior had to respect a rule number two and work with some student, this was important to give the undergraduate the opportunity to interact with a savvy professional in his area and keeping the knowledge flow in the project.
  43 +With the aforementioned rules we divided all the project into four different
  44 +teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible
  45 +for reducing the communication problem with the other teams and help the
  46 +members to organize itself in the best way for everyone (always respecting the
  47 +work time). The coach, was a normal student working in some of the teams with
  48 +the extra duty of register the current tasks developed in the sprint and with
  49 +the responsibility to talk with other teams. One important thing to notice is
  50 +the mutability of the team and the coach, during the project many students
  51 +changed between the teams to try different areas.
  52 +
  53 +One characteristic of the teams was the presence of (at least) one senior per
  54 +team. This was essential, because hard decisions and complex problems were
  55 +usually addressed to them, this relieved the coach duty to take a complicated
  56 +technical decisions and encouraged students to be a coach. Lastly, the senior
  57 +had to respect a rule number two and work with students, this was important to
  58 +gave the undergraduate the opportunity to interact with a savvy professional in
  59 +his area and keeping the knowledge flow in the project.
  60 +
  61 +Finally, we had to add two last elements of the team organization, that was
  62 +essential for the project harmony: the meta-coach and professors. The former
  63 +was a software engineer recently graduated and which wanted to keep working on
  64 +the project, the latter were professors that orchestrated all the interactions
  65 +between all members of the project. The meta-coach usually worked in one
  66 +specific team and had the extra task of knowing the current status of all
  67 +teams. Professors and meta-coaches worked together to reduce the communication
  68 +problem between all the teams. Lastly, all the bureaucracy tasks was
  69 +centralized in the professors.
22 70  
23 71 \subsection{Meetings}
24 72  
25   -Brazilian government used to work with software development in a very traditional way, frequently they focused on documents and does not focus on what really matter: running software. This way of thinking caused to us a communication noise with MPOG, because they constantly tried to leverage on our work style. It was especially hard to convince them to work with open scope and agile development, but after months of work and showing results they stopped resisting.
26   -
27   -After many interactions with MPOG, they started to trust in our work and we defined some level of meeting granularity to avoid to generate overheads to the developers. We had a strategical and validating meeting with MPOG (the former once in a month and the latter each 15th day), release plaining with the entire team (one per month), and finally a sprint planning (one each 15th day).
28   -
29   -In the strategical meeting we usually defined the priorities and new features with MPOG (we always had to negotiate a bunch of things). Normally the professors, the coach of each team, and the employees of MPOG join in this meeting. In this meeting we usually discussed what the team already produced since our last meeting, and we establish the new features for the next release. Notice that just part of the team join in this meeting to avoid generating unnecessary overhead to the developers, but all the students interested to participate was accepts (many students wanted this experience during the project).
30   -
31   -After the strategical meeting with MPOG, we had a planning turn with all teams together. In this part, each team works together to convert the MPOG wishes into small parts which was represented by the epics of the release. Each coach was responsible for conducting the planning, and after that register it on the project wiki (the wiki provided by Gitlab). With this epic, each 14th day the team have generated their sprint scheduler (with small achievements mapped in issues).
32   -
33   -To keep MPOG always as part of the project, we invited them to work with us to validate the new features in progress. Normally we had a meeting each 15th day. Basically, this was our work flow, we always kept everything extremely open to the MPOG (our way of work and open source projects) and to the team.
34   -
35   -To keep the track of all of those things we used the SPB, especially the Gitlab. Basically, we had:
  73 +Brazilian government used to work with software development in a very
  74 +traditional way, frequently they claim on documents and does not focus on what
  75 +really matter (running software). This way of thinking caused to us a
  76 +communication noise with MPOG, because they constantly tried to leverage on our
  77 +work style. It was especially hard to convince them to accept the idea of open
  78 +scope and agile development, but after months of labor and showing results they
  79 +stopped resisting.
  80 +
  81 +We defined some level of meeting granularity to avoid to generate overheads to
  82 +the developers. We had a strategical and validating meeting with MPOG (the
  83 +former once in a month and the latter each 15th day), release plaining with the
  84 +entire team (one per month), and finally a sprint planning (one each 15th day).
  85 +Figure X is a diagram that represents our meeting organization.
  86 +
  87 +In the strategical meeting we usually defined the priorities and new features
  88 +with MPOG (we always had to negotiate next steps with them). Normally the
  89 +professors, the coach of each team, the meta-coach, and some employees of MPOG
  90 +join in this meeting. We usually discussed what the team already produced since
  91 +our last meeting, and we establish the new features for the next release.
  92 +Notice that just part of the team join in this meeting to avoid generating
  93 +unnecessary overhead to the developers, but all the students interested to
  94 +participate was allowed to join (many students wanted this experience during
  95 +the project).
  96 +
  97 +After the strategical meeting with MPOG, we had a planning turn with all teams
  98 +together. In this part, each team worked together to convert the MPOG wishes
  99 +into small parts which was represented by the epics of the release. Each coach
  100 +was responsible for conducting the planning, and after that register it on the
  101 +project wiki (the wiki provided by Gitlab). With this epic, each 14th day the
  102 +team have generated their sprint scheduler (with small achievements mapped in
  103 +issues).
  104 +
  105 +To keep MPOG always updated, we invited them to work with us to validate the
  106 +new features in progress. Normally we had a meeting each 15th day. Basically,
  107 +this was our work flow, we always kept everything extremely open to the MPOG
  108 +(our way of work and open source projects) and to the team.
  109 +
  110 +To keep the track of all of those things we used the SPB, especially the
  111 +Gitlab. Basically, we had:
36 112  
37 113 \begin{enumerate}
38 114 \item Project repository: We have one organization with many repositories
... ...
opensym2017/content/05-architecture.tex
  Diff comments   View file @303f7df
1 1 \section{Architecture}
2 2 \label{sec:architecture}
3 3  
4   -%TODO: Kanashiro e Siqueira
5   -
6   -The two main requirements provided by the Brazilian Federal Government
7   -for the new platform were:
8   -%
9   -1) \textit{Integrate existing FOSS systems}, with minimal differences
10   -from their original versions. This way, the platform can benefit from
11   -improvements done by the upstream communities that provide those
12   -systems, and the maintenance effort that is specific for the SPB Portal
13   -should be reduced;
14   -%
15   -and
16   -2) \textit{Provide a consistent user interface} across the different
17   -systems, as well as centralized authentication.
18   -
19   -The first requirement was accomplished by dedicating specialized teams
20   -for each system that was being integrated. The teams would learn how to
21   -develop their assigned systems, and contribute the necessary features
22   -directly to the original communities, so that the version we used was
23   -not significantly different from the original. Of course, at times
24   -project deadlines forced us to use our own version before tho features
25   -were fully reviewed and integrated upstream to the original projects,
26   -but we managed to contribute the vast majority of the changes back.
27   -
28   -For the second requirement, we integrated a web integration platform
29   -called Colab\footnote{\url{https://github.com/colab/colab}}. Colab
30   -serves as a frontend for other web applications as a reverse proxy,
31   -manages authentication, and can apply changes to the HTML provided by
32   -the integrated applications in order to provide visual consistency.
33   -Colab had support for an initial set of applications (Trac, GNU Mailman,
34   -Apache Lucene) hard-coded; our team evolved Colab so that it can now
35   -receive plugins to add support for new applications with minimal changes
36   -to its existing core. We added support for the other applications used
37   -in the SPB platform: Noosfero, GitLab, and Mezuro.
  4 +At the point of view of the architecture, two main requirements was brought by
  5 +the Brazilian Federal Government for the new platform were:
  6 +
  7 +\begin{enumerate}
  8 + \item \textit{Integrate existing FOSS systems}, with minimal differences from
  9 + their original versions;
  10 + \item \textit{Provide a consistent user interface} across the different
  11 + systems, as well as centralized authentication.
  12 +\end{enumerate}
  13 +
  14 +Adopting existing FOSS systems by the government could bring the benefit of
  15 +improvements done by the upstream communities, and the maintenance effort. On
  16 +the other hand, integrating different tools with distinct intent, it is not an
  17 +easy task and it was important to have a consistent user interface which
  18 +justifies the last requirement.
  19 +
  20 +For the first requirement, we identified four main systems that required
  21 +specialized teams for work in the integration process. The teams learned how to
  22 +develop for their assigned systems and contributed to the original
  23 +communities, so that the version we used were not significantly different from
  24 +the original. Unfortunately, the deadlines at the end of the project forced us
  25 +to use our own version before the features were fully reviewed and integrated
  26 +into the upstream of the project.
  27 +
  28 +At the end of the project, SPB portal was composed of more than ten systems
  29 +among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix,
  30 +Munin, and so forth. The following sections explained a little bit of Colab,
  31 +Noosfero, Mezuro and Gitlab (the main tools which we contributed). Later, we
  32 +described how we integrated those tools and conclude with the deployment.
  33 +
  34 +\subsection{Colab}
  35 +
  36 +For the second requirement, we decided to use a web integration platform named
  37 +Colab\footnote{\url{https://github.com/colab/colab}}. It works as a frontend
  38 +for other web applications as a reverse proxy, manages authentication, and can
  39 +apply changes to the HTML provided by the integrated applications in order to
  40 +bring visual consistency.
  41 +
  42 +Initially, Colab had support for a small set of applications (Trac, GNU
  43 +Mailman, Apache Lucene) and all of them was hard-coded; our team evolved Colab
  44 +so that it can now receive plugins to add support for new applications with
  45 +minimal changes to its existing core. We developed plugins to be used in the
  46 +SPB platform: Noosfero, GitLab, and Mezuro.
  47 +
  48 +\subsection{Noosfero}
38 49  
39 50 Noosfero\footnote{\url{http://noosfero.org/}} is a software for building
40 51 social and collaboration networks. Besides the classical social
... ... @@ -42,12 +53,17 @@ networking features, it also provides publication features such as blogs
42 53 and a general-purpose CMS (Content Management System). Most of the user
43 54 interactions with SPB is through Noosfero: user registration, project
44 55 home pages and documentation, and contact forms.
  56 +
  57 +\subsection{Gitlab and Mezuro}
  58 +
45 59 GitLab\footnote{\url{http://gitlab.com/}} is a web-based Git repository
46 60 manager with wiki pages and issue tracking features.
47 61 Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code
48 62 metric to monitor the internal quality of softwares written in C, C++,
49 63 Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists.
50 64  
  65 +\subsection{System unification}
  66 +
51 67 \begin{figure}[hbt]
52 68 \centering
53 69 \includegraphics[width=\linewidth]{figures/arch.png}
... ... @@ -74,6 +90,8 @@ list posts, or source code.
74 90 \label{fig:architecture2}
75 91 \end{figure*}
76 92  
  93 +\subsection{Deploy}
  94 +
77 95 In real, the SPB platform was deployed in 7 virtual machines with different functions,
78 96 as we can see in Figure \ref{fig:architecture2}.
79 97  
... ... @@ -102,6 +120,3 @@ static analysis tools on source code stored in repository and provide this data
102 120 to Prezento. A social network and CMS (Content Manager System) is provided by
103 121 Noosfero in \textit{social}, and the databases of all tools with a cache
104 122 service are in \textit{database}.
105   -
106   -* Visão geral da implantação pode ser em um nível maior de abstração, mas deixar claro o esforço para automatizar (DevOps -> Chake)
107   -* Empacotamentos
... ...