Commit 303f7df27769bc1a578e2d1dfa437b44d043a2d6
1 parent
cfc06e6a
Exists in
master
and in
3 other branches
Small fixes on process and updated architecture
Showing
2 changed files
with
148 additions
and
57 deletions
Show diff stats
opensym2017/content/04-process.tex
1 | 1 | \section{Development Organization and Process} |
2 | 2 | \label{sec:process} |
3 | 3 | |
4 | -The SPB members were composed of a variety of professionals with different levels and skills. Most of them were undergraduate students with major in software engineering (from 4th semester or upper), hence they could not dedicate many hours in the project per week. In their daily lives in the project, they had programming and devops tasks. It is important to highlight that the students always had the flexibility to negotiate their work hours during the semester, because we tried to not cause any damage to their grades. | |
5 | - | |
6 | -The development of SPB project required a vast experience and background that usually undergraduate students do not have yet, because of this, some senior developer joined to the project to help with hard issues and to transfer knowledge to the students. Their tasks were technical and provide solution to hard problems, in other words, they worked as a developer. All the senior lived in a different states spread around Brazil, and all the communication was made via internet. Finally, those professionals are very skilful and the project could not afford a full time work for them and as a result, some of them worked partially on the project. | |
7 | - | |
8 | -Finally, the last group of actors of this project was composed of employees formally work for the Brazilian Government, in the Ministery of Planning, development and Management (MPOG is the Brazilian acronyms). All the project decisions, validations, and scope definitions were made by them. | |
4 | +The SPB members were composed of a variety of professionals with different | |
5 | +levels and skills. Most of them were undergraduate students with major in | |
6 | +software engineering (from 4th semester or upper), hence they could not | |
7 | +dedicate many hours in the project per week. In their daily lives in the | |
8 | +project, they had programming and devops tasks. It is important to highlight | |
9 | +that the students always had the flexibility to negotiate their work hours | |
10 | +during the semester, because we tried to not cause any damage to their grades. | |
11 | + | |
12 | +The development of SPB project required a vast experience and background that | |
13 | +usually undergraduate students do not have yet, for this reason, some senior | |
14 | +developer joined to the project to help with hard issues and to transfer | |
15 | +knowledge to the students. Their main tasks provide solutions for hard | |
16 | +problems, in other words, they worked as a developer. All the senior lived in a | |
17 | +different states spread around Brazil, and all the communication was made via | |
18 | +internet. Finally, those professionals are very skilful and the project could | |
19 | +not afford a full time work for them, as a result, some of them worked | |
20 | +partially on the project. | |
21 | + | |
22 | +Finally, the last group of actors of this project was composed of employees | |
23 | +formally working for the Brazilian Government, in the Ministery of Planning, | |
24 | +development and Management (MPOG is the Brazilian acronyms). All the project | |
25 | +decisions, validations, and scope definitions were made by them. As can be | |
26 | +seen, the project had many kinds of profiles that had to be organized and | |
27 | +synchronized. | |
9 | 28 | |
10 | 29 | \subsection{Teams organizations} |
11 | 30 | |
12 | -More than X\% of the teams was composed of undergraduate students and they worked physically in the same laboratory in opposite of the senior. Each student had their own scheduler based on their class, it made hard to implement pair programming. Also, they had a different area of interests. To cope with all this diversity we had two basic rules which guided the project organization: | |
31 | +More than X\% of the teams was composed of undergraduate students and they | |
32 | +worked physically in the same laboratory in the opposite of the senior. Each | |
33 | +student had their own scheduler based on their class, it made complicated to | |
34 | +implement pair programming. Also, they had a different area of interests. To | |
35 | +cope with those diversity, we had two basic rules which guided the project | |
36 | +organization: | |
13 | 37 | |
14 | 38 | \begin{enumerate} |
15 | 39 | \item Classes have to be the top priority for undergraduate students; |
16 | 40 | \item Always work in pair (locally or remotely). |
17 | 41 | \end{enumerate} |
18 | 42 | |
19 | -With the aforementioned ideas we divided all the project into four different teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible for reducing the communication problem with the other teams and help the members to organize itself in the best way for everyone (always respecting the work time). One important thing to notice is the mutability of the team and the coach, during the project many students changed between the teams to try different areas. | |
20 | - | |
21 | -One important characteristic in the team organization was the presence of (at least) one senior per team. This was essential, once hard decisions and complex problems were usually addressed to them, this relieved the coach of the duty to take a complicated technical decisions (this encouraged students to be a coach). Lastly, the senior had to respect a rule number two and work with some student, this was important to give the undergraduate the opportunity to interact with a savvy professional in his area and keeping the knowledge flow in the project. | |
43 | +With the aforementioned rules we divided all the project into four different | |
44 | +teams: Colab, Noosfero, Design, and DevOps. Each team had one coach responsible | |
45 | +for reducing the communication problem with the other teams and help the | |
46 | +members to organize itself in the best way for everyone (always respecting the | |
47 | +work time). The coach, was a normal student working in some of the teams with | |
48 | +the extra duty of register the current tasks developed in the sprint and with | |
49 | +the responsibility to talk with other teams. One important thing to notice is | |
50 | +the mutability of the team and the coach, during the project many students | |
51 | +changed between the teams to try different areas. | |
52 | + | |
53 | +One characteristic of the teams was the presence of (at least) one senior per | |
54 | +team. This was essential, because hard decisions and complex problems were | |
55 | +usually addressed to them, this relieved the coach duty to take a complicated | |
56 | +technical decisions and encouraged students to be a coach. Lastly, the senior | |
57 | +had to respect a rule number two and work with students, this was important to | |
58 | +gave the undergraduate the opportunity to interact with a savvy professional in | |
59 | +his area and keeping the knowledge flow in the project. | |
60 | + | |
61 | +Finally, we had to add two last elements of the team organization, that was | |
62 | +essential for the project harmony: the meta-coach and professors. The former | |
63 | +was a software engineer recently graduated and which wanted to keep working on | |
64 | +the project, the latter were professors that orchestrated all the interactions | |
65 | +between all members of the project. The meta-coach usually worked in one | |
66 | +specific team and had the extra task of knowing the current status of all | |
67 | +teams. Professors and meta-coaches worked together to reduce the communication | |
68 | +problem between all the teams. Lastly, all the bureaucracy tasks was | |
69 | +centralized in the professors. | |
22 | 70 | |
23 | 71 | \subsection{Meetings} |
24 | 72 | |
25 | -Brazilian government used to work with software development in a very traditional way, frequently they focused on documents and does not focus on what really matter: running software. This way of thinking caused to us a communication noise with MPOG, because they constantly tried to leverage on our work style. It was especially hard to convince them to work with open scope and agile development, but after months of work and showing results they stopped resisting. | |
26 | - | |
27 | -After many interactions with MPOG, they started to trust in our work and we defined some level of meeting granularity to avoid to generate overheads to the developers. We had a strategical and validating meeting with MPOG (the former once in a month and the latter each 15th day), release plaining with the entire team (one per month), and finally a sprint planning (one each 15th day). | |
28 | - | |
29 | -In the strategical meeting we usually defined the priorities and new features with MPOG (we always had to negotiate a bunch of things). Normally the professors, the coach of each team, and the employees of MPOG join in this meeting. In this meeting we usually discussed what the team already produced since our last meeting, and we establish the new features for the next release. Notice that just part of the team join in this meeting to avoid generating unnecessary overhead to the developers, but all the students interested to participate was accepts (many students wanted this experience during the project). | |
30 | - | |
31 | -After the strategical meeting with MPOG, we had a planning turn with all teams together. In this part, each team works together to convert the MPOG wishes into small parts which was represented by the epics of the release. Each coach was responsible for conducting the planning, and after that register it on the project wiki (the wiki provided by Gitlab). With this epic, each 14th day the team have generated their sprint scheduler (with small achievements mapped in issues). | |
32 | - | |
33 | -To keep MPOG always as part of the project, we invited them to work with us to validate the new features in progress. Normally we had a meeting each 15th day. Basically, this was our work flow, we always kept everything extremely open to the MPOG (our way of work and open source projects) and to the team. | |
34 | - | |
35 | -To keep the track of all of those things we used the SPB, especially the Gitlab. Basically, we had: | |
73 | +Brazilian government used to work with software development in a very | |
74 | +traditional way, frequently they claim on documents and does not focus on what | |
75 | +really matter (running software). This way of thinking caused to us a | |
76 | +communication noise with MPOG, because they constantly tried to leverage on our | |
77 | +work style. It was especially hard to convince them to accept the idea of open | |
78 | +scope and agile development, but after months of labor and showing results they | |
79 | +stopped resisting. | |
80 | + | |
81 | +We defined some level of meeting granularity to avoid to generate overheads to | |
82 | +the developers. We had a strategical and validating meeting with MPOG (the | |
83 | +former once in a month and the latter each 15th day), release plaining with the | |
84 | +entire team (one per month), and finally a sprint planning (one each 15th day). | |
85 | +Figure X is a diagram that represents our meeting organization. | |
86 | + | |
87 | +In the strategical meeting we usually defined the priorities and new features | |
88 | +with MPOG (we always had to negotiate next steps with them). Normally the | |
89 | +professors, the coach of each team, the meta-coach, and some employees of MPOG | |
90 | +join in this meeting. We usually discussed what the team already produced since | |
91 | +our last meeting, and we establish the new features for the next release. | |
92 | +Notice that just part of the team join in this meeting to avoid generating | |
93 | +unnecessary overhead to the developers, but all the students interested to | |
94 | +participate was allowed to join (many students wanted this experience during | |
95 | +the project). | |
96 | + | |
97 | +After the strategical meeting with MPOG, we had a planning turn with all teams | |
98 | +together. In this part, each team worked together to convert the MPOG wishes | |
99 | +into small parts which was represented by the epics of the release. Each coach | |
100 | +was responsible for conducting the planning, and after that register it on the | |
101 | +project wiki (the wiki provided by Gitlab). With this epic, each 14th day the | |
102 | +team have generated their sprint scheduler (with small achievements mapped in | |
103 | +issues). | |
104 | + | |
105 | +To keep MPOG always updated, we invited them to work with us to validate the | |
106 | +new features in progress. Normally we had a meeting each 15th day. Basically, | |
107 | +this was our work flow, we always kept everything extremely open to the MPOG | |
108 | +(our way of work and open source projects) and to the team. | |
109 | + | |
110 | +To keep the track of all of those things we used the SPB, especially the | |
111 | +Gitlab. Basically, we had: | |
36 | 112 | |
37 | 113 | \begin{enumerate} |
38 | 114 | \item Project repository: We have one organization with many repositories | ... | ... |
opensym2017/content/05-architecture.tex
1 | 1 | \section{Architecture} |
2 | 2 | \label{sec:architecture} |
3 | 3 | |
4 | -%TODO: Kanashiro e Siqueira | |
5 | - | |
6 | -The two main requirements provided by the Brazilian Federal Government | |
7 | -for the new platform were: | |
8 | -% | |
9 | -1) \textit{Integrate existing FOSS systems}, with minimal differences | |
10 | -from their original versions. This way, the platform can benefit from | |
11 | -improvements done by the upstream communities that provide those | |
12 | -systems, and the maintenance effort that is specific for the SPB Portal | |
13 | -should be reduced; | |
14 | -% | |
15 | -and | |
16 | -2) \textit{Provide a consistent user interface} across the different | |
17 | -systems, as well as centralized authentication. | |
18 | - | |
19 | -The first requirement was accomplished by dedicating specialized teams | |
20 | -for each system that was being integrated. The teams would learn how to | |
21 | -develop their assigned systems, and contribute the necessary features | |
22 | -directly to the original communities, so that the version we used was | |
23 | -not significantly different from the original. Of course, at times | |
24 | -project deadlines forced us to use our own version before tho features | |
25 | -were fully reviewed and integrated upstream to the original projects, | |
26 | -but we managed to contribute the vast majority of the changes back. | |
27 | - | |
28 | -For the second requirement, we integrated a web integration platform | |
29 | -called Colab\footnote{\url{https://github.com/colab/colab}}. Colab | |
30 | -serves as a frontend for other web applications as a reverse proxy, | |
31 | -manages authentication, and can apply changes to the HTML provided by | |
32 | -the integrated applications in order to provide visual consistency. | |
33 | -Colab had support for an initial set of applications (Trac, GNU Mailman, | |
34 | -Apache Lucene) hard-coded; our team evolved Colab so that it can now | |
35 | -receive plugins to add support for new applications with minimal changes | |
36 | -to its existing core. We added support for the other applications used | |
37 | -in the SPB platform: Noosfero, GitLab, and Mezuro. | |
4 | +At the point of view of the architecture, two main requirements was brought by | |
5 | +the Brazilian Federal Government for the new platform were: | |
6 | + | |
7 | +\begin{enumerate} | |
8 | + \item \textit{Integrate existing FOSS systems}, with minimal differences from | |
9 | + their original versions; | |
10 | + \item \textit{Provide a consistent user interface} across the different | |
11 | + systems, as well as centralized authentication. | |
12 | +\end{enumerate} | |
13 | + | |
14 | +Adopting existing FOSS systems by the government could bring the benefit of | |
15 | +improvements done by the upstream communities, and the maintenance effort. On | |
16 | +the other hand, integrating different tools with distinct intent, it is not an | |
17 | +easy task and it was important to have a consistent user interface which | |
18 | +justifies the last requirement. | |
19 | + | |
20 | +For the first requirement, we identified four main systems that required | |
21 | +specialized teams for work in the integration process. The teams learned how to | |
22 | +develop for their assigned systems and contributed to the original | |
23 | +communities, so that the version we used were not significantly different from | |
24 | +the original. Unfortunately, the deadlines at the end of the project forced us | |
25 | +to use our own version before the features were fully reviewed and integrated | |
26 | +into the upstream of the project. | |
27 | + | |
28 | +At the end of the project, SPB portal was composed of more than ten systems | |
29 | +among them we can highlight: Colab, Noosfero, Mezuro, Gitlab, Mailman, Postfix, | |
30 | +Munin, and so forth. The following sections explained a little bit of Colab, | |
31 | +Noosfero, Mezuro and Gitlab (the main tools which we contributed). Later, we | |
32 | +described how we integrated those tools and conclude with the deployment. | |
33 | + | |
34 | +\subsection{Colab} | |
35 | + | |
36 | +For the second requirement, we decided to use a web integration platform named | |
37 | +Colab\footnote{\url{https://github.com/colab/colab}}. It works as a frontend | |
38 | +for other web applications as a reverse proxy, manages authentication, and can | |
39 | +apply changes to the HTML provided by the integrated applications in order to | |
40 | +bring visual consistency. | |
41 | + | |
42 | +Initially, Colab had support for a small set of applications (Trac, GNU | |
43 | +Mailman, Apache Lucene) and all of them was hard-coded; our team evolved Colab | |
44 | +so that it can now receive plugins to add support for new applications with | |
45 | +minimal changes to its existing core. We developed plugins to be used in the | |
46 | +SPB platform: Noosfero, GitLab, and Mezuro. | |
47 | + | |
48 | +\subsection{Noosfero} | |
38 | 49 | |
39 | 50 | Noosfero\footnote{\url{http://noosfero.org/}} is a software for building |
40 | 51 | social and collaboration networks. Besides the classical social |
... | ... | @@ -42,12 +53,17 @@ networking features, it also provides publication features such as blogs |
42 | 53 | and a general-purpose CMS (Content Management System). Most of the user |
43 | 54 | interactions with SPB is through Noosfero: user registration, project |
44 | 55 | home pages and documentation, and contact forms. |
56 | + | |
57 | +\subsection{Gitlab and Mezuro} | |
58 | + | |
45 | 59 | GitLab\footnote{\url{http://gitlab.com/}} is a web-based Git repository |
46 | 60 | manager with wiki pages and issue tracking features. |
47 | 61 | Mezuro\footnote{\url{http://mezuro.org/}} is a platform to collect source code |
48 | 62 | metric to monitor the internal quality of softwares written in C, C++, |
49 | 63 | Java, Python, Ruby, and PHP. GNU Mailman is used for mailing lists. |
50 | 64 | |
65 | +\subsection{System unification} | |
66 | + | |
51 | 67 | \begin{figure}[hbt] |
52 | 68 | \centering |
53 | 69 | \includegraphics[width=\linewidth]{figures/arch.png} |
... | ... | @@ -74,6 +90,8 @@ list posts, or source code. |
74 | 90 | \label{fig:architecture2} |
75 | 91 | \end{figure*} |
76 | 92 | |
93 | +\subsection{Deploy} | |
94 | + | |
77 | 95 | In real, the SPB platform was deployed in 7 virtual machines with different functions, |
78 | 96 | as we can see in Figure \ref{fig:architecture2}. |
79 | 97 | |
... | ... | @@ -102,6 +120,3 @@ static analysis tools on source code stored in repository and provide this data |
102 | 120 | to Prezento. A social network and CMS (Content Manager System) is provided by |
103 | 121 | Noosfero in \textit{social}, and the databases of all tools with a cache |
104 | 122 | service are in \textit{database}. |
105 | - | |
106 | -* Visão geral da implantação pode ser em um nível maior de abstração, mas deixar claro o esforço para automatizar (DevOps -> Chake) | |
107 | -* Empacotamentos | ... | ... |