Compare View

switch
from
...
to
 
Commits (11)
Showing 13 changed files   Show diff stats
opensym2017/content/01-introduction.tex
  Diff comments   View file @d92e024
1 1 \section{Introduction}
2 2 \label{sec:intro}
3 3  
4   -The Brazilian Federal Government has been
5   -improving its processes for software contracting and development. For
6   -instance, in 2003, the recommendation to adopt Free/Libre/Open Source
7   -Software (FLOSS) became a public policy. In 2007, the Brazilian
8   -Government released the Brazilian Public Software Portal
9   -(\textit{Portal do Software Público Brasileiro}, in Portuguese), with the
10   -goal of sharing FLOSS projects developed by, or for, the Brazilian
11   -Government. Additionally, the Brazilian legal instrument on software
12   -contracting (known as IN 04/2012) mandates that public agents must
13   -prioritize solutions available on the SPB Portal. The
14   -acquisition of a proprietary solution must be explicitly justified by
15   -demonstrating that there is no suitable alternative on the SPB Portal.
16   -In 2013, the Brazilian Federal Court issued a ruling
17   -(\textit{Acórdão 2314/2013}) about contracts between the public administration
18   -and suppliers using agile methodologies in software development.
  4 +The Brazilian Government released in the year 2000 the Eletronic Government
  5 +program (eGov) aiming at democratizing information access and improving the
  6 +public provision quality of service and information. In 2003, the Brazilian
  7 +President created a committee for implementation of free
  8 +software(\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/
  9 +DecretoComite}}) and thereafter the Chief of Staff of Brazil sent a circular-
  10 +letter to all Ministries in which the recommendation to adopt Free/Libre/Open
  11 +Source Software (FLOSS\footnote{In this work, the acronym ``FLOSS'' is
  12 +used as a representative for ``Free Software'', ``Open Source Software'' (OSS),
  13 +and``Free/Open Source Software'' (FOSS).}) became a public policy
  14 +(\footnote{\url{http://www.softwarelivre.gov.br/documentos-oficiais/circulardoministro}}).
  15 +In 2007, the Brazilian Public Software Portal (\textit{Portal do Software
  16 +Público Brasileiro}, in Portuguese) was released with the goal of sharing FLOSS projects developed by, or for, the Brazilian Government. Additionally, the
  17 +Brazilian legal instrument on software contracting (known as IN 04/2012)
  18 +mandates that public agents must prioritize solutions available on the SPB
  19 +Portal. The acquisition of a proprietary solution must be explicitly justified
  20 +by demonstrating that there is no suitable alternative on the SPB Portal. In
  21 +2013, the Brazilian Federal Court issued a ruling (\textit{Acórdão 2314/2013})
  22 +about contracts between the public administration and suppliers using agile
  23 +methodologies in software development.
19 24  
20 25 Despite of these legal advancements, in practice, Brazilian government agents
21 26 still do not practice, or even understand,
... ... @@ -47,12 +52,7 @@ project during 30 months, UnB was funded by a grant
47 52 of 2,619,965.00 BRL (about 750,000.00 USD in June 2016)
48 53 from the Federal Government.
49 54  
50   -\begin{figure*}[hbt]
51   - \centering
52   - \includegraphics[width=.9\linewidth]{figures/home-SPB_2.png}
53   - \caption{The new SPB Portal.}
54   - \label{fig:spb}
55   -\end{figure*}
  55 +
56 56  
57 57 The project was developed by a team of three professors, two masters students,
58 58 about fifty undergraduate students (not all of them at the same time,
... ... @@ -70,7 +70,6 @@ FLOSS~\cite{kon2011,deKoenigsberg2008, fagerholm2013, fagerholm2014} and
70 70 Agile~\cite{steghofer2016, harzl2017} practices for Software Engineering
71 71 education.
72 72  
73   -Figure \ref{fig:spb} shows the home page of this integrated platform.
74 73 All the code was developed as open source. The changes we needed in the
75 74 FLOSS tools were implemented by ourselves and
76 75 contributed back to their respective communities. Our
... ... @@ -78,7 +77,7 @@ process was based on agile practices and FLOSS communities interaction.
78 77 We incrementally released five versions of the new SPB
79 78 Portal. The first release (beta) was in September 2014, only 9 months
80 79 from the beginning of the project. The old portal was shut down in
81   -September 2015. Finally, the last version, illustrated in Figure 1, was
  80 +September 2015. Finally, the last version was
82 81 released in June 2016.
83 82  
84 83 In this paper, we present an overview of the new SPB Portal.
... ...
opensym2017/content/02-spb.tex
  Diff comments   View file @d92e024
... ... @@ -4,27 +4,27 @@
4 4 Since the beginning of computing the majority of developers worked in the way
5 5 that we now identify as free software, that is, sharing code openly. This
6 6 openness makes the code available for inspection, modification, and use by any
7   -person or organization \cite{hippel2003,kon2012}.
  7 +person or organization \cite{hippel2003,kon2011}.
8 8  
9 9 The elements that distinguish FLOSS from other types of software are the
10 10 reasoning about the development process, the economic context, the relationship
11 11 between developers and users, as well as the ethical and legal characteristics
12 12 that relate to the software. In the context of FLOSS, user freedom is promoted
13 13 and its development is based on open collaboration and development practices
14   -\cite{meirelles2013}.
  14 +\cite{kon2011}.
15 15  
16 16 From the economic point of view, unlike what happens with proprietary software,
17 17 FLOSS promotes the establishment of several suppliers that can compete with
18 18 each other based on the same software. This stronger competition among
19 19 suppliers brings benefits to users because it gives better assurances regarding
20   -the evolution of the system and induces a reduction in prices \cite{kon2012}.
  20 +the evolution of the system and induces a reduction in prices \cite{kon2011}.
21 21 These freedoms and assurances on software are guaranteed in Brazil by Law
22 22 9610/98 (copyright law). Most of the time, this protection from the law
23 23 complies with the terms conferred by a contract related to certain software.
24 24 This contract is called ``license''. A software license determines a list of
25 25 rights that are given to, and duties that are imposed on a user of the
26 26 software. In particular, what differentiates FLOSS from proprietary software is
27   -just the way they are licensed \cite{sabino2009}. The FLOSS licenses guarantee
  27 +just the way they are licensed \cite{kon2011}. The FLOSS licenses guarantee
28 28 the right to execute, study, adapt, and improve the software. Example of common
29 29 FLOSS licenses are the \textit{GPL (GNU General Public License)}, the Apache
30 30 license, the MIT license, and the BSD license.
... ...
opensym2017/content/03-relatedworks.tex
  Diff comments   View file @d92e024
... ... @@ -2,62 +2,65 @@
2 2 \label{sec:related}
3 3  
4 4 The new SPB platform is a fully integrated environment,
5   -being very advanced in comparison with related projects and initiatives.
  5 +very advanced in comparison with related initiatives.
6 6 %
7 7 For instance, the USA government has a platform designed to improve access to
8   -the federal government developed software\footnote{\url{https://code.gov}}.
9   -Code.gov is an interface to organize the USA government projects and, in short,
10   -make it easy for users and developers to obtain information and access their
11   -source code repositories at GitHub. However, there are not social networking
12   -and CMS features, as well as, other communication resources provided by that
13   -platform.
  8 +federal government software\footnote{\url{https://code.gov}}.
  9 +Code.gov is an interface to organize USA government projects,
  10 +making it easy for users and developers to obtain information and to access their
  11 +source code repositories at GitHub. However, it does not provide social networking
  12 +and CMS features, as well as, other communication resources.
14 13  
15 14 Additionally, there are two initiatives in Europe:
16 15 OSOR\footnote{\url{https://joinup.ec.europa.eu/community/osor}} and
17 16 OW2\footnote{\url{http://ow2.org}}. The Open Source Observatory (OSOR) is a
18 17 community hosted in the JoinUp platform powered by the European Commission.
19   -OSOR aims at exchanging information, experiences and best practices around the
20   -use of FOSS in the public administration. It helps to find a FOSS made
21   -available by other public administrations, providing access to information such
22   -as news, events, studies and solutions related to implementation of open source
23   -software. It also offers forum discussions and community mailing lists, but it
24   -does not have an integrated source code repository manager and for the each
25   -project there is a link to its own external repository (or its tarball file).
  18 +OSOR intends to exchange information, experiences, and best practices around the
  19 +use of FLOSS in the public administration. It supports the discovering of FLOSS projects made
  20 +available by public agencies, providing information about these projects, such
  21 +as news, events, studies, and solutions.
  22 +It also offers discussion forums and community mailing lists. But it
  23 +does not have an integrated source code repository manager, so for each
  24 +project there is a link to its own external repository.
26 25 %
27   -OW2 is a FOSS community to promote the development of FOSS middleware, generic
28   -business applications, cloud computing platforms and foster a community and
29   -business ecosystem. In short, it aims to support the development, deployment
30   -and management of distributed applications with a focus on FOSS middleware and
31   -related development and management tools.
32   -
33   -Moreover, in 2007 the European Comission published a study examined the impact
34   -the development and distribution of FLOSS by public bodies has on eGovernment
35   -services, the economy, and the information society \cite{ghosh2004}. And there
36   -was between 2007 until 2011 the QualiPSo project that aimed at providing FLOSS
37   -users, developers, and consumers, with quality resources and expertise on the
38   -various topics related to FLOSS. The QualiPSo project also had planned to
39   -develop a platform called QualiPSo Factory but it was not fully completed.
  26 +OW2 is an organization that promotes the development, deployment and management
  27 +of FLOSS middleware, generic
  28 +business applications, and cloud computing platforms, besides fostering a community and
  29 +business ecosystem around such projects.
  30 +
  31 +\leo{E aí, o que a OW2 tem a ver com o SPB?}
  32 +
  33 +Moreover, in 2007 the European Comission published a study examining the impact
  34 +that the development and distribution of FLOSS by public agencies have on eGovernment
  35 +services and the economy~\cite{ghosh2004}. And, from 2007 to 2011, there
  36 +was the QualiPSo project, whose objective was to provide FLOSS
  37 +users, developers, and consumers with resources and expertise on FLOSS quality.
  38 +The QualiPSo project also had planned to
  39 +develop a platform called QualiPSo Factory, but it was not fully completed.
40 40  
41 41 Finally, in Latin American there is an initiative based on the SPB project
42   -called ``Software Publico
43   -Regional''\footnote{\url{http://softwarepublicoregionalbeta.net}}. From a
44   -practical point of view, it provides a customized Gitlab instance to share the
45   -source code and documentation of the project from the involved countries.
  42 +called ``Software Público
  43 +Regional''\footnote{\url{http://softwarepublicoregionalbeta.net}}.
  44 +It provides a customized Gitlab instance to share
  45 +source code and documentation of projects developed by the involved countries.
46 46 %
47   -Like Brazil, Chile has its own portal also called ``Software
48   -Publico''\footnote{\url{http://www.softwarepublico.gob.cl}}. Users can create
49   -content in the communities (news items, documents, wiki pages), but source code
  47 +Like Brazil, Chile has its own portal, called ``Repositorio Software
  48 +Público''\footnote{\url{http://www.softwarepublico.gob.cl}}. In the communities of the portal,
  49 +users can create content such as news, documents, and wiki pages, but source code
50 50 repositories are available at the Bitbucket
51 51 platform\footnote{\url{https://bitbucket.org/softwarepublico}}.
52 52  
53 53 The Brazilian government needed to evolve the SPB platform that existed since
54 54 2007. When we started this project, the SPB Portal had about 200 thousand
55   -registered users. For example, we could not just contact these users and ask
56   -them to register an account at Github as well. Moreover, after the Edward
57   -Snowden's case, the Brazilian government approved a specific law decree
58   -(8.135/2013) to rule its communication services, requiring the public
59   -administration to host its information systems to be provided by itself, what
60   -rules out usage of private platforms, specially ones provided by foreign
  55 +registered users. We could not just do something like contacting these users and asking
  56 +them to register an account at Github. Moreover, after the Edward
  57 +Snowden's case, the Brazilian government issued a decree
  58 +(8.135/2013) requiring public
  59 +agencies to host their information systems by themselves,
  60 +avoiding the usage of private platforms, especially the ones provided by foreign
61 61 companies. Therefore, we needed to develop our own solution to cover all the
62 62 requirements, producing a complete governmental integrated platform for
63 63 collaborative software development.
  64 +
  65 +\leo{Me parece que o decreto fala sobre sistemas de comunicação. Acho que o foco era e-mail. Não me parece que o SPB tenha a ver com isso. A menos que se considere algo pelos fóruns... para refletir.}
  66 +
... ...
opensym2017/content/04-researchdesign.tex
  Diff comments   View file @d92e024
1   -\section{Challenging Questions}
  1 +\section{Open Questions}
2 2 \label{sec:researchdesign}
3 3  
4   -In this paper, we aims to share our experience designing and developing the new
  4 +In this paper, we aim to share our experience designing and developing the new
5 5 SPB Portal by reporting, alongside the technical efforts carried out, our
6 6 empirical work process and the lessons learned. In the begins of the new SPB
7 7 Portal project, we had in mind 3 main challenges to overcome, as explained in
... ... @@ -10,7 +10,7 @@ the following open questions.
10 10 \begin{description}
11 11  
12 12 \item [Q1:] \textit{Which strategy could be used to integrate several existing
13   -FOSS tools to promote the collaborative software development?}
  13 +FLOSS tools to promote the collaborative software development?}
14 14 %
15 15 Based on an extensive list of functional requirements defined by the Brazilian
16 16 Federal Government, we selected some FLOSS systems to form our solution,
... ... @@ -20,7 +20,16 @@ were fully aware that we would need to improve those systems in order to
20 20 provide the rest. We were also convinced that it would be impossible to provide
21 21 all of those requirements with a single tool.
22 22  
23   -\item [Q2:] \textit{How to introduce the FLOSS collaborative and agile
  23 +\item [Q2:] \textit{How to involve students in real-world projects interacting with
  24 +real customers?}
  25 +%
  26 +Our team was mainly composed of software engineering undergraduate
  27 +students, who had the opportunity to interact with senior developers and
  28 +designers on the team, as well as with the team of technicians and
  29 +managers from the Brazilian Government, and the management team from
  30 +UnB. For the majority of the students, this was a first professional experience. We have define an approach to involve the undergraduate students in this project with a central role in our development process.
  31 +
  32 +\item [Q3:] \textit{How to introduce the FLOSS collaborative and agile
24 33 practices to governmental development process?}
25 34 %
26 35 The Brazilian government works based on a very traditional way regarding
... ... @@ -31,8 +40,4 @@ certain expectations about the development of project according to RUP
31 40 approaches, what not match our work based on agile and FLOSS practices. We have
32 41 created strategies that would support different these organizational cultures.
33 42  
34   -\item [Q3:] \textit{How to involve undergraduate students in real-world projects?}
35   -%
36   -
37   -
38 43 \end{description}
... ...
opensym2017/content/05-requirements.tex
  Diff comments   View file @d92e024
... ... @@ -44,7 +44,7 @@ requirements were:
44 44  
45 45  
46 46 There were other requirements based on the experience of the IT
47   -stakeholders from the Brazilian government and from the Brazilian FOSS
  47 +stakeholders from the Brazilian government and from the Brazilian FLOSS
48 48 community (that UnB and USP were representing too in this project). The
49 49 new platform would only work properly if there is a unique
50 50 authentication to use the provided tools. Additionally, a unified
... ... @@ -71,7 +71,7 @@ web-based source code static analysis monitor. By analyzing all of these
71 71 requirements, we proposed the technological requirements
72 72 illustrated in Figure \ref{fig:requirements} to guide the development of
73 73 the new SPB platform. In other words, we have designed the SPB evolution
74   -project based on existing FOSS tools. However, the integration of
  74 +project based on existing FLOSS tools. However, the integration of
75 75 several existing systems that were already implemented in different
76 76 programming languages and frameworks, adding features such as a
77 77 centralized authentication, unified interface, and a search engine, as
... ...
opensym2017/content/06-architecture.tex
  Diff comments   View file @d92e024
... ... @@ -5,13 +5,13 @@ From the point of view of the architecture, two main requirements was
5 5 brought by the Brazilian Federal Government for the new platform:
6 6  
7 7 \begin{enumerate}
8   - \item \textit{Integrate existing FOSS systems}, with minimal differences from
  8 + \item \textit{Integrate existing FLOSS systems}, with minimal differences from
9 9 their original versions;
10 10 \item \textit{Provide a consistent user interface} across the different
11 11 systems, as well as centralized authentication.
12 12 \end{enumerate}
13 13  
14   -Adopting existing FOSS systems and minimizing locally-made changes had
  14 +Adopting existing FLOSS systems and minimizing locally-made changes had
15 15 the objective of being able to upgrade to newer versions of the original
16 16 software, benefiting from improvements and maintenance done by the
17 17 existing project communities. Providing a consistent user interface on
... ... @@ -76,7 +76,7 @@ home pages and documentation, and contact forms.
76 76 \subsection{Gitlab}
77 77  
78 78 GitLab\footnote{\url{http://gitlab.com}} is a web-based Git repository
79   -manager with wiki pages and issue tracking features. Gitlab is a FOSS
  79 +manager with wiki pages and issue tracking features. Gitlab is a FLOSS
80 80 platform and focuses on delivering a holistic solution that will see
81 81 developers from idea to production seamlessly and on a single platform.
82 82  
... ... @@ -131,7 +131,7 @@ as we can see in Figure \ref{fig:architecture2}.
131 131  
132 132 \begin{figure*}[hbt]
133 133 \centering
134   - \includegraphics[width=.8\linewidth]{figures/arch3.png}
  134 + \includegraphics[width=\linewidth]{figures/arch3.png}
135 135 \caption{Instanciation view of the SPB architecture.}
136 136 \label{fig:architecture2}
137 137 \end{figure*}
... ...
opensym2017/content/09-process.tex
  Diff comments   View file @d92e024
... ... @@ -21,7 +21,7 @@ communication to be online.
21 21  
22 22 In short, our work process was based on open and collaborative software
23 23 development practices. The development process was defined based on the
24   -adaptation of different agile and FOSS communities practices, highlighting the
  24 +adaptation of different agile and FLOSS communities practices, highlighting the
25 25 high degree of automation resulting from DevOps practices. Thus, the work
26 26 process was executed in a cadenced and continuous way.
27 27  
... ...
opensym2017/content/11-conclusion.tex 0 → 100644
  Diff comments   View file @d92e024
... ... @@ -0,0 +1,220 @@
  1 +\section{Conclusion}
  2 +\label{sec:conclusion}
  3 +
  4 +In this work, we presented and discussed issues experienced during a government-funded
  5 +project, in partnership with the University of Brasilia and the University of
  6 +São Paulo, to evolve the Brazilian Public Software Portal.
  7 +Its contributions are twofold. First, we present the strategy used to develop
  8 +and to deliver an unprecedented platform to Brazilian government. Second,
  9 +based on the results of the SPB Portal project, we point out that it is
  10 +possible to mitigate conflicts experienced in the development environment
  11 +and to conciliate governmental and academy cultures. To summarize our main contributions, we answered in this section the three open questions those guided this paper.
  12 +
  13 +
  14 +\textbf{Q1 -- Which strategy could be used to integrate several existing
  15 +FLOSS tools to promote the collaborative software development?}
  16 +%
  17 +The SPB portal integrates more than 10 FLOSS tools and provides several features,
  18 +such as social network, mailing list, version control, content management and
  19 +source code quality monitoring. Concerned with the platform sustainability and
  20 +maintainability, the aforementioned 10 FLOSS tools were integrated with minimum
  21 +differences from their official versions and the new developed features were
  22 +sent upstream to ensure an alignment between the portal systems and their
  23 +respective official versions. In the integration process, the main softwares
  24 +were identified, specific teams were formed to work with each one of them
  25 +and each team was composed of students with different levels of skills and at
  26 +least one senior professional.
  27 +
  28 +\textbf{Q2 -- How to involve students in real-world projects interacting with
  29 +real customers?}
  30 +%
  31 +In terms of mitigating conflicts, we tried to show that, as long as the
  32 +university can provide a healthy and challenging environment to its students,
  33 +one may conciliate studies and professional training in universities.
  34 +%
  35 +The students interacted with professionals of diverse fields of expertise, and they were
  36 +able to participate in all levels of the software development process.
  37 +This contributed to a great learning opportunity.
  38 +%
  39 +In our work process, based on open and collaborative software development
  40 +practices, students could negotiate their work schedule as well as count on IT
  41 +professionals to solve development issues.
  42 +%
  43 +Among the students, we have defined coaches for each team and a meta-coach
  44 +(coach of whole project). All coaches, together with professors, have
  45 +intermediated the communication between client (Ministry of Planning of Brazil)
  46 +and the rest of the group.
  47 +After the end of the project, some students successfully
  48 +embraced opportunities in public and private sectors, within national borders
  49 +and abroad. Some other students went further and started their own companies.
  50 +
  51 +\textbf{Q3 -- How to introduce the FLOSS collaborative and agile
  52 +practices to governmental development process?}
  53 +With some adaptations, what we called the ``translation processes'', it is feasible
  54 +to conciliate agile methodologies and FLOSS practices to develop software to
  55 +governmental organizations with functional hierarchical structures that use
  56 +traditional development paradigm.
  57 +%
  58 +Aiming at reducing client questions about workconclusion, a DevOps front was
  59 +created to automate all deploy process and also to work in continuous
  60 +delivery. The government was brought to our work environment and interacted
  61 +with our management and communication tools. For the project success, we
  62 +focused on providing a friendly working environment as well as on showing to
  63 +governmental agents another way to interact with the FLOSS community and the
  64 +university.
  65 +
  66 +\subsection{Lessons Learned}
  67 +\label{sec:lessons}
  68 +
  69 +From the answers of our initial open questions, we can also highlighted six lessons learned to make easier to share our experience during the development of the new SPB Portal.
  70 +
  71 +\textbf{L1 -- The participation of experienced professionals is crucial to
  72 +the success of the project.}
  73 +One important factor for the students was the composition of the teams
  74 +with the participation of experienced professionals.
  75 +%
  76 +On the technical side, the senior developers and designers would handle
  77 +the more difficult technical decisions, creating a work environment
  78 +where the students could develop their skills in a didactic way without
  79 +pressure.
  80 +%
  81 +On the management side, the active participation of professors -- who
  82 +are, in the end, the ones responsible for the project -- is crucial to
  83 +make sure students participation is conducted in a healthy way, and it is an
  84 +instance of leading by example.
  85 +
  86 +\textbf{L2 -- A balanced relationship between academia and industry.}
  87 +The experience of the SPB project led UnB to develop a work style which
  88 +proved to be appropriate for an educational environment that brings
  89 +academia and industry together.
  90 +%
  91 +The highest priority from the university's point of view is the
  92 +students. Considering this, the activities of the project were never
  93 +prioritized to the detriment of classes and other pedagogical
  94 +activities. In summary, we had students working at different times, part
  95 +time, remotely or locally, always respecting their individual
  96 +conditions, but doing the work in a collective, collaborative and open
  97 +way.
  98 +%
  99 +And even under a potentially adverse environment, the project delivered
  100 +the desired solution with success.
  101 +%
  102 +At the end of the project, we noted that the skills developed by the
  103 +students were at the software engineering state of the art.
  104 +After the project ended, we had team members successfully embracing
  105 +opportunities in public, private, national, and international
  106 +organizations, in addition to those students that
  107 +opened their own companies.
  108 +
  109 +\textbf{L3 -- Managing different organizational cultures.}
  110 +In the beginning of the project, the Brazilian Government stakeholders
  111 +had certain expectations about the project development that, let's
  112 +say, didn't exactly match our work style based on agile and FLOSS practices.
  113 +%
  114 +We had to develop strategies that would support these different
  115 +organizational cultures. The
  116 +MP is organized in a functional hierarchical organizational structure,
  117 +typically adopting a traditional development paradigm. Therefore, we had to
  118 +create a translation process between our team and the MP managers who
  119 +managed the project on their side assuming a traditional waterfall
  120 +process.
  121 +
  122 +\textbf{L4 -- Managing higher-level and lower-level goals separately.}
  123 +During the initial phase of the project, the MP team has brought
  124 +strategic discussions to technical/operational meetings that
  125 +were supposed to be about practical technical decisions.
  126 +%
  127 +This produced a highly complex communication and management environment,
  128 +overloading the professors because they were supposed
  129 +for maintaining the MP strategy synchronized with the
  130 +implementation plans of the development team. This was hard, especially because the
  131 +aforementioned cultural mismatch. Mixing both concerns in the same
  132 +discussions caused confusion on both sides.
  133 +%
  134 +From the middle of the project we were able to keep those
  135 +concerns separated, what eased the work of everyone involved.
  136 +
  137 +\textbf{L5 -- Living with ill-advised political decisions.}
  138 +At the initial phases of the project, by political and personal
  139 +motivation, the main stakeholders from the Brazilian government imposed
  140 +the use of Colab to guide the development of the new SPB platform. Our
  141 +team was totally against the idea because we already knew that Colab was
  142 +a very experimental project and its adoption could dramatically increase
  143 +the project complexity. Even though we provided technical reasons to
  144 +not utilize Colab, the MP was adamant and we had to manage this problem. We did massive changes to
  145 +Colab, and at the end of the project we have completely rewritten it to make
  146 +it stable. It is important to notice that the MP compelled us to accept
  147 +a technical decision based only on political interests, without considering
  148 +all the resources that would be spent due to that decision. At the end
  149 +of the project, we verified that Colab indeed consumed a vast amount of
  150 +the budget and increased the project complexity. At the end of the
  151 +project and after our analysis on the decision made by the MP, we
  152 +understand that MP managers are capable of ignoring technical reasons
  153 +in favor of political decisions.
  154 +
  155 +\textbf{L6 -- Consider sustainability from the beginning.}
  156 +In the process of deploying the SPB platform in the MP infrastructure we had
  157 +to interact with the MP technicians. We did several workshops, training
  158 +and a meticulous documentation describing all the required procedures to
  159 +update the platform, however, we realized that the MP technicians
  160 +constantly avoid the responsibility. After noticing the aforementioned
  161 +situation, we organized a DevOps team that completely automated all
  162 +the deployment procedure. We simplified all the platform deployment to a
  163 +few steps: (1) initial configurations (just ssh configuration) and (2)
  164 +the execution of simple commands to completely update the platform. By
  165 +the end of the project, we observed that the MP technicians invariably
  166 +still depended on our support to update the platform even with all the
  167 +automation provided by us. We were sadly left with a feeling of
  168 +uncertainty about the future of the platform after the project ended. In
  169 +hindsight, we realize that the MP dedicated system analysts and
  170 +managers to the project, but not operations technicians. The later
  171 +should have been more involved with the process so they could at least be
  172 +comfortable in managing the platform infrastructure.
  173 +
  174 +
  175 +\textbf{Final remarks and future work}
  176 +
  177 +The portal is available at \url{softwarepublico.gov.br}. All
  178 +documentation, including detailed architecture and operation manuals are
  179 +also available\footnote{\url{https://softwarepublico.gov.br/doc/}
  180 +(in Portuguese only at the moment)}.
  181 +%
  182 +All the integrated tools are FLOSS and our contributions were published
  183 +in open repositories, available on the SPB Portal itself. We also
  184 +contributed these features back to the respective communities, which
  185 +benefits both those communities and us, since we can share future
  186 +development and maintenance effort with other organizations that
  187 +participate in these projects.
  188 +
  189 +Future work should use data produced by the project to validate and evaluate
  190 +how the used FLOSS and Agile practices have impacted the students and the
  191 +governmental development process. For this, we would conduce a \textit{postmortem}
  192 +analysis using the project open data and a survey targeting the involved actors.
  193 +
  194 +%===========
  195 +% Conclusion
  196 +%===========
  197 +
  198 +% * Gestão dos recursos: Fizemos mais por menos (2.6M de 3.2M) --- sem os dados
  199 +%% (escopo, custo, tempo e qualidade) bem discutidos é difícil sustentar essa
  200 +%% afirmação, embora eu e Paulo consigamos perceber isso.
  201 +
  202 +
  203 +%* utilização do projeto para formação de recursos humanos (alunos)
  204 +
  205 +%* dados da verificação dos repositório para a análise da qualidade dos código via Mezuro e CodeClimate
  206 +
  207 +%* o que achamos que irá acontecer com o SPB no futuro breve (acabar)
  208 +
  209 +%* 69 projetos marcados como SPB, de 81 no total na plataforma.
  210 +
  211 +%* 47\% é desenvolvido em PHP.
  212 +
  213 +% foi constatado que aproximadamente 75\% dos softwares \textbf{não} possuem seus códigos-fonte versionados nesta ferramenta. Realizado algumas pesquisas, foi encontrado o código-fonte em outros serviços (Github, Bitbucket).
  214 +
  215 +% Foram adicionados 31 softwares do SPB em ambas as ferramentas (Mezuro e Code Climate), desenvolvidos em PHP e Python. Estas adições resultaram na análise descrita nos próximos parágrafos. No Mezuro, dos 31 softwares adicionados, somente 4 obtiveram sucesso na avaliação. No Code Climate, 16 softwares realizaram a \textit{build} da avaliação com sucesso. Nos que falharam, alguns dos erros foram encontrados em três das \textit{engines}: ora em \textit{duplication}, ora na \textit{phpmd}, ora na \textit{eslint}.
  216 +
  217 +% também foram inseridos no Mezuro para avaliação, 5 projetos dos 17 desenvolvidos em Java, com o intuito de ser um contraponto ao Code Climatepor esta não compreender a análise de projetos em Java, C, ou C++. Infelizmente nenhuma das \textit{builds} resultou em resultados concretos.
  218 +
  219 +%* Debater economia de recursos em orgão públicos
  220 +
... ...
opensym2017/spb.bib
  Diff comments   View file @d92e024
... ... @@ -8,7 +8,7 @@
8 8 }
9 9  
10 10 @inproceedings{kon2011,
11   - author = {Kon, Fabio and Meirelles, Paulo and Lago, Nelson and de Azevedo Terceiro, Antonio Soares and Chavez, Christina and Mendonça, Manoel G.},
  11 + author = {Kon, Fabio and Meirelles, Paulo and Lago, Nelson and Terceiro, Antonio and Chavez, Christina and Mendonça, Manoel},
12 12 booktitle = {SBES},
13 13 ee = {http://doi.ieeecomputersociety.org/10.1109/SBES.2011.19},
14 14 interhash = {cb27e5a1aa6816e36e92a1d3b9011615},
... ... @@ -96,18 +96,18 @@
96 96 }
97 97  
98 98 @article{ducheneaut2005,
99   - author = {Ducheneaut, N.},
100   - date-added = {2008-01-20 20:37:12 -0800},
101   - date-modified = {2008-01-20 20:37:12 -0800},
102   - interhash = {8cfe59af50070854ba2a8383ad75d492},
103   - intrahash = {8a17826d55e41bcc35851ee8493f0b66},
104   - journal = {Computer Supported Cooperative Work (CSCW)},
105   - number = {4},
106   - pages = {323--368},
107   - publisher = {Springer},
108   - title = {{Socialization in an Open Source Software Community: A Socio-Technical Analysis}},
109   - volume = {14},
110   - year = {2005}
  99 + author = {Nicolas Ducheneaut},
  100 + title = {Socialization in an Open Source Software Community: {A} Socio-Technical Analysis},
  101 + journal = {Computer Supported Cooperative Work},
  102 + volume = {14},
  103 + number = {4},
  104 + pages = {323--368},
  105 + year = {2005},
  106 + url = {https://doi.org/10.1007/s10606-005-9000-1},
  107 + doi = {10.1007/s10606-005-9000-1},
  108 + timestamp = {Sun, 28 May 2017 13:18:01 +0200},
  109 + biburl = {http://dblp.uni-trier.de/rec/bib/journals/cscw/Ducheneaut05},
  110 + bibsource = {dblp computer science bibliography, http://dblp.org}
111 111 }
112 112  
113 113 @article{fagerholm2013,
... ... @@ -179,12 +179,17 @@
179 179 }
180 180  
181 181 @article{bobr2003,
182   - author = {Booch, Grady and Brown, Alan W.},
183   - journal = {Advances in Computers},
184   - pages = {1--27},
185   - title = {Collaborative Development Environments},
186   - volume = {59},
187   - year = {2003},
  182 + author = {Grady Booch and Alan W. Brown},
  183 + title = {Collaborative Development Environments},
  184 + journal = {Advances in Computers},
  185 + volume = {59},
  186 + pages = {1--27},
  187 + year = {2003},
  188 + url = {https://doi.org/10.1016/S0065-2458(03)59001-5},
  189 + doi = {10.1016/S0065-2458(03)59001-5},
  190 + timestamp = {Sat, 20 May 2017 00:22:37 +0200},
  191 + biburl = {http://dblp.uni-trier.de/rec/bib/journals/ac/BoochB03},
  192 + bibsource = {dblp computer science bibliography, http://dblp.org},
188 193 publisher={Elsevier}
189 194 }
190 195  
... ... @@ -234,31 +239,42 @@
234 239 year = {2008}
235 240 }
236 241  
237   -
238 242 @book{refactoring,
239   - title={Refactoring for software design smells: Managing technical debt},
240   - author={Suryanarayana, Girish and Samarthyam, Ganesh and Sharma, Tushar},
241   - year={2014},
242   - publisher={Morgan Kaufmann}
243   -}
  243 + author = {Suryanarayana, Girish and Samarthyam, Ganesh and Sharma, Tushar},
  244 + title = {Refactoring for Software Design Smells: Managing Technical Debt},
  245 + year = {2014},
  246 + isbn = {0128013974, 9780128013977},
  247 + edition = {1st},
  248 + publisher = {Morgan Kaufmann Publishers Inc.},
  249 + address = {San Francisco, CA, USA},
  250 +}
244 251  
245   -@article{mezuro_oss,
  252 +@inproceedings{mezuro_oss,
246 253 title={Mezuro: Understanding source code metrics},
247   - author={Guedes and Meirelles and Manzo and Camarinha},
248   - journal={International Conference on Open Source Systems},
249   - volume={01},
  254 + author={Guedes, Dylan and Meirelles, Paulo and Manzo, Rafael and Camarinha, Diego},
  255 + booktitle={Poster Proceedings of the 13th International Conference on Open Source Systems},
  256 + pages={15--18},
250 257 year={2017},
251   - publisher={OSS}
  258 + url = {http://oss2017.lifia.info.unlp.edu.ar/blog/wp-content/uploads/2017/03/PosterProceedings.rar},
  259 + organization={OSS Conference}
252 260 }
253 261  
254 262 @article{collaboration_tools,
255   - title={Collaboration tools for global software engineering},
256   - author={Lanubile, Filippo and Ebert, Christof and Prikladnicki, Rafael and Vizca{\'\i}no, Aurora},
257   - journal={IEEE software},
258   - volume={27},
259   - number={2},
260   - year={2010},
261   - publisher={IEEE}
  263 + author = {Filippo Lanubile and
  264 + Christof Ebert and
  265 + Rafael Prikladnicki and
  266 + Aurora Vizca{\'{\i}}no},
  267 + title = {Collaboration Tools for Global Software Engineering},
  268 + journal = {{IEEE} Software},
  269 + volume = {27},
  270 + number = {2},
  271 + pages = {52--55},
  272 + year = {2010},
  273 + url = {https://doi.org/10.1109/MS.2010.39},
  274 + doi = {10.1109/MS.2010.39},
  275 + timestamp = {Thu, 08 Jun 2017 09:06:58 +0200},
  276 + biburl = {http://dblp.uni-trier.de/rec/bib/journals/software/LanubileEPV10},
  277 + bibsource = {dblp computer science bibliography, http://dblp.org}
262 278 }
263 279  
264 280 @article{parker2007wiki,
... ... @@ -273,10 +289,20 @@
273 289 }
274 290  
275 291 @inproceedings{opensourcestyle,
276   - title={Open source-style collaborative development practices in commercial projects using github},
277   - author={Kalliamvakou, Eirini and Damian, Daniela and Blincoe, Kelly and Singer, Leif and German, Daniel M},
278   - booktitle={Proceedings of the 37th International Conference on Software Engineering-Volume 1},
279   - pages={574--585},
280   - year={2015},
281   - organization={IEEE Press}
  292 + author = {Eirini Kalliamvakou and
  293 + Daniela E. Damian and
  294 + Kelly Blincoe and
  295 + Leif Singer and
  296 + Daniel M. Germ{\'{a}}n},
  297 + title = {Open Source-Style Collaborative Development Practices in Commercial
  298 + Projects Using GitHub},
  299 + booktitle = {37th {IEEE/ACM} International Conference on Software Engineering,
  300 + {ICSE} 2015, Florence, Italy, May 16-24, 2015, Volume 1},
  301 + pages = {574--585},
  302 + year = {2015},
  303 + url = {https://doi.org/10.1109/ICSE.2015.74},
  304 + doi = {10.1109/ICSE.2015.74},
  305 + timestamp = {Sun, 04 Jun 2017 01:00:00 +0200},
  306 + biburl = {http://dblp.uni-trier.de/rec/bib/conf/icse/KalliamvakouDBS15},
  307 + bibsource = {dblp computer science bibliography, http://dblp.org}
282 308 }
... ...
opensym2017/spb.tex
  Diff comments   View file @d92e024
... ... @@ -187,8 +187,7 @@
187 187 \input{content/08-ux}
188 188 \input{content/09-process}
189 189 \input{content/10-contributions}
190   -\input{content/11-lessons}
191   -\input{content/12-conclusion}
  190 +\input{content/11-conclusion}
192 191  
193 192 %------------------------------------------------------------------------------
194 193 \bibliographystyle{SIGCHI-Reference-Format}
... ...
sbqs2017/reviews.txt 0 → 100644
  Diff comments   View file @d92e024
... ... @@ -0,0 +1,100 @@
  1 +Reviewer 1
  2 +---
  3 +
  4 +Resumo (Summary): Uma breve descrição do artigo
  5 +
  6 +O artigo apresenta uma visao geral da experiência do desenvolvimento do novo Portal SPB, explicando algumas questões técnicas relacionadas e algumas lições aprendidas durante a execução do projeto.
  7 +
  8 +Pontos Fortes (Paper Strengths): Quais os pontos positivos do artigo?
  9 ++ desenvolvimento de uma plataforma computacional de uso geral em larga escala capaz de gerar um nicho de desenvolvimento de software nacional
  10 +
  11 +Pontos Fracos (Paper Weakness): Quais os aspectos negativos do artigo?
  12 +- falta de embasamento teórico mais detalhado (incluindo referências)
  13 +- falta de visões compartilhadas entre equipe desenvolvedora e equipe consumidora (o viés é da equipe desenvolvedora)
  14 +- questões de qualidade de escrita (informal e de Português)
  15 +- lições aprendidas mais específicas do processo de desenvolvimento
  16 +
  17 +Comentários para os Autores (Comments to Authors): Comentários e sugestões para o autor melhorar a qualidade do artigo
  18 +Alguns comentários gerais e específicos:
  19 +- nome do LAPPIS: "Sofware" -> Software
  20 +- "30 months work on the SPB project" -> revisar Inglês
  21 +- "Nesse contexto, criamos uma plataforma baseada na integração e evolução de ferramentas livres existentes" -> este fragmento informa que os autores foram os desenvolvedores do Portal SPB. No entanto, o MP não participou do desenvolvimento com estudos, sobretudo considerando a visão ágil? E sobre a autoria da versão antiga, ou legado do SPB? Nada foi aproveitado de modo que a autoria do novo portal seja somente resultante deste projeto?
  22 +- A última frase do do primeiro parágrafo da introdução, que menciona sobre metodologias ágeis no governo, está desconexa do restante do parágrafo, que trata de software público. Sugiro revisar isso e melhor argumentar o contexto que os autores pretendem trazer para motivar o relato.
  23 +- "as metodologias de desenvolvimento de software livre ou ágeis, isto é, os métodos colaborativos e empíricos de desenvolvimento de software" -> esta equivalência não é adequada, pois assume-se que metodologias de desenvolvimento de software livre ou ágeis = métodos colaborativos e empíricos de desenvolvimento de software. Além disso, métodos empíricos, que acredito fazer alusão à aplicação da base de experimentação, parece não fazer sentido algum nesta sentença.
  24 +- "vários problemas técnicas." -> técnicos
  25 +- "nao estava mias sendo desenvolvido" -> mais
  26 +- Figura 1 é inserida antes de ser citada no texto e há um espaço em branco após a figura, fora do formato da SBC.
  27 +- "lançamos uma plataforma sem precedentes para
  28 +o governo brasileiro aplicar métodos colaborativos de desenvolvimento de software." -> qual a novidade em relação a outros conjuntos ou kits ou sets de ferramentas que podem ser utilizados para o desenvolvimento de software? Por que "sem precedente"? Qual o impacto prático do Portal SPB? Quais as estatísticas de uso? Qual é a sua motivação ou alvo?
  29 +- "desde o início da computação, a maioria dos desenvolvedores trabalhavam da maneira que agora identificamos como software livre" -> sentença confusa.
  30 +- "Essa forte concorrência entre os fornecedores traz benefícios para os usuários, porque dá melhores garantias em relação à evolução do sistema, levando a uma redução dos custos" -> isso se dá em um cenário de software livre ou FOSS? Existe esta concorrência tão significativa como no mercado de software tradicional/proprietário? Este trecho carece de melhores explicações.
  31 +- "Haviam outros requisitos" -> Havia
  32 +- "funcionaria corretamente se houver" -> se houvesse
  33 +- "Assim, no primeiro momento, desejamos disponibilizar uma versao inicial que poderia substituir o antigo portal SPB" -> mas já não foi disponibilizada? Ainda há módulos a serem construídos? E quanto aos estudos de usabilidade e ajustes pós-implantação?
  34 +- "Analisando todos esses requisitos, criamos o projeto de evolução do SPB baseado em ferramentas de software livre existentes. No entanto, a
  35 +integração de vários sistemas existentes que já foram implementados em diferentes linguagens de programação e arcabouços" ... "o portal SPB foi composto por mais de dez sistemas" -> como se deu a escolha destas ferramentas e/ou sistemas? Que estudos embasaram esta escolha? Foi realizado algum survey com a comunidade de desenvolvedores e usuários? Foi definido pelo cliente? Ou foi uma seleção por conveniência ou experiência da equipe? Isso não está claro.
  36 +- "Nós também estávamos convencidos de que seria impossível fornecer todas as funcionalidade com uma única ferramenta" -> como remediar este cenário? A comunidade de desenvolvedores não poderia participar da evolução do próprio portal? Uma vez que o portal tem por finalidade apoiar o desenvolvimento de software público e com foco em FOSS, por que não foi preparado para que a própria comunidade pudesse manter e evoluir a sua estrutura?
  37 +- "sem confundí-lo" -> confundi-lo
  38 +- "Temos 3 grandes" -> Temos três grandes
  39 +- "No novo portal SPB, cada software tem um conjunto padrao de páginas e ferramentas. Além de acessar as páginas de suporte (como FAQ e guia de instalação) dentro da plataforma" -> deve ser uma frase só que está quebrada em duas. Revisar.
  40 +- "sua evolução pelos resultados métricas" -> ?
  41 +- "cada análise métrica sao públicos" -> ?
  42 +- "informação do no portal SPB" -> do portal
  43 +- "a arquitetura ... re-projetada para fornecer uma navegação
  44 +transparente e alcançar usuários com perfis diferentes" -> o que se entende por navegação transparente? Isso não está claro. E que perfis de usuários são esses?
  45 +- "Um processo de harmonização foi empregado nos modelos de interação de cada ferramenta para reduzir a curva de aprendizado" -> o que se entende por estes modelos de interação? Novamente, não está claro e não permite ao leitor compreender como foi a avaliação com usuário.
  46 +- "Além disso, ferramentas de origens diferentes, que em muitos casos fornecem funcionalidades semelhantes, o que nos levou ao desenvolvimento de uma interface unificada." -> Reescrever.
  47 +- "dos ferramentas" -> das
  48 +- "Nós nos engajamos em seu desenvolvimento" -> ?
  49 +- A utilização de DevOps deveria ser melhor introduzida e explicada.
  50 +- "Como se pode ver, o projeto tinha muitos tipos diferentes de stakeholders que tinham de ser organizados e sincronizados." -> Quais? Por quê?
  51 +- "Essa dinâmica incentivamos aos demais alunos" -> Reescrever.
  52 +- "Entretanto, depois de meses de trabalho e mostrando resultados, eles deixaram de resistir aos métodos empíricos." -> que métodos são estes? O que os autores realmente entendem por este conceito?
  53 +- "... o mapeamento sistemático das ..." -> ?
  54 +- "Observe que este fluxo de trabalho proporcional a nós e aos envolvidos pelo governo brasileiro uma rastreabilidade completa de uma visao de alto nível de cada funcionalidade (requisito) até para o nível mais baixo (código)." -> Reescrever.
  55 +- A seção 8 não traz as conclusões frente ao objetivo do artigo, introduzindo diretamente apenas a listagem de lições aprendidas, sem de fato trazer um balanço sumarizado da experiência relatada. Além disso, muitas lições aprendidas não trazem novidade para a comunidade: quais são as lições específicas e que não foram reportadas até o momento?
  56 +- "No final do projeto, notamos que as habilidades desenvolvidas
  57 +pelos alunos estavam no estado da arte da prática de engenharia de software. Depois que o projeto terminou, tivemos membros da equipe que abraçaram com sucesso oportunidades em organizações públicas, privadas, nacionais e internacionais, além dos estudantes que entraram no empreendedorismo e abriram suas próprias empresas." -> este é um exemplo de trechos trazidos nas lições aprendidas com uma discussão rasa e informal, sem dados mais concretos sobre os pontos colocados. Qual o resultado concreto para os alunos envolvidos? Além disso, há diversos trabalhos sobre a relação academia-indústria e o relato não se preocupou em fazer menção. Para além de diversos trabalhos já publicados em toda a história do SBQS, sugiro este do SBES 2012: "Strategic Alignment between Academy and Industry: A Virtuous Cycle to Promote Innovation in Technology".
  58 +- "Nao aceitar más decisoes técnicas por questoes políticas." -> ao descrever essa "lição aprendida", os autores mencionam que o Colab foi reescrito para ser um "sistema de sistema". Acredito que este conceito deveria ser utilizado com mais atenção. Buscar trabalhos publicados pelo grupo da Elisa Nakagawa e José Maldonado (USP-SC). Ainda nesta lição, os autores apontam que "o MP os obrigou a aceitar uma questão técnica". Esta afirmação é forte. Uma vez que o relato é de autoria somente de um lado do projeto e se a lição é não aceitar, como proceder então com uma avaliação mais crítica? Acredito que o relato deveria envolver os dois lados do projeto (desenvolvedor e adquirente), especialmente neste caso (Portal SPB).
  59 +- "Considerar a sustentabilidade desde o início." -> o seguinte trecho desta lição precisa de mais explicações: "os técnicos MP constantemente evitavam essa responsabilidade". Outro ponto está no trecho "nós organizamos uma equipe de DevOps específica". Por que DevOps é colocado como uma bala de prata?
  60 +- "observamos que os técnicos MP sempre dependiam do nosso apoio para atualizar a plataforma, mesmo com toda a automaçao fornecida por nós. Ficamos tristemente com um sentimento de incerteza sobre o futuro da plataforma após o término do projeto." -> ajustar para "técnicos do MP". Além disso, este trecho não traz uma lição de caráter técnico (ver o uso de "tristemente"). Como se configurou este cenário? Qual o contexto? Por que a incerteza, se o projeto terminou? Quais os próximos passos? Novamente, a experiência carece de uma visão compartilhada entre as partes envolvidas.
  61 +- "da infra-estrutura plataforma" -> da infraestrutura da plataforma (?)
  62 +review
  63 +
  64 +Reviewer 2
  65 +---
  66 +
  67 +Resumo (Summary): Uma breve descrição do artigo
  68 +Apresentação da arquitetura, do processo de trabalho e das lições aprendidas no desenvolvimento de um portal de software público brasileiro.
  69 +
  70 +Pontos Fortes (Paper Strengths): Quais os pontos positivos do artigo?
  71 +Trata-se de uma experiência em um grande projeto de bastante relevância que envolveu equipes mistas.
  72 +
  73 +Pontos Fracos (Paper Weakness): Quais os aspectos negativos do artigo?
  74 +Não fica evidente a contribuição com a qualidade de software.
  75 +Algumas opiniões muito pessoais, há que ter uma certa impessoalidade em determinadas afirmações.
  76 +Algumas descobertas que já são de conhecimento comum.
  77 +
  78 +Comentários para os Autores (Comments to Authors): Comentários e sugestões para o autor melhorar a qualidade do artigo
  79 +Realizar uma revisão de gramática geral.
  80 +Reescrever determinadas opiniões que estão muito pessoais, como por exemplo, "tristemente"
  81 +Acho que faltou um survey para coletar as impressões dos envolvidos neste processo. Acredito que as constatações ficariam menos subjetivas.
  82 +
  83 +Reviewer 3
  84 +---
  85 +Resumo (Summary): Uma breve descrição do artigo
  86 +O artigo apresenta a "historia" do desenvolvimento da nova plataforma do Portal do Software Publico Brasileiro.
  87 +
  88 +Pontos Fortes (Paper Strengths): Quais os pontos positivos do artigo?
  89 +- artigo bem escrito e super agradável de ler
  90 +- lições aprendidas mostra o grande valor da parceria academia-industria
  91 +- a apresentação do portal dos requisitos definidos às grandes funcionalidades desenvolvidas
  92 +
  93 +Pontos Fracos (Paper Weakness): Quais os aspectos negativos do artigo?
  94 +- no resumo na introdução é comentado que foi utilizado práticas ageis. No entanto. nada é apresentado durante o artigo. Por exemplo, nenhuma lição aprendida sobre a aplicação da metodologia ágil é apresentada
  95 +- a ausência de uma parte mais técnica do ponto de vista de desenvolvimento e avaliação da qualidade do sistema. O artigo se limita a descriçãod das funcionaldiades.
  96 +- ausência da discussão sobre manutenção evolutiva.
  97 +- não se explica como foi avaliado o sistema (seria interessante para a comunidade SBQS)
  98 +
  99 +Comentários para os Autores (Comments to Authors): Comentários e sugestões para o autor melhorar a qualidade do artigo
  100 +Trabalho muito bonito e muito interessante. Parabenizo toda a equipe pela realização. Como artigo científico sugiro que coloquem em valor as dificuldades técnicas de manutenção evolutiva de um software livre e o uso de metodologias ágeis na pratica. Terminei a leitura sem saber exatamente quais as práticas foram realmente aplicadas.
... ...