Problems and Purpose

This case study describes the combination of two democratic innovations: the Deliberatorium and the ‘Doparie’. The Deliberatorium is a software that allows large groups of individuals to deliberate using the principles of large scale argumentation maps. The ‘doparie’ is an Intra-Party Deliberative Referendum mechanism that is meant to improve the internal decision making process within parties in between elections.

In the project analyzed by this case study, these two democratic innovations were combined. The project was implemented in 2012 by a partnership between a group of academics and a group of politicians aimed to study the feasibility of introducing online discussions as a decision-making mechanism in the Italian Democratic Party. Additionally, this project aimed to compare the difference in deliberation between an argument-centric environment and a discussion forum.

Background History and Context 

In 1993, Italy changed electoral law from a proportional system to a mixed system. Then in 2005, Silvio Berlusconi, then Prime Minister of Italy, again changed the law back to a proportional system with closed lists (Spada et al. 2015). The electoral reform generated lots of controversy because it reduced the ability of voters to select representatives and did not generate stable majorities.

During this period, the rise of the Five-Star Movement created a new political force in Italy (Spada et al. 2015). It began as a blog by comedian Beppe Grillo to criticize the corruption in Italian politics. The protest movement argued that all the political parties in the country were corrupt old cliques. Most of their political campaigning was conducted online using the meet up platform, blogs and forums. They built their platform based on an anti-establishment, anti-corruption, green agenda. Between 2010 - 2012, the Five-Star Movement increased its influence in local and regional politics. 

In 2011, a million signatures were gathered to propose a referendum to abolish the current electoral law introduced by Berlusconi and to go back to the previous majoritarian system. A few months into 2012, before the beginning of the experiment described in this case study, the proposal was rejected by the Supreme Court in 2012 as not constitutional.

During this time, the rise of the Five-Star Movement and its success based on innovative engagement strategies centred on social media altered the Italian political equilibrium. Traditional parties began to explore possibilities to counter these tactics.

The highly controversial electoral law introduced in 2005 and its’ regional mechanism of Senate seat allocation resulted in a coalition government in the 2013 elections. To everyone’s surprise, the 5-Star Movement allocated enough votes to become the opposition party after the coalition. The traditional parties became alarmed with the 5-Star Movements’ success using the Internet and social media to recruit supporters. This led to the willingness of the Italian Democratic Party to explore the possibilities of online discussion with their supporters over the topic of electoral law. This created the opportunity for Spada, Callabretta and Klein to run this experiment with the online community, Insieme per il PD, using the Deliberatorium and the doparie.

Organizing, Supporting, and Funding Entities 

The project was initiated by Paolo Spada who was a post-doctoral fellow in the Ash Center for Democratic Governance and Innovation at the Harvard Kennedy School at the time. Spada connected Mark Klein with Raffaele Calabretta to build a team that could manage the democratic innovation. While Klein’s research on the Deliberatorium was funded by the National Science Association, this project was not funded by monetary means, but rather an investment of time.

Raffaele Calabretta convinced an online community of the Italian Democratic Party (Insieme per il PD) to explore an online version of the Intra-Party Deliberative Referendum. Mark Klein recruited former colleague Luca Iandoli and his student Ivana Quinto to manage the team of 10 moderators and develop additional survey tools. 

Participant Recruitment and Selection

For this particular experiment, members of the online community, Insieme per il PD (Together for the Democratic Party), were asked to participate. Calabretta used his network to advertise for participants on social media, through the Insieme per il PD website, and Facebook page, which had around 5000 members at the time. The Italian Democratic Party also advertised the process airing a small video produced by Italian Filmaker Alberto Bougleux (see https://www.youtube.com/watch?v=noMaHClrrbU). The campaign highlighted the democracy enhancing concepts of this process, and the possibility of bringing different ideas to the attention of the party elite. However, the advertising campaign was very clear in acknowledgment that the experiment was for research purposes and it would have a limited impact over the program of the Italian Democratic Party (Spada et al. 2015). 

Participants were asked to enrol beforehand and provide information regarding their age, gender, level of education, and level of internet usage. The participants had an average age of 48, 25% over 60 years old and 15% under 30 years old. This pool is not representative of the overall Italian population, with 2/3 males and 1/3 females. There was also a higher numbers of people over 60 years old which is possibly due to more free time. 58% of the participants had education higher than high school, and this is higher than the national average of 33% (Spada et al. 2015).

Finally, the 640 enrolled person participants were divided into 4 different groups of 160 each and balanced based on their demographic statistics using block randomization. Two groups were placed in a traditional discussion forum, while the other two in the argument maps (Spada et al. 2015). 

Methods and Tools Used

The Deliberatorium: A solution to the problems of online deliberation

The most commonly used tools for online deliberation are time-centric tools (emails, chat rooms, forum) where content is organized based on post times. These tools generate numerous issues:

• scattered content: coverage is not systematic which makes it difficult to find content 
• low signal-to-noise ratio: there is too much repetition of the same points which makes it hard to find more mature points 
• balkanization: users will self-group based on common opinions which makes it difficult to be broadly informed
• dysfunctional argumentation: arguments tend to be biased and not evidence or logic based (Klein et al 2012)

There are also topic-centric tools (wikis and idea sharing systems) where ideas that are unpopular are able to be edited by other users who disagree. Controversial topics are then moved back to discussion, leading back to the issues of time-centric tools (Klein 2012). 

Finally, there are idea-sharing tools which center around specific questions and the community is asked to comment or rate on the proposed solutions. However, these tools generate a lot of redundancy, where ideas are mostly simplistic, and most comments are similar to one another (Klein et al. 2012).

To counter the shortcomings of online deliberation, Mark Klein, a researcher at MIT’s Collective Centre for Intelligence, developed the Deliberatorium. The Deliberatorium is an argument-centric tool designed to address the issues of traditional discussions forum.

Argument-centric tools allow users to systematically deliberate in an organized tree structure. Each online posting is limited on only one point. It should be either a specific issue (question to be answered), an idea (possible answers for a question), or an argument (statement that support or detract from an idea or argument) (Spada et al. 2015). Before the development of the Deliberatorium, argument-centric systems had only been used on a small scale, and rely on a moderator to capture free-form dialogue. These systems have had moderate success to enable deliberation over the internet as most user are uneducated and unfamiliar with an argument mapping system. The purpose for the Deliberatorium is to enable web users to more effectively express logic and evidence in favour or against an argument, idea, or an issue. Rather than having ideas spread out in a disorganized fashion, the Deliberatorium helps user systematically arrange deliberation in a tree structure using “the IBIS argumentation formalism” (Klein 2012). 

The ‘Doparie’ (Intra-Party Deliberative Referendum)

The second democratic innovation used in this experiment is an Intra-Party Deliberative Referendum, or the ‘doparie’. Raffaele Calabretta, a senior researcher at the Italian National Research Council developed this mechanism to be used by party members in between elections to deliberate, make decisions, and vote (Callabretta 2012). The doparies are open to all the supporters of a party, and are centred in providing balanced information and promoting good deliberation. The doparies are designed to improve the communication between party members and voters, to increase accountability, and to allow parties to consult the base on difficult decisions that split the elected representatives (e.g., end of life laws).

Deliberation, Decisions, and Public Interaction

The experiment took a total of 2 months, and was divided into 3 different stages: 1) advertising and enrolment, 2) deliberation, and 3) voting. The advertising phase lasted one month. During this first month, participants were invited to re-post information on their social media pages to encourage further enrolment. The deliberation stage took 3 weeks in which the Deliberatorium and the discussion forum were open for use. The final week allowed participants to vote. (Spada et al. 2015). 

The design of the user interface was similar for both the discussion forum and the Deliberatorium to eliminate most differences, and also provided with a set of standard tools for the users. For both mediums, three questions were asked regarding Italian electoral reform: 

• What electoral law should Italy adopt?
• What other important questions should be asked about the electoral system?
• What topics should be deliberated about in the future?” (Spada et al. 2015)

In the traditional forum, users have the usual functions of adding comments, adding sub-forums, and the ability to rate other posts. In the argument map, every issue, idea, or argument appears once to reduce repetition, making it easier to look for what has or hasn’t been posted already. Users are asked to create posts with a single point and place them onto the appropriate section of the argument map. The post will be pending until a moderator has verified that it has met all the guidelines. Moderators are responsible for educating users on how the deliberation process works, and also quality control by certifying posts. This ensures that the argument maps are well structured for maximum utility. Moderators in the discussion forums were only responsible for ensuring that content was SPAM-free and not abusive. Once a post was certified by a moderator, it was then available to be rated, viewed, and edited by other users (Klein et al. 2012).

For this particular project, the moderators were chosen among the graduate students of Professor Luca Iandoli with previous knowledge of argument mapping. The moderators were tasked only to make able to implement simple changes, such as fixing where a post belonged, or whether it was incorrectly labelled as a pro or a con. If a user’s post was not straightforward, the moderators were instructed to send them a message, educating them on how to fix the problem. However, the moderators were not engaged with the content in any other way. This not only taught users the structure of argument mapping, but also helped the tree to be structured for optimal use and easier idea harvesting (Spada et al. 2015).

The data retrieved from this project is still being compiled and studied. At the initial analysis, there were over a couple hundred thousand user events, and the researchers are still in the process creating measures to analyze the data. Currently, Mark Klein is coding an algorithm to apply to the map’s event stream in order to analyze how mature the discussion was, the number of ideas discussed, the number of rebuttals, the depth of the map, and the depth of sub-idea, and the depth of the arguments using a series of metrics and signifiers. Luca Iandoli is also doing a social network analysis on the data.

Influence, Outcomes, and Effects

The experiment report provides results for the following event types: edit, create, rate, view, vote, view resources. The report shows that the huber of ratings is significantly higher in the map treatment that in the forum treatment. This may be because each post on a map is only one point, which makes rating it much easier and more meaningful (Spada et al. 2015).

In the 4 groups of 160 participants, 373 out of the 640 (just over 50%) logged onto the website at least once. It was noted that participation increased over time and did not plateau. As the process only lasted 3 weeks for the purposes of this study, it is theorized that more people would have logged if given enough time. 30% (194 out of the 373) created one or more posts. Compared to the general rule of thumb of internet participation, this exceed the 1/9/90 rule of 1% creators, 9% editors, and 90% readers. The researchers all agreed that this high level of participation is contributed by the controversial topic, the media coverage, and the novelty of the process (Spada et al. 2015).

One of most surprising results of this experiment was that the retention rate was not impacted by the map. Due to the fact that the map appears to require added work from the users, the researchers had initially hypothesized that the difficulty would negatively impact the rate of activity and retention. This makes argument mapping a viable substitute in the future over discussion forums. The added effort that the map requires for the user may be overshadowed by the improved organizational structure and clarity of information (Spada et a. 2015). This can also be attributed to the fact that the participants knew they were part of a research project. 

For Klein, he has not noticed any major issues regarding the slightly more complicated style of argument mapping. He theorizes that the maps users were willing take their time to read through the tree in order to place their post on a specific area. This methodology added structure and clarity to the deliberation and compensated the increased efforts to participate. Also, users were found to be more respectful to other opinions due to the tree’s logical mechanism.

During the deliberation period, the forum only required 3 hours of moderation while the map required 42 hours. However, most of the moderation was used in the first week and the users became much more adept in navigating the map afterwards. Thus, it can be hypothesized that the moderation time required would decrease after the initial educative phase. 

Post deliberation, the data from the forum was recoded into an argument map to compare the argumentation quality. The forum required an additional 160 hours to process the ideas. This resulted in a significant difference between the two treatments for the overall man hours needed moderate and harvest the information (163 vs 42) (Spada et al. 2015). 

The discussion forums yielded 3 times as many ideas as the argument map (290 vs 78), but it generated less than half as many arguments per idea than the map. Therefore the map is seen to promote higher exchange of arguments than a forum. The authors conclude that argument mapping software might not be appropriate for small environment that focus on brainstorming, while it is more appropriate for environments with hundreds of participants in which ideation is easy to achieve, but argumentation is difficult to sustain (Spada et al. 2015).

The voting process at the end of the experiment allowed users to vote on what electoral law they preferred. Of course the vote was in no way legally binding. For now, the impact of the discussion on the Italian Democratic Party is unclear.

Despite the Intra-Party Referendum being partial to in person deliberation due to his experience working with James Fishkin, Calabretta, during our interview, revealed that the deliberation occurring on the Deliberatorium was surprisingly good. This project provided the participants with a deeper consultative mechanism between party members and their voters. For Klein, he feels that while the Deliberatorium allows for better idea generating, it does not necessarily result in more decision making. 

Analysis and Lessons Learned

This process provided an first step to scale-up deliberation online. The maps’ restricted nature did not hinder the participant’s retention rate or daily average activity during the process. The discussion remained also respectful and organized. The facilitation provided by the moderators also reduced the time needed to process the information afterwards.

While the Deliberatorium is a beneficial tool for large scale deliberation on complex issues, it is questionable if all participants in a broader scheme would be able to use the sophisticated software. Also, the experimental nature of this democratic innovation or the highly controversial topic may have led to the high user participation for this project. Thus while this experiment is a significant improvement from previous studies employing students, the self-selected nature of the sample might have affected results. It is unknown what the maximum size of users would be in this system. It would be possible for the tree to get too big for users to find the specific branch that they would like to comment on. It is unknown where the tradeoff between organizational structure and the ease of use ends. 

As for user interface, a more streamlined approach may encourage further posting. The current system appears to be bulky and difficult to navigate. Improvements can be made to make navigation more seamless and intuitive. As well, an improvement in the aesthetics would increase the universal appeal. Users are much likely to use software that is simple and easy to navigate. 

Since the experiment was done with a group of users with similar political views, it is unknown if the pool was more diversified if this system would work. Groups with different political views may not deliberate in a such a respectful manner. Nonetheless, the introduction of controversial issues is much more effective in garnering participation. 

To date, the 640 participants is the largest pool that the researchers have worked with using large-scale deliberation. Therefore, it is difficult to gauge how successful the deliberation would be with an even larger group. Furthermore, as the role of the moderator is quite significant in facilitating the conversation, there may be the possibility of personal biases on highly controversial topics. This bias can lead to certain posts being placed in areas where the original author did not intend.

Finally, the adoption of both the Deliberatorium in online conversations and the Intra-Party Mechanism in political parties has not yet been significant. There remains questions on how to implement these innovations into more areas. Once these innovations have been adopted by a greater number of polities, then a discussion on how to bridge the information harvest to actual democratic and policy changes should be next. 

See Also 

Deliberatorium 

Doparie 

Online Deliberation 

References

 Calabretta, R. (2010). Doparie dopo le primarie. Nutrimenti, Rome. 

 Calabretta, R. (2011). Doparies: a novel party deliberative and aggregative decisione-making mechanism to improve the quality of representative democracy. Sage Open DOI:10.1177/2158244011433699 http://sgo.sagepub.com/content/1/3/2158244011433699.full

 Klein, Mark. (2011). How to Harvest Collective Wisdom on Complex Problems: An Introduction to the MIT Deliberatorium. Center for Collective Intelligence Working Paper. http://cci.mit.edu/docs/working_papers_2012_2013/kleinwp2013.pdf.

 Klein, M. (n.d.). Mark Klein's Home Page. Retrieved May 20, 2015, from http://cci.mit.edu/klein/deliberatorium.html

 Klein, M., Spada, P., & Calabretta, R. (2012). Enabling Deliberations in a Political Party Using Large-Scale Argumentation: A Preliminary Report. 10th International Conference on the Design of Cooperative Systems, 236-259. 

Spada, P., Klein, M., Calabretta, R., Iandoli, L., & Quinto, I. (2015). A First Step toward Scaling-up Deliberation: Optimizing Large Group E-Deliberation using Argument Maps. Retrieved May 20, 2015, from http://www.spadap.com/

External Links

http://doparie.it

https://www.facebook.com/doparie

Mark Klein's Home Page https://cci.mit.edu/klein/research/

Paolo Spada's Home Page http://www.spadap.com/

Raffaele Calabretta's Home page http://laral.istc.cnr.it/rcalabretta

Notes

Lead Image: Deliberatorium https://goo.gl/qJwL43 and Italy https://goo.gl/HXD1QE