Ranking comment sorting policies in online debates

1.1.Background and motivations

The Internet has enabled people to share their views about many topics online, ranging from commenting on important news events11 to debating philosophical stances.22 Given that the number of Internet users grew eight-fold to over three billion worldwide between 2000 to 2016 [27], it is not surprising that such discussions have also grown both in the number of comments posted and the range of topics discussed [19]. Many online discussions now contain so many comments that it is unrealistic for a normal Internet user to read every single point being made; it is expected that a user interested in a given topic of debate would read only a fragment of the entire discussion. To help users, many online platforms that host such discussions allow for the comments to be sorted according to some policy, such as displaying the comments from the most liked to the least liked.

Further, comments exchanged over such discussions are often argumentative in nature, and can be modelled with argumentation theory (e.g. [7,31]); this can resolve disagreements arising from such online debates by finding the winning arguments based on normative yet intuitive criteria. Suppose that a user would like to know which arguments have prevailed at a given moment, but does not have time to read every comment. Such a reader may think that a given argument has won when it has been rebutted by further comments which have not yet been read. The goal of this paper is to compare how various comment sorting policies can affect the number of actually winning arguments exposed to a reader, depending on how much of the debate has been read. A precise way to compare such policies will bring visibility to which arguments should win, thus helping readers to better navigate and understand large and argumentative online debates (e.g. [1,8,14,28,32,36]).

1.2.Overview of contributions

We apply argumentation theory, data mining and statistics to build a pipeline that compares comment sorting policies by measuring the number of actually winning arguments each policy displays to a reader who has only read a part of the debate. Intuitively, we mine online debates (e.g. such as [13,17,35]), and represent them as directed graphs (digraphs), whose nodes are the comments made during the debates and the edges denote which comment is replying to which other comment. As such debates begin with an initial comment, and each subsequent comment replies to exactly one other comment, these debates are trees, which is what we will restrict our analysis to in this paper, leaving other debate network topologies for future work (see Section 6).

Argumentation theory (e.g. [31]) is the branch of artificial intelligence concerned with the rational and transparent resolution of disagreements. Abstractly, arguments and their disagreements (called attacks) are represented as digraphs, called argumentation frameworks (AFs) [15]. However, in real debates, arguments can agree with each other as well as disagree. Subsequent theoretical developments have incorporated support between arguments, which can be interpreted as “agreement”. An AF where each directed edge is either an attack or a support is called a bipolar argumentation framework (BAF) [9,10]. There are principled ways of determining from various combinations of attacks and supports which arguments are winning. Therefore, subject to an appropriate representation of the debate one is interested in, resolving the disagreements is modelled by finding a subset of winning arguments.

To enable the application of argumentation theory, the pipeline makes two idealised assumptions: (1) each comment qualifies as a self-contained argument, and (2) each reply is either supporting or attacking. Of course, the vast majority of online debates are not so “clean”, but the pipeline does not preclude methods that allow for the identification of whether a comment qualifies as an argument (as opposed to, e.g. an insult), or whether a reply between comments is an attack or a support (e.g. [3,11]). Assuming that there is a relatively “clean” online debate, or that we can incorporate sophisticated data cleaning techniques to the pipeline, then we can treat the online debate formally as a BAF. Furthermore, if this debate is a tree, then the set of winning arguments is unique [15, Theorem 30].

To model the idea of a comment sorting policy, the pipeline linearly sorts all comments based on the direction of replies, such that if comment b replies to comment a, then a is read prior to b, while preserving the structure of each thread (see Section 3.1). However, when multiple comments b1,b2,…,bn are replying to a given comment a, we will sort the bi’s based on a given attribute. For this paper, we will only consider the attributes likes (represented by some number, see Section 5.2) and time (represented e.g. by Unix time). Therefore, a comment sorting policy determines the order by which a reader is exposed to the arguments made in a debate with respect to an attribute.

Suppose the debate has N∈N+ comments (N is a positive integer). The reader will read the first n (0⩽n⩽N) comments according to the policy, and for each such n, we will obtain an induced BAF of those comment-nodes (“induced”, i.e. keeping all edges between those n nodes). Each such induced BAF will have its own unique winning set of arguments Gn, which may or may not contain the set of actually winning arguments GN, i.e. the winning arguments after reading the whole debate. To quantitatively compare policies, the pipeline compares Gn with GN for each n, which gives a sequence of numbers J0,J1,J2,…,JN, where Jn∈[0,1] is the Jaccard coefficient between the sets Gn and GN (see Section 3). Intuitively, the larger Jn is, the higher the proportion of actually winning arguments the policy exposes to a reader who has read n comments. The pipeline calculates, for each policy, an (N+1)-tuple of Jn’s.

The pipeline then aggregates the Jn’s of each policy into a single number, such as taking the average (see Section 3.3), in order to compare which policy is more effective at displaying the actually winning arguments. Alternatively, one can fix n and just compare Jn across policies (see Section 5.4), which represents the effectiveness of reading the first n comments given the policies being evaluated. These values can be calculated for a single debate, or across an ensemble of debates, for a given policy. For the latter case, we can compare summary statistics of the distributions of values of the averages or Jn’s and use statistical tests to see which policy has a greater such summary statistic.

As mentioned earlier, if for simplicity one does not wish to incorporate argument mining techniques, then these calculations can only make sense if we do have a dataset of “clean” online debates, i.e. where each comment qualifies as a self-contained argument, and each reply is classified as either an attack or a support. Fortunately, the online debating platform Kialo33 provides such a “clean” dataset, thanks to its moderation policy that ensures every comment is a concise and relevant argument to the topic being discussed that is replying either positively (supporting) or negatively (attacking) to another comment (or no such reply if it is the root comment of a discussion). We have mined Kialo data until March 2019 (see Section 4), and processed this data via our proposed pipeline. Subject to suitable aggregation and identification of what “likes” and “time” mean for each comment (see Section 5.2), the pipeline shows that reading Kialo debates from the most to least “liked” comment displays more actually winning arguments, on average, than reading Kialo debates from the earliest comment to the most recent comment. We close by conjecturing why this is and speculate on how robust this result is to less “clean” online debates.

1.3.Structure of this paper

In Section 2, we will summarise the relevant background in argumentation theory and related work. In Section 3, we will explain the pipeline and clarify its assumptions and limitations. In Section 4, we overview Kialo, explain how it validates the pipeline’s assumptions, explain how we have mined the data, and offer summary statistics. In Section 5, we articulate the comment sorting policies based on likes and time for Kialo, and have the pipeline measure which policy is more effective. Section 6 concludes with future work.

2.1.Abstract argumentation theory

From [15]: An abstract argumentation framework (AF) is a digraph ⟨A,R⟩, where A is the set of arguments under consideration and R⊆A2 is the defeat relation, where A2:=A×A (and similarly, Xn denotes X×⋯×X n times). We write R(a,b) to denote (a,b)∈R, which means a defeats (disagrees with) b. In what follows let a,b∈A and S⊆A. Let S+:={b∈A∣(∃a∈S)R(a,b)}, S−:={b∈A∣(∃a∈S)R(b,a)} and a±:={a}±. Define the neutrality function, which maps sets of arguments to sets of arguments, to be nf:P(A)→P(A):S↦A−S+, i.e. the set of arguments not defeated by S. We say S is conflict-free (cf) iff S⊆nf(S), and S is self-defending (sd) iff S⊆nf∘nf(S), where ∘ denotes composition of functions (if well-defined).

We say S is admissible iff S is both cf and sd. There are four basic notions of “winning arguments”, each of which builds on admissibility [15]. We say S is a complete extension iff S is admissible and that nf∘nf(S)⊆S. We say S is a preferred extension iff S is a ⊆-maximal complete extension. We say S is a stable extension iff nf(S)=S. We say S is the grounded extension, denoted G, iff it is the ⊆-least complete extension; G always exists, is unique, and is empty iff the set of unattacked arguments (i.e. a∈A where a−=∅) is empty [37, Corollary 6.10]. Recall that if an AF is well-founded, i.e. that there exists no sequence of arguments {ai}i∈N such that (∀i∈N) R(ai+1,ai) [15, Definition 29], then G is the unique stable, preferred and complete extension, i.e. there is only one set of winning arguments [15, Theorem 30]; this is certainly the case when ⟨A,R⟩ is a tree [37, Corollary 2.46].

Example 2.1.

Let A={a,b,c,d,e} and R={(c,b),(c,a),(d,b)}. ⟨A,R⟩ is well-founded with G={c,d,e}. This AF is shown in Fig. 1.

Fig. 1.

The AF in Example 2.1; argument e is an isolated node.

2.2.Bipolar argumentation theory

Bipolar argumentation theory extends AFs with a support relation [9]. A bipolar argumentation framework (BAF) is a structure ⟨A,Ratt,Rsup⟩ where A is the set of arguments, Ratt⊆A2 denotes when two arguments attack each other, and Rsup⊆A2 denotes when two arguments support each other, and Ratt∩Rsup=∅. A BAF can be visualised as a digraph where edges are either red (Ratt) or green (Rsup).

There are principled ways to absorb Rsup into Ratt, resulting in an associated AF ⟨A,R⟩. Following [9], for arguments a,b∈A, we say that a support path from a to b is a path in the underlying digraph that only traverses support edges. We write a→supb iff a support path exists from a to b. We say a support-defeats b iff (∃c∈A)[a→supc,Ratt(c,b)] and a indirectly defeats b iff (∃c∈A)[Ratt(a,c),c→supb]. The associated AF of a given BAF is ⟨A,R⟩, where R(a,b) iff a support-defeats b or a indirectly defeats b. The winning sets of arguments for a BAF are the extensions of its associated AF.

Example 2.2

Example 2.2(Example 2.1 continued).

Figure 2 illustrates the BAF where A={a,b,c,d,e}, Ratt={(c,b)} and Rsup={(d,c),(e,b),(b,a)}.

Fig. 2.

The BAF from Example 2.2, where green (dotted) edges denote supports and red (solid) edges denote attacks.

The support paths of length 1 in this BAF are (e,b), (b,a) and (d,c), and the support paths of length 2 in this BAF consist of only (e,b,a). Therefore, e→supb, b→supa, d→supc and e→supa. As c attacks b, we can see that c indirectly defeats a, and d support-defeats b. Therefore, the associated AF of this BAF has the same arguments, but the defeat relation is R={(c,b),(c,a),(d,b)}. This resulting AF is already illustrated in Fig. 1. The set of winning arguments for this BAF is also G={c,d,e}.

2.3.Argument mining and the analysis of online debates

Argument mining (e.g. [23,24]) seeks to identify and extract structured, self-contained arguments and inter-argument relationships from natural language text to (amongst many other applications) resolve conflicting points of view expressed in online debates (e.g. [6,7]). However, in order to successfully apply argumentation theory to real data, one would need a relatively “clean” dataset of online debates. This is non-trivial, because most online debates are very “noisy”. For example, Twitter Tweets are often too short to be treated as self-contained arguments so a preprocessing step is required to determine whether Tweets can be treated as such [3]. Further, one must formally represent the relationships between the mined arguments. In the context of applying BAFs to model online debates, we are interested in classifying whether a reply from comment a to comment b (assuming both a and b can be treated as arguments) is attacking or supporting. Existing techniques include textual entailment, which infers relevance and contradiction between pieces of natural language (e.g. [2,12]) and hence identify attacks between comments. Another way of identifying attack or support can be to calculate affective or emotional scores from key words in the comments using Linguistic Inquiry and Word Count [29,34], such that replies between comments with very similar such scores can be related to supports. A more sophisticated method for identifying attacks and supports involves deep learning [11]. Therefore, there are many argument mining techniques at our disposal if one wishes to investigate a “noisy” online debate dataset, where it cannot be assumed that each comment made by users constitutes a well-defined argument, or that every reply is either an attack or a support. We envisage that such techniques are used to first render the “noisy” online debate “clean”, then one can use the usual methods in identifying the winning set(s) of arguments.

In the next section, we will apply ideas from argumentation theory to articulate a pipeline that allows one to rank the effectiveness of various comment sorting policies in online debates, assuming that the input data is already “clean”, either because it is “naturally clean”, or that it has been “cleaned up” by techniques such as those just mentioned. Therefore, the pipeline does not preclude the use of such techniques. We will outline how such techniques can be integrated into this pipeline in Section 6.

3.1.The stages of the pipeline

The pipeline consists of the following stages, where the first two stages involve validating our assumptions on the data.

From now, we make the assumption of time coherence of a debate network – that if argument b replies to argument a, then tb>ta, i.e. a comment must exist first before it can be replied to. Furthermore, we will use the terms root and leaf as defined in Example 3.1.

We illustrate the above six steps, for a fixed value of n, with Fig. 3 [38]:

Fig. 3.

A schematic of our pipeline used to evaluate comment sorting policies.

3.2.Formal properties of the pipeline

The following are straightforward consequences of our pipeline.

Note that the converse to Corollary 3.5 is false, as indicated in Example 3.3 for n=2, say. This means it is possible to read a non-zero portion of the debate and think (at least normatively) that the actually winning arguments are all losing, therefore “totally missing the point”.

Intuitively, Corollary 3.5 states that if you read nothing you get nothing, and Corollary 3.6 states that if you read everything you get everything, regardless of the comment sorting policy. One further result concerns the case where all edges are supports and the policy is based on DFS.

This means that in a debate where there is no disagreement, the more one reads, the more of actually winning arguments is obtained, because every additional argument read is actually winning. Further, this strictly monotonically increasing situation is only possible in the case where every reply is supporting.

3.3.How are comment sorting policies compared?

As discussed so far, the pipeline takes as input a BAF and a comment sorting policy, and returns a vector J:=(J0=0,J1,J2,…,JN−1,JN=1), where each Jn∈[0,1]. The larger the Jn, the larger the proportion of actually winning arguments one is exposed to by only reading the first n comments according to that comment sorting policy. How can we aggregate the components of J for each policy in order to compare different policies? One idea is to take the average across all N+1 components:

Intuitively, as the Jaccard coefficient is a measure of how many actually winning arguments the apparently winning arguments contain, having read the debate up to a certain point, ⟨J⟩ for a policy is the typical proportion of actually winning arguments the policy exposes. We now give an example.

The following corollary of Theorem 3.7 shows for the special case of debates where there is no disagreement, every policy based on DFS gives an average Jaccard of 0.5. This makes sense as we are taking the centre of mass of the diagonal line that joins (nN,Jn)=(0,0) and (1,1).

3.4.Summary of the pipeline

The first contribution of this paper is a pipeline that accepts as input an online debate represented as a BAF, and a comment sorting policy, and calculates how this policy, on average, exposes the reader to the actually winning arguments, especially when the reader has not read the entire debate. We have given a running example of how the pipeline works (Examples 2.1 to 3.9).

However, the fundamental assumption of the pipeline – that a real-life online debate can be represented as a BAF – is suspect. One cannot just assume that comments posted on online debates are self-contained arguments, nor that every reply can be classified as an attack or a support. Of course, the pipeline does not preclude the use of argument mining techniques (Section 2.3) to structure online debates into a BAF for our pipeline to ingest (Section 6). In the next section, we present a dataset that we have mined, which seems to satisfy the property of being “clean”, and thus being immediately suitable for input into the pipeline as a proof of concept of how might various comment sorting policies be evaluated.

4.1.What is Kialo and why is Kialo “clean”?

Kialo is an online debating platform that helps people “engage in thoughtful discussion, understand different points of view, and help with collaborative decision-making.”66 In a Kialo debate, users submit claims. The starting claim is a thesis, which has at least one, but possibly more, indexing tags. Further claims then reply to existing claims, and are classified as either pro or con the claim being replied to.

Kialo debates are “clean” and easily-represented as BAFs without the use of argument mining tools (Section 2.3). Firstly, Kialo has a strict moderation policy, which aims to keep users engaged and promotes a well-structured discussion.88 Central to this is the enforcement of writing good claims. For example, each claim should be concise, self-contained, based on logic and facts, and make a single point that is relevant to the debate topic – in other words, claims are arguments.99 Claims should not be questions, or mere comments, or duplicated within a discussion – this excludes the possibility that claims are insults, say. Discussions are structured such that more general claims are closer to the root, and more specific claims are closer to the leaves. The conciseness of each claim allows for fine-grained support or attacks when replied to.1010 Claims that fail to meet such criteria are marked for review.1111 Either the claim is deemed unsuitable and therefore removed from the debate, or that the claim is conditionally approved after some discussion between the claim’s author and the moderators, and that the claim’s author must make satisfactory improvements for the claim to be published.1212 This validates the pipeline’s assumption that all claims made in Kialo debates can be treated as an argument.

In Kialo, every claim b that is replying to another claim a is classified by the author as pro or con a; the polarity of the reply is also suitably moderated as part of auditing the claim. Therefore, all replies are classified as either attacking or supporting. This validates our assumption that Kialo debates can be modelled as BAFs. Further, the mechanism of how claims reply to at most one other claim and that each claim is moderated to ensure that it makes a unique point mean we can assume Kialo debates are trees.

In summary, Kialo is an online debating platform. Its highly moderated nature validates our assumptions that every claim made in Kialo debates can be treated as self-contained arguments. Further, every reply between two claims in a Kialo debate must either be a support (pro) or an attack (con). Lastly, the structure of Kialo debates guarantees that the underlying digraph is a tree. It is in this sense that Kialo debates are sufficiently “clean” for us to represent them straightforwardly as BAFs without the need for further data cleaning methods.

4.2.Mining Kialo

We scraped all Kialo debates dated up until March 2019 as follows: we use the publicly visible Kialo application programming interface (API), which is used by Kialo’s front end, to acquire all the available tags used to tag individual debates; this resulted in a list of over one thousand unique tags. We then used the same API to perform a tag-based search for debates using the previously acquired list of tags. For each debate we obtained its claims, replies, and various claim attributes such as the username of the author, its text, time of posting, and votes (related to likes, see Section 5.2). This returned 1,056 debates. We then use independent means of verifying this dataset [4], which gives us a high degree of confidence that this is almost all of the debate activity on Kialo, as of March 2019.

4.3.The resulting dataset

When the debates dataset was cleaned, we noticed that some nodes (less than 1% of the total) have empty text and therefore could not be considered arguments; this is possibly due to how the data is stored in the back end of Kialo has failed to synchronise with moderators asking for a claim to be removed. Further, some reply edges violate time coherence (Section 3.1). We delete both such nodes and edges and focus on the resulting sub-discussions, on the assumption that Kialo’s moderation policy implies that sub-discussions are also self-contained, despite being separated from the thesis, due to the conciseness of each reply. This results in 49,040 weakly-connected BAFs that are also trees, containing 1 to more than 500 arguments.1313 By excluding BAFs with 9 arguments or less, we obtain 4,365 BAFs to analyse with our pipeline. These BAFs have an average of 28.71 arguments (sample standard deviation (SSD) 35.21). Amongst these BAFs, the in-degrees of all nodes present range from 0 (leaves) to 58, with a mean in-degree of 0.965 (SSD 2.00). Amongst all root nodes, the average in-degree is 7.34 (SSD 4.58). This gives some idea of the typical activity of discussion in the average debate.

Example 4.2

Example 4.2(Example 4.1 continued).

We can visualise the debate in Fig. 4. This has 115 arguments where 57% of replies are attacks, and 43% of replies are supports.

Fig. 4.

The debate as a BAF from Example 4.2, where red edges are attacks and green edges are supports.

5.1.Validating the pipeline assumptions

As stated in Section 4.1, Kialo debates are sufficiently clean such that they are readily representable as BAFs due to its design and moderation policy. Further, in our dataset, all debates are trees hence all debates (and all induced subgraphs thereof) have a well-defined set of winning arguments via the grounded extension that the pipeline calculates. This verifies Steps (1) to (3) of the pipeline (Section 3.1), apart from defining the policies to sort comments by. Lastly, we do not need to perform further natural language processing, because the BAFs are already given.

5.2.Which policies to evaluate?

To apply Step (4) of the pipeline to Kialo data, we still assume that an interested reader would read each replying comment from the root in a DFS manner as this models how the reader is getting whole conversation threads without skipping comments before backtracking to the next possible thread.

For the purposes of illustrating how the pipeline works on “cleaned” real data in this paper, we compare sorting along two attributes: likes and time; other attributes and policies will be considered in future work (Section 6), as these two attributes are readily available from Kialo data.

Time refers to the time of posting the claim, while likes is a measure of how “good” the claim is. In our dataset, each comment in a Kialo debate is already time-stamped, hence time for each comment is well-defined. However, likes is not well-defined. Conceptually, the closest attribute available in Kialo comments is the number of votes, which is a five-tuple v=(v0,v1,v2,v3,v4), where each vk∈N is the number of votes made by users for the kth impact category of the comment. Intuitively, impact is a function of the truth, persuasiveness and relevance of the claim to the parent it is replying to.1414 Claims are assigned votes close to the end of the discussion (as deemed by the moderator), and when each user votes, the other votes claims have already received are hidden, which avoids bias.1515 Impact category 0 means the comment has no impact, while category 4 denotes very high impact.

How can we relate the five-vector of votes v to a single number to sort claims by, which is also a proxy for likes? We use the following weighted average: define a weight vector w:=(w0,w1,w2,w3,w4)∈(R0+)5, which weighs each impact category by the corresponding non-negative weight wk. Given such a w, define μ(w):=15∑k=04wk; if there is no risk of ambiguity about w we will write μ instead. Let · denote the usual (Euclidean) dot product. We define a normalised weighted average, parameterised by w, as

Notice if w=0, then (∀v∈N5) nwaw(v)=0. If all components of w are equal to 1, then (∀v∈N5) nwaw(v)=1. Further, if all wk of w satisfy wk⩽1, then (∀v∈N5) nwaw(v)⩽1. Therefore, we consider w where all components wk lie strictly between 0 and 1.

We now define the following two w’s to form two “sort by likes” policies. Informally, the first w is a priori and the second is data-driven. Suppose we interpret these five impact categories in terms of a rational reader’s degrees of belief about the truth of the claim (e.g. [21]), so v0 denotes the number of votes by users who think the claim is false… etc. and v4 denotes the number of votes by users who think the claim is true. In this case, we let wr:=(0,0.25,0.5,0.75,1), which weighs the five categories equally in the sense that moving up between two categories corresponds to an equal amount of increase in the voters’ degrees of belief concerning the claim’s truth. We call nwawr(v) the rational likes of v.

One criticism of rational likes is that this idea of equal weight increases across the five impact categories is unjustified in that it assumes all readers interpret these impact categories rationally. Another approach is to weigh each impact category according to the number of votes each has received across all 4,365 debates in the dataset, which is based on real readers’ perceptions of impact. We find that 13.73% of votes cast were impact 0, 13.75% of votes cast were for impact 1, 23.72% of votes cast were for impact 2, 22.19% of votes cast were for impact 3, and 26.61% of votes cast were for impact 4. Suppose we set we=(0.1373,0.1375,0.2372,0.2219,0.2661). We call nwawe(v) the empirical likes of v.

Therefore, we enhance all nodes in each BAF of our dataset with rational and empirical likes, based on their votes vector. We define the following four policies to evaluate with our pipeline (Section 3.1):

5.3.Results

We now measure the effectiveness of the above four policies as described in Steps (5) and (6) of Section 3.1. For each debate in our dataset, we calculate and plot {(nN,Jn)}n=1N for each policy. Plotting Jn against nN instead of n will allow us to make comparisons across debates of different sizes, i.e. in terms of the fraction of the debate read with respect to a policy. The following example shows each policy’s {(nN,Jn)}n=1N plot for the debate in Example 5.4.

Example 5.4

Example 5.4(Example 4.2 continued).

The data cleaning process has assigned the index 2,768 to this debate. As n increases from 0 to N, Fig. 5 displays how Jn changes with respect to n/N.

Fig. 5.

The Jn vs. nN curve of Example 4.2 for all four policies.

Visually, it is difficult to judge how each policy differs as all cluster around the diagonal. For each debate, we plot the detrended Jaccard, DJn:=Jn−nN against nN, for 0⩽n⩽N. Notice DJ0=DJN=0.

Example 5.5

Example 5.5(Example 5.4 continued).

Figure 6 plots the detrended Jaccard vs. nN for Example 4.1.

Fig. 6.

The DJn:=Jn−nN vs. nN curve of Example 4.2 for all four policies.

Visually, all policies are the same when reading up to ∼2% of this debate. Further, Policy 3 (descending rational likes with DFS) and Policy 2 (ascending time with DFS) seem to track each other, and both seeming to beat Policy 3 (ascending time without DFS), while Policy 1 (ascending time, no DFS) seems to capture the least proportion of the actually winning arguments.

The conclusions we have drawn so far only apply to debate 2,768 (Examples 4.2 to 5.5). We are now interested in using our entire dataset of 4,365 BAFs to determine which of our four policies – Policies 1 and 2 are based on time (with or without DFS) and Policies 3 and 4 are based on likes (rational and empirical, with DFS) – is the most effective, i.e. maximises Jn. We will take the average DJn across 0⩽n⩽N for each debate under each policy, as it is easy to show that this is equal to ⟨J⟩−12 (the proof is analogous to that of Corollary 3.10). Therefore, comparing average values of DJn is the same as comparing ⟨J⟩. We then plot the distribution of such values for each policy.

5.4.For Kialo, sorting by likes is, on average, better than sorting by time

We repeat the calculation in Example 5.5 for all 4,365 BAFs in our dataset. Each version of Fig. 6 for each BAF will have some pattern of fluctuations about the horizontal for each policy. Instead of describing these fluctuations directly, we evaluate the policies over our dataset of debates in two ways: (1) across the entire range of nN (Section 3.3), and (2) evaluating each policy by calculating J5, which models the effect of reading the top five claims given the policy. Admittedly five is arbitrary; we assume that a hurried reader only has time to read five claims. The distributions of the averaged DJn across all four policies for (1) are shown in Fig. 7. The analogous distributions for (2) are shown in Fig. 8.

Fig. 7.

The DJn distributions for all policies averaged over nN (“one segment”), for all 4,365 trees.

Fig. 8.

The distribution of the values of J5 for all debates in our dataset, for each policy.

Visually, we can see that Policy 1 is the worst policy to sort by, according to its averaged DJn distribution. Policy 2 seems much better, while Policies 3 and 4 (both based on likes), overlap a lot. We can make this more precise with the summary statistics of the distributions in Fig. 7 in Table 3.

We can see that comparing the medians and means of the four policies does support that sorting by likes is better than sorting by time, and further, that time with DFS is better than time without DFS. Let us make this even more precise with an appropriate statistical test. Figure 7 displays skewed distributions, therefore we should make no assumptions on the underlying population distribution for average DJn per policy (non-parametrics). Indeed, performing a Kolmogorov–Smirnov test [22,33] allows us to conclude that none of these distributions are Gaussian, and hence we should compare medians rather than means.

As this is a non-parametric situation, we apply the Mann–Whitney test [25] to compare whether the observed distributions of average DJn are sampled from the same underlying population distribution (our null hypothesis), or different such distributions (our alternative hypothesis). At a significance level of α=0.05, we perform 6=4C2 such tests. The results for the data plotted in Fig. 7 are shown in Table 4.

For Fig. 7, Policy 2 beats Policy 1 (average DJn −0.067>−0.107), Policies 3 and 4 beat Policy 2 (average DJn −0.052,−0.053>−0.067), while Policies 3 and 4 are not significantly better than each other (average DJn −0.052∼−0.053). Overall, sorting by likes, averaged over all debates in our dataset, beats sorting by time.

We now perform the same calculation for the case where instead of averaging DJn over all of 0⩽n⩽N, we evaluate each policy using J5. This is the proportion of winning arguments obtained by reading the first five arguments given the sorting policy, regardless of the size of the debate. This gives us the series of partially-overlapping histograms in Fig. 8.

The summary statistics of the distributions of J5 for the four policies are shown in Table 5.

When comparing medians, Policy 1 is again the least effective. Policy 2 is more effective than Policy 1 (i.e. sorting by time with DFS is better than without DFS). Both Policies 3 and 4 are more effective than Policy 2, and they seem as effective as each other. Again, we can make this precise with the same statistical tests, hypotheses and significance levels as for the summary statistics in Fig. 7 (see Table 6).

We conclude that the ordinal ranking the four policies by ⟨J⟩−12 and by J5 do not change. Therefore, when seen through our two ways of measuring the effectiveness of these policies, sorting Kialo claims by likes-DFS (as defined in Section 5.2), on average, contains a higher proportion of the actually winning arguments than sorting by time (DFS or not).