Probabilistic interpretations of argumentative attacks: Logical and experimental results¹

1.Introduction

Various disciplines study argumentation [8,85], including artificial intelligence (e.g., [10,75]), computer science (e.g., [11,30]), philosophy (e.g., [34,52,84,86]), and psychology (e.g., [50,62]). The motivation of this paper is to bring together logical, probabilistic, and psychological points of views to better understand specific rationality principles, which refer to the logical form of claims, but ignore the support part of arguments. Our approach is hence an interdisciplinary one, as we combine elements of Dung-style abstract argumentation [30], logical argument forms, coherent conditional probability, and also present a psychological experiment to assess the descriptive validity of selected formal principles.

Argumentation is a highly complex and dynamic process that proceeds dialectically by presenting arguments and counter-arguments, i.e., attacks on arguments. Like in Dung-style abstract argumentation [30], we take a static view that ignores temporal aspects of argumentation and focuses on the attack relation. Usually, arguments are conceived as premise (“support”) and conclusion (“claim”) pairs. Here, following [21–23], we focus on the interplay between argumentative attacks and the logical form of claims formalized by classical propositional formulæ. Since we ignore the support part of arguments, our attack relation operates not between arguments, but between propositions (claims of arguments). This relation can be understood as the result of an existential abstraction: a claim A attacks a claim B, if there exists an argument with claim A that attacks an argument with claim B in an underlying instantiated argumentation framework. The term ‘semi-abstract argumentation framework (SAF)’ was coined in [22] to emphasize the fact that corresponding attack principles operate on a level that is situated between Dung’s (fully) abstract argumentation frameworks and (fully) instantiated argumentation framework. This corresponds to the claim-centered view on argumentation [35]; SAFs are called ‘claim augmented argumentation frameworks’ in [35].

In [22] logical attack principles have been introduced that are motivated by considerations like the following: if an argumentation framework contains arguments that feature claims A, B, as well as A∧B, respectively, then it seems reasonable to expect that for any argument that attacks an argument with claim A or B there is also an argument attacking an argument with claim A∧B. However, it is much less clear, whether one is also entitled to expect an attack against an argument with either claim A or with claim B if there exists an argument attacking an argument with claim A∧B. In [23] such (qualitative) logical attack principles where generalized to quantitative principles, where the attack relation between claims is endowed with weights in [0,1]. For example, the following principle was considered there: if there is an attack with weight x on an argument claiming A and an attack with weight y on an argument claiming A, then an attack against an argument with claim A∧B should carry a weight ⩾max(x,y). Both, the qualitative and the quantitative scenario, call for a systematic assessment of logical attack principles of the indicated type. The distinguishing feature of our paper is that we endow qualitative as well as quantitative versions of logical attack principles with a probabilistic interpretation that allows us to distinguish between plausible and implausible forms in a principled way, which we also assess empirically. Let us also clarify at the outset that we are interested in principles here that are independent of the concrete content of arguments, but rather only refer to the logical form of the involved claims. A similar proviso applies to weights of attack: we neither propose any particular method of assigning weights nor impose any particular meaning of attack strength that may depend on the given context. Rather we suggest and evaluate principles that potentially apply to any (normalizable) notion of strength of attack.

The outline of the paper is as follows: Section 2 explains the specific level of abstraction of our approach. Section 3 gives a brief survey of qualitative attack principles which were investigated in [22]. We propose a probabilistic interpretation of attack between claims. Specifically, we use coherent conditional probabilities to systematically evaluate the rationality of logical attack principles: coherence serves as rationality criterion for selecting attack principles: “good” principles should be coherent, i.e., they should not violate laws of probability. In Section 4 we show how to model the qualitative attack principles in probabilistic terms. An attractive feature of our probabilistic semantics is that it naturally leads to an interpretation of weighted attacks. The corresponding generalization of qualitative to quantitative attack principles and their probabilistic interpretation is discussed in Section 5. Section 6 presents an experiment which aims to explore the psychological plausibility of selected features of the proposed approach. Then, we present in Section 7 some observations concerning the special case of logical argumentation, where the underlying attack relation is defined in terms of classical logical entailment. Section 8 contextualizes our contributions by indicating some relations to other approaches in computational argumentation and AI. We conclude in Section 9 with some remarks on future research.

2.Argumentation frameworks: Abstract, concrete, and semi-abstract

Dung’s seminal paper [30] introduced abstract argumentation frameworks (AFs). An AF is a directed graph, where the vertices represent arguments and the edges represent attacks between these arguments. Given an AF, the primary task is to compute extensions or admissible sets; i.e., sets of arguments that don’t attack each other and that moreover defend the arguments in the extension by attacking those arguments outside the extension that attack them. Various further properties, in particular maximality conditions, imposed on extensions lead to a plethora of so-called semantics for AFs, including complete, preferred, grounded, and stable semantics. We will not be concerned with these types of extensions here and refer the interested reader to, e.g., the handbook [8] for details.

Dung and his followers showed that the indicated lean and mathematically elegant abstract approach, based on graph theoretic properties of the attack relation, allows one to computationally handle various reasoning tasks arising for nonmonotonic reasoning, including, e.g., forms of logic programming. It is indeed impressive to observe to what extent purely structural properties of graphs (AFs), compiled from large, in general inconsistent sets of statements, assist the assessment of information entailed by such data bases. However, it has also been recognized that one has to pay attention to the logical structure of arguments themselves in order to be able to determine whether a given argument indeed attacks another argument or not. Various logical formats for arguments have been suggested in the literature. For example, Besnard and Hunter [11] have popularized a widely followed approach in which arguments are conceived as pairs ⟨Φ,A⟩, where the support Φ consists of a finite, consistent set of formulæ entailing the conclusion or claim A. Moreover, Φ is required to be minimal (with respect to the subset relation) among sets of formulas with these properties. Other authors (e.g., [7,25,48]) have argued that both, the consistency as well as the minimality condition, are problematic. In particular, Arieli and Strasser [6,83] explore sequent-based argumentation, where arguments are identified with (single-conclusioned) sequents Φ⊢A. In this approach the support part Φ, i.e. the formulas on the left hand side of the sequent, neither needs to be consistent nor minimal. Yet another quite popular approach is ASPIC+ [61], where the structure of arguments is more involved, featuring not only formulas expressing facts, but also default rules as well as strict (logical) rules in the support part of arguments. All of the mentioned formats refer to logical argumentation, where the claim of an argument has to be logically entailed by its support. Moreover, also the attack relation between arguments is defined in terms of logical consequence in various ways. While this is in line with the indicated computational approach to argumentation, logical argumentation is arguably too restrictive to support realistic models of informal argumentation, where attack between arguments is, in general, not a logical relation, but a material one that depends on given interpretations and contexts and that might admit degrees. Although the attack principles that are in the focus of this paper refer to the logical form of claims of arguments, they are not confined to logical argumentation. In particular, except for the specific remarks on logical argumentation in Section 7, we will not be concerned with the specific type of attack that relates two arguments.22 For the logical attack principles introduced in [22] and described in Section 3, below, the specific form of attack is immaterial. In fact, these principles amount to (possible) rationality constraints for AFs also for collections of arguments, where the attack relation between pairs of arguments is not of a logical nature at all.

As outlined above, Dung-style argumentation theory can be thought of as referring to two quite different levels. On the one hand, there are the abstract AFs, where arguments are represented simply as nodes in a directed graph and edges between nodes represent attacks between arguments. On the other hand, there are concrete (instantiated) AFs, where arguments are structured compounds of specific logically complex statements and, possibly, rules of different kinds. The logical attack principles, introduced in [22], that we study in this paper neither operate on abstract AFs nor on the level of concrete AFs. These principles rather focus on the logical form of claims, i.e. on the outermost logical connective of the formula representing the claim of an argument. A particularly simple example of an attack principle of this kind is the following: if an argument γ attacks an argument α that features a claim A, then γ implicitly also attacks an argument β, if the claim of β is A∧B. (Actually, as we will see in Section 3, the attack principles considered in this paper are somewhat less restrictive: rather than requiring that γ itself attacks β, the principle is satisfied if there exists an argument that attacks β.)

Formally, following [22,23], we thus consider semi-abstract AFs, which are ordinary AFs, where each node is annotated with a propositional formula featuring the claim of the represented argument. The expression ‘semi-abstract’ is meant to signal that we are not interested in the possibly quite complex internal structure of concrete arguments; rather, we add information about the logical form of claims to the abstract AF. The emphasis on claims of arguments, rather than on full arguments is by no means new, of course. In fact, the standard approach to computational argumentation in the wake of Dung features the focus on claims as its final step of information extraction: once the appropriate (e.g., complete, preferred, grounded, or stable) extensions have been computed for a given collection of arguments, one usually asks whether a given formula appears as claim A of some argument in either all or at least in some extensions. In the former case, A is skeptically accepted; in the latter case A is credulously accepted. In the context of investigating the computational complexity of corresponding reasoning tasks, Dvořák and Woltran [35] speak of a claim-centric view on AFs. Although we aim at a different target, namely the interpretation of certain rationality constraints on the attack relation, our investigation is claim-centric as well. Note that what has been called semi-abstract AF (SAF) in [22] is called claim-augmented AF (CAF) in [35].

It may seem obvious that in order to attack (an argument with claim) A∧B it suffices to attack either A or B. But it is much less clear whether also the inverse holds, i.e., whether attacking A∧B entails attacking A or attacking B. Neither is it immediately obvious which principles of this type are plausible for disjunctive, implicational, and negated claims. The situation gets even more challenging when, as in [23], one generalizes from semi-abstract AFs to weighted semi-abstract AFs, where a weight is attached to each edge of the graph, intended to signal the strength of the corresponding attack. Still, it seems intuitively justified to impose constraints like the following: if an argument γ attacks an argument with claim A∨B with a certain degree of strength, then either γ itself or some related argument in the given framework attacks arguments with claims A and B, respectively, to at least the same degree. In order to assess the plausibility of this and similar principles in a systematic manner one needs a concrete interpretation of the notion of (weighted) attacks between claims of arguments. After introducing a range of candidates for qualitative (i.e., unweighted) attack principles in the next section, we will provide a probability-based interpretation for them in Section 4. This interpretation straightforwardly generalizes to quantitative principles (i.e., involving weighted attacks) as will be shown in Section 5.

3.Qualitative attack principles

Following [22], we write “A⟶B” to denote that there is an argument claiming A that attacks some argument with the claim B. This notion implicitly refers to a given collection Λ of arguments (i.e., a given AF). However, we neither care about the particular form of the arguments in Λ nor about the nature of the attack relation (defeat, rebuttal, undercut, etc.) defined for Λ. Rather, we abstract away from the given arguments and corresponding attacks and focus on formulas that appear as claims of arguments. Therefore we can safely drop the reference to Λ. Although, strictly speaking, arguments and not claims get attacked, we will express A⟶B as “A attacks B”, which, as just explained, is to be understood as short for “there exists an argument α with claim A in Λ that attacks at least one argument β in Λ with claim B”.

From a computational point of view, one may think of ‘semi-abstraction’ as follows: given Λ one compiles a graph, called semi-abstract argumentation framework (SAF) in [22,23], where the set of vertices is the set of formulas that appear as claims in arguments in Λ. The edges of the SAF are readily computed as indicated above: whenever an argument with claim A attacks an argument with claim B, then there is an edge A⟶B. Clearly, extracting an SAF from an underlying instantiated argumentation framework Λ can be done in polynomial time. The attack principles investigated below refer to the extracted SAF, rather than the underlying AF. In general, the AF is much larger than the corresponding SAF. In any case, checking whether an argumentation framework satisfies logical attack principles, like those presented below, is decidable in polynomial time. This should be contrasted with the complexity of checking, e.g., whether a given set of claims is logically consistent (i.e., satisfiable) or with the intractability results regarding preferred or stable semantics (see, e.g., [33]).

Throughout the paper we will refer to the following example where we illustrate some relevant notions by appealing to a meteorological argumentation framework M.

In the following we will assume that the claims of arguments are presented as classical propositional formulas. Using classical logic allows us to identify propositions with events, which we will be used in Section 4. It is natural to assume that attacking a claim A triggers an implicit attack to any claim B that classically logically entails A (denoted by B⊧A):

Concerning negation, the following principle is intuitively plausible.

One can also formulate inverse forms of the above principles:

The results of [22] imply that imposing all of the above (connective specific) attack principles amounts to an alternative characterization of classical logic, while proper subsets of the full set of these principles lead to weaker logics that result from discarding some of the logical inference rules of Gentzen’s classical sequent calculus LK [40].

Systematic criteria for accepting or rejecting attack principles call for a robust interpretation of the attack relation that is capable of formally supporting (or questioning, as appropriate) informal intuitions about the varying strength of the attack principles.66 In the next section we will tackle this problem by applying coherence-based probability theory for developing a semantics of our qualitative attack principles. This will also provide a natural and straightforward basis for investigating quantitative attack principles.

4.Probabilistic semantics

Probabilistic semantics for argumentation became popular in recent years (see, e.g., [49,53,54,63,76,87]). In light of the results of [22], as sketched in Section 3, the challenge is to come up with an intuitively convincing and formally sound interpretation of the attack relation between claims of arguments. This motivates us to explore to which extent one may employ coherence-based conditional probability (see, e.g., [20,41,71]) for this purpose. The basic intuition of coherence is usually explained in betting terms, specifically in terms of avoiding Dutch books. Accepting a Dutch book implies sure loss, thus making sure to avoid such bets is the basic rationality requirement.

Coherence amounts to the solvability of a suitable finite sequence of systems of linear equations (for corresponding algorithms to check coherence and for further technical details see, e.g., [12,20]). A conditional event C|A is the (conditional, trivalent) object which is measured by the corresponding conditional probability p(C|A).

In betting terms, Definition 2 can be read such that you win the bet on C|A when A∧C is true, you lose when A∧¬C is true, and you get your money back when ¬A is true. Because of its trivalence, C|A cannot be expressed by any Boolean function. Within the coherence approach, conditional probability is primitive (and not defined by the fraction, p(A∧C)/p(A), which—in order to avoid fractions over zero—requires positive-probability antecedents, p(A)>0) and allows for properly managing zero-probability antecedents. The latter property is important, for example, to avoid counterintuitive inferences, like the following paradox of the material conditional:

Concretely, we suggest to read “F attacks A” as the assertion that it is likely that A does not hold, given that F holds. More precisely, we suggest an interpretation of F⟶A as p(¬A|F)⩾t, which is parameterized for some threshold 0.5<t⩽1. We note that p(¬A|F)⩾t is equivalent to p(A|F)<t. The latter formulation is simpler, but since we are interested in measuring the strength of attack, we prefer the former formulation, which provides a lower bound on the strength of attack. Throughout the rest of the paper, we assume that F is not a logical contradiction (i.e., F is not equivalent to ⊥, where ⊥ denotes the truth constant falsum). This assumption is not only intuitively plausible (because assuming ⊥ to be true does not make sense) but also technically important for us, since although zero-probability antecedents are allowed within the framework of coherence, p(A|⊥) is undefined. Note that coherence requires that p(A) must be equal to zero if A is a logical contradiction (since ⊥ cannot be true, in betting terms, you can never win when you bet on the truth of ⊥), while the reverse does not hold: if P(A)=0, this does not mean that A is a logical contradiction, i.e., A could be contingent. Approaches to probability, where the values 0 and 1 are reserved for contradiction and tautology, respectively, are sometimes called “regular”. The coherence approach is more general than approaches using regular probabilities, as 0 and 1 can also be assigned to contingent events.

We emphasize that the suggested interpretation of F⟶A does not determine a specific interpretation of the attack relation between arguments of Λ itself. In particular, attack between arguments does not have to be defined in terms of probability. Only the relation between corresponding claims of arguments is interpreted probabilistically.

Note that our probabilistic interpretation operates on the semi-abstract (or claim-centric) level that shifts attention from individual attacks between arguments to the mere existence of attacks between arguments featuring certain claims. In this manner, adopting our interpretation amounts to imposing certain rationality constraints about the overall coherence of a given AF on the level of considered claims. So far, we only demand that each claim should always be possibly true according to some interpretation, i.e. it should not be self-contradictory. Further constraints of this kind will arise from the corresponding interpretation of our logical attack principles.

Translating the attack principles that refer to conjunction, disjunction, and negation according to the suggested interpretation is straightforward. The following probabilistic constraints correspond to the principles (C.∧), (C.∨), and (C.¬):

If p(A) and p(B) are stochastically independent, then p(A∧B)=p(A)·p(B). However, if stochastic independence cannot be presupposed, the probability of the conjunction of A and B is given by the lower and upper Fréchet bounds, max{0,p(A)+p(B)−1} and min{p(A),p(B)}, respectively. The same bounds hold for corresponding conditional probabilities. The corresponding rule coincides with the probabilistic version of the (And) Rule of System P, which is among the most prominent systems of nonmonotonic reasoning [58]. It is proven to be coherent in [41]:

We now turn to arguments featuring conditionals as claims. As we use classical logic, A⊃B is equivalent to ¬A∨B. The corresponding translations of (C.⊃) and (C.⊃)’ are as follows:

We note that interpreting attack principles involving the implication connective is delicate in general, since it is widely agreed that the natural language conditional (‘if …, then …’) should not be identified with classical (truth-functional) implication. Actually, as argued, e.g., in [42,66], coherence-based conditional probability itself provides a sound and robust semantics for the conditional. Moreover, the coherence approach turned out to be very useful for modeling argument strength [65,70]. Following this insight would force us to use degrees of beliefs in nested conditionals (e.g., in terms of previsions in conditional random quantities; see, e.g., [44,77–79]) to interpret principles like (C.⊃). While this is an interesting topic for future research, here we only want to check how our probability-based interpretation of the attack relation classifies (C.⊃) and (C.⊃)’ when classical logic is assumed. Therefore, we have chosen to use the material conditional interpretation of conditionals in our analysis.

In [22] also logically contradictory claims are considered by formulating the following corresponding attack principle:

The principle (C.⊥) is probabilistically interpreted by

5.Quantitative attack principles and their semantics

So far, we have only discussed qualitative attack principles, i.e., principles that only care for the presence or absence of an attack between (claims of) given arguments. However it is natural to refine such an analysis by considering weights or varying strength of attacks. Various suggestions regarding weighted AFs can be found in the literature on argumentation in AI, see, e.g., [1,3,5,9,17,24,26,32]. But, to our best knowledge, there is no investigation yet of rationality postulates that systematically relates weights of explicit and implicit attacks to the logical form of the involved claims of arguments. However, see Section 8 for some remarks on, at least vaguely, related work.

A first step in that direction has been attempted in [23], where the principles introduced in [22] are generalized to the context of weighted AFs. The aim of [23] is to explore under which assumptions one can characterize various t-norm-based fuzzy logics in terms of ‘weighted attack principles’. As expected, it turns out that some of the principles that are needed to recover a truth-functional (fuzzy) semantics are implausible from an intuitive, argumentation-based point of view. In any case, the situation, once more, calls for a systematic interpretation of the relevant principles, that enables one to formally judge their respective plausibility. Fortunately, the probabilistic interpretation of the qualitative attack principles, developed in Section 4, generalizes in a very direct and natural manner to the quantitative scenario.

Rather than just distinguishing between F⟶A and F⟶̸A (“F attacks / does not attack A”), we will use F⟶wA to denote that F attacks A with the weight (or to the degree) w. Let us stress again that “attack”, here, is a relation between propositions and not between arguments. In the literature, there are various suggestions for generalizing ordinary AFs to weighted AFs (or systems), where real numbers attached to attacks between arguments are intended to represent degrees of strength of such attacks (see, in particular, [32]). In analogy to the qualitative scenario of Sections 3 and 4, one may understand F⟶wA to refer to an underlying weighted AF in some specific manner. For example, one might want to identify w with either the average or with the minimum of weights of all attacks of arguments with claim F on arguments with claim A and set w=0 if no corresponding attack exists. However, since we are only interested in rationality constraints arising for logically complex antagonistic claims, we will treat weights of attacks between propositions as primitive, here. These weights are understood to be normalized, with 1 being the maximal weight of any attack, whereas F⟶0A means that F in fact does not attack the claim A at all. Note that this stipulation entails that the qualitative scenario discussed in Sections 3 and 4 amounts to an instance of the weighted case, where the only possible weights are 0 and 1. We deliberately refrain from prescribing specific, context-dependent methods for determining concrete weights of attacks, since we are interested in principles that do not depend on the specific content of the involved statements, but only on their logical form.

An attractive feature of the probabilistic approach taken here is the fact that it immediately leads to a quantitative refinement of the qualitative case: interpreting attacks in terms of coherent conditional probabilities suggests to directly attach weights, instead of using thresholds to judge whether a given statement attacks another one. As pointed out in [23], there are several non-equivalent ways in which the qualitative attack principles reviewed in Section 3 can be generalized to ‘weighted attack principles’. The most straightforward generalization of principle (C.∧) to weighted attacks is arguably the following:

Alternative weighted attack principles for conjunction, formulated in the same manner, are:

As already indicated, in contrast to the qualitative case of Section 4, we do not have to involve threshold values in interpreting a weighted attack relation between claims, but simply identify the weight with which F attacks A with the conditional probability that A does not hold, given that F holds. More formally, our probabilistic semantics interprets F⟶wA by p(¬A|F)=w. (Remember that this is only viable if we exclude the possibility that F is a logical contradiction; although, we allow for the possibility that p(F)=0.) Once more, we point out that interpreting weights between claims of arguments as probabilities does not mean that we have to interpret also weights of the underlying attacks between arguments probabilistically. The suggested semantics operates on the semi-abstract level that deliberately ignores the fully instantiated level of attacks between concrete arguments, which may well depend on the support part of arguments and not just their claims.

According to the probabilistic semantics, the above versions of weighted attack principles translate into the following statements:

We now investigate which of the various possible weighted attack principles for conjunction should indeed be adopted as rationality principles constraining the underlying AFs, if we follow the interpretation of weights of attacks between claims as coherent conditional probabilities. We obtain the following corresponding classification.

Although principles (Ł⩾w.∧), (P⩾w.∧), (G⩽w.∧), and (P⩽w.∧) do not hold generally under coherence, the corresponding conditions (Ł⩾w.∧)p, (P⩾w.∧)p, (G⩽w.∧)p, and (P⩽w.∧)p, respectively, may hold for particular probability assignments. Consider, for example, the following propositions:

From the just outlined evaluation for the attack principles involving conjunction, we now turn to attack principles involving disjunction, which are of course, dual to those for conjunction.

Likewise, we use (G⩽w.∨), (Ł⩽w.∨), and (P⩽w.∨) to refer to the principles that arise by just replacing ‘⩾’ by ‘⩽’ in the respective constraint and we obtain the following proposition:

Analogous results can be obtained for principles involving conditionals, since the material conditional A⊃B is logically equivalent to the disjunction ¬A∨B. For example, for Gödel logic we obtain the following two principles:

Concerning negation, our semantics naturally suggests the following attack principle:

We recall that according to the semantics of Łukasiewicz logic, if the truth value of A is x, then the truth value of ¬A is 1−x, which coincides with the negation in probability theory as expressed in (Ł.¬)p. However, negation in Gödel and Product logic is different: in both logics the truth value of ¬A is 0 if the truth value of A is positive and 1 otherwise. Corresponding principles would not be justified within coherence-based probability semantics when classical negation is used.

Regarding falsum we obtain the following principle:

Regarding implication, one may of course extract corresponding principles from the above mentioned ones, under the stipulation that A⊃B is understood, classically, as equivalent to ¬A∨B. But, as already indicated, it would actually be more adequate to model (informal) implication not as a disjunction but as a proper conditional. This leads to the tricky and, as yet, only partially explored terrain of iterated conditional probabilities (for an approach within coherence, see, e.g., [43–46,77,78]); thus providing a challenging topic for future research.

6.Psychological experiment

In this section we present a first experiment which serves to explore empirically the psychological plausibility of the interpretation of the attack principles in our approach. Table 1 gives an overview on the investigated argument forms/formulas. Coherence-based probability logic received empirical support in recent years (e.g., [57,64,68,69,72]). However, principles governing the strength of attacks have not yet been investigated empirically (neither within nor outside the coherence framework; for an overview on empirical work on abstract argumentation see, e.g., [16]).

Participants. The sample consists of 139 computer science students who took part in the lecture Formale Modellierung at the TU Wien (Technical University of Vienna, 18 female, 116 male, and 5 who chose not to reveal their gender) with a mean age of 21.1 years (SD=3.2). Only German native speakers were included in the data analysis. Seven participants were excluded from the analysis because of missing data in the target tasks. Most students were in their second semester and did not receive a thorough training in logic yet.

Table 2 shows that, on the average, the participants rated the overall task clearness88 and difficulty99 on an intermediate level (M=4.9 and M=4.3, respectively, on a rating scale out of 10). The intermediate task difficulty ratings indicate that the tasks were neither perceived to be too easy nor too difficult: this is good, as extreme values on this scale could indicate a decreased motivation for solving the tasks. However, concerning the overall comprehensibility of the tasks, ratings closer to the maximum (10) would be preferable compared to the observed average close to 5, which hampers the interpretability of the data and restricts its conclusiveness. The intermediate comprehensibility of the tasks could be due to the implicit and explicit negations in the task material: psychological reasoning research indicates that negations are harder to process (see, e.g., [36]), which may thus negatively impact perceived task comprehensibility. More specifically, recall the distinction between reasoning to an interpretation and reasoning from a fixed interpretation [82]. This distinction refers to two general reasoning processes: firstly, participants reason how to interpret the task; secondly, after fixing their interpretation, participants reason from their interpretation to the conclusion [82]. If the process of fixing the interpretation is hard (presumably because of negations), then the perceived task comprehensibility is lower compared to straightforward tasks. Moreover, processing problems during the reasoning to an interpretation may also explain why the participants were not highly confident in the correctness1010 of their solutions (M=4.1 out of 10) even if in general they tend to like solving logical/mathematical problems1111 (M=7.5 out of 10). Overall, we observed positive correlations between confidence in correctness and task difficulty ratings, r(137)=.52, p<.001, and between confidence in correctness and task comprehensibility ratings, r(136)=.37, p<.001: the higher the confidence in correctness ratings, the easier and more comprehensible the tasks were rated.1212 Interestingly, there was no statistically significant correlation between task comprehensibility and difficulty ratings, r(136)=.16, p=0.058: whether the tasks were rated as comprehensible or not had no impact on how difficult the tasks were rated.

Fig. 1.

Sample Task A1. Task type: quantitative, judgment of correctness task. Response format: forced-choice. For the corresponding argument form (Conjunction elimination) and the competence response, see Table 1.

Fig. 2.

Sample Task C1. Task type: quantitative, attack strength rating task. Response format: open. For the corresponding argument form (Conjunction elimination) and the competence response, see Table 1.

Method and materials. Each participant was given a A4 sheet of paper, containing an introduction on the first page and the target tasks on both pages. There were three between-participant conditions (to test inter-group differences), two with forced-choice (group A: n1=44 and group B: n2=48; see, e.g., Fig. 1) and one with an open choice response format (group C: n3=47; see, e.g., Fig. 2). We hypothesised that the forced-choice tasks were easier compared to the open response format tasks, since judging attack strength candidates requires less cognitive effort and is hence considered to be easier compared to generating attack strengths. After showing that the degree of attack can be expressed on a scale form 0 to 10 and that claims can also be compounded (like [A and B]), the participants were presented with tasks corresponding to the argument forms described in Table 1. The participants’ task consists in evaluating possible consequents of the respective conditional, either in terms of judging the correctness of presented consequent candidates (groups A and B, forced-choice format, illustrated in Fig. 1) or in terms of generating strengths (group C, open response format, illustrated in Fig. 2). The Conjunction elimination tasks A1 and C1, for example, present the antecedent of the conditional “If A attacks with exactly the strength 7 the claim [B and C], then …” and differ in the way how the participants complete the conditional’s consequent (compare Figs 1 and 2).

For the sake of simplicity, we omit references to underlying sets of arguments featuring such claims. Therefore, we assume that attacks can be viewed as directly relating claims rather than requiring reference to the possible complex underlying arguments. Moreover, we are interested in how logical form impacts on reasoning and not how contexts may influence the participants’ interpretation of the attack principles. Therefore we have chosen abstract task materials. Since the sample consists of computer science students we did not expect a priori problems of understanding the (semi-)formal character of the tasks. Moreover, concrete task materials derived from everyday life examples may yield belief biases, i.e., that people may ignore the logical form and draw their inferences based on what they think is factually true (see, e.g., [37]).

For communicating quantitative degrees of attack we decided to use point-values and intervals constructed from the numbers 0, 3, 7, and 10 only. The reason for this restriction is to keep the task simple for the participants while still allowing for investigating quantitative attack principles. Tasks A9, B10, and C16 requested participants to respond in terms of probability judgments. Consequently, the values of the attack strength candidates were replaced by probabilities and the labels “0” and “10” in the illustrative scales were replaced by “P(B|A) = 0” and “P(B|A) = 1”, respectively.

In the tasks of conditions A and B, nine consequent candidates were presented, which completed the conditional. Eight consequents were of the form “… attacks A with [M] with the strength [S] the claim B”, where “[M]” indicates a precise value (“exactly”), a lower (“at least”), or an upper bound (“at most”) on the strength [S]. [S] was either 0, 3, 7, or 10. All possible point and interval options were formulated in ascending order (see Table 3 for the attack strength options we used). Except for the interval [0,10] we used “nothing follows about how strong …attacks …”, as the ninth response option. The participants were asked to tick for each of the nine items whether the according sentence is wrong (“falsch”) or correct (“richtig”). In the strength generation condition C, the participants were instructed to fill in “exactly”, “at least”, or “at most”, the value of the strength, and additionally had to mark the strength of attack (either as a point value or as an interval) on a scale as introduced at the beginning of the instructions.

All participants were presented with quantitative and with qualitative task types (see Table 1). The quantitative task types were of the kind we just described. In the qualitative task types the participants were asked to choose one among the three options “wrong”, “correct”, or “undetermined” (unbestimmt) by ticking a corresponding box. The qualitative Negation attack′ task A5, for instance, asked the participants whether the following assertion is in principle wrong, correct or undetermined (Ist folgende Behauptung grundsätzlich falsch, richtig oder unbestimmt?): It is not the case that: A attacks not-A.

Procedure. The experiment took place during the last part of the first lecture of the course (Formale Modellierung). The students were informed that the experiment aims to investigate systematic relationships between logical form and argumentation, that their participation makes an important contribution to research, that the experiment is (not about testing skills but) about finding out how people deal with certain argument forms, that participation is voluntary, and that anonymity is guaranteed. Moreover, to foster independent processing of the tasks, we informed participants that there are different versions of task sets. We also told the participants that they should first read carefully the introductions and then the tasks. We stressed that the individual claims differ: sometimes only in detail. The participants were asked to think carefully and to take as much time as they need for answering the questions. Then we made the importance of answering the questions independently explicit by also explaining that it would be very unfavorable for the statistical analysis and would distort the data if the participants influence each other during the experiment. We expected that not everyone will finish the filling in process at the same time. Thus, in order to further prevent conversations during the experiment and to keep the noise in the lecture hall down, we asked the participants to keep quiet and remain seated until the sheets are collected.

Then we continued with administering the task pages. The three conditions were administered in a systematically alternated way to reduce the chance of plagiarized responses.

Results and discussion. The main results are presented in Tables 3–7. First, we observe that most people are unaware of the best possible (or tightest) coherent bounds (marked in italics and bold). Responses which are within the best possible coherent bounds are of course also coherent, like in the Conjunction elimination task A1 where 45% of the participants judged that “precisely 7” is correct. In this task, 43% judged that the interval “at most 7” is correct, which corresponds to the best possible coherent interval. The response patterns in the corresponding strength generation task C1 were analogous: most participants responded with a precise value equal to the coherent upper bound. Thereby, the participants neglect that the value zero is the best possible coherent lower bound. Second, we observe that compared to direct tests of coherence-based probability logic (e.g., [64,68,69,72]), the agreement between the predictions concerning the quantitative attack principles and the participant’s responses are modest, especially in the correctness judgment conditions (A and B). For condition C, which requires to generate strengths of attacks, the majority of the participants hit some of the optimal bounds as predicted (see Table 6). In particular, looking at the median values, we observe that out of all 11 quantitative tasks of condition C, more than half of generated strengths correspond to the best possible coherent intervals in four tasks (C5, C6, C13, C16), the best possible coherent upper bounds (while most lower bounds were incoherent) in five tasks (C1, C2, C4, C7, C11), the best possible lower bound in one task (C17), and a coherent but not optimal upper bound in one task (C3). From the seven qualitative tasks in condition C, more than 50% of the responses confirmed our predictions in four tasks (i.e., C8, C10, C12, C19; see Table 7). Moreover, there is some tendency towards the predicted responses in the tasks C14 and C18.