Working on the argument pipeline: Through flow issues between natural language argument, instantiated arguments, and argumentation frameworks

2.1.Sources and examples

[61] present a policy-making discussion adapted and abstracted from a BBC Have Your Say discussion list. The BBC’s Have Your Say introduces a question for discussion, provides brief background points, then allows users to write in comments in a monitored, unthreaded discussion list.44 For the particular study, [61] considered comments in response to the question Should people be paid to recycle?, where 697 comments were published (and 35 were rejected).55

The comments as they are present a knowledge engineer with a wealth of unstructured textual data since the discussion list is unthreaded and has very few substantive constraints on what participants can contribute or how. Among the many issues such a discussion raises for the analyst, we see the following. Participants may use any vocabulary or syntactic structure they wish (so long as it does not lead to the rejection of their comment); they may use novel words or expressions, metaphors, misspellings, and ungrammatical sentences. There is no check that the meaning as interpreted is the meaning as intended. Implicit or pragmatic information in the comments must be inferred by the reader. There are no over indications of inter-comment relationship (unless the commenter explicitly refers to one or more previous comments). The comments can represent uncertain or inconsistent information. The comments can be vague and ambiguous. The interpretation of comments is left to the analyst to infer.

Beyond these empirical observations, no current computational linguistic or text-mining systems accurately parse the large majority of sentences, yield the acceptable correlated semantic interpretation, and can be used to support the acquisition of the knowledge for argument representation. Text-mining can extract some useful information (e.g., sentiment analysis [33], and some aspects of rules [56,57], and contradictions [47]), but is still limited, error prone, and does not capture the complex meaning of each comment, much less the complex web of meanings comprised of the semantic relationships among the comments. Large corpora can be queried for extraction of argument elements, thought this is not parsing and semantic representation [39,58,59]. There are parsers with semantic interpretation that work on large scale [10], but require detailed examination and refinement to produce appropriate output [55]. Other parsers with semantic interpretation work on a fixed range of corpus-specific constructions [38,46]. There is work on textual entailment [18], some aimed at argumentation at a coarse-grained level [13,14]; however, this is insufficient to represent the internal structure of arguments and reconstruct relevant relationships. Uncertainties and inconsistencies (other than explicit contradictions) are difficult to identify in current approaches. It remains a very significant, manual task to take a list of comments from a blog and transform them into a knowledge base that is suitable for a KR task. There has been intensive recent interest in and progress on argument mining, which is automatic extraction of structured arguments from unstructured textual documents (see [30] and citations therein). It remains highly challenging to extract the components of structured arguments and contrasting statements from source text.

Given the current state of the art in argument mining, an alternative approach to input of sentences and rules is to use a controlled language, e.g., ACE used here (also see [52,60]. This allows us to input homogenised, structured information with respect to the lexicon and syntactic forms, allowing inferences and contrast identification. We can take controlled expressions as an intermediate form between source unstructured text and what is needed for formal argument evaluation; expressions mined from unstructured text might be mapped to the intermediate form so as to facilitate argument evaluation. Moreover, the focus of this paper is on the flow through issues rather than NLP or sentence input per se.

[61] scope the issues, which is a common practice in semantic studies in Linguistics. By scoping the issues, the data is filtered, normalised, and reduced in scale so as to define a problem space that is on the one hand close to or derived from the source data and is on the other hand amenable to yet challenging for the theoretical and implemented techniques. It allows us to address and solve particular problems, abstract issues, and make progress which otherwise is prohibited by an ill-defined scope. While the resulting example is relatively far from the source material, it is still a useful starting point. By addressing relatively artificial examples, we can highlight issues, then extend the approach incrementally to solve additional and more natural problems and examples.

With these programmatic points in mind, we adopt and adapt the approach and sentences in [61], where 16 sentences are automatically translated into First-order logic (with some modal operators), manually indicating where there are rules and contradictions: users input sentences using the controlled language editor, then indicate rule relationships between sentences such as premise, claim, or contradiction. The result is a (possibly inconsistent) knowledge base.

In the current paper, we have 19 example sentences; in Section 2.3, we justify our revisions and discuss the sentences. For the purposes of this paper, we only work with Propositional Logic because the theory of argument formation for expressions in First-order logic is more complex than we need here and because the implemented system that we use for generating arguments and relations only works for Propositional Logic. A representation in First-order Logic is crucial in order to identify and reason with contradiction, redundancy, implication, ontological argument, fine-grained information extraction, and query; in future work, we anticipate working with First-order Logic Statements. We have, therefore, assigned each sentence a lower case propositional variable. Each rule has a lower case rule name.66

Example sentences. 

As in [61], where a rule has more than one premise, we take it that the premises conjunctively hold for the claim to hold. The rules are definite clauses – Horn clauses with exactly one positive literal. We assume that users construct rules from the propositions.

Rule set. 

In addition to rules, we have what we express here as several constraints (or meaning postulates) on models which express contradiction between propositions. For example, p18 Tom is an objective expert about recycling and p11 Tom is not an objective expert about recycling are intuitively contradictory, so we write ¬p11 ∨ ¬p18. We discuss below further issues relating to contradiction

Constraints. 

We can represent the structure of the literals and rules into a graphic where solid lines indicate a premise relation and dashed lines indicate a contradiction (see Fig. 1).

Fig. 1.

Relationships between statements – premises, claims, contradictions.

In the next section, we briefly outline issues bearing on the syntactic parsing and semantic interpretation provided by ACE.

2.2.Parsing and semantic representation with Attempto Controlled English (ACE)

[61] provide a detailed discussion of the application of ACE to our candidate sentences. In the context of this paper, where the emphasis is on providing the relationship between the input statements, rules, and an argumentation system which works only on propositions, the importance of using ACE is reduced. Yet, it is relevant here to give a brief overview in that ACE controls the language of the policy discussion and also provides expressions in First-order Logic (with some modal expressions) that can be used in future implemented argument instantiation systems [5,21].

To facilitate the processing of sentences, we use a well-developed controlled natural language system – ACE (for an overview of the properties and prospects of controlled natural languages, see [52]). A controlled language has a specified vocabulary and a restricted range of grammatical constructions which are a subset of a natural language (e.g., English). Sentences written and read in the controlled language appear as normal sentences, but can be automatically translated into a formal representation.

ACE supports a large lexicon, a range of grammatical constructions, discourse anaphora, and correlated semantic interpretations: negation on nouns or verbs, conjunction, disjunction, conditionals, quantifiers, adjectives, relative clauses, discourse anaphora, modals (e.g., necessity and possibility), possessives, prepositional phrases, verbs with three arguments, and verbs with subordinate clauses. ACE checks that the sentences input to the system satisfy the constraints of the syntax and semantics of the language, thus the user is only able to input grammatically acceptable and semantically interpretable sentences in building the knowledge base. ACE has an associated reasoning engine to test for consistency and draw inferences.

As discussed in [61], the syntactic form must be crafted to generate the intended semantic interpretation (see further discussion below). Our 19 sentences are grammatically well-formed in ACE and have the intended interpretations.77

2.3.Considerations

In developing the sentences for input to a generator of instantiate arguments, four issues arose – material development/user groups, making contradictoriness explicit, the introduction of implicit premises, and the construction of well-formed rules in the LRKB. We discuss these in turn.

As mentioned above, input to ACE must be crafted, which highlights the problem of engaging members of the public as knowledge engineers and using a natural language since this gives rise to a range of expectations which must be addressed. The user must accommodate to the restrictions of the language, which limits the spectrum of users of the system; however, research indicates the language is not unduly difficult to learn or use [29], university students (among others) could learn to use it. On the other hand, the system allows users to communicate in a form that they are most familiar with, and in this aspect the approach improves over other policy-making tools. As a controlled language, we work with a (large) fragment of natural language that can be incrementally developed with further linguistic capabilities. Finally, it is our assumption that the system is used in “high value” contexts by participants who are willing and able to adapt to some of the constraints (topic, expressivity, explicit marking of statement relations) in order to gain the advantages of collaboratively building a clear, explicit knowledge base which represents a range of diverse, and possibly conflicting statements. Related issues arise with wide-scale parsers [55].

To generate instances of arguments and their relations, we require explicit indicators of contradictoriness. Consider statements such as Bill exercises daily and Bill does not exercise daily; the lexical items and syntactic form are such that it is relatively easy to identify these as contradictory. Systematically translating from natural language expressions with negation to logical forms with negation is a complex issue itself, though we do not discuss it here. However, rather than stating Bill does not exercise daily, suppose a participant claims Bill is a couch potato, which is also contradictory with Bill exercises daily. Yet, the lexical items and syntactic form are significantly different; it requires lexical semantic and syntactic knowledge to know that these expressions are contradictory. Indeed, there are a broad range of linguistic forms for expressing contradictoriness [16]. While ACE easily identifies the inconsistency between Bill exercises daily and claims Bill does not exercise daily, it misses the inconsistency between Bill exercises daily, and Bill is a couch potato.

There are two approaches to addressing such contradictions. One strategy is for the user to introduce an explicit constraint which simply states which two sentences are contradictory. If Bill exercises daily is represented as the propositional variable P, Bill does not exercise daily is ¬P, and Bill is a couch potato is R, ¬[P ∧ R] is a constraint – P and R cannot both be true in a model. Yet, unlike the logical contradiction [P ∧ ¬P] of Propositional Logic, non-logical constraints must be independently given in the knowledge base. Besides having to be specified, an additional problem of constraints is that there is no explanation of the contradiction; that is, they are facts of the knowledge base and follow from or relate to no other elements of the knowledge base.

To overcome this, an alternative strategy is to explicitly express the rules that represent the implicit knowledge which gives rise to the inconsistency. Such rules may be encoded in auxiliary KBs, e.g., ontologies or lexical semantic structures. For instance, Bill is a couch potato implies that Bill does not exercise daily. Thus, in stating Bill is a couch potato one ought to introduce as well the rule that it implies that Bill does not exercise daily; then, where it is asserted that Bill is a couch potato and Bill exercises daily, we would find, by inference, that an inconsistency arises. Such an approach relies on some pre-existing and accepted terminology by the users, that is, that exercise daily is in the lexicon already such that the contradiction of it is meaningful. More broadly, contradictoriness is a rich, complex, research domain that is only recently yielding results [3,12,16,17,47].

A final issue relates to the construction of well-formed rules. Given a set sentences, participants can, in principle, introduce any combination of literals to produce a rule. Yet, some of these appear to be well-formed and meaningful, while others do not. For example, the following argument is formally and syntactically well-formed as well as semantically coherent: Socrates is a man and Every man is mortal therefore Socrates is mortal. In contrast, the following is formally and syntactically well-formed, but semantically incoherent, we would not want to accept it as an argument: Bill is rich and Jill sails therefore Socrates is mortal. Clearly in the latter example, we are missing some semantically relevant relationships between the premises themselves and between the premises taken together and the claim; in this regard, the former example is more coherent. While the former example may have a relatively simple explanation for this coherence (terms are shared), the coherence of more complex examples such as argumentation schemes which represent normative patterns of presumptive reasoning are more difficult to explain [48]. A related issue is that some patterns of argumentation may contain enthymemes in addition to those used to identify contradictions [8,49].

On the construction of well-formed rules, we have some of the following general questions. What are the conditions on semantic well-formedness for the rules provided to the knowledge base? What makes a set of statements semantically cohere as premises of a claim? What makes a set of statements semantically imply a conclusion? While the mathematical or syntactical conceptions of a rule may be clear, and logicians or knowledge engineers may implicitly know how to write well-formed rules, the principles which underlie the linguistic semantic instantiations are not clear. If we allow non-logicians or knowledge engineers to build rules, what guidance can be provided to build meaningful rules? That is, how can we make explicit what logicians and knowledge engineers implicitly know?

These topics touch directly on complex issues relating to the interface of human reasoning, language, and formal reasoning [20,44]; indeed, in addition to the issues that arise about controlled languages, it is problematic to presume that participants can, without training, contribute to a normative rational discussion in a way which supports automated reasoning; a similar point applies to the reconstruction of arguments from texts by participants. In other words, a degree of literacy in operating the system and interpreting the results is going to be required.

While all of these issues are substantial, they are partially addressed in our approach to collaborative, incremental, distributive knowledge base construction. For example, suppose there are two sentences in the knowledge base which are mutually incompatible (e.g., our Bill exercises daily and Bill is a couch potato) but were not overtly recognised or marked as such by users; this could arise simply because of what the users attend to or the size of the knowledge base. Some subsequent users could (on separate occasions and without being aware of the input of other participants) introduce the relevant implications to yield the representation of contradiction. Similarly, rules which are constructed by some parties could subsequently be rendered inapplicable by asserting that a premise does not hold. Finally, as implicit knowledge is solicited, it is explicitly introduced into the knowledge base which can then support further explicit reasoning. In this industrialised vision of knowledge engineering, individual participants may be logically imperfect and partially ignorant, yet contribute to the construction of globally perfect knowledge bases which arise after the calculation of argument extensions. In this regard, the system does not so much rule out erroneous statements or rules as to allow them to be modified by the input of other participants and, more importantly, reasoned with to determine consequences which would highlight the problems.

Finally, we should touch on one representational issue. At different stages of the pipeline, different forms of representation are useful to support distinct reasoning processes. There is a balance to be struck at every step between abstraction and expressiveness. For instance, while we have used lower case letters and numbers, we could have used the sentence strings themselves as propositional variables. While we have left resolution of these issues to future work, it is not particularly problematic so long as we are clear about and consistent about the conversions at each state.

3.1.JArgue

JArgue is an implementation in Java of logic-based argumentation [6]; given an LRKB, JArgue generates arguments, attacks between arguments, and an argument tree.99 One provides JArgue with an LRKB as a text file of propositional clauses (conjunctive normal form); implicational rules such as p → q are rewritten as !p|q, where ! is negation and | is disjunction. Where the antecedent of a rule is a conjunction of literals as in [r ∧ s] → q, we rewrite the negation of the antecedent as a disjunct of negated literals – !r|!s|q. Starting JArgue and inputting an LRKB, one can request the generation of arguments where a particular literal is the claim; given the arguments, one can then select among the arguments and request the generation of the argument tree. We discuss JArgue and illustrate it with our example.

From Section 2.1, we have the propositions p1–p19 and rules r1–r8. Several preliminary comments are needed before presenting the input LRKB. First, since incompatible literals is a significant issue, we did not presume to represent statements which were incompatible as logical contradictions, e.g., p1 and p2, p11 and p18, and p15 and p17, even though ACE translates these pairs as contradictory. In other words, we presumed the problematic examples of contradiction rather than the unproblematic.

However, to determine attack relations, the logic-based approach requires the explicit expression of contradiction as in classical logic. We therefore make the following substitutions: p2 ≡ ¬p1, p11 ≡ ¬p18, and p17 ≡ ¬p15. This is an issue relevant to the illustration, not to the functionality of either the logic-based approach or JArgue since expressions of contradiction can be added to the knowledge base as it is built. However, the choice here is to represent such incompatibilities as logical contradiction rather than as constraints of the form ¬[p1 ∧ p2] (or in conjunctive normal form for JArgue !p1|!p2). We explore this choice in future research.

Given these equivalences, the related rules are adjusted by substituting in equivalent expressions:

Our LRKB for JArgue will be comprised of literals and (conjunctive normalised) rules. However, there is a somewhat tangential issue of whether we need include all literals since some serve only as intermediaries between rules (see the discussion of intermediate concepts in [51]); that is, if we have a language comprised of p, q, r, p → q, q → r, there is no reason to include in the KB q (or r) itself since it is enough to include all the rules and only those literals which are terminal leaves in a graph of all the rules as in Fig. 1; in other words, the KB need only represent what is asserted along with the rules, not what is simply expressed in the language. In a dynamically constructed KB, what is asserted and what is inferred may change over time. We presume this simplification and leave further discussion to future research. Thus, we have rules r1–r8 as revised above and only p6, p7, p8, p9, p10, p12, p13, p14, p16, p18, and p19. Other literals are inferred.1010

With these points in mind, our input LRKB for JArgue is as follows:

To run JArgue, we add this LRKB, then query the KB to generate an argument. Querying the KB can only be done for one literal at a time. For our example, if we query for p1, we get the following argument, which we label a1:1111

JArgue then requests the user to select from an argument with respect to which it should generate an argument tree. Selecting a1, JArgue generates an argument tree with two counterarguments a2 and a3 along with one counterargument a4 to a3; for the sake of discussion, we discuss each node and their relationship rather than present the graph. The first counterargument a2 is an undercutter of a1 since the claim of a2, !p18, is the negation of a literal which appears in the support of a1:1212

If we query the LRKB for !p1 and keep the argument labeling the same, we have a3. The argument tree for a3 gives us a1 as a counterargument; this shows that where we have contradictory claims, the arguments attack one another. Otherwise, a2 is a counterargument of a1, and a4 is a counterargument for a3 as before.

Querying just the argument trees for p1 and !p1, we can create an AKB, which is a knowledge base of generated arguments in attack relations (indicated with att), which is given in the argument tree. Such an AKB is suitable for input to an abstract argumentation framework to generate an AEKB, which allows us to reason abstractly over arguments. This is AKB1:

3.2.Considerations

Where we choose to argue for and against literals at the root of an argument tree such as about p1 and !p1, the arguments and relations among arguments appear relatively straightforward. It is important to note that we have used JArgue to generate argument trees with respect to a query on a proposition. However, we are using it as well to creating AKBs for input to abstract argumentation frameworks, which is the reason to investigate the argument trees for a proposition and its negation; however, using JArgue to generate arguments and relations to support abstract argumentation frameworks is somewhat beyond the brief, as we see in a moment.

A relevant general question arises – how many arguments and attack relations among them can be generated from the underlying LRKB? Given that we have classical logic, contrapositive reasoning applies; that is, given ¬q and p → q, we can infer ¬p. Thus, if we query p3, which is implied by p10, which is itself asserted, we can generate an argument and an argument tree.

For our purposes, we only include those arguments and relations relative to a debate under investigation, here p1 and !p1; these are the arguments and relations as generated by JArgue. Other arguments and counterarguments such as for p3 are not clearly relevant. Indeed, perhaps all the other arguments and relations arise via contraposition are problematic; indeed, it is questionable whether contraposition is a natural reasoning rule (see [20,44]).

The hedge, however, bears on how we use AKB as input to the abstract argumentation framework discussed in Section 4, where we find the AKB with four arguments and four attacks is not sufficient to get a correct result. This is so because while a2 attacks a1, and a4 attacks a3, nothing attacks a2 or a4; thus, these arguments attack and defeat a1 and a3. The extensions leave just a2 and a4, which is contrary to our desired goal, which is that a1 and a4 could hold together, while a3 and a2 could hold together. We need, then, to generate an attack on a2 where we query for the claim of a1, and to generate another attack on a4 when we query for the claim of a3.

This would seem to require that where we query for a proposition and its negation, we want to generate at least one additional argument to counterattack attackers, if we can. In other words, we want to generate attacks (either rebuttals or undercutters) to a2 and a4. For example, we could do so by finding a rebuttal which uses (in a sense to be explained) the claim of the queried argument. In other words, we want to find one rebuttal a5 where the claim of a5 is the negation of the claim of a2 and where the claim of a1, namely p1, is used is found among the support for the claim of a5; similarly, we want to find another rebuttal a6 where the claim of a6 is the negation of the claim of a4 and where the claim of a3, namely ¬p2, is used. Such considerations yield the following arguments:

Fig. 2.

Arguments and relationships.

The result is a graph as in Fig. 2. Note that arguments a1–a4 are clearly related to our earlier representation of the relationships among statements in Fig. 1, but that a5 and a6 are not.

However, to find a5 and a6, we have some additional issues to consider, in particular the argument trees of each of these arguments. We consider the issues by example. The argument a2 has as claim ¬p18, thus the rebuttal a5 must have p18 as claim. In fact, we have an argument by assertion for this claim: ⟨{p18},p18⟩. However, this has a counterargument (derived by contraposition) as well as counter-counterarguments:

While these are remarks about what is required to have an AKB suitable for reasoning with afs, we leave to future research how this is theoretically developed and implemented. In any case, adopting these additional arguments gives more satisfactory results in terms of afs. The important point is that we must carefully consider what results we might want with respect to afs in order to create the appropriate arguments and relations, then build into our argument generator the mechanisms to generate the arguments and relations, otherwise the desired goal from the af will not be achieved.

4.1.afs

For our purposes, we consider the af of [19], where there is one set of undifferentiated objects, arguments, which are nodes in a graph as well as one undifferentiated relationship between the nodes, the attack relation, which can be represented as a graph in which attacks are arcs between nodes representing the arguments (for related work, see [4,9,15], among others).

Some of the relevant auxiliary definitions are as follows, where S is a subset of X. There are a range of semantics, which are definitions of extensions (sets) of arguments; we give only admissible and preferred semantics:

In the next section, we use the ASPARTIX system to generate argument extensions.

4.2.Calculating extensions with ASPARTIX

The ASPARTIX program computes extensions of argumentation frameworks [22];1313 there are related tools that carry out similar computations [11,23,42] It is, then, very useful for our purposes, where we have created arguments in relations and then wish to compute extensions. ASPARTIX is implemented in the answer set programming language Disjunctive Datalog System (DLV).1414 Using ASPARTIX, one can compute any of the standard extensions for the classical AF (following Dung) including admissible and preferred extensions. Extensions for preference-based, value-based, and bipolar argumentation frameworks can also be generated.

To run ASPARTIX, we input to the program a text file for the AKB, containing the arguments and relations, as well as the desired sort of extension. We used our previously defined AKB, which has the arguments and attack relations:

arg(a1).                att(a3,a1).
arg(a2).                att(a2,a1).
arg(a3).                att(a4,a3).
arg(a4).                att(a5,a2).
arg(a5).                att(a2,a5).
arg(a6).                att(a6,a4).
att(a1,a3).             att(a4,a6).

ASPARTIX generates the following preferred extensions, where in indicates that the argument is in the extension:

{in(a1), in(a5), in(a6)}.
{in(a2), in(a3), in(a6)}.
{in(a3), in(a5), in(a6)}.
{in(a1), in(a4), in(a5)}.
{in(a2), in(a4)}.

Such extensions which are generated by ASPARTIX we refer to as the Argument Extensions Knowledge Base (AEKB).

This seems to be an attractive result. As we are only interested in extensions with p1 and ¬p1, we will not look at {in(a2), in(a4)}. Preferred extensions, which are maximal sets of “consistent” arguments (and related propositions), correlate to the notion of normative positions of deontic logic [40], which are maximal sets of consistent deontic expressions. Therefore, we call such extensions policy positions.

5.1.Considerations

The policy positions of PP1 and PP3 seem to have some additional superfluous information. For instance, PP1 contains the propositions p14 and p15, which are propositions in the support of !p1, which itself is the rebuttal of the main point of PP1. Since other portions of the support for !p1 do not hold, !p1 does not hold. Yet, it would seem, in an intuitive argument, odd to have as part of one’s position statements in support of the opposite point, even if these statements are not inconsistent with one’s position. We see, then, that afs overgenerate policy positions. In addition, the output sentences of the policy positions are without discourse markers, so the relations among them are unclear. We leave for future research just how to refine the results to generate all and only the policy positions that correlate with an intuitive notion as well as how to use natural language generation technology to provide well-formed, easy-to-understand output paragraphs of the policy positions.