The convergence of computational thinking, computational intelligence, and multi-agency

2.1Bounded rational agency

Intelligent systems do not have infinite temporal or computational resources to devote to decision making. Bounded rational agency deals with the decision making activities of autonomous entities under such constraints.

The development of intelligent systems has had a troubled history moving through a very pronounced series of booms and busts as the limits of their theoretical underpinnings, computational hardware, and the availability of sufficient quantities of machine-tractable data were encountered placing development at odds with the expectations and aspirations of both practitioners and the public at large. As a result, long periods of so-called AI-winters of reduced funding and development followed periods of overoptimistic hype. Consequently, the techniques for developing such systems have spanned a considerable portion of the development of the discipline of software development itself and as such a need for a unified approach to framing the development of such systems was needed.

The divergent approaches to the development of such systems were not only manifest at an architectural level but were also present in terms of the fundamental approaches used to represent both knowledge and reasoning processes ranging from purely symbolic reasoning, logical inference, to todays dominant data-driven approaches. As such a way to abstract over these differences was needed in order to create an effective container for intelligent processes. The concept of rational agency (Russell & Norvig, 2020) has emerged a useful tool for the containerization of intelligence.

The lack of a formal definition of intelligence has necessitated the adoption of more pragmatic working definitions for the functional and non-functional requirements and other externally observable properties of intelligent systems. These definitions are often very problem specific (for example maximising accuracy, recall, and precision in classification systems or finding an acceptable set of local minima / maxima in a constraint satisfaction problem). The lack of precise definition for intelligence, while troubling on an existential level, is not unique to artificial intelligence research and is reflected in the study of human psychology where the debate regarding the nature of multiple, task-specific, intelligences versus a generalized intelligence (known as g) is ongoing (Visser, Ashton, & Vernon, 2006).

As such research tends to focus on developing systems which exhibit rationality in their decision-making processes rather than true intelligence. Rationality being taken to mean making the best decision (or at least an acceptable one) given the information available to at the time. This allows for task-specific quality measurements to be defined for each problem domain.

As implied these systems are often modelled as decision making processes where the intelligence of the system is captured in a so-called agent function which maps a sequence of observable precepts together with the state of the environment and any knowledge the agent has embedded within it on an a priori basis onto the selection of an action from a finite set of possibilities. Agent oriented systems are considered to be embedded within an environment as opposed to being non-localized inference engines common in the design of older expert-systems. This environmental embedding is a natural extension of the need for modularity and complexity management as discussed later.

The exact mechanism for mapping a percept sequence onto action selection varies in complexity from simple reflexive mappings up to and including complex goal-oriented behaviours incorporating an understanding of the environment within which they are embedded and its evolution over time and in response to the selected actions themselves. Table 1 summarised the main agent architectures while the accompanying Fig. 1 shows more details regarding those structures.

The exact nature of agent environments also varies considerably depending on the problem domain which has profound impact on the design decisions involved in agent architecture selection, percept representation, knowledge representation, and the details pertaining to the agent function implementation. The following describes the most common taxonomy of agent environments which may be represent as environmental N-tuples according to Equation (1). Table 2 shows the composition of such an N-tuple in detail.

Regardless of the nature of environment or the selected agent architecture decision making is subject to constraints both in terms of the time available for computation of the agent function as well as access to required data and communication resources. The end result of decision making under such constraints is the concept of bounded relationality which accounts for the cognitive limits of the agents as well as the consequent opportunity costs of deliberation in a real-time environment (Byrant, 2021). Such opportunity costs arise as a result of actions not being taken due to incomplete / ongoing evaluation of possible action selections.

2.2The principal-agent problem

Real world systems are both complex and multi-faceted and often require the interaction between human-driven and software-oriented processes. As automation becomes increasingly prevalent and the amount of time available for deliberation is reduced human beings will less able to meaningfully participate in the modern economy unless assisted to do so by artificially intelligent systems (Tegmark, 2017). This gives rise to the principle-agent problem wherein the agent must represent the interests of its principal (the human) while maintaining some degree of autonomy in the execution of its duties (Alon, Dütting, & Talgam-Cohen, 2021) as the mismatch in terms of temporal scales operated within by both agent and principal does not allow for real-time human participation.

Unfortunately, the nature of bounded rationality and the inability to completely specify human interests in a formal manner leads inevitably to the emergence of the alignment problem (Hadfield-Menell & Hadfield, 2019). The alignment problem is considered a wicked problem in terms of its multi-faceted nature. Weaker versions of the problems are often addressed for example by focussing on confirming that an agent has been authorized to act on behalf its principle rather than the degree to which those actions will necessarily benefit the principle.

3.1Biological emergence

The definitions of life are numerous and often imprecise ranging from broadly information theoretical approaches centring around replicative patterns through to thermodynamical based definitions focused on the local minimization of entropy. Such definitions are often framed as the importation of negative entropy, through to more conventional biological definitions based (Macklem & Seely, 2010).

Early proto-biological systems took the form self-replicative chains of nucleic acids in the form of either Ribonucleic acid (RNA) or Deoxyribonucleic acid (DNA) in either shallow bodies of water or deep oceanic vents. These replicators were subject to damage due to the varying conditions within these bodies of water. A common theory regarding the transition from prebiotic to cellular life centres around phospholipids suspended within these primordial bodies of water have a marginally hydrophilic head and marginally hydrophobic tail i.e. they exhibit an attractive force to water at one end and a repulsive force at another. As a result, membranes which are formed out of these will find a spherical least energy configuration trapping a small amount of water in their centre. Replicators which through chance and replicative drift happened to code for the production of these compounds therefore enjoyed a survival advantage due to the protection offered by these membranes and become the progenitors of cellular life. Over time these membranes developed to become more selectively permeable with chemically regulated gated channels allowing for control over the passage of chemicals to and from the external environment.

Cooperation between these early cells, mediated by way of chemical signalling, leads to a distinct survival advantage for participating cell lines in terms of risk mitigation and function specialisation. Multi-cellular life is thus an instance of the emergence of cooperative multi-agency.

It is important to note that similarities and differences do exist between natural properties and similar properties in engineered systems. Such distinctions have been considered in the context of natural adaptability in contrast with adaptability in systems (Horváth, Rivero, & Castellano, 2019).

3.2Early software modularization

Software systems were originally written using very low-level flow control primitives in order to create branching and iterative logic. These goto statements allowed the programmer to directly modify the program counter register used by the CPU to determine which byte in memory would be subject to the fetch-decode-execute cycle. While this directly matched the operation of the underlying hardware and provided the programmers with maximal freedom regarding how to sequence the logical steps in their algorithms the lack of defined boundaries between each logical component in the system meant that the logical intent was exposed to a high degree of entropy both during the initial development and subsequent maintenance.

Such spaghetti-code systems were replaced by more structured flow control and procedural constructs after urging from well known computer scientists such as Edsger Dijkstra (Dijkstra, 1968). Eventually these components were grouped into modules such as software libraries with well defined interfaces of programmer exposed functionality. Since each of these components now had distinct boundaries between themselves and their environments, they were able to lower their exposure to high degrees of entropy as were systems constructed therefrom. By considering each of these as units of functionality the overall cyclomatic complexity of systems developed from these building blocks was significantly lowered (Sommerville, 2018). While not yet able to be considered multi-agent systems the initial stages of the separation between the system and their external environment had begun.

3.3Object modelling

The process started by the early modularisation of software systems continued with the emergence of object-orientation in the late 1990s and its widespread adoption in the development of mainstream systems in the 2000s. Object orientation (i.e. the encapsulation of related data and the operations legal upon those data into a single entity together with other related techniques) was built upon further enhancing the boundary between system components and the internal environment of the system itself in addition to the external environment the system as a whole operates in.

While there are several principals underpinning the object-oriented design of systems the fundamental one is the concept of encapsulation which unifies the data relating to a single problem domain entity along with the operations legal on those data into a single unit. Additional relationships are possible between objects and between classes (the design-time analogue of an object prior to its instantiation). These well-defined relationships are standardised by the Object Modelling Group in the Unified Modelling Language (The Object Modelling Group, 2017), key examples of which are shown in Table 3, and further enforce the boundary between objects and the internal system environment as object-oriented systems accomplish their goals by way of message passing between objects in the system. Direct access to the internal state of an object is not possible and such values may only be accessed or modified by way of such message passing.

Table 3

A selection of UML relationship types (The Object Modelling Group, 2017)

Relationship	Symbol	Description
Association	∸	A set of messages passed with a common purpose may be summarized as an association between the classes
has-a (composition)		A containment relationship representing a whole-part relationship with the implication that the container and contained do not have a meaningful independant existence.
has-a (aggregation)		A containment relationship where container and contained may have meaningful existence independantly of each other.
is-a Realisation		One object implements the interface defined by another, more abstract class.
is-a Inheritance	⇾	A more general base class is specialized into a more concrete derived class.

While object orientation represented a firm step in the direction of realising multi-agent system design the key contribution the methodology is the encapsulation of entity attributes and related operations. In order for true agent-orientation to be possible autonomy regarding the selection of those operations (i.e., the implementation of an agent-function) needs to be included.

It is worth noting that there is evidence that imperative approaches to programming such as exist in conventional object oriented programming may not be ideal for comprehension and skill acquisition and that those oriented around pure immutable mathematical functions may be preferable (Mirolo & Izu, 2019). Approaches and dedicated development languages exist which attempt to unify both paradigms (Richard-Foy & Doeraene, 2021).

3.4Microkernels

The trend of defining boundaries between systems and the environment within which they are embedded coupled with imbuing them with a certain degree of autonomy has manifested across many categories of software system. This section and the subsequent one will continue two such important cases where this has emerged in the form of operating system design and web stack backend frameworks.

Operating Systems serve to fulfil a variety of complex needs, but their primary purpose is to serve as unified abstraction layer separating the application developers from the complexities and differences found in the underlying hardware as well as to provide fair access to those application programs to the limited resources available.

As such a distinction is drawn between code running in the core kernel of the operating system, which is able to execute a selection of privileged instructions unavailable to conventional application software, and user-mode code. This distinction, while necessary to allow the operating system to play its management role, does come at a cost in terms of context switching as values are transferred between user and kernel space and the traps necessary to switch into a mode necessary for the execution of privileged instructions. In a sense this can be seen as the boundary between the operating system kernel and the environment within which it is embedded. Operating systems differ in terms of the modularity of their internal architectures and the degree to which those modules may operate autonomously of each other. These approaches range from the monolithic architecture followed by the Linux kernel, microservice-oriented approaches followed by kernels such as the Mach kernel used within OSX and the hybrid approaches used by the Windows NT family of operating systems (Tanenbaum, 2016).

Monolithic kernels may be internally modular as is the case with the Linux kernel but still be considered monolithic due to the lack of internal autonomy of those components as well as their existing in a shared address space (Novikov & Zakharov, 2018). Truly microkernel-based approaches have independent modules for each functional area within the system while hybrid systems exist a compromise between the two. It is worth noting that the microkernels are themselves often connected via message passing systems in a manner analogous to the object-oriented approach discussed previously, the microservice approach discussed subsequently, and the actor and agent-frameworks discussed thereafter.

Although there are numerous advantages to the microkernel-based approach in terms of increased fault tolerance and scalability operating systems are very performance critical pieces of software and as such the advantages of a monolithic approach in terms of minimizing transitions to and from kernel space outweigh the advantages of loose coupling.

3.5Microservices

The development of web-based systems has echoed similar trends towards adopting aspects of agent orientation as shown previously. Initially web-based systems were largely monolithic and although being externally facing systems that are therefore de facto embedded within an environment their high degree of internal coupling coupled with their lack of autonomy left them far from the agent-oriented ideal. Recent trends in the development of such systems have favoured the creation of systems as bundles of independently invokable web-services echoing the development of library-based modularity in conventional application development (The Open Group, 2014).

The latter part of the last decade has seen an increase in the implementation of service-oriented software systems by way of microservices architectures (Bravetti, Giallorenzo, Mauro, Talevi, & Zavattaro, 2020). In such approaches each service in the bundle is implemented by way of a microservice. Microservices themselves are implemented as cohesive independent processes which communicate with each other by way of message passing and have their own distinct data storage. There are numerous advantages to implementing service-oriented architectures in this manner.

Each microservice is its own unique process or sufficiently isolated thread of control which means they may be implemented in different programming languages and distributed across multiple compute nodes in cloud environments. Such isolation has security advantages in that their address spaces are disjoint as well as advantages in terms of parallel development in addition to parallel deployment. Scalability is also more naturally archived as more instances may be spun up and down as needed. Microservice based systems are often deployed in cloud contexts using containerisation technologies adding a further layer of modular abstraction. With a sufficient degree of abstraction, and the sequence of messages taking the place of a percept sequence microservices may be considered as multi-agent systems.

Figure 2 shows an example of a microservice oriented architecture with distinct data storage on a per problem domain entity basis (Song, Chauvel, & Nguyen, 2020).

Fig. 2

A microservice architecture (Song, Chauvel, & Nguyen, 2020).

3.6Agent development platforms

Due to their nature as critical information infrastructure (Presente, Rolim, & Moreto, 2015), telecommunication systems have many of the same needs in terms of robustness, distribution, fault tolerance as are offered by multi-agent architectures (Dorri, Kanhere, & Jurdak, 2018). This is evidenced by the contributions to multi-agent technologies made by service provides in the telecommunication space as is discussed in the section and the following one.

Explicitly agent-oriented standards and software development frameworks have been developed in the past two decades and successfully deployed. The most important standard for the development of software-based agents is ironically the communication standard defined by the Institute for Electrical and Electronic Engineering Computer Society’s (IEEE-CS) Foundation for Intelligent Physical Agents (FIPA). While the degree to which this standard is used in the development of physical agents may not align with the name of the foundation however the standards they defined have formed the backbone for the development of software-based agents where the need for interoperation is a concern as they have defined an implementation-neutral standard in the form of the Agent Communication Language (ACL) specification (Foundation for Intelligent Physical Agents, 2002).

The ACL is based around Searle’s Speech Act Theory (Searle, Kiefer, & Bierwisch, 1980) in which each message is strongly typed with a performative meaning allowing for both information to be conveyed as well as instructions to be issued within the same construct. A layer-based decomposition of the agent communication problem is made in a manner analogously to the decomposition used by the International Standards Organisation (ISO) in their model of internet communication (Coulter, 2014) such a decomposition is shown in Table 4.

Table 4

FIPA ACL Modular Layered Decomposition (Coulter, 2014)

The Java Agent Development Environment (JADE) is the most mature and well-established implementation of the FIPA standard and was developed by Telecom Italia under an open-source licence (Telecom Italia, 2021). The portability of the system due to its implementation on the Java Virtual Machine (JVM) has made it easy to deploy across heterogenous hardware configurations common in real world infrastructure environments. Development of agent-oriented systems using the framework is accomplished by specializing the abstract base classes provided by the framework and specifying the logic of each of a set of behaviours of which an agent is considered to always be running one. Agent behaviour logic consists of the sending and receiving of ACL messages and occasional switching between running behaviours selected from a pool of possible behaviours.

Figure 3 shows the static structure of such a specialisation for the purpose of creating a collaborative multi-agent system while Fig. 4 illustrates the dynamic behaviour of such a running system.

Fig. 3

Static structure of class specialisation in JADE (Coulter, 2014).

Fig. 4

Dynamic behaviour selection in JADE (Coulter, 2014).

3.7The actor model

The Actor Model is a system for the modelling of concurrent systems which has converged on a set of features making it well suited to the creation of multi-agent systems. The Actor Model was defined in 1973 by Hewitt et al to express a modular, concurrent, message-oriented model of distributed compotation (Hewitt, Bishop, & Steiger, 1973). The Actor Model has been applied successfully in the telecommunication industry for similar reasons that lead to the applicability of standard multi-agency discussed previously. As an example of this consider that the Actor Model is deeply integrated into the Erlang programming language developed by Ericsson (Chechina, et al., 2017)

Actors, together agents within multi-agent systems, can be viewed as a realization of the object-oriented paradigm in that each actor encapsulates a single processing unit together with a set of data and communicate by way of message passing. Interestingly enough, the Actor Model has been successfully implemented using non imperative languages, as was the case with Erlang, hybrid object-oriented / functional languages such as Scala along with more conventional object-oriented languages such as Java. The canonical implementation of the Actor Model on the Java Virtual machine is the Akka library (Typesafe Inc, 2021) which is officially supported by both the Scala and Java languages (in fact it supplanted Scala’s original actor implementation of the model) although the framework should be usable by any JVM language that supports Java interoperability.

Actors, like objects and agents, communicate through a set of messages and do not have direct access to each other’s internal states. Within the context of the Akka framework actors are grouped within actor systems and form nested hierarchies of actors Each actor is represented by an actor reference which encapsulates the actor’s state and behaviour, its mailbox, references to each of its immediate child actors.

When an actor receives a message, it may react by either sending further messages to other actors to which it has access and/or by changing its internal behaviour. This behaviour switching approach is analogous to how agent behaviours are modified in the JADE framework supporting multi-agency as a natural endpoint for the development of distributed complex systems. An actor may also create a set of child actors over which it assumes an oversight role with errors being propagated up to parent actors in order for them to handle by way of their supervision strategy. These actors and their children form a hierarchy of nested actors akin to a process hierarchy or file system in operating systems. Actors may additionally be located by way of their ActorPath which represents their location within the actor system hierarchy. Figure 5 shows the actor hierarchy within an actor system.

Fig. 5

Akka actor system hierarchies (Typesafe Inc, 2021).

Fig. 6

A holonic multi-agent implementation of Map Reduce (Cullinan & Coulter, 2021).

3.8Holonic multi-agency and holarchies

As the previous section shows actor-oriented systems where individual actors are invested with a degree of autonomy are equivalent to software based multi-agent systems whose percept sequences are implemented as a series of messages.

Many problem domains have self-similar properties reflecting the fact that examples of these problems may be recursively defined as smaller instances of the same problem in their own right. The philosophical concept of a Holon represents an entity which is simultaneously a part of something and a whole in and of itself. A nested hierarchy of holons is referred to as a holarchy. Ongoing work involving the implementation of holonic multi-agent systems which build upon these ideas using the Akka framework’s actor systems to realize holarchies of multi-agent systems has been explored (Cullinan & Coulter, 2021). Although complete details of the work can only be made fully known following the publication of the work the basic concept is that the self-similar nature of the widely used Map Reduce approach to structuring Big Data solutions is exploited to create a holonic multi-agent system on the Akka platform. Figure 7 illustrates this concept in abstract.

Fig. 7

Multi-agent artificial immune algorithms (Coulter & Ehlers, 2012).

4.1Core competencies

The cognitive competencies required for effective software development have been discussed under many names since the inception of the discipline itself. The modern cycle of the required competencies debate was rekindled by an article in the Communications of the Association for Computing Machinery (Wing, 2006), furthered across a number of important and influential venues such as the Royal Society (Wing, 2008), and continues to this day (Bradshaw & Milne, 2021).

Computational competencies in this discussion have focussed on algorithmic thinking, abstraction, and automation. These competencies are often kept at a fairly high level as many mechanisms exist for realizing abstraction and over specifying these details is dangerous in a field as rapidly evolving as information technology. The position adopted here is that previously important aspects of developing complex systems should have been explicitly included as an aspect of computational thinking and that its omission has proven harmful. Likewise thinking in terms of multi-agent systems has become increasingly important and as discussed previously is a natural progression across multiple disparate branches of software development. The nested natures of holonic multi-agent systems necessitates the development of referential thinking capabilities as discussed in the following subsection.

For the purposes of this discussion the core competencies considered are referential thinking, parallel problem solving, and multi-agent problem decomposition.

4.2Referential thinking, parallel problem solving, and multi-agency

A particular competency required in the development of complex information systems is the ability to reason about systems with high degrees of indirection i.e., those whose architecture makes use of multiple layers of references. Such thinking is particularly important in lower-level programming languages which directly expose the underlying memory model of the underlying hardware in the form of explicit pointer variables to store and modify memory addresses. This holds true in principle even if the actual physical addresses are abstracted behind virtual memory addresses. Errors in reasoning about such matters has resulted in a significant number of logical errors and security vulnerabilities across the years.

For this reason many attempts to solve the issue have focussed on removing direct access to pointers in favour of system managed references and while this is commendable in that it does reduce the total number of errors produced it does not negate the need for developing competency in this manner of thinking. Many problems require referential thinking (the ability to follow and correctly reason about layers of indirection) that do not involve pointers such as reasoning about deeply abstracted generic type systems (such as those involving templated code) and highly relational databases. Previous attempts to offload aspects of computational thinking onto the programming languages themselves in the form of garbage collected memory management has had similar issues in that they only considered the management of one type of resource which follows the acquire-use-release pattern namely dynamically allocated memory. There are many resources which follow such usage patterns such as file handles, both database and network connections, mutexes and semaphores all of which need well-developed computational thinking skills to be managed correctly. For this reason resource exhaustion remains an issue in long running systems even in the face of garbage collected memory.

Parallel problem solving is qualitatively different from conventional sequential algorithmic thinking. A large number of error classes such as deadlocks and livelocks are only possible in environments which consist of concurrently executing components. Due to the physical limits of expected future improvements regarding computational power as well as the growing volumes of data being processed means that most systems going forward will tend to be distributed in some form or another. The methods for reasoning about concurrently executing processes and the management of shared resources between those processes is well studied but often presented in a manner focussed on the development of multi-threaded applications in a shared memory environment. As argued throughout this paper multi-agency is a commonly reoccurring pattern in the development of distributed systems indeed if not the natural end point for such development. Therefore, we should consider including multi-agent reasoning as one of the core competencies of computational thinking.

5.1Immunological computation

Complex natural and biological systems exhibit properties of recursive self-similarity and multi-agency and therefore require the computational thinking core competencies as described in order to be meaningfully reasoned about.

Due to the global COVID-19 pandemic appreciation has grown regarding the ability of the vertebrate adaptive immune system to respond to previously unknown pathogens and, in general, for the quality of its response to improve over time. This learning process has served as the inspiration for numerous learning algorithms over time however, in practice, most do not capture the truly multi-agent nature of the underlying process. The immune system comprises of several layers which have evolved over millions of years. The system is incredibly complex and, as is common in biological systems, both highly coupled and highly redundant with functionality duplicated as well as the same system elements playing very different context specific roles depending in circumstances.

It is therefore natural that any tractable algorithms derived from the immune system must be highly abstracted and simplified, seeking to capture the essence of the learning process without the unnecessary complexity.

Unfortunately, the vast majority of implementations ignore the inherently multi-agent nature of the immune system. The immune system can be viewed as consisting of actors in the form of lymphocytes (most algorithms incorrectly focus on the antibody as the fundamental representation of the algorithms when they are more correctly viewed as effectors), whose percept sequences are made up of the antigens presented to them coupled with the degree of affinity exhibited to those antigens by receptors on their cell membranes (the boundary between the agent and its environment is easily defined by way of the cell membranes of the lymphocytes), and as stated previously the effectors are the various immune responses such as the antibodies produced.

In addition to not using a multi-agent approach most algorithms derived from the immune system are not implemented in a parallel manner (i.e. do make use of concurrent threads of execution) which loses the major reason for the efficacy of the biological systems upon which they are ostensibly based. The chemical and replicative processes which govern the adaptive response are in fact terribly slow it is only the fact that they are carried out in a massively parallel manner which allows the system to function reasonably at all.

The exact implementation of the agent function depends on the aspect of the immune system under consideration. Figure 7 shows both B-Cell and T-Cell mediated immunity implemented in a parallel manner while Fig. 8 shows the agent function architecture of a single T-Cell based agent.

Fig. 8

A T-Cell based agent function (Coulter & Ehlers, 2012).

5.2Swarm intelligence

A similar issue occurs in particle swarm optimisation in which each potential solution to a problem is mapped onto a position in a high-dimensional problem space. Effectively each candidate solution can be viewed as an agent encapsulating a utility function and a set of vectors representing its position and velocity within that space. Agents then communicate the values of that utility function, using their current location as input, to a limited subset of the population of agents. Their velocity and hence position is updated by calculating a resultant vector of their local neighbourhoods’ best positions as well as the best position encountered by the agent itself. Because the neighbourhood definitions for each agent overlap incompletely information is transmitted between different subsections of the population reducing the likelihood of prematurely converging on a local, rather than global, optimum.

While the narrative descriptions of such algorithms use the language of multi-agency the actual algorithms themselves do not exhibit multi-agency or indeed even concurrency in most cases. The following listing presents a truly multi-agent implementation of a particle swarm system (Thomas & Coulter, 2021).

5.3Reinforcement learning

As a counterpoint let us consider reinforcement learning which is an explicitly agent-oriented learning paradigm in which an agent embedded within an environment selects actions from a set of possible actions given their environment state. They then receive feedback in the form of an immediate reward percept (which may be either positive or negative) and must then develop a policy mapping future percept sequences onto action selections which are expected to maximize its total cumulative reward over time. Once such family of algorithms is known as Q-Learning (Clifton & Laber, 2020).

Unlike other branches of computational intelligence reinforcement learning adheres to the agent-oriented paradigm in both the terminology used as well as the fundamental approach to realising the paradigm. Multi-agent reinforcement learning incorporate game theoretic underpinnings in addition to the Markov chains of the single agent version of the approach.

5.4Connectionist learning

The one subdomain of artificial intelligence which has experienced the greatest surge in both interest, funding, and expectations are those based around connectionist learning approaches such as neural networks. The improvement in utilisation and applicability of these techniques is a result of simultaneous cost-effective access to computational processing power (both locally in the form of cheaper microprocessors and remotely in the form of cloud computing) as well as access to large amounts of data with which to train such models.

The most natural intersection between artificial neural networks is in the implementation of the agent function. This would typically take the form of percepts being mapped onto the first layer of such a network and each node in the output layer representing a different possible action. The most strongly activated node after a feed-forward pass through a trained network would represent the action to be selected. Training of the network would be conducted using either conventional backpropagation based supervised approaches or a hybrid computational training approach using evolutionary or particle swarm algorithms (assuming a dataset of correct action selection training examples were available).

Such approaches are of course limited in that they assume that people’s interests lie in the creation of multi-agent systems in and of themselves rather than simply their being the most appropriate tool for implementing distributed intelligent systems. For this reason, research has been undertaken in implementing neural network architectures themselves in a multi-agent manner akin to how intelligence is an emergent property arising from the interaction of inherently non-intelligent components (Cullinan & Coulter, 2018). Figure 9 demonstrates such a multi-agent-oriented implementation of a convolutional neural network.

Fig. 9

An actor/agent-oriented implementation of a convolutional neural network (Cullinan & Coulter, 2018).