Multi-agent based manufacturing: current trends and challenges

—The advent of Industry 4.0 and the development of future industrial applications can be achieved using Cyber-Physical Systems (CPS). This technological development invokes high levels of communication and computation in the form of an interconnected network of industrial resources. Multi-agent systems precisely can empower such technological evolution by introducing properties like decentralisation, autonomy, ﬂexibility, social ability and modularity to the industrial context. In this regard, the current work surveys recent multi-agent based manufacturing approaches and provides a general vision of current trends focusing on frameworks/architectures, complementary technologies and common applications. This article, ends with an integrated discussion of emerging agent-based industrial challenges, a general conclusion and ﬁnal remarks.


I. INTRODUCTION
With the development of various technologies in the field of computer engineering, the manufacturing sector has witnessed a significant shift in the past decade.This shift can be attributed to the application of high-performance sensing, computing & networking devices.A common term used to represent this new paradigm is called "smart manufacturing".This paradigm has supported the manufacturers to withstand market turbulence like competition from developing markets and need for mass customization.
One of the emerging technology that support this new manufacturing paradigm is agent-based computation.Agents are cognitive entities which exhibits properties like autonomy, This work has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant No. 814078.reactivity, pro-activeness and social ability.A commonly accepted definition for an agent is that it's "an encapsulated computational system that is situated in some environment and that is capable of flexible, autonomous action in that environment in order to meet its design objectives [1]".A single agent has its computing and knowledge limitations in complex and large problems like manufacturing systems.Multi-Agent System (MAS) is one way to solve these problems where each agent uses its knowledge to solve it's particular problem but also co-ordinates with other agents to solve interdependent problems.
In the last half decade, there were very limited reviews which gives a general overview of the current frameworks, technologies, applications and challenges of multi-agent based manufacturing.Falco and Robiolo [2] presented a detailed systematic review of the patterns and trends in MAS with focus on all application domains including transport, healthcare and manufacturing.Calegari et al [3] presented a systematic review of MAS but mainly focused on logical technologies.Some researchers have focused their review on a specific area or sectors like agent-based programming [4], micro-grid systems [5] [6], shared transport services [7], energy sector [8], image segmentation [9], smart homes [10] etc.The aim of this work is to give researchers in manufacturing domain a brief overview of the current trends and challenges in MAS.We also wish this work would act as a guide for manufacturers in implementing MAS for their specific requirements.

II. METHODOLOGY
To understand the current trends and challenges in multiagent system a structured literature review was carried out.The first and the most important step was to identify relevant research questions (RQs) which would satisfy the requirements of the review.The structured literature review methodology is given in Figure 1.Our review will answer these four research questions and analyze & discuss their results.Most of the work on framework involves developing the concepts around application environments.As per the literature review it is found that mostly there doesn't exist a standard framework/architecture to target general manufacturing environment but is usually application specific.Applications of multi-agent frameworks vary from supply-chain negotiations [11], [12] all the way to manufacturing operation scheduling [13], [14], [15].
Influx of different technologies have laid foundations of multiple frameworks and architectures.Service oriented architecture (SOA) coupled with semantic web ontologies presented an effective management approach for IoT devices in manufacturing processing [16].SOA in an Holonic Manufacturing Cell (HMC) generalised decision making strategies for distributed manufacturing systems [17] along with possibility of reconfiguration of manufacturing cells [18].Often these technologies are aided by others to achieve application purpose.For instance multi-agent frameworks based on SOA and HMC technologies are linked with webservice agent, XML schema and data formats (STEP) to produce a potential case for virtual production system [19].
Multi-agent frameworks and architectures have been integrated with cloud and edge technologies.These technologies in manufacturing are being more widely accepted, attributed to their computational competence and evolving decreasing latency requirements.A common data structure among all platforms further leverages the idea of adoption in production.Using of prior knowledge and data gathered during production could lead to zero-defect manufacturing.[20] studied this concept while in cooperation of cloud and edge technology.High computational capability can be utilised in cloud based platforms for employing reinforcement learning approach, that could aid in making highly effective distributed intelligence based decisions in manufacturing environment [21].Further applications to this approach involve process planning [22] and dynamic task scheduling [15].
Biological insights have also been incorporated in some frameworks and architectures.These mainly involve studying their patterns for smart decision making in manufacturing.Ant based optimisation techniques have been used for control based system decision making [23], [24].The action, reward and decision model is very useful in establishing effective decision criteria in distributed manufacturing environments [25].Additional, remarks of main frameworks and architectural characteristics are detailed in Table 1.

B. Technologies that support the development of MAS [RQ2]
While the theory and concepts behind the application of MAS in manufacturing are paramount for its understanding, it is through the tooling that its potential can be fully realised to solve real world problems.The various recent applications and frameworks present a large ecosystem of technological awareness.They are riding and increasing the potential of smart manufacturing bringing intelligence, interoperability and integration of systems.
The answer to RQ2 is derived from two figures.The first one (Figure 2) clusters eight topics that support MAS i.e. interoperability, internet of things, learning based methods, enabling technologies, simulation, production management, modelling and control and optimisation.The second one (Figure 3) presents the frequency of the appearance of such technological enablers.
Results of this question do not aim to provide a very strict statistical analysis but to offer a general vision of what scope of technologies and with what strength are supporting MAS in manufacturing.
At least 53% of works (25) are supported by interoperability-related technologies i.e. in infrastructure, standards and semantic topics and represent the greatest technological interest.MAS are utilized to provide resource communication in the supply chain through the cloud, Three layer architecture : User layer, System Layer and Service Layer.User layer is the interaction layer with operator, customers, supervisors, and suppliers.it's connected to cloud server in the service layer through web server.The system layer encapsulates the MAS which is connected to IoT System (IoT Devices).Decision making is a functionality of MAS at the system layer that provides a gateway to edge computing and then to manufacturing cloud.MAS architecture consists of Production Agent (PA), Maintenance Agent (MA), Quality Agent (QA) and Logistics Agent (LA).PA consists of Process Agent and Scheduling Agent, MA consists of Monitor Agent and Repair Agent, Quality Agent consists of Analysis and Assurance Agent along with LA which contains Transfer Agent and Resource Agent.

R. Lu et al [27] : Multi-Agent Deep Deterministic Policy Gradient (MADDPG) Algorithm
Critic and Actor network are operated with weights for each agent.The functionality involves target network parameter optimised with each episode.
Each agent selects action as per policy & exploration noise.Each agent action leads to reward and the next state is observed.The learning phase incorporates sample batches of optimisation variables (minimising loss) , updates critic and gives target network parameters..The actor updates as per sampled policy gradient.S. Baer et al. [28] : Petrinet based FMS Multi-Agent RL (MARL) framework An online scheduling approach for flexible manufacturing system was developed using reinforcement learning.Virtual representation was achieved by Petri Net modelling for plant topology and product flow.A four stage training approach aided by agents proposed The deep reinforcement learning agents guide the product through the setup to achieve nearoptimal processing times and optimal re4source allocation.Agents forsee machine failures, plant topology re-configuration and optimisation goals.Rafaella de Souza [29] : MAS system for Supply Chain The system developed to study the optimisation of supply chain efficiency.The assumption of proposed system is the individual and global objective function.The negotiation mechanism in system is catered around dutch auctions and monotonic concession protocol.
The environment involves agents as intelligent actors.Agent learning in system is broached by ε -heuristic.D' Aniello et al. [30] : CM based architecture for operation management Architecture developed for production planning of decentralised resources.Distributed task scheduling addressed.
Three type of agents: Task Agent (TA), Master Agent (MA) and Printer Agent (PA).These agents collaborate to manage and monitor homogeneous manufacturing services.R. Wang et al. [31] : Multi Agent Manufacturing Process Optimization Method MATLAB enforced MAS method for production scheduling in interest of energy conservation was developed.
MAS method involves agents that utilised Quantum Particle Swarm Optimization (QPSO) algorithm for analysing and making decision on real-time energy data for optimisation.Yang et al. [12] : Negotiation model for supply chain The framework is a negotiation method for solving conflict and realisation of cooperation in supply chain environment.
Agents use request bid system via an Agent Name Server (ANS) for negotiation.Constraint conditions can be incorporated Li et al. [32] : Multiobjective Particle Swarm Optimization Algorithm MAS algorithm Catered to process industry for complex working condition control.Model is divide into control and execution layer.
MAS technology works with MOPSO algorithm to optimise system model and reduces time consumption due to information interaction.
Cagnin et al. [16] : Architecture based on MAS, SOA and Semantic Web Technology Process automation for applications to coordinate and execute tasks by autonomous devices.Multiagent Architecture consists of reactive and cognitive layer.agent implementation at physical layer at IoT devices followed by agent to agent connection for multi agent system and enforced by Knowledge base.Ontologies assist in dynamically and automatically select and execute tasks.Thomas et al. [17] : HMC SOA agent framework Discusses emerging holonic and multi-agent system technology in service oriented and cloud based approach.Presented a case for utilisation of ICT technologies such as agents, virtualization, big data and data analytics.
Presented a agent-based optimisation technique, that used fuzzy-multiagent systems to enforce decision maker's strategies to value chain environment.HMC and SOA applications using agents proposed.Abid et al. [18] : SysML (Systems Modelling Language) framework agent-based RMS System integrating using holonic paradigm.Increases productivity through the simulation of reconfigurability.
Implementing the holonic architecture in the agent-based platform.Guizzi et al [13] : decentralized multiagent optimisation approach Dynamic integration is discussed for process planning and scheduling operations in open job-shop manufacturing systems.Decision making process is distributed at level of agents that overcome local disturbance to reach overall target.
FIPA CNP protocol and MAS combination proposes a method for solving OJSSP under uncertainty.MAS architecture divides scheduling problem of a big job into constituent smaller problems.Composite-dispatching rule is developed for scheduling, decreasing in job mean waiting time.Li et al [36] : Distributed MultiAgent Cooperation Collaborative Control Method and HD MAS (Hierarchal Distributed) architecture Provides control mechanism for process industry and process control optimisation.This leads to a collaborative control model among production units.A DS MADDPG (Distributed Multi Agent Deep Deterministic Policy Gradient) framework is proposed and utilises MADDPG algorithm to achieve intelligent distributed collaborative control.
Experiments done with two agent environment gradually learning policy and learn to collab to manipulate task agent.MADDPG vs DS MADDPG (better results shown).
Leitao et al. [20] : Multi-Agent System Architecture for Zero Defect Multi-agent CPS for targeting ZDM in multi-stage production system.
MAS architecture for distributed data collection and analysis.Monitoring and adaptation among cloud/edge layers.It promotes process and product variability with generation of optimised knowledge based on aggregated data.Yang et al. [11] : Negotiation of Manufacturing enterprise supply chain Multi-objective negotiation model, negotiating tactics and steps between purchasing agent and supplier agent Negotiation model and tactics: agent coordinator communicates through negotiation thread with suppliers.Taurino et al. [37] : Multi-agent Systems for Production Management A model of the mid-layer multi-agent management negotiation, according to the "game theory" viewpoint.
The requirement from any agent of the network is to understand the payoff of the part of production system.To evaluate this payoff, it is necessary to adopt a network model based on "cooperative game theory" that shows the different way of "players" (agents) to interact and cooperate.Giret et al. [38] : Sustainable intelligent manufacturing control systems Engineering method that helps researchers to design sustainable intelligent manufacturing systems Identification of the manufacturing components and the design and integration of sustainabilityoriented mechanisms in the system specification, providing specific development guide lines and tools with built-in support.Mantravadi et al. [39] : Multi-agent Manufacturing Execution System (MES) A combination of different elements such as hardware, software, organisational practices and other tool boxes like ML could include MES software as a main actor.Such a collaborating system can derive benefit for the enterprise.
Multi agent MES consist on sub agents that run on raspberry pi and central agent (a middleware running on MES server).Sub agent collects data to detect abnormal behaviour and aid MES to execute.Yongkui et al. [15] : Multi-agent-based scheduling in cloud manufacturing Scheduling issues in cloud manufacturing using multi-agent technologies.An architecture for scheduling in cloud manufacturing is proposed.
A model is presented that incorporates many-to-many negotiations based on an extended contract net protocol.It takes into account dynamic tasks arrivals.Mezgebe et al. [40] : Algorithm for multiagent-based manufacturing system Consensus algorithm for multi agent based manufacturing system (CoMM) -to control rush order and minimize a makespan.consensus.
Each agent decides when to broadcast its state -controlling decision depends on this state behaviour.Liu et al. [14] : Multi-agent architecture for scheduling complete manuf.system -distributed physical manuf.and virtual manufacturing mapped from physical ones resulting in platform-based smart manufacturing systems (PSMSs) agents used for platfrrm and enterprise level scheduling.
discovering and orchestrating services on demand [41] and running agent technology in the network or edge if needed [20].The integration, of agents is also supported by standards.Some examples are XML language that formalizes knowledge base methods for queries and data storage [42] or RFID tags that provide opportunities for precise data acquisition [23].Finally, semantic communication and in general ontologies and semantic web provide methods for data understanding, knowledge and automatic discovering of information [41].
In terms of modelling and control, at least 38% of works (18), provide a specification of system modelling, either with the formalization of methods or with various sources of inspiration.Some examples are symbolic artificial intelligence with formal methods [1], behavior modeling [35], Markov process [28], contract net protocols [15], graphs, etc.Most of agent-based approaches have inspiration in Holonic systems to model and design digital agents with a high level of compositionability and granularity [43].With this method, the level of control is normally possessed by a superior digital entity (hierarchy) and entities below representing intermediate decision making.In the lower level, holons abstract physical objects and information of sensors and actuators.Also, some of these works, include bio-inspiration (e.g.ants' pheromones) as a mechanism to provide indirect agent influence for process adaptation [23].
To a lesser extent, learning based approaches (approx.25% or 12 works) present learning mechanisms that guarantee the utilisation of experience-based knowledge.Generally, supervised and unsupervised techniques support agent-based decision-making.This integration provides also embedded functionalities: anomaly detection, data storage, continuous adaptation of parameters and information [39].Additionally, Fig. 2: Scope of relevant technologies that support MAS Fig. 3: Technology frequency appearance reinforcement learning can efficiently guide agent policies to achieve near-optimal production specification e.g.scheduling or resource allocation of resources [28].
Similarly, optimisation methodologies (approx.17% or 8 works) provide necessary support for process and parameter optimisation.Normally, these techniques are based on evolutionary computation [44] e.g.particle swarm optimisation (PSO) [25] to adjust agent behaviour considering influencing factors, modelling of the system and manufacturing restrictions.We consider at the number of works that have simulation as means to test or enhance their approaches (15% 7 works).Simulations are used for testing in a risk-free environment and for enhancing real-time decision-making.In the latter case, simulations work as a future prediction mechanism.A popular tool for agent implementation is the JADE framework.In addition, NetLogo provides a graphical and interactive language programming for agent implementation and testing.
The number of works that utilised production management tools and technologies is reduced (8%, 4 works).Here we refer to agents with high level of abstraction, mostly related to supply chain or operations out of shop floor.In this context, agents provide support for intelligent communication and autonomous decision making in applications like Product Life-cycle Management (PLM), Manufacturing Execution System Operations [39], Virtual Enterprises [19], etc.
Finally, multi-agent technologies work in symmetry with other technological enablers.Undoubtedly, this integration provides necessary tools to potentiate manufacturing expectations and serve as example for future practitioners e.g.industrial robots, augmented reality, virtual reality, internet of things, etc.The high relevance of industrial robotics should be emphasised and their integration with agent technologies to provide an intelligent manufacturing control.Agents act as the digitisation mechanism for such resources.

C. Applications of MAS in manufacturing [RQ3]
In our analyses, we found that majority of papers (64%) address 5 major applications namely: production planning (16%), energy and emission reduction (16%), manufacturing scheduling (14%), logistics & supply chain management (10%) and decision support system (8%).Considering the ISA 95 levels, the results shows that most of the publications addresses the higher levels i.e., Manufacturing Operations Management (L3) and Business Planning and Logistics (L4).Others in Figure 4 represent applications which are addressed in less than or equal to 5% of publications.These include applications like maintenance, quality control, manufacturing control, performance and efficiency improvement, Anomaly detection, self-organisation etc.Here we are presenting some of the most relevant works in each application.Sarkar et al. [41] proposed an agent framework for developing a generic planning agent based on Belief, Desire and Intention mechanism (BDI).The proposed model can be used for planning at any level of aggregation dimension.At each level, the lower level resource agent presents specifications and upper level agent proposes matching capabilities for lower level agent to accept or reject it.Wang et al. [45] compared communication between each agent using Particle Swarm Optimization (PSO) and Quantum-behaved PSO algorithm for achieving energy conservation and emission reduction.The authors used real time energy and production data from the production of 1000kg of glass fiber in a kiln.They concluded that use of Quantum-behaved PSO reduces oxygen, natural gas and machine power consumption.Liu et al [46] presented a multi-agent architecture for scheduling with a centralized management.The architecture manages scheduling process for task executed in both platform and enterprise-level.The process is based on bidding and negotiations between different types of agents.Farsi et al [47] proposed a 3layer Multi-Agent Cyber Physical Manufacturing System for interaction between shop floor phases and external stakeholders within a supply chain.The 3-layers represent multi-layer agents from micro to macro level for global manufacturing supply chain.Takahashi et al. [48] constructed a sharing and non-sharing durable goods market model.The duopoly model has two groups of agents; Manufacturer agents and Consumer agents.The authors used multi-agent simulation using Qlearning for agents' decision making.Table II presents various technologies which support each MAS application.In analysing the table we notice that certain technologies are used in almost all types of applications.These includes reinforcement learning, Edge & Cloud technologies and simulation techniques.These technologies could be identified as promising technologies for the future development of MAS.Interoperability and modelling & control are the technological enablers which was used for the realisation of all applications indicating the importance of focus in its development for a better implementation of MAS.

D. Challenges in MAS in smart Manufacturing [RQ4]
On the level of framework a significant challenge that exists for adoption of agent technology in manufacturing setting is related to lack of framework and architecture that targets general manufacturing environment.MAS on the other hand are seen to fail in dealing with real-time properties, as they typically go for the best-effort approach.This approach although feasible does not account for worst-case scenario or prepare the system well in advance before such a case occurs.An issue in this regard is to ensure real-time compliance that could be achieved by interoperablility.The next missing link in the MAS framework is the missing rules and mechanisms that may be triggered due to any timing errors causing delay in synchronisation and scheduling of tasks.Real-time constraints can cause MAS frameworks to be incompatible across different platforms, therefore requiring certain functionalities within standards and open protocols are needed to support this combination transition.In terms of control technology, a significant challenge is due to lack of vendor support in terms of design tools and run-time support for agent-based IEC 61499 deployments.In manufacturing setups, disturbances produced cause deviation from plan and degradation of performance in overall systems.The frameworks usually developed do not account for these.Future works on agent frameworks should also factor in treatment of disturbances and exceptions for intelligent manufacturing operation execution.
Regarding the technological level, agent-based cyberphysical infrastructures normally lack of a connectivity analysis.This is necessary due to high number and sometimescomplex negotiation mechanisms applied.This can generate high levels of latency and thus communication bottlenecks.Therefore, the measuring and optimisation of related KPIs are necessary.Learning based methods should be highly considered in future applications.So far, most of approaches rely in predefined knowledge based models.However, the imple-mentation of more innovative techniques has been elusive.For example, the inclusion of deep reinforcement learning could be utilised for the creation of dynamic learning agents.This can reduce the dependence from classical negotiation protocols and potentiate the development of more open architectures.In addition, the consideration of the context where agents are applied or the human-agent interactions as a way to reinforce policies or rules seems to be not highly applied.This still open issue can enhance framework adaptability and thus provide continuous process improvement.In terms of enterprise management, there is a clear gap among the integration of low level agents (e.g.shop floor resources) with agents representing high level enterprise actors.Also, few works consider inter-enterprise communication.Thus, the need of generating more detailed models that can showcase the integration of all actors in manufacture value chain and the development of more complete generic framework in stead of application specific.
In application of MAS in manufacturing, one challenge which was clear while reviewing the literature was that distributed task scheduling problem is yet to be addressed.Giuseppe D'Aniello et.al. [49] has partially tacked this issue in the context of cloud manufacturing by considering an architecture based on autonomous scattered manufacturing resources with dynamically re-configurable network.There is still a lack of MAS based decision support capabilities for manufacturing execution systems on shop floor.

IV. CONCLUSION
The paper presents an overview of the applications of multiagent systems in manufacturing, the current trends in research, academia and industry along with underlying challenges that must be addressed for further enforcing of the concept.This paper breaks down the focus into four major research questions namely discussing frameworks, technologies,application and challenges.The fundamental methodology of the review of the topic in the research domain is presented for reference.The paper presents an in-dept overview of different frameworks and architectures developed in order to target multi-agent system implementation.This gives an idea on the current focus and trends that are applicable to the area of research in-terms of techniques that solutions are based on, most common areas targeted, frequent problems identified, role of agent interaction and consideration.This in-depth review also gives rise to an observation on technologies used to support development of MAS.The most common technologies used for realisation are broken into fields of application or scope of application and presented.Common overlaps of technologies and characteristics of MAS enablers are highlighted.The application section breaks down the gathered research into the targeted areas in manufacturing.The analysis leads to an idea on the most targeted areas in manufacturing for MAS implementations, and assists in mapping of the reasons for those areas to be targeted along with major challenges witnessed for current and future research.
Finally, the section on challenges presents the gaps that exists in frameworks, technologies and application that hinder the progression of MAS integration in manufacturing.The main reasons are primarily attributed to lack of standard architectures/frameworks, lack of real-time adaptable execution and compliance, interoperability, lack of synchronisation and scheduling methodologies, real-time constraints, reference models and lack of methods of specification requirement association in current standards.Compatibility issues across technologies can restrict integration and in others if a means is present then the protocols, standards and specifications are needed to support such transitions.In addition to this, connectivity issues between components of platforms needed for agent-negotiation, improvement of system based on learning techniques and instantiating context-awareness in the manufacturing setting are significant challenge.Also, distributed systems in manufacturing see a general lacking of cloud service models.Future works should target these areas for better MAS integration in manufacturing.
The 4 Research Questions identified for this review paper are mentioned below, RQ1: What are the various frameworks and architectures developed in recent years for MAS in manufacturing?RQ2: What are the different technologies that support the development of MAS in recent years?RQ3: What are the various applications of MAS in manufacturing ?RQ4: What are the current challenges in the development of MAS in smart Manufacturing ?

TABLE I :
Frameworks and Architectures for MAS Tonelli et al. [33]: Manufacturing sustainability framework Block libraries, base building blocks used for developing models.Sustainability issues and key performance indicators (KPIs) are considered.The model defined in environment consists of: agents, behaviours, etc.A model agent simulation developed for use-case of food producing plant for animals.Semiautomatic line conveyors based automated line, buffer, manual line, lift , shaker and finally packaging.Kovalenko et al. [34] : Distributed MAS task Negotiation Distributed MAS proposed to improve flexibility.Two important components (product and resource agents).Architecture consists of: Resource agent Knowledge, RA provided with model of their capabilities: information of resource capability and Neighbouring.RAs: Information of the states shared with other resources.Product Agent Knowledge, process plan and product history.Pas communicate with RA teams through bids.RAs use them to enable task negotiation and PA to formulate better environmental understating.Dhokia et al. [35] : generative multi-agent design methodology Methodology that relied on termite behavior to simultaneously design, optimise and evaluate parts produced by additive manufacturing.Agent based design by 200 termites to optimise design problem under objective conditions.Iterations are performed by feedback loop to finite element solver to demonstrate AM parts.Agent based generative design tool developed.

TABLE II :
Technologies that support MAS Applications