Tackling the "Gremlins Paradox" : Autonomous Hot Swap Healing Protocol for Chronic Performance Problem Aversion in On-board Mission Critical Systems for Lethal Autonomous Weapons

Nyagudi Musandu Nyagudi

DOI: https://doi.org/10.48550/arXiv.1409.4409

2014-09-16

Abstract:This paper gives insights into the issue of Chronic Performance Problem Aversion in On-board Mission Critical Systems for Lethal Autonomous Weapons

Cryptography and Security

What problem does this paper attempt to address?

Adversarial Attacks and Mitigation for Anomaly Detectors of Cyber-Physical Systems

Yifan Jia,Jingyi Wang,Christopher M. Poskitt,Sudipta Chattopadhyay,Jun Sun,Yuqi Chen

DOI: https://doi.org/10.1016/j.ijcip.2021.100452

IF: 3.683

2021-01-01

International Journal of Critical Infrastructure Protection

Abstract:The threats faced by cyber-physical systems (CPSs) in critical infrastructure have motivated research into a multitude of attack detection mechanisms, including anomaly detectors based on neural network models. The effectiveness of anomaly detectors can be assessed by subjecting them to test suites of attacks, but less consideration has been given to adversarial attackers that craft noise specifically designed to deceive them. While successfully applied in domains such as images and audio, adversarial attacks are much harder to implement in CPSs due to the presence of other built-in defence mechanisms such as rule checkers (or invariant checkers). In this work, we present an adversarial attack that simultaneously evades the anomaly detectors and rule checkers of a CPS. Inspired by existing gradient-based approaches, our adversarial attack crafts noise over the sensor and actuator values, then uses a genetic algorithm to optimise the latter, ensuring that the neural network and the rule checking system are both deceived. We implemented our approach for two real-world critical infrastructure testbeds, successfully reducing the classification accuracy of their detectors by over 50% on average, while simultaneously avoiding detection by rule checkers. Finally, we explore whether these attacks can be mitigated by training the detectors on adversarial samples.
Ancillary Mechanism for Autonomous Decision-Making Process in Asymmetric Confrontation: a View from Gomoku

Chen Han,Xuanyin Wang

DOI: https://doi.org/10.1080/0952813x.2022.2067249

2022-01-01

Abstract:This paper investigates how agents learn and perform efficient strategies by trying different actions in asymmetric confrontation setting. Firstly, we use Gomoku as an example to analyse the causes and impacts of asymmetric confrontation: the first mover gains higher power than the second mover. We find that the first mover learns how to attack quickly while it is difficult for the second mover to learn how to defend since it cannot win the first mover and always receives negative rewards. As such, the game is stuck at a deadlock in which the first mover cannot make further advances to learn how to defend, and the second mover learns nothing. Secondly, we propose an ancillary mechanism (AM) to add two principles to the agent's actions to overcome this difficulty. AM is a guidance for the agents to reduce the learning difficulty and to improve their behavioural quality. To the best of our knowledge, this is the first study to define asymmetric confrontation in reinforcement learning and propose approaches to tackle such problems. In the numerical tests, we first conduct a simple human vs AI experiment to calibrate the learning process in asymmetric confrontation. Then, an experiment of 15*15 Gomoku game by letting two agents (with AM and without AM) compete is applied to check the potential of AM. Results show that adding AM can make both the first and the second movers become stronger in almost the same amount of calculation.
Humanitarian Algorithms : A Codified Key Safety Switch Protocol for Lethal Autonomy

Nyagudi Musandu Nyagudi

DOI: https://doi.org/10.48550/arXiv.1402.2206

2014-02-11

Abstract:With the deployment of lethal autonomous weapons, there is the requirement that any such platform complies with the precepts of International Humanitarian Law. Humanitarian Algorithms[9: p. 9] ensure that lethal autonomous weapon systems perform military/security operations, within the confines of International Humanitarian Law. Unlike other existing techniques of regulating lethal autonomy this scheme advocates for an approach that enables Machine Learning. Lethal autonomous weapons must be equipped with appropriate fail-safe mechanisms that locks them if they malfunction.

Cryptography and Security
Dynamic Control Allocation between Onboard and Delayed Remote Control for Unmanned Aircraft System Detect-and-Avoid

Asma Tabassum,He Bai

DOI: https://doi.org/10.1016/j.ast.2021.107323

2021-03-14

Abstract:This paper develops and evaluates the performance of an allocation agent to be potentially integrated into the onboard Detect and Avoid (DAA) computer of an Unmanned Aircraft System (UAS). We consider a UAS that can be fully controlled by the onboard DAA system and by a remote human pilot. With a communication channel prone to latency, we consider a mixed initiative interaction environment, where the control authority of the UAS is dynamically allocated by the allocation agent. In an encounter with a dynamic intruder, the probability of collision may increase in the absence of pilot commands in the presence of latency. Moreover, a delayed pilot command may not result in safe resolution of the current scenario and need to be improvised. We design an optimization algorithm to reduce collision risk and refine delayed pilot commands. Towards this end, a Markov Decision Process (MDP)and its solution are employed to create a wait time map. The map consists of estimated times that the UAS can wait for the remote pilot commands at each state. A command blending algorithm is designed to select an avoidance maneuver that prioritizes the pilot intention extracted from the pilot commands. The wait time map and the command blending algorithm are implemented and integrated into a closed-loop simulator. We conduct ten thousands fast-time Monte Carlo simulations and compare the performance of the integrated setup with a standalone DAA setup. The simulation results show that the allocation agent enables the UAS to wait without inducing any near mid air collision (NMAC) and severe loss of well clear (LoWC) while positively improve pilot involvement in the encounter resolution.

Systems and Control,Human-Computer Interaction
Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms

Sheila Schoepp,Mehran Taghian,Shotaro Miwa,Yoshihiro Mitsuka,Shadan Golestan,Osmar Zaïane

2024-07-22

Abstract:Industry is rapidly moving towards fully autonomous and interconnected systems that can detect and adapt to changing conditions, including machine hardware faults. Traditional methods for adding hardware fault tolerance to machines involve duplicating components and algorithmically reconfiguring a machine's processes when a fault occurs. However, the growing interest in reinforcement learning-based robotic control offers a new perspective on achieving hardware fault tolerance. However, limited research has explored the potential of these approaches for hardware fault tolerance in machines. This paper investigates the potential of two state-of-the-art reinforcement learning algorithms, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), to enhance hardware fault tolerance into machines. We assess the performance of these algorithms in two OpenAI Gym simulated environments, Ant-v2 and FetchReach-v1. Robot models in these environments are subjected to six simulated hardware faults. Additionally, we conduct an ablation study to determine the optimal method for transferring an agent's knowledge, acquired through learning in a normal (pre-fault) environment, to a (post-)fault environment in a continual learning setting. Our results demonstrate that reinforcement learning-based approaches can enhance hardware fault tolerance in simulated machines, with adaptation occurring within minutes. Specifically, PPO exhibits the fastest adaptation when retaining the knowledge within its models, while SAC performs best when discarding all acquired knowledge. Overall, this study highlights the potential of reinforcement learning-based approaches, such as PPO and SAC, for hardware fault tolerance in machines. These findings pave the way for the development of robust and adaptive machines capable of effectively operating in real-world scenarios.

Machine Learning,Artificial Intelligence,Robotics
Adaptive Optimization of Autonomous Vehicle Computational Resources for Performance and Energy Improvement

Saurabh Jambotkar,Longxiang Guo,Yunyi Jia

DOI: https://doi.org/10.48550/arXiv.2102.07000

2021-07-30

Abstract:Autonomous vehicles usually consume a large amount of computational power for their operations, especially for the tasks of sensing and perception with artificial intelligence algorithms. Such a computation may not only cost a significant amount of energy but also cause performance issues when the onboard computational resources are limited. To address this issue, this paper proposes an adaptive optimization method to online allocate the onboard computational resources of an autonomous vehicle amongst multiple vehicular subsystems depending on the contexts of the situations that the vehicle is facing. Different autonomous driving scenarios were designed to validate the proposed approach and the results showed that it could help improve the overall performance and energy consumption of autonomous vehicles compared to existing computational arrangement.

Systems and Control
War Elephants: Rethinking Combat AI and Human Oversight

Philip Feldman,Aaron Dant,Harry Dreany

2024-04-30

Abstract:This paper explores the changes that pervasive AI is having on the nature of combat. We look beyond the substitution of AI for experts to an approach where complementary human and machine abilities are blended. Using historical and modern examples, we show how autonomous weapons systems can be effectively managed by teams of human "AI Operators" combined with AI/ML "Proxy Operators." By basing our approach on the principles of complementation, we provide for a flexible and dynamic approach to managing lethal autonomous systems. We conclude by presenting a path to achieving an integrated vision of machine-speed combat where the battlefield AI is operated by AI Operators that watch for patterns of behavior within battlefield to assess the performance of lethal autonomous systems. This approach enables the development of combat systems that are likely to be more ethical, operate at machine speed, and are capable of responding to a broader range of dynamic battlefield conditions than any purely autonomous AI system could support.

Computers and Society,Artificial Intelligence
Resolving responsibility gaps for lethal autonomous weapon systems

Patrick Taylor Smith

DOI: https://doi.org/10.3389/fdata.2022.1038507

2022-12-06

Frontiers in Big Data

Abstract:This paper offers a novel understanding of collective responsibility for AI outcomes that can help resolve the “problem of many hands” and “responsibility gaps” when it comes to AI failure, especially in the context of lethal autonomous weapon systems.
Autonomous Military Robotics: Risk, Ethics, and Design

Patrick Lin,George Bekey,Keith Abney

DOI: https://doi.org/10.21236/ada534697

2008-12-20

Abstract:Imagine the face of warfare with autonomous robotics: Instead of our soldiers returning home in flag-draped caskets to heartbroken families, autonomous robots-mobile machines that can make decisions, such as to fire upon a target, without human intervention-can replace the human soldier in an increasing range of dangerous missions: from tunneling through dark caves in search of terrorists, to securing urban streets rife with sniper fire, to patrolling the skies and waterways where there is little cover from attacks, to clearing roads and seas of improvised explosive devices (IEDs), to surveying damage from biochemical weapons, to guarding borders and buildings, to controlling potentially-hostile crowds, and even as the infantry frontlines. These robots would be 'smart' enough to make decisions that only humans now can; and as conflicts increase in tempo and require much quicker information processing and responses, robots have a distinct advantage over the limited and fallible cognitive capabilities that we Homo sapiens have. Not only would robots expand the battlespace over difficult, larger areas of terrain, but they also represent a significant force-multiplier-each effectively doing the work of many human soldiers, while immune to sleep deprivation, fatigue, low morale, perceptual and communication challenges in the 'fog of war', and other performance-hindering conditions.
Subverting Skynet: The Strategic Promise of Lethal Autonomous Weapons and the Perils of Exploitation

Lennart Maschmeyer

DOI: https://doi.org/10.23919/CyCon55549.2022.9811008

2022-05-31

Abstract:Lethal Autonomous Weapons Systems (LAWS) promise a revolution in warfare by increasing the lethality of force while reducing the costs of war. Yet these gains come at the cost of significant yet underappreciated perils. LAWS are vulnerable to subversion, allowing adversaries to degrade or disable these systems or even turn them against their makers. Subversion involves exploiting flaws in complex systems to make them behave in unexpected ways. It is possible because, like other computer systems, the behavior of LAWS is determined by logical rules and routines. These rules and routines inevitably contain flaws, creating vulnerabilities that adversaries can exploit. This paper identifies three avenues of subversion: (1) manipulating the algorithm itself during the design process, (2) poisoning the data used to train the artificial intelligence operating the LAWS, and (3) manipulating physical objects LAWS are trained to respond to. This potential for subversion creates fundamental uncertainty for strategic planners, military commanders, and soldiers in the field. LAWS are powerful capabilities that may win wars, yet they may also become liabilities that lead to defeat against crafty adversaries. Hence, fulfilling the strategic promise of LAWS requires mitigating their vulnerabilities. Examining possible mitigations, the paper shows that technical fixes are currently unavailable, necessitating strategic solutions. It identifies two possible solutions. The first is employing counterintelligence strategies and tactics to detect, neutralize, and pre-empt attempts at subversion. The second is adopting a force structure model that maintains human superiority to neutralize rogue LAWS if necessary. However, both solutions reduce operational effectiveness and the strategic value of LAWS, thus forfeiting some of the core advantages these new systems promise. Consequently, the paper concludes that the strategic challenges of deploying LAWS currently outweigh the opportunities, necessitating a cautious approach and a greater prioritization of strategy development.

Political Science,Computer Science
Multi-Agent Deep Reinforcement Learning-Driven Mitigation of Adverse Effects of Cyber-Attacks on Electric Vehicle Charging Station

M. Basnet,MH Ali

DOI: https://doi.org/10.48550/arXiv.2207.07041

2022-07-15

Abstract:An electric vehicle charging station (EVCS) infrastructure is the backbone of transportation electrification. However, the EVCS has myriads of exploitable vulnerabilities in software, hardware, supply chain, and incumbent legacy technologies such as network, communication, and control. These standalone or networked EVCS open up large attack surfaces for the local or state-funded adversaries. The state-of-the-art approaches are not agile and intelligent enough to defend against and mitigate advanced persistent threats (APT). We propose the data-driven model-free distributed intelligence based on multiagent Deep Reinforcement Learning (MADRL)-- Twin Delayed Deep Deterministic Policy Gradient (TD3) -- that efficiently learns the control policy to mitigate the cyberattacks on the controllers of EVCS. Also, we have proposed two additional mitigation methods: the manual/Bruteforce mitigation and the controller clone-based mitigation. The attack model considers the APT designed to malfunction the duty cycles of the EVCS controllers with Type-I low-frequency attack and Type-II constant attack. The proposed model restores the EVCS operation under threat incidence in any/all controllers by correcting the control signals generated by the legacy controllers. Also, the TD3 algorithm provides higher granularity by learning nonlinear control policies as compared to the other two mitigation methods. Index Terms: Cyberattack, Deep Reinforcement Learning(DRL), Electric Vehicle Charging Station, Mitigation.

Systems and Control
Moving to Advanced Research on Human Machine Interface Design

Keith Brawner

DOI: https://doi.org/10.32473/flairs.37.1.135458

2024-05-13

The International FLAIRS Conference Proceedings

Abstract:This poster paper will discuss a variety of items related to Human-Machine Teaming and research in support in-creasing control of autonomous machines present in phys-ical problem domains of interest. Many military tasks can be decomposed into their primary elements – intelligence preparation, reconnaissance, movement, maneuver, fires, and and support across the combat domains of interest – air, land, sea, etc. There are an increasing number of au-tonomous and semi-autonomous ground-based systems available, such as the Multi-Utility Tactical Transport (MUTT) Unmanned Ground Vehicle, for movement of materials, or Quadrupedal Unmanned Ground Vehicle (QUGV) for the disposal of explosive ordinance, comple-mented by aerial platforms considering of a wide variety of Unmanned Aerial Systems (UAS) for the gathering and transmitting of information, held together by a common backbone and network. The preponderance of new systems and capabilities brings new issues, one of which is critical to research - how to control and manage a large number of systems. Commercial systems of significantly less capability, such as light-up drone shows, involve approximately 2 people per 100 drones; meaning that a single controller for a sin-gle drone is simply a non-starter. How can research be applied to scale the complexity of operations upwards without additional demands of personnel? This poster discusses several portions of early-stage research into a variety of applications, including: Reasoning about mixed-team processes, includ-ing real/synthetic teammates, and how infor-mation gathered about the use of synthetic teammates within simulation can be transitioned into utilization for a robotic teammate Embedding affective information into dialogue channels in order to save cognitive bandwidth Repurposing of foundational dialogue models for specific tasks and purposes Utilizing psychological research to design sys-tems in order to infer user intent Theory of mind research for autonomous systems regarding their human operators Simulated environments and agents in order to test the simulations in representative areas. Common-Sense reasoning augmented by Large Language Model technology for instructing ro-botic platforms The following things will be discussed at the poster ses-sion, representing a portfolio of ongoing work addressing the problems of large-scale multi-agent command and control. While the near-term application of these tech-nologies is into military problem domains; the poster presentation is cleared for public release.
Rise of the HaCRS: Augmenting Autonomous Cyber Reasoning Systems with Human Assistance

Yan Shoshitaishvili,Michael Weissbacher,Lukas Dresel,Christopher Salls,Ruoyu Wang,Christopher Kruegel,Giovanni Vigna

DOI: https://doi.org/10.48550/arXiv.1708.02749

2017-08-09

Abstract:As the size and complexity of software systems increase, the number and sophistication of software security flaws increase as well. The analysis of these flaws began as a manual approach, but it soon became apparent that tools were necessary to assist human experts in this task, resulting in a number of techniques and approaches that automated aspects of the vulnerability analysis process. Recently, DARPA carried out the Cyber Grand Challenge, a competition among autonomous vulnerability analysis systems designed to push the tool-assisted human-centered paradigm into the territory of complete automation. However, when the autonomous systems were pitted against human experts it became clear that certain tasks, albeit simple, could not be carried out by an autonomous system, as they require an understanding of the logic of the application under analysis. Based on this observation, we propose a shift in the vulnerability analysis paradigm, from tool-assisted human-centered to human-assisted tool-centered. In this paradigm, the automated system orchestrates the vulnerability analysis process, and leverages humans (with different levels of expertise) to perform well-defined sub-tasks, whose results are integrated in the analysis. As a result, it is possible to scale the analysis to a larger number of programs, and, at the same time, optimize the use of expensive human resources. In this paper, we detail our design for a human-assisted automated vulnerability analysis system, describe its implementation atop an open-sourced autonomous vulnerability analysis system that participated in the Cyber Grand Challenge, and evaluate and discuss the significant improvements that non-expert human assistance can offer to automated analysis approaches.

Cryptography and Security,Human-Computer Interaction
Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers

Aryaman Gupta,Kaustav Chakraborty,Somil Bansal

2024-09-28

Abstract:Autonomous systems, such as self-driving cars and drones, have made significant strides in recent years by leveraging visual inputs and machine learning for decision-making and control. Despite their impressive performance, these vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. Such errors can cascade to catastrophic system failures and compromise system safety. In this work, we introduce a run-time anomaly monitor to detect and mitigate such closed-loop, system-level failures. Specifically, we leverage a reachability-based framework to stress-test the vision-based controller offline and mine its system-level failures. This data is then used to train a classifier that is leveraged online to flag inputs that might cause system breakdowns. The anomaly detector highlights issues that transcend individual modules and pertain to the safety of the overall system. We also design a fallback controller that robustly handles these detected anomalies to preserve system safety. We validate the proposed approach on an autonomous aircraft taxiing system that uses a vision-based controller for taxiing. Our results show the efficacy of the proposed approach in identifying and handling system-level anomalies, outperforming methods such as prediction error-based detection, and ensembling, thereby enhancing the overall safety and robustness of autonomous systems.

Robotics,Computer Vision and Pattern Recognition,Machine Learning,Systems and Control
Adaptive Online Fault Diagnosis in Autonomous Robot Swarms

James O'Keeffe,Danesh Tarapore,Alan G. Millard,Jon Timmis

DOI: https://doi.org/10.3389/frobt.2018.00131

2018-11-30

Frontiers in Robotics and AI

Abstract:Previous work has shown that robot swarms are not always tolerant to the failure of individual robots, particularly those that have only partially failed and continue to contribute to collective behaviors. A case has been made for an active approach to fault tolerance in swarm robotic systems, whereby the swarm can identify and resolve faults that occur during operation. Existing approaches to active fault tolerance in swarms have so far omitted fault diagnosis, however we propose that diagnosis is a feature of active fault tolerance that is necessary if swarms are to obtain long-term autonomy. This paper presents a novel method for fault diagnosis that attempts to imitate some of the observed functions of natural immune system. The results of our simulated experiments show that our system is flexible, scalable, and improves swarm tolerance to various electro-mechanical faults in the cases examined.
Algorithms for the Greater Good! On Mental Modeling and Acceptable Symbiosis in Human-AI Collaboration

Tathagata Chakraborti,Subbarao Kambhampati

DOI: https://doi.org/10.48550/arXiv.1801.09854

IF: 14.4

2018-01-30

Artificial Intelligence

Abstract:Effective collaboration between humans and AI-based systems requires effective modeling of the human in the loop, both in terms of the mental state as well as the physical capabilities of the latter. However, these models can also open up pathways for manipulating and exploiting the human in the hopes of achieving some greater good, especially when the intent or values of the AI and the human are not aligned or when they have an asymmetrical relationship with respect to knowledge or computation power. In fact, such behavior does not necessarily require any malicious intent but can rather be borne out of cooperative scenarios. It is also beyond simple misinterpretation of intents, as in the case of value alignment problems, and thus can be effectively engineered if desired. Such techniques already exist and pose several unresolved ethical and moral questions with regards to the design of autonomy. In this paper, we illustrate some of these issues in a teaming scenario and investigate how they are perceived by participants in a thought experiment.
When Autonomous Intelligent Goodware will Fight Autonomous Intelligent Malware: A Possible Future of Cyber Defense

Paul Théron,Alexander Kott

DOI: https://doi.org/10.48550/arXiv.1912.01959

2019-11-26

Abstract:In the coming years, the future of military combat will include, on one hand, artificial intelligence-optimized complex command, control, communications, computers, intelligence, surveillance and reconnaissance (C4ISR) and networks and, on the other hand, autonomous intelligent Things fighting autonomous intelligent Things at a fast pace. Under this perspective, enemy forces will seek to disable or disturb our autonomous Things and our complex infrastructures and systems. Autonomy, scale and complexity in our defense systems will trigger new cyber-attack strategies, and autonomous intelligent malware (AIM) will be part of the picture. Should these cyber-attacks succeed while human operators remain unaware or unable to react fast enough due to the speed, scale or complexity of the mission, systems or attacks, missions would fail, our networks and C4ISR would be heavily disrupted, and command and control would be disabled. New cyber-defense doctrines and technologies are therefore required. Autonomous cyber defense (ACyD) is a new field of research and technology driven by the defense sector in anticipation of such threats to future military infrastructures, systems and operations. It will be implemented via swarms of autonomous intelligent cyber-defense agents (AICAs) that will fight AIM within our networks and systems. This paper presents this cyber-defense technology of the future, the current state of the art in this field and its main challenges. First, we review the rationale of the ACyD concept and its associated AICA technology. Then, we present the current research results from NATO's IST-152 Research Task Group on the AICA Reference Architecture. We then develop the 12 main technological challenges that must be resolved in the coming years, besides ethical and political issues.

Cryptography and Security
Mitigating undesirable emergent behavior arising between driver and semi-automated vehicle

Timo Melman,Niek Beckers,David Abbink

DOI: https://doi.org/10.48550/arXiv.2006.16572

IF: 6.4588

2020-06-30

Human-Computer Interaction

Abstract:Emergent behavior arising in a joint human-robot system cannot be fully predicted based on an understanding of the individual agents. Typically, robot behavior is governed by algorithms that optimize a reward function that should quantitatively capture the joint system's goal. Although reward functions can be updated to better match human needs, this is no guarantee that no misalignment with the complex and variable human needs will occur. Algorithms may learn undesirable behavior when interacting with the human and the intrinsically unpredictable human-inhabited world, thereby producing further misalignment with human users or bystanders. As a result, humans might behave differently than anticipated, causing robots to learn differently and undesirable behavior to emerge. With this short paper, we state that to design for Human-Robot Interaction that mitigates such undesirable emergent behavior, we need to complement advancements in human-robot interaction algorithms with human factors knowledge and expertise. More specifically, we advocate a three-pronged approach that we illustrate using a particularly challenging example of safety-critical human-robot interaction: a driver interacting with a semi-automated vehicle. Undesirable emergent behavior should be mitigated by a combination of 1) including driver behavioral mechanisms in the vehicle's algorithms and reward functions, 2) model-based approaches that account for interaction-induced driver behavioral adaptations and 3) driver-centered interaction design that promotes driver engagement with the semi-automated vehicle, and the transparent communication of each agent's actions that allows mutual support and adaptation. We provide examples from recent empirical work in our group, in the hope this proves to be fruitful for discussing emergent human-robot interaction.
A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?

Nathan I. N. Henry,Mangor Pedersen,Matt Williams,Jamin L. B. Martin,Liesje Donkin

2024-02-12

Abstract:The value-loading problem is a significant challenge for researchers aiming to create artificial intelligence (AI) systems that align with human values and preferences. This problem requires a method to define and regulate safe and optimal limits of AI behaviors. In this work, we propose HALO (Hormetic ALignment via Opponent processes), a regulatory paradigm that uses hormetic analysis to regulate the behavioral patterns of AI. Behavioral hormesis is a phenomenon where low frequencies of a behavior have beneficial effects, while high frequencies are harmful. By modeling behaviors as allostatic opponent processes, we can use either Behavioral Frequency Response Analysis (BFRA) or Behavioral Count Response Analysis (BCRA) to quantify the hormetic limits of repeatable behaviors. We demonstrate how HALO can solve the 'paperclip maximizer' scenario, a thought experiment where an unregulated AI tasked with making paperclips could end up converting all matter in the universe into paperclips. Our approach may be used to help create an evolving database of 'values' based on the hedonic calculus of repeatable behaviors with decreasing marginal utility. This positions HALO as a promising solution for the value-loading problem, which involves embedding human-aligned values into an AI system, and the weak-to-strong generalization problem, which explores whether weak models can supervise stronger models as they become more intelligent. Hence, HALO opens several research avenues that may lead to the development of a computational value system that allows an AI algorithm to learn whether the decisions it makes are right or wrong.

Multiagent Systems,Artificial Intelligence,Theoretical Economics,Machine Learning,Computers and Society
CRASH: Challenging Reinforcement-Learning Based Adversarial Scenarios For Safety Hardening

Amar Kulkarni,Shangtong Zhang,Madhur Behl

2024-11-26

Abstract:Ensuring the safety of autonomous vehicles (AVs) requires identifying rare but critical failure cases that on-road testing alone cannot discover. High-fidelity simulations provide a scalable alternative, but automatically generating realistic and diverse traffic scenarios that can effectively stress test AV motion planners remains a key challenge. This paper introduces CRASH - Challenging Reinforcement-learning based Adversarial scenarios for Safety Hardening - an adversarial deep reinforcement learning framework to address this issue. First CRASH can control adversarial Non Player Character (NPC) agents in an AV simulator to automatically induce collisions with the Ego vehicle, falsifying its motion planner. We also propose a novel approach, that we term safety hardening, which iteratively refines the motion planner by simulating improvement scenarios against adversarial agents, leveraging the failure cases to strengthen the AV stack. CRASH is evaluated on a simplified two-lane highway scenario, demonstrating its ability to falsify both rule-based and learning-based planners with collision rates exceeding 90%. Additionally, safety hardening reduces the Ego vehicle's collision rate by 26%. While preliminary, these results highlight RL-based safety hardening as a promising approach for scenario-driven simulation testing for autonomous vehicles.

Machine Learning,Robotics

Tackling the "Gremlins Paradox" : Autonomous Hot Swap Healing Protocol for Chronic Performance Problem Aversion in On-board Mission Critical Systems for Lethal Autonomous Weapons

Adversarial Attacks and Mitigation for Anomaly Detectors of Cyber-Physical Systems

Ancillary Mechanism for Autonomous Decision-Making Process in Asymmetric Confrontation: a View from Gomoku

Humanitarian Algorithms : A Codified Key Safety Switch Protocol for Lethal Autonomy

Dynamic Control Allocation between Onboard and Delayed Remote Control for Unmanned Aircraft System Detect-and-Avoid

Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms

Adaptive Optimization of Autonomous Vehicle Computational Resources for Performance and Energy Improvement

War Elephants: Rethinking Combat AI and Human Oversight

Resolving responsibility gaps for lethal autonomous weapon systems

Autonomous Military Robotics: Risk, Ethics, and Design

Subverting Skynet: The Strategic Promise of Lethal Autonomous Weapons and the Perils of Exploitation

Multi-Agent Deep Reinforcement Learning-Driven Mitigation of Adverse Effects of Cyber-Attacks on Electric Vehicle Charging Station

Moving to Advanced Research on Human Machine Interface Design

Rise of the HaCRS: Augmenting Autonomous Cyber Reasoning Systems with Human Assistance

Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers

Adaptive Online Fault Diagnosis in Autonomous Robot Swarms

Algorithms for the Greater Good! On Mental Modeling and Acceptable Symbiosis in Human-AI Collaboration

When Autonomous Intelligent Goodware will Fight Autonomous Intelligent Malware: A Possible Future of Cyber Defense

Mitigating undesirable emergent behavior arising between driver and semi-automated vehicle

A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?

CRASH: Challenging Reinforcement-Learning Based Adversarial Scenarios For Safety Hardening