Kyriakos G. Vamvoudakis - Research Projects

Our research draws from the areas of control theory, cyber-physical systems, game theory, and reinforcement learning. Our recent research interests lie in the design of robust, safe, and secure multi-agent networked systems. Below is a brief description of some recent research projects.

Learning-enabled systems, e.g., autonomous systems, cyber-physical systems, and high-performance computing applications, have been rapidly increasing in size and acquiring new capabilities. These systems are typically deployed in complex operating environments, so their safety becomes extremely important. Ensuring safety requires that systems are robust to extreme events while we can monitor them for anomalous and unsafe behavior. Using model-free reinforcement learning (RL) to train the agent by directly mapping raw observations into actions or state-action values usually requires a large amount of training data. Model-based RL algorithms obtain a large amount of supervision from every sample, since they can use each sample to learn better how to predict the system dynamics that is, to learn the physics of the problem. Once the dynamics are learned, near-optimal behavior can in principle, be obtained by planning through these dynamics.

The overarching goal of the proposed research is to establish a framework that leads to the design and implementation of learning-enabled systems in which safety is ensured with high levels of confidence. The framework will leverage tools from control theory, multi-agent autonomy, and formal methods for rigorously probabilistic reasoning to yield safe learning-enabled systems. The expected outcome of this project will yield safe model-free, mode-based, and interacting model-free and model-based learning-enabled systems with sound design principles that practitioners could leverage to achieve safety specifications. The proposed research could effectively facilitate safe learning-enabled systems even within complex environments while monitoring them for anomalous and unsafe behavior. For more details see our published work.

The recent boom in artificial intelligence and learning has brought autonomy to a new level, especially in the military domain: we now have unmanned battles, carried out by autonomous systems that are able to learn and adapt in unknown environments and under unforeseen circumstances, defense systems are trained with machine learning methods to accurately distinguish between allies and threats, and human decisions are being assisted by learning-based tools that are fed real-time measurements and provide recommendations for the best course of action. This significant expansion of automation via machine learning has increased the attack surface of almost any device/agent entailing a digital component.

The goal of the proposed research is to ensure the autonomy, efficiency, and safety, of complex adaptive systems with and without humans-on-the-loop. We will develop learning-based rules for agents that are constantly changing their internal structure and are thus unpredictable and hard to decipher by adversaries. We will also be able to answer the following question: how can an intelligent agent use the information it has learned thus far, to gain an advantage over a reinforcement learning based adversary that has less information. For more details see our published work.

The vulnerability of closed-loop reinforcement learning cannot be exploited by existing techniques that build upon heuristics, require knowledge of the system and excessive (offline) training, and do not provide quantifiable optimality given an adversary. To the best of our knowledge, there has not been any work on closed-loop adversarial reinforcement learning with formal stability and robustness guarantees that can effectively quantify adversarial vulnerabilities inside the learning mechanisms.

This project will develop the next generation of agile, highly adaptive autonomous systems that utilize mechanisms from cognition and learning to process information from distributed sensors. In particular, looking to autonomous systems appearing in nature for inspiration, we will develop robust and safe reinforcement learning algorithms to address agent interactions, cooperative and non-cooperative tasks, task assignments, and resource allocations. The framework will be validated to ensure its maximal possible impact and implementability. The team will work closely with MathWorks to develop a toolbox and host internships. For more details see our published work.

In structural design and qualification applications for multi-physics environments, engineers frequently need to evaluate an expensive model and/or experiment many times in order to assess the structure's response to the various environments (thermal, vibration electrical, etc), and then make design changes to mitigate any potential structural failure mechanisms. This process is subject to the so-called curse of dimensionality, i.e. a rapid growth in the number of evaluations of the model/experiment as the number of environmental/design parameters grows. We propose to instead develop classification methods that can address these problems in reverse; that is, we assume structural failure has occurred and look backwards to assess which environments could lead to that failure.

A variety of classification methods in the literature such as Support Vector Machines have been developed to address the curse of dimensionality and reduce the number of samples needed to separate a design space into pass/fail bins. However, a relatively unexplored mechanism for solving classification problems is through reinforcement learning. This project will develop novel classification methods using reinforcement learning and related concepts for these types of structural engineering applications. For more details see our published work.

Reinforcement learning, an emerging branch of machine learning (ML) and control, has a lot to promise for autonomy with its capacity to address unpredictable changes in the system and environment. However, the field still has many research gaps and also suffers from the lack of practical research evaluation. Most AI based autonomous vehicles research is performed in simulations, on simple platforms and for simplified cases that are far from reflecting real-life uncertainties and convincing responses to changing road conditions, especially at high speeds.

This project will support selected students from US universities to tackle the open challenges of fully autonomous vehicles within a cohort experience at Istanbul Technical University (ITU) under the mentorship of subject-matter experts from ITU, from the Royal Institute of Technology (KTH, Sweden), and a US based autonomous bus company (ADASTEC Corp). The algorithm tests will be performed around the ITU Campus within real-world scenarios. Each year, a different student cohort will be selected for this unique research and professional development opportunity, thereby contributing to the US leadership in the future of vehicle technologies with a well-prepared workforce. Special recruitment efforts are planned for broadening participation and recruitment of students from underrepresented communities. For more details see our published work.

Cyber-physical critical infrastructures provide management and control capabilities for mission-critical utilities such as power grids. Programmable logic controllers (PLCs) play a key role as they serve as a convenient bridge between the cyber and the physical worlds. PLCs' critical roles have made them the target of sophisticated cyberattacks that are designed to disrupt their operation, which creates both social unrest and financial losses. In this context, honeypots have been shown to be highly valuable tools for collecting data to better understand the many different strategies and objectives of the attackers. The project's novelty is to develop a new domain-specific stealthy honeypot for cyber-physical critical infrastructures and specifically PLC controllers. The solutions allow for active data collection using autonomous interactions with the malware to activate its malicious capabilities The project's broader significance and importance are to provide guidelines for researchers and practitioners looking to incorporate honeypots and security methods into cyber-physical systems (CPS) and embedded controllers.

For complete stealth, the solutions of this project: (i) leverage air-gapped observations of the malware behavior through physical side channels such as PLC processor power signal produced by the on-device malicious code execution and (ii) use mathematical models and physics-informed neural networks for a realistic emulation of the physical dynamics and to provide a misleading physical process interface to the PLC input-output ports. The research outcomes will address the above-mentioned semantic gap via an automated binary reverse engineering of the malicious controller code to extract the high-level adversarial objectives from low-level controller software execution traces. This enables the classification of adversaries dynamically using online data-driven meta-learning algorithms. For more details see our published work.

The gradual deployment of self-driving cars will inevitably lead to the emergence of a new important class of cyber-physical-human systems where autonomous vehicles interact with human-driven vehicles via on-board sensors or vehicle-to-vehicle communications. Reinforcement learning along with control theory can help meet the safety requirements for real-time decision making and Level 5 autonomy in self-driving vehicles. However, it is widely known that conventional reinforcement learning policies are vulnerable to adversarial or non-adversarial perturbations to their observations, similar to adversarial examples for classifiers and/or reward (packet) drops of the learning. Such issues are exacerbated by concerns of addressing resiliency as the use of open communication and control platforms for autonomy becomes essential, and as the industry continues to invest in such systems. Decision making mechanisms, designed to incorporate agility with the help of reinforcement learning, allow self-adaptation, self-healing, and self-optimization. This research will contribute and unify the body of knowledge of several diverse fields including reinforcement learning, security, automatic control, and transportation for resilient autonomy with humans-in-the-loop.

This project will leverage proactive switching policies that aim (i) to provide robustness to adversarial inputs and reward drops in the closed-loop reinforcement learning mechanisms, (ii) to increase the cost of manipulation by deception, (iii) to limit the exposure of vulnerable actions and observations, and (iv) to provide stability, optimality, and robustness guarantees. It will contribute to future engineering curricula, pursue a substantial integration of research and education, and provide opportunities to engage students from underrepresented groups. For more details see our published work.

Substantial improvements in artificial intelligence, computing resources, and software tools have enabled tremendous capabilities to mobile robots and autonomous systems. The problem of motion planning and navigation is a core topic in robotics and autonomous vehicles, as the majority of robotic applications require safe path planning and obstacle avoidance. Ideally, a solution to this problem considers collision-free navigation in dynamic environments, computationally affordable algorithms for real-time implementation, and optimal control strategies. Safe motion planning of mobile robots in dynamic environments is a challenging task with applications in autonomous driving and robotic navigation in pedestrian environments.

This project will develop a safe, model-free, kinodynamic motion planning framework that can be implemented online. It will consist of five phases: (i) the theoretical and algorithmic formulation; (ii) the rigorous Lyapunov-based stability proof; (iii) the optimality/robustness quantification; (iv) the safety-obstacle awareness; and (v) the implementation. For more details see our published work.

Increasing wildfire costs---a reflection of climate variability and development within wildlands---drive calls for new national capabilities to manage wildfires. The great potential of unmanned aerial systems (UAS) has not yet been fully utilized in this domain due to the lack of holistic, resilient, flexible, and cost-effective monitoring protocols. This project will develop UAS-based fire management strategies to use autonomous unmanned aerial vehicles (UAVs) in an optimal, efficient, and safe way to assist the first responders during the fire detection, management, and evacuation stages. This proposal's objective is to develop an integrated framework satisfying unmet wildland fire management needs, with key advances in scientific and engineering methods by using a network of low-cost and small autonomous UAVs along with ground vehicles during different stages of fire management operations including: (i) early detection in remote and forest areas using autonomous UAVs; (ii) fast active geo-mapping of the fire heat map on flying drones; (iii) real-time video streaming of the fire spread; and (iv) finding optimal evacuation paths using autonomous UAVs to guide the ground vehicles and firefighters for fast and safe evacuation.

This project will advance the frontier of disaster management by developing: (i) an innovative drone-based forest fire detection and monitoring technology for rapid intervention in hard-to-access areas with minimal human intervention to protect firefighter lives; (ii) multi-level fire modeling to offer strategic, event-scale, and new on-board, low-computation tactics using fast fire mapping from UAVs; and (iii) a bounded reasoning-based planning mechanism where the UAVs identify the fastest and safest evacuation roads for firefighters and fire-trucks in highly dynamic and uncertain dangerous zones. The developed technologies will be translational to a broad range of applications such as disaster (flooding, fire, mud slides, terrorism) management, where quick search, surveillance, and responses are required with limited human interventions. This project will also contribute to future engineering curricula and pursue a substantial integration of research and education while also engaging female and underrepresented minority students, developing hands-on research experiments for K-12 students. For more details see our published work.

Due to the autonomous nature of learning-enabled CPS, it has become increasingly difficult and expensive to design evaluation and testing scenarios that verify the robustness of their operation and the satisfiability of the various specifications. This, in turn, hinders the development of algorithms that allow model-free learning strategies to be implemented while the system remains within given safe areas. Thus, the problem of assured (or safe) autonomy has been in the forefront of research activities in both the control and the learning communities. It has become apparent that alongside the systems themselves, evaluation strategies need to be informed by learning components as well, allowing for black-box paradigms to be introduced as solutions to the verification problem.

The objective of this project is to develop mechanisms for increased safety, security and fault tolerance of cyber-physical systems with increased autonomy capabilities. We will investigate potential faults that can occur due to the existence of learning mechanisms in the system due to the structure of those mechanisms or the opportunities of attack they provide to malicious agents. We will further analyze those faults in terms of the effects they have on the system, in order to design a complete taxonomy and hierarchy that will be integrated in the SAE AADL (Architecture Analysis and Design Language). Subsequently, we propose various detection mechanisms for faults in the learning-based components, as well as isolation rules that will be able to remove the potentially corrupted ones and maintain safe operation of the system. Finally, we will validate the proposed approaches in a unmanned aerial vehicle experimental platform. Our goal is to integrate continuous-time system-theoretic methods of learning and security with logic-based contracts that will express the safety, security and robustness specifications for the platform in a formal way. For more details see our published work.

The last few decades have witnessed uninterrupted growth in "traditional" commercial aviation, creating modernization requirements for the National Airspace System (NAS) to meet increasing needs for capacity with safety through such initiatives as NextGen. The recent introduction of unmanned systems into the NAS will bring further challenges and opportunities for the nation's aviation system. The integration of such a complex transportation system creates a clear need to develop new technologies and innovative operational concepts for secure and safe assured autonomy. An unmanned systems future will see the integration of a wide variety of Unmanned Aerial Systems (UAS), personal air vehicles, Urban Air Mobility (UAM) vehicles, and cargo and special mission aircraft into the NAS. These developments can leverage UAS Traffic Management (UTM) advancements for the unique requirements of UAM airspace management.

This vision must include robust human-machine decision-making systems for increasingly autonomous traffic management. These unmanned systems and the associated decision-making infrastructure systems must handle atmospheric, traffic, and airspace uncertainties and must operate securely and safely at different levels of system complexity. Adopting this vision will provide greater diversity and richness for the NAS; however, critical technical challenges must be addressed to ensure secure and safe cyber-physical aviation systems. The main challenges include: (i) sensing and understanding complex operational surroundings, coordinating different types of aerial vehicles, planning and navigation through highly dynamic and uncertain environments; (ii) securing the NAS against a wide range of malicious adversarial threats, specifically cyber-physical attacks; (iii) verifying and validating autonomous system operation; and, finally, (iv) properly integrating new vehicles and traffic management approaches in the midst of autonomy. Our primary goal is to ensure safe, secure and robust integration of autonomous vehicles into a UAM-tailored transportation infrastructure while maintaining compliance with existing commercial and civil air transportation safety standards. For more details see our published work.

Utility trucks are the first responders in areas of extreme climate and severe weather situations for recovering life beings from destroyed properties, for cutting trees to restore traffic, and for repairing electric posts and restoring power. It is thus crucial to create an advanced and effective scientific framework for maintaining productivity and safety while simultaneously eliminating accidents. Towards that direction, this project aims to establish a novel and integrated framework to monitor and predict weather conditions and feed such information into reinforcement learning mechanisms that will enable us to autonomously shape the utility trucks.

Unlike conventional approaches, the A-IMS will integrate model-free shape-morphing learning mechanisms with model-based interactive design to manage air-fluid flows, based on the road conditions, meteorology, speed limit, wind speed, and direction. This potentially transformative framework for A-IMS will: (1) bring new perspectives of learning to enhance smartness of natural-engineering systems that leverage physical and information processes; (2) establish an integrated design framework for hazardous environments to achieve resilience, and productivity through integrated adaptation of morphological properties while also mitigating the effects of potentially adversarial learning agents that can exist in the cloud; (3) investigate the interactive physical components of the A-IMS, that will simultaneously operate in two different mediums of multi-phase fluids, and solids (i.e., the air/fluid and road). The A-IMS framework shall be evaluated through, (i) hardware/software implementation, and in (ii) real-world conditions in the unique facilities of the Wall of Wind at the Florida International University. For more details see our published work.

The networks and information systems that are being constructed today are extremely complex. A cyberphysical attack against a network could have cascading and devastating effects on other portions of the information enterprise. To defend against a network wide attack, it is imperative that we know what is on our systems and their composition. We need to know what computers we have, what applications are running, what vulnerabilities exist and what networks are related to other networks. Only when armed with this type of information can we possibly adapt our countermeasures and protection procedures to counter an attack.

The objective of this project is to develop novel learning frameworks for non-equilibrium game-theoretic adaptive algorithms for systems in adversarial, dynamic and uncertain environments. The technical approach is to be inspired by interdisciplinary ideas from different fields, such as game theory, nonlinear adaptive learning theory, cognitive science and network security. The anticipated outcomes include the development of (i) level-k learning architectures for security, where decision makers are not perfectly rational and do not possess infinite intelligence; (ii) a learning-based moving target defense that reduces predictability; and (iii) a resilient learning framework to bandwidth depletion attacks. The contributions and impact on Army's objectives will be the ability to multiply engagement capability, and enable coordination of distributed, heterogeneous teams of manned, unmanned vehicles and humans in dynamic, adversarial and completely uncertain environments. The models, theories, techniques, and tools developed as part of the proposed research will lay the foundation for the DoD to develop more effective techniques against intelligent attackers with different levels of rationality and capabilities. For more details see our published work.

Current learning algorithms cannot be easily applied in CPS due to their need for continuous and expensive updates, with the current triggered frameworks having fundamental limitations. Such limitations lead to the following questions. How can we incorporate and fully adapt to totally unknown, dynamic, and uncertain environments? How do we co-design the action and the intermittent schemes? How can we provide quantifiable real-time performance, stability and robustness guarantees by design? And how do we solve congestion and guarantee security? We will build on our multiyear experience and develop fundamental contributions to CPS by providing novel frameworks that will allow the fully autonomous operation in the face of unknown, bandwidth restricted, and adversarial environments.

As specific merits, the project will, (i) unify new perspectives of learning in engineering to enable smart autonomy, resiliency, bandwidth efficiency, robustness, real-time optimality and adaptation that cannot be achieved with the state-of-the-art approaches; (ii) develop intermittent deep learning methods for CPS that can mitigate sensor attacks by dynamically isolating the suspicious components and can handle cases of limited sensing capabilities; (iii) incorporate nonequilibrium game-theoretic learning in CPS with components that do not share similar mechanisms for decision making and do not have the same level of rationality due to heterogeneity and may differ in either their information obtained or their ability in utilizing it; and (iv) investigate ways to transfer intermittent learning experiences among the agents. For more details see the project's webpage and our published work.

The objective of this project is to investigate and develop novel data-driven learning techniques for realtime and adaptive cyber-physical situation awareness of adversarial activities, and feed information about the impact of those activities back to the security analysts. This will significantly enhance awareness and safety in defense systems by providing fast decision making along with, guarantees for mission completion and conditions for robustness. The guaranteed robustness of the proposed data-driven learning techniques will ensure fast prompt and/or automated decision making without reaching back to higher command levels.

The technical approach is to be inspired by interdisciplinary ideas from different fields, such as game theory, cognitive learning (specifically reinforcement learning), cyber-physical systems, and network security. The anticipated outcomes include transformative data-driven learning algorithms to engage the DoD security analysts in the characterization, recognition and mitigation of adversarial actions, with the support of rapid decision-making through prompt action, and recommendations/actions from "automated assistants." For more details see our published work.

The adoption of services and applications that rely on Unmanned Aircraft Systems (UAS) is expected to continue its acceleration for the foreseeable future. Such services and applications include search and rescue, crop monitoring, real-time traffic monitoring, critical infrastructure inspections, and others. As UAS continue to proliferate, aviation regulatory agencies around the world are facing a challenging dilemma in terms of safely integrating UAS operations into their respective national airspace systems and enforcing their compliance to relevant regulations, including enforcement of no fly-zones. Current UAS lack effective on-board flight restrictions that would prevent them from entering critical airspace, e.g., airport runways and nuclear power plants. Furthermore, it is essential that the safety and privacy of members of the public be safeguarded from accidental and non-accidental UAS incursions. This project aims to address these critical needs by developing offensive and defensive measures for detecting, identifying, and bringing down UAS that violate controlled/restricted airspace in an automated, controlled, and reliable manner.

A three phase approach to thwarting a UAS encroaching on restricted airspace will be used: (1) detect the incursion using passive radar that exploits signals of opportunity and radio frequency (RF) communications/emissions from the encroacher itself; (2) deploy one or more defender UAS, capable of learning the motivations and behaviors of the encroacher, to pursue the UAS and gain and maintain proximity to it even should the encroacher take evasive action; and (3) have the defender UAS launch offensive measures that manipulate/compromise the sensor and actuation signals the encroacher relies on to stay airborne. For more details see our published work.

Extremely fast charging (XFC) has the potential to reduce the charging time of battery electric vehicles (BEV) to be equivalent to the filling time of internal combustion engine vehicles (ICEV), thus eliminating one of the few advantages ICEV still posses for light- and heavy-duty vehicles. Enabling XFC will, however, require coordination and cooperation between the grid, charging stations, and the vehicles themselves, which leads to an inevitable increase in the attack surface for all systems combined.

In securing the overall system, we must not only embrace traditional cybersecurity, which is chiefly concerned with communications and the operation of digital systems, but also cyber-physical systems security as the proper operation of XFC is critically dependent on systems' abilities to know about (sense) and interact with (actuate) the physical world. For more details see our published work.

The proposed project's effort is to develop and demonstrate a new technology to radically enhance vehicle tactical and operational mobility and survivability by establishing agile and interactive dynamics of innovative vehicle sub-systems. The scope of the proposed project is fundamental and is based on applied research in the multi-disciplinary mechatronics areas including ground vehicle engineering, sensor design and application, electronics and controls.

The new technology will enable vehicles of any weight class, with any number of wheels, independently driven wheels, or through a drive-train a unique ability to maintain mobility through high-speed and rapid maneuvers in complex off-road environments. Ramifications from this technology will greatly increase crew safety, reduce roll-overs, and enables vehicles too rapidly and aggressively veer/steer or experience external blast/force in on-road, off-road, and poor terrain conditions while maintaining vehicle control and mobility. The proposed system is a vehicle by-wire sub-system that controls and monitors the vehicle's mobility based on input programs of the motions via any autonomous controller or human operator input. For more details see our published work.

The convergence of systems on modern UUVs and the reliance of the industry on connectivity leads to a broad space for exploitable vulnerabilities. Unfortunately, due to current re-certification practices, software and/or hardware fixes to address such vulnerabilities could take months to appear. The goal of the proposed research is to increase the security of underwater cyber-physical systems (UCPS), which include the UUVs themselves and necessary supporting infrastructure, through reductions in attack surface, decreased time to patch, and increased resiliency to attacks, thus ensuring safety, integrity, and efficiency.

The overall objectives of this project are to assess cybersecurity risks for UUVs and to propose practical countermeasures. For more details see our published work.

Adaptive control and optimal control represent different philosophies for designing feedback control systems. These methods have been developed by the Control Systems Community of engineers. Optimal controllers minimize user prescribed performance functions and are normally designed offline by solving Hamilton-Jacobi (HJ) design equations, for example, the Riccati equation, using complete knowledge of the system dynamical model. However, it is often difficult to determine an accurate dynamical model of practical systems.

We propose new techniques based on approximate dynamic programming that allow the design of adaptive control systems with novel structures that learn the solutions to optimal control problems and Nash/Stackelberg games in real time by observing data along the system trajectories. Complicated Lyapunov stability proofs ensure boundedness of the closed-loop signals. For more details see our published work.