Reinforcement Learning Based Decision Making for Self Driving & Shared Control Between Human Driver and Machine
Abstract
This study presents solutions to decision making for autonomous driving based on reinforcement learning and shared control between human driver and machine. The main objective of this research is to propose decision making models in highway driving and to study how to share control authority between two controllers which are human driver and machine in the vehicle control loop. Therefore, the research consists of two sub-topics 1) shared control between human driver and machine, and 2) reinforcement learning based decision making model.
First of all, for shared control between human driver and machine, game theoretical model predictive control (MPC) approach is studied. Four game frameworks - non-cooperative game with the simultaneous move, non-cooperative game with leader and follower, non-cooperative game with sequential move, and cooperative game - are investigated, and then several driving situations are studied under different game frameworks. Shared control strategy for fully mixed driving authority is proposed considering collision probability based on Time-to-Collision (TTC) and the weighted square sum of tracking error. Also, game transition between different game frameworks is studied in consideration of driving situations. Simulations for cooperative driving and inter-game transition driving were conducted and the simulation results show that the control authority is shared continuously by the proposed shared control strategy based on collision probability, and realistic driving with game transition is studied and analyzed from the simulation results.
For decision making models in highway driving, we develop Deep Reinforcement Learning (DRL) based decision making models. The problem is formulated as a Reinforcement Learning (RL) problem with three different types of state definitions, and several Deep Q Network (DQN) based RL approaches are applied to design decision makers for highway driving. The three different types of state definitions are 1) relative maneuvers based state with respect to surrounding vehicles, 2) surrounding inter-vehicles gap based state, 3) occupied grid based state (image-like state definition with three channels). For the highway driving decision making problem, designed DQN based algorithms show similar performances with three different types of state definitions. To verify the performance of RL based decision model, its performance is compared with the performances of other decision models which are the conventional human driver behavior model, rule based decision model, and data-driven decision model. The mean velocities and moving distances from highway driving simulations are compared and analyzed to compare their performances, and RL based decision model shows the best performance among the decision models.
Finally, RL based decision model is applied as machine's decision model with game theoretic MPC based shared controller in several driving situations to evaluate the performance of designed RL based decision model and shared controller under a hierarchical architecture, and it is found that the proposed RL based decision model and shared controller are capable of dealing with undesired driving situations in order to prevent a collision and to guarantee vehicle safety.
Subject
Shared controlDecision making for self-driving
Game theory
Model predictive control
Reinforcement learning
Vehicle dynamics control
Citation
Ko, Sangjin (2021). Reinforcement Learning Based Decision Making for Self Driving & Shared Control Between Human Driver and Machine. Doctoral dissertation, Texas A&M University. Available electronically from https : / /hdl .handle .net /1969 .1 /195426.