Automating Voltage Control of the Electric Transmission Grid with PowerWorld and Deep Reinforcement Learning

Abstract

This thesis applies a subset of machine learning known as deep reinforcement learning (DRL) to the problem of steady state voltage control in the electric power transmission system. In support of this work, both a new Python package that interfaces with PowerWorld Simulator and a set of modular DRL environments were developed. A state-of-the-art open-source DRL algorithm which leverages “deep Q networks” (DQN) was used along with the developed DRL environments in order to apply DRL to voltage control.

An experiment from a recent publication which applies DRL to voltage control was reproduced and its shortcomings were discussed. This led to experimentation with algorithm and environment modifications. It was found that a novel change to the DQN algorithm in which agents are not allowed to take the same action twice in any given training or testing episode leads to significant improvements. Additionally, it was found that using min-max scaled voltages in observations provided to the DRL agent rather than per unit voltages leads to marked improvements in successfully solving the voltage control problem.

After initial exploration using the IEEE 14 bus test system, use of DRL for voltage control was tested on synthetic 200 and 500 bus systems developed at Texas A&M University. Results for these systems were mixed, and instabilities were observed during training. For the 200 bus system with single line contingencies present in all training and testing episodes, DRL agents were able to achieve success near to that of a heuristically-driven graph-based agent that was developed for comparison. By contrast, with the 500 bus system the DRL agents’ success rates were approximately half that of the graph-based agents.

This thesis demonstrates that there is promise in the application of deep reinforcement learning to power system voltage control, but more research is needed in order to enable DRL-based techniques to consistently outperform conventional methods.