Optimal control limit policy for a Partially Observable Markov Decision Process model

Abstract

In this research, we consider the problem of determining an optimal replacement policy for stochastically deteriorating systems for which only incomplete state information is available. When the deterioration is governed by a Markov process, such a process is known as a Partially Observable Markov Decision Process, which is a generalization of a completely observable Markov Decision Process. This research investigates a three-state partially observable Markov Decision Process in which only deterioration can occur and for which the only actions possible are to replace or not to replace the machine. The goal of this research is to first prove that a control-limit policy is optimal, and then incorporate such a policy into the policy iteration algorithm given by Sondik, in order to enhance its computational efficiency. Two conditions are presented which guarantee that the optimal replacement policy can be limited to control-limit policies for the partially observable case. One condition is a slight modification of Derman's first condition, and the other one is the same as Derman's second condition. A solution algorithm which adopts the basic idea of Sondik's policy iteration algorithm is proposed. Finally, computational comparisons are carried out to demonstrate the efficiency of the proposed algorithm.