Reinforcement learning for patient-specific propofol anesthesia: A human volunteer study

Classical control methods, such as the PID controller, enjoy widespread industrial application due to their simplicity in design and implementation, as well as their success in controlling a wide range of systems. However, some processes may not lend themselves to regulation via these conventional control methods. Challenging control problems may involve high-dimensional inputs, nonlinear transfer functions, nonstationary time responses, and general uncertainty – all properties that may confound conventional control techniques. In response, more robust methods, including fuzzy logic, nonlinear control, and optimal control theories, have been applied to these tasks; however, a corresponding increase in complexity in design and implementation accompanies these methods. Efforts to achieve ideal control in challenging problems have expanded to include techniques commonly associated with intelligent systems. As a result, systems employing neural networks, fuzzy control, and evolutionary systems have demonstrated good performance in difficult applications of industrial control. With limited application, these successes have also been observed in life-critical clinical control problems. For example, both classical control methods and neuro-fuzzy techniques have demonstrated clinically-relevant abilities to manage complex physiological processes. Reinforcement learning (RL), another intelligent control method, has demonstrated proficiency in difficult control tasks within the intelligent systems community. Although RL has no reported application to closed-loop control in human patients, the method appears well-suited to such problems. RL presents a mathematical framework for optimal decision making in goal-directed stochastic control problems, and the technique has shown success in time-delayed and highly-dimensional problems. As such, RL appears uniquely suited to managing the complexities of physiological control systems. To investigate the suitability of RL control to a life-critical control task, a human volunteer study was performed to evaluate closed-loop control of intravenous propofol anesthesia. Fifteen healthy human subjects were enrolled in an IRB-approved volunteer study performed at the Stanford School of Medicine, Palo Alto, CA. The volunteers underwent approximately 90 minutes of intravenous propofol anesthesia in a fully-equipped operating room under the supervision of practicing anesthesiologists. Under these conditions, reinforcement learning demonstrated clinically-suitable control of propofol anesthesia in all volunteers as determined by accepted metrics of closed-loop anesthesia literature. These observations demonstrate that RL control may be applied to challenging domains outside of intelligent systems research; furthermore, this study suggests that RL control is suitable for surgical patients, and further investigation is warranted.

This dissertation won 2nd Place in the Texas Tech University Outstanding Thesis and Dissertation Award, Mathematics, Physical Sciences & Engineering, 2010.

Embargo status: Restricted from online display. To request an access exception from the author, click on the PDF link to the left.