We study the continuous-time counterpart of Q-learning for reinforcement learning (RL) under the entropy-regularized,exploratory diffusion process formulation. As the conventional (big) Q-function collapses in continuous time, we consider its first-order approximation and coin the term ``(little) q-function. This function is related to the instantaneous advantage rate function as well as  the Hamiltonian. We develop a ``q-learning theory around the q-function that is independent of time discretization. We jointly characterize the q-function and value function by martingale conditions of certain stochastic processes, in both on-policy and off-policy settings. We then apply the theory to devise different actor--critic algorithms for solving underlying RL problems, depending on whether or not the density function of the Gibbs measure generated from the q-function can be computed explicitly. One of our algorithms interprets the well-known Q-learning algorithm SARSA, and another recovers a policy gradient (PG) based continuous-time  algorithm proposed in Jia and Zhou (2022). Finally, we conduct simulation experiments to compare the performance of our algorithms with those of PG-based algorithms and time-discretized conventional Q-learning algorithms. This is a joint work with Yanwei Jia. 

学术海报.pdf