Li Meng

• Meng, Li; Goodwin, Morten; Yazidi, Anis & Engelstad, Paal E. (2023). Unsupervised State Representation Learning in Partially Observable Atari Games. In Tsapatsoulis, Nicolas (Eds.), Computer Analysis of Images and Patterns. CAIP 2023. Springer. ISSN 9783031442391. p. 212–222. doi: 10.1007/978-3-031-44240-7_21. Full text in Research Archive Show summary

  State representation learning aims to capture latent factors of an environment. Although some researchers realize the connections between masked image modeling and contrastive representation learning, the effort is focused on using masks as an augmentation technique to represent the latent generative factors better. Partially observable environments in reinforcement learning have not yet been carefully studied using unsupervised state representation learning methods. In this article, we create an unsupervised state representation learning scheme for partially observable states. We conducted our experiment on a previous Atari 2600 framework designed to evaluate representation learning models. A contrastive method called Spatiotemporal DeepInfomax (ST-DIM) has shown state-of-the-art performance on this benchmark but remains inferior to its supervised counterpart. Our approach improves ST-DIM when the environment is not fully observable and achieves higher F1 scores and accuracy scores than the supervised learning counterpart. The mean accuracy score averaged over categories of our approach is 66%, compared to 38% of supervised learning. The mean F1 score is 64% to 33%. The code can be found on https://github.com/mengli11235/MST_DIM.

• Meng, Li; Yazidi, Anis; Goodwin, Morten & Engelstad, Paal (2022). Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples. Proceedings of the Northern Lights Deep Learning Workshop. 3. doi: 10.7557/18.6237. Full text in Research Archive Show summary

  In this article, we propose a novel algorithm for deep reinforcement learning named Expert Q-learning. Expert Q-learning is inspired by Dueling Q-learning and aims to incorporate semi-supervised learning into reinforcement learning through splitting Q-values into state values and action advantages. We require that an offline expert assesses the value of a state in a coarse manner using three discrete values. An expert network is designed in addition to the Q-network, which updates each time following the regular offline minibatch update whenever the expert example buffer is not empty. Using the board game Othello, we compare our algorithm with the baseline Q-learning algorithm, which is a combination of Double Q-learning and Dueling Q-learning. Our results show that Expert Q-learning is indeed useful and more resistant to the overestimation bias. The baseline Q-learning algorithm exhibits unstable and suboptimal behavior in non-deterministic settings, whereas Expert Q-learning demonstrates more robust performance with higher scores, illustrating that our algorithm is indeed suitable to integrate state values from expert examples into Q-learning.

• Meng, Li; Goodwin, Morten; Yazidi, Anis & Engelstad, Paal (2022). Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise. IEEE Transactions on Games (TG). ISSN 2475-1502. 15(4), p. 580–589. doi: 10.1109/TG.2022.3185330. Full text in Research Archive Show summary

  Q-learning is one of the most well-known Reinforcement Learning algorithms. There have been tremendous efforts to develop this algorithm using neural networks. Bootstrapped Deep Q-Learning Network is amongst them. It utilizes multiple neural network heads to introduce diversity into Q-learning. Diversity can sometimes be viewed as the amount of reasonable moves an agent can take at a given state, analogous to the definition of the exploration ratio in RL. Thus, the performance of Bootstrapped Deep Q-Learning Network is deeply connected with the level of diversity within the algorithm. In the original research, it was pointed out that a random prior could improve the performance of the model. In this article, we further explore the possibility of replacing priors with noise and sample the noise from a Gaussian distribution to introduce more diversity into this algorithm. We conduct our experiment on the Atari benchmark and compare our algorithm to both the original and other related algorithms. The results show that our modification of the Bootstrapped Deep Q-Learning algorithm achieves significantly higher evaluation scores across different types of Atari games. Thus, we conclude that replacing priors with noise can improve Bootstrapped Deep Q-Learning’s performance by ensuring the integrity of diversities.

View all works in NVA