In this episode, we dive into the fascinating world of Vision Language Models (VLMs) and their potential to revolutionize how AI interacts with complex environments, like video games. Our focus is on a groundbreaking study that explores how an AI agent can learn to play the action-packed role-playing game Black Myth: Wukong using only visual input. The researchers behind this work developed a new framework, VARP, which allows the AI to navigate the game's challenges by mimicking human actions and planning its own strategies. Amazingly, this AI was able to master 90% of the game's easy and medium-level combat tasks! We’ll discuss how the framework works, the importance of the human gameplay data they’ve shared, and what this means for the future of AI in gaming and beyond. Tune in to learn how this research could shape the next generation of intelligent agents in complex, visually-rich environments!

Original paper:

Chen, P., Bu, P., Song, J., Gao, Y., & Zheng, B. (2024). Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case. https://arxiv.org/abs/2409.12889

Demo video: https://varp-agent.github.io/