Tongyao Zhu

I am a PhD student in the National University of Singapore (NUS) WING Lab and Sea AI Lab. I am fortunately supervised by Prof Kan Min-Yen from NUS and Dr Liu Qian.

I am generally interested in (1) how knowledge is acquired, processed, and used by language models (2) the intersection between information retrieval and LLMs 🤔 and (3) how to make pretraining more efficient and effective.

Prior to my PhD study, I graduated with my Bachelor’s degree in Computer Science from NUS in 2021, after which I worked as a machine learning engineer at the chatbot team in Shopee for one and half years. I enjoy mountain hiking ⛰️, playing badminton 🏸️, and classical music 🎵.

🔥 News

• [2025-03] 🎉🎉 Our work “SkyLadder: Better and Faster Pretraining via Context Window Scheduling” has been accepted by ICLR 2025 Sci-FM Workshop! Checkout the paper and the code.
• [2024-05] 🎉 Our work “Beyond Memorization: The Challenge of Random Memory Access in Language Models” has been accepted by ACL 2024 (oral)!
• [2024-03] ⬆️ I passed my PhD Qualifying Examination, and become a PhD candidate!
• [2023-01] 🏫 I am back to the National University of Singapore (NUS) as a PhD student.

📖 Publications

Tongyao Zhu, Qian Liu, Haonan Wang, Shiqi Chen, Xiangming Gu, Tianyu Pang, Min-Yen Kan (2025). SkyLadder: Better and Faster Pretraining via Context Window Scheduling. (ICLR 2025 Sci-FM Workshop)

Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, Tianyu Pang (2024). When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training. (TMLR)

Tongyao Zhu, Qian Liu, Liang Pang, Zhengbao Jiang, Min-Yen Kan, Min Lin (2024). Beyond Memorization: The Challenge of Random Memory Access in Language Models. (ACL 2024, oral)

Lizi Liao, Tongyao Zhu, Le Hong Long, Tat Seng Chua (2021). Multi-domain Dialogue State Tracking with Recursive Inference. (WWW 2021)

📄 Preprints

Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, Manling Li (2025). Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas

Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kydlíček, Zeyi Liu, Qunshu Lin, Sittipong Sripaisarnmongkol, Kridtaphad Sae-Khow, Nirattisai Thongchim, Taechawat Konkaew, Narong Borijindargoon, Anh Dao, Matichon Maneegard, Phakphum Artkaew, Zheng-Xin Yong, Quan Nguyen, Wannaphong Phatthiyaphaibun, Hoang H Tran, Mike Zhang, Shiqi Chen, Tianyu Pang, Chao Du, Xinyi Wan, Wei Lu, Min Lin (2025). Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

Yixi Ding, Jiaying Wu, Tongyao Zhu, Yanxia Qin, Qian Liu, Min-Yen Kan (2024). CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization

Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, Harold Soh (2023). Translating Natural Language to Planning Goals with Large-Language Models.

🏆 Awards

• 2019-2021: USP Honour Roll, Senior Honour Roll, President’s Honour Roll, University Scholars Programme
• 2019, 2020: Dean’s List, School of Computing, NUS
• 2016: NUS Science & Technology Undergraduate Scholarship (SM2)

🎓 Education

• 2023.1-Present: Ph.D in Computer Science, National University of Singapore
• 2017.8-2021.5: Bachelor of Computing (Highest Distinction) in CS, National University of Singapore