Mastering the game of Go with deep neural networks and tree search

企业开发 2018-05-24 20:41:11 阅读次数: 2

Mastering the game of Go with deep neural networks and tree search

Received:

11 November 2015

Accepted:

05 January 2016

Published:

27 January 2016

Abstract

The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of state-of-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.

  • Subscribe to Nature for full access:

    $199

    Subscribe

  • READCUBE ACCESS:

    $8.99rent

    $22.00buy

    Buy/Rent now

Additional access options:

Already a subscriber?  Log in  now or  Register  for online access.

References

  1. 1.

    Allis, L. V. Searching for Solutions in Games and Artificial Intelligence. PhD thesis, Univ. Limburg, Maastricht, The Netherlands (1994)

  2. 2.

    van den Herik, H., Uiterwijk, J. W. & van Rijswijck, J. Games solved: now and in the future. Artif. Intell. 134, 277–311 (2002)

  3. 3.

    Schaeffer, J. The games computers (and people) play. Advances in Computers 52, 189–266 (2000)

  4. 4.

    Campbell, M., Hoane, A. & Hsu, F. Deep Blue. Artif. Intell. 134, 57–83 (2002)

  5. 5.

    Schaeffer, J. et al. A world championship caliber checkers program. Artif. Intell. 53, 273–289 (1992)

  6. 6.

    Buro, M. From simple features to sophisticated evaluation functions. In 1st International Conference on Computers and Games, 126–145 (1999)

  7. 7.

    Müller, M. Computer Go. Artif. Intell. 134, 145–179 (2002)

  8. 8.

    Tesauro, G. & Galperin, G. On-line policy improvement using Monte-Carlo search. In Advances in Neural Information Processing, 1068–1074 (1996)

  9. 9.

    Sheppard, B. World-championship-caliber Scrabble. Artif. Intell.134, 241–275 (2002)

  10. 10.

    Bouzy, B. & Helmstetter, B. Monte-Carlo Go developments. In10th International Conference on Advances in Computer Games, 159–174 (2003)

  11. 11.

    Coulom, R. Efficient selectivity and backup operators in Monte-Carlo tree search. In 5th International Conference on Computers and Games, 72–83 (2006)

  12. 12.

    Kocsis, L. & Szepesvári, C. Bandit based Monte-Carlo planning. In15th European Conference on Machine Learning, 282–293 (2006)

  13. 13.

    Coulom, R. Computing Elo ratings of move patterns in the game of Go. ICGA J. 30, 198–208 (2007)

  14. 14.

    Baudiš, P. & Gailly, J.-L. Pachi: State of the art open source Go program. In Advances in Computer Games, 24–38 (Springer, 2012)

  15. 15.

    Müller, M., Enzenberger, M., Arneson, B. & Segal, R. Fuego – an open-source framework for board games and Go engine based on Monte-Carlo tree search. IEEE Trans. Comput. Intell. AI in Games2, 259–270 (2010)

  16. 16.

    Gelly, S. & Silver, D. Combining online and offline learning in UCT. In 17th International Conference on Machine Learning, 273–280 (2007)

  17. 17.

    Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 1097–1105 (2012)

  18. 18.

    Lawrence, S., Giles, C. L., Tsoi, A. C. & Back, A. D. Face recognition: a convolutional neural-network approach. IEEE Trans. Neural Netw. 8, 98–113 (1997)

  19. 19.

    Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

  20. 20.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015)

  21. 21.

    Stern, D., Herbrich, R. & Graepel, T. Bayesian pattern ranking for move prediction in the game of Go. In International Conference of Machine Learning, 873–880 (2006)

  22. 22.

    Sutskever, I. & Nair, V. Mimicking Go experts with convolutional neural networks. In International Conference on Artificial Neural Networks, 101–110 (2008)

  23. 23.

    Maddison, C. J., Huang, A., Sutskever, I. & Silver, D. Move evaluation in Go using deep convolutional neural networks. 3rd International Conference on Learning Representations (2015)

  24. 24.

    Clark, C. & Storkey, A. J. Training deep convolutional neural networks to play go. In 32nd International Conference on Machine Learning, 1766–1774 (2015)

  25. 25.

    Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)

  26. 26.

    Sutton, R., McAllester, D., Singh, S. & Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, 1057–1063 (2000)

  27. 27.

    Sutton, R. & Barto, A. Reinforcement Learning: an Introduction(MIT Press, 1998)

  28. 28.

    Schraudolph, N. N., Dayan, P. & Sejnowski, T. J. Temporal difference learning of position evaluation in the game of Go. Adv. Neural Inf. Process. Syst. 6, 817–824 (1994)

  29. 29.

    Enzenberger, M. Evaluation in Go by a neural network using soft segmentation. In 10th Advances in Computer Games Conference, 97–108 (2003). 267

  30. 30.

    Silver, D., Sutton, R. & Müller, M. Temporal-difference search in computer Go. Mach. Learn. 87, 183–219 (2012)

  31. 31.

    Levinovitz, A. The mystery of Go, the ancient game that computers still can’t win. Wired Magazine (2014)

  32. 32.

    Mechner, D. All Systems Go. The Sciences 38, 32–37 (1998)

  33. 33.

    Mandziuk, J. Computational intelligence in mind games. InChallenges for Computational Intelligence, 407–442 (2007)

  34. 34.

    Berliner, H. A chronology of computer chess and its literature.Artif. Intell. 10, 201–214 (1978)

  35. 35.

    Browne, C. et al. A survey of Monte-Carlo tree search methods.IEEE Trans. Comput. Intell. AI in Games 4, 1–43 (2012)

  36. 36.

    Gelly, S. et al. The grand challenge of computer Go: Monte Carlo tree search and extensions. Commun. ACM 55, 106–113 (2012)

  37. 37.

    Coulom, R. Whole-history rating: A Bayesian rating system for players of time-varying strength. In International Conference on Computers and Games, 113–124 (2008)

  38. 38.

    KGS. Rating system math.http://www.gokgs.com/help/rmath.html

  39. 39.

    Littman, M. L. Markov games as a framework for multi-agent reinforcement learning. In 11th International Conference on Machine Learning, 157–163 (1994)

  40. 40.

    Knuth, D. E. & Moore, R. W. An analysis of alpha-beta pruning.Artif. Intell. 6, 293–326 (1975)

  41. 41.

    Sutton, R. Learning to predict by the method of temporal differences. Mach. Learn. 3, 9–44 (1988)

  42. 42.

    Baxter, J., Tridgell, A. & Weaver, L. Learning to play chess using temporal differences. Mach. Learn. 40, 243–263 (2000)

  43. 43.

    Veness, J., Silver, D., Blair, A. & Uther, W. Bootstrapping from game tree search. In Advances in Neural Information Processing Systems (2009)

  44. 44.

    Samuel, A. L. Some studies in machine learning using the game of checkers II - recent progress. IBM J. Res. Develop. 11, 601–617 (1967)

  45. 45.

    Schaeffer, J., Hlynka, M. & Jussila, V. Temporal difference learning applied to a high-performance game-playing program. In 17th International Joint Conference on Artificial Intelligence, 529–534 (2001)

  46. 46.

    Tesauro, G. TD-gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6, 215–219 (1994)

  47. 47.

    Dahl, F. Honte, a Go-playing program using neural nets. InMachines that learn to play games, 205–223 (Nova Science, 1999)

  48. 48.

    Rosin, C. D. Multi-armed bandits with episode context. Ann. Math. Artif. Intell. 61, 203–230 (2011)

  49. 49.

    Lanctot, M., Winands, M. H. M., Pepels, T. & Sturtevant, N. R.Monte Carlo tree search with heuristic evaluations using implicit minimax backups. In IEEE Conference on Computational Intelligence and Games, 1–8 (2014)

  50. 50.

    Gelly, S., Wang, Y., Munos, R. & Teytaud, O. Modification of UCT with patterns in Monte-Carlo Go. Tech. Rep. 6062, INRIA (2006)

  51. 51.

    Silver, D. & Tesauro, G. Monte-Carlo simulation balancing. In 26th International Conference on Machine Learning, 119 (2009)

  52. 52.

    Huang, S.-C., Coulom, R. & Lin, S.-S. Monte-Carlo simulation balancing in practice. In 7th International Conference on Computers and Games, 81–92 (Springer-Verlag, 2011)

  53. 53.

    Baier, H. & Drake, P. D. The power of forgetting: improving the last-good-reply policy in Monte Carlo Go. IEEE Trans. Comput. Intell. AI in Games 2, 303–309 (2010)

  54. 54.

    Huang, S. & Müller, M. Investigating the limits of Monte-Carlo tree search methods in computer Go. In 8th International Conference on Computers and Games, 39–48 (2013)

  55. 55.

    Segal, R. B. On the scalability of parallel UCT. Computers and Games 6515, 36–47 (2011)

  56. 56.

    Enzenberger, M. & Müller, M. A lock-free multithreaded Monte-Carlo tree search algorithm. In 12th Advances in Computer Games Conference, 14–20 (2009)

  57. 57.

    Huang, S.-C., Coulom, R. & Lin, S.-S. Time management for Monte-Carlo tree search applied to the game of Go. InInternational Conference on Technologies and Applications of Artificial Intelligence, 462–466 (2010)

  58. 58.

    Gelly, S. & Silver, D. Monte-Carlo tree search and rapid action value estimation in computer Go. Artif. Intell. 175, 1856–1875 (2011)

  59. 59.

    Baudiš, P. Balancing MCTS by dynamically adjusting the komi value. ICGA J. 34, 131 (2011)

  60. 60.

    Baier, H. & Winands, M. H. Active opening book application for Monte-Carlo tree search in 19×19 Go. In Benelux Conference on Artificial Intelligence, 3–10 (2011)

  61. 61.

    Dean, J. et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, 1223–1231 (2012)

  62. 62.

    Go ratings. http://www.goratings.org

Download references

Acknowledgements

We thank Fan Hui for agreeing to play against AlphaGo; T. Manning for refereeing the match; R. Munos and T. Schaul for helpful discussions and advice; A. Cain and M. Cant for work on the visuals; P. Dayan, G. Wayne, D. Kumaran, D. Purves, H. van Hasselt, A. Barreto and G. Ostrovski for reviewing the paper; and the rest of the DeepMind team for their support, ideas and encouragement.

Author information

Author notes

    • David Silver
    •  & Aja Huang

    These authors contributed equally to this work.

Affiliations

  1. Google DeepMind, 5 New Street Square, London EC4A 3TW, UK

    • David Silver
    • , Aja Huang
    • , Chris J. Maddison
    • , Arthur Guez
    • , Laurent Sifre
    • ,George van den Driessche
    • , Julian Schrittwieser
    • , Ioannis Antonoglou
    • ,Veda Panneershelvam
    • , Marc Lanctot
    • , Sander Dieleman
    • , Dominik Grewe
    • ,Nal Kalchbrenner
    • , Timothy Lillicrap
    • , Madeleine Leach
    • , Koray Kavukcuoglu
    • , Thore Graepel
    •  & Demis Hassabis

  2. Google, 1600 Amphitheatre Parkway, Mountain View, California 94043, USA

    • John Nham
    •  & Ilya Sutskever

Contributions

A.H., G.v.d.D., J.S., I.A., M.La., A.G., T.G. and D.S. designed and implemented the search in AlphaGo. C.J.M., A.G., L.S., A.H., I.A., V.P., S.D., D.G., N.K., I.S., K.K. and D.S. designed and trained the neural networks in AlphaGo. J.S., J.N., A.H. and D.S. designed and implemented the evaluation framework for AlphaGo. D.S., M.Le., T.L., T.G., K.K. and D.H. managed and advised on the project. D.S., T.G., A.G. and D.H. wrote the paper.

Competing interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to David Silver or Demis Hassabis.

Extended data

Extended data tables

  1. 1.

    Details of match between AlphaGo and Fan Hui

  2. 2.

    Input features for neural networks

  3. 3.

    Supervised learning results for the policy network

  4. 4.

    Input features for rollout and tree policy

  5. 5.

    Parameters used by AlphaGo

  6. 6.

    Results of a tournament between different Go programs

  7. 7.

    Results of a tournament between different variants of AlphaGo

  8. 8.

    Results of a tournament between AlphaGo and distributed AlphaGo, testing scalability with hardware

  9. 9.

    Cross-table of win rates in per cent between programs

  10. 10.

    Cross-table of win rates in per cent between programs in the single-machine scalability study

  11. 11.

    Cross-table of win rates in per cent between programs in the distributed scalability study

Supplementary information

Zip files

  1. 1.

    Supplementary Information

    This zipped file contains game records for the 5 formal match games played between AlphaGo and Fan Hui.

Rights and permissions

To obtain permission to re-use content from this article visitRightsLink.

Comments

By submitting a comment you agree to abide by our Terms andCommunity Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

猜你喜欢

转载自my.oschina.net/u/3797187/blog/1818286
今日推荐
基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
周排行
BPM为企业带来的实际利益
好程序员web前端分享css常用属性缩写
Java文件下载（excel）
css样式的动态添加及显示和隐藏等零碎用法
axios全局配置以及拦截器
使用Logstash来实时同步MySQL和log日志数据到ES
C++获取当前时间（年月日、时分秒、毫秒）
Odoo产品分析 (四) -- 工具板块(11) -- 网站即时聊天(1)
Java环境配置正确，但是java、javac、java -version均返回“不是内部或外部命令，也不是可运行的程序或批处理文件”？
01 官网下载各种CentOS教程（超详细版）
每日归档
更多
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022