I talked about two different projects that I’ve worked on at Sony AI.

1. GT Sophy A reinforcement learning agent that plays Gran Turismo at a superhuman level.

2. Value Function Decomposition for Iterative Design of Reinforcement Learning Agents By decomposing the value function into components, we can gain insight into the agent’s learning behavior and use this to improve agent design.

A guiding theme to the talk is the importance of reward design in reinforcement learning, something that has largely been ignored in the community.

On that note, I’m officially on the market for a real adult job. I’m looking for a research scientist position in any of the following areas: robots, human augmentation, neural interfaces, computational neuroscience, drug discovery, genetic editing, and synthetic biology.

Sherstan, C., MacGlashan, J., Pilarski, P. M. (2018) Generalizing Value Estimation over Timescale. FAIM Workshop: Prediction and Generative Modeling in Reinforcement Learning (PGMRL). Stockholm, Sweden. July 15.

General value functions (GVFs) are an approach to representing models of an agent’s world as a collection of predictive questions. A GVF is defined by: a policy, a prediction target, and a timescale. Traditionally predictions for a given timescale must be specified by the engineer and each timescale learned independently. Here we present γ-nets, a method for generalizing value function estimation over timescale, allowing a given GVF to be trained and queried for any fixed timescale. The key to our approach is to use timescale as one of the network inputs. The prediction target for any fixed timescale is then available at every timestep and we are free to train on any number of timescales. We present preliminary results on a simple test signal.

Learning the SR

• Dayan, P. (1993). Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Computation, 5(4), 613–624.
  
• Gehring, C. A. (2015). Approximate Linear Successor Representation. In Reinforcement Learning Decision Making. Retrieved from http://people.csail.mit.edu/gehring/publications/clement-gehring-rldm-2015.pdf
  
• White, L. M. (1996). Temporal Difference Learning: Eligibility Traces and the Successor Representation for Actions. University of Toronto.
  
• Pitis, S. (2018). Source Traces for Temporal Difference Learning. In AAAI Conference on Artificial Intelligence. New Orleans, Louisiana, USA.
  

Transfer

• Barreto, A., Dabney, W., Munos, R., Hunt, J., Schaul, T., Silver, D., & van Hasselt, H. (2017). Transfer in Reinforcement Learning with Successor Features and Generalised Policy Improvement. In Lifelong Learning: A Reinforcement Learning Approach Workshop @ICML. Sydney, Australia. 
  
• Barreto, A., Munos, R., Schaul, T., & Silver, D. (2016). Successor Features for Transfer in Reinforcement Learning. arXiv: 1606.05312.
  
• Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., … Munos, R. (2018). Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement. International Conference on Machine Learning (ICML).
  
• Zhang, J., Springenberg, J. T., Boedecker, J., & Burgard, W. (2017). Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments. In The International Conference on Intelligent Robots and Systems (IROS) (pp. 2371–2378). Vancouver, Canada.
  
• Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L., Gupta, A., … Farhadi, A. (2017). Visual Semantic Planning using Deep Successor Representations. International Conference on Computer Vision, 2(4), 7.
  
• Sherstan, C., Machado, M. C., & Pilarski, P. M. (2018). Accelerating Learning in Constructive Predictive Frameworks with the Successor Representation. IROS. Madrid, Spain.
  
• Kulkarni, T. D., Saeedi, A., Gautam, S., & Gershman, S. J. (2016). Deep Successor Reinforcement Learning. arXiv: 1606.02396.
  
• Lehnert, L., Tellex, S., & Littman, M. L. (2017). Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning. arXiv: 1708.00102
  
• Ma, C., Wen, J., & Bengio, Y. (2018). Universal Successor Representations for Transfer Reinforcement Learning. arXiv: 1804.03758.
  

Exploration

• Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., & Campbell, M. (2018). Eigenoption Discovery Through The Deep Successor Representation. In International Conference on Learning Representations. Vancouver, Canada.
  

Neuroscience

• Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017). The Hippocampus as a Predictive Map. Nature Neuroscience, 20, 1643–1653. 
  
• Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2014). Design Principles of the Hippocampal Cognitive Map. Advances in Neural Information Processing Systems, 1–9.
  
• Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017). Predictive Representations Can Link Model-based Reinforcement Learning to Model - free Mechanisms. PLoS Computational Biology, 13(9), 1–42.
  
• Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012). The Successor Representation and Temporal Context. Neural Computation, 24(6), 1553–1568. 
  
• Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., … Kumaran, D. (2018). Vector-based Navigation using Grid-like Representations in Artificial Agents. Nature, 26.
  
• Ducarouge, A., & Sigaud, O. (2017). The Successor Representation as a Model of Behavioural Flexibility. ?
  
• Foster, D. J., Morris, R. G. M., & Dayan, P. (2000). A Model of Hippocampally Dependent Navigation, Using the Temporal Difference Learning Rule. Hippocampus, 10(1), 1–16.
  
• Gershman, S. J. (2017). Predicting the past, remembering the future. Current Opinion in Behavioral Sciences, 17, 7–13.
  
• Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017). The Successor Representation in Human Reinforcement Learning. Nature Human Behaviour, 1(9), 680–692.
  

Other

There is also a handful of literature that does not necessarily directly make the connection to the SR but uses it nonetheless.

• Yao, H., Szepesvári, C., Sutton, R., Modayil, J., & Bhatnagar, S. (2014). Universal Option Models. Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NIPS), 1–9.
  

There are a number of papers in the field of imitation learning that may be related:

• Apprenticeship Learning Using Linear Programming - Syed, Bowling & Schapire, ICML ‘08 
  
• Abbeel, P., Ng. A. Y. (2004). Apprenticeship Learning Via Inverse Reinforcement Learning. ICML
  
• Syed, U. & Schapire, R. A. (2007). Game-Theoretic Approach to Apprenticeship Learning. NIPS
  

I’ve got a new paper with Marlos C. Machado and Patrick M. Pilarski on using the successor representation to speed up learning of incrementally added general value functions.

I’ve got a new paper on arXiv on using TD methods to directly estimate the variance of the lambda-return.