Successor Representation Literature

This is a list of literature related to the successor representation. It is not exhaustive and I have not read it all. Right now it is just a list; if I have time I’ll add summaries for those papers I’ve read.

Learning the SR

• Dayan, P. (1993). Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Computation, 5(4), 613–624.
  
• Gehring, C. A. (2015). Approximate Linear Successor Representation. In Reinforcement Learning Decision Making. Retrieved from http://people.csail.mit.edu/gehring/publications/clement-gehring-rldm-2015.pdf
  
• White, L. M. (1996). Temporal Difference Learning: Eligibility Traces and the Successor Representation for Actions. University of Toronto.
  
• Pitis, S. (2018). Source Traces for Temporal Difference Learning. In AAAI Conference on Artificial Intelligence. New Orleans, Louisiana, USA.
  

Transfer

• Barreto, A., Dabney, W., Munos, R., Hunt, J., Schaul, T., Silver, D., & van Hasselt, H. (2017). Transfer in Reinforcement Learning with Successor Features and Generalised Policy Improvement. In Lifelong Learning: A Reinforcement Learning Approach Workshop @ICML. Sydney, Australia. 
  
• Barreto, A., Munos, R., Schaul, T., & Silver, D. (2016). Successor Features for Transfer in Reinforcement Learning. arXiv: 1606.05312.
  
• Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., … Munos, R. (2018). Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement. International Conference on Machine Learning (ICML).
  
• Zhang, J., Springenberg, J. T., Boedecker, J., & Burgard, W. (2017). Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments. In The International Conference on Intelligent Robots and Systems (IROS) (pp. 2371–2378). Vancouver, Canada.
  
• Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L., Gupta, A., … Farhadi, A. (2017). Visual Semantic Planning using Deep Successor Representations. International Conference on Computer Vision, 2(4), 7.
  
• Sherstan, C., Machado, M. C., & Pilarski, P. M. (2018). Accelerating Learning in Constructive Predictive Frameworks with the Successor Representation. IROS. Madrid, Spain.
  
• Kulkarni, T. D., Saeedi, A., Gautam, S., & Gershman, S. J. (2016). Deep Successor Reinforcement Learning. arXiv: 1606.02396.
  
• Lehnert, L., Tellex, S., & Littman, M. L. (2017). Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning. arXiv: 1708.00102
  
• Ma, C., Wen, J., & Bengio, Y. (2018). Universal Successor Representations for Transfer Reinforcement Learning. arXiv: 1804.03758.
  

Exploration

• Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., & Campbell, M. (2018). Eigenoption Discovery Through The Deep Successor Representation. In International Conference on Learning Representations. Vancouver, Canada.
  

Neuroscience

• Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017). The Hippocampus as a Predictive Map. Nature Neuroscience, 20, 1643–1653. 
  
• Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2014). Design Principles of the Hippocampal Cognitive Map. Advances in Neural Information Processing Systems, 1–9.
  
• Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017). Predictive Representations Can Link Model-based Reinforcement Learning to Model - free Mechanisms. PLoS Computational Biology, 13(9), 1–42.
  
• Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012). The Successor Representation and Temporal Context. Neural Computation, 24(6), 1553–1568. 
  
• Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., … Kumaran, D. (2018). Vector-based Navigation using Grid-like Representations in Artificial Agents. Nature, 26.
  
• Ducarouge, A., & Sigaud, O. (2017). The Successor Representation as a Model of Behavioural Flexibility. ?
  
• Foster, D. J., Morris, R. G. M., & Dayan, P. (2000). A Model of Hippocampally Dependent Navigation, Using the Temporal Difference Learning Rule. Hippocampus, 10(1), 1–16.
  
• Gershman, S. J. (2017). Predicting the past, remembering the future. Current Opinion in Behavioral Sciences, 17, 7–13.
  
• Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017). The Successor Representation in Human Reinforcement Learning. Nature Human Behaviour, 1(9), 680–692.
  

Other

There is also a handful of literature that does not necessarily directly make the connection to the SR but uses it nonetheless.

• Yao, H., Szepesvári, C., Sutton, R., Modayil, J., & Bhatnagar, S. (2014). Universal Option Models. Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NIPS), 1–9.
  

There are a number of papers in the field of imitation learning that may be related:

• Apprenticeship Learning Using Linear Programming - Syed, Bowling & Schapire, ICML ‘08 
  
• Abbeel, P., Ng. A. Y. (2004). Apprenticeship Learning Via Inverse Reinforcement Learning. ICML
  
• Syed, U. & Schapire, R. A. (2007). Game-Theoretic Approach to Apprenticeship Learning. NIPS