Publications

You can also find my articles on my Scholar Profile.

Transformers Use Causal World Models in Maze-Solving Tasks

Spies, A.F., Edwards, W., Ivanitskiy, M.I., Skapars, A., Räuker, T., Inoue, K., Russo, A., Shanahan, M.

ICLR 2025 World Models Workshop 2024

We identify World Models in transformers trained on maze-solving tasks, using Sparse Autoencoders and attention pattern analysis to examine their construction and demonstrate consistency between feature-based and circuit-based analyses.

PDF

@inproceedings{spies2024transformers,
  title={Transformers Use Causal World Models in Maze-Solving Tasks},
  author={Spies, Alex F. and Edwards, William and Ivanitskiy, Mikhail I. and Skapars, Adrians and Räuker, Tilman and Inoue, Katsumi and Russo, Alessandra and Shanahan, Murray},
  booktitle={ICLR 2025 World Models Workshop},
  year={2024}
}

Figure from Transformers Use Causal World Models in Maze-Solving Tasks

Structured World Representations in Maze-Solving Transformers

Ivanitskiy, M.I.*, Spies, A.F.*, Räuker, T.*

NeurIPS 2023 UniReps Workshop 2023

Transformers trained to solve mazes form linear representations of maze structure, and acquire interpretable attention heads which facilitate path-following.

PDF Poster GitHub

@InProceedings{pmlr-v243-ivanitskiy24a,
 title = {Linearly Structured World Representations in Maze-Solving Transformers},
 author = {Ivanitskiy, Michael and Spies, Alexander F. and Räuker, Tilman and Corlouer, Guillaume and Mathwin, Christopher and Quirke, Lucia and Rager, Can and Shah, Rusheb and Valentine, Dan and Behn, Cecilia Diniz and Inoue, Katsumi and Fung, Samy Wu},
 booktitle = {Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models},
 pages = {133--143},
 year = {2024},
 editor = {Fumero, Marco and Rodolá, Emanuele and Domine, Clementine and Locatello, Francesco and Dziugaite, Karolina and Mathilde, Caron},
 volume = {243},
 series = {Proceedings of Machine Learning Research},
 month = {15 Dec},
 publisher = {PMLR},
 pdf = {https://proceedings.mlr.press/v243/ivanitskiy24a/ivanitskiy24a.pdf},
 url = {https://proceedings.mlr.press/v243/ivanitskiy24a.html}
}

Figure from Structured World Representations in Maze-Solving Transformers

A Configurable Library for Generating and Manipulating Maze Datasets

Ivanitskiy, M.I., Shah, R., Spies, A.F.

arXiv preprint 2023

A dataset to generate simple Maze-like environments for use with Transformers.

PDF GitHub

@misc{maze-dataset,
 title={A Configurable Library for Generating and Manipulating Maze Datasets}, 
 author={Michael Igorevich Ivanitskiy and Rusheb Shah and Alex F. Spies and Tilman Räuker and Dan Valentine and Can Rager and Lucia Quirke and Chris Mathwin and Guillaume Corlouer and Cecilia Diniz Behn and Samy Wu Fung},
 year={2023},
 eprint={2309.10498},
 archivePrefix={arXiv},
 primaryClass={cs.LG},
 url={http://arxiv.org/abs/2309.10498}
}

Figure from A Configurable Library for Generating and Manipulating Maze Datasets

Sparse Relational Reasoning with Object-Centric Representations

Spies, A.F., Russo, A., Shanahan, M.

ICML 2023 DyNN Workshop (Spotlight) 2022

Assessing the extent to which sparsity and structured (Obect-Centric) representations are beneficial for neural relational reasoning.

PDF Talk Poster

@inproceedings{spies2022sparse,
  title={Sparse Relational Reasoning with Object-Centric Representations},
  author={Spies, Alexander F. and Russo, Alessandra and Shanahan, Murray},
  booktitle={ICML 2023 DyNN Workshop},
  year={2022}
}

Figure from Sparse Relational Reasoning with Object-Centric Representations

Nonlocal Thresholds for Improving the Spatial Resolution of Pixel Detectors

Nachman, B., Spies, A.F.

Journal of Instrumentation 2019

Investigating the potential use of charge sharing between neighboring pixels in HEP sensors to increase resolution and radiation hardness.

PDF

@article{Nachman_2019,
 doi = {10.1088/1748-0221/14/09/P09028},
 url = {https://dx.doi.org/10.1088/1748-0221/14/09/P09028},
 year = {2019},
 month = {sep},
 publisher = {},
 volume = {14},
 number = {09},
 pages = {P09028},
 author = {Nachman, B. and Spies, A.F.},
 title = {Nonlocal thresholds for improving the  spatial resolution of pixel detectors}, journal = {Journal of Instrumentation}, url={https://iopscience.iop.org/article/10.1088/1748-0221/14/09/P09028}
}