Hi, I'm Fazl Barez

AI safety and Interpretability researcher.

I’m a Research Fellow at Torr Vision Group (TVG), University of Oxford, where I lead safety research. Additionally, I’m a Research Scientist (consultant) at Anthropic and advisor at Apart Research. My affiliations include two University of Cambridge institutions: the Centre for the Study of Existential Risk (CSER) and the Kruger AI Safety Lab (KASL), as well as the Future of Life Institute. I’m a member of ELLIS (European Laboratory for Learning and Intelligent Systems), a pan-European AI network of excellence. I also serve as an Adjunct Research Fellow at the Digital Trust centre (Singapore AI Safety Institute) at Nanyang Technological University, and a Visiting Researcher at the School of Informatics, University of Edinburgh.

Previously, I worked as Technology and Security Policy Fellow at RAND Corporation, on Interpretability at Amazon and The DataLab, recommender systems at Huawei, and on building a finance tool for budget management for economic scenario forecasting at RBS Group.

Sep 20, 2024	Our paper Interpreting LFPs in Large Language Models has been accepted at NeurIPS 2024! See you in Vancuver 🍁🇨🇦
Sep 20, 2024	Two papers accepted at EMNLP 2024! See you in Miami ❤️
Aug 11, 2024	Excited to be presenting our work on LLMs Relearn Removed Concepts at ACL 2024! See you all in Bangkok 🇹🇭!
Aug 10, 2024	Co-organised the first workshop on Mechanistic Interpretability at ICML 2024! 🎉
Aug 9, 2024	Presented 2 main confrence and 1 workshop paper at ICML 2024! 🎉

This paper presents an in-depth analysis of a one-layer Transformer model trained for n-digit integer addition. We reveal that the model divides the task into parallel, digit-specific streams and employs distinct algorithms for different digit positions. Our study also finds that the model starts calculations late but executes them rapidly. A rare use case with high loss is identified and explained.

As artificial intelligence (AI) systems become increasingly integrated into various domains, ensuring that they align with human values becomes critical. This pa- per introduces a novel formalism to quantify the alignment between AI systems and human values, using Markov Decision Processes (MDPs) as the foundational model. We delve into the concept of values as desirable goals tied to actions and norms as behavioral guidelines, aiming to shed light on how they can be used to guide AI decisions. This framework offers a mechanism to evaluate the degree of alignment between norms and values by assessing preference changes across state transitions in a normative world. By utilizing this formalism, AI developers and ethicists can better design and evaluate AI systems to ensure they operate in har- mony with human values. The proposed methodology holds potential for a wide range of applications, from recommendation systems emphasizing well-being to autonomous vehicles prioritizing safety.

Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming. Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size. Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.

Hi, I'm Fazl Barez

Research Interests

Publications

2023

2021

Keynotes and Invited Talks

Invited Talk - Digital Trust Centre (Singapore AI Safety Institute) - Mechanistic Interpretability for AI Safety (NTU Singapore)

Invited Talk - Foresight - AGI: Safety & Security Workshop (San Francisco 2024, USA)

Invited Panelist - Mechanistic Interpretability (ICLR 2024, Austria)

Invited Talk - Technical AI Safety Conference 2024 (TAIS Conference, Japan)

Invited Talk - Introduction to AI safety: Can we remove undesired behaviour from AI? (KAIST, South Korea)

Keynote Speaker - Personalization of Generative AI Systems (EACL 2024, Malta)

Invited Talk - Interpretability for Safety and Alignment (Foresight Institute, USA)

Selected Awards and Honors