Hi, I'm Fazl Barez

AI safety and Interpretability researcher.

I’m a Research Fellow at the University of Oxford, leading research in AI safety and interpretability. Through collaboration across academia, AGI labs, AI Safety Institutes, government organizations and public sector, I work to accelerate progress in AI safety research and translate technical insights into practical solutions that benefit society. My research has helped shape both academic directions and industry safety practices.

My academic affiliations include the Centre for the Study of Existential Risk (CSER) and Kruger AI Safety Lab (KASL) at University of Cambridge, the Digital Trust Centre at Nanyang Technological University, and the School of Informatics at University of Edinburgh. As a member of ELLIS European Laboratory for Learning and Intelligent Systems (ELLIS), I contribute to advancing AI safety research across Europe. I also serve as a Research consultant at Anthropic.

Prior to my current roles, I served as Co-Director and Advisor at Apart Research, Technology and Security Policy Fellow at RAND Corporation, developed interpretability solutions at Amazon and The DataLab, built recommender systems at Huawei, and created financial forecasting tools at RBS Group.

Nov 20, 2024	Excited to speak about unlearning and Safety at the How To Evaluate AI Privacy Tutorial at NeurIPS 2024!🍁🇨🇦
Sep 20, 2024	Our paper Interpreting LFPs in Large Language Models has been accepted at NeurIPS 2024! See you in Vancouver 🍁🇨🇦
Sep 20, 2024	Two papers accepted at EMNLP 2024! See you in Miami ❤️
Aug 11, 2024	Excited to be presenting our work on LLMs Relearn Removed Concepts at ACL 2024! See you all in Bangkok 🇹🇭!
Aug 10, 2024	Co-organised the first workshop on Mechanistic Interpretability at ICML 2024! 🎉

Reinforcement learning from human feedback (RLHF) is widely used to train large language models (LLMs). However, it is unclear whether LLMs accurately learn the underlying preferences in human feedback data. We coin the term Learned Feedback Pattern (LFP) for patterns in an LLM’s activations learned during RLHF that improve its performance on the fine-tuning task. We hypothesize that LLMs with LFPs accurately aligned to the fine-tuning feedback exhibit consistent activation patterns for outputs that would have received similar feedback during RLHF. To test this, we train probes to estimate the feedback signal implicit in the activations of a fine-tuned LLM. We then compare these estimates to the true feedback, measuring how accurate the LFPs are to the fine-tuning feedback. Our probes are trained on a condensed, sparse and interpretable representation of LLM activations, making it easier to correlate features of the input with our probe’s predictions. We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies as related to LFPs. Understanding LFPs can help minimize discrepancies between LLM behavior and training objectives, which is essential for the safety of LLMs.

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential risks and resulted in calls for tighter regulation, in particular from some of the major tech companies who are leading in AI development. While regulation is important, it is key that it does not put at risk the budding field of open-source Generative AI. We argue for the responsible open sourcing of generative AI models in the near and medium term. To set the stage, we first introduce an AI openness taxonomy system and apply it to 40 current large language models. We then outline differential benefits and risks of open versus closed source AI and present potential risk mitigation, ranging from best practices to calls for technical and scientific contributions. We hope that this report will add a much needed missing voice to the current public discourse on near to mid-term AI safety and other societal impact.

While transformer models exhibit strong capabilities on linguistic tasks, their complex architectures make them difficult to interpret. Recent work has aimed to reverse engineer transformer models into human-readable representations called circuits that implement algorithmic functions. We extend this research by analyzing and comparing circuits for similar sequence continuation tasks, which include increasing sequences of Arabic numerals, number words, and months. By applying circuit interpretability analysis, we identify a key sub-circuit in both GPT-2 Small and Llama-2-7B responsible for detecting sequence members and for predicting the next member in a sequence. Our analysis reveals that semantically related sequences rely on shared circuit subgraphs with analogous roles. Additionally, we show that this sub-circuit has effects on various math-related prompts, such as on intervaled circuits, Spanish number word and months continuation, and natural language word problems. Overall, documenting shared computational structures enables better model behavior predictions, identification of errors, and safer editing procedures. This mechanistic understanding of transformers is a critical step towards building more robust, aligned, and interpretable language models.

Hi, I'm Fazl Barez

Research Interests

Selected Publications

2024

2023

2021

Keynotes and Invited Talks

Invited Panelist - Dialogue on Digital Trust and Safe AI 2024: Building bridges for a safe future (Singapore)

Invited Talk - Digital Trust Centre (Singapore AI Safety Institute) - Mechanistic Interpretability for AI Safety (NTU Singapore)

Invited Talk - Digital Trust Centre (Singapore AI Safety Institute) - Unlearning and Relearning in LLMs

Invited Talk - Foresight - AGI: Safety & Security Workshop (San Francisco 2024, USA)

Invited Panelist - Mechanistic Interpretability (ICLR 2024, Austria)

Invited Talk - Technical AI Safety Conference 2024 (TAIS Conference, Japan)

Invited Talk - Introduction to AI safety: Can we remove undesired behaviour from AI? (KAIST, South Korea)

Keynote Speaker - Personalization of Generative AI Systems (EACL 2024, Malta)

Invited Talk - Interpretability for Safety and Alignment (Foresight Institute, USA)

Selected Awards and Honors