Hi, I’m [ˈheɪmɪʃ]! I’m a PhD student in the University of Washington’s H2Lab, advised by Hannaneh Hajishirzi, and a student researcher at Google DeepMind. My research focuses on post-training for language models: making them more useful to more people, improving them beyond next-token prediction (especially with reinforcement learning), and understanding better data mixtures. I also dabble in alternative approaches to language modelling.
I’m from Sydney and completed my undergraduate studies at the University of Sydney, earning degrees in Arts and IT with majors in Linguistics, Classical Greek, and Computer Science. I also worked with the university’s natural language processing group on multi-hop question answering. During and just after my undergraduate studies, I spent time at the Commonwealth Bank of Australia, a few startups, and Optiver. Before my PhD, I was a predoctoral researcher at AI2 on the AllenNLP team.
If you have questions about my work, academia, software, or research—or just want to chat—feel free to reach out at hamishiv [at] cs [dot] washington [dot] edu. I’m generally happy to answer questions. You can also find me as @hamishivi.
@misc{ivison2026tmax,
title = {Tmax: A simple recipe for terminal agents},
author = {Ivison*, Hamish and Yin*, Junjie Oscar and Shao, Rulin and Xiao, Teng and Lambert, Nathan and Hajishirzi, Hannaneh},
year = {2026},
eprint = {2606.23321},
archiveprefix = {arXiv},
primaryclass = {cs.CL},
url = {https://arxiv.org/abs/2606.23321},
code = {https://github.com/hamishivi/tmax}
}
Terminal-using agents have quickly become the most popular downstream application of language models (LMs). Despite their prevalence, relatively little academic work has examined RL-based training of these models, likely due to difficult benchmarks, a lack of data, and a lack of simple baseline recipes. We present Tmax, the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. While simple, our recipe achieves 27% on Terminal-Bench 2.0 with only 9B parameters, outperforming much larger models from prior work. Concretely, we generate data using a novel taxonomy, combining difficulty control, personas, and verifier diversification, which allows us to cheaply generate large amounts of terminal environments for RL and SFT training. We open-source our terminal dataset, which is over 2.5x larger than previously released terminal-agent datasets. We then train open-weight models using RL with our data, using a simple, outcome-only recipe. We release our data, models, and code as a strong baseline for future open academic work on terminal agents.
Meta-Reinforcement Learning with Self-Reflection for Agentic SearchTeng Xiao, Yige Yuan, Hamish Ivison, Huaisheng Zhu, Faeze Brahman, Nathan Lambert, Pradeep Dasigi, Noah A. Smith, and Hannaneh Hajishirzi. 2026. COLM.
@article{xiao2026metareinforcementlearningselfreflectionagentic,
title = {Meta-Reinforcement Learning with Self-Reflection for Agentic Search},
author = {Xiao, Teng and Yuan, Yige and Ivison, Hamish and Zhu, Huaisheng and Brahman, Faeze and Lambert, Nathan and Dasigi, Pradeep and Smith, Noah A. and Hajishirzi, Hannaneh},
year = {2026},
journal = {COLM},
eprint = {2603.11327},
archiveprefix = {arXiv},
primaryclass = {cs.LG},
url = {https://arxiv.org/abs/2603.11327},
code = {https://github.com/tengxiao1/MR-Search}
}
This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks.
DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep ResearchRulin Shao*, Akari Asai*, Shannon Shen*, Hamish Ivison*, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Sam Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldani, Faeze Brahman, Scott Yih, Sherry Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hanna Hajishirzi, Pang Wei Koh. 2026. ICML.
@article{drtulu,
title = {{DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research}},
author = {{Rulin Shao*, Akari Asai*, Shannon Shen*, Hamish Ivison*, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Sam Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldani, Faeze Brahman, Scott Yih, Sherry Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hanna Hajishirzi, Pang Wei Koh}},
year = {2026},
journal = {ICML},
url = {https://www.datocms-assets.com/64837/1763476533-dr_tulu.pdf},
code = {https://github.com/rlresearch/dr-tulu}
}
Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.
RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable EnvironmentsZhiyuan Zeng*, Hamish Ivison*, Yiping Wang*, Lifan Yuan*, Shuyue Stella Li, Zhuorui Ye, Siting Li, Jacqueline He, Runlong Zhou, Tong Chen, Chenyang Zhao, Yulia Tsvetkov, Simon Shaolei Du, Natasha Jaques, Hao Peng, Pang Wei Koh, and Hannaneh Hajishirzi. 2026. ICML.
@article{zeng2025rlve,
title = {{RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments}},
author = {Zeng*, Zhiyuan and Ivison*, Hamish and Wang*, Yiping and Yuan*, Lifan and Li, Shuyue Stella and Ye, Zhuorui and Li, Siting and He, Jacqueline and Zhou, Runlong and Chen, Tong and Zhao, Chenyang and Tsvetkov, Yulia and Du, Simon Shaolei and Jaques, Natasha and Peng, Hao and Koh, Pang Wei and Hajishirzi, Hannaneh},
year = {2026},
journal = {ICML},
url = {https://arxiv.org/abs/2511.07317},
code = {https://github.com/Zhiyuan-Zeng/RLVE}
}
We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model’s capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM’s original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly.
We introduce Olmo 3, a family of state-of-the-art, fully open language models at the 7B and 32B parameter scales. Olmo 3 model construction targets long context reasoning, function calling, coding, instruction following, general chat, and knowledge recall. The release includes the entire model flow, i.e., the full lifecycle of the family of models, including every stage, checkpoint, datapoint, and dependency used to build it. Our flagship model, Olmo 3 Think-32B, is the strongest fully open thinking model released to-date.
Generalizing Verifiable Instruction FollowingValentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. 2025. NeurIPS Datasets and Benchmarks Track.
@article{pyatkin2025generalizing,
title = {{Generalizing Verifiable Instruction Following}},
author = {Pyatkin, Valentina and Malik, Saumya and Graf, Victoria and Ivison, Hamish and Huang, Shengyi and Dasigi, Pradeep and Lambert, Nathan and Hajishirzi, Hannaneh},
year = {2025},
journal = {NeurIPS Datasets and Benchmarks Track},
url = {https://arxiv.org/abs/2507.02833},
code = {https://github.com/allenai/IFBench}
}
A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like “only answer with yes or no" or “mention the word ‘abrakadabra’ at least 3 times" that the user adds to craft a more useful answer. Even today’s strongest models struggle with fulfilling such constraints. We find that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. We introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, we perform an extensive analysis of how and on what data models can be trained to improve precise instruction following generalization. Specifically, we carefully design constraint verification modules and show that reinforcement learning with verifiable rewards (RLVR) significantly improves instruction following. In addition to IFBench, we release 29 additional new hand-annotated training constraints and verification functions, RLVR training prompts, and code.
The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong GainsScott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, and Pang Wei Koh. 2025. COLM.
@article{geng2025delta,
title = {{The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains}},
author = {Geng, Scott and Ivison, Hamish and Li, Chun-Liang and Sap, Maarten and Li, Jerry and Krishna, Ranjay and Koh, Pang Wei},
year = {2025},
journal = {COLM},
url = {https://arxiv.org/abs/2507.06187}
}
Improvements in language models are often driven by improving the quality of the data we train them on, which can be limiting when strong supervision is scarce. In this work, we show that paired preference data consisting of individually weak data points can enable gains beyond the strength of each individual data point. We formulate the delta learning hypothesis to explain this phenomenon, positing that the relative quality delta between points suffices to drive learning via preference tuning–even when supervised finetuning on the weak data hurts. We validate our hypothesis in controlled experiments and at scale, where we post-train 8B models on preference data generated by pairing a small 3B model’s responses with outputs from an even smaller 1.5B model to create a meaningful delta. Strikingly, on a standard 11-benchmark evaluation suite (MATH, MMLU, etc.), our simple recipe matches the performance of Tulu 3, a state-of-the-art open model tuned from the same base model while relying on much stronger supervisors (e.g., GPT-4o). Thus, delta learning enables simpler and cheaper open recipes for state-of-the-art post-training. To better understand delta learning, we prove in logistic regression that the performance gap between two weak teacher models provides useful signal for improving a stronger student. Overall, our work shows that models can learn surprisingly well from paired data that might typically be considered weak.
Large-Scale Data Selection for Instruction TuningHamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. 2025.
@article{ivisondata2025,
title = {{Large-Scale Data Selection for Instruction Tuning}},
author = {Ivison, Hamish and Zhang, Muru and Brahman, Faeze and Koh, Pang Wei and Dasigi, Pradeep},
year = {2025},
eprint = {2503.01807},
archiveprefix = {arXiv},
primaryclass = {cs.CL},
url = {https://arxiv.org/abs/2503.01807},
code = {https://github.com/hamishivi/automated-instruction-selection}
}
Selecting high-quality training data from a larger pool is a crucial step
when instruction-tuning language models, as carefully curated datasets often produce
models that outperform those trained on much larger, noisier datasets. Automated data
selection approaches for instruction-tuning are typically tested by selecting small
datasets (roughly 10k samples) from small pools (100-200k samples). However, popular
deployed instruction-tuned models often train on hundreds of thousands to millions of
samples, subsampled from even larger data pools. We present a systematic study of how
well data selection methods scale to these settings, selecting up to 2.5M samples from
pools of up to 5.8M samples and evaluating across 7 diverse tasks. We show that many
recently proposed methods fall short of random selection in this setting (while using
more compute), and even decline in performance when given access to larger pools of data
to select over. However, we find that a variant of representation-based data selection
(RDS+), which uses weighted mean pooling of pretrained LM hidden states, consistently
outperforms more complex methods across all settings tested – all whilst being more
compute-efficient. Our findings highlight that the scaling properties of proposed automated
selection methods should be more closely examined.
TESS 2: A Large-Scale Generalist Diffusion Language ModelJaesung Tae*, Hamish Ivison*, Sachin Kumar, and Arman Cohan. 2025. ACL.
@article{taeivison2025tess2,
title = {{TESS 2: A Large-Scale Generalist Diffusion Language Model}},
author = {Tae*, Jaesung and Ivison*, Hamish and Kumar, Sachin and Cohan, Arman},
year = {2025},
journal = {ACL},
url = {https://arxiv.org/abs/2502.13917},
code = {https://github.com/hamishivi/tess-2}
}
We introduce TESS 2, a general instruction-following diffusion language model
that outperforms contemporary instruction-tuned diffusion models, as well as matches and
sometimes exceeds strong autoregressive (AR) models. We train TESS 2 by first adapting a
strong AR model via continued pretraining with the usual cross-entropy as diffusion loss,
and then performing further instruction tuning. We find that adaptation training as well
as the choice of the base model is crucial for training good instruction-following diffusion
models. We further propose reward guidance, a novel and modular inference-time guidance
procedure to align model outputs without needing to train the underlying model. Finally,
we show that TESS 2 further improves with increased inference-time compute, highlighting
the utility of diffusion LMs in having fine-grained controllability over the amount of
compute used at inference time.
We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes
dense autoregressive models with improved architecture and training recipe, pretraining data mixtures,
and instruction tuning recipes. Our modified model architecture and training recipe achieve both
better training stability and improved per-token efficiency. Our updated pretraining data mixture
introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model
capabilities across many downstream task benchmarks when introduced via late-stage curriculum
training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best
practices from Tülu 3 to develop OLMo 2-Instruct, focusing on permissive data and extending our
final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the
Pareto frontier of performance to compute, often matching or outperforming open-weight only models
like Llama 3.1 and Qwen 2.5 while using fewer FLOPs and with fully transparent training data, code,
and recipe. Our fully open OLMo 2-Instruct models are competitive with or surpassing open-weight
only models of comparable size, including Qwen 2.5, Llama 3.1 and Gemma 2. We release all OLMo
2 artifacts openly—models at 7B and 13B scales, both pretrained and post-trained, including their
full training data, training code and recipes, training logs and thousands of intermediate checkpoints.
The final instruction model is available on the Ai2 Playground as a free research demo.
Tülu 3: Pushing Frontiers in Open Language Model Post-TrainingNathan Lambert*, Jacob Morrison*, Valentina Pyatkin*, Shengyi Huang*, Hamish Ivison*, Faeze Brahman*, Lester James V. Miranda*, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, et al. 2025. COLM.
@article{lambert2024tulu3,
title = {Tülu 3: Pushing Frontiers in Open Language Model Post-Training},
author = {Lambert*, Nathan and Morrison*, Jacob and Pyatkin*, Valentina and Huang*, Shengyi and Ivison*, Hamish and Brahman*, Faeze and Miranda*, Lester James V. and Liu, Alisa and Dziri, Nouha and Lyu, Shane and Gu, Yuling and Malik, Saumya and Graf, Victoria and Hwang, Jena D. and Yang, Jiangjiang and Bras, Ronan Le and Tafjord, Oyvind and Wilhelm, Chris and Soldaini, Luca and Smith, Noah A. and Wang, Yizhong and Dasigi, Pradeep and Hajishirzi, Hannaneh},
year = {2025},
journal = {COLM},
email = {tulu@allenai.org},
url = {https://arxiv.org/abs/2411.15124},
code = {https://github.com/allenai/open-instruct}
}
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tülu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tülu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tülu 3, we build a multi-task evaluation scheme for post-training with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. The Tülu 3 release includes model weights, a demo, and the complete recipe — datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tülu 3 approach to more domains.
Personalizing Reinforcement Learning from Human Feedback with Variational Preference LearningSriyash Poddar*, Yanming Wan*, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. 2024. NeurIPS.
@article{Poddar2024PersonalizingRL,
title = {Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning},
author = {Poddar*, Sriyash and Wan*, Yanming and Ivison, Hamish and Gupta, Abhishek and Jaques, Natasha},
year = {2024},
url = {https://arxiv.org/abs/2408.10075},
code = {https://github.com/WEIRDLabUW/vpl_llm},
journal = {NeurIPS}
}
Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.
Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference FeedbackHamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. 2024. NeurIPS.
@article{ivison2024unpacking,
title = {Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback},
author = {Ivison, Hamish and Wang, Yizhong and Liu, Jiacheng and Wu, Zeqiu and Pyatkin, Valentina and Lambert, Nathan and Smith, Noah A. and Choi, Yejin and Hajishirzi, Hannaneh},
year = {2024},
eprint = {2406.09279},
journal = {NeurIPS},
url = {https://arxiv.org/abs/2406.09279},
code = {https://github.com/allenai/open-instruct}
}
Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training and evaluating our models, along with the models and datasets themselves.
OLMo: Accelerating the Science of Language ModelsDirk Groeneveld, Iz Beltagy, ..., Hamish Ivison, ..., Noah A. Smith, and Hannaneh Hajishirzi. 2024. ACL.
Language models (LMs) have become ubiquitous in both NLP research and in
commercial product offerings. As their commercial importance has surged, the
most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development
undisclosed. Given the importance of these details in scientifically studying these
models, including their biases and potential risks, we believe it is essential for the
research community to have access to powerful, truly open LMs. To this end, this
technical report details the first release of OLMo, a state-of-the-art, truly
Open
Language Model and its framework to build and study the science of language
modeling. Unlike most prior efforts that have only released model weights and
inference code, we release OLMo and the whole framework, including training
data and training and evaluation code. We hope this release will empower and
strengthen the open research community and inspire a new wave of innovation.
Backtracking Mathematical Reasoning of Language Models to the Pretraining DataYasaman Razeghi*, Hamish Ivison*, Sameer Singh, and Yanai Elazar. 2024. The Second Tiny Papers Track at ICLR 2024.
@article{backtracking,
title = {Backtracking Mathematical Reasoning of Language Models to the Pretraining Data},
author = {Razeghi*, Yasaman and Ivison*, Hamish and Singh, Sameer and Elazar, Yanai},
booktitle = {The Second Tiny Papers Track at ICLR 2024},
year = {2024},
url = {https://openreview.net/pdf?id=otHhLO7GZj}
}
In-context learning and chain-of-thought prompting have demonstrated surprising performance improvements on mathematical reasoning benchmarks. Therefore, understanding the underlying factors enabling these capabilities is crucial. However, the specific aspects of pretraining data that equip models with mathematical reasoning capabilities remain largely unexplored and are less studied systematically. In this study, we identify subsets of model pretraining data that contribute to math reasoning ability of the model, and evaluate it on several mathematical operations (e.g. addition, multiplication) and tasks (e.g. the asdiv dataset). We measure the importance of such subsets by continual training of the model on pretraining data subsets, and then we quantify the change in performance on the mathematical benchmark to assess their importance. If a subset results in an improved performance, we conjecture that such subset contributes to a model’s overall mathematical ability. Our results unveil that while training on math-only data contributes to simple arithmetic abilities, it does not solely explain performance on more complex reasoning abilities like chain-of-thought reasoning. We also find that code data contributes to chain-of-thought reasoning while reducing the arithmetic performance.
TESS: Text-to-Text Self-Conditioned Simplex DiffusionRabeeh Karimi Mahabadi*, Hamish Ivison*, Jaesung Tae, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2024. EACL.
@article{tess,
author = {Mahabadi*, Rabeeh Karimi and Ivison*, Hamish and Tae, Jaesung and Henderson, James and Beltagy, Iz and Peters, Matthew E. and Cohan, Arman},
title = {TESS: Text-to-Text Self-Conditioned Simplex Diffusion},
journal = {EACL},
url = {https://arxiv.org/abs/2305.08379},
year = {2024},
code = {https://github.com/allenai/tess-diffusion}
}
Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various domains with continuous-valued inputs. Despite the promises of fully non-autoregressive text generation, applying diffusion models to natural language remains challenging due to its discrete nature. In this work, we propose Text-to-text Self-conditioned Simplex Diffusion (TESS), a text diffusion model that is fully non-autoregressive, employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the typical learned embedding space. Through extensive experiments on natural language understanding and generation tasks including summarization, text simplification, paraphrase generation, and question generation, we demonstrate that TESS outperforms state-of-the-art non-autoregressive models and is competitive with pretrained autoregressive sequence-to-sequence models.
Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2Hamish Ivison*, Yizhong Wang*, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. technical report.
@article{ivison2023camels,
title = {Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2},
author = {Ivison*, Hamish and Wang*, Yizhong and Pyatkin, Valentina and Lambert, Nathan and Peters, Matthew and Dasigi, Pradeep and Jang, Joel and Wadden, David and Smith, Noah A. and Beltagy, Iz and Hajishirzi, Hannaneh},
year = {2023},
url = {https://arxiv.org/abs/2311.10702},
eprint = {2311.10702},
journal = {technical report},
primaryclass = {cs.CL},
code = {https://github.com/allenai/open-instruct}
}
Since the release of TÜLU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into TÜLU, resulting in TÜLU 2, a suite of improved TÜLU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) TÜLU-V2-mix, an improved collection of high-quality instruction datasets; (2) TÜLU 2, LLAMA-2 models finetuned on the V2 mixture; (3) TÜLU 2+DPO, TÜLU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (TÜLU 2+DPO 70B); (4) CODE TÜLU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the TÜLU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks. We release all the checkpoints, data, training and evaluation code to facilitate future open efforts on adapting large language models.
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open ResourcesYizhong Wang*, Hamish Ivison*, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. NeurIPS Datasets and Benchmarks Track.
@article{tulu,
title = {How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources},
author = {Wang*, Yizhong and Ivison*, Hamish and Dasigi, Pradeep and Hessel, Jack and Khot, Tushar and Chandu, Khyathi Raghavi and Wadden, David and MacMillan, Kelsey and Smith, Noah A. and Beltagy, Iz and Hajishirzi, Hannaneh},
year = {2023},
url = {https://arxiv.org/abs/2306.04751},
eprint = {2306.04751},
journal = {NeurIPS Datasets and Benchmarks Track},
primaryclass = {cs.CL},
code = {https://github.com/allenai/open-instruct}
}
In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with state-of-the-art proprietary models, these claims are often accompanied by limited evaluation, making it difficult to compare models across the board and determine the utility of various resources. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets ranging from manually curated (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, and open-ended instruction following abilities through a collection of automatic, model-based, and human-based metrics. We further introduce Tülu, our best performing instruction-tuned model suite finetuned on a combination of high-quality open resources. Our experiments show that different instruction-tuning datasets can uncover or enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preference-based evaluations fail to reflect differences in model capabilities exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average 83% of ChatGPT performance, and 68% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including a fully finetuned 65B Tülu, along with our code, data, and evaluation framework at https://github.com/allenai/open-instruct to facilitate future research.
HINT: Hypernetwork Instruction Tuning for Efficient Zero-Shot GeneralisationHamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, and Matthew Peters. 2023. ACL.
@article{hint,
author = {Ivison, Hamish and Bhagia, Akshita and Wang, Yizhong and Hajishirzi, Hannaneh and Peters, Matthew},
title = {HINT: Hypernetwork Instruction Tuning for Efficient Zero-Shot Generalisation},
journal = {ACL},
url = {https://arxiv.org/abs/2212.10315},
year = {2023},
code = {https://github.com/allenai/hyper-task-descriptions}
}
Recent NLP models have the great ability to generalise ‘zero-shot’ to new tasks using only an instruction as guidance. However, these approaches usually repeat their instructions with every input, requiring costly reprocessing of lengthy instructions for every inference example. To alleviate this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples using a pretrained text encoder into parameter-efficient modules inserted into an underlying model, eliminating the need to include instructions in the model input. Compared to prior approaches that concatenate instructions with every input instance, we find that HINT models are significantly more compute-efficient and consistently outperform these approaches for a given inference budget.
Data-Efficient Finetuning Using Cross-Task Nearest NeighborsHamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. 2023. Findings of ACL.
@article{deft,
author = {Ivison, Hamish and Smith, Noah A. and Hajishirzi, Hannaneh and Dasigi, Pradeep},
title = {Data-Efficient Finetuning Using Cross-Task Nearest Neighbors},
journal = {Findings of ACL},
code = {https://github.com/allenai/data-efficient-finetuning},
url = {https://arxiv.org/abs/2212.00196},
year = {2023}
}
Language models trained on massive prompted multitask datasets like T0 (Sanh et al., 2021) or FLAN (Wei et al., 2021a) can generalize to tasks unseen during training. We show that training on a carefully chosen subset of instances can outperform training on all available data on a variety of datasets. We assume access to a small number (250–1000) of unlabeled target task instances, select their nearest neighbors from a pool of multitask data, and use the retrieved data to train target task-specific models. Our method is more data-efficient than training a single multitask model, while still outperforming it by large margins. We evaluate across a diverse set of tasks not in the multitask pool we retrieve from, including those used to evaluate T0 and additional complex tasks including legal and scientific document QA. We retrieve small subsets of P3 (the collection of prompted datasets from which T0’s training data was sampled) and finetune T5 models that outperform the 3-billion parameter variant of T0 (T0-3B) by 3–30% on 12 out of 14 evaluation datasets while using at most 2% of the data used to train T0-3B. These models also provide a better initialization than T0-3B for few-shot finetuning on target-task data, as shown by a 2–23% relative improvement over few-shot finetuned T0-3B models on 8 datasets. Our code is available at https://github.com/allenai/data-efficient-finetuning.
Hyperdecoders: Instance-specific decoders for multi-task NLPHamish Ivison and Matthew E. Peters. 2022. Findings of EMNLP.
@article{hyperdecoders,
url = {https://arxiv.org/abs/2203.08304},
author = {Ivison, Hamish and Peters, Matthew E.},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Hyperdecoders: Instance-specific decoders for multi-task NLP},
journal = {Findings of EMNLP},
year = {2022},
code = {https://github.com/allenai/hyperdecoders}
}
We investigate input-conditioned hypernetworks for multi-tasking in NLP, generating parameter-efficient adaptations for a decoder using a hypernetwork conditioned on the output of an encoder. This approach produces a unique decoder for every input instance, allowing the network a larger degree of flexibility than prior work that specializes the decoder for each task. We apply our method to sequence classification tasks, extractive QA, and summarisation and find that it surpasses previous parameter efficient fine-tuning methods and often outperforms fully finetuning the underlying model. An analysis of the embeddings used by our hypernetwork shows that they are sensitive to output label and type, suggesting that our approach better maps from encoder representations to output labels.
Local Interpretations for Explainable Natural Language Processing:
A SurveySiwen Luo*, Hamish Ivison*, Soyeon Caren Han, and Josiah Poon. 2021. ACM Computing Surveys.
@article{localinterp,
author = {Luo*, Siwen and Ivison*, Hamish and Han, Soyeon Caren and Poon, Josiah},
title = {Local Interpretations for Explainable Natural Language Processing:
{A} Survey},
year = {2021},
url = {https://arxiv.org/abs/2103.11072},
journal = {ACM Computing Surveys},
eprint = {2103.11072},
timestamp = {Wed, 24 Mar 2021 15:50:40 +0100}
}
As the use of deep learning techniques has grown across various fields over the past decade, complaints about the opaqueness of the black-box models have increased, resulting in an increased focus on transparency in deep learning models. This work investigates various methods to improve the interpretability of deep neural networks for natural language processing (NLP) tasks, including machine translation and sentiment analysis. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work. The methods collected and summarised in this survey are only associated with local interpretation and are divided into three categories: 1) explaining the model’s predictions through related input features; 2) explaining through natural language explanation; 3) probing the hidden states of models and word representations.
Would you like fries with that? Modular Multi-hop ReasoningHamish Ivison. 2020. November.
@thesis{thesis,
author = {Ivison, Hamish},
title = {Would you like fries with that? Modular Multi-hop Reasoning},
school = {University of Sydney},
type = {Honours Thesis},
year = {2020},
month = nov,
url = {/assets/static/thesis.pdf}
}
In this work, we investigate an interpretable, modular approach to multi-hop question answering by adapting a popular visual question answering architecture, the MAC cell, to the task of multi-hop reading comprehension. In multi-hop reading comprehension, a model must answer questions by collating facts from multiple text sources. Our augmented MAC cell design outperforms existing modular approaches to multi-hop QA with less supervision and provides interpretable insights into its reasoning process. We then investigate integrating our cell with the highly popular BERT model and design a novel model which iteratively reads and retrieves documents in an interpretable fashion, allowing scalable and interpretable multi-hop question answering. Alongside this, we investigate the behaviour of generic BERT-based models on multi-hop QA and show that several existing approaches to multi-hop QA fail to significantly beat a naive BERT baseline. Our work shows the promise of MAC networks for multi-hop reasoning and outlines future paths for both MAC networks and multi-hop reasoning as a whole.
