Skip to content
Projects

Project

LLM: Diffusion, MBO, and LLM Pretraining

A CPIL research theme bringing diffusion/generative models and offline model-based optimization (MBO) to large language models, including an LLM-oriented Design-Bench 2.0.

  • Large Language Models
  • Diffusion Models
  • Offline Model-Based Optimization
  • Black-Box Optimization
  • LLM Pretraining
  • Generative Models
Visualization for LLM: Diffusion, MBO, and LLM Pretraining

Overview

This theme studies the intersection of diffusion and flow-based generative models, offline black-box / model-based optimization (MBO), and LLM pretraining. A current focus is Design-Bench 2.0, an LLM-oriented benchmark that adapts offline MBO algorithms to LLM-related tasks, alongside diffusion- and flow-based methods for black-box and multi-objective optimization.

Motivation

This theme sits at the intersection of three lines of work that increasingly reinforce one another: diffusion and flow-based generative modeling, offline model-based optimization (MBO), and large language model pretraining. Offline MBO methods learn to propose high-performing designs purely from a static dataset of past evaluations, and diffusion/flow models have proven to be powerful tools for representing and editing those design distributions. The theme explores how these optimization ideas transfer to the LLM setting, where the “design space” becomes language- and sequence-structured.

Project Goals

  • Develop diffusion- and flow-based estimators and samplers for offline black-box and multi-objective optimization.
  • Bridge offline MBO algorithms with LLM-related tasks, treating language and sequence generation as an optimization problem.
  • Build an LLM-oriented benchmark that standardizes evaluation of these methods.

Recent Progress

The team has started an LLM-oriented Design-Bench 2.0, with the goal of adapting offline MBO algorithms to LLM-related tasks. This extends the classic offline black-box optimization benchmarking setup toward language-model settings, providing a common ground to test diffusion-, flow-, and optimization-based methods on LLM tasks.

Recent published results span diffusion estimation for offline black-box optimization (SPADE, ICML 2026), training diffusion language models directly for black-box optimization (ICML 2026 Spotlight), and a preprint on diffusion large language models for black-box optimization, building on earlier work in design editing (TMLR), guided flows for multi-objective optimization (ICLR 2025), and importance-aware co-teaching (NeurIPS 2023). See the publications list above for details.

Related Publications

Impact Holders

Impact holders and user communities will be added as the project scope becomes clearer.