Joseph Suh

Hi there! I'm a third year Ph.D. student at BAIR in UC Berkeley, advised by Serina Chang and John Canny.
Most of my recent research is on user simulation and synthetic data generation for interactive evaluation and training of language models. I'm betting on user simulation as a scalable path to align language models with human goals.

Email  /  CV  /  Scholar  /  X/Twitter  /  Github

Research

Human-LLM conversation datasets shape how we understand AI use and train user models, but how different are the interactions they capture? We study seven datasets and find distinctive dataset signatures: classifiers can identify a conversation's source from user messages alone, well above chance.

We argue that the current dominant practice in LLM human simulation, prompting instruction-tuned “assistant” language models to role-play personas, is inaccurate and produces stereotyped predictions. We propose and explore tandem models which combine a pre-trained model with an instruction-tuned supervisor.

With controlled experiments, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs when interacting with real humans.

We present an open-source interactive platform for demographically controllable survey simulation using backstory-conditioned large language models.

We analyze here how deep binding of pretrained base models with extended backstories leads to more faithful replication of identity-based behaviors.

We identify a broad class of human simulation tasks currently addressed with LLMs—discrete choice prediction—and propose a graph-based modeling that matches or exceeds LLM-based approaches while offering added advantages.

Expanding evaluations of LLM virtual personas to include ingroup / outgroup and meta-perception, "backstories" serve as a stepping stone to higher-order reflections in social context.

Fine-tuning LLMs on response distributions from public opinion survey questions enables the models to predict opinions across different subpopulations, survey waves, and survey families.

Constructing latent dimensions of personality using log-probabilities from language models, inspired by the methodology psychologists used to develop the Big Five model.

We propose naturalistic bodies of text describing individual life stories, namely "backstories", as prefix to model prompts for persona conditioning.

A photonic lattice with non-nearest-neighbor interactions that breaks the tradeoff between topological channels and channel bandwidths.

Theoretical modeling of Laughlin's topological pump in synthetic frequency dimensions by the interplay of frequency mode-dependent and independent gauge fields.

Education and Work

About me

Honors and Awards

Blog

Powered by Jon Barron's theme.