Eve Fleisig

I'm an incoming postdoctoral fellow at Princeton CITP. I received my PhD from Berkeley EECS, where I was advised by Dan Klein in BAIR's Berkeley NLP group.

My research combines natural language processing (NLP) and AI ethics: how do we develop language models that we trust to benefit all users, and create safeguards against societal harms? To do so, I work on training and evaluating language models to serve complex distributions of users with varied needs. This includes training on informative disagreement among users, evaluating discrimination against users who speak differently, and designing frameworks for LLMs that serve populations with many perspectives. Broadly, I am interested in topics related to NLP, societal impacts, preference learning, AI + sociolinguistics, and the science of ML evaluation.

I previously earned a BSE in computer science and minor in linguistics at Princeton University, advised by Christiane Fellbaum. I've received the NSF Graduate Research Fellowship, Berkeley Chancellor's Fellowship, and paper awards at NAACL and EMNLP. My work has been covered in outlets including Nature and The Verge.

Recent News

• ✈️ [Jun '26] Visiting Angelina Wang at Cornell Tech this summer.
• 💬 [Aug '25] Invited talk at UW NLP on AI leaderboard manipulation.
• 💬 [Dec '25] Invited panelist for the NeurIPS '25 Science of Benchmarking tutorial.
• 💬 [Nov '25] Invited panelist for NLPerspectives at EMNLP '25.
• ✈️ [Oct '25] Visiting Dirk Hovy at the Milan NLP lab this fall.
• 💬 [Aug '25] Invited talk at Edinburgh NLP on GRACE.
• 🏆 [July '25] My 3-minute thesis won 2nd place at the LSA Summer Institute.
• 🏆 [May '25] AdvScore won Outstanding Paper at NAACL '25.
• 💬 [May '24] Invited talk at Stanford NLP on Linguistic Bias in ChatGPT.

Selected Work

Please see Google or Semantic Scholar for an up-to-date list.

Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell, Dirk Hovy [ PREPRINT ]

Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions Eve Fleisig*, Matthias Orlikowski*, Philipp Cimiano, Dan Klein [ PAPER ]

GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration Yoo Yeon Sung*, Eve Fleisig*, Yu Hou, Ishan Upadhyay, Jordan Boyd-Graber (ACL 2025). [ PAPER ]

Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness Yoo Yeon Sung, Maharshi Gor, Eve Fleisig, Ishani Mondal, Jordan Boyd-Graber (NAACL 2025 - Outstanding Paper Award). [ PAPER ][ TWITTER ]

Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination Eve Fleisig*, Genevieve Smith*, Madeline Bossi*, Ishita Rustagi*, Xavier Yin*, Dan Klein (EMNLP 2024). [ PAPER ][ TWITTER ][ BLOG ]

Mapping Social Choice Theory to RLHF Jessica Dai, Eve Fleisig (R2FM @ ICLR 2024). [ ARXIV ][ TWITTER ]

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks Eve Fleisig, Rediet Abebe, Dan Klein (EMNLP 2023). [ PAPER ][ TWITTER ]

Incorporating Worker Perspectives into MTurk Annotation Practices for NLP Olivia Huang, Eve Fleisig, Dan Klein (EMNLP 2023 - Outstanding Paper Award). [ PAPER ]

Ghostbuster: Detecting Text Ghostwritten by Large Language Models Vivek Verma, Eve Fleisig, Nicholas Tomlin, Dan Klein (NAACL 2024). [ PAPER ][ TWITTER ][ BLOG ]

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat (NAACL 2024). [ PAPER ]

Hedges and Apologies in ChatGPT Responses to African-American English Eve Fleisig (NWAV 2023). [ SLIDES ][ ABSTRACT ]

First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models Naomi Saphra, Eve Fleisig, Kyunghyun Cho, Adam Lopez (NAACL 2024). [ PAPER ][ TWITTER ]

FairPrism: Evaluating fairness-related harms in text generation Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach (ACL 2023). [ PAPER ][ DATASET ]

Mitigating Gender Bias in Machine Translation through Adversarial Learning Eve Fleisig, Christiane Fellbaum Outstanding Senior Thesis Award; Sigma Xi Book Award [ ARXIV ]

Mentorship

I've mentored some wonderful undergraduate students, including Samuel Ghezae (→JHU), Olivia Huang (→Citadel), Harbani Jaggi (→Applied Intuition), Kayla Lee (→YC startup founder), Kashyap Murali (→Anthropic), Vyoma Raman (→Stanford, Cornell Tech), Mahathi Ryali, Zaina Shaik (→Amazon), Vivek Verma (→OpenAI), and Xavier Yin (→CMU).

I'm not currently taking on new undergraduates for research projects. However, I maintain a resource guide for students interested in NLP research, and I'm happy to chat about research or anything else!

Interested in chatting or collaborating?

Reach out to me at efleisig :at: berkeley :dot: edu

Miscellaneous

My surname is pronounced /'flʌɪsɪg/ ("fly"-sihg).

I'm a member of ACF, a volunteer-run organization that produces high-quality collegiate quizbowl tournaments. Very proud to have been on the Berkeley team that won both 2026 national championships! 🎉

🇦🇷¡Siempre estoy feliz de charlar con otros latinoamericanos!