Breaking Down EvalGen: Who Validates the Validators?

Update: 2024-05-13

Description

Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators often inherit the problems of the LLMs they evaluate, requiring further human validation.

This week’s paper explores EvalGen, a mixed-initative approach to aligning LLM-generated evaluation functions with human preferences. EvalGen assists users in developing both criteria acceptable LLM outputs and developing functions to check these standards, ensuring evaluations reflect the users’ own grading standards.

Read it on the blog: https://arize.com/blog/breaking-down-evalgen-who-validates-the-validators/

Paper: https://arxiv.org/abs/2404.12272

Learn more about AI observability and evaluation in our course, join the Arize AI Slack community or get the latest on LinkedIn and X.

Comments

Top Podcasts

The Best New Comedy Podcast Right Now – June 2024 The Best News Podcast Right Now – June 2024 The Best New Business Podcast Right Now – June 2024 The Best New Sports Podcast Right Now – June 2024 The Best New True Crime Podcast Right Now – June 2024 The Best New Joe Rogan Experience Podcast Right Now – June 20 The Best New Dan Bongino Show Podcast Right Now – June 20 The Best New Mark Levin Podcast – June 2024

In Channel

Merge, Ensemble, and Cooperate! A Survey on Collaborative LLM Strategies

2024-12-1028:47

Agent-as-a-Judge: Evaluate Agents with Agents

2024-11-2324:54

Introduction to OpenAI's Realtime API

2024-11-1229:56

Swarm: OpenAI's Experimental Approach to Multi-Agent Systems

2024-10-2946:46

KV Cache Explained

2024-10-2404:19

The Shrek Sampler: How Entropy-Based Sampling is Revolutionizing LLMs

2024-10-1603:31

Google's NotebookLM and the Future of AI-Generated Audio

2024-10-1543:28

Exploring OpenAI's o1-preview and o1-mini

2024-09-2742:02

Breaking Down Reflection Tuning: Enhancing LLM Performance with Self-Learning

2024-09-1926:54

Composable Interventions for Language Models

2024-09-1142:35

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

2024-08-1639:05

Breaking Down Meta's Llama 3 Herd of Models

2024-08-0644:40

DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines

2024-07-2333:57

RAFT: Adapting Language Model to Domain Specific RAG

2024-06-2844:01

LLM Interpretability and Sparse Autoencoders: Research from OpenAI and Anthropic

2024-06-1444:00

Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models' Alignment