AI-Enhanced Audit and Correspondence Experiments - A Short Course

By the end of the course, you will be able to:

• • Design audit and correspondence experiments with a clear understanding of identification, signal minimalism, and information equivalence.
  • Distinguish which parts of an audit-study workflow can be responsibly delegated to AI agents and which require direct human control.
  • Use AI agents to help construct tightly controlled signals, including names, emails, résumés, cover letters, and application materials.
  • Recognize how agent-produced text and behavior can introduce unintended class, racial, gendered, cultural, or stylistic cues.
  • Develop validation and pre-testing protocols tailored to AI-assisted correspondence.
  • Evaluate whether agent-generated variation functions as a meaningful treatment signal or as uncontrolled noise.
  • Use AI agents to support the fielding of audit studies, including structured treatment delivery, message sequencing, follow-up management, and instrument administration, without compromising design integrity.
  • Distinguish between bias embedded in the agentic system and bias revealed through field responses.
  • Clearly document the use of AI agents for replication, peer review, and ethical oversight.

To participate fully, you’ll need to download and set up Warp, a terminal-driven agent interface. No programming setup is required, though you’re welcome to use a laptop and a current project to apply the workflows to your own work.

This seminar is designed for:

• • Researchers who use or plan to use audit and correspondence experiments in studies of discrimination, gatekeeping, or social evaluation.
  • Faculty members, graduate students, and applied researchers interested in integrating AI agents into experimental design without sacrificing causal clarity.
  • Scholars who want practical guidance on using AI agents to support stimulus generation, treatment delivery, and study implementation while maintaining control over equivalence and interpretation.
  • Research teams seeking transparent, defensible workflows for AI-assisted audit studies.

This course is not for those seeking a general introduction to AI, chatbot-based experimentation, or AI-assisted statistical analysis. It is not intended as a course on fully autonomous field experiments. The emphasis throughout is on careful design, bounded delegation, and credible inference.

Audit-study logic and why AI agents create both promise and risk

• • What audit and correspondence experiments identify
  • Signal minimalism and the logic of controlled difference
  • Information equivalence and why it is easy to break
  • How AI agents can help and how they can distort
  • The difference between useful augmentation and uncontrolled variation

Using AI agents to generate correspondence without losing design control

• • Constrained stimulus generation with AI agents
  • Building templates for names, résumés, emails, and application materials
  • Iterative refinement with human oversight
  • Detecting unintended cues in style, fluency, tone, and content
  • Designing pre-tests and diagnostics for agent-assisted materials

Using AI agents to field studies: treatment and instrument delivery

• • When AI agents can be used to deliver treatments or administer instruments in correspondence designs
  • Standardization versus responsiveness in agent-mediated delivery
  • Risks introduced by adaptive behavior at the point of contact
  • Building bounded workflows for message delivery, follow-up, and interaction management
  • Logging, auditing, and validating what was actually delivered in the field

Inference, ethics, and reviewer-proofing AI-enhanced audit studies

• • Conducting audit studies when stimuli or delivery mechanisms are partly agent-assisted
  • Interpreting results when the agentic system may itself introduce bias
  • Separating treatment effects from agent artifacts
  • IRB review, ethical burden, and institutional concerns
  • Writing methods sections that anticipate skepticism from reviewers and editors