Can You Trust Your AI Research Assistant? Evaluating LLMs and AI Agents - A Short Course

The seminar follows one evaluation workflow, from research task to reproducibility record, applied to three running examples. First, an LLM codes open-ended text and is validated against expert human coders. Second, that pipeline is stress-tested, and an LLM is then used as a judge applying a scoring rubric, with planted fabrications it must catch. Third, a transparent R research agent assembles cited evidence from a document corpus and is deliberately sabotaged to see whether it notices.

Along the way you will learn the measurement ideas that make an evaluation defensible: why agreement between human coders sets the ceiling for any model, which agreement statistics fit which tasks, how to tell systematic failure from random noise, and why an agent’s process matters as much as its answer. The emphasis is on repeatable evaluation experiments rather than informal spot-checks. This is not a prompt-engineering course, and it does not teach you to build agent frameworks; it teaches you to audit what such tools do and to decide, on the evidence, whether their output can count as research evidence.

Everything is done in R with the ellmer package and familiar tidyverse tools, so each method transfers directly to your own data. You will leave with reusable notebooks, benchmark and audit templates, an error taxonomy, a human-review protocol, and a reporting template. The final session is a guided clinic, so bring a dataset, a corpus, or a research question you are considering using an LLM for.

This is a hands-on seminar with instructor-led demonstrations and guided exercises in R. You will need a recent version of R (4.1 or later) and RStudio, or Posit Cloud, with the ellmer and tidyverse packages; a short list of supporting packages will be included in the setup instructions.

You will also need an API key for a commercial LLM provider. The standard configuration uses the OpenAI API, and the total cost of API calls for the seminar is expected to be below $20 USD; other providers can be used with minor changes to the code. No GPU is required, and no models need to be installed locally. Setup instructions will be distributed before the seminar, and all notebooks, datasets, and templates will be provided.

To follow along with the course exercises, you should have good familiarity with the use of R: running scripts, working with data frames, and using basic tidyverse functions.

If you’d like to take this course but are concerned that you don’t know enough R, there are excellent online resources for learning the basics. Here are our recommendations.

This seminar is for researchers and analysts, including faculty, postdocs, doctoral students, and research analysts, who use, or are preparing to use, LLMs or AI agents for text coding, literature screening, document review, evidence extraction, or research automation. The examples come from the social and behavioral sciences, health, business and management, education and the humanities; the methods apply to any field where text is data. Reviewers, editors, and research-integrity staff who assess AI-assisted work will also find the framework and reporting standards directly usable.

No prior experience with LLMs or AI is assumed. No background in machine learning or natural language processing is required. If you have already taken a course on using LLMs or agents in your research, this seminar is the natural next step; if you have not, it will still equip you to evaluate any AI-assisted output you are asked to trust.

Module 1: From AI-assisted work to research evidence

• Where LLMs and agents enter the research workflow: screening, coding, extraction, classification, and multi-step automation.
• Why fluent output is not evidence: instability, model dependence, weak grounding, and irreproducibility.
• The evaluation workflow used throughout the seminar, and what reviewers, editors, and ethics boards will ask.

Module 2: Calling LLMs from R

• Setting up ellmer and connecting to a provider.
• Prompts and structured output: getting typed, analyzable results rather than free text.
• Logging models, settings, and calls so that an evaluation can be rerun.

Module 3: Building a benchmark

• Defining the task, the unit of analysis, and the coding scheme.
• Creating a human reference standard: sampling, coding instructions, and expert coders.
• The human-agreement ceiling: why inter-coder agreement sets the bar for any model.
• Choosing and interpreting agreement statistics: percent agreement, kappa, Krippendorff’s alpha, precision, and recall.
• Hands-on: scoring a first run of the LLM coding pipeline against expert human codes.

Module 4: Stress-testing the pipeline: reliability, robustness, and error taxonomy

• Repeated runs: does the model give the same answer twice?
• Prompt perturbation: paraphrases, reordering, and instruction variants.
• Cross-model comparison and what disagreement between models tells you (instructor demonstration).
• Building an error taxonomy from model-human disagreements: fabrication, omission, over-inference, and drift.
• Hands-on: a reliability and robustness audit for the Module 3 pipeline.

Module 5: Auditing LLM-as-a-judge

• When researchers use an LLM to score, rank, or rubric-grade material.
• Why the judge is not neutral: position, length, and self-preference effects.
• Designing an empirical bias test, including planted fabrications the judge must catch.
• Hands-on: auditing an LLM judge and recording an admissibility decision.

Module 6: Evaluating an AI research agent

• Outcome versus process: why the final answer is not enough.
• Trajectory inspection: each step, each source, each decision.
• Fault injection: sabotaging the corpus to test whether the agent detects and recovers.
• Hands-on: inspecting a transparent R research agent that assembles cited evidence.

Module 7: Reproducibility records and reporting

• What to document: models and versions, prompts, settings, intermediate outputs, and human interventions.
• Writing the methods section: meeting reviewer, editor, and ethics-board expectations.
• Hands-on: completing the AI Research Assistant Evaluation Report template.

Module 8: Guided clinic: your own research

• Applying the benchmark, stress-test, and audit steps to your own data or materials.
• Troubleshooting setup, prompts, and metrics.
• Drafting a personal evaluation plan for a current or planned project.