Automatic Item Generation and Validation - A Short Course

Automatic Item Generation and Validation via Network Integrated Evaluation (AI-GENIE) leverages large language models (LLMs) and advanced network psychometric techniques to streamline the item generation and validation process without the need to collect data from human participants. (See the scientific preprint for technical details.) Traditional scale development is resource-intensive, time-consuming, and costly, often requiring extensive human expert intervention and costly data collection for psychometric validation.

Recent advancements in AI and LLMs offer promising solutions to generate expert-quality text for scale items. The challenge lies in efficiently selecting and validating non-redundant, high-quality items that accurately represent intended psychological constructs, and that present adequate dimensionality (structural validity) AND item/dimension stability. AI-GENIE automates the entire process, from item generation to validation, enhancing efficiency and scalability in psychological assessment creation.

Previous research has shown that AI-generated items can create adequate psychological assessments, but the item selection process remains resource-intensive (with rounds of data collection from human participants). AI-GENIE eliminates the need for extensive human expert involvement in generating, selecting, and validating items, potentially saving researchers significant time and money.

The methodology combines open and closed-source LLMs, generative AI, and network psychometrics to facilitate scale generation, selection, and validation. AI-GENIE is the first fully automated methodology to generate, assess, and validate the quality of AI-generated items for psychometric scales.

By the end of this course, you will be able to:

1. Understand the principles and applications of AI-GENIE in scale development.
2. Utilize various LLMs to generate new items using R.
3. Apply network psychometric techniques for item validation and selection in silico (i.e., without the use of human subjects) using R.
4. Critically evaluate the effectiveness and limitations of AI-generated items.
5. Design and implement a full-scale development project using AI-GENIE methodology.

This is a hands-on course with instructor-led software demonstrations and guided exercises. These guided exercises are designed for the R language, so you should use a computer with a recent version of R (version 4.1.3 or later) and RStudio (version 2022.02.1+461 or later).

NOTE: An OpenAI API and a Groq API will be required for the seminar (more details will be provided before the course). Registration for an OPENAI API is free but requires a credit card on file. No credit card will be required for the Groq API.

To follow along with the course exercises, you should have good familiarity with the use of R, including opening and executing data files and programs, as well as performing very basic data manipulation and analyses.

If you’d like to take this course but are concerned that you don’t know enough R, there are excellent online resources for learning the basics. Here are our recommendations.

This course is perfect for researchers, marketers, or anyone interested in creating more efficient surveys or tests. No prior knowledge of AI or advanced statistics is required, but you should have:

• Basic understanding of psychometrics and scale development.
• Familiarity with R programming, like from an introductory seminar such as Introduction to R for Data Analysis, R for SPSS Users, R for Stata Users, or R for SAS Users.
• Knowledge of statistical analysis and concepts.

Day 1

Introduction to AI-GENIE and LLMs

• Overview of traditional scale development challenges
• Introduction to AI-GENIE methodology
• Exploration of LLMs: New models (OSS-120b, OSS-20b), LLAMA-3, as well as Gemma 2, Mixtral 8x7b, GPT-3.5, and GPT-5
• Ethical considerations in AI-assisted scale development

Prompt engineering for item generation

• Principles of effective prompt design
• Few-shot prompting techniques
• Crafting prompts for item development
• Hands-on practice with prompt engineering

Item generation and initial pool creation

• Implementing LLMs for item generation
• Managing temperature settings and their effects
• Strategies for creating diverse and balanced item pools
• Quality assessment of generated items

Network psychometrics and item embedding

• Introduction to network psychometrics
• Text embedding techniques
• Exploratory graph analysis (EGA)
• Normalized Mutual Information (NMI) in item analysis

Day 2

Item reduction and redundancy analysis

• Unique Variable Analysis (UVA)
• Weighted topological overlap for identifying locally dependent items
• Iterative item reduction techniques
• Optimizing Walktrap algorithm step size

Stability analysis and final item selection

• Introduction to bootEGA for stability assessment
• Implementing stability thresholds
• Iterative stability analysis and item removal
• Finalizing the item pool

Project implementation and evaluation

• Designing a complete scale development project using AI-GENIE
• Implementing the full AI-GENIE pipeline
• How to use the AI-GENIE R package, and how to deal with OpenAI’s API and Groq’s API keys
• Evaluating results and comparing to traditional methods
• Discussion of potential improvements and future directions

“The instructor was very knowledgeable and helped participants with their questions on a number of topics. I appreciated the clarity and detail used in the prompts and how small changes in wording had big impacts. It’s exciting to consider the possibility of developing psychometric measures without the traditionally high costs of both money and time!”
  Deepalika Chakravarty, University of California

“I really liked how the workshop brought theory and practice together. It was eye-opening to see how LLMs can be used in the scale development process—from writing prompts to selecting high-quality items.”
  Omer Faruk Sen, Kirikkale University

“I can’t choose just one thing that I liked the most about this course! Everything was explained well and the slides were amazing. The presentation was top quality and the instructor knew what they were talking about.”
  Peter Lititaris, University of Athens

“I am highly excited about AI-GENIE and its potential application in the development of new psychometric measures. The course structure was well-balanced in terms of theoretical content and practical demonstrations. The material was explained very clearly and all questions were answered thoroughly.”
  Kirill Miroshnik, University of Trieste

“It was a very well-designed seminar. Difficult concepts were explained in an accessible way, and the tools provided seem easy to use. I will definitely utilize it in my work.”
  Alexander Rodríguez, Basque Government