Automatic Item Generation and Validation: A Short Course

Automatic Item Generation and Validation via Network Integrated Evaluation (AI-GENIE) leverages large language models (LLMs) and advanced network psychometric techniques to streamline the item generation and validation process without the need to collect data in humans. (See the scientific preprint for technical details.) Traditional scale development is resource-intensive, time-consuming, and costly, often requiring extensive human expert intervention, and costly data collection for psychometric validation.

Recent advancements in AI and LLMs offer promising solutions to generate expert-quality text for scale items. The challenge lies in efficiently selecting and validating non-redundant, high-quality items that accurately represent intended psychological constructs, and that present adequate dimensionality (structural validity) AND item/dimension stability. AI-GENIE automates the entire process, from item generation to validation, enhancing efficiency and scalability in psychological assessment creation.

Previous research has shown that AI-generated items can create adequate psychological assessments, but the item selection process remains resource-intensive (with rounds of in-human data collection). AI-GENIE eliminates the need for extensive human expert involvement in generating, selecting, and validating items, potentially saving researchers significant time and money.

The methodology combines open and closed-source LLMs, generative AI, and network psychometrics to facilitate scale generation, selection, and validation. AI-GENIE is the first fully automated methodology to generate, assess, and validate the quality of AI-generated items for psychometric scales.

By the end of this course, you will be able to:

1. Understand the principles and applications of AI-GENIE in scale development.
2. Utilize various LLMs to generate new items using R.
3. Apply network psychometric techniques for item validation and selection in silico (i.e., without the use of human subjects) using R.
4. Critically evaluate the effectiveness and limitations of AI-generated items.
5. Design and implement a full-scale development project using AI-GENIE methodology.

This is a hands-on course with instructor-led software demonstrations and guided exercises. These guided exercises are designed for the R language, so you should use a computer with a recent version of R (version 4.1.3 or later) and RStudio (version 2022.02.1+461 or later).

To follow along with the course exercises, you should have good familiarity with the use of R, including opening and executing data files and programs, as well as performing very basic data manipulation and analyses.

If you’d like to take this course but are concerned that you don’t know enough R, there are excellent on-line resources for learning the basics. Here are our recommendations.

This course is perfect for researchers, marketers, or anyone interested in creating more efficient surveys or tests. No prior knowledge of AI or advanced statistics is required, but you should have:

• Basic understanding of psychometrics and scale development.
• Familiarity with R programming, like from a introductory seminar such as Introduction to R for Data Analysis, R for SPSS Users, or R for Stata Users.
• Knowledge of statistical analysis and concepts.

Day 1

Introduction to AI-GENIE and LLMs

• Overview of traditional scale development challenges
• Introduction to AI-GENIE methodology
• Exploration of LLMs: Gemma 2, Llama 3, Mixtral 8x7b, GPT 3.5, and GPT 4
• Ethical considerations in AI-assisted scale development

Prompt Engineering for Item Generation

• Principles of effective prompt design
• Few-shot prompting techniques
• Crafting prompts for item development
• Hands-on practice with prompt engineering

Item Generation and Initial Pool Creation

• Implementing LLMs for item generation
• Managing temperature settings and their effects
• Strategies for creating diverse and balanced item pools
• Quality assessment of generated items

Network Psychometrics and Item Embedding

• Introduction to network psychometrics
• Text embedding techniques
• Exploratory Graph Analysis (EGA)
• Normalized Mutual Information (NMI) in item analysis

Day 2

Item Reduction and Redundancy Analysis

• Unique Variable Analysis (UVA)
• Weighted topological overlap for identifying locally dependent items
• Iterative item reduction techniques
• Optimizing Walktrap algorithm step size

Stability Analysis and Final Item Selection

• Introduction to bootEGA for stability assessment
• Implementing stability thresholds
• Iterative stability analysis and item removal
• Finalizing the item pool

Project Implementation and Evaluation

• Designing a complete scale development project using AI-GENIE
• Implementing the full AI-GENIE pipeline
• Evaluating results and comparing to traditional methods
• Discussion of potential improvements and future directions