Generative AI in Human-AI Collaboration: Validation of the Collaborative AI Literacy and Collaborative AI Metacognition Scales for Effective Use

Checked 30 Sept 2026. Non-commercial or no-derivatives license: full text kept internally; only metadata and abstract are indexed and used.

Advancements in AI’s conversational capabilities and situational awareness mean that now, humans work with AI collaboratively. However, these developments affect the skills needed to use AI effectively. In this study, we address the need to update measures of AI-related knowledge and skills to reflect the collaborative capability of advanced AI tools by developing and validating two new scales focusing on collaboration and metacognition. A survey of 292 users of collaborative AI tools was conducted. Both the Collaborative AI Literacy and Collaborative AI Metacognition scales showed good internal consistency and predictive validity. Structural equation modeling supported their convergent and discriminant validity. Both measures correlated with users’ assessments of the benefits from working with collaborative AI tools. As predicted, Collaborative AI Metacognition explained significant variance beyond that explained by general Metacognition. These validated scales provide an important resource for assessing and researching knowledge and skills for working with Collaborative AI tools.

• Two recent scale-development papers converge that legacy instruments miss what matters in AI collaboration: Dai et al. 2026 "redefine student agency as a process of human-AI coagency grounded in adaptability, epistemic responsibility, and distributed inquiry," and Sidra & Mason 2025 show "Collaborative AI Metacognition explained significant variance beyond that explained by general Metacognition." Trace →

• Supervision and doctoral education