Optimising Human-AI Collaboration
Lessons from Scaffolding Theory
The increasing popularity of large language models (LLMs) like Chat-GPT has brought us into a new era of AI-assisted writing tools that can generate coherent sentences and paragraphs on demand. However, the challenge remains; how can we seamlessly integrate this technology into the human writing process? A key consideration is the level of "scaffolding" — the supportive guidance provided by AI to augment the writer's capabilities. This scaffolding spectrum ranges from low support like individual sentence suggestions to high support via complete paragraph generation. The level of support that yields the optimal balance between human agency and AI augmentation is not yet well understood.
A recent study by Dhillon et al. (2024) provides insight into this by systematically evaluating how varied AI scaffolding impacts writing quality, user satisfaction, cognitive load, productivity and more. Their findings reveal important nuances and repeatedly underscore the need for human-centred, adaptive scaffolding approaches.
Scaffolding in Learning
The concept of scaffolding has deep roots in education theory. Bruner (1975) pioneered this metaphor, describing scaffolding as the temporary supportive structures given to learners to help them achieve skills they cannot yet independently master. Think of this like training wheels on a children’s bike. They temporarily help them keep the bike steady as they’re learning to balance it. Some key principles outlined by Bruner were the following:
Support: The instructor offers support in various forms, such as hints, encouragement, or direct teaching, to assist the learner in achieving a higher level of understanding or skill.
Collaboration: The learning process is a collaborative effort between the learner and the instructor. The instructor's role is to facilitate learning by adapting the support provided as the learner's competence increases.
Problem-solving: Scaffolding often focuses on problem-solving tasks. The learner is encouraged to engage with problems that are just beyond their current capability, promoting cognitive development and understanding.
Gradual removal of support: As the learner becomes more proficient, the support is gradually removed, a process known as "fading." This encourages independence and ensures that the learner does not become overly reliant on the support provided.
Learning sequence: Bruner believed that learning should proceed in steps, with each step building upon the previous one. Scaffolding helps to organise these steps in a way that makes the learning process more accessible and effective for the learner.
In practice, scaffolding can take many forms, depending on the learning context and objectives. Vygotsky's (1978) highly influential idea of the "Zone of Proximal Development" (ZPD) complemented this. The ZPD represents the gap between what a learner can achieve unaided versus with guidance from a capable instructor. Optimal learning is thought to occur when scaffolding is calibrated to challenge a student within their ZPD - not so little support that they struggle, but not so much handholding that they don't actively learn.
In writing pedagogy, similar ideas have been explored. Strategies like feedback, peer review, portfolios and explicit instruction have all been studied as means of providing guided practice and assisting skill development in various writing contexts.
Generative AI now opens up new ways to scaffolding writing, creating unique research opportunities. Dhillon et al's work described below contributes to this growing body of inquiry into AI-supported writing instruction.
Evaluating Scaffolding from LLMs
To systematically evaluate scaffolding from LLMs in co-writing tasks, the researchers developed a custom AI-integrated tool using the GPT-3 language model. Their study had three conditions:
No AI assistance (control)
Next-sentence suggestions (low scaffolding)
Next-paragraph suggestions (high scaffolding)
Using a within-subjects design, each of the 131 participants experienced all three conditions in different sequences to support analysis of order effects. For each argumentative writing prompt, participants wrote a minimum of 250 words, optionally accepting AI suggestions of sentences or paragraphs depending on the condition.
To comprehensively assess the impact, the study considered diverse outcome variables such as writing quality (via human scoring, lexical analysis and text coherence metrics), user experience (NASA-TLX cognitive load, satisfaction, sense of ownership), productivity (words per unit time), AI's influence on content, and more. This multi-faceted evaluation provides a deeper understanding of the complex human-AI collaborative writing dynamics.
A key takeaway from this study was the U-shaped relationship between scaffolding level and writing quality/productivity. Compared to the control, low (sentence-level) scaffolding decreased output quality, whereas high (paragraph) scaffolding significantly improved it, especially benefiting non-regular and less tech-savvy writers.
This idea aligns with existing scaffolding theories, which suggest that minimal support — like offering just single sentences — might not give enough context for effective learning, according to Pea (2004). This approach could interrupt a writer’s flow, as it requires them to frequently switch between coming up with ideas and evaluating them, leading to a fragmented thought process. On the other hand, providing paragraph-level suggestions can ease the cognitive load by supplying a cohesive narrative structure. This allows writers to seamlessly integrate their own ideas, while still retaining the freedom to be creative. Interestingly, the study found an opposite trend for user satisfaction and sense of ownership — both decreased as AI assistance increased, despite quality improvements.
The findings indicate an expertise asymmetry, where high scaffolding is more beneficial for novices but not substantially advantageous for professional writers. This aligns with Vygotsky's ZPD concept - skilled experts may find paragraph-level AI suggestions unnecessary or restrictive compared to less-practiced writers.
Another insightful observation was how transitioning between scaffolding conditions affected writing performance. Moving from sentence to paragraph suggestions led to quality improvements, suggesting a beneficial scaffolding progression where initial low support familiarises users with AI capabilities, preparing them for more intensive AI-human collaboration later. However, reverting from intensive AI writing assistance back to no support resulted in performance drops. This shows the importance of gradually fading AI scaffolds to avoid negatively affecting users' independent skills - a principle from scaffolding theory about progressively transferring responsibility to learners.
Suggestions for UX Professionals
For UX professionals working on human-AI interactions across various fields, insights from AI-assisted writing offer guidance for creating more effective and user-friendly interfaces. These insights lead to a couple of key strategies:
Understand who your users are: Conduct research with your users to understand their profiles, existing knowledge, and needs.
Adaptive user interfaces: Design interfaces that adapt to the varying levels of expertise among users. This could mean dynamic suggestion systems that measure user comfort and skill over time, offering more or less scaffolding as needed.
Personalisation and customisation: Implement features that allow users to personalise the level of AI assistance. This caters to individual preferences and writing styles, enhancing the user's sense of control and ownership over the writing process.
Gradual scaffolding adjustment: Integrate mechanisms for gradually increasing or decreasing AI support based on user interaction patterns. This helps mimic the natural learning progression, fostering user growth and independence.
Foster agency and ownership: Users should feel in command of the AI interaction, capable of customising their experience and overriding AI suggestions as needed. Ensuring users retain control over the interaction encourages a partnership with the AI, rather than a dependency.
Design for continuous learning: Incorporate user feedback mechanisms to refine AI suggestions over time, making the tool more responsive to the evolving needs of the user.
By focusing on these aspects, we can significantly enhance user experience, making AI tools not just helpful but indispensable in achieving goals across a diverse range of activities.