A Ten-Stage Methodology for Complex Learning Solutions
Scott J. Warren, University of North Texas
Annette Fog, Globe Life
Janetta Boone, NASA
Brent Tincher, Lockheed Martin
Stephanie L. Robinson, University of North Texas
Decision Sciences Conference
Organizations struggle with developing effective training media that addresses:
Analysis → Design → Development → Implementation → Evaluation
Linear, phase-based approach
(Bichelmeyer, 2005; Wang & Hsu, 2009)
Analyze learners → State objectives → Select media → Utilize → Require participation → Evaluate
Media-focused methodology
(Reigeluth, 1999)
(Wang & Sun, 2022; Warren et al., 2021)
Origin: Jay Forrester's work at MIT
Analyzes complex systems by exploring how macro, meso, and micro level subsystems interact
(Forrester, 1996, 2007)
Valuable for understanding big-picture components but has limitations for detailed design guidance (Featherston & Doolan, 2012; Harrop et al., 2012)
Origin: Peter Checkland's research
Addresses "wicked problems" - complex, ill-defined challenges with multiple stakeholders and competing objectives
(Mingers, 1980, 2015; Checkland, 1981; Checkland & Scholes, 1990)
Integrating soft systems methodology with systems dynamics for rigorous corporate training media design and production
Engage in sense-making through data collection: performance analytics, surveys, interviews, observations
Create narrative describing what the problem is, organizing for proposed solutions aligned with identified challenges
Create common set of root definitions so all stakeholders communicate about the same structures with shared understanding
Work with stakeholders to build narrative model providing detailed explanations of system structure, subsystems, purposes, activities, interactions
Design team compares ideal structure with current state to determine which components should be changed, removed, added, or improved
Determine what to include and what is possible to implement successfully. Create performance goals aligned with business elements
Develop dynamic model (rapid prototype) of the training media showing how the product addresses identified problems. Create planning documents, draft media products, and user support materials
Document engineering specifications ensuring each media component is clearly defined for repeatability in future training media projects
End user implements the training media product in intended organizational settings. Business value and media effectiveness are tested against real world. Capture data on user perceptions and alignment with performance goals
Use assessment and evaluation data to iteratively improve the training media product or store knowledge for future media development projects. Build organizational capacity for addressing complex training media challenges
Goal: Develop immersive training simulation media using VR technology to teach complex cybersecurity concepts through engaging scenarios
Incidents driving need:
Financial losses from security incidents plus regulatory fines, reputation damage, and decreased customer confidence
Data collected: Security incident reports, help desk tickets, compliance audits, employee surveys, interviews, observational studies
Key finding: 78% of successful attacks involved human error or manipulation
| Attack Type | Percentage |
|---|---|
| Phishing emails | 45% |
| Social engineering (phone/in-person) | 23% |
| Improper password management | 10% |
Created CATWOE analysis defining Clients (employees/managers), Actors, Transformation (simulation creation), Worldview (scenario-based learning), Owner (Security & L&D), and Environment (corporate/remote/mobile settings)
Created board with security managers to identify critical competencies
Compared ideal security behavior model with current state
Determined desirable and feasible components given resources and skills
Created performance goals aligned with business objectives for measurable outcomes
Media Approach: Immersive VR simulation technology for training media
Rationale: VR tools allowed rapid prototyping of media, iterative editing of simulation scenarios, and realistic presentation of security situations
Budget consideration: VR development tools made creation of multiple media prototypes feasible for testing with stakeholders
Built comprehensive repository with engineering specifications, root definitions, and media design elements for organizational knowledge sharing and future media projects
Recognizes training media exists within complex organizational ecosystems where media success depends on integration with existing systems and culture
Ensures media solutions address real organizational needs through continuous involvement of content experts, designers, and end-users throughout media development
Iterative media prototyping identifies problems early when they can be addressed cost-effectively before full production
Systematic documentation creates organizational assets for future media projects and continuous improvement of media development processes
Links training media objectives to business outcomes, building evaluation into media design from the beginning
Well-defined processes support scaling successful media solutions to different contexts and audiences
Successful DSE adoption for media development requires:
Opportunities for integrating DSE with emerging media technologies:
As organizations face increasingly complex challenges, DSE offers a valuable framework for creating training media—from immersive simulations to interactive digital experiences—that truly serves both employee development and business performance goals.
Questions & Discussion