WP6: Application of OpenScout in large-scale Reference Scenarios and Evaluation

Main Objectives

The main objective of WP6 is to to create large-scaled best practice or so-called reference scenarios for the OpenScout services. Within these scenarios, the use of the services shall be demonstrated and evaluated on a very large scale for many users. The scenarios will be leveraged over the whole community and by this multiply the use of OpenScout services already within the project runtime to a broad number of users.

Task 6.1 Good Practice Scenario Collection

Many online collaboration systems fail to deliver the value expected. Thus one of the first tasks of WP6 was to undertake a study to try to understand why some systems fail and others succeed, in order to guide the design, community involvement and organizational aspects of OpenScout.

We collected the actual experiences of OpenScout project consortium partners with real life, productive, skill-based learning environments in higher education and enterprises. Partners provided information about eighteen experiences with Learning Management Systems (LMS) and Collaborative Environments (CE).

From these and related research, we derived practice-relevant recommendations to guide and improve the development of learning management systems and collaborative environments such as OpenScout.  This study was published in OpenScout Deliverable D6.1 Good Practice Scenario Collection.

Task 6.2 Evaluation & Case Study Validation

In this task we define validation and evaluation scenarios to enable feedback, experience exchange and optimisation. This task also includes the continuous project evaluation. The “Refined Evaluation Plan” deliverable D6.2.1 has been completed and presents the overall evaluation methodology, the initial prototype evaluation plan, and the evaluation plan for the project as a whole.

The OpenScout evaluation plan is part of a continuous improvement process. Our overall evaluation methodology consists of five main steps (see Figure below):

1. Define evaluation objectives: WP6 will set objectives in collaboration with the other work packages.
2. Select methodology: Methods will be defined based on the evaluation objective. The selection of methods will be adapted according to the changing requirements, progress and results of the project.
3. Collect data: Data will be collected during the entire project in cooperation with all WP partners.
4. Analyze data: In this phase data will be interpreted by WP6 and other WP participants who have expert knowledge 
5. Provide feedback: Suggestions for improvement will be made.

Evaluation Methodology Overview

As OpenScout is targeted at accelerating the use, improvement and distribution of OER, a user-centric approach is used to achieve project aims. User involvement is crucial from the very beginning of the project. At all stages of the evaluation stakeholders will be involved in order to provide continuous feedback and assessment as well as to refine objectives.

User feedback will be collected during three Evaluation cycles:

Cycle 1 - Formative Evaluation: Before the first prototype becomes available, the very early validation with users will be done mainly with non-technical OpenScout project members.  This formative evaluation is aimed at guiding the design process.

Cycle 2 - Pilots: Once the first prototype is ready, the Partners will run Pilots in real contexts with users and organizations.  Their feedback will be used to improve the prototype.

Cycle 3 - Large Scale Trials: In this cycle, large-scale adoption and improvement, contents will be improved, adapted and localized by different user groups.  Users will be encouraged to work with the learning contents and tools as well as to evaluate further versions of the OpenScout prototype.

Pilots will be run in Higher Educational Institutions including Universities, Business Schools, Research Centres and Vocational Training Centres, and various sizes of Enterprises. There are two core user groups, teachers (e.g. professors, trainers, instructors, coaches) and learners (students, executive education participants and SME owners/employees). Feedback will also be obtained from those who counsel these groups, namely librarians and consultants.