Learning with Vince =)

Assignment 3- Additional part
The evaluation of an e-learning course/program becomes essential in understanding an e-learning course’s objectives, outcomes, effectiveness and application when teaching or educating. Reeves (1993, p. 97) highlights evaluation as an important component in judging any course ‘it may be used for a variety of functions such as needs assessment, refining goals, improving products and estimating costs’. In an organisational setting such as BMW’s international business sector (BMW, 2007), e-learning course can provide interactive and engaging learning that would cater to all staff ages and learning styles evident in organisational practice of JISC (2007). It can actively promote distance learning and establish a foundation or extension on learning a language synchronously; simulating visual and audio tutorials (Boyd, 2002).

Learning a language online can prepare staff to communicate on business trips or through international communication meetings based on training in practice with JISC (2007). However in order to promote effective learning practitioners must measure and outline the outcomes the course aims to achieve. By using evaluation criteria practitioners can breaks down the course components to establish a framework that can understand the engagingness, relevance and user friendliness (Athanasou, 1998). The evaluation criteria can not only be used as a measure of outcomes that defines competencies that staff members aim achieve but can be used to record advantages and disadvantages the staff/ learner’s experiences. The criteria tool allows practitioners such as facilitators or managers to monitor learning difficulties and challenges to assist in problems that staff may encounter. Additionally this acts as a tool that develops constructive feedback for improvements such as additional support material or changes in course design to suit different individual learning styles (Boyd, 2002).

In developing an evaluation criteria for an online language course such KSB World Radio (2007) ‘Let’s learn Korean’ (Korean online language course) practitioners must measure the effectiveness which aims to educate or prepare staff international business communication (JISC, 2007). The key areas to be assessed would include a measure on:

o General course information- name of course, estimated time for learning course, actual learning time for course, and origin of course
o Content- what you have learnt and what course teaches?
o Quality of delivery/Presentation – how well information was delivered?
o Relevance- how well the learner could apply skills or knowledge practically?
o Engagingness- interests to learner?
o Level of Difficulties encountered- what were they?
o Level of Technology Accessibility/ Navigation – how easy was it to access and use?
o Level of Assistance/ Support- Was feedback that was provided? What type?
o Learning dynamics- different skill levels beginner, intermediate and advanced?
o Additional comments/ Improvements on course design/ content

Additionally the evaluation criteria would include a section for additional comments explaining how or why (reasoning) for the staff member allocated grade to different areas. The assessment criteria would be a survey involving a grade system ranging from the 1-3; 1 being poor, 2 being average, and 3 being excellent which has been adapted by Athanasou (1998) frameworks for evaluating effectiveness of technology. The evaluation would therefore contain criteria based on numbers ranging from 1-3 with the additional comment section for constructive justification or reasoning for providing each grade (Boyd, 2002). Similarly it would define each category being assessed; identifying ‘ideal’ levels of learning achievements and sections that would provide insights on how the course could be improved, based on Athanasou (1998) frameworks design for evaluating effectiveness of technology.

In applying the evaluation criteria (survey) to KSB World Radio (2007) ‘Let’s learn Korean’ online learning program practitioners can highlight the general aspects that the course aims to provide. For example the estimated time could be 4 hours however this may not take into account practice time, time for revision or learning difficulties making the course greater than 4 hours. The content component compares the individual’s learning outcome with the actual outcomes of the course. It may measure or predict what has been individual is capabilities of learning and establishes what the learner has not managed to achieve this based on Kirkpatrick (1994) four levels of evaluation including a measure on reaction, learning, behaviour and results. The KSB World Radio (2007) ‘Let’s learn Korean’ thus is broken down into 3 parts, if the learner identifies learning part 1 and part 2 up to chapter 10 the evaluation criteria can deduce that the whole of part 2 was not completed and part 3 needs to be worked on based on Athanasou (1998) frameworks for evaluating effectiveness of technology.

The quality, relevance and engagingness test measures the way the content was delivered and how effective it was to the learner. This breaks down whether the skills can be used practical used to addresses the design of the course Kirkpatrick (1994) focusing predominately on progression of learning and the individual’s experiences. For example KSB World Radio (2007) ‘Let’s learn Korean’ uses the tutorial’s visual and audio components such flash dialogue for learning; this outlines the relevance that can further allow the learner to demonstrate what he/she has learnt. This identifies the components that motivate the learner such as animation or interactivity based on Najjar (1996) understanding of interactivity which emphasizes the learner and the learning systems mutual action.

The levels of difficulties, technology, accessibility, navigation and assistance and support target the course structure, analysing the interactivity of the learner which can be used to assist in understanding changes in the program and areas that need assistance. Additionally this shows the challenges the learner can encounter which can be a way to target problems such as limited feedback or technology restriction on obtaining information (Herrington & Oliver, 1997). Learning dynamics thus become crucial aspect in evaluating a course as it can help develops a degree of expertise that can allow the learner to progress further making the skills evolve from basic to advance. This can becomes a measure of two components; a measure of the core skills obtained and what the learner has progressed to obtain (additional skills) can influence the measure that provides feedback on how the course can be made more challenging or less difficult; understand how to emphasis the content promoting retention of information (Stemler, 1997) . This can be combined with additional comments; improvements on course design content to specifically target certain aspects to promote improvement.

In evaluating my personal insights and individual experiences of ‘Let’s learn Korean’ KSB World Radio (2007) I found the course needs an additional component that targets feedback. I would recommend active feedback which could facilitate pronunciation as well as progress in order to target learning difficulties that can hinder the Korean language conversation component. This can be in the form of forums, questions and virtual programs that document progress and voice mechanisms such as speech for correction. Additionally I would suggest more exercises for practices such as reading and writing exercises that could be integrated into the tutorial. Overall the course time for learning needs to be greater than 4 hours as different learners can respond differently to the pace of learning. Finally recommendation for the online language course would be a greater diversity in accessing material this can include learning through mobile phone application such as conversation files, interactive games to simulate learning and a combination of personal contact with an educator, practitioner or friend that could actively provide or facilitate constructive feedback to clarify language pronunciation, writing and reading. These evaluation components and improvements are based on Stemler (1997) means to promote greater understanding and retention of information and skills whilst has been developed through Kirkpatrick (1994) framework for understanding reaction, learning, behaviour and results.

In Conclusion evaluation criteria’s become an essential component in understanding the learner’s experience. In an organisational context such a training BMW staff to communicate languages with international clients (BMW, 2007) , evaluation criteria’s establish guidelines for evaluating the learner’s outcomes or objectives that the course aims to achieve. The evaluation criteria (survey) tool can test or measure general course information the name of course, estimated time for learning course, actual learning time for course, and origin of course, the content of what the course teaches, the quality of delivery or presentation, the relevance to the learner (whether it can be used practically), engagingness, interest to learner, level of difficulties, technological accessibility, navigation, assistance or supporting tool, learning dynamics (skill level) and overall recommended improvements. Through evaluating the course content practitioners can measure the level of skills obtained and provide constructive feedback to improve the learner’s experience (Boyd, 2002). This can be through additional resources and supporting tools such as corrections or test; targeting challenges or difficulties and highlighting e-learning components that motivate the learner. For example interactive learning through animation or audio can motivate or suit different learning styles while set a pace. In the Korean online language course ‘Let’s learn Korean’ KSB World Radio (2007) feedback and testing through exercise and correction actively improves learning content; emphasizing Stemler (1997) concept of effective learning and retention.

Evaluation criteria tool [Based on individual experience with ‘Let’s learn Korean’ KSB World Radio (2007)]

General course information
Name of course: ‘Let’s learn Korean’

Estimated Time for learning course: 4 hours
Actual Time for learning course: 4 hours +
Origin of course: Websites ‘Let’s learn Korean’ KSB World Radio (2007)]

Content- what you learnt/ what the course teaches?
Outcomes achieved- Part 1 and Part 2, chapter 12 basic conversation
Course outcomes-Understanding the Korean language fundamentals and basic and advanced conversation

In the following identify the types of grade for each area and specify why?
Poor-1 (Failed to provided area or was limited)
Average-2 (Satisfactory but could use improvement)
Good-3 (Contained helpful tools for area and needed none of next to any improvement)

Quality of delivery/Presentation – how well information was delivered?
Grade: Good
How/ Why?
Used both audio and visual aspects to entertain me.

Relevance- how well the learner could apply skills or knowledge practically?
Grade: Average
How/ Why?
Restricted by people I can have conversations with (application).

Engagingness- interests to learner?
Grade: Good
How/ Why?
Used both audio and visual aspects to entertain me.

Level of Difficulties encountered- think about what were they and why they were difficulties?
Grade: Average
How/ Why?
No feedback, but had audio which could be imitated.

Level of Technology Accessibility/ Navigation – how easy was it to access and use?
Grade: Average
How/ Why?
It lagged my computer.

Level of Assistance/ Support- Did the course provide feedback? Outline what type?
Grade: Poor
How/ Why?
No active feedback received.

Learning dynamics- Did the course have different skill levels beginner, intermediate and advanced?
Grade: Good
How/ Why?
Had basic, intermediate and advanced.

Additional comments/ Improvements on course design/ content:
More feedback and greater interaction in which learner can set the pace.

Download Link to blank Evaluation criteria tool: http://any1wish.edublogs.org/files/2007/11/evaluation-criteria-tool-blank.doc 

Reference List
o Athanasou, J. A. (1998). “A Framework for Evaluating the Effectiveness of Technology-Assisted Learning”, Industrial and Commercial Training 30: 3, pp. 96-103.
o Herrington, J. & Oliver, R. (1997). Multimedia, magic and the way students respond to a situated learning environment. Australian Journal of Educational Technology 13: 2, pp.127-143.
o Kirkpatrick, D. L. (1994). “Evaluating Training Programs”, The Four Levels, Berrett-Koehler Publishers.
o Najjar, L. J. (1996). Multimedia information and learning. Journal of Educational and Multimedia and Hypermedia 5: 2, pp.129-150.
o Reeves, T.C. (1993) Evaluating interactive multimedia: Multimedia for Learning, Development, Application and Evaluation, Educational Technology Publications, pp. 97-112.
o Stemler, L.K. (1997). Educational characteristics of multimedia: A literature review. Journal of Educational Multimedia and Hypermedia 6: 3/4, pp. 339-359

Websites
o BMW (2007) , last updated: 2007 <http://www.bmw.com/com/en/insights/corporation/mgmbh/individual.html> Accessed: 29th October 2007
o Boyd, S. (2002) ‘Tips to Make E-Learning Stick’, last Updated: 2002 <http://www.learningcircuits.org/2002/may2002/elearn.html> Accessed: 25th October 2007
o JISC (2007) ‘Understanding your practice’, last updated: 2007 <http://www.elearning.ac.uk/effprac/html/e_practice.htm> Accessed: 27th October 2007
o KSB World Radio (2007) ‘Lets Learn Korean’, last updated: 2007 <http://rki.kbs.co.kr/learn_korean/lessons/e_index.htm> Accessed: 19th October 2007