Has your company’s internal training provided anything valuable to its employees? Has it helped to improve their performance and yield business results? Most online training reports may just give an assessment of “yes” or “no,” being complete or not. Yet, it’s still not informative enough to help you rate the level of training effectiveness. Actually, many F.Learning’s clients of Digital Training Departments are struggling with how to measure online training effectiveness more quantitatively.
The key is that you need to include some effectiveness metrics to make the training evaluation report more numerically measurable. Keep reading to figure out what these metrics are and how the metrics will help in an online training effectiveness evaluation.
What are common metrics to measure online training effectiveness?
There are a variety of metrics that an online training effectiveness evaluation should cover, based on your training objectives. The starter kit usually includes the following ones:
- Learner attendance rate: tracking how popular or favored a certain course is at your company.
- Training completion rate: showing the portion of learners who join and complete the course.
- Training dropout rate: tracking the number of learners who join but then decide to leave before completing the course.
- Pass or fail rate: the final results that are claimed for tests, quizzes, and course completion of learners.
- Average scoring: estimating how good the learners’ performance is on tests, quizzes, or final examination.
- Training experience satisfaction: showing how difficult or easy the learners perceive the courses, whether they enjoy the course or not. That helps to identify which factors need improvements.
- Post-training performance: assessing the training effectiveness by observing how employee performance has improved after training courses. They can be a work efficiency boost or an increasing number of sales, etc.
- Return on investment (ROI): comparing the financial value coming from the training results against the investment to achieve those milestones.
You have to consider your online training types to determine which metrics are the most relevant to make the evaluation. Whether training courses are synchronous, asynchronous, or blended? For webinars or online classes, for example, you should take the metrics indicating attendance and participation into account. Otherwise, in an asynchronous training course, you can trach the progression rate of learners throughout the course. The number of completed modules, time to complete, and average scores of final tests are some metrics to cover.
Now you have a list of suggested metrics. So when and how to use those metrics to measure training effectiveness?
In this article, we’d like to recommend the Kirkpatrick Model, which is among the most popular models for training. Let’s see which metrics you can implement with this training effectiveness evaluation model at every level.
Preparation Checklist to Power your Digital Transformation
Wondering if you miss anything in your preparation to digitize your training? This is just what you need.
How to measure online training effectiveness with the Kirkpatrick Model?
Dr. Donald Kirkpatrick, former president of The Association for Talent Development, first introduced the Kirkpatrick Model back in the 1950s. Even though the model was created a long time ago, it’s still applicable to almost all current online training courses.
Basically, the Kirkpatrick Model guides trainers and operators to measure training effectiveness following four levels:
- Level 1: Reaction
- Level 2: Learning
- Level 3: Behavior
- Level 4: Results
1. Level 1 – Reaction
Level 1 aims at discovering how learners feel about the training by getting their immediate feedback after the course. The point is to evaluate training materials, which are considered relevant and which need to be revised for better use.
Note that the stage is just about the reaction or learning experience of learners itself. We will not talk about how the training results in any outcomes or business impacts yet.
Which metrics to measure?
- Attendance or Participation rate;
- Training completion rate;
- Training drop-out rate;
- Training experience satisfaction
How to measure?
- Post-training survey, questionnaires, or online poll
Use a scaling or Likert method to set up all these survey questions for online training effectiveness for measurable results.
- In-person interviews with open-ended questions to get learners’ insights.
For example, did you feel comfortable participating? Did you like the venue, the presentation style, timing, etc.? How do you think you can apply the knowledge you received?
TIPS:
When adopting online internal training, you should implement a Learning Management System (LMS) to automate all the processes and management. In this way, you can integrate the tracking tool into every digital content to capture learner activities and behaviors.
Additionally, using training content in the form of videos is also a great idea to track the learners’ progress.
You can collect summary data from the LMS of how many learners viewed a video and how long they watched it. Or any added in-video notes and comments will show whether learners engaged with the content or not. Then, analyze the viewership and engagement data to find how you can improve for better content.
Take a look at:
2. Level 2 – Learning
At this level, we measure how employees understand and remember the provided knowledge or skills after the training. In other words, we identify what content and how much learners have learned, along with how much they have missed.
That means level 2 focuses on the amount of knowledge as the training result rather than learners’ perceptions and feelings.
Which metrics to measure?
- Progress and completion rates;
- Average score results;
- Pass or fail rate;
- Attempts & Answer Breakdown: In particular, that a high number of learners answer incorrectly to the same question can indicate a problem. The questions or requirements may not be clear, or the training content is not sufficient or misunderstood.
How to measure?
- Online quizzes or tests;
- Scenarios or simulation activities;
- Related-topic group discussions.
For a more objective assessment of learners’ progress, it’s highly recommended to use the test before and after the training. In this way, you can evaluate how much the learners improve throughout the training.
Moreover, also use interactive online training elements such as quizzes or mini-tests as recurrent checkpoints after each section to continuously recall what employees are learning.
Recommend reading:
3. Level 3 – Behavior
After assessing the Learning level, at level 3, we will evaluate the impact of training courses on employees’ performance. It’s critical to review if employees can apply intended knowledge in their workplace or not.
This stage of evaluation helps you to decide whether other training courses are in need to further support employees.
However, the lack of application on the stage does not necessarily mean that employees haven’t retained the training well enough. It can be caused by other reasons of resistance to change, organizational stagnation, workplace barriers, lack of motivation, etc.
Which metrics to measure?
To evaluate at the Behavior stage, you need to ask the following questions:
- Which modules of the course have learners put into work?
- How many learners can guide other colleagues with that knowledge or skills?
- How many learners are aware that they’ve changed their behavior?
How to measure?
- online self-assessment questionnaires for learners;
- discussion with department supervisors and managers;
- observation checklists to follow
- Data analysis from work processes (errors rates or calling time, etc.)
As a long-term process, behavior evaluation takes time to be comprehensive and effective. This stage should occur over three to six months after the end of the course.
Recommend reading:
4. Level 4 – Results
The final level focuses on the tangible training result, which shows how employees’ behavioral improvements affect the whole business. At this level, you’re measuring and getting a clearer sense of ROI for the online internal training program.
Which metrics to measure?
- Business development results
- Increased productivity and quality of work
- Employee retention
- Lower costs for training
How to measure?
- surveys to measure the perceptions of customers and stakeholders;
- comparing data on employee turnover and retention rate;
- analyzing sales and profits data before and after the training.
5 best practices to measure online training effectiveness
- Choose a reasonable number of metrics
- Identify KPIs before the training development phase
- Schedule data collection timeline in the training design phase
- Customize evaluation framework for each course
- Put your finding into practices
1. Choose a reasonable number of metrics.
It’s good to get different metrics for a detailed evaluation, but do not be overwhelmed with too many because the more metrics that you measure, the more information and data you have to work with.
So when choosing metrics to include, be selective. Which are necessary will depend on the types and topics of the internal training.
2. Identify KPIs before the training development phase.
You need to discuss with key stakeholders to know what their expectations are and which metrics are of high priority to them. This allows you to clarify what needs to be measured and the suitable method to measure effectiveness.
Recommend reading:
3. Schedule data collection timeline in the training design phase.
Building a data collection timeline keeps you always organized to manage stakeholders’ expectations. You can know exactly when and how you have to track and measure the effectiveness.
4. Customize evaluation framework for each course.
The Kirkpatrick Model is obviously a long-term and costly process. In practice, it’s not necessary to follow all its four levels to evaluate a particular training course. In some cases, you might determine that just levels 2 and 3 of the Kirkpatrick model, for example, make sense. So just go for those two levels.
Analyze all the training requirements and factors to make your best-customized framework for the training effectiveness evaluation.
5. Put your findings into practice.
This practice is definitely the most important and efficient one. Measure and evaluate, and then take action, such as making modifications and adjustments if necessary.
To sum up
Do not overlook the training effectiveness evaluation. Evaluation helps you to figure out whether the objectives have been met and how you can make the courses better. Focusing on improving employees’ overall performance also contributes to boosting the outcomes and success of your business. More importantly, measuring online training effectiveness has to be a continuous process, making employees always feel supported and motivated at the workplace.
Read more
Sean Bui, the founder and creative director of F.Learning Studio, is a respected leader in the e-learning and multimedia production industry. With over 10 years of experience, he has dedicated his career to helping organizations create engaging and impactful learning experiences.
Under his leadership, F.Learning Studio has grown into a trusted partner for organizations in the education, healthcare, and corporate training sectors, producing over 2,000 minutes of educational animation.
