Assessing And Testing Training Effectiveness

There is no question about it. Training and development is gaining attention in Corporate America. The industry is growing and changing to achieve a variety of goals, from satisfying immediate employment skills shortages to meeting organizations' long-term strategic needs. According to a survey conducted by Chief Learning Officer magazine and Fairfield Research Inc., enterprise firms in the United States spend, on average, $3.7 million every year on learning and training. Total spending on corporate learning by enterprise companies is forecast to increase to $11.8 billion—a projected 4.3 percent increase in 2003.

Although many CEOs value training because they believe it strengthens the organization and serves as a retention tool, not many are clear on how to measure the return on the investment (ROI). With increased pressure to justify expenses, CLOs are looking for ways to show improved bottom-line results.
"We are seeing more and more CLOs link the development of a training program to the strategic direction of the company," said Linda Gookin, senior consultant at the Hay Group, a professional services firm that helps companies worldwide develop their employees. "To maximize the effectiveness of a training program, an organization needs to use ongoing assessment to establish learning outcomes, link them to a performance plan, define measures and finally evaluate learning—and this must be an integral part of the corporate strategic plan."

There is a huge gap between the learning programs that produce results and many of those being implemented today. Certainly, there are many reasons this gap exists, including failure to link training to core business strategies, uncertain business and economic conditions, lack of acknowledgement or reward for training accomplishments and failure to make training an integral part of an employee's job. These and many other factors can hinder the development of a learning program that produces results.

However, one of the main reasons CLOs have a difficult time proving the value of a corporate training program is that testing and assessment are not fully integrated into the program. All too often, testing and assessment are viewed as separate from the learning process and as afterthoughts to the development and implementation of the learning program. But without ongoing integration of testing and assessment into a learning program, there is no reliable way to know how much learning has taken place, if any at all. A renewed focus on testing and assessment is necessary to improve the quality of instruction, the effectiveness of the curriculum, and the validity of the outcomes.

When testing and assessment are held completely tangential to learning, the entire organization is affected by the negative impact on the learning process, the learning outcomes and the ability to measure ROI. Only when we hold learners and the learning process accountable to themselves will we see verifiable proof that learning has taken place. When you are able to fully integrate testing and assessment into the development, management and evaluation of your organization's learning system, you will be able to spend your training dollars knowing—not hoping, not guessing, not assuming, but knowing—that you are getting something in return.

Certainly, an organization's advertising team would not set aside money for a print ad campaign without knowing something about the publications in which the advertisements will appear. The team would want to know about the audience that reads the publication. Creative ideas would be tested using a variety of techniques such as focus groups, image studies and positioning research. Copytesting would be done to diagnose the effectiveness of the proposed ad. All of this would take place before the publication hits the newsstand. Once the advertisement is printed, market-tracking evaluations would be conducted to assess the impact on sales. Changes in customer attitudes and behaviors would be measured. The advertising team wouldn't dream of ignoring this information because it understands the value and usefulness in determining how to allocate resources.

Imagine that you have been asked to implement a new enterprise-wide training program to ensure that each employee masters a new skill and can successfully demonstrate competency within a month. Maybe you have five employees, maybe you have 50,000, but the idea is that all employees have to master the skill. Would you ignore data that told you which employees knew the information already? What about data that told you how your employees learned most efficiently? How about data that told you what to change about your current learning programs to make them more effective? Of course you wouldn't, you would welcome this information with open arms. Without this information, your training program will never achieve its full value.

No organization can afford to take a hit-or-miss approach to training. And no organization can afford training that doesn't pay off for the business.
Oracle Corp., the world's largest enterprise software company, was searching for a way to enhance the reach of its certification program and increase the value of the curriculum offered at its 125 Oracle University Training Centers located around the world. Oracle sought to analyze options and develop a customized, technology-enabled testing solution.

"We were able to more broadly link our certification training with Internet-based testing solutions to measure the impact of the training and to expand the value and reach of our certification programs," said John Hall, senior vice president, Oracle University. "We wanted to maximize the learning experience for Oracle candidates, and in order to qualify and quantify the effectiveness of a learning strategy, effective assessment must be in place."

Because of the implementation of the new testing solution, Oracle University's learning system has unparalleled efficiency and scalability. Assessment results are instantly available to the instructor and give the learner immediate feedback, verifying knowledge and competency levels or pinpointing additional training and development needs.

Testing and assessment are keys to providing the information needed to build and sustain learning initiatives. Testing and assessment are absolutely integral to the learning process—just as content development, instructional design and methods of delivery are integral. The proper use of testing and assessment vehicles—prior to learning, throughout learning and after learning takes place—will give you the means to provide standards of consistency, achieve accountability and ensure that your organization gains value from training investments.

Well-designed and well-integrated testing and assessment strategies provide very real benefits to the learner, the instructor and the sponsoring organization. Training empowers workers. Testing and assessment empower training. It is a fundamental law of learning: People want to know that they have accomplished a task, that they have achieved the objective of the training. Every person needs to know what is expected of him, what abilities are needed to be successful on the job. And employees must know that they will be held accountable to these expectations. Good tests, based on consistent standards, provide one objective measurement of the skills and knowledge of the individuals who take them. Employees can return to their jobs with the added skills, increased morale and the knowledge that their employers value them enough to invest in their development.

An integrated assessment program also provides insight and development goals for the instructor and course design. Assessment is a diagnostic tool that ultimately helps the learners confirm what they have learned and helps the instructor plan for future learning or remediation. A well-built test is a quick and objective way to identify the strengths and weaknesses of the delivery of the learning program.

Some of the most profound benefits integrated testing and assessment programs can provide are to the organization. Think about the four levels of evaluation: participant reaction, participant learning, behavior change and impact on the business. By incorporating these systematic evaluations consistently into a training program, you can prove that learning within the program has a significant impact on the driving concerns of the business—revenue, profits, employee retention, market share and, ultimately, customer satisfaction. Testing and assessment as part of the learning process is efficient and reliable and provides immediate results.

You may argue that you do not have the time, money or incentive to implement an integrated testing and assessment program into your learning system. If you take training seriously, you need to take the evaluation of training seriously by asking whether the training works. If you want employees to walk away from a training program with actual job skills that can translate to benefits for the employee and the organization—not just a certificate and a fancy binder—you need to provide proof of effectiveness.

It is time to require staff and vendors to embed testing and assessment into each curriculum and learning opportunity. You should be asking your e-learning providers how they do this, and you should be evaluating them based on their answer. It is time to encourage employees to embrace testing and assessment as a tool that promotes and demonstrates continued learning and performance improvement.

So, where do you begin? It is important to remember that an assessment instrument is most effective when used as part of the total training system, but it is likely to be very counterproductive when used inappropriately. Often inappropriate use comes from not having a clear understanding of what is being measured and why. You can begin integrating testing and assessment into your learning programs by identifying what it is you want to accomplish with the assessment program in order to select the appropriate tools to achieve your mission. Just as you wouldn't use a knife to eat tomato soup, you would find a certification test quite useless in measuring the effectiveness of the instructor. It is absolutely critical to have a clear understanding of what needs to be measured and for what purpose.

Sun Microsystems Inc., a leading provider of hardware, software and services for network computing solutions, has also developed a worldwide corporate learning system. In July 2000, Sun began the implementation of Sun Sigma, the company's adaptation of a worldwide business phenomenon called Six Sigma—a comprehensive program for building and sustaining business performance, success and leadership. The company used the Sun Enterprise Learning Platform to manage the training of 39,000 globally dispersed employees on Sun Sigma concepts.

To date, more than 24,000 employees have been trained on the fundamentals of Sun Sigma. More than 1,000 executives have been trained as Sun Sigma "Champions" through online learning and classroom experiences. And more than 100 employees are Sun Sigma "Black Belts," signifying the highest level of leading Sun Sigma projects.

To effectively deploy a global learning program for thousands of employees, the company needed to integrate testing and assessment tools throughout the learning process. Program coordinators began by identifying the desired skills for a successful outcome and then identified where the gaps needed to be filled. They measured the performance of employees as they completed learning segments. They rigorously tracked results of on-the-job improvements through verifiable cost reduction, product improvement and customer satisfaction.

"In today's corporate world, most learning takes place outside of a managed learning environment, and the ROI for learning is universally overlooked," said Mike Wenger, senior director, Center for Learning Innovation and Effectiveness at Sun Microsystems. "Companies need to design an overall plan for bringing learning systems into the corporate vision. Testing and assessment is a key component to building accountability into the system. Organizations need a clearly developed assessment strategy with measured results to achieve this desired accountability."

How effectively organizations can measure the results of their training objectives has caught the attention of the financial community. What a company knows—what its employees know—and what those employees do with that knowledge has real value. Yes, knowledge is intangible, but much of today's corporate value is associated with intangible factors. The more effectively you can measure and prove knowledge, the more likely you can turn learning into a tangible corporate benefit.

Consider this: A four-year study by the American Society for Training and Development (ASTD) shows that firms that invest $1,500 per employee in training compared to those that spend $125 experience on average 24 percent higher gross profit margins and 218 percent higher income per employee.

In today's economy, all expenses require justification. You are faced with a daunting challenge. You need to implement a learning program that will successfully educate employees on key objectives, provide course designers and facilitators with timely feedback to improve quality and offer senior management bottom-line justification. Furthermore, you must implement twice as many programs with half as many resources as you had last year. It seems an insurmountable task. Consider that the complete integration of testing and assessment tools and programs can give you the data and feedback you need to prove not only that your program is meeting its objectives of training employees effectively, but also that the organization is receiving some real, tangible benefits because of its existence.