Using Learner Segmentation in Online Hospitality Training Programs

This post was originally written by Peter Matamala who works with our friends at Matchstick Inc. We're cross posting this post with his permission. We hope you find this content useful. Let us know what you think in the comments below! 

The core objective of any quality training program is to create relevant and meaningful content that drives performance results and positive change for the learner and the organization. To achieve this, one must take into account company culture and employee profile. This is a critical consideration in today’s hospitality workplace which is a vibrant collage of - cultures, languages, experiences, beliefs and age groups.  Unfortunately, the diversity we so value creates complexities and can prevent training programs from achieving their desired outcome.  Our job as instructional designers is to accommodate all learner needs and acknowledge this diversity in the hospitality training programs we create.  In this article we will explore how instructional designers can leverage the organization’s data about our diverse organizations to design more relevant and targeted training.

Obtain your Organization’s Data

The first step in tackling the challenge of diverse learning needs is to obtain an understanding of the learner requirements.  It is essential that during the assessment phase of your project, instructional designers 'peel' back the layers on your training audience to understand their needs better.

Today’s organizations are data rich, collecting employee and technology data from many sources and making it available to various departments.   The instructional designer that successfully evaluates and assembles this data can create a vivid picture of the organization’s learner landscape. Consider the data available in a typical IT department.  End user computing and security teams would almost certainly have data pertaining to employee devices, browsers, and wireless access at learner locations.  This useful information allows your ID to leverage existing technology, determine optimal learning modalities and implement effective design techniques.  Human Resources will likely be able to provide learner location, language and job level among many other employee attributes, thereby providing key information that could influence translation costs, accessibility to training content and desired training topics.

The designer needs only to assemble this data to paint a picture of the learner population.  Let’s now take a look at how that would be accomplished.


Create a Matrix

After obtaining learner related data, begin assembling it into a matrix.  Choose a primary attribute and compare that against the other attributes and capabilities. For example, take Job Role and compare against computing device, location and language.  Use spreadsheets and pivot table analysis to filter and summarize the data.

The output should look something like the table below.  Notice we have placed the role on the y-axis and learner attribute on the x-axis.  


Where x and y intersect, we show the counts of learners meeting that criteria.  Use of a spreadsheet and pivot table analysis will make this task much easier.  

Analyze the Matrix

Once you have created your matrix, take a look at the data and volume counts and begin to develop some core and outlier requirements for this online program.  In our matrix example a few things stand out:

1.     Heavy use of mobile devicevs. PC's

2.     First language of this training population is predominately the local language.

3.     In comparison with the total population, the number of learners with an advanced degree is about half.  

As a response to finding #1 noted above, the designer would absolutely need to consider designing for mobile.  In finding #2, voice overs and on screen text would need to be in the local language and could impact on screen text and voiceovers.  For finding #3, the matrix shows that there are large portions of both manager and non-managers who have advanced degrees.  

A savvy Instructional Designer can take the analysis further to determine mobile device type, types of languages and degree levels of the manager/non-manager bases.  This more detailed assessment and review will help further refine the training and, based on the above, allows the designer to confidently design a mobile program in the local language.  Perhaps the training should be offered in manager and non-manager versions.  In instances where there are large percentages of foreign languages spoken, a specific language matrix can help prioritize which languages are most widely spoken and would be in scope for translation.Delivering the Online Training

For the eLearning program outlined in our example above, the instructional designer can use the matrix to create segments of learners.   Building segmentations can be a great tool in determining how best to deliver and deploy the training.  The segment analysis of this data shows an array of learner device and PC access.  We also know that there are various job roles in the learner population. Finally we also know the HR data can show the number of learners in certain regions or locations.

With this information at hand, the instructional designer can work with the LMS team, training managers and regional staff to deploy the online training in a targeted and deliberate fashion.  For example, designers might use this information to deploy to the segments of:

  • Managers only
  • Non-Managers only
  • Back office staff
  • Mobile devices users
  • PC users
  • Location
  • Language

Designers could use this data to deploy to managers first so they can take the training prior to their staff to encourage top-down support.  Perhaps the operations teams want to avoid training deployments to back office staff during quarter end or year end activities.   Knowing which learners have only PC’s or mobile devices can help the deployment team target instances where a learner should receive the Flash or HTML5 version of the training.

Segmentation of the learner population can be a powerful tool in the reporting and analytics for any hospitality training program. We will follow up in a later post and explore that in greater detail.


This method of identifying smaller, homogenized groups, from a larger and potentially highly diverse learner population is highly valued by designers hoping to tailor the learning to the target audience or control the rate and reach of how hospitality training is deployed.    While this does represent some additional effort, the time spent at the beginning of a training project can help designers deliver training with an exceptional understanding of their learner environment, thereby positively impact the take rate and application of knowledge learned.

To learn more about Match Stick and their e-learning solutions check out their site



As a consultant, I travel both domestically and internationally. Two weeks ago, during a four day trip to NY, I realized that I have amassed a small collection of items that I truly need when away from home and other items that just make travel more comfortable. It also dawned on me that I pack almost all of these items whether I travel a few hours away by car or many hours in the air. I realized that I’m a bit obsessed with my travel gear!  See the last pic below, that seems like my life at times!

Along with many of my fellow SIOP members, I am burned out from writing conference submissions and finishing research over the last month.  So I decided to do something easier (AKA lazy) and just share my “travel checklist” with hopes that other management consultants/IO practitioners/hobos may enjoy the light reading and may even find something useful! Here’s the larger list of items, in no particular order:

  • Power pack
  • Tri-fold garment bag/duffel
  • Business cards
  • TSA Precheck
  • Quirky All purpose rubber bands w/hooks
  • Laptop bag
  • Space saving travel bags
  • High quality USB charging cable
  • Travel Electric Adaptor
  • Elevator Speech

Here are some of my favorite items:

Trifold garment bag- Having this bag eliminates the possibility of being asked to check my bag at the gate (which happens often when traveling using a traditional carry-on suitcase or folding garment bag). Overhead compartments get crowded on domestic flights and especially with the smaller bins on international flights. I find it comforting to know that I can avoid the luggage carousel when I arrive at my destination and keep my stuff within reach at all times. I can carry my suits and shirts all day and unpack them with minimal wrinkles upon arrival.

TSA PreCheck- This may be the best thing I have purchased for travel. If I get held late with a meeting, wait on line to return a rental car, or simply want to get a few more minutes of sleep, the PreCheck makes getting to my gate much easier. I get to keep my laptop in its case and remain fully dressed (which is helpful during business travels, but also when traveling with the family (including two young kids) for vacation. Its like a two-for-one benefit!

Business cards and elevator speech- I lump these together because they are equally important and serve the same purpose: communicating your brand as a consultant. Sometimes you only have a few moments to chat with someone and the opportunity to convey your passion for what you do. I have conversed with many interesting people when I have my hands full of luggage or in the middle of snacking during my travel.  I ask for their card and they say, “Oh, I don’t have any on me”. I try to keep their name on my mind, but most of the time when I finally get a chance to settle down, it is lost forever. The speech is important, but the business card is key.


High quality USB cable- more specifically USB-to-micro USB cable. This 6 foot cable allows me to charge my phone from a myriad of USB ports on computers, rental cars, alarm clocks, hotel televisions, etc that I come across. Regardless of location, I can always charge!!! It can be bent, folded, twisted, and otherwise abused and will not leave me stranded with a frayed cord or damaged connector like the cheap ones that all smartphones come with…looking at you Apple and Samsung!

Power bank- For the times when I can’t get to an outlet, don’t have the right power adaptor and do not have access to a 2.1A or higher USB port (which current smart phones need to charge with any sort of speed), the power bank is a life saver. My phone can be charging while I sit at the airport, while in flight, while I drive, or even when sitting in day-long meetings sans computer. Using the power bank, I can charge anywhere without a wall socket! 


There are things that I am leaving out like headphones, USB drives, my smartphone, and a passport holder wallet which I also take with me on every trip. I would be interested in hearing what others bring that they simply can’t do without.  Please share your best travel accessories in the comments below!

Ensuring your 360 appraisals doesn't suck!

Over the last few months, I have been drawn into conversations with colleagues and clients about leadership assessment, performance management, survey design, linkage analytics, and 360 feedback. During these conversations, the recent trend to change performance management or even eliminate performance appraisals in their most typical form often comes up. Despite this trend, I believe the process is still useful as part of the larger overall assessment of individual's strengths and developmental opportunities. When implemented correctly, it can drive employee development and key business outcomes. Below, I provide tips on how to optimize 360 degree feedback based on experience and evidence accumulated in the literature.

In my experience over the last decade or so, I have seen 360 appraisals range from very successful to almost completely ineffective. Most of the time, it was not the actual survey or items that were ill conceived (though that idea could be its own blog post), but rather a lack of clear direction and understanding of the 360 appraisal process that may have doomed the intervention before it began.

I put forth these 10 statements and questions as a practical “birds eye view” guide to implementing 360 appraisal processes that can be applied across companies and industries.

 1.      Identify the purpose of the appraisal.

            Why is it being conducted? Is it part of a larger performance management process? Is it to identify areas of coaching/training/development? Identifying the purpose of the appraisal will aid in the development of an effective 360 tool.

2.      Evaluate and increase stakeholder/leader support.

            Are all leaders informed about the process and its importance? Are they supportive? If not, try to gain buy-in through an informational session about 360 appraisal and its benefits (financial, workload, etc). A 360 appraisal process cannot succeed without managerial and stakeholder support.

3. What behaviors/competencies/values need to be covered?

            Focus on the competencies that align with the key business drivers and current strategy/direction of the organization. If necessary, develop questions related to the competencies and a scoring scale.  Review the appraisal form with managers and subject matter experts to determine if the listed competencies are representative of the job.

4. Who will receive feedback and who will provide it?

In a 360 appraisal process, supervisors, the employees themselves, co-workers, subordinates, and possibly customers will provide feedback. In order to get the most out of the appraisal process it is important to determine which co-workers, which supervisors, and which customers or subordinates will provide feedback. In addition to the question of who will evaluate, an organization may give different weights or values to the feedback received. This should be determined prior to collecting the 360 appraisal data.

5. Do the raters know what they are rating?

Quite often, organizations will purchase 360 appraisal tools and let raters merely rate employees in whatever manner the employees choose. In order for the ratings to be effective, these raters must be able to observe performance the same way. This can be achieved through frame of reference training where trainers are taught what high, moderate, and low quality performance actually mean.

  6.  How will you communicate with the employees and managers about the appraisal process?

            Let employees know about the process, why is it being used, how it works, how it will benefit them, and what is expected from them. Communicating to the employees and managers about the appraisal process is very important. Employees and managers must be told when the 360 appraisal process will begin, what the goals are for the appraisal process, and how the process should be conducted. The results of the appraisal process must be communicated…it is very important that employees know when and how these results will be used.

7. Will there be reports and what will they look like?

            Each person receiving feedback should receive a report with aggregated information/unidentifiable open responses (except for direct feedback from their leader). The aggregated information helps keep individual responses anonymous. Having this report helps the recipient understand the feedback provided by everyone involved.

8.      Will there be a follow-up meeting?

            While receiving feedback is wonderful, reports need to be reviewed with all associates and leaders. Employees and managers should create development plans that specifically target areas of need. Provide training to leaders about how to create a developmental plan or, if this is too cumbersome, hire an external consultant to work with your organization's employees to create useful developmental plans.

9.      When will the organization provide coaching?

        The feedback report by itself will not result in sustainable behavior change unless the organization provides coaching. This coaching must be provided by either a skilled coach, a manager, or someone from human resources.

            The report should help recipients understand their strengths and highlight areas where there are opportunities to grow or improve. Many organizations see the report as the final product that will result in improved performance but sustained reminders about performance are necessary.

10.  What did you learn?

            After the 360 process is completed, there should be a process review. Not only should  feedback produce the desired changes in employee performance, but an analysis of the process of administering the 360 appraisal should be conducted. Along with this, reviewers should provide feedback about the process possibly through a short post survey.


A word of advice:  As a manual process, 360 degree feedback and the associated reporting and follow-up analytics are exceptionally time consuming. Using technology (such as HRIS systems like ADP Workforce Now or Oracle HRMS) can help automate the process, allowing companies to focus on gaining insight and driving results. If your human resources department does not have the necessary expertise, consider hiring external consultants who know how to leverage 360 performance appraisals for maximum results.

Post-SIOP 2016 Reflection


I had almost forgotten to write my post SIOP 2016 thoughts until I came across this set of pictures from the 2016 conference. SIOP 2016 still feels like a whirlwind and while I made it to many sessions, I did not attend as many as I would have liked.  Here are my thoughts after experiencing IO psychology’s biggest annual conference:

1)      Data science- Data science was one of the prominent themes at this year’s SIOP conference. I attended a number of sessions about machine learning, big data, deep learning and related topics. There seemed to be much concern among session attendees that IO psychology was falling behind in the area of statistics and data science.  However, as David Morgan from Facebook stated it will take data scientists much longer to learn psychology then it will for IO psychologists to learn data science. Ultimately, big data, deep learning, and machine learning are newer techniques that IO psychology practitioners and academics are in a unique position to take advantage of.  IO practitioners are poised to take the lead in workplace research. We just need to learn how to leverage these new tools and techniques to generate new insights.

2)      Translators- Many of the sessions I attended and many of the conversations that I had with friends, colleagues, and students were around the increased attention on our field. Whether that attention has come from the work of Lazlo Bock, Adam Grant, or the Re:Work initiative. This increased interest in our field means that we have a unique opportunity to effectively communicate the fundamental ideas of IO psychology to our students, clients, and stakeholders. Whether it was a panel on survey design, situational judgment tests, or data analytics, speakers focused on the importance of IO psychology practitioners’ ability to communicate to those outside of our field. There was great excitement about new journals such as Bowling Green’s Personnel Assessment and Decisions. This journal's primary objective is to bridge the gap between science, practitioners, and business stakeholders. The message from SIOP 2016 was clearer than ever: we must all advocate for evidence-based management.

3)      Technology- There were many panels on technological innovations that were shaping the way we have done traditional IO work. Whether it was mobile pulse surveys, web-based simulations or e-learning we are finding new ways of using our methods and assessments. You couldn't walk around the SIOP exhibition hall without seeing new and exciting tools for simulations or assessment centers. This is an exciting time to be in the field because we are able to collect data using these technologies which will ultimately help us develop even more robust tools and statistical techniques. 

4)      ‘Classic' Methodology- Even though we are seeing innovations in the amount of data, technology, and demand for IO services there was a sense at this year’s SIOP that we have to stay true to ourselves. That we need to remain true to what we know about selection, training, performance management, and organizational development. None of the technological innovations are changing what constitutes good science. Nor does it change our goals as IO psychology practitioners but the technology does expand our tool box and may help us become more efficient.

5)      Teaming up- If it’s good enough for the Avengers then its good enough for SIOP! It was wonderful to see that SHRM was a sponsor of this year’s conference. It was also exciting to hear about the many wonderful local IO groups that have been recently formed around the country. My local IO group, Metro, had a strong presence at the conference this year and I learned from some of my co-panelists about local IO groups in Dallas, DC, and Minnesota. These partnerships and organizations (at the local and national level) will only help to strengthen the field of IO psychology and help us make an even greater impact in the workplace. It’s great to see SIOP take such a leading role at recognizing the benefits of partnership and collaboration.

6)      The expanding scope of IO Psychology- When I first started studying IO psychology in 2007 I initially felt that our science’s only application could be found in corporate America. Over the years, I have learned otherwise. IO psychology’s principles of methodological rigor paired with evidence-based decision making can be found in the worlds of market research, program evaluation, healthcare (just to name a few applications). One of the most exciting areas is the work of the Global Organization for Humanitarian Work Psychology. The GOHWP is a coalition of individuals from low- to high-income countries devoted to the field of humanitarian work psychology “whose purpose is to further the synthesis of organizational, industrial, work, and other areas of psychology with deliberate and organized efforts to enhance human welfare”.  The GOHWP provides our field an opportunity to extend our science into unforeseen areas of practice. IO psychology has much to offer non-profit and governmental organizations. The work of the GOHWP includes expanding research into areas involving different samples (i.e. Nicaraguan garment workers) working in varied organizations (businesses, governments, and non-governmental organizations) and using research science to tackle real-world issues (i.e. employability, poverty). Talent Metrics proudly offers its services to local non-profit organizations in NY and Florida. GOHWP is a wonderful endeavor and it continues to grow year after year. Ashlee Hoffman, Dr. Stuart Carr, and Laura Sywulak of GOHWP lead a lively discussion at the GOHWP SIOP party. There are exciting Humanitarian Work Psychology initiatives happening at organizations like the  UN (and other non-profit organizations) that will allow IO practitioners to expand their skillset. If you are a student looking for an opportunity to practice what you learned in class, a practitioner looking to try your hand at a new challenge, or  an academic looking to conduct research in an exciting new area, humanitarian work psychology may have something to offer to you. Reach out to GOHPW on Twitter or at their website  to get involved!

These are my takeaways from SIOP, but I would love to hear yours! The time period right after the conference is when I’m most energized and excited about the field. Please post them in the comments below or send me a message on Twitter at @IOSyIslam

Upcoming Speaking Engagement and Congratulations to Dr. Sy Islam!

Talent Metrics, LLC would like to congratulate Principal Consultant, Sy Islam on his SIOP presentation “Mythbusters: Debunking Common Career Mythconceptions which presented at the 2016 SIOP Conference this past weekend in Anaheim.

We would also like to congratulate him on his upcoming presentation at Baruch College! Dr. Islam will be leading a discussion about his dissertation research, “The effect of the timing of a resource gain on team adaptive behaviors and objective team performance.”

This is a great event for anyone who is interested in the field of workplace science and teams, or for any of Dr. Islam's students who wish to learn more about the process of a dissertation!

Register here for this 4/22 event which runs from 6-8:30PM at:

Baruch College - Information & Technology Building
(Library Building)
151 E. 25th St.
New York, NY 10010
Room: Room 763 

Thank you,

Mike Chetta, Andrzej Kozikowski, and Sy Islam

pre SIOP 2016 thoughts


                  As I prepare to embark on another journey to the annual meeting of the Society for Industrial Organizational Psychology (SIOP), I started to reflect on how the purpose and meaning of the conference evolves each year as I further develop and make progress in my I/O career. I find that SIOP reminds me of how far I have come in the journey of a scientist-practitioner.

                  My first SIOP conference was in 2007, when the conference fortuitously took place right where I live in New York City. I remember the feeling of overwhelming excitement at attending a national conference. I was overly enthusiastic for the opportunity to listen to some of the great scholars whose papers I had read and been inspired by. That first conference was a whirlwind of activity and I often remember feeling exhausted at the end of each day from expending too much energy attempting to attend every session and listen to every speaker that I was interested in. I often wonder how much I got out of that first conference other than just being utterly enervated.

                  I look back at that youthful naiveté of the individual who was running from one conference session to another and I smile at that young exuberance. After 9 years of attending SIOP conferences off and on, here’s what I look forward to most.

1)     Conference sessions-the program at SIOP is usually very good and full of sessions that I enjoy attending. Unlike other conferences that I attend (Association for Psychological Science, Eastern Psychological Association, etc.), I love attending sessions that are geared towards my particular research interests. It’s also great to hear about what’s happening in other areas of IO psychology that I might not have much expertise in. That cross-pollination of ideas is key to future innovations in the field.

2)     Expand your network- I love meeting new people at SIOP…If you’re attending this year’s conference just swing by and say hello! If attending sessions outside of your interest area can spark an idea, the same is true from speaking to new people at the conference. Whether you’re an academic who is the only IO in your department, or if you’re a practitioner who works with people who have only the vaguest notion of what IO psychology is, SIOP can be a great way to reinvigorate your approach to your work. Part of that reenergizing comes from meeting new people. If you’re an experienced conference attendee, provide mentorship to those newbies who are attending the conference for the first time. If you’re a first-timer, don’t be shy about speaking to someone you don’t know. Go to the poster session and meet people who are there to talk about their research. It may come across as common sense, but speaking to others is the simplest and most direct way to get to know your colleagues in the field. It may give you some ideas for your own research or even where you want your career to go. One of my best SIOP experiences was serving as a conference ambassador. The ambassador program is a great resource for newcomers seeking advice on attending the conference and making the most of the offerings.

3)     Reconnect- One of the best parts of SIOP is reconnecting with old colleagues and friends from graduate school or previous jobs. Sometimes life is so busy that SIOP is the only time where you get a chance to reconnect. As you grow in your careers, you’ll find that reconnecting with old colleagues is as important as meeting future colleagues. Networking isn’t just about meeting new people, but maintaining strong relationships with those with whom you’ve already worked.

4)     Have fun-Make sure to carve out time to go to the different parties and events. Conferences aren’t just about attending sessions they’re about building relationships and learning from other scientist-practitioners in the field. Whether you’re a seasoned IO veteran, or a young graduate student just making their way through their first conference, listening to someone else’s lived human experience is invaluable. You’ll gain insights that you can carry back home with you.

If you’d like some further reading on the topic, Richard Landers has a great post about attending the SIOP conference this year. I’ll be live tweeting during some of the sessions that I attend please reach out feel free to reach out to me @IOSyIslam

I’ll also be writing a post-SIOP review where I’ll discuss some of my takeaways from the conference.

Have a wonderful SIOP!


Talent Metrics, LLC would like to congratulate Principal Consultant, Mike Chetta on his new position with the Greater Orlando Organization Development Network! Dr. Chetta is now Co-Chair of the External Consultant Community of Practice and will coordinate activities for the GOOD Network involving external consulting.

The GOOD Network (Greater Orlando Organization Development Network) is a regional OD network affiliated with the International Organization Development Network. The membership includes a diverse mix of consultants and OD practitioners from a variety of industries. The organization strives to uphold an open and welcoming environment for all members that maintains a strong focus on providing tools, techniques, and insights which members can apply in their professional endeavors. The GOOD Network was selected as a 2015 Outstanding Regional Network by ODN.

About Talent Metrics:

Talent Metrics is a human capital management consulting firm devoted to solving organizational problems and improving performance. We help organizations make informed decisions talent management, and organizational culture through customized tools, workplace research, and advanced analytics. Talent Metrics drives business through people analytics. 

Talent Metrics is open to discussions with potential new clients to add to our portfolio of Non-Profit, Private Company and Fortune 500 partnerships. Our firm employs evidence-based practices and analytics to provide solutions and inform business decisions.

In today’s fast-paced and volatile business climate, companies may not have the time, resources, or specific capabilities in their workforce to apply evidence-based solutions. Given these constraints, organizations may employ practices that are costly and unproductive. Talent Metrics is committed to helping organizations make the most informed decisions.  Our clients choose to work with us because we…

· Offer customized analytics services appropriate for businesses of all sizes.

· Offer a broad range of evidence-based consulting services including structured interview development, executive/leadership assessment, survey design, and data analytics.

· Offer a team of consultants with a wide range of experience in different areas of human capital management (i.e. leadership development, selection, training, executive assessment).

We invite you to check out our website (www.TalentMetrics.IO) to learn more about who we are, what we do, and how we can drive business results in your organization.  This is also where you can find blog posts, tweets, and white papers from our team of consultants and business partners.

Please reach out to Talent Metrics if you have a question, challenge, or business decision where we may be able to provide insight. You can send a request on our website’s contact page, email one of our consultants, send a LinkedIn message, or tweet @TalentMetrics!

Thank you,

Michael Chetta, Ph.D., Sy Islam, Ph.D. & Andrzej Kozikowski, Ph.D.                                               Co-Founders & Principal Consultants                                                                                       Talent Metrics


Are you maximizing the use of your data?

Perhaps one of these situations is familiar to you.

Situation 1

Client: “We just completed this training program and I’m glad you’ve collected all this data using different multiple choice and open-ended questions but it’s overwhelming. I just want overall numbers that I can show my bosses.”

Situation 2

Client: “We need to do a training evaluation but I want something as simple as possible. Just use a smile sheet. We just need to know if the trainees liked the training enough to continue this program.”

Situation 3

Client: “I’m really concerned about the training program and I’d love to look at the qualitative and quantitative data we collected but I don’t know how to make sense of it. I’m not sure what to do with it or if I even its useable.”

These are some situations that I have encountered when dealing with clients (i.e. HR directors, business owners, line managers). I hope I have captured some element of the post-intervention (in these examples a training program) challenges that many consultants face when dealing with evaluation.

One of the most frequent challenges of post-training data analysis is that you might collect an enormous amount of data and then not be sure what to do with it. The data you have start to look like vacuum cleaner attachments. Sometimes you look at your data and all you see are extra attachments for a device that you are not sure how to use.

Can we use all of these attachments? Of course!

Do we always use them? No!

We’re not certain what the attachments are for. All we know is that we have them. 

In many situations, I find that HR managers, generalists, training managers, directors etc. often have the same reaction when faced with the survey data they collected from trainees. They feel overwhelmed. This is especially true if the clients did not participate in developing the evaluation process. Clients are a necessary component to any evaluation because they know what they are looking for out their training program.

One of the most common sources of overlooked data are qualitative comments from trainees. In many cases, this data is not overlooked because the clients feel that the data is not valuable. In general, my clients have found the qualitative comments invaluable. However, there is typically hesitation about how to analyze the comments made by trainees and how to use the comments to develop next steps.

A recent study by Harman, Ellington, Surface, and Thompson (2015) illustrates the importance of comments to an evaluation of training program effectiveness. The researchers conducted three field studies in a series of simultaneous training classes. In each study they assessed the commenting behavior of these classes. I strongly recommend that any learning, HR, OD, or IO practitioner to read the entire study. However, I wanted to highlight some of my major takeaways.

1)      Classroom experiences affected the likelihood of commenting. As individuals who have experienced training in a variety of contexts, we intuitively know this. However, it’s great to know that the comments you receive in your training evaluations will reflect real differences in classroom experience. Pay attention to the comments because they will tell you what happened in these training programs.

2)      As class-level learning decreased commenting increased. In other words, there was a negative correlation between learning and commenting. This is a very powerful finding as it relates to the first big takeaway. If trainees do not feel like they are learning in the classroom they will say something about it. That kind of data is important to pay attention to.

3) Trainee reactions are multidimensional. Many times we want to reduce the data to a single number or a single value i.e. “What percent of the trainees liked the program? 60%.” However there’s a lot more going on in any training program than just what the trainees felt about it overall. Trainee reactions can lead us to understand many changes that are important to be made in subsequent training classes. If we look deeper at the data we can learn about components of the training program, the trainer, the class environment, and the relevance of the material.

4) If there is no expectation of change, you will receive no comments. In other words, if your employees feel that your organization won’t change anything based on what they have to say, they won’t say anything at all. If you are conducting a training evaluation (or any evaluation) your organization needs to be committed to making the necessary changes and that should be communicated to your employees. You depend on data from your employees and by communicating your commitment to making changes you will elicit comments from those who have experienced your training program.


The message is clear from these research findings. Your employees want great training programs and want your programs to improve. It’s up to us to leverage the comments made in addition to the quantitative ratings to get the most benefit from the metrics we employ.

If you do feel like you’re missing out on some data or you have training evaluation data that you feel like you don’t know what to do with, what are your options? Here are some suggestions:

1)      Reach out to colleagues and discuss your options. Some of my best ideas have come from discussions at ATD meetings or at Metro Applied Psychology meetings. As IO practitioners, we live for this discussion.

2)      Talk to your vendor. If you are using a vendor, ask them about options they offer for data analysis. Much like the vacuum cleaner, they probably have options you have not fully investigated.

3) Reach out to a consultant. If you are truly lost in the evaluation process, reach out to a consultant that specializes in training evaluation who can guide you through using this data.


Can you think of other situations where you may be missing out on this hidden data? Is there data that you collect that you feel you do not get the most out of? Feel free to list these in the comments below. I would love to hear your thoughts. Make sure to read the full article. The reference is below!



Harman, R. P., Ellington, J. K., Surface, E. A., & Thompson, L. F. (2015). Exploring qualitative training reactions: Individual and contextual influences on trainee commenting. Journal of Applied Psychology, 100(3), 894.

 This post has been cross-posted from