Ensuring your 360 appraisals doesn't suck!

Over the last few months, I have been drawn into conversations with colleagues and clients about leadership assessment, performance management, survey design, linkage analytics, and 360 feedback. During these conversations, the recent trend to change performance management or even eliminate performance appraisals in their most typical form often comes up. Despite this trend, I believe the process is still useful as part of the larger overall assessment of individual's strengths and developmental opportunities. When implemented correctly, it can drive employee development and key business outcomes. Below, I provide tips on how to optimize 360 degree feedback based on experience and evidence accumulated in the literature.

In my experience over the last decade or so, I have seen 360 appraisals range from very successful to almost completely ineffective. Most of the time, it was not the actual survey or items that were ill conceived (though that idea could be its own blog post), but rather a lack of clear direction and understanding of the 360 appraisal process that may have doomed the intervention before it began.

I put forth these 10 statements and questions as a practical “birds eye view” guide to implementing 360 appraisal processes that can be applied across companies and industries.

 1.      Identify the purpose of the appraisal.

            Why is it being conducted? Is it part of a larger performance management process? Is it to identify areas of coaching/training/development? Identifying the purpose of the appraisal will aid in the development of an effective 360 tool.

2.      Evaluate and increase stakeholder/leader support.

            Are all leaders informed about the process and its importance? Are they supportive? If not, try to gain buy-in through an informational session about 360 appraisal and its benefits (financial, workload, etc). A 360 appraisal process cannot succeed without managerial and stakeholder support.

3. What behaviors/competencies/values need to be covered?

            Focus on the competencies that align with the key business drivers and current strategy/direction of the organization. If necessary, develop questions related to the competencies and a scoring scale.  Review the appraisal form with managers and subject matter experts to determine if the listed competencies are representative of the job.

4. Who will receive feedback and who will provide it?

In a 360 appraisal process, supervisors, the employees themselves, co-workers, subordinates, and possibly customers will provide feedback. In order to get the most out of the appraisal process it is important to determine which co-workers, which supervisors, and which customers or subordinates will provide feedback. In addition to the question of who will evaluate, an organization may give different weights or values to the feedback received. This should be determined prior to collecting the 360 appraisal data.

5. Do the raters know what they are rating?

Quite often, organizations will purchase 360 appraisal tools and let raters merely rate employees in whatever manner the employees choose. In order for the ratings to be effective, these raters must be able to observe performance the same way. This can be achieved through frame of reference training where trainers are taught what high, moderate, and low quality performance actually mean.

  6.  How will you communicate with the employees and managers about the appraisal process?

            Let employees know about the process, why is it being used, how it works, how it will benefit them, and what is expected from them. Communicating to the employees and managers about the appraisal process is very important. Employees and managers must be told when the 360 appraisal process will begin, what the goals are for the appraisal process, and how the process should be conducted. The results of the appraisal process must be communicated…it is very important that employees know when and how these results will be used.

7. Will there be reports and what will they look like?

            Each person receiving feedback should receive a report with aggregated information/unidentifiable open responses (except for direct feedback from their leader). The aggregated information helps keep individual responses anonymous. Having this report helps the recipient understand the feedback provided by everyone involved.

8.      Will there be a follow-up meeting?

            While receiving feedback is wonderful, reports need to be reviewed with all associates and leaders. Employees and managers should create development plans that specifically target areas of need. Provide training to leaders about how to create a developmental plan or, if this is too cumbersome, hire an external consultant to work with your organization's employees to create useful developmental plans.

9.      When will the organization provide coaching?

        The feedback report by itself will not result in sustainable behavior change unless the organization provides coaching. This coaching must be provided by either a skilled coach, a manager, or someone from human resources.

            The report should help recipients understand their strengths and highlight areas where there are opportunities to grow or improve. Many organizations see the report as the final product that will result in improved performance but sustained reminders about performance are necessary.

10.  What did you learn?

            After the 360 process is completed, there should be a process review. Not only should  feedback produce the desired changes in employee performance, but an analysis of the process of administering the 360 appraisal should be conducted. Along with this, reviewers should provide feedback about the process possibly through a short post survey.


A word of advice:  As a manual process, 360 degree feedback and the associated reporting and follow-up analytics are exceptionally time consuming. Using technology (such as HRIS systems like ADP Workforce Now or Oracle HRMS) can help automate the process, allowing companies to focus on gaining insight and driving results. If your human resources department does not have the necessary expertise, consider hiring external consultants who know how to leverage 360 performance appraisals for maximum results.

Post-SIOP 2016 Reflection


I had almost forgotten to write my post SIOP 2016 thoughts until I came across this set of pictures from the 2016 conference. SIOP 2016 still feels like a whirlwind and while I made it to many sessions, I did not attend as many as I would have liked.  Here are my thoughts after experiencing IO psychology’s biggest annual conference:

1)      Data science- Data science was one of the prominent themes at this year’s SIOP conference. I attended a number of sessions about machine learning, big data, deep learning and related topics. There seemed to be much concern among session attendees that IO psychology was falling behind in the area of statistics and data science.  However, as David Morgan from Facebook stated it will take data scientists much longer to learn psychology then it will for IO psychologists to learn data science. Ultimately, big data, deep learning, and machine learning are newer techniques that IO psychology practitioners and academics are in a unique position to take advantage of.  IO practitioners are poised to take the lead in workplace research. We just need to learn how to leverage these new tools and techniques to generate new insights.

2)      Translators- Many of the sessions I attended and many of the conversations that I had with friends, colleagues, and students were around the increased attention on our field. Whether that attention has come from the work of Lazlo Bock, Adam Grant, or the Re:Work initiative. This increased interest in our field means that we have a unique opportunity to effectively communicate the fundamental ideas of IO psychology to our students, clients, and stakeholders. Whether it was a panel on survey design, situational judgment tests, or data analytics, speakers focused on the importance of IO psychology practitioners’ ability to communicate to those outside of our field. There was great excitement about new journals such as Bowling Green’s Personnel Assessment and Decisions. This journal's primary objective is to bridge the gap between science, practitioners, and business stakeholders. The message from SIOP 2016 was clearer than ever: we must all advocate for evidence-based management.

3)      Technology- There were many panels on technological innovations that were shaping the way we have done traditional IO work. Whether it was mobile pulse surveys, web-based simulations or e-learning we are finding new ways of using our methods and assessments. You couldn't walk around the SIOP exhibition hall without seeing new and exciting tools for simulations or assessment centers. This is an exciting time to be in the field because we are able to collect data using these technologies which will ultimately help us develop even more robust tools and statistical techniques. 

4)      ‘Classic' Methodology- Even though we are seeing innovations in the amount of data, technology, and demand for IO services there was a sense at this year’s SIOP that we have to stay true to ourselves. That we need to remain true to what we know about selection, training, performance management, and organizational development. None of the technological innovations are changing what constitutes good science. Nor does it change our goals as IO psychology practitioners but the technology does expand our tool box and may help us become more efficient.

5)      Teaming up- If it’s good enough for the Avengers then its good enough for SIOP! It was wonderful to see that SHRM was a sponsor of this year’s conference. It was also exciting to hear about the many wonderful local IO groups that have been recently formed around the country. My local IO group, Metro, had a strong presence at the conference this year and I learned from some of my co-panelists about local IO groups in Dallas, DC, and Minnesota. These partnerships and organizations (at the local and national level) will only help to strengthen the field of IO psychology and help us make an even greater impact in the workplace. It’s great to see SIOP take such a leading role at recognizing the benefits of partnership and collaboration.

6)      The expanding scope of IO Psychology- When I first started studying IO psychology in 2007 I initially felt that our science’s only application could be found in corporate America. Over the years, I have learned otherwise. IO psychology’s principles of methodological rigor paired with evidence-based decision making can be found in the worlds of market research, program evaluation, healthcare (just to name a few applications). One of the most exciting areas is the work of the Global Organization for Humanitarian Work Psychology. The GOHWP is a coalition of individuals from low- to high-income countries devoted to the field of humanitarian work psychology “whose purpose is to further the synthesis of organizational, industrial, work, and other areas of psychology with deliberate and organized efforts to enhance human welfare”.  The GOHWP provides our field an opportunity to extend our science into unforeseen areas of practice. IO psychology has much to offer non-profit and governmental organizations. The work of the GOHWP includes expanding research into areas involving different samples (i.e. Nicaraguan garment workers) working in varied organizations (businesses, governments, and non-governmental organizations) and using research science to tackle real-world issues (i.e. employability, poverty). Talent Metrics proudly offers its services to local non-profit organizations in NY and Florida. GOHWP is a wonderful endeavor and it continues to grow year after year. Ashlee Hoffman, Dr. Stuart Carr, and Laura Sywulak of GOHWP lead a lively discussion at the GOHWP SIOP party. There are exciting Humanitarian Work Psychology initiatives happening at organizations like the  UN (and other non-profit organizations) that will allow IO practitioners to expand their skillset. If you are a student looking for an opportunity to practice what you learned in class, a practitioner looking to try your hand at a new challenge, or  an academic looking to conduct research in an exciting new area, humanitarian work psychology may have something to offer to you. Reach out to GOHPW on Twitter or at their website  to get involved!

These are my takeaways from SIOP, but I would love to hear yours! The time period right after the conference is when I’m most energized and excited about the field. Please post them in the comments below or send me a message on Twitter at @IOSyIslam

Upcoming Speaking Engagement and Congratulations to Dr. Sy Islam!

Talent Metrics, LLC would like to congratulate Principal Consultant, Sy Islam on his SIOP presentation “Mythbusters: Debunking Common Career Mythconceptions which presented at the 2016 SIOP Conference this past weekend in Anaheim.

We would also like to congratulate him on his upcoming presentation at Baruch College! Dr. Islam will be leading a discussion about his dissertation research, “The effect of the timing of a resource gain on team adaptive behaviors and objective team performance.”

This is a great event for anyone who is interested in the field of workplace science and teams, or for any of Dr. Islam's students who wish to learn more about the process of a dissertation!

Register here for this 4/22 event which runs from 6-8:30PM at:

Baruch College - Information & Technology Building
(Library Building)
151 E. 25th St.
New York, NY 10010
Room: Room 763


Thank you,

Mike Chetta, Andrzej Kozikowski, and Sy Islam

pre SIOP 2016 thoughts


                  As I prepare to embark on another journey to the annual meeting of the Society for Industrial Organizational Psychology (SIOP), I started to reflect on how the purpose and meaning of the conference evolves each year as I further develop and make progress in my I/O career. I find that SIOP reminds me of how far I have come in the journey of a scientist-practitioner.

                  My first SIOP conference was in 2007, when the conference fortuitously took place right where I live in New York City. I remember the feeling of overwhelming excitement at attending a national conference. I was overly enthusiastic for the opportunity to listen to some of the great scholars whose papers I had read and been inspired by. That first conference was a whirlwind of activity and I often remember feeling exhausted at the end of each day from expending too much energy attempting to attend every session and listen to every speaker that I was interested in. I often wonder how much I got out of that first conference other than just being utterly enervated.

                  I look back at that youthful naiveté of the individual who was running from one conference session to another and I smile at that young exuberance. After 9 years of attending SIOP conferences off and on, here’s what I look forward to most.

1)     Conference sessions-the program at SIOP is usually very good and full of sessions that I enjoy attending. Unlike other conferences that I attend (Association for Psychological Science, Eastern Psychological Association, etc.), I love attending sessions that are geared towards my particular research interests. It’s also great to hear about what’s happening in other areas of IO psychology that I might not have much expertise in. That cross-pollination of ideas is key to future innovations in the field.

2)     Expand your network- I love meeting new people at SIOP…If you’re attending this year’s conference just swing by and say hello! If attending sessions outside of your interest area can spark an idea, the same is true from speaking to new people at the conference. Whether you’re an academic who is the only IO in your department, or if you’re a practitioner who works with people who have only the vaguest notion of what IO psychology is, SIOP can be a great way to reinvigorate your approach to your work. Part of that reenergizing comes from meeting new people. If you’re an experienced conference attendee, provide mentorship to those newbies who are attending the conference for the first time. If you’re a first-timer, don’t be shy about speaking to someone you don’t know. Go to the poster session and meet people who are there to talk about their research. It may come across as common sense, but speaking to others is the simplest and most direct way to get to know your colleagues in the field. It may give you some ideas for your own research or even where you want your career to go. One of my best SIOP experiences was serving as a conference ambassador. The ambassador program is a great resource for newcomers seeking advice on attending the conference and making the most of the offerings.

3)     Reconnect- One of the best parts of SIOP is reconnecting with old colleagues and friends from graduate school or previous jobs. Sometimes life is so busy that SIOP is the only time where you get a chance to reconnect. As you grow in your careers, you’ll find that reconnecting with old colleagues is as important as meeting future colleagues. Networking isn’t just about meeting new people, but maintaining strong relationships with those with whom you’ve already worked.

4)     Have fun-Make sure to carve out time to go to the different parties and events. Conferences aren’t just about attending sessions they’re about building relationships and learning from other scientist-practitioners in the field. Whether you’re a seasoned IO veteran, or a young graduate student just making their way through their first conference, listening to someone else’s lived human experience is invaluable. You’ll gain insights that you can carry back home with you.

If you’d like some further reading on the topic, Richard Landers has a great post about attending the SIOP conference this year. I’ll be live tweeting during some of the sessions that I attend please reach out feel free to reach out to me @IOSyIslam

I’ll also be writing a post-SIOP review where I’ll discuss some of my takeaways from the conference.

Have a wonderful SIOP!


Talent Metrics, LLC would like to congratulate Principal Consultant, Mike Chetta on his new position with the Greater Orlando Organization Development Network! Dr. Chetta is now Co-Chair of the External Consultant Community of Practice and will coordinate activities for the GOOD Network involving external consulting.

The GOOD Network (Greater Orlando Organization Development Network) is a regional OD network affiliated with the International Organization Development Network. The membership includes a diverse mix of consultants and OD practitioners from a variety of industries. The organization strives to uphold an open and welcoming environment for all members that maintains a strong focus on providing tools, techniques, and insights which members can apply in their professional endeavors. The GOOD Network was selected as a 2015 Outstanding Regional Network by ODN.

About Talent Metrics:

Talent Metrics is a human capital management consulting firm devoted to solving organizational problems and improving performance. We help organizations make informed decisions talent management, and organizational culture through customized tools, workplace research, and advanced analytics. Talent Metrics drives business through people analytics. 

Talent Metrics is open to discussions with potential new clients to add to our portfolio of Non-Profit, Private Company and Fortune 500 partnerships. Our firm employs evidence-based practices and analytics to provide solutions and inform business decisions.

In today’s fast-paced and volatile business climate, companies may not have the time, resources, or specific capabilities in their workforce to apply evidence-based solutions. Given these constraints, organizations may employ practices that are costly and unproductive. Talent Metrics is committed to helping organizations make the most informed decisions.  Our clients choose to work with us because we…

· Offer customized analytics services appropriate for businesses of all sizes.

· Offer a broad range of evidence-based consulting services including structured interview development, executive/leadership assessment, survey design, and data analytics.

· Offer a team of consultants with a wide range of experience in different areas of human capital management (i.e. leadership development, selection, training, executive assessment).

We invite you to check out our website (www.TalentMetrics.IO) to learn more about who we are, what we do, and how we can drive business results in your organization.  This is also where you can find blog posts, tweets, and white papers from our team of consultants and business partners.

Please reach out to Talent Metrics if you have a question, challenge, or business decision where we may be able to provide insight. You can send a request on our website’s contact page, email one of our consultants, send a LinkedIn message, or tweet @TalentMetrics!

Thank you,

Michael Chetta, Ph.D., Sy Islam, Ph.D. & Andrzej Kozikowski, Ph.D.                                               Co-Founders & Principal Consultants                                                                                       Talent Metrics


Are you maximizing the use of your data?

Perhaps one of these situations is familiar to you.

Situation 1

Client: “We just completed this training program and I’m glad you’ve collected all this data using different multiple choice and open-ended questions but it’s overwhelming. I just want overall numbers that I can show my bosses.”

Situation 2

Client: “We need to do a training evaluation but I want something as simple as possible. Just use a smile sheet. We just need to know if the trainees liked the training enough to continue this program.”

Situation 3

Client: “I’m really concerned about the training program and I’d love to look at the qualitative and quantitative data we collected but I don’t know how to make sense of it. I’m not sure what to do with it or if I even its useable.”

These are some situations that I have encountered when dealing with clients (i.e. HR directors, business owners, line managers). I hope I have captured some element of the post-intervention (in these examples a training program) challenges that many consultants face when dealing with evaluation.

One of the most frequent challenges of post-training data analysis is that you might collect an enormous amount of data and then not be sure what to do with it. The data you have start to look like vacuum cleaner attachments. Sometimes you look at your data and all you see are extra attachments for a device that you are not sure how to use.

Can we use all of these attachments? Of course!

Do we always use them? No!

We’re not certain what the attachments are for. All we know is that we have them. 

In many situations, I find that HR managers, generalists, training managers, directors etc. often have the same reaction when faced with the survey data they collected from trainees. They feel overwhelmed. This is especially true if the clients did not participate in developing the evaluation process. Clients are a necessary component to any evaluation because they know what they are looking for out their training program.

One of the most common sources of overlooked data are qualitative comments from trainees. In many cases, this data is not overlooked because the clients feel that the data is not valuable. In general, my clients have found the qualitative comments invaluable. However, there is typically hesitation about how to analyze the comments made by trainees and how to use the comments to develop next steps.

A recent study by Harman, Ellington, Surface, and Thompson (2015) illustrates the importance of comments to an evaluation of training program effectiveness. The researchers conducted three field studies in a series of simultaneous training classes. In each study they assessed the commenting behavior of these classes. I strongly recommend that any learning, HR, OD, or IO practitioner to read the entire study. However, I wanted to highlight some of my major takeaways.

1)      Classroom experiences affected the likelihood of commenting. As individuals who have experienced training in a variety of contexts, we intuitively know this. However, it’s great to know that the comments you receive in your training evaluations will reflect real differences in classroom experience. Pay attention to the comments because they will tell you what happened in these training programs.

2)      As class-level learning decreased commenting increased. In other words, there was a negative correlation between learning and commenting. This is a very powerful finding as it relates to the first big takeaway. If trainees do not feel like they are learning in the classroom they will say something about it. That kind of data is important to pay attention to.

3) Trainee reactions are multidimensional. Many times we want to reduce the data to a single number or a single value i.e. “What percent of the trainees liked the program? 60%.” However there’s a lot more going on in any training program than just what the trainees felt about it overall. Trainee reactions can lead us to understand many changes that are important to be made in subsequent training classes. If we look deeper at the data we can learn about components of the training program, the trainer, the class environment, and the relevance of the material.

4) If there is no expectation of change, you will receive no comments. In other words, if your employees feel that your organization won’t change anything based on what they have to say, they won’t say anything at all. If you are conducting a training evaluation (or any evaluation) your organization needs to be committed to making the necessary changes and that should be communicated to your employees. You depend on data from your employees and by communicating your commitment to making changes you will elicit comments from those who have experienced your training program.


The message is clear from these research findings. Your employees want great training programs and want your programs to improve. It’s up to us to leverage the comments made in addition to the quantitative ratings to get the most benefit from the metrics we employ.

If you do feel like you’re missing out on some data or you have training evaluation data that you feel like you don’t know what to do with, what are your options? Here are some suggestions:

1)      Reach out to colleagues and discuss your options. Some of my best ideas have come from discussions at ATD meetings or at Metro Applied Psychology meetings. As IO practitioners, we live for this discussion.

2)      Talk to your vendor. If you are using a vendor, ask them about options they offer for data analysis. Much like the vacuum cleaner, they probably have options you have not fully investigated.

3) Reach out to a consultant. If you are truly lost in the evaluation process, reach out to a consultant that specializes in training evaluation who can guide you through using this data.


Can you think of other situations where you may be missing out on this hidden data? Is there data that you collect that you feel you do not get the most out of? Feel free to list these in the comments below. I would love to hear your thoughts. Make sure to read the full article. The reference is below!



Harman, R. P., Ellington, J. K., Surface, E. A., & Thompson, L. F. (2015). Exploring qualitative training reactions: Individual and contextual influences on trainee commenting. Journal of Applied Psychology, 100(3), 894.

 This post has been cross-posted from psychologyofwork.wordpress.com