Future of work

How do I evaluate an AI hiring tool?

Will AI be like Skynet? Find out below!

Will AI be like Skynet? Find out below!

Note: Thanks to Dr. Vivian Woo and Dr. Daniel Meltzer for their helpful feedback

            AI has been a hot topic in business over the last few years. The automation movement has been changing how we shop, work, and live our day-to-day lives, from self-checkout lanes to new phone apps. AI and automation also affect our hiring practices. In the human capital management world, AI/automation has become the topic du jour of many conferences and a variety of newspaper articles. Some recent Washington Post articles about the use of AI in hiring have caught the attention of many in the HR world, who may not be aware of how automated hiring practices are being implemented in organizations. The first article is about assessing nanny personality and fitness for hire based on social media posts and the second is a post about AI-based video interviewing tools that uses facial scanning technology to make determinations about a candidate’s fitness to be hired. Both articles seem to sound an alarm around AI and automated tools in hiring.

            Among HR professionals, there’s growing concern about how to use these tools, and among applicants, there’s concern about whether an automated tool will fairly assess their skillset. Some larger organizations, such as Hilton, have already implemented AI and automated tools into their hiring practices. One of the most common questions I receive is how AI will change the workplace and whether these tools have value. While I am by no means an expert on the AI/automation tools that exist, I do know how to design and develop selection systems. Here are some tips that may be helpful to those interested in implementing or purchasing these types of products.


Some important notes about algorithmic hiring processes that should be covered prior to any discussion of AI/automation. First, algorithmic hiring has existed for almost 100 years in the form of multiple regression and predictive hiring models using traditional pre-hire assessments (i.e., interviews, work sample tests, cognitive ability tests, etc.). Data-driven hiring techniques are the cornerstone of HR practice and can improve organizations in terms of financial performance, diversity, and a variety of other business-related outcomes. Overall, the process you would use to decide on an AI/automation tool is the same approach you would use for any other pre-hire assessment. I would recommend using the Uniform Guidelines on Employee Selection procedures as a reference for evaluating any pre-hire assessment.

The second note is one of nomenclature. I have been referring to these tools as AI/automation because many tools are not the type of artificial intelligence, we hear about in pop culture. Instead these tools mostly automate existing HR practices through technology. Automation has its own value to HR processes, and for the sake of brevity, I will not treat AI and automation as different things, I will merely address them as part of the same category. If you have questions beyond what is covered below, reach out to an HR consultant or an employment lawyer.

My recommendation to anyone considering a change in their human resources practice whether it involves AI or not, is to start with the needs assessment. The needs assessment process allows an HR professional to determine the business problem that must be solved by the potential change. Before beginning the process of vetting AI/automation vendors, conduct a needs analysis to determine what the real problem is. Once you have determined what the problem is then you can assess the viability of a potential AI/automation solution.  This process is especially important given some of the claims that AI/automation tools make. Once you have a clear idea of what the problem is, you and your organizational stakeholders can make a clearer judgement about the AI/automation tools. Many HR professionals struggle when deciding about solutions because the problem has not been specificed.


            The simplest step to begin your evaluation of any potential AI/automation tool is to determine whether the content of the assessment matches the content of the job itself. Review your job description and job analysis or competency model to determine whether the AI/automation tool assesses a relevant knowledge, skill, or ability. In some cases, the tool may replicate on-the-job tasks. By establishing that the AI/automation tool is related to the job, this establishes that the content of the assessment matches the content of the job. This should be established for any pre-hire assessment. If the tool does not replicate on-the-job tasks, then you may ask if it assesses the necessary knowledge, skills, abilities, or other characteristics (KSAOs) of the job.


            While matching the content of the job to the job itself is a key element of any hiring practice, the proclaimed value of AI/automation by many organizations is in the tools' ability to predict future job performance. This is referred to as predictive validity and is a correlation between the pre-hire assessment and future job performance. SHRM offers a toolkit on this topic that may be useful to those working in a selection practice. Ultimately, we must ask ourselves does the AI/automation tool make better predictions and allow your organization to hire more effectively.

            Many HR professionals may wonder where to get this kind of information. First, any vendor of pre-hire assessments should be able to provide you with a technical report that explains how their tool helps organizations make better decisions and reduces bias in the hiring process. These reports should include information on the validation study that was conducted and the predictive capability of the assessment. This information is usually in the form of a research study showing the sample of employees used and the quality of hire and job performance. Be mindful of the outcome variables that are measured. If the potential vendor does not have that information, I would be wary of the services they provide.

            In addition to accepting the predictive validity offered by the vendor, it may be in your best interest to conduct an internal validation study using your own employees. Give the AI/automation tool to your employees and correlate the tool’s scores/recommendations with the performance of your employees. This will allow you to assess the predictive capability of the tool itself. Even after you have implemented the AI/automation tool, it is in your best to interest to evaluate how much improvement you are seeing in decision quality. There are some existing approaches to assessing decision quality that may be valuable such as Taylor-Russell tables. If you have any concerns about assessing decision quality, your vendor maybe able to help or reach out to another data driven HR consultant.

Fit within Selection Battery

Your organization should use more than one pre-hire assessment in its selection battery (i.e., the group of pre-hire assessments given to job applicants). When adding an AI/automation tool to your selection battery, there are some important considerations. First, how does the AI/automation tool fit into your overall hiring strategy? In other words, does it make the hiring process faster, more effective in decision quality, or is it merely another way to add predictive power? Another important consideration is whether the AI/automation tool is an addition or a replacement for an existing part of your selection battery. Compare the predictive capability of your different tools, and determine whether the AI/automation tool adds any incremental validity (also known as increased predictive power). If the AI/automation does not add predictive power or does not fit your hiring strategy, then it may not be useful to add it. There may be other ways to improve your hiring processes.

When considering fit with your selection battery, an important consideration is employer brand. Review how your organization promotes itself as an employer and evaluate whether the AI/automation tool fits into your employer brand. The selection process is a two-way street. The organization is getting to know the applicant, but the applicant is also getting to know the organization. How an organization hire sends a clear message about the employer brand and about how the organization values people. If much of the process is automated, it may convey to an applicant that people aren't so important to the organization. Human beings generally recognize we allocate time to things we deem important. If the organization can't take enough time out of its day to speak to the candidate, it may leave job seekers with a negative impression. The Washington Post articles spend quite a bit of time on the concerns of applicants who experience these tools. Applicants should be told how the tools will be used and what the AI/automation tool assesses. One of the nannies who had been interviewed about her experience with the AI social media tool had not realized what the tool was evaluating her on. This can create later resentment. Clear communication can allay fears, but if your industry and potential job applicants are not amenable to the use of an AI/automation tool, then it may impact your recruitment efforts.

Applicant reactions may also determine the likelihood of legal action. New laws governing the use of AI/automation hiring tools are coming (see the recent law in Illinois) If applicants think the AI/automation tool is unfair or if the way the tool will be used in the hiring process is not clearly communicated, this may result in anger and legal action. Remember that as an employer, you are engaging in a psychological contract with job applicants who want a fair hiring system. If your organization communicates your process clearly, applicants know what to expect and will be less surprised by the AI/automation tool.

Technology Implementation

Implementation is also an important element of fitting the tool into your selection battery. Is the AI/automation tool able to fit it into your existing hiring process easily? Or does the tool not work with your applicant tracking software (ATS) or other technology? A technology solution should fit as smoothly as possible in your existing HR technology. The tool should be easy for you to use on the back end and appear seamless to applicants who are taking the AI/automation tool as part of the application process.


            The reality of using any of pre-hire assessment is that they involve costs. Costs can be financial in terms of how much it costs to give the AI/automation tool per applicant. Cost can also be related to how the AI/automation tool fits into your system. There are important questions around this issue to consider such as the learning curve for your HR team in the use of the tool. Does it fit seamlessly with your ATS? Is it easier for your applicants to access? Does the tool truly make your system faster? As noted in a recent post about AI myths on the SHRM site, Hilton has increased the number of offers using an AI/automation tool, and that is a valuable outcome for Hilton. If something goes wrong with the AI tool will there be support for you to fix it. Finally, if there is bias that is found in the way the tool provides decisions, can you detect it before it becomes a problem.

If your organization does not have the budget for an AI/automation tool, there are less expensive ways of improving hiring that your organization may want to consider. Cost should be balanced against return on investment, which can be calculated based on the predictive power of the AI/automation tool. New tools can lead to changes in your HR strategy. An HR professionals must balance the cost of the hiring tools with the strategic needs of the organization.


The gist of all of this is to say that before adopting any pre-hire assessment (whether AI or otherwise) consider the following issues:

1)      Does the content of the assessment match the job?

2)      Does the assessment predict job performance effectively?

3)      Does the assessment fit your organization’s selection system?

4)      Is the tool cost effective?

5)      Does the tool solve the business problem?

5 qs ai tm.jpg

YouTuber: Career or Fad?

by Darla Van Govan

Note from Sy: We're lucky to be running our first guest post by one of our interns Darla Van Govan. Enjoy! Be sure to follow her at @IO_Psych101 

The Rise of YouTubers

Over the last 12 years, YouTube has become a daily destination for content. Since its inception in 2005, YouTube has transitioned from a video sharing platform to a content creating space. There has been an influx of individuals creating content on YouTube in recent years. Many of these individuals do it as weekend entertainment or as a secondary source of income. However, there has been a growing population of individuals who become full-time YouTubers. With the incorporation of AdSense and third party sponsorships, it has become possible to make a modest income. These individuals leave their more traditional occupations to devote time to their YouTube channels. Their goal in becoming full-time YouTubers is to grow their channel and their influence on other social media platforms, including Twitter, Instagram, and Snapchat.

In recent years, advancements in technology and science have created demands for jobs that did not exist a few decades ago. It can be said that being a YouTuber is one of the newer career choices available to individuals, particularly millennials. However, since it is a relatively new platform, it is often debated whether if being a YouTuber can even be a viable long-term career choice. This question has a complex answer that is both yes and no.  

There is a perception that being on YouTube is not a job, that it does not require or develop any useful skills typically found in “normal” jobs. A common complaint is that the typical YouTuber stays home all day, sits in front of the camera to talk about a topic (or themselves) or to demonstrate something. This is not the case at all.

Careful consideration must be taken with how a YouTuber is labeled. It is important to acknowledge that “YouTuber” is not a job title. For example, engineers at Google do not call themselves Googlers. Chemists at L’Oreal do not call themselves L’Orealers. Similarly, a YouTuber is not simply a YouTuber. This is extremely limiting and does not communicate YouTubers’ varied skills. Rather, it is more accurate to call them content creators. YouTube is a not a career or a job title, it is a platform for content creation.


Myth 1: Anyone Can Do YouTube

Content creators on YouTube have skills rooted in multiple disciplines. Three things are required for a successful YouTube channel: personality, niche content, and quality control. That is not to argue that having these three things will guarantee an aspiring Youtuber fame and fortune. It is unwise to make such promises. However, they are three important ingredients to success as a content creator on YouTube.

  • ·      Personality: This is a no brainer. An interesting and engaging personality will attract more views and subscribers. Everyone has a favorite YouTuber or two. We watch these favorites religiously. We watch them because we like them as a person and we like the content that they produce. The biggest and the most popular YouTubers are ones with recognizable personalities - PewDiePie, JennaMarbles, iJustine, Phillip DeFranco, and KSI to name a few. An opinionated personality makes a YouTuber easy to connect with; it makes them more than a face on a computer screen.      

  • ·         Niche Content: No doubt there are strong communities,, even industries in the YouTube space. A quick search on YouTube and you can easily find the heavy hitters in the Beauty Guru community and the experts in the Tech community. Other genres include Gaming, Lifestyle, and Pranks/Comedy. Having a defined channel ensures that the target audience is reached. It also makes it possible to network and collaborate with other YouTubers within the same genre. It also gives the YouTubers the ability to build camaraderie among the subscribers. Building a community with viewers is extremely vital to creating a loyal fanbase. Some fans are devoted enough to claim that they know their favorite YouTubers better than friends and family. Some even claim their favorite YouTubers are major influencers in their lives.

  • ·         Quality Control: It takes both time and money to create professional-level quality videos on YouTube. Many YouTubers invest in high-end microphone and camera set-ups as their viewership increases. There is also the need to learn how to use professional-grade editing software. Some of the larger channels, such as Unbox Therapy, even invest in separate filming studios. As many of these content creators rely on YouTube as either a primary or secondary source of income, producing quality content is regarded as extremely important.       


The purpose here is not to dissuade future aspirants from pursuing YouTube. It is imperative to remember that becoming a YouTuber in the comforts of one’s own home is not a get rich quick scheme. It requires unwavering dedication, time, and even money to truly have a chance at becoming successful on the platform. Those who start Youtube channels with the intention of becoming the next PewDiePie or IISuperwomanII will find themselves burnt out and discouraged when they realize their growth is not moving as quickly as they had hoped.   


Myth 2: I Can Do YouTube Forever

YouTubers can make an income, even at small amounts, from uploading videos to YouTube. But is it a career? Or even a job? Not quite so. The relationship between YouTubers and YouTube is not equal. Content creators are not employees of YouTube or Google. YouTube, in the end, does not owe content creators anything. There is no salary; there is no employer-employee contract vetted by a HR professional. The income YouTubers make is dependant on ad revenue and sponsorships, not a wage determined by YouTube. Rather than an employee, content creators are users of a service. That is all.  

This unilateral relationship was apparent in the recent YouTube policy changes termed “Adpocalypse” by many Youtubers. This began when the Wall Street Journal targeted PewDiePie after couple videos were deemed controversial. They ran with a sensational “PewDiePie is a Nazi” story. Further investigations led to the discovery that there are more controversial content on YouTube with ads enabled. This instigated a massive exodus of YouTube advertisers.

Another incident was when YouTube tried to be more transparent with their ad revenue policies. The new policy stated that if videos are demonetized by YouTube from that point onward, the content creators would be notified. This, of course, meant that prior to this new transparency policy, YouTube did not notify content creators when videos were demonetized. How ironic!  

This, as one can imagine, caused a ripple throughout the entire YouTube community as many well-known YouTubers’ earnings have dwindled; some by 60-80%. Many content creators have looked far and wide for other sources of income. These include Patreon and other donation set-ups, merchandise stores, moving onto other platforms or creating their own, and seeking sponsorships. A handful have already decided to leave content creation permanently. The moral of these recent events show that YouTubers may not always be able to stay YouTubers full-time. YouTube may not always exist as a viable source of income.

Another concern is the longevity of being a YouTube content creator. Using the trajectory of an educator’s career as an example, we can map out roughly the career path, from a student teacher to associate teacher to senior teacher. Some may enter the administrative side by becoming principals, joining school district administrations, or even, ahem, vying for the position of Secretary of Education. The changes in roles and responsibilities keep things relatively fresh over the course of a career. Variety is the spice of life. The same can be said about work life.

However, a YouTuber is always a YouTuber. The only way YouTubers can grow professionally is by finding new opportunities - which may not be readily available to a beginner YouTuber - and creating a variety of content. Making videos may get monotonous over time. Switching niches may be risky. A tech expert who wants to suddenly make videos about the vegan lifestyle may alienate a portion of their viewership. A career as a YouTube content creator can be inflexible as they are locked inside a bubble. This has been seen many times with content creators from BuzzFeed. In the last couple years, many BuzzFeed content creators have gone rogue in order to create their own unaffiliated content. However, their new content is similar to their BuzzFeed portfolio. Being known for one style of content or for a specific niche effectively typecasts content creators into that bubble.         


Myth 3: #YouTubeIsDying

Given what is discussed above, it may come across as a surprise that the Twitter hashtag is presented as a myth. Being an industrial-organizational psychologist is not just about identifying problems, it is also about figuring how to address them. Creating content for YouTube is not entirely a waste of time, especially for the younger individuals who are still identifying personal strengths and weaknesses. YouTubers develop skills that are relevant to and are valued in other careers. YouTubers are masters of content creation. This particular set of skills is valued in the digital marketing and branding spaces, for example. Another skill everyone must develop is verbal communication, regardless of industry. Learning to use editing software or a new piece of equipment can be accomplished over time. However, verbal communication is the first skill YouTubers need to master.

Soft skills should also be acknowledged. This includes time management, ability to be a self-starter, and manage a project or product development from start to finish. These skills are extremely needed in a variety of jobs. YouTube can be used to learn skills which are useful outside the video sharing platform. If YouTube does cease to exist one day, with careful personal re-branding and portfolio building, a YouTuber could easily find a job that fit their content creation skills.       


The Future of Youtube

What it means to be a YouTuber has already begun to see a transformation. In its humble beginnings, YouTube was used to share content for entertainment. Then it progressed into an income generating platform for content creators. Many YouTubers consider producing content a full-time occupation. Now, society is experiencing the age of influencers, which is a new tactic employed by marketing teams.    

Many YouTubers are already involved in large scale marketing campaigns. More and more YouTubers are becoming “Influencers.” Impacted by their internet celebritydom, YouTubers and other individuals with large social media followings are often targeted as brand and marketing tools. In the beauty industry,  there are many makeup, skin- and hair-care brands who are exclusively YouTube or Instagram-famous. One such brand famously known for YouTube affiliates is Morphe. Among the non-niche content creators, sponsorships, such as those from Crunchroll, LootCrate, SquareSpace, and GlassesUSA, are commonplace. In a political twist, Hillary Clinton sought out endorsements from influential YouTubers during the campaign trail. This does not mean that YouTubers are no longer trustworthy in their opinions and in their content. Most of them continue to create trustworthy content that are enjoyable. This only reveals a growing trend in how online marketing is achieved today.

     So what is the “TL;DR” short answer to the question of whether if YouTubing is a viable career choice? Being a content creator on a video-sharing platform is not a lifelong career move. There is no guarantee that income generating, video-sharing platforms will still be present in 15-20, or even 30 years. However, the skills one develops as a content creator may open up opportunities elsewhere.

One such opportunist is Michelle Phan, one of the original beauty gurus on YouTube. Once she gained large enough of a following, she was able to pursue business ventures. On a smaller scale, other beauty gurus often collaborate in makeup collections with established brands. Philip DeFranco, who creates daily news show content, is in the process of establishing his own independent news network. Casey Neistat, another well known giant on YouTube, controversially sells his company to CNN for $25 million. Neistat’s fans now joke that CNN now stands for Casey Neistat Network.   

All of these examples show that even now, YouTubers are smartly reaching for opportunities that go beyond making videos on YouTube. They acknowledge that in order to survive with a social media-driven career, they must become entrepreneurs and create opportunities for themselves, just as the Kardashians famously (or infamously, depending on your views) did in their rise to fame. Just as how Kylie Jenner is now on her way to $1 billion from her cosmetics line.