Psych Tech @ Work

This episode explores the various ways social media information is being used to evaluate job applicants and provides expert opinions on their viability and general acceptance. Special guest Dr. Shawn Bergman’s expertise in the area of social media for hiring, as well as his own research provide a great backdrop for filtering general perceptions about this hot topic. The conclusions drawn provide a useful context for anyone interested in exploring the use of social media data as an assessment tool. Before looking at the various ways social data is used in hiring, it is important to look at the some of the issues that apply to using social media data to make employment decisions.ConcernsThere are four overall areas of concern when it comes to using social data as an assessment. 1. Problems with source materials- There are a great deal of individual differences in the use of social media. For instance, both Dr. Bergman and Dr. Handler admit that they are not very active on social platforms. It seems logical that a lack of data based on infrequent posting or not using a particular channel- could have a negative impact on an applicant’s evaluation. The potential differences in how much people post and where- represent a challenge when comparing them to others who may post frequently to many channels. 2. Accuracy- The accuracy of social media data in measuring human traits is speculative at best. Without confidence in construct related measures- the value of social data as a predictor of job performance is suspect. Of course it is possible to simply correlate patterns extracted from social data with performance metrics- but this brings us right back to the same issues raised with most AI based evaluation tools. Unfortunately, many companies making social media assessment tools do not involve I/O psychologists - adding additional concerns when it comes to the measurement of actual job related constructs. 3. Privacy- The default assumption with social data in hiring is that the individual being evaluated has not given specific consent for their information to be used. This is probably the most talked about issue when it comes to social media data for hiring, and one that will not go away anytime soon. One of the main problems in the realm of privacy is that it is often impossible for an individual to know their data is being used to evaluate them. While opting in is becoming a standard requirement- it is virtually impossible to police the use of tools that harvest and evaluate social media data. There is also the question of who owns social data. Questions of data ownership can turn the concept of privacy on its ear- requiring legal precedent and legislation to sort out. 4. Bias- There are several ways bias can enter the room when it comes to hiring and social media data. Any process in which profiles are reviewed manually presents a serious snakepit when it comes to bias. Automated tools are also famous for creating systematic biases when making evaluations. While there is some great work being done to train AI/ML to actually reduce bias, the fact remains that social media data helps keep the very real issue of bias alive and well. Use CasesDr. Bergman reports that 70% of organizations use some form of social or public data when evaluating applicants. So how are they using these tools?Manual review of profiles/postsWhile the sexiest and most talked about use case centers around the use of AI based tools to systematically evaluate data from social media accounts and posts, the lowest hanging fruit when it comes to the use of social data for hiring decisions is the simple review of profiles by humans. Social media data is often used for a very low tech and manualized process of reviewing profiles of job candidates to look for inappropriate information, revealing posts, etc. This process is accessible to anyone with a computer, setting up many a disaster when it comes to subjectivity, accusations of immorality, etc.Human review of profiles for problematic behavior definitely opens up a great deal of concern. It is accessible and there is no accountability for reporting the results of these often ad hoc evaluations. While these evaluations can be outsourced to firms that specialize in the evaluation and return a report, this does not legitimize the method. The available research in this area shows that there is no relation between these evaluations and performance on the job. Furthermore, this type of easily disadvantages protected classes, and its job relevance is often hard to demonstrate. AI based toolsThe core of all these tools is tech that spiders the web to find profiles and information, scrapes the data, and then interprets and processes it into an output that can be used to support decision making. The tech can work passively- based on open web searches with no opt ins, or more actively - with applicants opting in to share information that can be used to evaluate the applicant. Social media data is often use

In this episode listeners are treated to 30 minutes of storytelling and inspiration from our special guest Mark Newman, Managing Partner of Liam June Ventures, a prolific investor and founder of pioneering video interviewing company, HireVue.As a founder of HireVue, a trailblazer in video interviewing and video assessment, Mark has been at the intersection of hiring and technology for most of his entire career. The episode kicks off with the fascinating story of HireVue's inception and early days and moves on to discuss the role of advanced tech in shaping the future of I/O psychology and selection science.The entire conversation is inspiring. From Mark’s perspective creating the future is about embracing change and maintaining a long term focus while being patient enough to accept the limitations of the moment. Mark brims with optimism about the future of selection science and assessments, stating that.."From an I/O perspective, we're entering into a golden age of the principles of I/O Psychology being applied in ways that no one that entered the field, 10, 20, 50 years ago every imagined."The conversation then turns to the journey.We are still in a time when AI based assessment is not appealing to every employer. Finding the future is about progressive employers who are willing to team up with good scientists to thoughtfully test tools and technology.Mark’s take on the bias question is to remind us not to be scared of the present state, to understand that current tools have bias and that the desire to remove it will ensure that technology is managed to this end. It is important to understand that the careful oversight and expertise of psychologists exert a powerful directional force on the management of unintended consequences from AI based hiring tools.In the end we are still in the larval stages of building the organizations of the future. These organizations will use I/O psych and data science at scale to ensure hiring is inclusive - allowing people an easier path to the job of their dreams. For now- there is still lots of work to be done. When it comes to creating the future- perhaps the hardest work for many will be checking their fear at the door. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

While the EEOC’s Uniform Guidelines on Employee Selection Procedures (aka UGES) provide a solid foundation when it comes to the do’s and don'ts of employment testing- it is always nice to hear about the EEOC’s stance on key issues straight from the horse’s mouth.In this episode special guest Dr. Romella El Kharzazi, of the Equal Employment Opportunity Commission teaches us about the EEOC’s mission and their stance on all things employment testing. This enlightening conversation also includes a good deal of talk about the agency’s stance on AI based assessment tools. This episode is a must listen for anyone who is considering using AI based assessment tools but is concerned about the legal risk.Here is a quick outline of the key messages that arose during the conversation.Getting to know the EEOCThe EEOC sees themselves as “ A force for good”- focuses on the core mission of fighting discrimination and making sure that everyone has an equal shot when it comes to employment.The EEOC’s priorities are: to protect vulnerable populations, to educate businesses on discrimination and how to avoid it, and to “build opportunities” for everyone no matter who they are or where they come from The EEOC is not just about enforcement- they also create policy and do research into all areas of employment discrimination with a special focus on monitoring the employment climate at all federal agencies.The EEOC and pre-hire testingWhen it comes to testing, the EEOC’s core focus is on working with employers to monitor, manage, and eliminate bias from the hiring process. This is done via a formal complaints and investigation process that is geared towards working with employers to eliminate bias more than it is towards dragging them into court.The requirements for compliance in hiring and testing are laid down by the UGES. The good news is that the UGES has been around a long time and legal and I/O Psych fields know them well and have been building testing products and programs that are compliant. Beyond the specifics of the UGES- the advice for staying on the EEOC’s good side includes: Job analysis- Establishing job relatedness of selection procedures is critical in all situations and should not be ignored. Especially when it comes to AI based tools, the absence of a job analysis will create exposure and risk.Record keeping- Companies are on the hook for keeping applicant records- with no exceptions. It is not acceptable to claim exemption from the rules because your firm does not keep any records. While it is not mandatory for applicants to supply demographic data, compliance requires that there is a place in the application process for applicants to provide it. Failure to attempt to collect applicant demographic data creates exposure and risk Be proactive- It is always required that employers seek alternative measures to replace those that may have adverse impact. Claiming business necessity for tests that show adverse impact only goes so far. Instead - it is your obligation to search for alternatives that have less adverse impact. Employers should be proactive and when aware of a problem- seek to fix it instead of keeping status quo and hoping they are not challenged.AI based toolsWhile the EEOC does not yet have a big data policy, usual rules apply to AI based tools. It is really important to understand that the EEOC does not feel that all AI based tools have no merit. As long as the development, calibration, and use of the tools meet the requirements of the UGES, all is good. While there has yet to be a case related to the use of AI based assessments- the EEOC is paying attention and is active in monitoring how these tools are used.So, how do we leverage the benefit of AI based tools without increasing exposure and risk?First and foremost - there is a need to be sure that companies building and using these tools create a seat at the table for persons who understand the ins and outs of employment testing. Companies who rely purely on data science and empirically driven relationships are at risk. Adding I/O psychologists to the mix makes a ton of sense both for ensuring compliance and helping ensure the human side of hiring is properly represented. A hiring assessment company with no I/O on staff immediately sends up a red flag. I/Os can help ensure that employers don’t overfit models to data sets, creating unreliable prediction across locations and over time. It is paramount that employers do not simply push the blame for bias associated with a specific tool. Humans program and train AI based tools and it is people who make the ultimate decision to hire an applicant or reject them. Bias can easily enter the equation when unsupervised learning is used so there is a need for caution when using these tools. For instance features such as social media data can be chock full of landmines that are biased against protected classes such as zip code and consumer behaviors. It is really important not to confuse the use of AI for custome

Dr. Ed Levine, Professor Emeritus in Industrial/Organizational Psychology at the University of South Florida, and a long time friend of Dr. Charles Handler joins us in this episode.Dr. Levine literally wrote the book on job analysis. In this episode Dr. Levine shares decades of wisdom about the how and why of job analysis. The conversation takes an existential turn and provides really refreshing perspective about the true essence and value of job analysis.So what is job analysis anyway and why is it so important? To answer this questions Episode 4 includes a great discussion about the key inputs and outputs of job analysis, but then reaches into a refreshing philosophical territory.Points of interest in the discussion include:Essentially job analysis translates what people do on a job into words. While job analysis may seem cold and boring on the surface, it is actually a spiritual thing in many ways because it involves a search for meaning within the work people do.Job analysis highlights the attributes needed for success at a job- providing a measuring stick for job applicants. The ability to hire based on the fit between applicants’ match to the human traits required for job success is of course critical to business outcomes. But job analysis has even more value because it helps individuals find work that is meaningful to them- therefore offering a larger benefit to society.The entire concept of a job is now getting outmoded so the view is turning to “work analysis” this expands the domain to include emotions, teams, personality factors, etc. This covers the work process not just a job. The future of job analysis is really bright. Beyond the applications for selecting employees, job analysis will continue to be a foundational aspect of the relation between humans and work. As the concept of a job changes, so too will job analysis. We can expect to see the scope expanded beyond just the study of one person’s role in one discreet job. Instead we can expect to see job analysis extending into “work analysis” that transcends the walls of one organization. Work analysis will focus on the study of the things required for various types of work. As work continues to become more team based in nature, we can also expect job analysis to be applied to the study of what teams need to successfully accomplish valued objectives. New technologies such as sensors and IOT devices will allow us a much deeper view on how teams interact in the workplace. Insights from this work will help promote a better understanding of the traits required to optimize the human elements that impact the success of work teams.We can expect to see advances in technology contribute to the evolution of job analysis. In the future we can expect that job analysts will have access to database that contain the results of thousands of job analysis studies. Think of an evolution of the O*net system to include access to accumulated job studies will allow analysts to instantly dial up job profiles that are fairly complete. Imagine this data being yoked to AI based systems that provide a deeper level of intelligence about the human traits required for success. Job analysis is a foundational part of any talent assessment program. Building an assessment without a job analysis is like building a house without a blueprint. But as we see in this episode, job analysis has a lot more to offer.All in all- Job analysis will continue to be a bright star in the understanding of humans and their work.Dr. Levine is a prolific author, speaker and consultant, working with governmental agencies, across the private sector and most recently with NASA on Astronaut selection. You can learn more about Ed by visiting his site: http://luna.cas.usf.edu/~elevine/And watch for the 3rd Edition of his book, "Job and Work Analysis" here: http://sk.sagepub.com/books/job-and-work-analysis This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

Ben Hawkes is an I/O psychologist who is a thought leader in area of game based pre-employment assessment tools. Ben is currently the co-Founder of BlackHawke Behavior Science and has served as the Selection Assessment Lead for Shell International since 2016. Ben’s background and experience make him a perfect resource for helping Dr. Handler answer the question, "Are game-based assessments the future of employment testing?"The short answer as revealed in this episode is: Don’t believe the hype!Game based assessments have a ton of value and are are PART of the future, but they have the same limitations as traditional assessments (if not more). Game based assessments will make a positive and essential contribution to the future of assessments because they will help us learn how to make assessments more lifelike and enjoyable. This episode touches on few key points to help educate and orient those interested in game based assessments1. Gamified assessments are not true games It is important to know what you are looking at. It is important to be careful not to believe the hype that a gamified assessment is really a game. Doing so is made harder because it is difficult to say exactly what is a game. Games have to meet certain requirements such as the presence of rules, an enjoyment factor, suspense, and control by the player. But there are grey areas around the true definition and it is common for things that have game like elements to be called games when they really aren’t.Gamified assessments- that is assessments with game like elements, are becoming increasingly common and are more likely to be an integral part of the future of assessment.Gamification has many advantages because they make assessments more enjoyable but allow them to maintain the characteristics of a good measurement tool. This is essential because when all the dressing is stripped away- assessments must be psychometrically sound measurement devices.Assessments that provide gamified elements such as: feedback, choices, dynamic user interfaces, and realistic environments definitely have a lot going for them. The upgraded user experience provided by gamification means we can expect to see an increase in their use. Just know they are not truly games and that gamification alone is not enough to make an assessment legit.2. Where does the risk lie?A good rule of thumb when looking at game based assessments is don’t buy based on the sizzle because the steak may taste like crap.For those shopping they first need to know that to be usable as an assessment a game must meet the minimum standards of being fair, reliable, and valid. Many tools may look like or be called an assessment but may not fit these minimum requirements. One type of game that is commonly mistaken for real assessment is the “attraction games”. These are branded experiences that allow the job seeker to interact with a job or organization. They are great for employment branding but typically are not designed to be measurement tools.Another type of game that must be approached with caution are “non-contextualized” games. These games take place in a simulated environment such as outer space or under the ocean. While these are sold as being attractive to applicants they may actually have the opposite effect. Candidates actually value assessments that appear job relevant more than they value being entertained by the experience. In the world of candidate experience, fairness is king and candidates value job relevance over fun. Personality tests alone, be they games or regular measures, are not strong predictors of work performance. So it is not a surprise that personality games have struggled to be effective as selection tools and their ability as strong predictors of applicant performance should be met with a healthy dose of skepticism. Yes, it is possible for games to measure aspects of personality but they have not proven to do so with at the accuracy level of more traditional assessments. Many games, including those that claim to measure personality, are sold based on the number of data points they generate about an applicant. These data points are called “paradata” and they are the byproduct of all the actions within the gaming experience. The sheer number of data points does not mean an assessment is a good measurement tool. Measuring constructs that underlie work performance requires understanding what you are measuring first and foremost and then creating accurate and reliable measurement tools. Just because you have millions of data points does not mean the patterns that belie personality are found within it. Finally- games may not meet accessibility standards required of selection tools. It is important to fully evaluate any assessment games for compliance to ADA standards.3. So what can we trust?Gamified assessments are great as long as they are held to the same standards as regular assessments. Currently the strongest type of assessment games are cognitive games. These games hav

There is no one better to weigh in on this most timely of topics than Gerry Crispin, co-founder of CareerXroads and the TalentBoard and long time champion of those who demand more from their hiring experience.In this episode provides an honest answer to the question:"Can assessments and a positive candidate experience peacefully coexist?"Gerry makes the answer simple: Candidates do not hate assessments at all. What they hate is engaging in a hiring process that seems unfair. Perceptions of fairness are driven by many things- where the assessment falls in the talent acquisition workflow, selling the value of the assessment to the candidate, and feedback from recruiters. But the #1 factor in job candidate’s positive feelings about an assessment is a clear link between the content of the assessment and the job they are applying to (aka “face validity”). The idea that candidates drop out of the process due to the length of an assessment is a myth. Candidates disengage from the process when they feel their time is being wasted by irrelevant questions delivered in what appears to be a vacuum.Beyond this revelation, Gerry provides some great ideas for those who want to make pre-hire assessments a value add to the candidate experience and delivers the good news that they can actually play a starring role. You can follow Gerry on Twitter at https://twitter.com/gerrycrispinConnect on LinkedIn here: https://www.linkedin.com/in/gerrycrispin/And Learn more about CareerXroads here: https://cxr.works This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

Dr. Richard N. Landers is the John P. Campbell Distinguished Professor of Industrial-Organizational Psychology at the University of Minnesota, and directs the TNTLAB (Testing New Technologies in Learning, Assessment and Behavior). Dr. Lander’s is one of the foremost authorities on advanced assessment methods. His research on advanced gamification, AI, and Virtual reality based assessments techniques make him the perfect person to answer the question,"Are advanced assessment technologies overhyped in 2018?"This episode’s conversation seeks separate fact from fiction when it comes to AI based pre-hire assessment tools. In discussing the current state of advanced assessment techniques- consideration is paid to both the benefits and drawback of techniques that are being hyped as revolutionary (e.g., Natural Language Processing, Machine Learning, Gamification, and Virtual Reality).Dr. Landers shares his view that these tools are still in their early days and a rush to widespread application over more trusted methods may be premature. Dr. Landers also highlights bright spots in the current state of advanced assessment tools and provides suggestions for getting the most out of them.Through TNTLAB, Dr. Landers trains Ph.D. students and undergraduates in this research area and I/O psychology more broadly. If you are interested in becoming a part of TNTLAB in either of these roles, please see this page http://rlanders.net/join/. If you are interested in learning more about TNTLAB research, please see their Research Areas to learn more: http://rlanders.net/research/ This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

Coming December 2018, Science 4-Hire with Dr. Charles Handler. The first and only podcast about everything talent assessment! A new episode featuring experts in the field discussing the latest developments in hiring every two weeks where ever you find Podcasts and always at Rocket-Hire.com This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com