Discover
Psych Tech @ Work
Psych Tech @ Work
Author: WRKdefined Podcast Network
Subscribed: 19Played: 882Subscribe
Share
© All rights reserved by WRKdefined
Description
A podcast about safe innovation at the intersection of psychology, tech, and the future of work. Host Dr. Charles Handler, an I/O psychologist and hiring science expert, and his guests take a practical look at the challenges and opportunities AI brings to hiring and talent transformation.
114 Episodes
Reverse
Alex Alonso, Chief Knowledge Officer at SHRM and a Fellow of SIOP, joins Charles Handler to work out how industrial-organizational psychology, artificial intelligence and the HR profession fit together as work is remade by AI. You will learn why HR and IO psychology both have a brand problem, how to make the business case for people science with a proof of concept instead of a budget request, what SHRM means by AI plus human intelligence, and why the resume, the interview and the work sample no longer prove that a candidate can do the job. The episode ends with Alex's view of what assessment has to become now that candidates can fake almost any skill with AI.
Key Takeaways
Only 31 percent of HR professionals know what IO psychology is, and fewer than one in ten who have worked with an IO psychologist want to do it again, because IO rarely shows HR how it creates business value.
About half the behaviors in HRX, SHRM's new framework for an excellent HR function, are IO science, so the value is real but it is not being translated.
The skill that carried Alex's career is speaking data to humans: translating what the research shows into something the people doing the work can use.
Talent analytics earn their keep when they show that the work raised revenue per employee, lifted engagement in ways customers felt, or increased the return on the employer brand.
Do not bring the C-suite an idea and ask for an investment. Run a proof of concept on one site or one team, show the business result, then ask for the resources to scale it.
AI plus HI changes the ROI equation. HI is human intelligence, the judgment, creativity and context a person brings that the machine does not.
SHRM moved from warning members about AI's legal and bias risks to teaching them where a human and an AI working together produce more than either alone.
HR generalist roles are being automated away, and the CHRO is evolving toward a chief work officer responsible for the humans, the technology and how work gets done.
Hiring's proxies are shattered. Candidates use AI to fake resumes, interviews and work samples, and 80 percent of HR professionals say they have hired someone who could not do within three months what they showed in the process.
Assessment has to measure how well a person works with AI, and it has to be dynamic, meaning something a candidate cannot plan for or fake.
Timestamps
01:13 – Speaking data to humans
10:05 – HR's brand problem, and IO's
11:42 – Talent analytics that make HR a value creator
18:49 – Why practitioners never get the performance data
21:06 – Proof of concept before you ask for budget
30:17 – A number for the board next to EBITDA
35:14 – AI plus human intelligence
44:35 – The shattered proxy, skill fishing and ghosts
45:40 – Assessing how people work with AI
50:10 – Dynamic assessment and the ChatGPT history test
Keywords
AI in hiring, candidates faking skills with AI, skills-based hiring, industrial-organizational psychology, making the business case for HR, return on workforce investment, AI plus HI, assessing AI skills in hiring, AI policy at work, future of HR
“There’s tons of plausible summaries of information out there… but is it valid? Is it structured? Is it giving the same opportunity to each of the individuals that are participating? That’s where that science comes in.”
— Tony Boyce
Why is this topic important?: AI now automates sensitive portions of high stakes people processes: hiring, employee listening, job analysis, customer feedback. Its output almost always looks credible, whether or not it is accurate, consistent, or fair, and the people relying on it usually can’t tell.
What is the one thing I will learn from listening?: Three questions to ask any AI tool that gathers information from people. Does it follow a structured guide? Does everyone get the same chance to be heard? Can every result be traced back to what the person actually said?
Episode Overview
I’m joined by my old friend Tony Boyce, an IO psychologist who has always worked at the frontier of science and technology: partner at Aon, principal research scientist at Amazon, and now co-founder of SAVO, a startup using AI-moderated conversations to gather information from people at scale. He also co-chaired this year’s SIOP Leading Edge Consortium on AI and assessment.
We cover how Tony proved AI could reliably score spoken responses at Amazon (and the six-page pitch that won him a 40-developer team), how he builds rigor into AI-moderated conversations at SAVO, why surveys, focus groups, and multiple choice tests may be on their way out, and why tools this powerful still need scientists.
Organizations are always trying to learn from people: what a job requires, why engagement dropped, what customers think, who is ready for a role. AI is moving into all of it. The gap between output that looks credible and output that is defensible is what this conversation is about.
Topics Discussed & Key Insights
1. Plausible Is Not Valid
AI tools that gather information from people are easy to build, and a vendor with no science team can produce output that looks identical to output built the right way. The difference is underneath: the same opportunity for everyone, a structured conversation, and results that can be traced and defended. Since the good and the bad look alike, buyers have to ask how the output was produced.
“You can have a bot and a bot can ask you questions… but in order to actually do it in a way that it stays on the rails, that it follows the right threads, that it structures the conversation, is where the measurement science comes in.”
2. Where the Rigor Lives in an AI-Moderated Conversation
SAVO isn’t a pre-hire assessment tool. It uses AI drive interviews to gather credible information from employees, customers, job experts, and candidates. The rigor lives in three places:
* The spec. An instruction sheet for each conversation: the information needed, the follow-ups, what to listen for. Writing one by hand took three to four days, so an AI tool now drafts about 80% and a scientist finishes it.
* The delivery. One agent builds rapport and runs the conversation. A second “metacognitive” agent tracks what has been learned and steers toward the gaps, within set guardrails.
* The analysis. Each response is broken into evidence units and scored by multiple judges, building a report you can hang your hat on.
“You’ve got all of this amazing unstructured data… but how do you process it in a reliable, valid, traceable way?”
3. What the Evidence Looks Like
At Amazon, Tony’s team trained a model on roughly 2,000 hours of human-labeled recordings to score candidates’ spoken answers to a situational judgment scenario. The evidence had two parts. A number anyone can check: the model agreed with the human raters more closely than they agreed with each other. And known limits: when LLMs arrived in early 2025, the scoring carried over to new content and populations without retraining, but only within similarity limits the team defined.
“Holy moly, this stuff actually works… and it’s reliable.”
4. The Old Trade-offs Are Breaking
Gathering information from people has always meant trading depth (interviews, focus groups, surveys) against scale. AI is removing that trade-off.
Within two years, Tony expects employee listening, customer listening, and user testing to see fewer focus groups, fewer human interviews, and almost no surveys. When an engagement score drops, a survey that held a short conversation, instead of offering an empty text box, will already tell you why. Job analysis interviews can run at scale instead of one at a time. And in hiring, he expects AI conversations and role plays to all but kill the multiple choice test.
“For a century, anytime we’ve wanted to grab some information from humans, we’ve had to choose either depth and rigor… or scale and rigor… But you could never get all three together.”
5. Fragile Systems Need Scientists
AI is powerful and brittle. Tony asked a decision model how confident it was that it would give the right answer: 90%. The wrong answer: also 90%.
I’ve seen the same in AI pipelines for writing assessment items, where fixing one thing quietly breaks another. Trust is unresolved too: executive 360 raters said the experience felt fine but didn’t trust the result. None of this argues against AI. It argues for people whose job is to test it.
“A simple turn of phrase and the decision quality or confidence quality of this thing just fell apart.”
Final Takeaway
AI makes it easy to produce output about people that looks right. Science determines whether it is right. The tools to gather richer information from people at scale exist today. What separates useful from dangerous is whether someone designed for validity, tested for failure, and can show the evidence.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
“We need to have rigor adjusted for reality.” — Allie Wehling
Episode Overview
I am lucky to be joined by my friend Allie Wehling. I consider Allie a superstar. She has achieved what I call the gold standard for internal I/Os.
Allie is a rare example of having an I/O helping guide the business of hiring. At Cisco, she owns both the IO content and the end-to-end hiring process. She sees the whole system and how every piece affects the others.
Most hiring functions treat science as a bolt-on: a test added to a process nobody designed with measurement in mind. That approach has value, but it leaves results to chance. Consistent, defensible hiring requires a holistic approach, where science, process, and business needs are designed together.
Join us as we dig into what an embedded science lens brings to hiring, and why TA can’t afford to go without it.
Topics Discussed & Key Insights
1. TA Leaders: Embed Science, Don’t Bolt It On
Assessment usually gets added to hiring without anyone thinking about the whole process — piecemeal jamming assessments into legacy tools and processes. Allie’s role is built differently. Her teams cover both what gets measured and how it’s implemented, from requisition through pre-boarding. Having one leader across both is rare, and it gets results.
“We have the WHAT, the content, the assessments from our COE, and then the HOW. How are we actually implementing it?”
2. Science Sees the Whole Funnel, Not Just the Test
The assessment is one input. Every handoff, from assessment to interview to decisioning, introduces the potential for bias AND the potential for ROI. The trick is pulling both levers in the right direction. And we can’t forget that candidate experience is critical. When top candidates drop out because of slow scheduling or untrained interviewers, the process fails, not the test.
“because you can have the most beautifully designed and validated assessment and still walk away with a bad hire because the assessment is just one input to the much longer process.”
3. Science Has to Be in Lockstep With the Business
What’s scientifically best and what’s operationally viable are different questions, and both have to be solved. Process and technology have to be designed together. Allie’s team works “two in a box” with recruiting tech, and Cisco has hired an AI orchestration lead so the tools feel like one system.
“they’re two different optimization problems. And if we pretend like they’re one, our systems are going to die quiet deaths.”
4. Science Prepares the Org. for Transformation
Allie listens for the problems leaders haven’t voiced yet. When executives kept asking how Cisco assesses for AI, her team built a library of work samples to measure it. Bespoke assessments built from validated items give hiring managers a hand in the content. An internal AI tool sets cut scores against predicted pass rates instead of gut thresholds.
“I try to listen to what problem are they not even voicing yet.”
5. Science Keeps You Defensible
Job analysis documents job relatedness. Vendor claims get tested, not trusted. If a vendor can’t explain what its score predicts, or who’s on its science team, it’s not the right partner.
“They’re predicting something, but is it the thing that we want to predict?”
Final Takeaway
Speed, volume, and AI are making hiring harder to get right. Science is what turns a pile of tools into a system that works effectively, consistently and holds up to scrutiny. It’s not a test you add. It’s a lens you build in.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
“If you automate chaos you get chaos at scale.”
— Tomas Chamorro-Premuzic
Episode Overview
In this episode I’m joined for a third time by Tomas Chamorro-Premuzic — organizational psychologist, author, and now the first Chief Science Officer at Russell Reynolds Associates, one of the world’s top executive leadership advisory firms.
We set out to role play this conversation as if it were happening in 2027. We didn’t stick to it. We kept sliding between the future, the present, and whatever is right around the corner — which says more about this moment than the role play ever could.
We’ll come back in a year to see how we did. Underneath the predictions is one argument: machines can only take what has already been standardized.
Topics Discussed & Key Insights
1. The Future Is Already Here. It’s Just Not Evenly Distributed.
Tomas recently returned from Shenzhen — stores staffed by humanoid robots, selling humanoid robots, with shoppers carrying them out in boxes. No human employees. His point, borrowing William Gibson: “the future is already here, just not evenly distributed.” What looks weird and niche today becomes mainstream fast. That’s the honest baseline for any one-year prediction: 95% of work will look the same next year. The question is which 5% won’t.
2. You Can’t Automate What Hasn’t Been Standardized
The core flip of the episode. Automation doesn’t take jobs — it takes standardized work. As Tomas puts it: “you cannot automate what hasn’t been standardized already.” Which raises his uncomfortable question: are knowledge workers standardizing themselves? Leaning on AI for everything produces what he calls “work slop and cognitive surrender” — the intellectual version of elevator music. Designed to blend in. Designed to be ignored. The more predictable your output, the easier you are to model, and the easier you are to replace.
3. Hiring for Potential, Not Track Record
His first concrete prediction: organizations will hire leaders less for what they have done and more for what they could do. AI enriches the signals needed to model curiosity, humility, coachability, and EQ — the things good assessment science has been chasing for decades. Technical skills, including the AI literacy everyone is worried about, “will be far less important than we think.” Human and humane skills will matter more. Track record measures what’s already been commoditized. Potential is what’s left.
4. Autonomous Hiring Splits the Market
I ran a live query on my TA Tech Navigator during the episode. TA Tech Navigator is my online market research portal that contains 540 profiles of candidate evaluation and assessment tech. It has several frameworks that classify AI use and Science and can run trends reports using the aggregate data. The trend it flagged: fully autonomous hiring stacks are moving from fringe to viable. There are now vendors where no human touches the process until the final decision. Employers are splitting into two camps — buying efficiency or buying defensibility. Is the tool validated? Is there something here you can trust? That gap will widen, with regulation as the wildcard. The Navigator’s one-year read: faster, cheaper, more automated — but not fairer, more valid, or more defensible unless somebody demands it.
5. The Zombie Problems
Tomas closes with a warning. Obsessing about the future is often an excuse for not dealing with present problems. Two have been with us for decades, and AI doesn’t touch either one: measuring leadership performance objectively, so tools predict actual performance instead of a popularity contest — and our stubborn preference for intuition over validated assessment. The same world that worships data, science, and validity is the world in which the MBTI is the number one personality assessment. New technology doesn’t kill these problems. Only deciding to solve them does.
Final Takeaway
Machines don’t take work. They take standardized work. The defensible position — for a knowledge worker, a leader, or a hiring process — is to stay hard to predict: keep thinking, keep surprising, and measure the things that actually differentiate people. That’s been the job all along. AI just raised the price of skipping it.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
“The way the law is working — the Workday case is exhibit A here — is that vendors are going to be on the hook if their systems are producing biased outcomes. Period.”
— Marc Weinstein
Episode Overview
In this episode I’m joined by Marc Weinstein, an attorney with 27+ years of practice focused on organizations that use tests to make high-stakes decisions about people — licensure, certification, admissions, and employment selection. His work centers on the legal rights of people impacted by AI systems and the organizations that use them.
The legal ground under AI hiring is moving fast. In the past few months a state AI law got sued into a rewrite, a court handed down a ruling that shields bias audit data from view, and a new lawsuit opened a line of attack nobody was watching. If you buy, sell, or use AI hiring tools, this episode helps you stay on top of the shifting sands.
Topics Discussed & Key Insights
1. First, how to roll: governance before headlines.
Marc’s advice starts above the regulations. Your own governance is the key to successful risk mitigation. Get this part right and the rest gets much easier.
* Before you track any law, ask why you want to use AI in the first place. If you have good answers, then ask how
* Set principles for your organization — not for the whole world, for your organization. Let them drive your use cases, your vendor selection, and your evaluation criteria
* Operate by them for real. A policy that lives on paper is worth nothing
* Do this and you’re already in compliance with most of the laws that could touch you. Everything below is what’s shifting underneath that foundation
2. Vendor liability is shifting fast — get ready now.
For a long time, vendors selling hiring tools got a pass. How the customer used the tool was the customer’s problem. That arrangement is breaking down.
* In Mobley v. Workday, the court is letting claims proceed on the theory that the platform acts as an agent of its employer customers — which puts the same federal anti-discrimination laws that apply to employers on the platform
* Plaintiffs go where the deep pockets are. Mega vendors like Workday and 8fold are the deep pockets
* Employers are still independently on the hook. Vendor liability adds to yours, it doesn’t replace it
* Newer legislation points the same direction, and California’s new regulations make bias testing itself evidence in court — more on that below
* Since we recorded: the court refused to dismiss most of the remaining claims against Workday. The case keeps moving, and the core theory keeps surviving
3. Bias audit data can be withheld — and that says a lot.
A discovery ruling in the Workday case protected the company’s bias testing data from the plaintiffs. Worth sitting with what that means.
* The court held the data was privileged because lawyers curated it and the testing was done to provide legal advice — not for business use
* Publicly advertising that you conduct bias testing did not waive the privilege
* So the people alleging harm from the system may never see the data that would show whether the system harmed them
* The bigger question: if the most important evidence about how these tools perform can be shielded, what does accountability actually look like? Proof may have to come from other directions
4. There are new ways to hold these tools accountable.
The Eightfold case shows creative lawyering finding routes into AI hiring that didn’t exist a year ago.
* The claim is built on the Fair Credit Reporting Act and privacy law — the argument is that AI-generated candidate scores are consumer reports, compiled without the required disclosures and consent
* These theories don’t require proving discrimination as a legal element, which makes them easier cases to bring
* The target hasn’t changed. It’s still about how these tools treat people. There are just more ways to get there now
5. Colorado is the cautionary tale: get aggressive, get crushed.
Colorado passed the most rigorous state AI law in the country — mandatory risk management, impact assessments, a real duty of care. Then it got taken apart.
* xAI sued to block the law. The DOJ intervened on xAI’s side — the first time the federal government moved to invalidate a state AI law
* The legislature rewrote the law under pressure. The replacement is a notice-and-transparency framework, delayed to January 2027
* The lesson for everyone else: states that regulate AI aggressively are going to get sued by the federal government and the private sector. Plan around that reality
6. Meanwhile, California quietly became the strictest.
While everyone watched Colorado, California put real requirements on the books — and nobody has sued to stop them.
* Under the FEHA regulations, conducting a documented bias test supports an employer’s defense in discrimination litigation. Not conducting one supports the plaintiff. Records must be kept four years
* The privacy agency’s automated decision-making rules take effect January 2027: pre-use notice to applicants and employees, opt-out rights, and the right to demand how a system reached a decision about you
* That last one matters. Can anyone actually explain how a frontier AI model reached its decision? For most of these tools, the honest answer is no
Final Takeaway
The laws will keep shifting. Colorado got rewritten, California’s deadlines are coming, and the courts are redrawing vendor liability case by case. You can’t chase all of it. Build the governance foundation, make vendors show you their evidence — your data, not just their study — and treat every recommendation a machine makes as the decision it really is. Good governance gets you more than 90% of the way there, as long as it’s not just lip service.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com







