Psych Tech @ Work

“If you automate chaos you get chaos at scale.”— Tomas Chamorro-PremuzicEpisode OverviewIn this episode I’m joined for a third time by Tomas Chamorro-Premuzic — organizational psychologist, author, and now the first Chief Science Officer at Russell Reynolds Associates, one of the world’s top executive leadership advisory firms.We set out to role play this conversation as if it were happening in 2027. We didn’t stick to it. We kept sliding between the future, the present, and whatever is right around the corner — which says more about this moment than the role play ever could.We’ll come back in a year to see how we did. Underneath the predictions is one argument: machines can only take what has already been standardized.Topics Discussed & Key Insights1. The Future Is Already Here. It’s Just Not Evenly Distributed.Tomas recently returned from Shenzhen — stores staffed by humanoid robots, selling humanoid robots, with shoppers carrying them out in boxes. No human employees. His point, borrowing William Gibson: “the future is already here, just not evenly distributed.” What looks weird and niche today becomes mainstream fast. That’s the honest baseline for any one-year prediction: 95% of work will look the same next year. The question is which 5% won’t.2. You Can’t Automate What Hasn’t Been StandardizedThe core flip of the episode. Automation doesn’t take jobs — it takes standardized work. As Tomas puts it: “you cannot automate what hasn’t been standardized already.” Which raises his uncomfortable question: are knowledge workers standardizing themselves? Leaning on AI for everything produces what he calls “work slop and cognitive surrender” — the intellectual version of elevator music. Designed to blend in. Designed to be ignored. The more predictable your output, the easier you are to model, and the easier you are to replace.3. Hiring for Potential, Not Track RecordHis first concrete prediction: organizations will hire leaders less for what they have done and more for what they could do. AI enriches the signals needed to model curiosity, humility, coachability, and EQ — the things good assessment science has been chasing for decades. Technical skills, including the AI literacy everyone is worried about, “will be far less important than we think.” Human and humane skills will matter more. Track record measures what’s already been commoditized. Potential is what’s left.4. Autonomous Hiring Splits the MarketI ran a live query on my TA Tech Navigator during the episode. TA Tech Navigator is my online market research portal that contains 540 profiles of candidate evaluation and assessment tech. It has several frameworks that classify AI use and Science and can run trends reports using the aggregate data. The trend it flagged: fully autonomous hiring stacks are moving from fringe to viable. There are now vendors where no human touches the process until the final decision. Employers are splitting into two camps — buying efficiency or buying defensibility. Is the tool validated? Is there something here you can trust? That gap will widen, with regulation as the wildcard. The Navigator’s one-year read: faster, cheaper, more automated — but not fairer, more valid, or more defensible unless somebody demands it.5. The Zombie ProblemsTomas closes with a warning. Obsessing about the future is often an excuse for not dealing with present problems. Two have been with us for decades, and AI doesn’t touch either one: measuring leadership performance objectively, so tools predict actual performance instead of a popularity contest — and our stubborn preference for intuition over validated assessment. The same world that worships data, science, and validity is the world in which the MBTI is the number one personality assessment. New technology doesn’t kill these problems. Only deciding to solve them does.Final TakeawayMachines don’t take work. They take standardized work. The defensible position — for a knowledge worker, a leader, or a hiring process — is to stay hard to predict: keep thinking, keep surprising, and measure the things that actually differentiate people. That’s been the job all along. AI just raised the price of skipping it. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

“The way the law is working — the Workday case is exhibit A here — is that vendors are going to be on the hook if their systems are producing biased outcomes. Period.”— Marc WeinsteinEpisode OverviewIn this episode I’m joined by Marc Weinstein, an attorney with 27+ years of practice focused on organizations that use tests to make high-stakes decisions about people — licensure, certification, admissions, and employment selection. His work centers on the legal rights of people impacted by AI systems and the organizations that use them.The legal ground under AI hiring is moving fast. In the past few months a state AI law got sued into a rewrite, a court handed down a ruling that shields bias audit data from view, and a new lawsuit opened a line of attack nobody was watching. If you buy, sell, or use AI hiring tools, this episode helps you stay on top of the shifting sands.Topics Discussed & Key Insights1. First, how to roll: governance before headlines.Marc’s advice starts above the regulations. Your own governance is the key to successful risk mitigation. Get this part right and the rest gets much easier.* Before you track any law, ask why you want to use AI in the first place. If you have good answers, then ask how* Set principles for your organization — not for the whole world, for your organization. Let them drive your use cases, your vendor selection, and your evaluation criteria* Operate by them for real. A policy that lives on paper is worth nothing* Do this and you’re already in compliance with most of the laws that could touch you. Everything below is what’s shifting underneath that foundation2. Vendor liability is shifting fast — get ready now.For a long time, vendors selling hiring tools got a pass. How the customer used the tool was the customer’s problem. That arrangement is breaking down.* In Mobley v. Workday, the court is letting claims proceed on the theory that the platform acts as an agent of its employer customers — which puts the same federal anti-discrimination laws that apply to employers on the platform* Plaintiffs go where the deep pockets are. Mega vendors like Workday and 8fold are the deep pockets* Employers are still independently on the hook. Vendor liability adds to yours, it doesn’t replace it* Newer legislation points the same direction, and California’s new regulations make bias testing itself evidence in court — more on that below* Since we recorded: the court refused to dismiss most of the remaining claims against Workday. The case keeps moving, and the core theory keeps surviving3. Bias audit data can be withheld — and that says a lot.A discovery ruling in the Workday case protected the company’s bias testing data from the plaintiffs. Worth sitting with what that means.* The court held the data was privileged because lawyers curated it and the testing was done to provide legal advice — not for business use* Publicly advertising that you conduct bias testing did not waive the privilege* So the people alleging harm from the system may never see the data that would show whether the system harmed them* The bigger question: if the most important evidence about how these tools perform can be shielded, what does accountability actually look like? Proof may have to come from other directions4. There are new ways to hold these tools accountable.The Eightfold case shows creative lawyering finding routes into AI hiring that didn’t exist a year ago.* The claim is built on the Fair Credit Reporting Act and privacy law — the argument is that AI-generated candidate scores are consumer reports, compiled without the required disclosures and consent* These theories don’t require proving discrimination as a legal element, which makes them easier cases to bring* The target hasn’t changed. It’s still about how these tools treat people. There are just more ways to get there now5. Colorado is the cautionary tale: get aggressive, get crushed.Colorado passed the most rigorous state AI law in the country — mandatory risk management, impact assessments, a real duty of care. Then it got taken apart.* xAI sued to block the law. The DOJ intervened on xAI’s side — the first time the federal government moved to invalidate a state AI law* The legislature rewrote the law under pressure. The replacement is a notice-and-transparency framework, delayed to January 2027* The lesson for everyone else: states that regulate AI aggressively are going to get sued by the federal government and the private sector. Plan around that reality6. Meanwhile, California quietly became the strictest.While everyone watched Colorado, California put real requirements on the books — and nobody has sued to stop them.* Under the FEHA regulations, conducting a documented bias test supports an employer’s defense in discrimination litigation. Not conducting one supports the plaintiff. Records must be kept four years* The privacy agency’s automated decision-making rules take effect January 2027: pre-use notice to applicants and employees, opt-out rights, and the right to demand how a system reached a decision about you* That last one matters. Can anyone actually explain how a frontier AI model reached its decision? For most of these tools, the honest answer is noFinal TakeawayThe laws will keep shifting. Colorado got rewritten, California’s deadlines are coming, and the courts are redrawing vendor liability case by case. You can’t chase all of it. Build the governance foundation, make vendors show you their evidence — your data, not just their study — and treat every recommendation a machine makes as the decision it really is. Good governance gets you more than 90% of the way there, as long as it’s not just lip service. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

“You buy the hiring platform, and then you don’t have these AI workflows enabled... here you are three, four years later. You’re still not able to do that.”— Nicole MundyEpisode OverviewIn this episode I’m joined by Nicole Mundy, Senior Research Analyst at Talentech Labs — a research and advisory firm that helps its enterprise clients evaluate and procure hiring technology systems.Nicole brings a valuable perspective to the table because she sees the dynamic between vendor and buyer up close and personal. When combined with my experience in this same realm from the science side- our discussion shines light on the reality of what is happening in AI tech adoption for TA.Topics Discussed & Key Insights1. Companies are buying AI hiring tools at scale — and then leaving them switched off.Among the world’s top enterprise organizations, Nicole estimates a surprisingly small amount are actually using AI to automatically assess active applicants.Why?Approvals never come - Companies buy the platform intending to enable the AI, “once the right approvals are in place- but years often pass without any change.Pilots underperform - Big companies test these tools and often conclude they can’t really use them, or they just didn’t work.Lack of solid ROI evidence - Despite the best intentions- most companies do not do the follow up work needed to demonstrate the impact of these tools on the bottom line. Legal ambiguity freezes decisions - With regulations constantly in flux, risk management often takes priority over business needs.2. Validation is misunderstood and absent.What vendors without I/O science guidance call validation isn’t what legal compliance actually requires.* Vendors are quick to speak about the validity of their solution and talk endlessly about validating their AI models — running statistical checks that the model predicts consistently and de-biasing its outputs across groups. This is purely empirical work.* But that’s IT-style validation. It confirms the system runs as built; it says nothing about whether the tool is fair or job-related* Validation for legal compliance, and sound science, demands a blend of rational and empirical work to document the job-relatedness of any tool used to make employment decisions3. “Skills” are everywhere, and nowhere.Skills-based hiring is the headline everyone wants. The problem is what counts as a skill.* Most platforms apply the “skill” label with no objective framework to define it. A skill ends up being little more than a tag like “Excel” or “communication”* The definitions behind these labels are usually poorly organized and loosely constructed. * There is no connection between the skills a platform claims to measure and any outcome on the job. Without that link, there is nothing for a buyer to trust or defend.* The companies doing skills-based hiring well are not buying one vendor and flipping a switch. They run multi-year programs: define the skills objectively, inventory what the organization has against what it needs, curate tools carefully on the front end, and collect assessment data at multiple points.4. Cheating is a zero-sum game, so let’s change the rulesAI-assisted candidate fraud, from AI completing assessments to coaching candidates through interviews, is driving enterprises toward more dynamic evaluation that is harder to game. But chasing detection is largely a losing battle.* Trying to catch and block AI use is whack-a-mole. It’s a zero-sum game, and it’s frustrating, because the tools keep getting better and the detection never really gets ahead* There’s a more useful way to think about it. People are going to use AI on the job, so why not let them use it in the application process in a controlled way? The question stops being “did they use AI” and becomes “how well do they use it”* Most large organizations are still just trying to get visibility on how much cheating is happening and what it looks like, which says how early everyone is on this.* Across the approaches Nicole sees, one thing tends to hold up whether a process is locked down or fully AI-assisted: competency-based follow-up questions that make candidates explain their own reasoning in their own words.Final TakeawayEnterprise isn’t slow on AI in hiring because it doesn’t understand the technology. It’s slow because the tools are bought on vendor claims that were never reviewed against real science, and the danger only becomes clear once the tool is in play. The companies getting it right aren’t chasing tools. They’re building programs, science first. Held to that bar, many of the tools on the market today wouldn’t survive the review, and the ones that would wouldn’t be sitting switched off. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

In this episode I’m joined by Robert Newry, Founder & CEO of the assessment company Arctic Shores and long time champion of doing assessment right! Robert and I (and my AI co-host Mayda Tokens!) dig into one of the most urgent problems in hiring right now: the complete breakdown of traditional hiring signals.We ponder the question- “How do we find the truth in an age where AI has flooded the top of the funnel, made credentials and resumes unreliable, and put enormous pressure on organizations to find new ways to identify talent?”And we come up with some pretty good answers!1. The Top of the Funnel Is in ChaosThe numbers are staggering. Accenture’s global resourcing lead told Robert they’re on pace for 12 million applications this year for roughly 100,000 hires — up from 4 million just three years ago. Same size team. Two and a half times the volume. The culprit isn’t a surge in qualified candidates; it’s AI-powered application tools that let candidates apply to jobs while they sleep. The moral contract between candidates and employers has been broken: candidates assume companies are using AI to screen, so they’re using AI to apply.“It’s chaos out there. Candidates are using AI to fight AI — and we’re in a no-win scenario.”2. Traditional Assessment Is Increasingly GameableArctic Shores’ research from 18 months ago showed what most people didn’t want to admit: AI can ace virtually any traditional assessment format — personality tests, cognitive reasoning, multiple choice — with ease. And it’s not just about having a second screen open. Candidates can now point a phone at their screen, have the AI read the item, and get the answer instantly. Proctoring doesn’t solve this. The old protection mechanisms are obsolete.3. The Answer Is Better Signal, Not More AIThe solution isn’t to ban AI from the process — it’s to design assessments that AI can’t easily game because they’re rooted in authentic behavior. Robert’s framework: if AI is being used to evaluate signals, those signals have to be grounded in high-fidelity behavioral data — not scraped from job descriptions, not inferred from keyword matching, not built on garbage in. Job descriptions themselves are often the first failure point, and no amount of downstream AI sophistication fixes a weak foundation.4. Stop Counting Leaves — Look at the RootsRobert’s tree analogy is one of the sharpest frameworks in this episode. For decades, hiring has been obsessed with leaves — the skills on a resume, the credentials on a LinkedIn profile. But with the average shelf life of a skill now estimated at two and a half years, leaves are increasingly unreliable. What matters is the root system: the durable human capabilities that allow someone to grow new skills, adapt to changing roles, and thrive in uncertainty.5. Skills-Based Hiring Needs a Clearer Definition of “Skill”Both Robert and I agree: the skills-based hiring movement is directionally right, but conceptually messy. Calling “innovation” or “persistence” a skill conflates what can be learned with what is innate. Durable traits — personality, cognitive style, learning orientation — don’t expire the way technical skills do. Measurement strategy has to account for these differences, or skills-based hiring just becomes the next echo chamber.Final TakeawayThe hiring signal crisis is real — and it’s accelerating. AI has made it trivially easy to fake credentials, game traditional assessments, and flood the funnel with noise. The organizations that receive the best signal won’t be the ones that deploy the most AI. They’ll be the ones that invest in the right signal: behavior-based, validated, and rooted in the durable human traits that no machine can fake.*Claude.ai assisted with the creation of these show notes This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

“By the time you dot the final I’s and cross the final T’s, the assessment is already out of date.”— Taylor SullivanEpisode OverviewIn this episode I’m joined by rising I/O rockstar Taylor Sullivan, IO psychologist and the architect of Workera’s assessment strategy. With Taylor’s guidance Workera, a verified skills intelligence platform, is doing something most of the industry is still afraid to do: going all in on using AI to build, deliver, and validate AI-based assessments.Taylor and I (and my AI co-host Mayda Tokens) dig into how this actually works, why it’s scientifically defensible, and why the industry needs to stop waiting and start moving.Topics Discussed & Key Insights1. Traditional Assessment Development Is Already BrokenBy the time a traditional assessment clears all the I-dotting and T-crossing, it’s often already out of date. AI changes that — enabling dynamic content generation, richer construct understanding, and real-time iteration that keeps pace with how work actually evolves.2. Codifying Measurement Science Into a Multi-Agent SystemWorkera didn’t just bolt AI onto existing processes. They embedded IO psychology’s core principles — evidence-centered design, validity frameworks, job analysis — directly into a multi-agent authoring system. Experts define the standards. Agents execute to those standards. The science drives the machine, not the other way around.Here’s a brief sketch of how it works in practice* Define the purpose — Tell the agent what you’re measuring and why. This grounds everything that follows.* Extract the construct — The agent probes the skill space using critical incident techniques, identifying what great performance actually looks like.* Design the assessment — The agent selects question formats (multiple choice, drag and drop, voice interaction, sequencing) based on what will best elicit evidence of the skill.* Automated quality review — Before anything goes live, the system checks for bias, language issues, and content alignment to the original skill definition.* Monitor and improve — Once deployed, the agent tracks response patterns, flags problems, and learns from score appeals adjudicated by humans.The skill domain is flexible — it works for cheeseburgers or cybersecurity. The methodology behind it is the same either way.3. The “Harness” — Why This Is SafeThe key to responsible agentic AI isn’t less autonomy — it’s a well-designed harness (the constrained ecosystem where the agents do their thing). Human experts define what good looks like, set quality thresholds, and build in escalation points. The agents work within those constraints and loop back when they hit uncertainty. As Taylor puts it: “It’s not running completely autonomously unchecked.”4. This Is About Development, Not Just HiringWorkera’s primary focus is post-hire — workforce development, upskilling, and learning. Once an assessment identifies verified gaps in a person’s skills, the platform connects those gaps directly to personalized learning plans, curating from an organization’s existing content library. Two people can get the same score on an assessment and walk away with completely different development paths based on their specific pattern of strengths and gaps.5. Verified Skills Intelligence — What It Actually MeansIn a world where AI can write a perfect resume and LinkedIn profile for anyone, credentials are noise. Verified skills intelligence cuts through that — using assessment to generate actual evidence of what someone can do, fit for the stakes of the decision being made.Final TakeawayThe tools to move beyond multiple choice, beyond static assessments, and beyond slow validation cycles exist today. The bottleneck isn’t technology — it’s the will to trust well-designed systems. When the science is built into the machine from the start, speed and rigor aren’t in conflict. They’re the same thing. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com