Analyzing ABS Challenges With Signal Detection Theory, Pt. 1
Baseball
Signal Detection Theory
Bayesian Inference
MCMC
The first in a series of posts that go beneath the surface of ABS challenge decisions
View the Jupyter Notebook
Visit the GitHub Repo
Introduction
For years, there was an official rule in Major League Baseball that prohibited arguing with the umpire about whether a pitch was a ball or a strike. If a player, manager or coach left their position to argue a call, they could be expelled from the game. But for the 2026 season, Major League Baseball said, “ok, actually, you can argue them a little bit,” and instituted a new challenge system that uses Automated Ball-Strike (ABS) technology.
Baseball fans tend to be a pretty traditional bunch, so this caused some handwringing at the beginning of the year. However, after a full season, fans have mostly accepted the new system and in many cases embraced it. I, for one, welcome our new robot overlords.
For those who are unfamiliar with ABS, the basics are:
- Each team starts the game with two available ABS challenges. If they challenge correctly, they retain that challenge, otherwise they lose it.1
- Only the batter, pitcher or catcher can challenge the umpire’s call. Managers, position players, or other personnel are not allowed to challenge.
- Players initiate the challenge by tapping their cap or helmet. The challenge must be made immediately and without assistance from the bench or other players.2
- After a few seconds, the broadcast and the stadium video board show a graphic animation of the ball’s location as it crossed the plate. Everyone learns the outcome at the same time, which makes for some fantastic entertainment.
It did not take long for the baseball analytics community to start coming up with ways to measure ABS challenges.3 Many ABS analyses measure a player’s success by the percentage of correct challenges they’ve recorded (precision). Others go further by estimating the challenges a player left on the table (recall). The problem with this approach is it confounds two aspects of the challenge decision into one number.
Challenging a call is a judgement based on a physical stimulus. The judgement has two parts: perception and a decision. If two players have different success rates on challenges, is this because they see the strike zone differently, have different risk preferences, or both? Signal Detection Theory (SDT) answers these questions with a single model.
Signal Detection Theory
SDT lives at the intersection of psychophysics and decision theory, studying humans’ ability to differentiate between information (signal) and random activity, which is devoid of information (noise). You can apply SDT to virtually any yes/no task, and it’s the kind of thing you start to recognize everywhere once you’re aware of it. Hiring decisions, dating,4 identifying misinformation, and detecting AI content are all modern applications of SDT.
In an SDT study, subjects make a yes/no choice based on a decision variable. For a baseball player, the decision variable is their perceived location5 of the pitch relative to the strike zone. If the decision variable is sufficiently high, the subject will respond “yes” (they’ll challenge), and will otherwise respond “no” (they won’t challenge).
How much is “sufficient?” That’s defined by the criterion, represented by the variable, \(c\), which tells us how wrong they think the call needs to be in order to challenge it. Players with a low criterion are more likely to challenge a borderline call, while players with a high criterion are likely to challenge only the most egregious calls. The criterion is different for every player, and it partially explains why two players may have different responses to the same call.
In SDT terms, an incorrectly called pitch is called a signal trial, while a correctly called pitch is a noise trial. Challenging a missed call is a true positive, and letting it go is a false negative. Challenging a correct call is a false positive, and letting it go is a true negative. On every taken pitch, a player perceives the ball at different distances inside or outside the strike zone, and makes a judgement call to challenge or not. In aggregate, this forms two random distributions: one for signal pitches (missed calls), and one for noise pitches (correct calls). These distributions can be plotted and compared, like so:
These curves are not the distributions of actual pitch locations. We infer them from the player’s behavior, and they represent the way a player would treat a typical pitch in each category. The blue curve is the range of judged pitch locations for the typical correct call; the orange curve for a typical missed call.
In the diagram, about 36% of the orange signal distribution is below zero. For a hitter, these pitches are outside the zone by definition, but their behavior treats them as though they’re strikes. This might be because they see the pitch as a strike (just like the umpire), or it might be that they don’t think the pitch is worth challenging. The same goes for the blue curve, where roughly 10% is outside the strike zone. This means, given their challenge history, there’s a roughly 10% chance this player will treat a true strike as though it is outside the strike zone. That doesn’t guarantee they’ll challenge that pitch, though - that depends on their criterion, and another value called sigma (\(\sigma\)).
The dashed purple line is the inferred value of the player’s decision criterion. It represents the distance beyond the zone at which there’s a 50% probability the player will challenge.6 The consistency of a player’s behavior is governed by a single parameter: \(\sigma\). A player with a small \(\sigma\) is predictable: pitches beyond their criterion almost always get challenged, and pitches short of it almost never do. A player with a large \(\sigma\) is erratic: they’ll challenge some pitches that were actually strikes and let some obvious misses go.
The width of the curves is also governed by \(\sigma\). The larger \(\sigma\) gets, the more these curves overlap, and the less we’d expect a player to discriminate between correct calls and missed calls when challenging (that’s bad, fyi). A subject with \(\sigma\) near zero would have two sharp spikes that barely touch.
A player’s ability to discriminate signal from noise is known as sensitivity. Sensitivity is measured as the difference between the means of the two distributions, measured in units of \(\sigma\). This measurement is called \(d'\) (pronounced “dee prime”). For \(d'=0\), a player would have no ability to distinguish between a missed call and a correct one, and you’d see two completely overlapping curves.
To avoid sampling noise, each SDT plot in this analysis uses the averages for all correct calls and missed calls seen by hitters and catchers, rather than the pitches each player happened to see. The width of the curves (\(\sigma\)) and the location of the criterion line will vary from player to player based on model estimates. Each plot also shades the 95% credible interval around the criterion, which is wider for players with fewer challenge observations.
Note
In a controlled SDT experiment, participants have to make a yes/no choice every time. But an ABS challenge is optional, and a player who sees a missed call perfectly well might decline to challenge it for any number of reasons. We can’t know why they don’t challenge, only that they don’t. Controlled experiments make it easier to interpret \(\sigma\) and \(d'\) as measures of perception, but in this analysis, we are using them to measure behavior.
In Part 2, we’ll take a closer look at the factors that may or may not cause a player to refrain from a challenge.
The Data
MLB’s Stats API is an embarrassment of riches for sports analytics nerds. Every game comes with an event-by-event feed: every pitch, pickoff, substitution and replay review, each tagged with who was involved and what happened. For pitches and batted ball events, you also get kinematic data describing the velocity and acceleration of the ball in three dimensions. For our model, we need to know three things:
- The true location of the pitch, as measured by the Hawk-Eye camera system in every MLB stadium.
- Whether the pitch was called correctly or incorrectly by the umpire.
- Whether a player challenged a pitch, and the outcome of that challenge.
This ETL script calls MLB’s Stats API, retrieves the schedule, and loops through every pitch of every game, gathering these three data points, along with a lot of other data about the game that we’ll discuss in Part 2.
Once the ETL has run, we have a table with 714,483 rows and 66 columns, where each row is a pitch.
Occasionally, the API fails to record the kinematic data for a pitch. This data is critical for point number one above, but given our large full-season sample, dropping at-bats that contain these errors does not affect the quality of the sample. After dropping 110 affected at-bats (450 pitches), the table contains 714,033 pitches.
The universe of pitches we care about only includes taken pitches (those where the hitter did not swing), as all other pitch events are not challengeable via ABS. We also set aside pitches thrown after the player’s team had used up its challenges, since there’s no longer a challenge decision to be made.7 That leaves 106,721 called strikes a batter could have challenged, and 238,416 called balls a catcher could have challenged.
Pitch Geometry
This analysis treats the ABS measurement as the ground truth,8 but we only get to observe that truth when someone challenges a pitch. For our model, we also need to observe the truth when there was no challenge. This means our data needs to include the location of the pitch in the same place ABS would measure it. Unfortunately, this is not what’s in the data!
The Stats API reports a pitch’s location as it crosses the front of the plate, but ABS judges its location at the midpoint of the plate, 8.5 inches deeper. This means we need to solve some quadratic equations (I’m sorry to inform you your high school teacher was right) to estimate the location of the ball in the ABS plane. Statcast gives us nine parameters - \(x\), \(y\) and \(z\) for position, velocity and acceleration - at \(y_0 = 50\) ft from the plate. The ball’s depth along the pitch’s path at time \(t\) (measured in seconds) is:
\[y(t) = y_0 + v_{y0}\,t + \tfrac{1}{2}a_y t^2\]
Setting \(y(t)\) equal to the plate’s midpoint (\(y_{mid} = 8.5\text{in}/12\text{in} = 0.708\text{ft}\)) and solving the quadratic gives the time the ball crosses it:9
\[t_{mid} = \frac{-v_{y0} - \sqrt{v_{y0}^2 - 2a_y\,(y_0 - y_{mid})}}{a_y}\]
Plugging that back in gives us the ball’s location at the midpoint:
\[x_{mid} = x_0 + v_{x0}\,t_{mid} + \tfrac{1}{2}a_x t_{mid}^2, \qquad z_{mid} = z_0 + v_{z0}\,t_{mid} + \tfrac{1}{2}a_z t_{mid}^2\]
The estimate is pretty good: it matches ABS’s actual ruling on 99.3% of the season’s 10,556 challenged pitches.
Two more details: ABS calls a strike if any part of the ball touches the zone. This means we need to pad the zone by the ball’s radius (0.122 ft, about 1.5 in) on every side, so “center of the ball inside the padded zone” means exactly “some of the ball touched the real zone.” The zone is 17 inches wide for everyone, but its top and bottom are specific to each batter.10
Finally, each pitch gets a signed distance from the strike zone edge, calculated as the Euclidean distance to the nearest edge of the padded zone. For batters, positive means the pitch was outside by that much. For catchers, a positive value means the pitch was in the zone. In both cases, a positive value indicates a challengeable pitch, and \(d = 0\) is the point where the whole ball has just cleared the zone.
To calculate, we let the padded strike zone be the rectangle \([x_{\text{left}}, x_{\text{right}}] \times [z_{\text{bot}}, z_{\text{top}}]\), where the rule-book zone is expanded by the ball’s radius \(r = 0.122\) ft on every side, and let \((p_x, p_z)\) be the pitch location at the plate midpoint.
The horizontal and vertical excursions outside the zone are: \[ \Delta_x = \max\left(x_{\text{left}} - p_x,\; 0,\; p_x - x_{\text{right}}\right), \qquad \Delta_z = \max\left(z_{\text{bot}} - p_z,\; 0,\; p_z - z_{\text{top}}\right), \]
each of which is zero whenever the pitch falls within the zone’s span on that axis. The signed distance to the nearest edge is then:
\[ d = \begin{cases} \sqrt{\Delta_x^2 + \Delta_z^2} & \text{if } \Delta_x > 0 \text{ or } \Delta_z > 0, \\[2ex] -\min\left(p_x - x_{\text{left}},\; x_{\text{right}} - p_x,\; p_z - z_{\text{bot}},\; z_{\text{top}} - p_z\right) & \text{otherwise.} \end{cases} \]
Signed distance is a departure from traditional SDT, which treats the presence of the signal as binary. The underlying assumption is that a high signed distance value is a stronger signal than a low one. In other words, a ball just off the plate is much harder to detect than a ball that sails over a batter’s head, and the model takes that into account.
The Model
The Generative Story
A pitch crosses the plate at a true signed distance \(d\) from the zone edge.11 The model borrows the standard SDT story: the player doesn’t act on \(d\) directly, but on a noisy internal version of it, the true distance plus some random error. We assume that error is normally distributed, with mean zero and variance \(\sigma^2\):
\[X = d + \varepsilon, \quad \varepsilon \sim \mathcal{N}(0, \sigma^2)\]
\(d\) is fixed for a given pitch, but \(\varepsilon\) is redrawn on every trial, which is why the same pitch could produce different decisions from the same player on two occasions. In a laboratory task, \(\varepsilon\) would be pure perceptual error. Here, as discussed above, it absorbs anything that makes a player’s decision vary for a given pitch: an imprecise read, a threshold that shifts with the situation, a reluctance to challenge at all. The \(\sigma\) that sets its spread is the same quantity that governs the overlap of the signal and noise distributions.
The player challenges when \(X\) exceeds their idiosyncratic criterion:
\[X > c\]
We never observe \(X\), only whether the player challenged. So we ask: for a pitch at true distance \(d\), what is the probability of a challenge?
\[P(\text{challenge} \mid d) = P(d + \varepsilon > c) = P(\varepsilon > c - d)\]
Because \(\varepsilon\) is Gaussian, this is a tail probability of a normal distribution, which the normal CDF answers directly. For a \(\mathcal{N}(0, \sigma^2)\) variable, \(P(\varepsilon \le t) = \Phi(t/\sigma)\), where \(\Phi\) is the standard normal CDF and dividing by \(\sigma\) standardizes. Taking the complement:
\[P(\text{challenge} \mid d) = 1 - \Phi\!\left(\frac{c - d}{\sigma}\right)\]
And because the standard normal is symmetric around zero, \(1 - \Phi(u) = \Phi(-u)\). Flipping the sign of the argument gives:
\[P(\text{challenge} \mid d) = \Phi\!\left(\frac{d - c}{\sigma}\right)\]
This is a probit regression, a close cousin of logistic regression. Both turn a linear predictor into a probability; logistic regression uses the logistic curve, while probit uses the standard normal CDF, \(\Phi\), to take a z-score and return a probability.12 We’re already standardizing \(d - c\) by dividing by \(\sigma\), so \(\Phi\) turns it directly into a probability of challenging. The choice of \(\Phi\) isn’t arbitrary: it follows from assuming the error is normally distributed. When \(d\) is well above \(c\), the probability approaches 1; well below, it approaches 0; and at \(d = c\) it is exactly 0.5, which is what the criterion means: the distance at which a player challenges half the time.
The Hierarchy
The model is hierarchical: each player’s \(c\) and \(\sigma\) are themselves drawn from population distributions. \(c_i\) is normal, governed by a population mean \(\mu_c\) and standard deviation \(\tau_c\). Because \(\sigma_i\) must be positive, it is assumed to be log-normal, governed by \(\mu_{\log\sigma}\) and \(\tau_{\log\sigma}\) on the log scale:13
\[c_i \sim \mathcal{N}(\mu_c, \tau_c), \qquad \log \sigma_i \sim \mathcal{N}(\mu_{\log\sigma}, \tau_{\log\sigma})\]
This allows partial pooling: players with sparse observations shrink toward the population, while high-volume players stand on their own data. The amount of pooling is itself learned, through the \(\tau\) parameters.
To avoid sampling problems, the structure is implemented non-centered. We draw standardized offsets \(z \sim \mathcal{N}(0,1)\) and reconstruct each parameter from them:
\[c_i = \mu_c + z_{c,i}\,\tau_c, \qquad \sigma_i = \exp\!\left(\mu_{\log\sigma} + z_{\sigma,i}\,\tau_{\log\sigma}\right)\]
This is algebraically identical, but the sampler explores clean unit-normal space instead of a funnel.
Likelihood
Each pitch is one Bernoulli trial, so for player \(i\) and pitch \(j\):
\[y_{ij} \sim \text{Bernoulli}\!\left(\Phi\!\left(\frac{d_{ij} - c_i}{\sigma_i}\right)\right)\]
Written in Python with PyMC, the model is constructed as:
Show Code
with pm.Model(coords={'player': batter_ids.cat.categories.astype('str'), 'obs': obs_ids}) as hitter_sdt:
#hyperpriors
mu_c = pm.Normal('mu_c', 0, 0.5)
tau_c = pm.HalfNormal('tau_c', 0.5)
mu_logsig = pm.Normal('mu_logsig', -0.7, 0.5)
tau_logsig = pm.HalfNormal('tau_logsig', 0.5)
# player offsets
z_c = pm.Normal('z_c', 0, 1, dims='player')
z_sig = pm.Normal('z_sig', 0, 1, dims='player')
d = pm.Data('d', distance, dims='obs')
c = pm.Deterministic('c', mu_c + z_c * tau_c, dims='player')
sigma = pm.Deterministic('sigma', pm.math.exp(mu_logsig + z_sig * tau_logsig), dims='player')
# Likelihood
eta = ((d - c[batter_idx]) / sigma[batter_idx])
p = pm.math.invprobit(eta)
pm.Bernoulli('y', p=p, observed=challenge, dims='obs')And represented graphically, the model looks like this:
All of the variables in the model, with the exceptions of \(d\) and \(y\), are learned simultaneously through Markov Chain Monte Carlo sampling. In total, four sampling chains generate 1,000 draws each, which gives us posterior distributions for everything.14
The Hitters
The signed distance from the strike zone is different depending on your perspective (positive means outside the zone for hitters, positive means inside the zone for catchers), so the model runs on each group independently. Due up first: hitters.
Criteria
| mean | sd | hdi95_lb | hdi95_ub | |
|---|---|---|---|---|
| c_mean | 4.29 | 0.09 | 4.11 | 4.47 |
| c_sd | 1.03 | 0.06 | 0.92 | 1.14 |
c_mean is the typical hitter’s criterion; c_sd is how much criteria vary from hitter to hitter.
The table above shows the model’s estimate of the distribution \(c\), converted to inches. The mean column is the average across 4,000 posterior draws, sd is the standard deviation across those draws, and the hdi95 columns give the range that contains the parameter with 95% posterior probability.
c_mean and c_sd describe the distribution of \(c\) across all MLB hitters. For hitters, the average criterion is \(c = 4.29\) inches, meaning a typical hitter needs to perceive the ball 4.29” off the plate before they’ll initiate a challenge in 50% of cases. The average missed call, though, is \(d=1.06\) inches off the plate, so this is quite conservative!
The standard deviation of \(c\) across the hitter population is also fairly wide. Roughly 33% of players will have a \(c\) more than one inch above or below the mean. In other words, hitters’ thresholds for challenging a missed call are all over the place, and this is the primary differentiator of hitter challenge behavior. You can see that \(c\) varies widely in the plot below:
Sensitivity
| mean | sd | hdi95_lb | hdi95_ub | |
|---|---|---|---|---|
| d_prime_mean | 1.46 | 0.02 | 1.42 | 1.51 |
| d_prime_sd | 0.16 | 0.02 | 0.12 | 0.20 |
d_prime_mean is the typical hitter’s \(d'\); d_prime_sd is how much \(d'\) varies from hitter to hitter.
By contrast, hitters’ sensitivity barely varies at all. A typical hitter has a \(d' = 1.46\), and hitters vary around that mean by 0.16, or about 11%. The interval for the standard deviation - 0.12 to 0.20 - excludes zero, so we can confidently say they do vary some, but only by a small degree.
So almost all hitters have a similar ability to differentiate between missed calls and correct calls, and the variation we see in challenge behavior is derived almost entirely from the location of a player’s criterion.
Hitter Comparisons
Here are the ten most aggressive hitters, i.e., the players with the lowest \(c\). These are the players who will typically challenge even when the ball only appears a couple inches off the plate.
| player_name | c_mean | c_ci_lb | c_ci_ub | d_prime_mean | d_prime_ci_lb | d_prime_ci_ub | |
|---|---|---|---|---|---|---|---|
| player | |||||||
| 547180 | Bryce Harper | 1.21 | 0.42 | 2.08 | 1.56 | 1.28 | 1.89 |
| 596142 | Gary Sánchez | 1.51 | 0.59 | 2.50 | 1.45 | 1.18 | 1.75 |
| 668901 | Mark Vientos | 1.87 | 0.84 | 3.00 | 1.44 | 1.16 | 1.75 |
| 694384 | Nolan Schanuel | 2.00 | 1.09 | 2.90 | 1.53 | 1.24 | 1.83 |
| 701398 | Sal Stewart | 2.06 | 1.32 | 2.85 | 1.60 | 1.29 | 1.90 |
| 694249 | Cole Carrigg | 2.09 | 1.11 | 3.20 | 1.37 | 1.10 | 1.64 |
| 687282 | Gabriel Rincones Jr. | 2.21 | 0.78 | 3.79 | 1.43 | 1.15 | 1.76 |
| 665923 | Esteury Ruiz | 2.25 | 1.14 | 3.45 | 1.38 | 1.11 | 1.63 |
| 514888 | Jose Altuve | 2.31 | 1.41 | 3.25 | 1.46 | 1.19 | 1.73 |
| 808959 | Munetaka Murakami | 2.47 | 1.53 | 3.40 | 1.54 | 1.25 | 1.85 |
_ci_lb and _ci_ub are the bounds of each 95% credible interval.
Baseball fans may not be surprised to see Bryce Harper’s name at the top of the list. Harper’s \(c\) is 1.21, which is only 0.15 inches more than the average signed distance of a missed call. Not only is he the most aggressive, he’s 20% more aggressive than his nearest neighbor, Gary Sánchez, and twice as aggressive as the tenth hitter, Munetaka Murakami.
Here’s his SDT plot - note where his criterion line falls on the signal curve:
If you set your \(c\) that close to the strike zone, you’d better be ok with missing a few challenges here and there. Harper challenged 47 times in the 2026 regular season - the second most as a hitter after Sal Stewart’s 51 - but succeeded on only 45% of them.
Harper took 288 challengeable called strikes in the 2026 season, and 256 of them were called correctly. From those 256 pitches the model expects about 26 false positives. His actual false positive count was 26, so the model nails it. The model does undersell him on the missed calls, though: it expects about 15 successful challenges, and he had 21.
For contrast, here are the least aggressive players, those with the highest \(c\):
| player_name | c_mean | c_ci_lb | c_ci_ub | d_prime_mean | d_prime_ci_lb | d_prime_ci_ub | |
|---|---|---|---|---|---|---|---|
| player | |||||||
| 650490 | Yandy Díaz | 6.54 | 5.15 | 7.95 | 1.69 | 1.36 | 2.01 |
| 683734 | Andrew Vaughn | 6.25 | 4.84 | 7.77 | 1.66 | 1.33 | 1.98 |
| 596019 | Francisco Lindor | 6.11 | 4.70 | 7.43 | 1.59 | 1.30 | 1.89 |
| 807713 | Matt Shaw | 5.90 | 4.41 | 7.29 | 1.13 | 0.94 | 1.30 |
| 664770 | Nathan Lukes | 5.86 | 4.37 | 7.42 | 1.63 | 1.32 | 1.96 |
| 664040 | Brandon Lowe | 5.79 | 4.50 | 7.11 | 1.60 | 1.30 | 1.92 |
| 686948 | Drake Baldwin | 5.78 | 4.47 | 7.26 | 1.62 | 1.31 | 1.93 |
| 660688 | Keibert Ruiz | 5.71 | 4.21 | 7.26 | 1.60 | 1.30 | 1.94 |
| 694197 | Angel Genao | 5.69 | 4.11 | 7.21 | 1.58 | 1.26 | 1.90 |
| 670541 | Yordan Alvarez | 5.67 | 4.42 | 7.10 | 1.58 | 1.30 | 1.87 |
_ci_lb and _ci_ub are the bounds of each 95% credible interval.
Yandy Díaz is unusually conservative. He took 411 called strikes in the 2026 season. Of those, 52 were miscalled, and should have been balls. Díaz didn’t challenge a single one. As it turns out, for better or worse, Yandy refuses to challenge pitches as a matter of personal policy:
There are some things Diaz doesn’t budge on.
One is that he only hits in the cages rather than take batting practice on the field. Another is that he picks the design for his black face paint each game.
Of more note, he refuses to embrace the new ABS system to challenge calls on pitches, despite urging from coaches.
Given what seems a keen knowledge of the strike zone, Diaz has on several occasions given an umpire a long look, but not, as 421 other hitters have done, tapped his helmet for the computer-assisted chance for correction.
“I don’t like it,” he said. “When I go to home plate, I need a hit not a challenge. … Maybe in the World Series some time, but now (no).”
You can see his criterion is all the way at the edge of the signal distribution and effectively beyond the noise distribution (he has no false positives), with a wide credible interval. Díaz never challenged a single pitch, so his data can only tell us that his criterion is beyond the distance of every missed call he saw, but we can’t know how far beyond. The model makes a guess by borrowing from the population of hitters, which is why his estimate is high but uncertain.
On one hand, he’s extremely accurate: because most called strikes are correct, never challenging means being right about 87% of the time. But his conservative policy carries real costs, which we’ll measure in Part 2.
Nearly all hitters have \(d'\) credible intervals that overlap the population interval, which means we can’t really say that most of them are more or less sensitive than the typical hitter. As we discussed above, they all cluster very closely together. There are a few exceptions, though, and they are the hitters with the lowest \(d'\), aka, the least sensitive hitters:
| player_name | c_mean | c_ci_lb | c_ci_ub | d_prime_mean | d_prime_ci_lb | d_prime_ci_ub | |
|---|---|---|---|---|---|---|---|
| player | |||||||
| 691406 | Junior Caminero | 5.49 | 4.32 | 6.63 | 1.04 | 0.88 | 1.19 |
| 807713 | Matt Shaw | 5.90 | 4.41 | 7.29 | 1.13 | 0.94 | 1.30 |
| 699013 | Esmerlyn Valdez | 4.42 | 3.17 | 5.60 | 1.14 | 0.96 | 1.33 |
| 683748 | Victor Mesa Jr. | 4.04 | 2.72 | 5.37 | 1.19 | 0.99 | 1.41 |
| 669397 | Nick Allen | 5.10 | 3.66 | 6.54 | 1.20 | 1.00 | 1.41 |
| 691718 | Pete Crow-Armstrong | 4.54 | 3.54 | 5.67 | 1.20 | 1.03 | 1.39 |
| 800325 | Luis Lara | 5.07 | 3.69 | 6.40 | 1.21 | 1.01 | 1.41 |
| 575929 | Willson Contreras | 3.33 | 2.28 | 4.45 | 1.28 | 1.05 | 1.50 |
| 655316 | Andruw Monasterio | 5.58 | 4.21 | 6.83 | 1.28 | 1.08 | 1.49 |
| 650489 | Willi Castro | 2.85 | 1.73 | 3.99 | 1.29 | 1.06 | 1.54 |
_ci_lb and _ci_ub are the bounds of each 95% credible interval.
Yandy’s Tampa Bay teammate, Junior Caminero, has the lowest \(d'\) of any hitter. At \(d'=1.04\), he’s one of seven whose sensitivity is detectably below average. His record shows why. He challenged 24 times and won only 9, so most of his challenges were spent on pitches that were actually strikes. Yet he also let 43 genuine missed calls go without a challenge.
That combination is the signature of low sensitivity rather than a misplaced criterion. A player who sets their bar too low would win less often but catch more missed calls; a player who sets it too high would catch fewer but win more of the ones they tried. Caminero does poorly on both at once. His criterion is actually on the conservative side; the problem is that his challenges don’t line up closely with where the pitch was.
That said, 1.04 isn’t zero; he isn’t guessing. If he’d picked 24 pitches to challenge at random, the chance of winning 9 or more is about 0.2%. And it says nothing about how well he sees the ball when he swings. It’s a measure of how closely his challenge decisions track the location of the pitch, and on that measure, he’s at the bottom of the league. He was also 6th in slugging percentage (.524) and 4th in home runs (42),15 so he’s doing just fine.
The Catchers
There are far fewer catchers (111) than hitters (653), but catchers see many more pitches in each game, which means we get many more opportunities to observe their decisions. The hitter dataset contains 106,721 called strikes, but the catcher dataset is more than twice the size, at 238,41616 called balls. As you’ll see, this increases the fidelity of some of our inferences.
Note
What about the pitchers?
In short, there isn’t enough data from pitcher challenges to draw reliable conclusions. Pitchers only challenged 179 of the 256,738 called balls this season. Conventional wisdom across baseball seems to agree that catchers have the more advantageous vantage point for discerning balls and strikes, and some teams even went as far as banning their pitchers from challenging:
…At the start of spring training, Reds manager Terry Francona decided to completely take away the temptation for pitchers to challenge. “We don’t let our pitchers challenge,” he said in late May, “because they think everything’s a strike.”
Like Yandy Díaz, Chris Sale stated he won’t challenge as a matter of personal policy.
Of the 179 pitcher challenges, only 71 (39.7%) were overturned, so it may not be a bad idea to leave it to the catchers after all.
Criteria
| mean | sd | hdi95_lb | hdi95_ub | |
|---|---|---|---|---|
| c_mean | 2.02 | 0.05 | 1.92 | 2.12 |
| c_sd | 0.39 | 0.04 | 0.31 | 0.48 |
c_mean is the typical catcher’s criterion; c_sd is how much criteria vary from catcher to catcher.
As a group, catchers have much more aggressive challenge criteria. An erroneous ball only needs to be 2 inches inside the strike zone for the typical catcher to initiate a challenge. The average mis-called ball was \(d=0.9\) inches over the plate, which is slightly closer than the mean for hitters (1.06). Catchers’ criteria also vary far less than hitters’, with 67% of catchers’ criteria falling within 0.39 inches of the group mean. In the plot below, you can see that the distribution is tighter and closer to the strike zone edge when compared with Figure 3:
Sensitivity
| mean | sd | hdi95_lb | hdi95_ub | |
|---|---|---|---|---|
| d_prime_mean | 4.59 | 0.07 | 4.45 | 4.73 |
| d_prime_sd | 0.44 | 0.06 | 0.32 | 0.58 |
d_prime_mean is the typical catcher’s \(d'\); d_prime_sd is how much \(d'\) varies from catcher to catcher. Not comparable to Table 2.
As with hitters, sensitivity for catchers is based on the signal and noise pitches for all catchers, not just those each catcher saw. Catchers also have an upper bound on their signed distance measurement. A called strike can be infinitely beyond the strike zone for hitters (in theory), but called balls can’t be infinitely inside the strike zone - they can only be directly in the center. This means the signed distance is bounded at about 10 inches17. For these reasons, we cannot directly compare \(d'\) between hitters and catchers. Comparisons within each group are still fair game, though.
Catcher sensitivity varies even less than hitter sensitivity, with two-thirds of catchers falling within 9.6% of the group mean.
Catcher Comparisons
As a group, catchers are much quicker to challenge, but who’s the most aggressive?
| player_name | c_mean | c_ci_lb | c_ci_ub | d_prime_mean | d_prime_ci_lb | d_prime_ci_ub | |
|---|---|---|---|---|---|---|---|
| player | |||||||
| 605170 | Victor Caratini | 1.15 | 0.87 | 1.46 | 5.31 | 4.67 | 6.01 |
| 691019 | Kyle Teel | 1.28 | 0.85 | 1.69 | 4.40 | 3.79 | 4.99 |
| 669221 | Sean Murphy | 1.45 | 1.04 | 1.85 | 4.82 | 4.15 | 5.52 |
| 699024 | Leo Bernal | 1.51 | 0.95 | 2.01 | 4.47 | 3.76 | 5.20 |
| 691788 | Joe Mack | 1.52 | 1.23 | 1.83 | 5.05 | 4.43 | 5.62 |
| 645305 | Ali Sánchez | 1.52 | 1.09 | 2.02 | 4.24 | 3.64 | 4.84 |
| 669134 | Luis Campusano | 1.54 | 1.16 | 1.87 | 4.51 | 3.97 | 5.09 |
| 672275 | Patrick Bailey | 1.54 | 1.23 | 1.84 | 4.72 | 4.17 | 5.21 |
| 691011 | Drew Romo | 1.55 | 1.17 | 1.91 | 4.35 | 3.82 | 4.88 |
| 682663 | Agustín Ramírez | 1.55 | 1.12 | 1.99 | 4.69 | 3.98 | 5.38 |
_ci_lb and _ci_ub are the bounds of each 95% credible interval.
These results are actually a bit surprising. If you asked most baseball fans, they would probably say that William Contreras is the most aggressive challenger behind the plate. During the regular season, he challenged 185 pitches, succeeding on 105 (57%) of them. That’s 49 more challenges than the second most-prolific challenger, Tyler Stephenson. Neither of them makes the top ten, though, and only three from the list above also appear in the top ten by challenges issued.
Victor Caratini has a very low threshold - barely more than the average missed call. In some cases, a willingness to challenge close calls could be dangerous, but Caratini also has the 2nd-highest \(d'\) among catchers. So not only does he challenge pitches very close to the zone edge, he’s also unusually skilled at discriminating signal from noise.
On the other end of the spectrum, these are the most conservative catchers:
| player_name | c_mean | c_ci_lb | c_ci_ub | d_prime_mean | d_prime_ci_lb | d_prime_ci_ub | |
|---|---|---|---|---|---|---|---|
| player | |||||||
| 643376 | Danny Jansen | 2.68 | 2.27 | 3.11 | 5.00 | 4.34 | 5.69 |
| 699625 | Jimmy Crooks | 2.57 | 2.06 | 3.05 | 4.35 | 3.74 | 4.89 |
| 518595 | Travis d'Arnaud | 2.55 | 2.08 | 3.06 | 4.73 | 4.03 | 5.38 |
| 657136 | Connor Wong | 2.54 | 2.16 | 2.93 | 4.56 | 4.03 | 5.13 |
| 694212 | Samuel Basallo | 2.50 | 2.12 | 2.92 | 4.16 | 3.65 | 4.66 |
| 642020 | Chuckie Robinson | 2.50 | 1.81 | 3.18 | 4.96 | 4.06 | 5.91 |
| 668670 | Jake Rogers | 2.40 | 1.98 | 2.84 | 4.49 | 3.94 | 5.08 |
| 663584 | Hayden Senger | 2.39 | 1.76 | 2.98 | 4.65 | 3.94 | 5.45 |
| 663698 | Joey Bart | 2.37 | 1.94 | 2.86 | 4.56 | 3.97 | 5.19 |
| 687221 | Dalton Rushing | 2.37 | 1.94 | 2.80 | 4.62 | 3.98 | 5.22 |
_ci_lb and _ci_ub are the bounds of each 95% credible interval.
Danny Jansen’s \(d'\) is nearly as high as Caratini’s, but he’s much more conservative behind the plate. That difference shows up clearly in their challenge records:18
| Challenges won (precision) | Missed calls challenged (recall) | |
|---|---|---|
| Victor Caratini | 75 of 127 (59%) | 75 of 159 (47%) |
| Danny Jansen | 23 of 32 (72%) | 23 of 139 (17%) |
Judged by win rate alone, Jansen looks like the better challenger: he wins 72% of his challenges to Caratini’s 59%. He isn’t, really. He’s just much more selective when he challenges, preferring a virtually sure thing. Jansen’s higher criterion buys him a better win rate at the cost of challenging only about a third as many of the missed calls he sees (17% vs. 47%). Holding sensitivity equal, the consequence of a more conservative threshold is more opportunities left on the table.
As with the hitters, most catchers have a \(d'\) interval that overlaps the population interval, so it’s hard to say that most are distinctly above or below average. There is one exception, though, and that’s J.T. Realmuto.
| player_name | c_mean | c_ci_lb | c_ci_ub | d_prime_mean | d_prime_ci_lb | d_prime_ci_ub | |
|---|---|---|---|---|---|---|---|
| player | |||||||
| 592663 | J.T. Realmuto | 2.13 | 1.82 | 2.43 | 5.81 | 5.01 | 6.58 |
| 605170 | Victor Caratini | 1.15 | 0.87 | 1.46 | 5.31 | 4.67 | 6.01 |
| 668939 | Adley Rutschman | 1.98 | 1.65 | 2.35 | 5.18 | 4.58 | 5.94 |
| 665804 | Miguel Amaya | 1.58 | 1.23 | 1.92 | 5.15 | 4.42 | 5.83 |
| 696100 | Hunter Goodman | 1.66 | 1.35 | 1.97 | 5.14 | 4.50 | 5.77 |
| 680728 | Adrian Del Castillo | 2.35 | 1.84 | 2.88 | 5.06 | 4.22 | 5.93 |
| 691788 | Joe Mack | 1.52 | 1.23 | 1.83 | 5.05 | 4.43 | 5.62 |
| 608348 | Carson Kelly | 2.08 | 1.75 | 2.40 | 5.00 | 4.42 | 5.61 |
| 643376 | Danny Jansen | 2.68 | 2.27 | 3.11 | 5.00 | 4.34 | 5.69 |
| 642020 | Chuckie Robinson | 2.50 | 1.81 | 3.18 | 4.96 | 4.06 | 5.91 |
_ci_lb and _ci_ub are the bounds of each 95% credible interval.
J.T. Realmuto does the best job of discriminating signal from noise on challenges. During the season, he challenged 86 times and lost only 16. However, his moderate threshold caused him to leave 191 challenge opportunities on the table, second only to Tyler Stephenson’s 192. Just as we saw with Jansen, his threshold trades missed opportunities for a better win rate. He wins 81% of his challenges, the best of the three, but challenges only about a quarter of the missed calls he sees.
| Challenges won (precision) | Missed calls challenged (recall) | |
|---|---|---|
| Victor Caratini | 75 of 127 (59%) | 75 of 159 (47%) |
| Danny Jansen | 23 of 32 (72%) | 23 of 139 (17%) |
| J.T. Realmuto | 70 of 86 (81%) | 70 of 283 (25%) |
Conclusion
To recap:
- Hitters’ challenge thresholds are all over the place. The typical hitter wants a pitch about 4.3 inches off the plate before challenging, but that bar swings by about an inch from one hitter to the next, from Bryce Harper’s hair trigger to Yandy Díaz’s personal boycott.
- Hitters’ sensitivity barely varies. Whatever mix of eyesight and judgment goes into a challenge decision, nearly every hitter tracks pitch location about as closely as the next. The bar is what separates them.
- Catchers are more aggressive, and more alike. The typical catcher challenges once a ball looks about 2 inches inside the zone, with little variation from one catcher to the next.
- Catchers’ sensitivity varies even less than hitters’.
- There’s a tradeoff between precision and recall. Aggressive challengers with lower \(c\) values will have lower precision, but they will also leave fewer challenge opportunities on the table.
So, which is better - high precision, or high recall? The answer depends on the cost of a missed challenge opportunity compared with the benefit of a successful challenge. Throughout this post, we’ve been ignoring an elephant in the room: game context. In a baseball game, those costs and benefits change from inning to inning, plate appearance to plate appearance.
In Part 2, we’ll look at the way players account for game context in their challenge decisions, or in some cases, how they don’t. Not only will we account for the expected run value of their challenge decisions, we’ll measure the value of the challenge itself as a call option on a future pitch. Then we can compare those two values for any given challenge decision and understand whether the player created or destroyed value with their decision, independent from the actual outcome. All this and more, coming soon to a screen near you!
Footnotes
In extra innings, any team that begins the inning with zero challenges will get one (1) extra challenge. If they exhaust that challenge, they get another one in the next extra inning. The ceiling is two (if you didn’t use them during the first nine innings), and the floor is one.↩︎
The definition of “assistance” is something of a subjective grey area. Controversially, some umps have denied some challenges by claiming the player took too long or received assistance from the bench.↩︎
MLB’s Baseball Savant site has a particularly good dashboard.↩︎
I couldn’t find examples involving dating apps specifically, which seems like a ripe area for research. If you work at Hinge, call me.↩︎
This will almost always be different from the actual location of the pitch!↩︎
In traditional SDT, \(c\) is measured as the distance between the midpoint of the distributions (roughly -1.4 inches in the figure) and the decision boundary. In this analysis, we’ll measure \(c\)’s absolute location relative to the strike zone.↩︎
This removes about 7% of the called pitches, and it matches how Baseball Savant counts challenge opportunities. Leaving those pitches in would count each one as a decision not to challenge, pushing every player’s estimated criterion upward.↩︎
Some may take issue with this but I say it’s close enough.↩︎
\(v_{y0}\) is negative because the ball is traveling toward the plate, and \(a_y\) is positive because drag slows it down. The quadratic has two positive roots. The smaller one is the actual crossing; the other is where the fitted parabola would come back if the ball kept going.↩︎
MLB had to measure each player to obtain their strike zone measurements, with hilarious consequences.↩︎
Note that \(d\) is not the same as \(d'\), which I appreciate is confusing.↩︎
The probit function itself goes the other way, from probability to z-score.↩︎
Priors are weakly informative and expressed in physical units, centered on neutral values (a bar at the zone edge, about 6 inches of decision noise). The posteriors are dozens of times narrower than the priors, so the estimates are driven by the data.↩︎
For post-sampling model diagnostics, please see the notebook on GitHub.↩︎
Per Junior’s Baseball Reference page.↩︎
This number excludes called balls that occurred after a team had exhausted their challenges. In total, there were 256,738 called balls in the 2026 season.↩︎
A pitch dead center is about 10 inches from the nearest edge of the padded zone: half the plate’s 17-inch width, plus the ball’s 1.5-inch radius. The zone’s height varies by hitter, but even the shortest zone in 2026 was 17.3 inches tall, so it’s taller than it is wide for everyone, and the side edges always set the limit. In practice, the deepest called ball of 2026 was 6.3 inches inside the zone.↩︎
The column headers borrow two terms from machine learning, where they’re used to evaluate binary classifiers. Precision is the share of a classifier’s “yes” answers that turn out to be right (here, challenges won). Recall is the share of true cases it catches (here, missed calls challenged).↩︎