Another Danged Catcher Defense Study

[RonCo]

Taking the conversation with Tyler to heart, I went out looking to confirm the existence of impact of catcher defense, and hopefully get a flavor of the size of it. What I found was interesting, so I’m going to do this here. Since this is a little big, though, I’ll do it in a total of four different posts—this one, one on the process, one on the running game, and another on “Framing.”

The TLDR version of the results is:

• The answer to the age-old question of the value of an arm in the running game (is it better to stop guys from running or let them run?) still seems to be dependent on how successful the runners are, not how often they run.
• I confirm again Tyler’s assertion of an impact for “framing” or “game calling” or whatever you want to attribute to “Ability.”
• Due to the controlled set-up I used I can’t directly map the quantitative value I found into the BBA, but it’s clear that catcher defense has value both in his arm and in his ability to manage/call/frame pitchers. Bottom line: this info is about OOTP, not the BBA. But at the end of the day the game is, again, directionally done well, so this suggests BBA GMs should do their best to make judgements about how this translates to our world.

With that, let’s get going…

(this may take me all day to post, so if I go quiet a bit, I’ll be back)

[RonCo]

THE PROCESS

To do this, I set up a weird little test league of 18 teams all in the same division. I froze the rosters of the 18 teams, and set the game to make me in control of all rosters so the AI wouldn’t change things around. I turned off development, and injuries, and all other things that seemed like it might possibly alter the players. At this point, I set all hitters and pitchers to the same ratings and potentials:

• Pitcher STU, MOV, CON, Hold = 120
• Pitcher VEL = 96 MPH
• Pitcher GB = 54%
• Hitter Ratings = 120
• Runner Speed, Steal, Baserunning = 140

I then set catcher arm and ability ratings using 100, 140, 180 (5/7/9), with two teams of each pair (i.e. Team 1 = 100/100, Team 2 = 100/110, Team 3 = 140/100, Team 4 = 140/100, etc.). I then ran a 170 game schedule so every team could play every other team 10 times, and tabulated results.

NOTE: In tabulating run values, I used a table created by Tom Tango. These numbers are different than others I’ve used—which came from Bill James. I think Tango’s methods are stronger…your mileage may vary. For example, however, a SB becomes worth .16 runs and a CS becomes worth .45 the opposite direction. This changes some of our earlier conversations in the direction to make SB less valuable overall. (I've always used .34/.66 in the past).

[RonCo]

WHAT I FOUND: THE RUNNING GAME

To start, I need to say that given the ratings set-up, this turned out to be a highly unsuccessful environment—meaning the SB success rate was only 62%. I think this made for a fun and interesting finding, that being that teams that had great arms behind the plate lost runs (and therefor wins) because runners ran less often. This is why I included the third bullet in the TLDR list.

First, Ability.

Bottom line, the Catcher Ability Rating did not seem to impact the outcomes of steals. Here is the performance of catchers split by Ability:

• 100: 1229 SBA, 61.4% SB Success (66.7 Runs saved *)
• 140: 1307 SBA, 60.1% SB Success (78.1 Runs Saved)
• 180: 1276 SBA, 63% SB Success (55.3 Runs Saved)

(*) There are six teams with each rating, so you have to divide each Runs Saved by six to get a team-specific number.

So runners were actually more successful against catchers with great ability. Beyond that, there isn’t any real pattern to the data that I can see. It just kind of jumps all over.

What about Catcher Arm?

As expected, Catcher Arm shows a clear value in defending the running game. Runners run less often and with less success at each level of catcher’s arm rating.

• 100: 1,484 SBA, 64.1% Success
• 140: 1,341 SBA, 61.6% Success
• 180: 987 SBA, 58.8% success

But here’s where it gets interesting. If you quantify this data into Runs Saved, you get the following:

• 100: 64.4 Runs Saved
• 140: 71.9 Runs Saved
• 180: 63.8 Runs Saved

On the whole this data says that, in an environment that is not highly successful, a strong armed catcher hurts the team by preventing SB attempts. in this world, you’d rather have a “5” Arm than a “9” Arm—though it’s close. Ultimately, you’d rather have a “7” arm. My expectation is that the answer for how valuable catcher arm is in practice is a calculation that changes both by league and by league-year. (In the BBA this year, with SB% success in the 73% zone, Catcher Arm is probably paying off well). Our job is to figure out where we are in the BBA--an answer we may not all agree on, which continues to be part of the fun of the game I suppose.

Here’s a set of matrices to show this performance.

CATCH-SBA-STUDY.PNG (12.38 KiB) Viewed 2527 times

[RonCo]

FINALLY! LET’S GET TO FRAMING!

The bottom line is that I can confirm Tyler’s finding that catchers’ defensive ability affects pitcher performance. This makes me feel good. It’s the first time I’ve really attempted to quantify framing or game calling, but I’ve always assumed it’s been there because Markus says it is, but I admit I like seeing the data bear it out. You just never know, you know?

Here’s the kicker, though: while I find an impact on pitcher performance (more Ks, fewer walks, etc.) when I abstract out to actual wins for the teams in this environment, that affect seems to go away. Maybe I just need more sample size, or whatever.

Going straight to the tabulations rather than displaying the raw data tables, here’s the results:

CATCH-ABI-PITCH-RESULTS.PNG (3.81 KiB) Viewed 2523 times

You can see the steady improvement of pitcher results in the data here. One caveat, I didn’t normalize defense, so there’s a possible source of systemic error in the “Hits/9” category. But, if you buy this data, a pitcher in this environment who threw to a catcher with a 180 Ability struck out .08 batters more per 9 innings than one who threw to a catcher with a 100 Ability (7.38-7.26).

To get a better idea of what this meant in runs (again in this environment), I went to Tango’s tables and grabbed run values.

I’ll put these on the next rock.

[RonCo]

So, what does this stuff mean? Is it saying I should dump my current guy for anyone with a 9 or better rating? Well. Let’s look at run values (recalling that we’re using Tango values, which may or may not be good representatives to the BBA—but they are what we’ve got, so we’ll use them).

From the data above, we can calculate the following chart—which is a tabulation of the difference between pitchers results on teams that had 100 Ability catchers and teams that had 140 Ability catchers.

Catch-100-vs-140.PNG (5.31 KiB) Viewed 2516 times

In this data, the “Perf Gap” is the difference in pitching results, “Run Value” is the figure from Tango’s work, “Runs” is the product of the performance gap times it’s weight, and Wins is a quickie Runs/10.

Given this environment, the value of a 140 defensive catcher over a 100 catcher could be called about .9 wins. Compare this to any gap in offensive capability ( and his run prevention in the running game), and you’ve got a view of overall value.

I hear you ... what about that 180 value catcher? Hold your horses...it's coming...

[RonCo]

Here’s the same data for that total “crap 100” catcher vs. an “elite 9.” I’ll once again warn about taking data straight from this vanilla environment full of clones hitters and pitchers and using it in ours. But, again, the direction feels good.

Catch-180-vs-100.PNG (5.64 KiB) Viewed 2511 times

That's a nice number right there. 3.2 Wins.

And while I 'll again warn against direct conversion...among the thing I note most here is the relative gaps between the performances. Regardless of what the actual numbers are league over league, the idea in this data is that the step from a “5” to a “7” Ability is an improvement in 1 Unit (we’ll call it a Unit rather than a Win), and the step from a “7” to a “9” is two Units more. My pure guess is that this ratio is probably something useful to think about not just in C Ability, but in most ratings.

Anyway…there you have it.

I may have some more thoughts shortly, but now I’ve got to go to the grocery. [grin]

[Spiccoli]

Interesting.... especially about the arm ratings. It almost seems like it’s better to have an average catcher arm and let runners get themselves out.

It’s one of the reasons I fired the previous manger. I couldn’t dial back his tendency to send runners enough and we had way too many CS’s from runners who shouldnt be trying to steal.

The current manager seems more reasonable in his decision making. Maybe it’s the new version

[RonCo]

I always love thinking about the parameters of the running game because the inputs are so complex. Pitcher hold was held constant in this study, but it plays in there, too. To really "know anything" you need to do a lot more work and probably understand more about situational values.

But I admit I find it fascinating to try to pick out what point in the success curve that this inverse relation happens where you're actually better off with a bad arm behind the plate. It's one of those things where the illogical can be true. Realize, though, that with the success curve up where we are now, it's likely that those big arms are helping.

[RonCo]

AS A KICKER, HERE's WINS...

I went to the standings and looked to see how these values translated into real wins, I found some data to suggest that the values found here sorted into the actual win column--but the data is noisy in ways, so you make your own call. But here's the data:

CATCH-WINS.PNG (7.75 KiB) Viewed 2489 times

Note that along the ARM scale, those with 140 ("7") arm catchers won the most, those with 180 the least, and those with 100 just a bit more than those with elite arms--which is what you would expect if the numbers for controlling the running game had any merit. Oddly, though, the ABILITY scale is kind of wonky, with teams in the middle winning fewer games than the others, suggesting any framing help they got was overridden by some other impact.

So, there's no free lunch here. No guarantees.

Finally if I get time, I'd love to do the same thing and alter various things like Change "Stealing Bases" setting in STATS & AI to see if I can adjust the steal rate. Or alter Pitcher Hold and Runner Speed/Steal to see how and if they change the value of catcher's ratings (it's a big ol' system, so I assume they do), Pitcher ratings, Batter ratings, etc, to see how they affect that framing or game calling aspect of the results.

[agrudez]

Interesting.... especially about the arm ratings. It almost seems like it’s better to have an average catcher arm and let runners get themselves out.

Ron did this on a test league, NOT the MBBA. Look at the results he posted:

•100: 1,484 SBA, 64.1% Success
•140: 1,341 SBA, 61.6% Success
•180: 987 SBA, 58.8% success

Yes, in that league's environment, you want guys running as often as possible because they are failing at a very counterproductive rate - so toss out a guy with a subpar arm and laugh at the extra outs. In the MBBA, though, as of this writing, those success rates are much, much higher and so that wouldn't be the case.

[RonCo]

In the MBBA, though, as of this writing, those success rates are much, much higher and so that wouldn't be the case.

Yes. All this.

But note that -until this year- the success rate was always 65% (give or take a little). So at those rates it was still a questionable issue. And if those rates go back into play, that situation occurs again. At question is whether 2035 is an outlier or a new standard.

[RonCo]

In addition, note this study included WP reduced, but did not include PB (or base defense against bunts--which is another post I want to make soon). All these things combine to make a difference that is probably (or at least could be) much greater on average than the catcher's involvement in the running game.

[udlb58]

That it pretty much what I found in my research (I didn't break it down into runs, but the K/9 and BB/9 ratings looked similar). I did feel like ability had a slight improvement in CS%, but maybe that was just noise in my data.

As for the run prevention of a 140 vs 180 arm; that may be the case over a full season, but my line of thinking (right or wrong) was that the sample size would be too small in a playoff series to see the number even out, so I wanted to prevent as many free bases as possible. Not only did I have an 11 (220?) arm catcher (with 13 defense) but my starting pitchers the last year I won the title were 9/8/7/7 at holding runners. In fact, since the running revolution started in 2030, I've only used 8+ arm catchers and have only had two SPs with a hold runner below 6 (Totten, who started due to injury; and Howard, who only played one post-season). Has it worked? I don't really know.

[agrudez]

Has it worked? I don't really know.

Largely anecdotal since it's only comparing 2 teams over 5/6ths of a season, but...

HSV:
Average catcher arm/9 = 8.31
Average SP hold/game = 8.4
Total SBA/9 = 1.226
Aggregate CS% = .279
Total successful SB/9 = 0.884
Total failed SB/9 = .342

JAC:
Average catcher arm/9 = 7.84
Average SP hold/game = 7
Total SBA/9 = 1.41
Aggregate CS% = .266
Total successful SB/9 = 1.038
Total failed SB/9 = 0.376

So, that's a net increase of ~1.5 SP hold/game and ~0.5 C arm/game for HSV. It resulted in 1 extra out every 29.4 games for JAC and 1 extra successful SB against every 6.485 games for JAC.

The gain in failed SB/game ends up pretty well negligible (that's ~5.5 extra outs per season) since teams run considerably less often on HSV (1 extra SBA every 5.43 games for JAC). The 1 extra SB essentially per schedule week seems tangible, though, so let's explore it a bit. Using the following table (note: I'm not positive the vetting of this, but I wasn't trying to do an extensive paper study and it was one of the first hits on a google search): https://www.nssl.noaa.gov/users/brooks/ ... druns.html, we can try to extrapolate how many runs you can expect to give up from that extra SB/week (an extra 25 SBs/season). Namely, I'm interested in the "Prob >0" column - and I'm going to be assuming (for simplicity's sake) that all of the gained SBs are of the 1st to 2nd variety (without checking, I'm assuming that accounts for a vast majority of successful SBs).

Delta in probability of runner scoring between 1st no outs and 2nd no outs: 0.198
Delta in probability of runner scoring between 1st 1 out and 2nd 1 out: 0.138
Delta in probability of runner scoring between 1st 2 outs and 2nd 2 outs: 0.096

Since (again, not checking) I believe that there isn't some overarching trend of when a runner goes (ie. is he more likely to go with 2 outs than 0 outs or vice versa?) I'm going to distribute those extra 25 SBs/season evenly over the situations. So... 8.33 SBs/season with 0 outs, 1 out and 2 outs, respectively. That would then equate to 1.65+1.15+.8=3.6 runs/season

Yes, that's right. You give up 3.6 runs/season more than me because of your inferior battalion defense of SBs. In other words (assuming a 9*162=1458 inning season), a 0.0025 increase in your ERA. Muhaha, the Southeast is mine! /s

[RonCo]

Here are the SB totals against HNT/JAX by base. EDIT >> Note that you can ignore the "count" breakdown. Those aren't meaningful in this chart. And I know I'm missing a few steals in this data due to game log formats shifting in mid-season. Still, they should be pretty close.

[RonCo]

Kyle's use of run expectancy charts has me thinking I need to add "outs" to my script. That would be interesting. Again, it assumes that the BBA run scoring environment is like the MLB, which is probably right directionally if nothing else.

[RonCo]

Actually...Kyle's not using a run expectancy chart. He's using a run probability chart. Which is different. I'm fiddling with numbers using Tango's run expectancy chart for modern baseball, and they are interesting enough that I have to go back and run the numbers again to see if I am doing something wrong.

[RonCo]

When I use Tango’s actual run expectancy charts, and assume all steals of second, third, and home happen in isolation (with no one else on base), and then assume that all SB happen across all out situations equally, I get the following values for SB and CS:

SB 2B: .164
SB 3B: .190
SB Home: 1.278

CS 2B: -.413
CS 3B: -.577
CS Home: -.767

When I weight actual performance in the BBA with these values I get the following chart for stolen base defense:

Notes:

If this is true…

- Running has cost the league about 950 runs throughout the season.
- Every team is preventing more runs than they are allowing
- Rockville’s leading the league at -49.3 Runs, Yellow Springs is second at 45.2
- Huntsville has allowed 4.6 fewer runs than Jacksonville, but Jacksonville has prevented 4.4 more than Huntsville. They are effectively equal.

Some possible errors here:

- The BBA run expectancy is not the MLB run expectancy
- The simplification of out or on-base situation is screwing with the numbers
- And, yes, I'm still missing a few SBA, but not enough to fundamentally change the findings.