Why I Hate Overall Ratings, A Thread

Beat articles, power rankings, statistical analysis, etc. goes here.
User avatar
RonCo
GB: JL Frontier Division Director
Posts: 22147
Joined: Sat Nov 14, 2015 10:48 pm
Has thanked: 2437 times
Been thanked: 3990 times

Why I Hate Overall Ratings, A Thread

Post by RonCo » Fri Nov 13, 2020 11:24 am

(Or, How I Stopped Giving a Crap and Learned to Love the Component)


If you’ve listened to any of the various podcasts I’ve done, you have, by now heard me say something in the range of “I hate overall ratings” enough times. Still, there’s always room for one more, I figured, so what the hell, right?

First, a little process junk: Last night, for giggles, I grabbed a view in OOTP and exported out all of our ratings (1-10 scale, plus a bunch of other stuff—specifically including overalls. Then, after deciding to focus on pitchers for this one, and after stripping out potential info, I was left with every pitcher in the league. This I cut into a couple different segments—specifically “all SP” and “all relievers with 10 Stuff” (note the “10 Stuff” was a random thing). At that point, I then sorted each into ratings blocks by Stu/MOV/CON pairings—then these parings were sorted by OVR.

In other words, I ended up with the ability to put equally rated STU/MOV/CON pitchers together and see how their OVR varied.

I’ll add the two splits as a stand-alone spreadsheet at the end of this rock so you can play with the process yourself. And most likely I’ll be posting follow on cuts in different ways over the next few days.

That said, I’ll do a few cuts here.

For example:


Let’s Look at some 10/5/7 Relievers

I chose this block for a could reasons. First, there are no YS9 pitchers in here, so I can look at them “objectively.” In addition, it was the first grouping of pitchers that had enough sample size that it spanned FOUR rating points. (There are pitchers in this segment who score 35, 40, 45, and 50 in OVERALL Rating)

Here are the pitchers:

OVR-RP-10-5-7.PNG

Can you see why I hate Overall Ratings here? I mean…

Compare Aurelio Castillo (35) with Mitch Freeman (50), and you see two power pitchers with three pitches that are all pretty danged good. The only difference is that Freeman throws two MPH faster, and Castillo is an over the top fly ball pitcher. I note, though, that Castillo controls the running game at an elite level, while the barn door is always open for Freeman. Is the two MPH difference (from 99-101?) worth 15 overall points? Is it the FB thing? If so, other than Castillo’s very brief stint in the bigs, there’s nothing in his numbers to say his “5” movement is any worse than Freeman’s.

And, if you say, yeah, two MPH is worth 15 rating points in OVERALL, well, whot do you make when you compare Castillo to Portland’s Faical Engeitado? Engitaido tops out at 93 MPH and his repertoire would seem to pale in comparison to Castillo’s. Engeitado is a lefty sidearm thrower with good splits. Perhaps that’s bump-worthy, but three notches? 15 points?

Play the same game with Charlotte’s Robert Hobbs—who blazes at 101 MPH and has a four-pitch arsenal.

Bottom line: if all these guys were within a rating of each other, I could see it. But they get spread over four total ratings…which makes me scratch my head.

Perhaps it’s more interesting to look at the common elements here—the three guys in the list who are all rated 45s? I’ll do that later, too—or even go further and collect up all 45s (or whatever) and see what that pile looks like in total. Or you can. Feel free…


Now Let’s Look at a Couple Starters

Personally, I find looking at starters to be more interesting by comparing across different rating segments, and I’ll probably do that a little later, but while we’re on this process, let’s look at an interesting pair.

OVR-SP-2.PNG

Madison’s 25-yo Yorikane Miyamoto (55) matches with New Orleans’ Cristian Garcia (65). If you looked at their ERAs you’d probably say that makes sense—but compare FIPs and WARs and you get a different feel. Until this past season, when Miyamoto seemed to implode, the two were quite similar.

Which makes sense when I scan the components.

I mean…both are young, both throw pretty hard and have better than 3-pitch repertoires. Both deal with the running game. Both go medium deep into games. Does the velocity bump account for the overall rating delta? I dunno, but I don’t think it should. I mean, looking at these two pitcher’s I’d value them about evenly all other things considered. Put them behind the same defenses and in the same ballparks and I’d expect a similar performance. But the game says one is 55 and the other 65. Am I the crazy one? Possibly.

Anyway, my morning time is up, so I’ll stop here…but more to come … maybe. (grin)
Attachments
2044-SP-RP-Real-OVR-Rating-Study.xlsx
(203.5 KiB) Downloaded 126 times
GM: Bikini Krill
Nothing Matters But the Pacific Pennant
Roster

bcslouck
Ex-GM
Posts: 3252
Joined: Sun Oct 02, 2016 10:09 am
Location: Millersville, MD
Has thanked: 394 times
Been thanked: 322 times

Re: Why I Hate Overall Ratings, A Thread

Post by bcslouck » Tue Nov 17, 2020 9:25 am

You know, thinking about this a coupe days after checking this out, I may be more on board with getting rid of them. I think a lot of people, including myself, tend to hang on to players because of overalls instead of looking at the ratings of the player. Or seeing if they actually fit what you need. I think it would take a little bit, but you may see the trade market open up more if you got rid of them of the overall potential and current. Keep all other ratings as is.
Brandon Slouck
Rocky Mountain Oysters (2058 - present)
Cairo Pharaohs (2057)
Charm City Jimmies (2029 - 2049)
Paris Patriots (2028)

Ted
Ex-GM
Posts: 5630
Joined: Thu Jun 04, 2015 2:50 pm
Has thanked: 368 times
Been thanked: 378 times

Re: Why I Hate Overall Ratings, A Thread

Post by Ted » Tue Nov 17, 2020 9:38 am

bcslouck wrote:
Tue Nov 17, 2020 9:25 am
You know, thinking about this a coupe days after checking this out, I may be more on board with getting rid of them. I think a lot of people, including myself, tend to hang on to players because of overalls instead of looking at the ratings of the player. Or seeing if they actually fit what you need. I think it would take a little bit, but you may see the trade market open up more if you got rid of them of the overall potential and current. Keep all other ratings as is.
I see your point. My concern is that this is an added degree of difficulty for less experienced people. Less importantly, I personally like to have more data. Even if the overall ratings doesn't make sense, it's telling you something. I've been able to figure some things out about players before because of it that I wouldn't have been able to without it.
Ted Schmidt
Twin Cities Typing Nightmares(2044-present)
California Crusaders (2021-2038)
Image

User avatar
aaronweiner
BBA GM
Posts: 12398
Joined: Wed Jun 16, 2010 1:56 pm
Has thanked: 89 times
Been thanked: 935 times

Re: Why I Hate Overall Ratings, A Thread

Post by aaronweiner » Tue Nov 17, 2020 10:49 am

You didn't figure out that Garcia was a 65 because he's a lefty?

And I'd clearly rate Neumann above the others.

User avatar
Lane
GB: Vice Commissioner
Posts: 7500
Joined: Tue Feb 02, 2016 8:18 am
Location: Los Angeles
Has thanked: 665 times
Been thanked: 954 times

Re: Why I Hate Overall Ratings, A Thread

Post by Lane » Tue Nov 17, 2020 11:06 am

I don't think they're perfect, but I really appreciate having them as a quick point of reference.
Stephen Lane
Vice Commissioner / Historian
General Manager, Long Beach Surfers
Since 2026

Image


Ex-GM, Amsterdam Neptunes, 2025 EBA Champions

Link Generator Unified History Donate

User avatar
RonCo
GB: JL Frontier Division Director
Posts: 22147
Joined: Sat Nov 14, 2015 10:48 pm
Has thanked: 2437 times
Been thanked: 3990 times

Re: Why I Hate Overall Ratings, A Thread

Post by RonCo » Tue Nov 17, 2020 11:07 am

Yeah, I don't think they need to be nixed, but they are quite wonky to my eyes in many cases.
GM: Bikini Krill
Nothing Matters But the Pacific Pennant
Roster

bcslouck
Ex-GM
Posts: 3252
Joined: Sun Oct 02, 2016 10:09 am
Location: Millersville, MD
Has thanked: 394 times
Been thanked: 322 times

Re: Why I Hate Overall Ratings, A Thread

Post by bcslouck » Tue Nov 17, 2020 11:40 am

Ted wrote:
Tue Nov 17, 2020 9:38 am
I see your point. My concern is that this is an added degree of difficulty for less experienced people. Less importantly, I personally like to have more data. Even if the overall ratings doesn't make sense, it's telling you something. I've been able to figure some things out about players before because of it that I wouldn't have been able to without it.
I feel like having actual ratings with potentials is plenty enough. I feel like fog of war is something most people want, but the BBA is about as far from it as you can be.
Brandon Slouck
Rocky Mountain Oysters (2058 - present)
Cairo Pharaohs (2057)
Charm City Jimmies (2029 - 2049)
Paris Patriots (2028)

User avatar
niles08
BBA GM
Posts: 2541
Joined: Wed Mar 29, 2017 9:15 pm
Has thanked: 171 times
Been thanked: 471 times

Re: Why I Hate Overall Ratings, A Thread

Post by niles08 » Tue Nov 17, 2020 11:55 am

I think I have been pretty vocal about getting rid of overall/potential ratings for some time. Often enough they don't make tons of sense as Ron showcases above. I will be the first to admit even when I argue we should get rid of them, I have found myself hanging onto players that I shouldn't have because of their "overall rating". I would even be in favor of half removing these and just leaving the all mighty "potential" rating and hiding the overall. That way it's not total darkness on this number which doesn't really mean much anyhow.
Image

Ted
Ex-GM
Posts: 5630
Joined: Thu Jun 04, 2015 2:50 pm
Has thanked: 368 times
Been thanked: 378 times

Re: Why I Hate Overall Ratings, A Thread

Post by Ted » Tue Nov 17, 2020 12:17 pm

aaronweiner wrote:
Tue Nov 17, 2020 10:49 am
You didn't figure out that Garcia was a 65 because he's a lefty?

And I'd clearly rate Neumann above the others.
I'm pretty sure you're messing around, but for the sake of the less informed, handedness does not impact overall/potential ratings.
Ted Schmidt
Twin Cities Typing Nightmares(2044-present)
California Crusaders (2021-2038)
Image

User avatar
RonCo
GB: JL Frontier Division Director
Posts: 22147
Joined: Sat Nov 14, 2015 10:48 pm
Has thanked: 2437 times
Been thanked: 3990 times

Re: Why I Hate Overall Ratings, A Thread

Post by RonCo » Tue Nov 17, 2020 12:19 pm

Ted wrote:
Tue Nov 17, 2020 12:17 pm
aaronweiner wrote:
Tue Nov 17, 2020 10:49 am
You didn't figure out that Garcia was a 65 because he's a lefty?

And I'd clearly rate Neumann above the others.
I'm pretty sure you're messing around, but for the sake of the less informed, handedness does not impact overall/potential ratings.
Technically, this is true in that you can change handedness and it won't affect the overall ratings. But handedness does come into actual splits, which the game engine uses (and I assume the overall rating system uses). Splits are created upon player creation, and will also not change if you alter player handedness.
GM: Bikini Krill
Nothing Matters But the Pacific Pennant
Roster

Ted
Ex-GM
Posts: 5630
Joined: Thu Jun 04, 2015 2:50 pm
Has thanked: 368 times
Been thanked: 378 times

Re: Why I Hate Overall Ratings, A Thread

Post by Ted » Tue Nov 17, 2020 12:21 pm

Ted wrote:
Tue Nov 17, 2020 12:17 pm
aaronweiner wrote:
Tue Nov 17, 2020 10:49 am
You didn't figure out that Garcia was a 65 because he's a lefty?

And I'd clearly rate Neumann above the others.
I'm pretty sure you're messing around, but for the sake of the less informed, handedness does not impact overall/potential ratings.
Hmm, except in one possible way. Our league is set so that if overall performance exceeds potential, it will diplay overall for both overall and potential. For example, if say a developed 30 ovr/pot player puts up the stats the game would expect from an 80 ovr/POT player, it will show 80 OVR 80 POT. (The other option is to have it say 80/30. We do not have it set up that way).

So, in theory, if handedness gave a player a huge advantage and that player massively exceeded the games ovr/pot expectation, you could see handedness "affecting" the displayed ovr/pot. But it's not affecting the internal ratings, just what we see. This could be contributing to Ron's confusion in the initial post.
Ted Schmidt
Twin Cities Typing Nightmares(2044-present)
California Crusaders (2021-2038)
Image

User avatar
RonCo
GB: JL Frontier Division Director
Posts: 22147
Joined: Sat Nov 14, 2015 10:48 pm
Has thanked: 2437 times
Been thanked: 3990 times

Re: Why I Hate Overall Ratings, A Thread

Post by RonCo » Tue Nov 17, 2020 1:14 pm

Yea, that is true. If we alter the weightings to 100% ratings, rather than being influenced by performance than I suppose things would be clearer.
GM: Bikini Krill
Nothing Matters But the Pacific Pennant
Roster

User avatar
RonCo
GB: JL Frontier Division Director
Posts: 22147
Joined: Sat Nov 14, 2015 10:48 pm
Has thanked: 2437 times
Been thanked: 3990 times

Re: Why I Hate Overall Ratings, A Thread

Post by RonCo » Tue Nov 17, 2020 1:41 pm

Here's a little test snippet of what I mean (and I think Aaron & Ted mean). Overall ratings can be influenced by both actual ratings (with different splits vR and vL in the editor), and by performance.

I'm pretty sure ours are done by the standard set (but am not perfectly sure!), so:

Ratings: 30%
Current Year Stats: 50%
Last Year Stats: 15%
Two Years Ago Stats: 5%

If I create a test player in a standard test league I've got, and go play around with all the factors (batter handedness, batter splits vR/vL, and whether stats are included in the overall ratings), I get a chart that looks like this:

OVR-BATTING-SPLITS-HANDEDNESS.PNG

So, batter (or pitcher) handedness drives hitter splits upon player creation, but changing handedness does not do anything unless you also go into the editor and change the core ratings.

This means that Aaron is not exactly "right" when he says Player A is rated better than Player B because he is left=handed, but it is _likely_ true that his splits represent a split more normally attributed to a left-handed player. Alternately, though, a player's performance could have been up or down so far that the influence on stats is enough to make a difference. You just don't know.

All of this really goes to my point. It's not that overall ratings are "bad" or even wrong, but to Brandon's point above, I think we get complacent in using them as if they are real--or at least as if they are more solid than they really are.

At the end of the day, I don't actually 'hate" overall ratings. I like them as generic markers for my players. But when I read an overall rating I guess what I'm doing inside my mind is adding a "plus or minus 15 points" to the rating--and that sometimes, that +/- 15 can be pretty important.
GM: Bikini Krill
Nothing Matters But the Pacific Pennant
Roster

User avatar
Jwalk100
GB: FL Pacific Division Director
Posts: 4308
Joined: Sat Jan 11, 2020 9:42 pm
Has thanked: 2536 times
Been thanked: 1233 times

Re: Why I Hate Overall Ratings, A Thread

Post by Jwalk100 » Tue Nov 17, 2020 1:57 pm

Let's go stats only!
Image
ImageImageImageImageImage

User avatar
niles08
BBA GM
Posts: 2541
Joined: Wed Mar 29, 2017 9:15 pm
Has thanked: 171 times
Been thanked: 471 times

Re: Why I Hate Overall Ratings, A Thread

Post by niles08 » Tue Nov 17, 2020 2:47 pm

Jwalk100 wrote:
Tue Nov 17, 2020 1:57 pm
Let's go stats only!
now you going and saying something like that is going to end any discussion on any changes period lol.
Image

User avatar
RonCo
GB: JL Frontier Division Director
Posts: 22147
Joined: Sat Nov 14, 2015 10:48 pm
Has thanked: 2437 times
Been thanked: 3990 times

Re: Why I Hate Overall Ratings, A Thread

Post by RonCo » Tue Nov 17, 2020 2:52 pm

Stats-only is fun, but takes a different kind of GM mindset.
GM: Bikini Krill
Nothing Matters But the Pacific Pennant
Roster

User avatar
aaronweiner
BBA GM
Posts: 12398
Joined: Wed Jun 16, 2010 1:56 pm
Has thanked: 89 times
Been thanked: 935 times

Re: Why I Hate Overall Ratings, A Thread

Post by aaronweiner » Tue Nov 17, 2020 4:08 pm

Even MLB doesn't go stats-only. Why should we?

sjshaw
Ex-GM
Posts: 587
Joined: Sat Mar 30, 2019 4:09 pm
Has thanked: 292 times
Been thanked: 105 times

Re: Why I Hate Overall Ratings, A Thread

Post by sjshaw » Tue Nov 17, 2020 4:29 pm

I’ve been in leagues without OVR/POT ratings and it was a nightmare. They suck, but help you quickly separate wheat from chaff.
GM, Louisville Sluggers, end of 2038 - current

Image

bcslouck
Ex-GM
Posts: 3252
Joined: Sun Oct 02, 2016 10:09 am
Location: Millersville, MD
Has thanked: 394 times
Been thanked: 322 times

Re: Why I Hate Overall Ratings, A Thread

Post by bcslouck » Tue Nov 17, 2020 4:29 pm

aaronweiner wrote:
Tue Nov 17, 2020 4:08 pm
Even MLB doesn't go stats-only. Why should we?
That has always been my mindset on stats-only.
Brandon Slouck
Rocky Mountain Oysters (2058 - present)
Cairo Pharaohs (2057)
Charm City Jimmies (2029 - 2049)
Paris Patriots (2028)

bcslouck
Ex-GM
Posts: 3252
Joined: Sun Oct 02, 2016 10:09 am
Location: Millersville, MD
Has thanked: 394 times
Been thanked: 322 times

Re: Why I Hate Overall Ratings, A Thread

Post by bcslouck » Tue Nov 17, 2020 4:30 pm

sjshaw wrote:
Tue Nov 17, 2020 4:29 pm
They suck, but help you quickly separate wheat from chaff.
I disagree. This is the only league I've been in with overalls and the others were/are perfectly fine.
Brandon Slouck
Rocky Mountain Oysters (2058 - present)
Cairo Pharaohs (2057)
Charm City Jimmies (2029 - 2049)
Paris Patriots (2028)

Post Reply Previous topicNext topic

Return to “League Features”

Who is online

Users browsing this forum: No registered users and 6 guests