2038 Opening Day Relative Ratings Snapshot: SP

[RonCo]

With the move to relative ratings, I’ve been interested in how we are interacting with the rating system the game is showing us. WE did a set of ratings breakdowns in the early days of the transition, but that was mostly a picture of what the game did. Now we’ve got a full off-season and spring training behind us. So how are we adjusting? What do the ratings mean in actual practice? Who are we playing and not playing?
Some of this is about scarcity, right? How many guys of various ratings are we playing is the right answer, but the caveat here is to wonder if that’s because there are no other guys available.

THE PROCESS

I say that because in answering the questions, I’m going to take a snapshot that looks like this: give me a list of all players listed as SP and who have started a baseball game for a BBA team. This sounds good, but in practice it means I’ll miss a couple things. First, any player on the DL (like my own Hiroyasu Osaragi) doesn’t count right now. In addition, guys who are RP and starting don’t make it either. Both Madison and Brooklyn are working in this way, and I don’t want to include them because a player’s RP 20/80 rating will be different than his SP 20/80 rating. I’d rather not mess up the data.

So, yeah.

Bottom line is this…the charts below are essentially a histogram of the relative ratings of players we’re using. If they matched the concept, they’d be perfectly “normal” and focused around “50.” Of course, they are not. Instead, in practice today they are focused on 55, and jagged.

HEREs THE DATA

Code: Select all

POT	30	35	40	45	50	55	60	65	70	75	80	Total
Pitchers	3	10	11	11	25	24	22	8	12	4	9	139

2038-SP-RAT-Hist.PNG (9.49 KiB) Viewed 2175 times

HERE’s THE DATA BY TEAM

Sorted by average 20/80 potential.

A FEW COMMENTS

I think it’s important to understand that, by the usage (And assuming you trust the OOTP ratings…which is a different topic all-together), a “50” starting pitcher is NOT average. 55 is average. I’ll admit I’m deeply interested in how this will manifest itself in actual performance. How far of a drop is there in the FIP or ERA of a 55 vs. a 50? I have no real idea. It could well be that the OOTP algorithm for these calculations is so weird that it won’t really matter. Only time will tell.

I also think it’s interesting to see the team-by-team breakout—particularly how the scatter spreads across the chart, as well as noting that Madison and Brooklyn’s experimental approach is bending the concept a bit. Which is interesting, eh?

Anyway…there’s the data. What do you see? Feel free to educate me!

[Ted]

Do you think that the average and median being closer to 55 than 50 is due to OOTP's inability to contain our overall ratings bloat, or is it an fault in the relative ratings design itself? Or do you think this is working as intended.

[felipe]

wow...how shit were we last year, then?

[usnspecialist]

Do you think that the average and median being closer to 55 than 50 is due to OOTP's inability to contain our overall ratings bloat, or is it an fault in the relative ratings design itself? Or do you think this is working as intended.

we could also just be pickier with our pitchers and not willing to start as many 50s as the game expects.

[JimBob2232]

Very interesting analysis. Thank you.

[aaronweiner]

Do you think that the average and median being closer to 55 than 50 is due to OOTP's inability to contain our overall ratings bloat, or is it an fault in the relative ratings design itself? Or do you think this is working as intended.

I think it's just noise.

We haven't had the ratings long enough or done a sufficient cross-section - right now we're going on the eye test, and that often fails.

[RonCo]

Do you think that the average and median being closer to 55 than 50 is due to OOTP's inability to contain our overall ratings bloat, or is it an fault in the relative ratings design itself? Or do you think this is working as intended.

I don't know, but I've seen it before.

My point with all things relative ratings is that one should not assume a standard interpretation--which in ways makes sense, anyway, because as the scouting article linked a few weeks ago said, scouts themselves didn't/don't use it right.

[agrudez]

It'd be interesting to see this list done again at the end of the year with aggregate WAR or average FIP or something as the new 'sort by' feature.

[7teen]

Nice work Ron. About where I'd expect Madison to be.

Could you help me out a little and tell me who the 2 pitchers I have that are starters? HAHA HAd you done this feature after sim 2 there'd be 0 listed.

[ae37jr]

I highly recommend anyone that is still confused by the relative ratings to cycle through "ratings relative to...." while looking at your minor league teams. It does good job of demonstrating how we are "grading on a curve" so to speak. Basically there are a set amount of grades handed out(20-80) that will always stay constant.

Let's say someone, on any team, calls up an uber prospect who is the greatest player of all time. Every single player in the BBA will technically "lump" in the eyes of scouting. It will be slight and you won't notice it for anyone except those on the edge of a scouting level. So note that bumps and lumps are slightly less important now, cause they may not equal a player getting better or worse. Simply that the league is changing around them.

I suspect that the breakdown of grades will be relative to the scarcity of talent at each position. What this chart shows is that elite starters are hard to come by, there is a few really, really bad starters(see Brooklyn), while the rest is in a tight group of being relatively the same. Think of it as two players playing tug of war of 100 skill points. The cream of the crop beats the replacement player 80/20 while two league average players split 50/50. Do a chart for corner outfielders and I'm willing to be that it is much more equal across the board as the difference between a good one and a bad one is a lot less.

[RonCo]

Basically there are a set amount of grades handed out(20-80) that will always stay constant.

I think your view is basically on target except for this part.

[Ted]

Not to completely derail this, but I also find it interesting how OOTP comes up with the grades. It really does seem to just be a stuff+motion+control+some platoon split factor+etc, which may not be very accurate in some cases. An example is the game calling Aki Kondo an 80. I don't know about you, but there a a ton of the 70's I'd rather have. But Kondo's stuff is so high, and he he has very good control, so it overcomes the 1-2 more motion I'd like him to have.

This isn't really just a high motion low motion debate. We see this all the time in those AA caliber players who have a great glove and 10 speed/steal and get called 35 or 40, when there are players who have say AAAA ability, but every facet of there game is sub big league level getting called 25.

I don't know if there is a good solution to this. Adding some sort of "well roundedness" factor to the scouting math, or perhaps subtracting points for skills that are sub big league average would help, but I'm sure that's harder to do than it is to simply state.

I also wonder how much of the unreliability of the overall scouting grades is a product of our particular type of ratings bloat. Can the game simply not compensate for the huge disparities we have in the ranges and distributions of some of the skills? (i.e. 10+ stuff being more common than 8+ motion, 10+speed 10x as common as 10+ eye). Our particular OOTP player pool has developed a huge power/motion mismatch. It's why we're setting HR records left and right. 15 or so seasons ago we were in a relative deadball era. This changes over time, but I wonder if the overall scouting grades (relative ratings or otherwise) just don't handle when the trend gets extreme very well.

Maybe I'll make some "Kondo-esque" players in a default MLB league with relative ratings on and see if it does better.

[RonCo]

The question of both overall ratings and relative ratings--and how they work--is a decades old conversation among people deeply involved. My guess is that the valuations under the hood are more complex than you're suggesting, but regardless, they are valuations that the game is deciding, rather than you. By using overall/relative ratings, you're outsourcing a lot of information.

As I said, I know a lot of people like them better than the raw ratings--but my personal view is that they like the concept better than the execution. And they are better able to pretend that the game is mostly right than I am.

[RonCo]

Another area of uncertainty is how the ratings convert to performance. I say this because the ratings scale is not linear in performance--20 points in one place of the scale is not always worth the same performance as 20 points in another place on the scale. Is OOTP baking this into their 20/80 viewpoint and their "relative" calculations? I dunno.

There are several ways to consider the ratings structure. All have their upside and their downside.

As several have said, with our current scheme you can get a better ides of where a guy is relative to his current minor league level. That helps some folks, but to me is like bowling with bumper rails up. A large part of the fun in developing players is deciding when a guy is really ready to move. Blah, blah, blah.

[RonCo]

Ted used Kondo as an example. But let's look at a lesser pitcher...I'll pick on me...Luis Colon, who is a rookie pitcher rated 7/6/5 in talents, with two blue pitches and a third solid green "7" (and a 7 stamina).

At 7/6/5, that says his stuff should be considerably better than average, movement above average, and his control about average or a tick below. We all know this is probably wrong, but that's what the numbers should say. Or is it wrong? OOTP itself says it's wrong, calling him a "50" potential (which we now know is below average).

But Colon also has a really nice reverse split against lefties, which will protect him a bit. So he'll essentially be pitching with the platoon advantage all the time. He's a fly ball pitcher, though, which means if we could get him some real defenders in the outfield, my guess is he'd be above average. I'd personally grade him as a "50"-"55" or so in that situation. But our OF defense is not so good, so maybe he should be a "45". What's right? I dunno.

And I especially don't know because I have really no idea what base performance the game is calling 7/6/5. Give me the raw ratings and I'll make up my own mind. As it is, though, I'll mostly just throw him into the rotation and see what happens.

[Ted]

Good thoughts, Ron. I'll admit that I'm in the camp of liking the idea of relative ratings better than the execution. I was pretty excited at first, but as time has gone forward, I'm less so. I'm still okay with them. I'm trying to tell myself that this is the "right" kind of scouting fog of war, where you know which guys are really good, and which are really bad, but for the rest you have to rely on performance. I may just have to give up on the overall scouting grades though. Or at least give +/- 10 on all of them.

I know the guy with 80 has more of "something" then the guy with 70. Whether or not any of that "something" make him better is another thing entirely. As you said, in some places 20 points is worth much more than others.

I think the thing that bothers me so much is that this shouldn't be so hard to get right. It's just math. If the devs are trying to incorporate GB/FB, etc into this they're making it too complicated. Either it's overly simplistic (which yeah, I also doubt, despite my comments) or they've tried to be so precise that they've created an endless proliferation of error cases where the mechanism breaks down. (This is what I actually suspect is the case)

You should be able to find some combo of average/median for a skill that means something, and rate everything else in a meaningful distribution to it fairly easily. This isn't hard. That's standard statistical math. You should then be able to weight those skills appropriately and come up with an overall grade that makes sense. Again, standard math, especially when you know how the engine works.

I bet if I talked to the devs, they could humble me with why it is so hard and doesn't work out. But from the outside looking in, this just seems like it should be better.

[RonCo]

I think the thing that bothers me so much is that this shouldn't be so hard to get right. It's just math. If the devs are trying to incorporate GB/FB, etc into this they're making it too complicated. Either it's overly simplistic (which yeah, I also doubt, despite my comments) or they've tried to be so precise that they've created an endless proliferation of error cases where the mechanism breaks down. (This is what I actually suspect is the case)

You should be able to find some combo of average/median for a skill that means something, and rate everything else in a meaningful distribution to it fairly easily. This isn't hard. That's standard statistical math. You should then be able to weight those skills appropriately and come up with an overall grade that makes sense. Again, standard math, especially when you know how the engine works.

I bet if I talked to the devs, they could humble me with why it is so hard and doesn't work out. But from the outside looking in, this just seems like it should be better.

Take a day and pull all the csv files on a test league and try to make one up. It is actually harder than you might think. The fact that it's "pure" math makes it feel simple, but it's not. Raw ratings go all over the board, and (again) the performance maps aren't linear. The fact that it's just math means that, unless your data is actually normal (which sometimes it is and sometimes it isn't), the 20/80 scale can't possibly be right. Just for an example.

So, in these cases, do you re-jigger the overall ratings to force them to be normal? If you do that, then you're ratings are wrong. Or do you stay with your system, which won't normalize anything.

Overall assessment and relative ratings are actually quite a bit more complicated down in the math than you think on first blush. Heck, even human beings doing scouting don't use it right, and they're capable of forcing the fit if they need to. At the end of the day, the 20/80 scale is not really a statistically mathematical thing, though everyone talks about it like it should be.

[RonCo]

Mainly, I like the 20-80 scale mostly because it sounds like baseball.

Personally, I would prefer the 2-8 scale, done on raw ratings. They wuold feel very baseball-like, and would give lots of fog of war (a 7-point scale rather than a 10 point we used to use), and drive even more reliance on actual stats than we used to have. But that resolution would make a lot of players look similar, too. But I'm weird, I know. And it would make the ability to read stats even more important...which would separate GM skills further.

[agrudez]

I only skimmed the above conversation, but whether the game calls it 20-80 grades or 0.5-5 stars it's the same thing. And players having good grades/stars being lesser than others with bad grades/stars is a tale as old as time. Even in a neutral stadium a 40 grade LHB > 40 grade RHB. Maybe a bit less so, but still true, of LHP > RHP. That's because the game applies a double penalty/boost to handedness matchups (ie. a pitcher is lesser against a batter's handedness AND a batter is greater against a pitcher's handedness simultaneously), then amps it to 11 for 90% of LHBs (RHBs generally get even-ish splits while LHBs get anywhere from slight splits to extreme ones) and then pops out a disproportionate (but realistic) ratio of RHP:LHP.

Other ways that grades/stars don't tell the whole story is that they are based on OVR ratings, so guys with extreme splits are undervalued. Guys like Fernando Reyes that are elite against 70% of the pitchers in the MBBA and AAAA at best against 30%. It's pretty easy to platoon them and get production well above their grades/stars (which inevitably means that you get production well above their salaries since the game seems to use grades/stars as one of their primary means for setting contract extension demands - in my anecdotal experience / opinion).

[RonCo]

I definitely agree that reading overall ratings is dangerous, and to be honest that's one reason I don't like relative ratings--since I know they don't actually say what people think they say, it's more likely they'll make mistakes. But they'll feel better in the process.

I also clearly agree on the use of splits. But I think it's important also to realize that the splits for LHB are much deeper than for RHB, so many RHB can play every day while relatively few LHB can. So contract rates are interesting in that fashion. With our 27-man rosters, it's a little easier to platoon guys. But if you can find a good lefty who hits lefties, you've got a helluva player.

This is a widely simplified pass, but I don't have time to cut it any deeper...but I know I've posted actual performance data in earlier years to show how this translates to actual stats in more reasonable usage. But to show the magnitude of split deltas, I took a quick scrape from the game of all BBA hitters in the majors today, and averaged their splits.

2038-contract-vs.rhp-lhp.PNG (7.27 KiB) Viewed 2048 times

The average LHB platoon disadvantage is about three times larger than the average RHB platoon split, and the average RHB is about a half-point better against RHP than LHB are against LHP. The standard deviations are also interesting.

Good GMs can make wise decisions around these deltas.

All that said, these numbers above are operationally wrong because not all batters will go to the plate as often, so there's some usage/selection bias at play when you're trying to determine the "real" splits.