Options
4000+ Gauntlet Rounds
Peachtree Rex
✭✭✭✭✭
in Ready Room
Compiled Rounds Spreadsheet:
https://docs.google.com/spreadsheets/d/1sYlf7A6M-kaENSsxoHhiUrpNuv71zNRz50Izi-ARB6E/edit#gid=0
2500 Round Thread:
https://forum.disruptorbeam.com/stt/discussion/11125/2500-rounds-of-gauntlet-data#latest
1000 Round Thread:
https://forum.disruptorbeam.com/stt/discussion/10548/1000-gauntlet-rounds-of-data#latest
Unless something dramatically changes, this will be my final installment with this information. I utilized the round export functionality on the IAmPicard tool that has now been discontinued.
Something new I added to my analysis is the ability to create some amount of custom comparisons (available on the "Custom Analysis" tab).
I'm not really going to do any analysis of my own here. Partially because I didn't really expect to be writing this post until I hit 5000 rounds but also because there really isn't that much to say that I haven't written in one of the prior threads.
If I was to tie a bow on this entire analysis and offer a conclusion, it would be this: In terms of how rolls and crits are performed, I am pretty certain that everything is on the up-and-up. Even if they are biased, it is in a manner that is so small that it doesn't seem to actually impact wins/losses in any significant fashion.
https://docs.google.com/spreadsheets/d/1sYlf7A6M-kaENSsxoHhiUrpNuv71zNRz50Izi-ARB6E/edit#gid=0
2500 Round Thread:
https://forum.disruptorbeam.com/stt/discussion/11125/2500-rounds-of-gauntlet-data#latest
1000 Round Thread:
https://forum.disruptorbeam.com/stt/discussion/10548/1000-gauntlet-rounds-of-data#latest
Unless something dramatically changes, this will be my final installment with this information. I utilized the round export functionality on the IAmPicard tool that has now been discontinued.
Something new I added to my analysis is the ability to create some amount of custom comparisons (available on the "Custom Analysis" tab).
I'm not really going to do any analysis of my own here. Partially because I didn't really expect to be writing this post until I hit 5000 rounds but also because there really isn't that much to say that I haven't written in one of the prior threads.
If I was to tie a bow on this entire analysis and offer a conclusion, it would be this: In terms of how rolls and crits are performed, I am pretty certain that everything is on the up-and-up. Even if they are biased, it is in a manner that is so small that it doesn't seem to actually impact wins/losses in any significant fashion.
7
Comments
Shuttle success rates were evidently fine if we were to believe most aggregate result collections that were posted to the forum. So I guess shuttles were never bugged in any way either, right? Right.
It's amazing the number of players who are apparently not paying attention to what happens during match selection. Oh well. Everyone has their cognitive biases.
So, your first point on shuttle data is completely wrong and revisionist history. Large samples showed that there WAS a problem, but it was different between players/missions/etc. It wasn't until the AND behavior was identified that these differences were actually able to be reconciled.
To your second point, this is a classic moving of the goalposts. When I started this study, very few, if any, people were moaning about match selection. It was all complaints about bogus crit rates and alleged cheating during "reward" rounds. So, that is what I set out to quantify and I'm pretty sure I have provided sufficient evidence that both crits and rolls are operating as advertised.
I've been very open to what I have (and have not) been testing. As I've said before, I offer no comment on the functionality and fairness of matchup selection. However, even if matchup selection was biased against the player, the only net effect is "you can choose to spend more merits if you want to re-roll the matchup". This is far less malicious than the actual matchup being rigged.
I totally agree that a study on matchup selection would be worthwhile. The parameters of such a study would be very difficult to pin down as there are a lot more variables to consider than what I was looking at. However, I'm pretty much done with spending my time on community efforts for this game, so someone else will have to take up that effort.
This would be extraordinarily hard, as you point out w.r.t. multiple variables. It's a Bayesian problem with lots of unknown and basically unknowable priors, you can certainly throw any of your own estimates in (that's the point of priors) but they'd be lousy and would probably just reaffirm for you personally what you already suspect.
The most important is the probability that any given Gauntlet participant HAS a particular card to play. Maybe that could be proxied by surveying an enormous number of gauntlet players for their complete crew manifests. But even if you had that, you'd still have an unknown factor of gauntlet players' strategies in populating crew. Who shows up in your matchup, if unbiased, is something like P(opponent appears | (player base has opponent AND has fielded opponent)). This is why we have walls of certain characters, lots of people have them AND believe they're Gauntlet gods and field them all the time - often when they probably shouldn't.
Because of that, doing the study is just as impossible as divining bias from what you see on the screen.
Gosh I wish the search function on the forums functioned adequately. My remembrance of shuttle data is completely different and my impression is that the "everything is functioning as intended" crowd was much louder and more pervasive.
I'm happy to be concede on this though since it highlights the importance of shuttle mission selection tracking and it's impact on outcome statistics.
Furthermore, it's a great illustration of how coverage bias can affect the perceived results of data collection studies. I'd be curious to know how you've avoided coverage bias in your data collection efforts.
Hmmm... I seem to remember at least one post on your data collection posts suggesting that some of the observed vs. expected outcomes were potentially noteworthy. I don't recall if that particular poster ever followed up with a more detailed analysis.
I was, and remain, agnostic on match outcomes. I've been a critic of match selection since shortly after the gauntlet was released and have mentioned it in response to at least two of your data collection posts going back to the first one. You were rather dismissive of me raising this point earlier, so it's nice that you've moderated your stance since then even if nothing has come of it.
I'm in full agreement that such a study would be very difficult given the data we have access to. But if a player doesn't spend merits for a re-roll (or even if they do), a biased selection would impact that player's potential match outcome. Thus, your proposed net effect is incorrect.
Match selection data isn't a nice to have, it's essential for providing context to match outcomes. This is a fatal flaw to even the best aggregate outcome data collection.