Inexplicable Faction Failure Rates
Stormagedon
✭✭✭
in The Bridge
I have collected 12 faction shuttles in the last 12 hours.
5 failed.
All had 'Chance of Success' forecasts of over 90%
Before anyone says anything about the RNG and the chance of a series of 'bad rolls' you may want to know this: I have studied probability and statistics substantially at a post graduate level.
There's a little thing called 'regression to the mean'.
Simply put, it means that as the sample size (number of instances) increases, the observed probability will approach the predicted probability, assuming the prediction is accurate.
I have now been playing this game for about a year and a half, and have never even approached a success rate consistent with the posted 'Chance of Success'.
I play other games and do NOT encounter this issue with them, strictly here.
DB, either you will admit your dishonesty AND fix it, or I'm done.
You have until the next server update.
Removal or editing of this post will result in immediate termination and legal action.
5 failed.
All had 'Chance of Success' forecasts of over 90%
Before anyone says anything about the RNG and the chance of a series of 'bad rolls' you may want to know this: I have studied probability and statistics substantially at a post graduate level.
There's a little thing called 'regression to the mean'.
Simply put, it means that as the sample size (number of instances) increases, the observed probability will approach the predicted probability, assuming the prediction is accurate.
I have now been playing this game for about a year and a half, and have never even approached a success rate consistent with the posted 'Chance of Success'.
I play other games and do NOT encounter this issue with them, strictly here.
DB, either you will admit your dishonesty AND fix it, or I'm done.
You have until the next server update.
Removal or editing of this post will result in immediate termination and legal action.
6
Comments
A lot of players have had the suspicion for a while that the value used for shuttle success rate in incorrect.
It’s DBs deck of cards dealt on DBs table on DBs carpet inside DBs casino in DBs town. Players stumbled in and after seeing the incredibly ridiculous rate of return want out. Through DBs door that is...
Just curious. (: Would love to see the numbers.
On other sites I frequent such a post as the one that started this thread would have the poster being whacked over the head with a +5 banhammer.
What are you talking about? Simple logic suggests he's talking about his game account termination. What legal action can you take against a dead man or dead company? Such comments are really unnecessary.
I think the latter is much more likely to be the case; it would explain how the playerbase as a whole has seemed to have particular events where *everyone's* shuttle failure rates are much, much higher than they should be.
Don't be ridiculous. In a previous paragraph I wrote of leaving the game. Termination was in reference to my departure.
Wrong.
I've been tabulating data for over 8 months. I originally kept data on faction events separately, but stopped when I realized that those numbers were even more skewed.
My numbers refer only to non-event shuttles and are broken down into probability groupings; 6o-69%, 70-79%, 80-89% & 90-99%.
Out of 864 shuttles forecast between 90-99%, 227 failed.
That's less than 76% success rate, and the sample size is statistically significant.
And far from regressing toward the mean, the numbers have been getting worse over the last month (down to under 72% for the last 30 days).
The long and the short of it is that DB habitually posts bogus probabilities throughout the game. Now I'm pretty sure we can all figure out why, but I'm focusing on facts, not speculation.
There was someone in another thread with over a thousand data points as well, and his was also off by a very notable margin. I was by no means saying your data was insignificant, I’m just a numbers kind of person.
I completely understand and agree with you; bit of numeri-geek myself.
It was my intention to post pertinent data, but I didn't have time to break it down when I wrote the OP.
Your message gave me the perfect intro.
Thank you 😉
I know this game and the community that has grown around it has value, especially to those of us for whom the Star Trek properties represent important times of our lives and helped forge the people we've become.
I've submitted tickets about particularly laughable statistical anomalies in the past, and without naming names (which would be a violation of the forum's TOS), the responses I received were either initiated by someone with no understanding of statistics or probability, or by someone intending to create a smokescreen.
I chose to make this public because I want to encourage the members of this community to share their experiences and input on this subject.
There's been ongoing threads on this for over two years and DB's stance has never changed (although the last time I heard it addressed explicitly was by Nod in a livestream). It doesn't matter how well-supported the data is (we have had several discussion on statistical relevance and all of that) if DB can just ignore these threads... which they can.
I feel you, but I have seen too many threads on these forums to think that the n-th thread will change something.
By the way, very nice Doctor Who reference!
Thank you. I spoke of legal action in regards to any potential attempts to silence this discussion.
I have kept my posts strictly within the TOS of this forum, and will continue to do so.
I do not visit this forum on a daily basis, in fact there are times I only pop in for event info.
That said, I have tried to keep abreast of as many discussions of in game probabilities as possible.
I have not seen 'regression to the mean' brought up in any of those discussions. There are a number of other statistical concepts and 'rules' of probability I likewise haven't seen referenced.
I have found this fascinating.
Awe shucks, t'weren't nothin'
It's a term that I personally don't like using as it has slightly different meanings in pure statistics and in financial statistics (where, if I remember correctly, it is a principle that doesn't apply to stochastic processes but rather argues from periodicity). I admit that's just my personal preference, though.
I usually tend to compute errorbars (which is trivial assuming a counting statistics), as I think they are easier to understand for the less knowledgeable. Saying "regression to the mean" translates one-to-one to observing that the relative errorbar will mathematically go towards zero as we increase the size of the data - and that the error window should "home in" towards the true value.
That said, I am more interested in what we can do with the data. As said, I have seen dozens of statistically significant threads being ignored by DB. You have not to worry about DB editing or removing this thread (nor blocking it, which is a far more likely scenario than removal), as their modus operandi is entirely different: indifference.
Check out our website to find out more:
https://wiki.tenforwardloungers.com/
My training in probability and stats comes from scientific fields where 'error bars' are only used for back-of-the-envelope calculations. Much like the statistical 'sniff test' it's a useful tool while collecting data, but can't be used when publishing results.
As to DB's indifference, I have seen a number of times here where a general unwillingness of players to 'let it go' has resulted in DB fixing something.
I wish DB was willing to rely on the strength of their product and the fact that so many fans of Trek enjoy it.
I don't see any reason for them to fudge their forecasts. They are unnecessarily creating ill will within their user base.
If DB stops shooting themselves in the foot, they will find life far more pleasant.
As for "can't be used when publishing results", I don't know. I have definitely seen PRL and PRB papers that employed simple error estimations and expressed their results as a window of confidence.
But I also have no interest in this discussion, really. If you think that no discussion on statistics was carried out correctly until you posted on these forums... good on you
Forgive me, I'm merely remarking upon the fact that telling them these same words a hundred of times has not produced any change.
But maybe I'm just a jaded cynic and the 101st time will do it.
Would you mind posting the possible success rate on this before I hit confirm?
Success rate so far: 0%.
I ran 12 gauntlet rounds in a row... Each time, the lower stat toon won, whether it was me or the opponent. Now, I admit, 12 is a very small sample size to make any decision, so it could have been a fluke, but it seems suspicious.
Check it out for yourself
This was discussed on the old forums, before Star-Bases went active. I did my own testing and found that my results matched the consensus of other users, roughly 60/40 when stats were identical.
Roughly a month ago I revisited the topic after noticing changes in the Gauntlet RNG's.
I won't classify my results as significant, but out of 25 samples 11 went to the challenger (me) and 14 to my opponent.
Within the last 2 weeks there has been an increase in degrees of freedom in the opponent selection matrix. I believe this is an attempt by DB to alleviate the much commented on 'Wall of...' phenomenon, where every listed potential opponent is an immortalized Mirror Picard or Mirror Flox, etc. This hasn't completely eliminated the issue, and has resulted in numerous occasions where one opponent offers 140 trophies and the rest offer 10, even when I've first entered a Gauntlet and I'm ranked below 120th place.
It also appears they've expanded the potential range of the rolls, as I'm now seeing mirror matches and other instances of characters with nearly identical stats and crits finish with one instance having almost 3 times the score of the other. This is now not uncommon, where it was previously unheard of in my experience.
I absolutely understand DB's rights to jiggle the odds in the RNG's. My concern remains where they misrepresent those odds.
My specific education, training and experience lies in the fields of psychology and medicine. I also provided narration for a few statistics for social sciences texts for visually impaired students back in my undergrad days (we're talking audio cassettes).
In medical and psychology papers, approximations or windows of probabilities are acceptable in the Abstracts, estimations are not. Complete data and all relevant analysis are required in the Body, Conclusions, and Methods sections for everything but the most preliminary research.
I do not intend to cast aspersions on ANY prior discussions of these matters. If I didn't feel I could bring anything new to these discussions I would not have bothered to post in the first place.
I also can't help but wonder why you would bother to post in a discussion in which you profess to have no interest? 😉
You misunderstand. I'm here for the "what does the data say" and the "what to do", not to have a discussion that start with you describing all past discussions with the words "I have not seen 'regression to the mean' brought up in any of those discussions", whatever that meant.
To be honest I have now no intention of discovering what the purpose of that "ooh but no discussion ever mention the regression to the mean" statement. Anyway, I am happy that medical and psychology papers have standards. As a physicist, my past discussion with psychologists on the topic of statistics have left me a bit... ehr... unconvinced.
Anyway. Back to "what the data says" (you have not released the complete data, by the way, which I'd find interesting) and the "what to do".