EDIT: Final Results: 2nd Experiment results: AND shuttles are absolutely bugged
![[SSR] GTMET](https://us.v-cdn.net/6030345/uploads/userpics/724/nA97JZAFVHVNR.jpg)
[SSR] GTMET
✭✭✭✭✭
Summary: By looking at shuttles that have 1 crew slot, we can eliminate variables and more closely study the effects of the AND node. By doing this, 3 different testers, including @(FF) This Sisko1, @[TFA]Múspell and 1 other found nearly identical results where the 2nd skill of and nodes grossly under perform expected results. We can then suggest that you should treat AND nodes as 1st + 25% * 2nd, not Highest + 25% Lowest as was the prior communication.
Edit: Example for clarity:
in a DIPLOMACY AND COMMAND node, Cmdr Keras (890 CMD, 617 DIP) should score as:
890 + 25% * 617 = 1044 (highest of the 2 skills plus 25% of the second). This is the stat used to calculate the displayed chance of success.
Instead, when the game is calculating if you pass the shuttle, it uses the following formula:
617+ 25% * 890 = 839 (First skill (DIP) + 25% the second skill (CMD)).
Background: After investigating our findings from the last event and proposing the AND node failure mode, @Peachtree Rex suggested utilizing the federation shuttles to compare single skill shuttles against the single crew AND shuttles where there should be a huge difference in the data if AND nodes are being mis-calculated.
Experimental Design: To achieve this we looked at "Rescue Citizen Captives" DIP AND SEC and "Investigate corruption" DIP AND CMD. We needed to test for the hypothesis that the AND node is calculated 1st 2nd while controlling for potential other bugs in the mission or bugs in calculation. To do this:
3 testers ran the AND missions with only Skills in the 2nd of the AND skills (security and command) to see if our results far underachieved the displayed chance of success
1 tester ran the same AND missions with only a skill in the 1st spot (diplomacy and diplomacy) to see if the shuttles matched the chance of success
3 testers ran single skill missions to see if those matched the chance of success.
Results: The missions with crew in the 2nd AND slot drastically under performed expectations, showing a result of 13% compared to the expected 71%.
The crew in the 1st skill of the AND missions were successful 16/16 runs. The small sample size could account for the abnormally high success rate, although we cannot rule out that there is actually some error that is boosting the success of the 1st skill AND missions, although that was out of the scope of this experiment. We can however rule out that the shuttles are drastically under performing the displayed success rate
The crew in the single mission shuttles came back right in line with expected success rates across the 3 testers.
Conclusion: There is now a preponderance of data with other variables ruled out that the AND missions are clearly calculating as 1st + 2nd, not Highest + Lowest.
Additional notes: The expected success rates was an estimate from the testers, since the high success or failure rates would move the number and we all had slightly different crew, there was some variability, but in general all the shuttles had displayed ranges in the 68-75 range, sufficient for the scope of this trial.
ORIGINAL POST:
All, a team of 4 people led by @(FF) This Sisko1 have been comparing single slot AND shuttles with regular single slot shuttles. We don't have all the final data in, but with a faction event starting tomorrow we wanted to share the preliminary results.
While single skill missions are performing right around the displayed % of ~70%, the AND shuttles using only the 2nd skill have bombed at less than %10 success rate. We have a few more rounds of data to collect and some number crunching, but with an N for each group of between 50-100, there is just no doubt any more that the AND shuttles are not performing as intended.
It is our recommendation that you only use AND slots where you utilize the first skill in this next event. We will publish our final combined results soon.
Edit: Example for clarity:
in a DIPLOMACY AND COMMAND node, Cmdr Keras (890 CMD, 617 DIP) should score as:
890 + 25% * 617 = 1044 (highest of the 2 skills plus 25% of the second). This is the stat used to calculate the displayed chance of success.
Instead, when the game is calculating if you pass the shuttle, it uses the following formula:
617+ 25% * 890 = 839 (First skill (DIP) + 25% the second skill (CMD)).
Background: After investigating our findings from the last event and proposing the AND node failure mode, @Peachtree Rex suggested utilizing the federation shuttles to compare single skill shuttles against the single crew AND shuttles where there should be a huge difference in the data if AND nodes are being mis-calculated.
Experimental Design: To achieve this we looked at "Rescue Citizen Captives" DIP AND SEC and "Investigate corruption" DIP AND CMD. We needed to test for the hypothesis that the AND node is calculated 1st 2nd while controlling for potential other bugs in the mission or bugs in calculation. To do this:
3 testers ran the AND missions with only Skills in the 2nd of the AND skills (security and command) to see if our results far underachieved the displayed chance of success
1 tester ran the same AND missions with only a skill in the 1st spot (diplomacy and diplomacy) to see if the shuttles matched the chance of success
3 testers ran single skill missions to see if those matched the chance of success.
Results: The missions with crew in the 2nd AND slot drastically under performed expectations, showing a result of 13% compared to the expected 71%.
The crew in the 1st skill of the AND missions were successful 16/16 runs. The small sample size could account for the abnormally high success rate, although we cannot rule out that there is actually some error that is boosting the success of the 1st skill AND missions, although that was out of the scope of this experiment. We can however rule out that the shuttles are drastically under performing the displayed success rate
The crew in the single mission shuttles came back right in line with expected success rates across the 3 testers.
Conclusion: There is now a preponderance of data with other variables ruled out that the AND missions are clearly calculating as 1st + 2nd, not Highest + Lowest.
Additional notes: The expected success rates was an estimate from the testers, since the high success or failure rates would move the number and we all had slightly different crew, there was some variability, but in general all the shuttles had displayed ranges in the 68-75 range, sufficient for the scope of this trial.
ORIGINAL POST:
All, a team of 4 people led by @(FF) This Sisko1 have been comparing single slot AND shuttles with regular single slot shuttles. We don't have all the final data in, but with a faction event starting tomorrow we wanted to share the preliminary results.
While single skill missions are performing right around the displayed % of ~70%, the AND shuttles using only the 2nd skill have bombed at less than %10 success rate. We have a few more rounds of data to collect and some number crunching, but with an N for each group of between 50-100, there is just no doubt any more that the AND shuttles are not performing as intended.
It is our recommendation that you only use AND slots where you utilize the first skill in this next event. We will publish our final combined results soon.
41
Comments
Eh, I pretty much fail 2/4 every time anyway and I am sure Troi and Picard are going to need federation stuff, so probably wasted 100 merits. Besides, I love a good experiment
There was one mission last weekend called Adjusting the Emitters. It was:
ENG AND CMD
ENG AND SEC
ENG AND DIP
I filled each seat with a crew using the second skill (ex T'Mir, KalSpock, Sybok) and then used a 3* ENG boost even though none of these crew have ENG. Typically I would have a 90-95% displayed success chance. I didn't notice any unusual failures all weekend.
Unknown, at this point. Boost usage, to the best of my knowledge, was not tested. Boosts also add such a small % to overall success that it would require a LOT more shuttles to definitively say anything. Since the exact mechanics of the "AND" bug is not really known either, it's hard to say what might be broken with Boosts and what might be broken with the "AND" slots.
To test boosts, you'd probably want to use low-power crew (so the boost has a bigger impact on shuttle success). But we're still talking a modest impact to expected success with a number of different permutations:
AND Character Skill 1 with boost of Skill 1
AND Character Skill 1 with boost of Skill 2
AND Character Skill 2 with boost of Skill 1
AND Character Skill 2 with boost of Skill 2
(900+250)*2 = 2400 points per slot
It was calculating as:
(900/4+250)*2 = 950 per slot
Those of us tracking it were getting results on that mission approaching 50%
If this is actually a (putridly awfully described) feature rather than a bug, it does add a lot more strategy to choosing who goes on which shuttles. And, by extension, it puts even more strain on crew slots. Instead of having one crew who can do both permutations of a particular AND slot, you would theoretically need two separate crew to maximize your success rate unless both crew had relatively close primary and secondary skills (like how T’Kuvma or Mirror Picard have COM and SEC skills that are only a hundred or so points apart, while Robin Hood has a several hundred point disparity between his COM and SEC skills).
I leave open the possibility that DB has more than one person on the design team and that they communicate poorly with programmers, writers, and Shan. It would fit what many suspect already, no?
On iOS here, and I haven't yet noticed anything that resembles an anomalous pattern since I started keeping a closer eye on it...
Now to figure out that in a 3/4 crew mission, the AND setup was what caused the false fails. Totally makes sense that if the shown was 95%, but the actual was at 85%, then the difference was the slot or two that I used the wrong crew (1st vs 2nd skill). Had they explained to us that the first skill shown had to be the primary, I would have always setup my missions that way.
Anyways, so far I see 5/40 for the second skill missions. While I am personally at a 20% success rate, others haven't passed a single one. Shoutout to GTMET for getting the ball rolling and not only sacrificing the time but also lowering his Federation difficulty because of the fails lol.
I usually am 50/50 on weekly shuttles so this didn't hurt me and I have a ton of film as a result.
Strange. I did the exact same thing and failed that mission 5 times in a row (4 as I kept repeating 2250 points, and the last time at 2750 points), and wound up avoiding it for the rest of the weekend. My average success rate was much better without risking that mission.
They were changed because they used to be an average, not highest plus 25% lowest. The change displays correctly, and I am pretty sure there is code to calculate it both ways, but there is clearly some bug that always uses the first case, not the highest case.
Same results here. This specific shuttle failed 90% of time despite it shows 91% success chance.
Would I be right in assuming that even if a crew member has both skills required on an "and" shuttle, they better be most proficient in the first skill listed, in order to approach the posted success rate?
https://forums.disruptorbeam.com/stt/viewthread/63133
This was the info we got and have relied on for the last year
By June of last year I had gone bananas with all the fails and finally started to ask:
Grr if only we thought the of the issue sooner.....
Like for a "Dip And Com" node, you're showing that you shouldn't put someone with just high Com. But are you still better-off with a crew with high Dip & Com, as opposed to just a high Dip? Or does the second part not really matter at all?
Thanks for all your work on this!
First Skill + 25%* second skill
So you are always better using 2 skills, but if your crew is 900 command 400 security but the slot is security AND command then you are better off putting a 700 security crew in the slot
I always just chocked it up to programmers randomly assigning traits......
Someone was just having a fun time at the dartboard.
Has anyone ever asked why the points needed for a 4000VP shuttle (or any) isn't shown?
If the order is supposed to matter, then there is a bug with how the expected pass rate as well as sorting the ideal crew for the slot.
There is a discrepancy between the client and server. They cannot both be right.
Timelines Crew Cost Viewer
I however push buttons because they light up and make sound. I don't stop to ask why or even notice a small key idea. I just tappy tap. I hate expedition events with a passion but I sure do well in them because I know how to mindlessly tap lol.
However I have a lovable personality which I guess must be addictive to every great member in FF. Which is why I love our 100 members