You repeat the rep range where you had the miss. You keep progressing the other two rep ranges. That is what gets you unstuck.
So if your 5s get stuck but you keep PRing yours 8s and 2s, the 5s will eventually unstick.
On a program like 531 or 852 where you are changing the rep scheme and weight amount on a per workout basis I have a question-
Let's say on a 5x3 day you fail to complete last rep, but you continue on in the cycle and successfully pass the 3x3 and 1x5,3,1 scheme, what is the appropriate next step? Do you
-stay at same weight and try the entire cycle again?
-go up in weight and start a new cycle
-keep the weight the same for the failed part and increase the part you passed?
You repeat the rep range where you had the miss. You keep progressing the other two rep ranges. That is what gets you unstuck.
So if your 5s get stuck but you keep PRing yours 8s and 2s, the 5s will eventually unstick.
If you're doing 5/3/1 you should never fail a rep. In a PR set you should leave 1 rep in the tank (RPE 8-9), and if you fail to get even the minimum reps (5,3,1) you're using a too high TM for the current program, or having a shittiest training day ever.
Wendler has made a general rule that the TM should be around 85-90% for most lifters. Which means that you should normally get at least 5 reps with your 95%.
I was addressing mainly my 852 protocol, not Wendlers program. The way I do things, a set of 8 means a set of 8, not a set of 13. And a set of 2 means a set of 2, not a set of 6.