You shoot on, and every now and then a bullet strikes the target a couple inches to the left or right (or high or low) of your main point of impact. You start to get frustrated and think about moving the scope–Don’t!
As long as 2 bullet holes stay clustered within an inch, you’re in good shape. Any “flyers” were likely due to human error—you’ve shot too much and flinched a bit or maybe jerked the trigger and pulled the errant shot offline.
The worst thing you can do is worry about one bad shot and start fiddling with your scope, which can totally wreck a perfectly fine zero. Simply throw away a flyer (I cover mine with tape to get it out of sight and mind) and shoot one last group to confirm true zero and get your confidence back.