Yes it can; especially if your hand is slipping forward (which may not be enough to notice but can be enough to affect your group) or you're repositioning your hand every so often and not getting it back in the same exact place each time.
If at all possible jam your non-firing hand up against that sling swivel. If your body can't do that get some "position-in-a-can" and glue your mitt in place. I have to do that for Sitting. It holds the rifle so I can relax.
Also get youself a landmark for the position of the butt in your shoulder. Mine is slightly below the collarbone in the pocket (again just like in the movies
) Inconsistent placement can move your shots up and down too.
Maury