Your fine, the left side of the bearing is against the shoulder of the axle and the right against the fork. Don't "wack it", you torque the axle then tighten the two pinch bolts at the bottom. Putting side force on the fork to make it flush will end up binding it.