My only concern with taking off 34.25 thou per spacer or bearing carrier (whatever is the correct description) was that next time you are knocking out the bearing cup the spacers might buckle.
I didn't measure the width of the spacer, so maybe taking off some metal will not cause any problems.