the way it works is that the further you go down the disproportionally more out it gets. So anything you add is affecting it, however.... this can be completely offset with running taller roll center adjusters. Dave from AJPS sells a type of steering ball joint that has no grease nipple on the bottom. This allows for much more room before the bottom of the outer steering joint hits the rim of the wheel.
Here's the formula for you:
Height of roll center adjusters needed = Lowered height from unsagged original ride height + crossmember spacer thickness