[SHAGGS]

Maybe a down firing enclosure like this one? Lots of finishing/blending with the interior options, and you'll get some boundary loading to boost output.

2018 Porsche 911 GT3 - Custom Subwoofer Enclosure -...

Hey folks, I was recently lucky enough to land my dream car: a Guards Red 991.2 GT3 in manual. Though this car will live half its life on the track, the other half will be lived on the street. So, of course my mind went towards the audio system. I had met Vanguard's Nick Apicella and Matt Kim...

www.diymobileaudio.com

 

[SHAGGS]

Not sure there's a formula, but as a general rule aim for 1-3 inches.
Lower excursion subs won't move as much air and won't need as much clearance to the surround, so they can get by with less.
High excursion subs will need more.