I'm trying to think of options I have without sacrificing the work I've done so far.

What about reinforcing the baffle from the inside? I lose a bit of volume, but it's still within the recommended range specified by Stereo Integrity.

edit: I just measured and it's almost at 3/8's of an inch.