Group delay is calculated from the phase plot and if you know one, then you know the other one too. Group delay is generally much easier to read since the vertical axis is time measured in (usually) milliseconds, which makes much more sense to most people than degrees of phase.
If you have a phase shift in your system that means some frequencies will arrive at your ear after other frequencies in terms of time. Usually the bass frequencies trail the midrange and treble frequencies but this phenomenon occurs at each crossover too.
There are not many rigorous studies done on the audibility of group delay vs frequency, but the little data that is available suggests a group delay with a duration less than the time it takes for one to two cycles of a frequency to occur is "pretty safe".
So look at your group delay chart at 40hz. One cycle of 40hz will take (1/40)=0.025 or 25 milliseconds. Two cycles at 40hz will take twice as long, so 50ms. If your group delay at 40hz is less than ~25-50ms, then you should be okay.
The "pretty safe" amount of group delay decreases as the frequency increases. Also, more exotic bass enclosures (such as ported) usually have more group delay than simple boxes (such as sealed), like you just learned.