My uninformed opinion is that the reason ORF is somewhat subpar is because of PHF. We really don't have the population to support two competing airports. RIC is passing ORF because it's the only major airport in RIC. Imagine if all the flights from PHF and ORF came out of the same facility... Also Southwest cut wayback in Norfolk after they acquired Airtran, I'm assuming because they had a bunch of Airtran flights at PHF.
Check out these stats from 2014:
ORF 1488114
PHF 259082
PHF and ORF = 1747196
RIC 1671096
So build a second runway or relocate ORF. Close PHF. Become a hub to some airline. Done deal.
I know a lot of folks that are driving to RIC from Hampton Roads to fly due to cheaper flights and more availability, myself included. Tempted to not use Southwest anymore since they're making things more of a pain for me locally.