Just to update. I finally fixed this problem a few weeks ago. There was a chunk of dirt stuck in the hard line that went between the intake manifold and brake booster. Vacuum to the booster was present but not very strong. I ended up putting a replacement tube in place and it fixed up my booster problem. Now I just have to troubleshoot a spongy pedal which I think will require bleeding of my master cylinder.