Hmm that really is puzzling then. Probably something that when you figure it out you're going to have to smack yourself in the head for not seeing it sooner. I've done that a fair share of times
Try pinching off the air regulator hose and normally it should die then. It
almost seems possible that if it were wide open you could have an idle like that, but I don't know. I know my IACV is a POS and I am 90% sure my air reg is jammed open. Right now I have a clamp on the hose to keep a low smooth idle but when it isn't on there, it surges to like 2k rpm. And on a cold start it used to be up closer to 3k. But 5k seems like too much to me for it to just be a stuck regulator.
So basically you are boosting but not supplying enough energy out of the exhaust to maintain the boost level? Now I'm doubly confused because I had heard you can't just boost without engine load. And if you were surging shouldn't you read a positive pressure on your gage, not negative? Or at least it should be changing and not constant.
If you suspect the manifold gasket is bad you could direct a small amount of starter fluid around it and see if your engine speed varies.