Defender Levin Oztunali gave Mainz the lead when he beat Dortmund goalkeeper Roman Burki from 20 metres just after half-time.
Belgium defender Thomas Meunier equalised, but Dortmund's 16-year-old striker Youssoufa Moukoko deserves credit for keeping the ball alive in the area and providing the crucial pass.
But Reus fired his spot-kick wide after a foul on Meunier in the area, leaving Dortmund in fourth when it could have moved into third above Bayer Leverkusen after its loss to Union Berlin on Saturday (AEDT).
"I could have decided the game – I am very, very sorry," a crestfallen Reus, who buried his face in his shirt after his miss 15 minutes before the final whistle, said.
"The disappointment is huge – we wanted to climb the table and we've missed the chance to take another step."