Australia's Barty and America's Vandeweghe triumphed 3-6, 7-6 (7/2, 7-6 (8/6) in the fourth meeting between the teams this year.
It was a first Grand Slam title for Barty, who had reached the women's doubles finals at all four majors with Casey Dellacqua before her former partner retired.
She and Vandeweghe were in their first Grand Slam final together since teaming up for the first time at Indian Wells in March.
Barty and Vandeweghe, who upset top seeds Barbora Krejcikova and Katerina Siniakova en route to the final, saved two championship points in the 10th game of the second set to force the tiebreaker.
Babos and Mladenovic saved two match points in the 10th game of the final set as they forced the tiebreaker, giving themselves a third chance at 6-5 in the decider.
Barty and Vandeweghe saved it to earn a third match point of their own on Mladenovic's serve and the French player double faulted.