MISSED the match? Catch up via beIN SPORTS CONNECT | 2-WEEK free trial
In front of its own fans, LAFC took the lead through Kellyn Acosta's deflected free-kick but was pegged back just before the hour when Daniel Gazdag scored his 24th goal of the season.
Jesus David Murillo headed LAFC back into the lead in the 83rd minute only for Jack Elliott to respond two minutes later for the visiting side to send the game into extra-time.
English centre-back Elliott appeared to have won the title for Philadelphia when he struck from close range deep into extra-time, after LAFC goalkeeper Maxime Crepeau was sent off having suffered what looked like a serious leg injury that would have ended his involvement anyway, but substitute Bale headed the hosts level in the eighth minute of stoppage-time to force the shootout.
Then the drama went up another notch, with McCarthy, a Philadelphia-born former Union goalkeeper, pulling off saves from Jose Martinez and Kai Wagner, after Gazdag haplessly missed when he slipped as he struck his spot-kick.
Ilie Sanchez tucked away the clinching penalty in a 3-0 shootout triumph, sparking wild celebrations of the team's first MLS Cup triumph, with McCarthy named MVP.
The opening goal in the 28th minute owed plenty to fortune, with Acosta's 22-yard strike flicking off the head of 19-year-old Union midfielder Jack McGlynn and finding the bottom-left corner.
This was the first time since 2003 an MLS Cup has featured the teams that came top in the West and the East.
Both sides finished the regular season with 67 points, but LAFC took the Supporters' Shield due to having more wins than Philadelphia, and that earned it home advantage.
The game's first equaliser arrived in the 59th minute when what looked like a wildly ambitious 30-yard shot from Jose Martinez inadvertently turned into a through ball for Gazdag, who controlled the fizzing low ball well before firing past Crepeau.
Murillo headed LAFC in front again with seven minutes remaining as he powered home Carlos Vela's in-swinging corner from the right.
However, a terrific equaliser from Elliott, a fierce header after a free-kick from Kai Wagner on the left, brought Philadelphia level in the 85th minute.
Bale entered the fray in the seventh minute of extra time, replacing Vela.
Philadelphia then looked to have been dealt an outrageous injustice when Cory Burke raced through on goal and was scythed down by Crepeau, who was initially only yellow-carded for a clear professional foul, but it was soon upgraded to red.
Crepeau's injury appeared serious and had to be carried off.
The incident allowed for the introduction of McCarthy, who proved to be the difference.
