An Alexander Callens own goal was cancelled out by Maxi Moralez, and a late Talles Magno winner sent NYCFC to its first ever appearance in the championship game, which will take place in Portland on Sunday.
Ronny Deila's New York side were missing star player Valentin Castellanos through suspension and struggled to impose themselves on the game until they were forced to do so when falling behind.
A nervy contest burst into life just after the hour mark as Philadelphia took a 63rd-minute lead. Daniel Gazdag whipped a searching ball into the box and New York City defender Callens diverted it into his own net as he tried to stop striker Kacper Przybylko from getting on the end of it.
NYCFC replied swiftly as Moralez fired in an equaliser just two minutes later. Maxime Chanot hit a shot from range that was saved by Matt Freese, but the ball eventually fell to Moralez, who made no mistake.
Nathan Harriel should have put Philadelphia back in front with four minutes remaining but sent his free header just over the bar, and the Union were made to regret it as NYCFC won it in the 89th minute.
Gudmundur Thorarinsson stole the ball from Olivier Mbaizo before putting it on a plate for substitute Magno to score the winner.