Alexander Ring tapped in a 59th-minute winner after a poor error by Columbus at Red Bull Arena.
City moved up to 10th in the Eastern Conference with just its second win in seven MLS games, while Columbus remains top by a point despite their first loss.
Eight minutes before the opener, City went close through Alexandru Mitrita, whose long-range effort hit the crossbar.
But a terrible mistake by Aboubacar Keita handed New York City the opener – and winner.
Keita's pass across defence was easily intercepted by Heber, who cutback for Ring to tap in the easy winner.