Wally Downes' side, which sits bottom of League One and is eight points from safety, inflicted one of the biggest cup shocks on the Hammers in recent years thanks to a superb performance at Kingsmeadow.
Wimbledon deservedly went ahead in the 34th minute when Kwesi Appiah converted coolly from six yards out, and it was 2-0 before half-time thanks to a slick finish from Scott Wagstaff, who had been allowed to stride unchallenged towards goal.
Manuel Pellegrini made three changes at the break, with Felipe Anderson among those introduced, but Wagstaff flicked Appiah's cross into the bottom-right corner to make it 3-0 less than a minute after the restart.
Lucas Perez blasted home 57 minutes in as West Ham at last began to build some sustained attacking threat, and when Felipe Anderson's curling free-kick beat goalkeeper Aaron Ramsdale at his near post, the comeback looked to be on.
However, a clever passing move down the left allowed Anthony Wordsworth to cross for Toby Sibbick to head in a fourth in the 88th minute and send a jubilant Wimbledon into the fifth-round draw.