Michy Batshuayi's equaliser had seemingly done enough to force extra-time against a Watford side that was poor in the second half, but Gray found the net 11 minutes from the end to send the host to Wembley.
Watford generally controlled proceedings in the first half and was deserving of its 1-0 lead at the break courtesy of Etienne Capoue, after a goalkeeping error from Vicente Guaita.
Palace rallied after the interval and equalised through Batshuayi, but Gray, who scored the winners in each of Watford's past two home matches, prodded home shortly after his introduction to knock Roy Hodgson's men out of the FA Cup.
After a low-key start, Watford made the breakthrough in the 27th minute as Guaita failed to claim a corner, Craig Cathcart subsequently setting up Capoue to finish into an empty net.
Guaita went some way towards making amends a few moments later, producing a brilliant save to keep Gerard Deulofeu's fierce strike out of the top-right corner.
Palace looked re-energised early in the second half, Max Meyer forcing Heurelho Gomes into a fine save with a header early on, before an Andros Townsend free-kick soon after was tipped over the crossbar.
Batshuayi increasingly became a threat and, a few moments after seeing a stinging drive go just wide, he levelled, charging down Adrian Mariappa's clearance and clinically finding the bottom-right corner.
But Watford had the last laugh, confirming its trip to Wembley as Gray latched on to Roberto Pereyra's inch-perfect pass in behind Palace's defence and poked beyond the helpless Guaita.