Forest led inside two minutes when Sam Byram barrelled down the right flank to send in a cross former Newcastle striker Daryl Murphy nodded in.
It proved a frustrating outing for the much-changed visitor at the City Ground, after boss Rafael Benitez saw his defensive tactics in the 2-1 weekend defeat to Chelsea come in for criticism.
Close-season signing Salomon Rondon sharply found the bottom corner to seemingly secure a penalty shootout in stoppage time, only for Newcastle to collapse in the closing minutes.
Substitute Matthew Cash profited from Karl Darlow’s inability to hold a cross and, after the officials decided not to punish Forest keeper Luke Steele for a challenge on Ayoze Perez in the box, Gil Dias pounced on a loose pass to chip home and seal a place in round three for Aitor Karanka's men.
Meanwhile, Everton saw off Championship opposition in the form of Rotherham United by the same scoreline, with Sandro Ramirez laying on Gylfi Sigurdsson's opener ahead of a rumoured return to Spain with Real Sociedad.
Dominic Calvert-Lewin helped himself to a second-half brace – the Everton youngster's second snuffing out the hope granted to Rotherham by sub Will Vaulks' goal four minutes from time.
Isaac Success set Watford on the way to a 2-0 win at Reading, where teenage midfielder Domingos Quina scored a marvellous long-range second to preserve the Hornets' 100 per cent start to the season.
Elsewhere, Millwall saw off Plymouth Argyle 3-2 despite twice going behind, with Aiden O'Brien turning in a late winner at the Den.