The Giants' season came to an end on Sunday with a 34-17 defeat to the Philadelphia Eagles, with New York finishing third in the NFC East with a 4-12 record for the campaign.
It proved the final straw for the Giants, who on Monday confirmed Shurmur had been relieved of his duties.
Over his two seasons as head coach, Shurmur managed a 9-23 record. His winning percentage in four years as coach with the Giants and Cleveland Browns ranks as the seventh lowest in NFL history.
Team president John Mara and chairman Steve Tisch confirmed the decision to sack Shurmur and retain Gettleman in a statement released on the team's website.
"Steve and I have had many extensive discussions about the state of the Giants," Mara said.
"This morning, we made the very difficult decision that it would be in the best interest of the franchise that we relieve Pat of his duties. The last three seasons have been extremely disappointing for the organisation and our fans.
"Pat has been a successful and highly-respected NFL coach for 21 years and he is not solely responsible for our record. But we came to the conclusion it is best to have a fresh start with the coaching staff.
"We very much appreciate how much Pat has done for this franchise. He is a man of character and integrity and the team has conducted itself with pride and professionalism."
Tisch added: "The last two seasons have been a continuation of what has been a very difficult and disappointing period for our franchise."
With Shurmur's dismissal, the Giants are now searching for their fourth head coach in the space of six seasons.