The former RB Leipzig boss has been out of work since leaving his role as Red Bull's head of sport and development last year.
Rangnick had been in talks with Schalke, where he previously spent two spells as head coach, over returning to Veltins-Arena in a boardroom position.
"Schalke is very close to my heart," Rangnick said in a statement on Saturday.
"The overwhelmingly positive reactions from members of the Konigsblauen community have influenced me and confirmed my feelings for this very special club.
"I would have loved to have been involved in helping Schalke on its difficult road back to its former strength.
"Unfortunately, due to the numerous uncertainties within the club at the moment, I do not see myself being able to take on sporting responsibility at S04 at the moment.
"My wish for all members and supporters of the club is that they will successfully join forces to unify Schalke on and off the pitch and lead it back to the top."
Rangnick's rejection comes just two days after Schalke board members Jens Buchta and Peter Lange revealed they had held positive talks with the ex-Hoffenheim coach.
And Saturday's announcement came as a shock to Schalke, who are 11 points adrift at the bottom of the Bundesliga standings with nine games to play.
"We are surprised by Rangnick's rejection and especially the short notice," Buchta, chairman of the club's supervisory board, said in a statement.
"The board was determined to reach an agreement with Ralf Rangnick in a second round [of talks] agreed for the coming week. It is regrettable that this will not happen.
"At the same time, the supervisory board has a certain understanding of Rangnick's decision in view of the events of the last 10 days.
"The supervisory board will now strive for the position of sports director to be filled as quickly as possible."
Rangnick is considered one of the frontrunners to take over as Germany head coach after Low announced last week he will step aside after the rescheduled Euro 2020 tournament.