The Norwegian Ruud, who took a medical timeout after the second set, succumbed 6-3, 7-5, 6-7 (4/7), 6-2 to the 39th-ranked Brooksby on Rod Laver Arena.
His early exit left the tournament without its top two seeds in week one after Nadal's shock loss on the same court Wednesday to another American in Mackenzie McDonald.
It opens the door to Novak Djokovic winning a 10th Melbourne title and temporarily derails Ruud's dreams of becoming world number one.
"First and foremost, Casper is a warrior, I knew it would be a great battle out there," said Brooksby, 22.
"I'm just really proud of my mental resolve after the third-set battle didn't go my way, and to turn it around.
"I thought I was playing really strong and I just didn't want to lose my focus out there."
The 24-year-old Ruud's defeat ended any hope he had of assuming the world number one ranking from Carlos Alcaraz, who is absent from Melbourne with an injury.
Only two players can now achieve the feat if they lift the trophy -- Serbian superstar Djokovic or Greek third seed Stefanos Tsitsipas.
According to the governing body ATP, Ruud could have done so by winning or simply reaching the final, as long as neither Djokovic nor Tsitsipas claimed the title.
That now won't happen in a blow for a player who enjoyed a breakthrough 2022, reaching the final of both the French and US Opens.
Brooksby, who is on his Australian Open debut, will face Tommy Paul next after his fellow American beat Spain's Alejandro Davidovich Fokina in five sets.
Brooksby, whose best Grand Slam result was reaching the round of 16 at the 2021 US Open, took charge in the first set, breaking twice, with the Norwegian hitting 15 unforced errors to slump behind.
Ruud, who was scheduled to meet Djokovic in the semi-finals, regrouped at the changeover and earned two break points on Brooksby's opening serve in set two, but he failed to convert.
They traded breaks through a gruelling second set before the American earned a set point at 5-6 and after a 26-shot rally moved two sets clear on a lucky net cord.
Ruud left the court for a medical timeout at the changeover, apparently for an abdominal problem.
His game didn't improve when he returned, broken two more times before staging a remarkable comeback.
He saved three match points at 3-5 as Brooksby became agitated, slamming his racquet on the ground, before taking it to a tiebreak where he broke early to take it to a fourth set.
But, undeterred, the gutsy American reset to break Ruud immediately and race to a 3-0 lead.