Clermont had not suffered a defeat since October, but it came unstuck against the champion at Stade Pierre-Fabre.
Castres duo Armand Batlle and Martin Laveau scored first-half tries before Peceli Yato's five-pointer reduced the deficit to 15-8 at the interval.
Remy Grosso crossed early in the second half to make it a two-point game, but Benjamin Urdapilleta took his tally from the tee to 11 points and Rory Kockott was on target with a penalty to give Castres breathing space after Greig Laidlaw split the posts at the other end.
Defeat for Franck Azema's men was just their third of the season and Toulouse can replace them at the summit with a win over lowly Grenoble on Monday (AEDT).
La Rochelle is six points adrift of the leader after an entertaining 27-25 win over Montpellier.
Jone Qovu's try six minutes from time brought the home side level and Ihaia West added the extras to win it, with Alexis Bales, Romain Sazy, Vincent Rattez also going over for La Rochelle.
Tries from Henry Immelman, Jacques du Plessis and Jan Serfontein were in vain for Montpellier, which is languishing in ninth spot after crashing out of the European Rugby Champions Cup.
Lyon is also out of Europe but responded to back-to-back Champions Cup losses with a 32-11 over Racing 92, Noa Nakaitaci, Pierre-Louis Barassi and Toby Arnold touching down.
Pau beat beleaguered bottom side Perpignan 30-24, while Bordeaux-Begles is up to fifth after a 25-17 success over struggling Agen.