The CBF said the outbreak, which has hit Brazil especially hard, made it difficult to secure the public and private funding and other commitments needed to mount a competitive bid.
"Because of fiscal and economic austerity measures taken to respond to the impact of the COVID-19 pandemic, (the Brazilian government) indicated it would not be advisable at this time to sign the guarantees requested by FIFA," the CBF said in a statement Monday.
Its withdrawal leaves Colombia, Japan and a joint bid from Australia and New Zealand in a three-way race to host the tournament.
FIFA is expected to announce the decision on June 25.
South America has never hosted the tournament. The CBF said it believed a "single candidacy will increase South America's chances."
Brazil's economy has taken a beating as the country has emerged as the latest epicenter in the pandemic, with the World Bank forecasting a painful eight-percent recession this year.
The country has recorded 37,134 COVID-19 deaths so far, the third-highest toll in the world, after the United States and Britain.