The London-based oil giant accused the court-appointed administrator for the settlement, Patrick Juneau, of trying to rewrite the terms of the deal. BP said Juneau violated the settlement in the way he used a complex formula to determine the payments to businesses.
Last week, BP warned investors that the settlement's price tag will be "significantly higher" than initially estimated.
"Although the ultimate exposure is at this time inestimable, it grows daily and could cost BP billions," the company's lawyers wrote Friday.
U.S District Judge Carl Barbier appointed Juneau and has upheld his decisions for calculating payments. Juneau's spokesman declined to comment on BP's lawsuit.
Attorneys who worked on the class-action settlement with BP said the payments to businesses were spelled out in the agreement.
"Simply put, BP undervalued the settlement and underestimated the number of people and businesses that qualify under the objective formulas that BP agreed to," attorneys Steve Herman and Jim Roy said in a statement.
BP said Juneau made decisions in January that expose the company to fictitious losses that were never contemplated in the settlement.
Friday's court filing asked Barbier to block payments to any businesses whose awards are part of the January decisions. As an alternative, BP asked to block payments to businesses in certain industries, including agriculture, construction, professional services, real estate, manufacturing and retail.
Before Barbier ruled last week, BP had argued that Juneau's interpretation of the settlement would lead to "absurd results" and "false positives." The judge said the settlement anticipated that "such results would sometimes occur."
"Objective formulas, the possibility of 'false positives,' and giving claimants flexibility to choose the most favorable time periods are all consequences BP accepted when it decided to buy peace through a global, class-wide resolution," Barbier wrote.
BP estimated a year ago that it would spend roughly $7.8 billion to resolve tens of thousands of claims covered by the settlement. It revised its estimate earlier this year, saying it expected to pay $8.5 billion, but now says it can't give a reliable estimate.
Barbier also is presiding over a trial designed to determine the causes of BP's April 2010 well blowout and assign percentages of fault to the companies involved in the disaster, which killed 11 workers and spawned the nation's worst offshore oil spill. The trial will enter its fourth week Monday.