A week after the Hollywood heavyweight lawyer wanted the judge in Janice Dickinson’s defamation suit against him and Bill Cosby tossed off the case, Marty Singer now has made a rare step back and withdrawn the challenge. And a lot of that might have to do with the former America’s Next Top Model judge saying, “Too late, buddy.”

On Monday, Singer filed a curt notice dropping his efforts to get current Judge Debra Weintraub off the controversial case for being prejudicial against him (read it here). This latest move came four days after Dickinson’s attorneys challenged the Lavely & Singer partner’s November 30 effort with claims that he’d already used up his challenge card.

“Co-defendants Mr. Cosby and Mr. Singer are on the same side and their interests are aligned, as that term is used in Section 170.6(a)(4) of the Code of Civil Procedure,” says the December 3 opposition to Singer’s filing of late last month (read it here). Planning to counter, Singer formally was named a defendant in the initially May 20-filed complaint in an amended complaint by Dickinson on November 16. On June 11, Cosby successfully challenged the first L.A. Superior Court judge in the case. Dickinson made a similar challenge of her own on June 25, and that’s when the matter has assigned to Weintraub. “The instant case was brought after Mr. Singer made statements on behalf of Mr. Cosby defaming Ms. Dickinson. As such, Mr. Singer may not disqualify Judge Weintraub because Mr. Cosby had previously exercised a challenge,” adds the December 3 opposition to Singer’s desire to get a new judge.

Looks like Singer agreed with Lisa Bloom and Dickinson’s other lawyer and hence filed his recent notice. Late last week, Cosby’s ex-longtime attorney also agreed with another recent move in the case. On December 1, Cosby’s new lawyers moved to have Dickinson’s first amended complaint struck or dismissed because they say it shouldn’t have been filed before the actor’s previous motion to have the matter tossed out on free-speech grounds was ruled on. Singer, who ceased representing Cosby in late October, joined that motion on December 4.

On November 2, Weintraub ordered that Cosby and Singer sit for depositions in Dickinson’s allegation that statements calling her a “liar” were defamatory. The statements of November 2014 came after Dickinson said that she was raped by The Cosby Show actor in 1982.

Both the lawyer and the much-accused Cosby, who now has more than 50 women claiming they were sexually assaulted by him over the decades, were spared the depos mid last month. On November 12, the 2nd District Court of Appeals laid down a temporary stay on the November 19 and 23rd scheduled depositions of Cosby and Singer in the case. Both sides have since filed further briefs in the matter with Dickinson wanting the stay lifted and Cosby’s team wanting it keep in place and Weintraub’s ruling dropped. A decision from the Appeal Court is expected by the end of the month – around the same time another recent deposition by Cosby in a separate sexual assault related case could be made public.