Former Federal Housing Finance Agency Director Mark Calabria said Thursday that if the FHFA and Treasury were to pick up the effort from where he and previous Treasury Secretary Steve Mnuchin left off during the first Trump administration, it could happen in a couple years or so.

"From everything that we did, and what I believe needs to be finished, it can be done," Calabria, senior advisor at the Cato Institute and author of a book on the pandemic-era FHFA, said Thursday, expanding on remarks he made at a Commercial Real Estate Finance Council event this week.

He acknowledged a lot of stars would have to be aligned and logistical issues would have to be addressed including the asset-liability imbalances the government-sponsored enterprises have and the Treasury's senior preferred shares, but said there are mechanisms for handling them.

"You have a huge overhang on the liability side of the balance sheet, and that needs to be crammed down, so you either cram it down explicitly, or you essentially swap everything to common stock and let the market cram it down," he said.

Calabria favors the second approach.

"Ultimately, I think it's better to let the equity markets determine what that value is," he said.

But even if there's a second run of the Trump administration, Calabria said it's not a given that the exit will be a top priority given that tax reform would likely take precedence for the Treasury.

Conservatorship may be even far less likely if President Biden stays in office, but Calabria said he wouldn't rule it out even in that circumstance.

That said, if Janet Yellen remains Treasury secretary, it's unlikely based on her testimony and actions, he speculated.

It would be more likely if Mnuchin were to return to the post, Calabria said. Mnuchin told CNBC on Thursday that he would consider serving again if Trump is elected.

For his part, Calabria said he doesn't think he necessarily needs to be involved for an exit to happen.

"I have a high degree of comfort that there are a number of people that can come in, in a different administration, and complete the work I started. So I don't necessarily feel like 'only Mark can do it.' I have a sense that the groundwork has been laid," he said.

However, Calabria added that he would consider serving if called upon.

"If I was asked to serve, it would be hard for me to say no to a president that wants me to have some sort of positive impact," he said.

But he stressed that his interest in such a position has changed over time.

"I would certainly admit, to a degree, that while at the White House in 2018, I threw my hat in the ring for the FHFA job and thought it was something I could do. I'm not out there throwing my hat in the ring and saying, 'me, me, me,' because I don't think it needs to be me," Calabria said.

Another possible factor in whether or how Fannie and Freddie exit conservatorship could be the strength of the single- and multifamily housing markets, he said. 

Signs of weakness seen in apartment market indicators like delinquencies, starts and rents could have equivalents that materialize soon in single-family homes too, according to Calabria, who expects to see trends in the two markets converge.

He thinks that elevated debt-to-income ratios, particularly in the Federal Housing Administration-insured market, could be a concern given that both high loan-to-values and low FICO credit scores are simultaneously allowed for those loans.

While the mortgage market often watches the FHA market for signs of broader credit issues, Calabria said the signs may be less visible due to "FICO inflation" from temporary pandemic relief for consumers and related policies. Restrictions on reporting delinquencies in credit reports, and loss mitigation initiatives during the COVID-era "kick the can down the road" regarding decisions about whether a loan needs to proceed to foreclosure, he said.

He noted that pandemic specific assistance drawn up during his time at the FHFA "was never meant to be permanent" although there was some thought it could inform what might be done in the future or potentially "put on the shelf" for potential use in another emergency.

Calabria agrees with others who say credit concerns now emerging aren't on the order of the Great Recession, which was preceded by a period where underwriting was particularly loose and foreclosure prevention was far less efficient, but said they shouldn't be ignored.

"For people who say, 'well, it's not going to be 2008, I ask, 'would half of 2008 be pleasant?' My annoyance with that general kind of commentary is sometimes what they're saying is we shouldn't care at all, and I think that's ridiculous. You don't need 2008 to care about the market."

Calabria also said he is concerned about disaster risk and the related rise in insurance costs to a degree, noting that it's a circumstance that arises in part from consumer preferences for coastal living.

"You don't have to be a climate alarmist, but set that debate aside," he said. The reality is that more of us continue to live by the ocean. You don't even really need to get some of the broader debates of climate, this is a real issue regardless, because of where we choose to live."