This follows a letter written to Rishi Sunak from businesses in the retirement and housing markets, which called for a Help to Downsize scheme for retirees.
More than half (51%) of downsizers aged over 55 said the most challenging part of moving was finding the right property to move into; 11% wished they had made the move sooner.
Almost two-fifths (38%) downsized to find a more suitable property for their changing needs, 30% because they did not need the space any more, 19% to relocate to a better location, and 17% to move closer to friends and family.
36% wanted to see more suitable housing options for older people, and 34% recommended a reduction in Stamp Duty.
Nick Sanderson, CEO of Audley Group, said: “People want to live in homes which match their changing needs, but many either struggle to find the right property, or struggle to make the numbers add up.
“This is a problem, and one that has been made more evident by the devastating impact of coronavirus over the last three months.
“Promised radical housing and planning reforms must rid the government of its laser focus on first time buyers.
“Instead it should tackle the chronic shortage of properties which help people stay independent, out of hospital, out of care homes, with access to support and care as they get older.
“A report from CASS Business School this month found that we will have 20 million surplus bedrooms in this country by 2040, many in houses owned by people who would like to downsize.
“It’s abundantly clear then that the focus should be on specialist housing.
“Hand in hand with increasing supply should come incentives, like stamp duty relief or help to move packages, for people moving out of family homes into properties that meet their needs and lifestyle.
“The government has a small window to make a real difference and improve the lives of the population, while simultaneously easing pressure on the health and social care system and stimulating the housing market.
“Now is the time to take this opportunity with both hands.”