Pontypridd-based builder Matthew Davies, a spokesperson for the Home Builders Federation, said: “The reason generally that there is a big gap is due to the planning system.
“There is land availability to meet the target, but the planning system needs a complete overhaul eventually.
“A planning application is taking between three and five years. That needs to be reduced to one to two years in order to achieve this target.”
Wales understands the government will publish more details about how it will measure progress towards the 20,000-homes target early next week.
The target was a manifesto commitment by Plaid Cymru at the Senedd election, and is the same target Labour had in its last five-year term in government.
Plaid has promised to hit it within the next four years as well as to lower regulations required to build and buy social homes.
Social homes are provided by councils and housing associations. Their rents and the quality of housing are controlled by the government, which also funds new building.
More than £2bn of Welsh government money was spent on social housing in the last Senedd, with rising prices and higher building standards pushing up costs.
The government grant required for a home rose 75% from £93,000 to £163,000 in four years, spending watchdog Audit Wales says.