A Manus Island refugee says closing the detention centre is a public relations exercise that will leave men destitute in Papua New Guinea.
The Australian run centre is being gradually closed with 900 residents due to be evicted by October 31.
About 200 asylum seekers face deportation while about 700 refugees must consider relocation to the Transit Centre in nearby Lorengau town.
Iranian refugee Amir Taghinia said the Transit Centre only had beds for 280, meaning the majority of refugees would be stranded.
"They are trying to close this place so they can say Manus Island detention centre is closed and stop people protesting in Australia," he said.
"They are trying to keep people quiet."
Refugees would also be expected to settle in the PNG community it was announced this week when officials revealed plans for the closure.
However Mr Taghinia said most of the men had become mentally ill from years of imprisonment and were too afraid to leave the centre.
The men had been allowed out since April 2016, when the PNG supreme court ruled the centre was illegal, although only a few made regular day-trips to Lorengau.
"Even though the centre was announced open they have stayed in their rooms ... and they have not been able to leave because of the feeling of anxiety and depression," said Mr Taghinia.
"Their mental health has been affected badly. Most of the people are institutionalised because they were kept in this centre for a long time."
Documents leaked to Guardian this week revealed persistent self harm and suicide attempts by detainees requiring guards to carry 'cut-down knives' to avert hangings.
The documents also showed that companies running the detention centre had deliberately degraded living conditions to coerce the men into leaving.
Fear of conflict with Lorengau locals also deterred refugees from moving, according to Mr Taghinia.
He said the few refugees who had already gone to the Transit Centre regularly encountered such violence.
"We get reports of them being attacked," said Mr Taghinia. "People get robbed everyday.
"People are being assaulted and insulted and we have witnessed those things with our own eyes," he said.
"So people are certain that once they've been relocated to the Transit Centre they are going to be beaten-up in the Lorengau town, they are going to be robbed."
Only about 40 refugees from Manus have tried to settle in PNG including Loghman Sawari, who was deported from Fiji after trying to escape intimidation and violent attacks from PNG locals
Others like Masoud Ali Shiekhi who moved to the Transit Centre, had also been physically assaulted despite trying to integrate with the Manus community.
Manus MP Ronny Knight said his constituents felt intimidated by the hundreds of male refugees forced onto their island.
Kurdish Journalist and detainee Behrouz Boochani warned of more violence if refugees were forced to settle in PNG.