The report is produced by the Housing and Land Rights Network pressure group which says it used official documents obtained under India's right to information act.
The group says there should be an independent inquiry into how this was allowed to happen. The government says it is looking into the allegations.
The report says tens of millions of dollars have been diverted from funds which are supposed to help raise underprivileged low caste communities out of poverty, the BBC reports.
It also says spending on the Games has spiralled out of control: expenditure on sports infrastructure alone is more than 2,000% of the initial projected budget.
In addition, more than 100,000 poor families have already been evicted due to projects connected with the Games, and up to 40,000 families are likely to be displaced before the Games begin in October, the document says.