NEW ALBANY, Ind. (WHAS) -- For five years, an off-white brick building off Erni Avenue in New Albany, Indiana, has been home to Brandon Brown.

"I had no other place to go," he said. "I would have been homeless."

Lately, Brown has been having trouble sleeping, kept awake with worries about a new plan proposed by the city of New Albany that could lead to the demolition of his home and many others around him.

"This is not what we asked for," he said. "This is not what we asked for."

"If somebody's paying money to live here, their opinion should override somebody that's just working here or someone on a council or something," James Walker, the boyfriend of a Parkview Tower resident, said.

The city's plan would call for the demolition of more than 600 units, including those in Parkview Tower, Parkview Terrace, Vance Courts and Riverside Terrace, affecting around 1,500 people living in those at this time. The plan would also call for rehabilitation of certain units and would convert some others into project-based Section 8 housing.

"As a resident of New Albany, we need to get a clearer understanding of what's really happening or what's happening next," Brown said.

The New Albany Housing Authority has sent out letters to Brown and other residents, encouraging them not to move out of their homes at this time. The letter states that any action would require the Housing Authority to involve all residents before, during and after the process, but residents like Brown are still worried about their futures.

"There really ain't a whole lot of options out there," Brown said.

"There are other people who may not have family," Walker said.

The New Albany Housing Authority was expected to vote on a memorandum of understanding Monday evening, but according to the city, the executive director of the NAHA requested that morning for the vote to be delayed until the April 10 meeting, following feedback from residents concerning the proposal.

The memorandum of understanding is not legally binding, but it is a formal agreement between the city and the Housing Authority about their intention to move forward.