LOUISVILLE, Ky. (WHAS11) -- In 1954, Louisville was still a black and white world. We were separated.

The owner of Louisville's Legends Gym James Doolin remembers, "When I was a kid, you could walk up to a water fountain and it was whites only or blacks only."

Signs that pointed to "whites only" or "colored only" hung across the city.

At 9-years-old, with a fascination of boxing, Doolin, like a lot of Louisville's young boys, was about to have his life changed and at the same time, Louisville was slowly changing.

One underground spot on 4th Street, south of Broadway, was already ahead of its time in the nation, and in Louisville.

The separation of blacks and whites was not happening at the Louisville Service Club building, which was home to the Columbia Gym. The gym and the building were owned by the City of Louisville. The gym was operated by the Louisville Parks Department and run by a

Louisville Police Officer named Joe Martin.

Doolin said Martin was ahead of his time, "During that time in the city, blacks and whites couldn't train together. Joe Martin put a stop to all that."

The head librarian and archivist, Jackie Young, at Spalding University, which now owns the building, said, "This is also part of where it all began as inclusiveness, as a city and accepting everyone together. That vision of Joe Martin: have a spot for everyone."

Louisville's boxing scene was setting the tone for the city's early civil rights movement.

Doolin puts it in perspective, "Back then boxing was big in Louisville. Big! It was the number one boxing town in America, ahead of Chicago, LA and New York."

Across the city, boxing gyms were tucked into the corners of buildings everywhere.

Doolin remembers walking into Columbia, "Sounded like speed bag going off. All boxing gyms smell of sweat, gloves and everybody working out."

At the same time, a 12-year-old, at 89 pounds, was starting to
make his name.

"A skinny kid, tall. He was a comedian. He liked to talk and play games and you about him," Doolin said. "I didn't believe he'd make it, if you want to know the truth."

You can trace the very spot in Louisville where the legend began.

It was the summer of 1954, 12-year old-Cassius Clay is riding his brand new, $60 red bike along a 4th Street sidewalk with his brother and a friend. They park. Why are they here? They're heading into this building for what's called the Louisville Home Show.

The home show was also a segregated event but for African-Americans only. Jackie Young has researched the event that was underway.

"Cassius Clay came to attend the home show and see what was going on. He heard there was free food and snacks," Young said.

While inside, his bike was stolen. So he ran back into the building, running down the steps, looking for officer Martin.

Martin himself told WHAS TV's legendary Sports Director Cawood Ledford what happened next in this interview conducted in the early 1960's.

"So he was very disturbed about it and someone sent him downstairs to see me. I told him I would report the theft to police, but that if he really wanted to fight someone, he should come down and be taught properly," Martin said.

Clay's boxing career would begin thanks to the theft of his bike. He was also in the right building at the right time, meeting the right person.

The legend of course grew into the greatest.

But what happened to the Columbia gym? We found some national writers have reported that it's "a parking lot", or one a few years ago who wrote, it's now "a basketball court". All wrong.

There are few photographs existing of the gym, which opened in the late '20s and was still there in 1963 when the sisters of Nazareth bought the building for Nazareth College.

Our search led to one striking view unearthed by the UofL archives and special collection department. It shows a smoky, hazy scene of an actual fight underway in 938, 16 years before Clay would walk inside.

Doolin remembers the smell of the place to this day.

"It was very old, musty, it was like basement," Doolin said.

Even today, when you tour the building now operated as Spalding , very little has changed.

The sidewalk is still there. The original steps that he ran down to find Martin, as still there.

But in 1963 the place was purchased by the nuns of the Nazareth. Did a boxing gym fit into their mission? Not so says archivist Jackie Young.

"They wanted a bookstore. So they just eliminated the gym because most of their students didn't box," Young said.

And now in 2015, "We have it now as our student center, the exact space that was once the Columbia Gym," Young said.

The university now honors the space with a famous quote from Muhammad Ali about how he overcame his hate for training: a lasting tribute to the gym that launched the greatest. It says: "I hated every minute of training, but I said 'don't quit. Suffer now and live the rest of your life as a champion".

Doolin shakes his head and simply says, think about it without the Columbia Gym, "There wouldn't be anywhere he could have parked his bicycle, nowhere to walk down the steps, nowhere to learn to box you know."