Colony most likely was originally established soon after the emancipation of enslaved people after the Civil War, during Reconstruction. It is believed that the original settlers of the town were most likely freed enslaved people from Baltimore, Alabama. It is the only African American community in Cullman County, which was founded in 1873, and was considered a safe haven for Black people at a time when other cities with white majorities were dangerous for Black people, and referred to as 'sundown' cities because after sundown Black people were in danger there. Those that originally settled there were given land as compensation for being slaves, with over 8,000 acres eventually cultivated as farm land.
In the early days of Colony it was not considered part of Cullman County because the freed Black people were not welcome. But when coal mines began to open adjacent to Colony the leaders of Cullman County wanted to incorporate the coal mines into the county for economic reasons, and were forced to add Colony as well. Colony was incorporated as a town in 1981 during the tenure of Mayor Earlene Johnson. The town celebrates Colony Day annually in August.