Fort Taylor covers a total of 54 acres and is a National Historic Landmark that played a significant role in Florida's early development. Construction of the fort began in 1845, shortly after Florida became a state. In 1850, the fortress was named after U.S. President Zachary Taylor, who died in office earlier that year. Throughout the 1850s, construction on Fort Taylor was slow. Yellow fever, shortages of material and men, remoteness, and hurricanes had slowed down progress.

Prior to the start of the Civil War, Capt. John Brannan occupied the fortress, placing it in Union hands. The main role of Fort Taylor during the Civil War was to serve as headquarters for the U.S. Navy's East Gulf Coast blockade squadron. This squadron deterred numerous supply ships from reaching and leaving Confederate ports in the Gulf of Mexico. While Fort Taylor never saw any hostile action, many historians have argued that because of Fort Taylor and its mission, the War of Rebellion was not allowed to continue any longer than it did.

Yellow Fever once killed as many as 15 people a day at the fort. Over the years thousands of people died in the fort. Visitors have reported seeing soldiers line up in formation while hearing whistles and gunshots. Downstairs in the fort there are very cold spots that should be impossible in the tropics. Several executions did take place here and every day at noon it is said you can hear the trap door open when some one is hung. Screams and pleas for mercy have also been heard to come from the jail area. It is said that there are mass graves of yellow fever victims in under the forts parade ground. A young girl's ghost with severe burns is often seen near the old hospital area.