There was a substantial rail yard at Fitzroy, and originally there may have been a station building there. The yard mostly handled briquette and later wood traffic and in the end a box factory. The yard was bounded by Napier Street, Jamieson Street/Alfred Crescent and Queens Parade.
Part of the park was also the yard itself. The remainder of the yard is now a housing development. There are people around (on another forum) who actually drove a B Class diesel down to that yard.
This is a good photo of it taken back in the thirties: http://handle.slv.vic.gov.au/10381/20700
You can see the yard at the bottom left of the photo, with the existing Fitzroy Football ground prominent in the photo. The long light coloured shed was the briquette depot and you can see a steam locomotive in the bottom left.
If you have Google Earth, then go to the northern end of the gardens and where the old track crosses Alfred Crescent, place a Streetview marker at that point and you will see the old rails still in place, embedded on the road.