The first known large scale commemoration of Memorial Day took place in May 1865 in Charleston, SC organized by a group of freed black slaves. The event was called "Decoration Day" due to decorating the graves, which they had dug and buried the Union troops during the
previous two weeks. This image shows the graves of Union troops that died at the Race Course prison camp in Charleston, later Hampton Park. 
It was on 1 MAY of 1865, that the former slaves gave the fallen Union troops a day long funeral ceremony to honor their sacrifice. The event was documented in the Charleston Daily Courier as "Martyrs of the Race Course" after the ceremony.
The crowd was an estimated 10,000 people, mostly the recently freed slaves with some white missionaries. There were several black Union regiments, including the 54th Massachusetts which was the second such unit created in the Union Army commanded by Robert Gould Shaw.
Shaw was killed while commanding the 54th during the assault on Fort Wagoner, SC. This was depicted in the movie, "Glory" with Matthew Broderick, Denzel Washington, & Morgan Freeman. Great movie if you haven't seen it.
The black troops performed some drill and ceremony type activities, black pastors read Bible verses, and 3000 black school children carried bouquets of flowers and sang, "John Brown's Body." The ceremony began at 9:00AM on 1 MAY 1865.
Here is a clipping image of the article in the Charleston Daily News.
Here is an historical marker that commemorates the occasion.
The following year, on 5 MAY 1866, the first community-wide commemoration of Decoration Day took place in Waterloo, NY where businesses closed and residents decorated the graves of soldiers with flowers and flags.
The Decoration Day commemoration was originally to honor the fallen of the Civil War. It wasn't until WWI that it changed to honor the fallen of all wars. It did not become a national holiday until 1971.
The story of this event was lost to history for some time until it was found in some boxes at Harvard's Houghton Library after David Blight read about it in the uncatalogued writings of a Union soldier.
One of the reasons it was probably lost to history was Charleston was nearly destroyed after more than a year of shelling by the Union forces on top of the fears Sherman was on his way to burn the city to the ground prior to the CSA surrender & southerns weren't interested
in honoring the Union troops that had destroyed the city and after the city was rebuilt in the 1880s, it was probably a distant memory.
One last thing to add, for those wondering about the lyrics to "John Brown's Body" here they are. The most ironic thing is the the part about Harper's Ferry, where he was a traitor, but now referring to the south as traitors.
