Lincoln's First Inauguration, March 4, 1861
From Washington to Franklin Roosevelt, no fewer than 27 presidents took the oath of office on that day (four did not because they were vice presidents who succeed presidents, but didn't win their own terms). Only 12 presidents have taken the oath on January 20 (FDR is in both categories and Ford didn't win his own term). Eventually, assuming the office endures and the Constitution isn't changed again, January 20 will overtake March 4, but it will be well into the 21st century before it does.
The March 4 inaugurations are rich in history. In 1801 and 1877, disputed elections almost meant the inauguration would have been delayed -- but it's never happened. William Henry Harrison famously gave a long, long speech on March 4, 1841, picked up a virus (presumably) and went down in history as a one-month president. Two of Lincoln's most famed speeches, the First and Second Inaugurals, were on March 4 four years apart. FDR's 1933 First Inaugural, which came on the very last March 4 swearing-in, is also rightly remembered as a rhetorical gem.