Atchison was indeed president pro tem of the Senate -- third in line to the presidency in those days -- in the outgoing 30th Congress, which came to an end on March 3, and then selected for the same position in the 31st Congress, which also was sworn in March 5, but that timing is irrelevant to the presidency. Zachary Taylor had been elected to a term that began at midnight March 4, 1849, and so he was president that day. He didn't want to take the oath of office on the Lord's Day, however, and thus did so the next day.
The issue had come up before 1849, at the end of James Monroe's first term in 1821. March 4, 1821, was a Sunday, and it was wondered whether that meant a presidential interregnum of one day if Monroe also honored the Sabbath by not taking the oath that day. Secretary of State John Quincy Adams asked the Supreme Court for a dictum on the matter, and Chief Justice Marshall wrote to Adams: "As the Constitution only provides that the President shall take the oath it prescribes 'before he enters on the execution of his office,' and as the law is silent on the subject, the time seems to be in some measure at the discretion of that high officer. There is an obvious propriety in taking the oath as soon as it can conveniently be taken, and thereby shortening the interval in which the executive power is suspended. But some interval is inevitable. The time of the actual President will expire, and that of the President-elect commence at 12 in the night of the 3rd of March. It has been usual to take the oath at midday on the 4th. Thus, there has been uniformly and voluntarily and interval of twelve hours during which executive power could not be exercised... Undoubtedly on any pressing emergency the President might take the oath in the first hour of the 4th of March... If any circumstance should render it unfit to take the oath on the 4th of March, and the public business would sustain no injury by its being deferred till the 5th, no impropriety is perceived in deferring it till the 5th. Whether the fact that the 4th of March comes this year on Sunday be such a circumstance may, perhaps, depend very much on public opinion and feeling..."
Note that Marshall said that executive power is suspended, not that the presidency is vacant just because the oath of office is delayed. The president is the president, even if he can't exercise his powers until that simple ceremony is over. If Chief Justice Marshall said it, that ought to settle it. One of the remarkable things about the presidency is that it has never been vacant in more than two centuries.