The most famous -- and damaging -- attack was of course when a British force swept into town in 1814 and set fire to the recently completed presidential residence, among other public buildings. But there have been other, less destructive attacks during the two centuries since then.
In August 1841, President Tyler vetoed a bill to re-establish the Bank of the United States, inspiring a Whig riot outside the White House. The Whigs, ostensibly Tyler's own party, felt betrayed and, according to History.com, "The rioters hurled stones at the White House, shot guns into the air and hung an effigy of the president that they then set on fire. The protest is considered one of, if not the most violent demonstration held near the White House. As a result of the unrest, the District of Columbia decided to create its own police force."
In two separate incidents in 1974, a U.S. Army private in a stolen helicopter landed on the White House grounds (and was arrested) and a man crashed his car through one of the White House gates, holding off police for a few hours by claiming to have explosives strapped to his body (he too was then arrested).
In 1984, a man waved a samurai sword outside the White House grounds, another one brandished a shotgun on the sidewalk outside the property -- and was shot to death for his trouble -- and yet another climbed the fence surrounding the property.
In 1994 -- for some reason -- all kinds of violence broke loose near the executive mansion: one Frank E. Corder stole a Cessna and crashed it on the White House lawn in an apparent suicide; Francisco M. Duran let loose rounds from a semiautomatic weapon, putting 11 holes in the White House facade; and four shots were fired at the rear of the building by an unknown shooter.
But that's not all. Though not attacks, strictly speaking, over the years other people have tried to sneak past the Secret Service guards and wander around the grounds, and crash official functions. The White House is a popular place.