The earliest Castle at Duffus existed by 1151 and was long a principal seat of the De Moravia family. We believe that this castle had only timber buildings, on earthworks used for the later stone defences which survive in ruins.

The 14th century tower has broken and slipped on the made-up ground. The site was encircled by a wet ditch, still crossed by an ancient bridge on the east. The castle is now cared for by Historic Scotland.