Pennard Castle has a particularly beautiful location on the south side of the Gower Peninsula, at the edge of the grassy dunes of Pennard Burrows, right on the rim of the steep-sided valley containing Pennard Pill, a slow-moving stream that meanders lazily towards Three Cliffs Bay, meeting the sea half a mile south. There are no trees to block the panoramic views, and the place has a distinct feeling of isolation, not near any recent development apart from the local golf course, which now occupies most of the Burrows.
The castle was never very large, but although much has disappeared, the remaining sections - the gatehouse, parts of the curtain walls and the remains of two towers - are still substantial enough to give a good idea of the original layout. The first fort at this location, in the 12th century, was a timber structure, later replaced by stone, and occupied until the start of the 15th century, when the place was abandoned, in part due to problems caused by shifting sands. A similar fate befell the medieval church of St Mary's, the low foundations of which can be seen a few hundred feet east of the castle. There were also several other small buildings in the vicinity, from the associated settlement, but all the population was forced to move further inland, beyond the edge of the sands, where the modern village of Pennard is situated.
The ruins can be reached by a half mile walk from several places in Pennard, including at the golf club entrance and along a narrow lane at the northeast side of the course, but the easiest start point is probably the Gower Heritage Centre, along the A4118. A path starting on the opposite side of the road runs through a patch of woodland, crosses Pennard Pill on a footbridge and heads south along the valley floor, coming below the castle, which is reached by a short, steep, sandy connecting path. Another route to the ruins climbs the wooded slope near the footbridge then leads along the edge of the golf course.
The castle site was chosen since a block of limestone is exposed at this location, providing a stable foundation in the otherwise soft, sandy terrain. Although not directly on the coast, the land is high enough to overlook a sizeable area of the Bristol Channel. The deep valley of Pennard Pill affords natural protection to the west, while the eastern approaches were over open grassland, which allowed plenty of warning of approaching visitors. The timber fort was built in the early 12th century by Henry de Beaumont, Earl of Warwick (died 1119), one of the leaders of the Norman expansion across south Wales. The first stone building, a hall, was constructed at the centre of the wooden surrounds in the 13th century, followed a few decades later by a sturdy curtain wall and a relatively large gatehouse; all the masonry used a mix of the local white limestone and grey/red sandstone. The castle at this time was owned by the de Braose family, lords of Gower and Bramber; it later passed between the Mowbray, Despenser and Beauchamp families. Abandonment occurred sometime in the early 15th century, though the fabric remained intact for another 300 years, until several sections collapsed in the 1700s. Limited repairs were carried out in the 20th century, including shoring up some parts of the structure using concrete.
Pennard Castle occupies an approximately oval-shaped enclosure, with the gatehouse on the east side, facing the Burrows; this had a portcullised doorway between two half-circular, three story towers, with a further room above the entrance passageway. A reasonable amount of the north tower survives but the south tower is mostly missing, having collapsed in Victorian times. Only a 20 foot-long segment of the south curtain wall remains, yet all of the north wall is still standing, up to 15 feet tall, linking the gatehouse with the small, circular, northeast mural tower, which contains a single arrowslit, and adjoins the larger, rectangular west tower, probably used for additional living space. There is no trace of the great hall that once stood near the centre of the site.