It is an oddity of an outpost, a die-hard vestige of imperial pride wedged between Spain and Africa. A rocky English refuge enshrouded with a whiff of controversy and romantic aura in equal measure. At the confluence of the Mediterranean and the Atlantic Ocean, the Rock of Gibraltar is a British bolt-hole with huge appeal, pinned to Spain’s southernmost tip by a sandy isthmus and just 23km away from Morocco.

To the ancient Greeks, Gibraltar was known as one of the Pillars of Hercules, with legend crediting Hercules as opening up the straits between the two pillars – Gibraltar in the north and Jebel Musa on the Moroccan side.

For the Arabs, Gibraltar was the staging post for their conquest of the Iberian Peninsula, in which they ruled the roost for 800 years.  Then the Spaniards occupied it for two centuries from 1462, before it was taken over by the British in 1704. It has remained a resolute British bastion ever since. Spain has made various attempts to recover the Rock, to no avail.

The British dug the Great Siege Tunnels and installed guns in a comprehensive defence system that has safeguarded their hold on the Rock. 50 kilometres of tunnels were drilled and some are open so you can hike across the face of the rock and actually peer out of the cannon holes back at Spain. They were expanded during World War II to house 10,000 soldiers, under Churchill's orders. There’s still plenty of graffiti chipped into the walls by bored troops, much of it outside the women’s showers.

With a population of just 30,000, Gibraltar resoundingly voted to remain within the European Union in last year's Brexit referendum, by 94 per cent. That has given rise to renewed speculation as to whether Spain will try and woo the Rock back into its clutches, but Gibraltarians are unlikely to renounce their Britishness. Just 6km wide and 6km long, this speck of Britannia is packed with intriguing sights.

The humble town centre hugs a large harbour with a bustling marina and a spine-like main street groaning with inviting restaurants, cafes and shops. Iconic red phone boxes, pillar boxes and bacon butty shops are an unmistakable reminder of Gibraltar's call to the home-country. Old England history permeates the island in every direction. High above the port, you can spot the tiny breakwater where they pickled Admiral Nelson after the Battle of Trafalgar. Nelson died in victory, and his body was apparently preserved in a barrel of spirits for the trip back to London from Gibraltar.

There are two ways you can reach the summit of the Rock. Not for the faint-hearted, take a taxi up to the top, via the pencil-thin roads with dozens of hair-pin bends and unforgiving sheer drops from the cliffs. These featured prominently in James Bond films. The more soothing ascent is via the cable car, which departs from from the exotic Alameda Gardens. On all sides atop the Rock, the panorama is astonishing, as you take in the views of the Costa del Sol, the Sierra Nevada and across the Strait of Gibraltar to Africa. On a clear day, the Atlas Mountains loom in the distance.

After the panoramic uplift, stop off on the cable car ride half-way back down for some close encounters with the famous, free-ranging Barbary Macaques– today, numbering some 200. They are extra gregarious, very cute but certainly not cuddly. Don't try and touch them, unless you want to risk losing a finger or feel savaged by their voracious bite. Historians believe the Arabs first introduced these tailless apes into Gibraltar over a thousand years ago. It's the only place in Europe with resident free-roaming primates. The British garrison has an ‘Officer in Charge of Apes’ responsible for their general well-being. Folklore says that ‘when the apes disappear from the Rock, the British will leave.’ There’s no sign of that happening anytime soon. Apparently Churchill ordered more of the primates to be imported from Africa, to beef up their numbers. He insisted their population was not to drop below 24.

Check out the Moorish Castle, which is one of Gibraltar’s most important historic sites. Its Tower of Homage, floodlit in summer, attracts the eyes of both passengers passing through the Straits and visitors to the Rock alike. It has been a critical fort for Moorish, Spanish and British troops.

Pop into Gibraltar Museum, which was  built over a 14th century Moorish bath – Europe's best preserved Moorish bathhouse. The museum features a 30 foot scale model of the Rock and tells its story from prehistoric times until today – including the Great Siege of 1789 – 1783 and the Battle of Trafalgar.

St. Michael's Cave is one of Europe’s most striking grottoes with superb blooms of stalactites bursting from the ceiling like monster broccoli. Its spectacular main chamber is used as an auditorium for live performances, while skilled cave adventurers may explore the lower corridors that lead to a subterranean lake. Local legend has it that the caves are haunted by the ghosts of explorers who tried to find an undersea passage through which the apes found their way from Africa. Pure fiction of course, but fabulously fanciful.

The Upper Galleries are situated on the north side of the colony, they were blasted and hewn out of solid rock in 1782 to make gun emplacements during the Great Siege of Gibraltar. They command a spectacular view of the airport and harbour.

Clinging to the side of the vaulting limestone Rock, the Rock Hotel is a characterful art deco property and a prime spot for afternoon tea on the shady terrace that overlooks the bay. John Lennon married Yoko Ono in the hotel 1969. The line from The Ballad of John and Yoko goes, "You can get married in Gibraltar near Spain". Cocktail time? Head for the Rock Hotel's Barbary Bar, for ocean vistas and ape theatre.

Buses from Algeciras will whisk you to the border control from Spain, while there are frequent ferry connections with Tangiers, Morocco. To enter Gibraltar, you'll need to show your passport, as you walk across the border, into this littlest of Britans'.

Mike Yardley is our Travel Correspondent on Jack Tame Saturday Mornings.