In the middle of Casco Bay stands a fort defending Portland, Maine’s harbor. Built on the Hog Island Ledge, the massive fortification was designed to present an overwhelming array of firepower to any who dared to attack. No one ever did. In the absence of conflict and destruction, trees and shrubs took root and birds used it as a waypoint between the islands and the city. Starting in 1916, a caretaker and his family lived on the island. His granddaughter was born there.

Earlier this summer, while on vacation, my family and I stormed its walls. For an afternoon, we made the fort ours.

Portland thrusts out into the bay on a peninsula that looks like a sailor’s brawny arm curled to show off his tattooed bicep. The harbor runs along the outside of the arm to the south, curving around the elbow, and the Old Port—where the heart of the historic city and its nightlife are located—is near where the beefy seaman’s watch might be. There’s a more placid cove to the north, inside the peninsula.

The Old Port is flanked by two hills covered in homes. One residential area is on the fist of land sticking out into the port, and the winding streets of working class homes with breathtaking bay views are crowned by an historic lighthouse. The other neighborhood is inland from the port, and at its top is the Eastland Hotel with its 201 guestrooms and rooftop restaurant and lounge.

Looking for adventure of the nautical variety, my family and I headed to Portland, and we spent our first night in the Eastland.

This larger hill with the hotel is where the city’s wealthy lived in the 1800s, and there are rows of streets with magnificent brownstones and Victorian mansions to prove it. The biggest, most elaborate churches are here, and our hotel, built in the 1920s, looked right down on one of them. When we first got to room, my son and I spent a while peering down from our 11th-floor perch imagining a pirate assault on the granite chapel, plotting our course over its rooftops to the inner courtyard. As it turned out, this flight of fancy was as exciting as things would get during our first evening in town.

My wife and I made several day trips to Portland before we became parents, and we had fond memories of the place: beer, seafood, shopping and boats, all within a dozen or so blocks of some of the most interesting, well-preserved historic oceanfront architecture you’re likely to find in New England. Portland proved ideal for adult carousing, but the dynamic was different with a six-year-old along for the ride.

Commercial Street is the main drag along the docks where many of the best restaurants are situated, but instead of enjoying a leisurely reconnaissance to find the perfect place to dine, our weary charge insisted we settle for the first place we found. As soon as we sat down, we felt conned by the place’s name—The Farmer’s Table. In the place of hearty local produce, we got slick, curved architecture and a standard-issue diner food. Later, in an attempt to right that wrong, we reserved a table at The Grill Room & Bar for dinner. While the food was much improved, the meal was more expensive than memorable.

Before we knew it, we were back in our hotel cell. Our boy was cranky from the slog back and forth between the port and the hotel, and we were regretting that we’d gotten ourselves trapped on the 11th floor of the hotel without an escape plan.

*

Our fortunes improved the next day when we left the city and drove across the harbor bridge to the home of an old college buddy in South Portland. She was out of town that weekend, and we had the place to ourselves. After lunch, we walked to the community beach.

Though also a city, South Portland is a sleepy residential community compared to the urban center on the other side of the water. The area where we were staying was packed with small, well kept houses, gardens and lawns. Though new to the place, we were regularly waved to on the way to the water by people tending their gardens. We felt welcome. Our fortunes seemed to be changing.

From the white crescent beach, I first spotted the future object of our conquest: Fort Gorges.

Because Portland is an important, historic trading harbor, the entrances to its bay are protected by a series of massive granite and brick fortifications. Many date from the mid-1800s and were once armed with massive cannon. In some places, dug a few hundred yards further in from the coastline, there are more modern military installations from the 1940s which had their guns trained on the clouds for enemy aircraft, but it’s these Civil War-era fortifications which are the most impressive.

Built not only to give a raiding fleet a pounding, these mighty walls full of massive guns were intended to ward off such an assault by looking every bit as powerful as they were. The walls are formed of massive blocks of granite masterfully knit together by careful planning, precise craftsmanship, and gravity. Directly across from the beach where our young captain body surfed, the dark, empty eyes of Fort Levett stretched across the midsection of Cushing Island. At one end of the cove in which we swam there was another gauntlet of cannon, but it was the fort that stood in the center of the bay that held my closest attention.

Apparently rising out of the waves all by itself, this fort stood two thick layers tall with countless gun casements surveying the bay in an impressive 180-degree sweep. Most beguiling of all, the flat top of the fortifications seemed to spill over with a mountain of vegetation, reminding me of a broccoli casserole. Despite its flamboyant cap, the fort seemed utterly impregnable and menacing.

The longer I looked out at it there, standing alone in the center of the great bay, and tried to gauge its size and dimensions, the more I became certain the scent on the breezes was that of true adventure. I realized I couldn’t leave Portland without at least attempting to scale those walls and see what was inside.

*

Returning from the beach, I found a corner of our friend’s yard where I could sneak onto someone else’s wi-fi connection and consult Google. I discovered the castle in the middle of the harbor was named after the first colonial leader of Maine, Sir Ferdinand Gorges. Construction was completed in 1865, and while Fort Gorges was designed to support a garrison of 500 soldiers to man the guns, it never saw battle. After the Second World War the military designated the fort as surplus, and in 1960 the city acquired it.

Just as I’d hoped: their defenses were down, and it seemed likely our small crew could take it with little or no resistance.

The trick was getting there. In the late summer and fall there were a couple tours that visited the island for about $40 a person, but in late June the only option seemed to be a water taxi. When I called, I was told it was possible only at high tide, and I should call back tomorrow. If they weren’t too booked it might happen.

Having found a worthy objective for adventurous ambitions, we also fared better finding appropriate vittles that night for dinner. With the Old Port now choked with Friday night revelers, we found parking and a far better place to eat a short drive up the hill toward the lighthouse.

Located on the ground floor of an old house, the place was called Silly’s, and they did their best to live up to their name—utensils and condiments in a lunchbox on everyone’s table, a menu full of crazy-but-enticing names (Cradle to the Gravy and Tempeh of Doom, for instance) and beverages served in cracked tin cups. Still, even with the thick layer of whimsy, the place served seriously tasty dishes. I had the Big Fat Greek Dinner, which was a scrumptious medley of smoky flavors from a Grecian diner (lamb, feta, and stuffed olive leaves), and my wife ordered the Slop Bucket, which was a bowl of pulled pork on a bed of dirty rice. Including a couple of beers and the captain’s meal, we left very satisfied for under $50.

The winds had definitely changed and seemed to be in our favor.

*

I managed to reach the water taxi pilot on his cell phone first thing the next morning. He said he’d take us out there for $40. We were to meet him on the docks at noon, when the tide was at its highest. We were on time, but when the pilot spotted our crew, he seemed incredulous.

“The docks out at Gorges got washed away ages ago,” he said. “There isn’t really an easy place to land—it can get a little hairy. If you’re not 100 percent sure you’re ready, now’s your last chance.”

Back off now? During our moment of triumph? Hell, no. With some trepidation we piled into the small power boat, and in minutes we were off, pounding across the waves to the fort in the middle of the bay. While the side of the fort that faces the bay sweeps around in a five-sided arc, the side facing the city is a straight wall, and here we could still see some of the Hog Island Ledge upon which the battlements were constructed. Here, too, there was a long granite wharf that thrust out into the water. As the pilot had said, age and neglect had disturbed the straight, military lines of the dock, and huge blocks of granite had so shifted and broken there was no easy landing place.

Instead, the pilot maneuvered the boat along a clean expanse of wall and told us to hoist our belongings up and climb after them. As the boat rose and fell on the waves, it was clear that the climb could only be done when we were at the highest point, so, waiting between surges, we each made the scramble, and before we knew it, the pilot was pulling away, promising to be back in an hour and a half.

At the end of a rambling path that ran along the wrecked wharf, the door to the fort stood open and waiting. After heading though a short arched hallway, we emerged into the giant, grassy courtyard at the center of the fort, and all around us the walls climbed up to their tree-covered summit. Two floors of arched cannon casements spread all around us, and there was a path through the grass to a spiral staircase that could, it appeared, take us wherever we wanted.

Our son’s face flooded with delight at the historic fortifications before us, and after an awe-filled moment, we were all heading for the stairs. We climbed to the top and marveled at the view of the harbor and also found the last remaining cannon in the fort lodged in the wall and covered with vines. We walked through the granite and brick chambers where the cannons had once been positioned and poked our heads into dark rooms full of echoes.

After we scouted out the fort quickly, I went back and began photographing panoramas of the building, while my wife and son shared a picnic and did a more methodical examination of the place that included a swim along the wharf.

Initially, we were almost alone in the place, but as we explored, several groups of kayakers made shore and joined us. Even so, the enormity of the place and the sound-dampening thickness of the walls made us feel remote and alone. Though the place was designed to blast invaders to kingdom come, the precise masonry and immense vision of the builders gave the fort a certain cathedral-like quality. The lack of broken beer bottles and absence of graffiti—the fort’s far out of range for drunken vandals—added to this sense of reverence, and for the hour and a half we were there, we found the kind of real peace and joy we’d set out to find.

As much as we enjoyed our afternoon playing pirate, we hadn’t anticipated encountering any real ones. As we raced back into the harbor, our faces giddy with excitement, the pilot saw fit to explain his pricing more accurately. The amount he had quoted earlier for the five-minute trip to Fort Gorges was only one way. He’d be expecting twice that for getting us home.