Once an old whaling port, Sag Harbor has small town charm which is overlayed with wealthy Hamptons flair. One of the best places in the Hamptons for doing things in the summer, you will spot wealthy locals getting coffee or a bite to eat and you might spot a celebrity or two. At the same time it is easy to blend into its laid back and comfortable vibe.
We’ve gone there three times now on a Labor Day weekend when the town is bustling, but it’s also more expensive to stay there during that time so you may want to consider to visit off season or earlier in the summer. Following is most of what we have done over three trips, so there is plenty to choose from for a weekend itinerary.
Consider yourself lucky if you ever get a room at the American hotel on Main Street…more likely you’ll find a room at Baron’s Cove a little farther out. AirBnb is also a good option in the area, that is how we stayed both times.
Main Street
Downtown generally starts at the corner of Main Street and Madison and extends down to Long Wharf. The sidewalks are wide and lined with park benches making it a promenade for locals and visitors alike on weekend mornings, afternoons and into the evening.
Some of the best restaurants are located on Main Street, like Lulu Kitchen and Bar for upscale American and local farm stand fare, Paige at 63 Main for outstanding pastas with a vibrant bar and front dining room, or the restaurant at the American Hotel which has several dining rooms and classic French-style fine dining. The wine list there actually comes to you in a leather-bound book five inches thick. Sen is a longtime favorite destination for ramen and sushi.
There are numerous small shops and boutiques along Main Street, from hardware stores and souvenir shops to upscale housewares store and clothing for Hamptons chic.
If you want a quick bite you can get pizza at the popular Sag Pizza, or a bagel at Goldberg’s Famous Bagels. For a place to sit down with your shopping bags and chill for a bit, go to Sagtown Coffee, which is always buzzing with locals getting coffee drinks and shakes.
Sag Harbor has a deep history, and if looking at historic homes is your thing there is lots to see here. Walk all the way to the Long Wharf to grab pictures at the windmill, which is the Sag Harbor visitor’s center. There you can grab flyers for a self-guided walking tour of historic sites.
On your self-guided walking tour you will be able to view houses like the Stanton House & Hope House on Madison, the prior residences of Admiral Stanton who took a historic visit to Japan opening up the east to the west, and Benjamin Franklin Hope who was a skilled clockmaker. Both houses were built in the mid 19th century and are examples of early Greek Revival and Victorian architecture. At Garden Street is the Custom House, which has been restored by the Society for the Preservation of Long Island Antiquities and is open to the public. It is furnished faithfully to the period often with original pieces.
Along Union Street you will find the Old Whaler’s Church, erected as a monument to the town’s whaling prosperity and it is considered the best example of Egyptian Revival style in the U.S. A point of interest is the decoration of blubber spade motifs along the roofline. Along Division Street, you will find the location of the Bulova Watchcase factory, which played a major role in the industrial revival of Sag Harbor in the early 20th century. These are just a few of the 43 historical sites listed in the Sag Harbor Historical Societies’ self-guided walking tour.
To fuel up for your walking tour you can get breakfast or lunch sandwiches at the well-known organic market Provisions just steps away from the windmill. Close to that you can get scrumptious house made doughnuts at Grindstone Coffee and Donuts a must-do for every time I am in Sag Harbor.
I love to take my coffee and donuts to one of the park benches along the water with views of the bay.
The Sag Harbor Whaling & Historical Museum
For a true taste of what Sag Harbor was like during the time when the whaling industry supplied New England and grew whaling ports all along the eastern seaboard, check out the Sag Harbor Whaling & Historical museum. You can see artifacts from that time, and items from whaling ships like scrimshaw.
There are antique maps and videos of what an actual whaling voyage was like. Its ferocity might surprise you.
Sag Harbor Cinema
This much beloved landmark of Sag Harbor was closed after a fire, but has been restored and reopened. It now also has a lovely rooftop terrace called The Green Room, the perfect place to get a drink before or after dinner and a movie.
Sunsets in Sag Harbor
The sunsets are beautiful here with so many water views. Probably one of the best known sunset views can be seen from the terrace at the Beacon restaurant. Try the local fish with beets and citrus beurre blanc there.
You can also take a sunset cruise into the bay, like those offered by American Beauty Cruises.
Foster Memorial Beach
There are many small beaches along the coast but a good one just a few minutes drive from downtown is Foster Memorial Beach. There is a paid parking lot (which we got around by taking an Uber there), otherwise there are no fees to the public. It’s a small, uncrowded, pebbly beach on Sag Harbor Bay (not the ocean) and the calm, shallow waters were fun to splash around in to cool off after a few hours lounging around in the sun.
Another nearby public beach in the other direction is Haven’s Beach.
Sagaponack
A 30 minute Uber ride to the other side will take you to Sagg Main Beach in Sagaponack for views of the ocean. There is plenty of room on the wide expanse of soft golden sand and it was easy for me to grab a place near the surf and take my fill of gazing out into the waves.
Everyone stops by the newly renovated Sagaponack General Store for a bite to eat. I enjoyed their ham and cheese baguette made with the freshest salted butter, paper thin and melt in your mouth ham and cheese inside a crisp freshly baked baguette….enjoyed outside seated amongst their lovely garden.
Back in Sag Harbor, the Marina is also a nice place to hang out at when the weather is nice. Get a pastry, sandwich or lemonade from the beautiful Carissa’s Bakery and sit by the water, or get lobster or clam rolls at The Dock House.
A walk down Bay Street is also pleasant, where you will find Gwyneth Paltrow’s Goop, the high-end and exquisite restaurant Dopo la Spiaggia, and a house on the corner which contains Billy Joel’s recording studio.
A nice place to spend awhile browsing and maybe getting a good book to read is the beloved Sag Harbor Books. I love the many little notes tucked in here and there with staff picks and opinions.
For a pick me up after book shopping, or a sweet treat after dinner, head down Main Street to Buddhaberry.
Most times you will find a happy mix of kids, teenagers and adults milling about inside the tangerine colored walls trying to decide which flavor of frozen yogurt to pump into their cup (my favorite is Very Buddha Berry)…and more importantly, which of the dozens of toppings lining the walls and counters in jars to pick from ( how about crushed Oreos, chocolate covered marshmallows or pépitas?)
Wolffer Estate Vineyard
There are numerous vineyards on Long Island and a favorite activity in the Hamptons is a visit or tour of a vineyard. We went to Wolffer Estate Vineyard on one of our trips, as it was nearby, about a 15 minute drive from downtown Sag Harbor. There is a large wine tasting room which requires a reservation. On our wine tasting we chose four of their award winning wines which we sampled along with a mezze platter. We then walked outside along the edge of the vineyard for a bit, took nice pictures, and then went to their wine store onsite to buy our favorite picks from our tasting.
However you spend your summer weekend in Sag Harbor, doing any or all of the above, there will be plenty of charm and good vibes to soak in and take home with you.
Dates of trip:
September 2019, September 2023, and September 2025
The Full Itinerary 