Preheat the oven to 350ºF and line the bottom of a 9-inch cake pan with parchment paper.

Toss sliced peaches with 1 tablespoon of brown sugar and arrange them in the bottom of the cake pan.

In a saucepan, melt butter, honey, and remaining brown sugar to make the honey caramel. Pour the caramel over the peaches.

Whisk together dry ingredients in a bowl and set aside.

Cream butter, sugar, and almond extract with a mixer until fluffy. Mix in eggs, Greek yogurt, and milk.

Slowly add dry ingredients to wet ingredients and mix until just combined. Pour batter over peaches.

Bake for 35-40 minutes until a toothpick comes out mostly clean.

Toast almonds in a skillet, then add honey and cook until coated. Spread almonds to cool.

Let the cake cool briefly, then invert it onto a rack. Serve with whipped cream if desired.