Heat oven to 350°F. Line a 9-inch round cake pan with parchment paper and lightly grease the sides.
Combine butter and brown sugar in a small saucepan over medium heat. Stir until melted and bubbly. Remove from heat and stir in cinnamon, allspice, maple syrup (if using), and vanilla.
Arrange apple slices over the bottom of the pan on top of the caramel mixture in an even layer. Sprinkle with walnuts if using.
In a medium bowl, combine flour, baking powder, baking soda, salt, and spices. Set aside.
In a large mixing bowl, whisk together browned butter, both sugars, eggs, sour cream, milk, and vanilla until smooth.
Add the dry ingredients to the wet and mix with a spatula or on low speed using an electric mixer, scraping the sides of the bowl. Do not overmix.
Pour the cake batter over the apples in the prepared pan and smooth into an even layer.
Bake in the preheated oven for 40–45 minutes, or until the center of the cake tests done.
Cool for 15–20 minutes on a wire rack. Run a knife around the edges of the cake, place a serving plate on top, and invert to release the caramelized apples on top.
Let the cake cool slightly before slicing. Serve warm with vanilla ice cream.