Line with parchment and grease a 9 x 13-inch baking dish. Leave some parchment overhanging on both ends so you can lift cake out.
Toss plums with 2 tablespoons all-purpose flour and set aside.
Pour sugar into a bowl, scrape vanilla seeds from the halved pod, and rub into sugar with fingertips. Rub the pods into sugar as well, to catch any remaining beans. Discard pods or reserve for another use.
In the bowl of a stand mixer fitted with the paddle attachment or with a hand held mixer, beat butter until light and fluffy, 2 minutes. Add 1/2 cup plus 2 tablespoons vanilla sugar and beat 2 minutes more. Add Cognac, and then the egg, mixing until each is well combined and scraping down sides as needed.
Combine flours, baking powder, and sea salt in a small bowl. With mixer on low, add half the flour, then the milk, and finally, add the remaining flour.
Scrape batter into prepared dish. Smooth batter out with a silicone spatula. In neat rows, lightly press plums into the surface of the cake. Sprinkle top with remaining 2 tablespoons vanilla sugar.
Bake 35 - 40 minutes, or until the top is golden, the edges have started to pull away from the pan, and a toothpick comes out with just a few crumbs attached.
Cool completely on a wire rack before serving with a dollop of whipped cream or crème fraîche.