Preheat the oven to 325°F (165°C). Grease two 8-inch round cake pans, line the bottoms with parchment paper and line a small baking sheet with parchment as well.
For the crumb topping, rub the almond flour, powdered sweetener, cold butter, vanilla extract and salt together with your fingertips until small clumps form. Spread on the baking sheet and bake for 10-12 minutes, stirring once, until lightly golden. Let the crumbs cool completely so they turn crisp.
In a medium bowl, whisk together the almond flour, coconut flour, baking powder, xanthan gum and salt.
In a large bowl, beat the softened butter and granulated sweetener for 2-3 minutes until fluffy. Beat in the eggs one at a time, then mix in the sour cream, lemon juice, almond milk, lemon zest and vanilla extract.Add the dry ingredients and mix on low speed just until a smooth, thick batter forms.
Divide the batter between the prepared pans and smooth the tops. Bake for 25-30 minutes, until golden and a toothpick comes out clean. Cool in the pans for 15 minutes, then transfer to a wire rack and cool completely.
For the lemon cream, beat the cream cheese, powdered sweetener, lemon juice, lemon zest and vanilla extract until very smooth. In a separate bowl, whip the cold heavy cream to stiff peaks, then fold it into the cream cheese mixture in two additions.
Place one cake layer on a serving plate and spread with half of the lemon cream. Top with the second layer and spread the remaining cream over the top and sides.
Gently press the cooled crumbs over the top and around the sides of the cake. Chill for at least 2 hours before serving so the cream sets and the flavors come together.