Preheat the oven to 350°F (175°C). Lightly grease an 8-inch or 9-inch square baking pan with butter or cooking spray. Line the pan with parchment paper if desired to make removing the cake easier.
In a medium bowl, whisk together the flour, baking powder, baking soda, cinnamon, ginger, nutmeg, and salt. Mixing these ingredients first helps distribute the leavening agents and spices evenly throughout the cake.
In a separate large bowl, whisk the pumpkin puree, granulated sugar, brown sugar, oil or melted butter, eggs, milk, and vanilla extract until smooth. Make sure the ingredients are well combined, but avoid excessive mixing.
Gradually add the dry ingredients to the pumpkin mixture. Stir gently with a spatula or whisk just until no dry flour remains. Do not overmix, as this can make the finished cake dense rather than tender.
In a separate bowl, combine the flour, brown sugar, granulated sugar, cinnamon, and salt. Pour in the melted butter and mix with a fork until large, moist crumbs form. If the mixture looks too wet, add a little flour; if it looks too dry, add a small amount of melted butter.
Pour the pumpkin batter into the prepared baking pan and spread it evenly. Sprinkle the crumb topping generously over the surface. Keep the topping in small and medium-sized clusters for a classic bakery-style appearance.
Bake for approximately 40–50 minutes, or until a toothpick inserted into the center comes out clean or with a few moist crumbs. The topping should look golden brown, and the center should feel set. If the crumbs brown too quickly, loosely cover the cake with foil for the final part of baking.
Allow the cake to cool in the pan for at least 20–30 minutes before slicing. For the optional glaze, whisk powdered sugar, milk, and vanilla until smooth. Drizzle it over the cooled cake and let it set before serving.