The theory of life history evolution provides a powerful framework to understand the evolutionary dynamics of pathogens in both epidemic and endemic situations. This framework, however, relies on the assumption that pathogen populations are very large and that one can neglect the effects of demographic stochasticity. Here we expand the theory of life history evolution to account for the effects of finite population size on the evolution of pathogen virulence. We show that demographic stochasticity introduces additional evolutionary forces that can qualitatively affect the dynamics and the evolutionary outcome. We discuss the importance of the shape of pathogen fitness landscape and host heterogeneity on the balance between mutation, selection and genetic drift. In particular, we discuss scenarios where finite population size can dramatically affect classical predictions of deterministic models. This analysis reconciles Adaptive Dynamics with population genetics in finite populations and thus provides a new theoretical toolbox to study life-history evolution in realistic ecological scenarios.