Recent work in scientific machine learning (SciML) has focused on incorporating partial differential equation (PDE) information into the learning process. Most of this work has focused on relatively "easy'' PDE operators (e.g., elliptic and parabolic), with less emphasis on relatively ``hard'' PDE operators (e.g., hyperbolic). Within numerical PDEs, the latter need to maintain a type of volume element or conservation constraint for a desired physical quantity, which is known to be challenging. Delivering on the promise of SciML requires seamlessly incorporating both types of problems into the learning process. To address this issue, we propose ProbConserv, a framework for incorporating constraints into a black-box probabilistic deep-learning architecture. To do so, ProbConserv combines the integral form of a conservation law with a Bayesian update. We demonstrate the effectiveness of ProbConserv via a case study of the Generalized Porous Medium Equation (GPME), a parameterized family of equations that includes both easier and harder PDEs. On the challenging Stefan variant of the GPME, we show that ProbConserv seamlessly enforces physical conservation constraints, maintains probabilistic uncertainty quantification (UQ), and deals well with shocks and heteroscedasticity. In addition, it achieves superior predictive performance on downstream tasks.