Inductive biases in the form of conservation laws have been shown to provide superior performance for modeling physical systems. Here, we present Hamiltonian graph neural network (HGNN), a physics-informed GNN that learns the dynamics directly from the trajectory. We evaluate the performance of HGNN on spring, pendulum, and gravitational systems and show that it outperforms other Hamiltonian-based neural networks. We also demonstrate the zero-shot generalizability of HGNN to unseen hybrid spring-pendulum systems and system sizes that are two orders of magnitude larger than the training systems. HGNN provides excellent inference in all the systems providing a stable trajectory. Altogether, HGNN presents a promising approach to modeling complex physical systems directly from their trajectory.