Restless multi-armed bandits (RMABs) are widely used for resource allocation in dynamic environments, but they typically do not consider fairness implications. This paper introduces a fairness-aware approach for offline RMABs. We propose a Kullback-Leibler (KL) divergence-based fairness metric to quantify the discrepancy between the selected and the overall population. This is incorporated as a regularizer into the soft whittle index optimization. We evaluate our fairness-aware algorithm on a real-world RMAB dataset where initial results suggest that our approach can potentially improve fairness while preserving solution quality.