Counterfactual generation has achieved impressive performance in tasks like image editing and synthesis, thanks to the development of diffusion models. However, due to a lack of understanding of the latent space, existing diffusion-based counterfactual generation models exhibit instability: they either retain too much of the original information or make excessive modifications, sacrificing the original information, which is inefficient and inauthentic. In this paper, we propose a framework that balances the latent space by incorporating signals for transferring to new counterfactuals while preserving factual information. We first identify the cause of this imbalance as the uncontrolled signal from the counterfactuals. Then, we propose a balancing method within the diffusion process. Our method is evaluated on the colored MNIST dataset, a modified version of the standard MNIST dataset, with experimental results demonstrating performance improvements over the previous latent space.