In the case of OpenAI’s CarRacing-V2, Reinforcement Learning (RL) needs tosolve both the problem of world modeling and exploration. This work primarilyfocuses at solving the issues of efficient world modeling and exploration strategiesin RL for continuous control tasks by comparing different approaches for improv-ing the performance. It exhibits an experimental evaluation of three approaches:(i) standard World Models, (ii) NVAE-based hierarchical World Models, and (iii)NoisyNet-augmented World Models. We compare these methods based on cumu-lative reward performance, training stability, and computational efficiency. Thecomparison of the cumulative rewards and training stability in the experimentsshowed that the NVAE-based models improve the feature representation and thegeneralization of the models while the NoisyNet augmentation improves the adap-tive exploration. The work also shows trade-offs, for instance, the computationalcost versus the reward performance among these approaches. It also proposesthat a future model-based RL for autonomous driving should incorporate NVAEfor feature extraction and NoisyNet for exploration as they could yield the best results.The results show that standard World Models have the highest cumulative reward,whereas the NoisyNet-augmented models have similar performance with fewerrollouts, thus indicating better exploration efficiency.