Autonomous sequential decision making is of key importance in agriculture. For example, in anticipation of an extreme weather event (e.g., flash droughts, derechos), it is critical to make proper decisions (e.g., plant date, irrigation frequency and amount) in order to avoid potentially severe impacts on the crop yield. In this talk, I will introduce some of our recently developed techniques studying the tradeoff between exploitation and exploration in sequential decision making. I will start by introducing EE-net for contextual bandits, which leverages two neural networks for learning the reward function and for adaptively learning the potential gains compared to the currently estimated reward respectively. Then I will introduce PageRank Bandits, which adapts the neural exploration strategy from EE-net to address link prediction problems on graphs. Furthermore, in the presence of adversarial attacks or corruptions on the received rewards, I will present our recent work utilizing a novel context-aware Gradient Descent training strategy to improve the robustness of contextual bandits. Towards the end, I will also share my thoughts regarding future directions.