Cellular networks are becoming increasingly complex, requiring careful optimization of parameters such as antenna propagation pattern, tilt, direction, height, and transmitted reference signal power to ensure a high-quality user experience. In this paper, we propose a new method to optimize antenna direction in a cellular network using Q-learning. Our approach involves utilizing the open-source quasi-deterministic radio channel generator to generate radio frequency (RF) power maps for various antenna configurations. We then implement a Q-learning algorithm to learn the optimal antenna directions that maximize the signal-to-interference-plus-noise ratio (SINR) across the coverage area. The learning process takes place in the constructed open-source OpenAI Gym environment associated with the antenna configuration. Our tests demonstrate that the proposed Q-learning-based method outperforms random exhaustive search methods and can effectively improve the performance of cellular networks while enhancing the quality of experience (QoE) for end users.