Network traffic produces large volumes of data every second, and traditional security tools often struggle to detect new or unknown attacks hidden within this traffic. Anomaly-based intrusion detection systems address this problem by learning normal network behavior and identifying suspicious deviations. This literature review examines recent studies that use machine learning, deep learning, and hybrid machine learning-deep learning approaches for network traffic anomaly detection. The review focuses on feature selection, model complexity, dataset use, evaluation metrics, and the practical challenges that still limit real-world deployment. The reviewed studies show that traditional machine learning models can remain efficient when supported by careful feature selection, while deep learning models are useful for learning more complex spatial and temporal traffic patterns. Hybrid approaches often report stronger performance because they combine the speed and simplicity of machine learning with the representational power of deep learning. However, the literature also shows continuing weaknesses, including reliance on static benchmark datasets, class imbalance, computational cost, limited explainability, and uncertainty about performance in live networks. The review concludes that hybrid approaches are promising, but their future value depends on making them lighter, more explainable, and more reliable outside controlled experimental settings.