This book explores the integration and interplay of model-based optimization and model-free deep reinforcement learning (DRL). It addresses the growing complexity of future wireless networks. This book begins with a concise overview of foundational DRL algorithms and then delves into advanced frameworks, including optimization-driven DRL, hierarchical DRL, multi-agent DRL, Bayesian-enhanced DRL, and Lyapunov-guided DRL. Each framework is illustrated through case studies in emerging scenarios such as intelligent reflecting surface (IRS)-assisted wireless communications, UAV-assisted wireless networks, backscatter-assisted relay communications, and mobile edge computing. Each chapter of this book demonstrates how partial system knowledge, inherent structural properties, and problem decomposition can dramatically accelerate learning convergence. It also improves sample efficiency, and enhance robustness in decentralized, dynamic, and large-scale wireless networks. Tailored for researchers and graduate students focused on wireless communications and AI-driven networking, it bridges theoretical innovation with practical implementation challenges. It provides a roadmap for designing intelligent, autonomous, and resource-efficient next-generation wireless systems. Engineers and professional specializing in AI and machine learning for wireless systems will also find this book useful as a reference.