Visual scene understanding is an important and fundamental field for advanced application scenarios such as autonomous driving and robotics. This course majorly focuses on delivering deep learning-based visual scene understanding techniques in both 2D and 3D perspectives. In the 2D part, it introduces topics including image and scene classification, semantic segmentation, and object detection. In the 3D part, it shows how 3D scene understanding can be performed through learning from 2D inputs, involving topics such as scene depth estimation, camera pose prediction, 3D scene reconstruction, and visual SLAM. Several representative deep scene understanding architectures and frameworks in supervised or self-supervised settings together with the 2D/3D tasks are also presented in the course.