This PhD tackles the challenge of camera-based methods for navigation and environmental sensing in dynamic environments. The goal is to design a robust real-time localization and mapping algorithm which can reliably cope with dynamic (e.g. people, cars) and changing (e.g. structural changes, weather) environments. We plan to introduce semantics into the traditional geometric processing cues, which allows for explicit treatment of dynamic and changing environments in order to improve mapping and, consequently, pose estimation. As a second goal we leverage the semantic information to introduce enhanced image retrieval techniques to improve the large-scale localization and mapmaintenance in multi-session scenarios.