Periodical inspection is the dominant form of structural health monitoring (SHM). However, civil engineering societies in North America have expressed the common consent that the current inspection practice is not sufficient to ensure infrastructure safety. Moreover, the increasing number of aged infrastructures will require an advanced form of inspection systems. The processes of vision-based methods for identifying damage using image processing algorithms (IPAs) are similar to human inspections because both use visual information. The outcomes of vision-based methods are much more intuitive than systems with traditional contact sensors. Accordingly, researchers have proposed a variety of different methods. For example, early research adopted IPAs directly into damage detection problems. The results from IPAs are intuitive but require manual decision-making processes. Further attempts have been made to establish automated decision-making systems using machine learning algorithms (MLAs). However, real-life applications are rare. The unavailability is mainly rooted in the fact that IPAs were developed and tested in controlled circumstances, while real-world situations often cannot be controlled. Mobile units with cameras have attracted great attention in the SHM discipline. This type of inspection can improve accessibility to infrastructures but still lacks automated damage detection. Even if IPAs and MLAs are integrated, the combined system (mobile units, IPAs, and MLAs) will likely be invalid in practice because this system inherits the limitations of IPAs. To overcome these challenges, IPAs should be replaced by advanced computer vision techniques. In this thesis, deep learning (DL) is considered the key for surpassing the current state of vision-based approaches. Deep learning models are capable of learning features from raw data. Instead of manually developing IPAs, feeding raw data that were collected in uncontrolled environments and leading a machine to learn the features of the data may be a better approach. A deep learning model for classifying images for damage detection into binary classes is introduced, and its performance is compared with IPAs. The results of the classification DL model demonstrate the possibility of replacing IPAs with DL models. A segmentation DL model is also introduced that demonstrates faster, more robust, more flexible, and more intuitive than competitive methods.