Surface defects in hot rolled steel strip poses quality and performance risks. Classifying defects enables for the rapid identification and removal of the causes of their occurrence, as well as the provision of appropriate treatment to fix them. This Deep Learning-based solution identifies four classes of salient surface defects: slag inclusion, oxide scale, scratches, and iron sheet ash. This solution analyses the user provided image data, identifies the best performing deep learning model architecture, and predicts the defect class with the highest probability score. This can assist metal products manufacturing companies to improve their quality control process.