Fluency recognition from speech signals plays a vital role in computer-assisted voice analysis. The proposed work presents a computational framework using an audio processing system capable of classifying the fluency of speech such as fluency, non-fluency pause, and non-fluency stammer. The proposed model comprises preprocessing, spectrogram generation, and classification of speech fluency by the VGG16 pre-trained model. This model consists of convolutional layers and these layers extract discriminative features from spectrogram images of the speech signal. In this work, speech datasets such as Libri Speech, Crosslinguistic Corpus of Hesitation Phenomena (CCHP) English, and University College London’s Archive of Stuttered Speech (UCLASS) were used to find speech fluency. The performance of the proposed model was compared with the existing pre-trained network and state of art methods.