The early detection of diseases is important in agriculture for an efficient crop yield. The bacterial spot, late blight, septoria leaf spot and yellow curved leaf diseases affect the crop quality of tomatoes. Automatic methods for classification of plant diseases also help taking action after detecting the symptoms of leaf diseases. This paper presents a Convolutional Neural Network (CNN) model and TSOA (Tomato Stem Optimization Algorithm) algorithm based method for tomato leaf disease detection and classification. The dataset contains 500 images of tomato leaves with four symptoms of diseases. We have model CNN for automatic feature extraction and classification. Color information is actively used for plant leaf disease researches. In our model, the filters are applied to three channels based on RGB components. The TSOA has been fed with the output feature vector of convolution part for training the network. The experimental results validate that the proposed method effectively recognizes four different types of tomato leaf diseases