In recent days, the mortality rate is occurred due to the rising number of brain tumor patients around the world. The severity of brain tumor is huge while comparing with other cancer types, and thus, there is a need of early identification and precise treatment for saving a life. However, detection of such tumor cells is complicated owing to the formation of the tumor cells, which results in complications regarding brain tumor classification for evaluating the tumors. Various imaging approaches have been utilized for detecting brain tumors in recent studies. On the other hand, there is a complication regarding the extraction of abnormalities in the brain via easy imaging approaches. To alleviate all the challenges, this paper implements a tumor classification model using intelligent segmentation and deep learning techniques. The input images are obtained from benchmark data sources and fed to pre-processing stage, which is accomplished by filtering methods and Contrast Limited Adaptive Histogram Equalization (CLAHE). These images are given to VGG16-Ensemble Network (VGG16- Ensemble) with three classifier models as Auto Encoder (AE), Deep Belief Network (DBN), and Long-Short Term Memory (LSTM), where the results are obtained as normal and abnormal images. The attained abnormal images forward to segmentation. Here, the Convolutional Neural Network (CNN), Modified CNN, and U-Net are used, where the parameters in CNN are optimized to propose modified CNN by Adaptive Distance-based Sea Lion Optimization Algorithm (AD-SLnO) to acquire the segmented images. Finally, the optimized ensemble classification is developed known as OVGG16-Ensemble, in which hyper parameters of VGG16, LSTM, DBN, and AE are tuned optimally by AD-SLnO algorithm. Thus, it is developed for classifying the abnormal images into benign stage or malignant stage. Consequently, the result evaluation is done, and its performance is estimated by diverse measures.