Abstract: To deal with the growing amount of videos on the Internet, this paper presented a scheme for automatic video classification based on the combination of MPEG7 descriptors and secondprediction strategy. Nine MPEG7 descriptors such as color, texture, shape and motion were extracted from five different genres of videos and combined as a whole representative feature. Then it was put into an SVM classifier to train the model and predict. The traditional 11 method was modified with a secondprediction strategy to improve the classification accuracy. The experiments on a broad range of video data demonstrate that the accuracy of our classification scheme is higher than other existing schemes and the scheme is suitable for the largescale video classification task under a complex environment.