An image-driven model based on spatiotemporal neural network (STNN) is proposed. And prediction of crack growth is carried out based on fatigue experiments of aluminum alloy. Fatigue experiments with an initial edge crack angle of 0° and a 15.0% limit load level are designed. Images of specimen deformation are taken by DIC. And 5511 frames of displacement fields are obtained. These images are used as datasets of STNN after interpolation, augmentation and dimension-raising. Convolutional long short-term memory (Conv-LSTM) and SimVP neural networks are used to predict fatigue crack growth. The prediction accuracies of these two methods are compared in terms of the structural similarity index measure (SSIM) and the root mean square error (RMSE). The results show that SimVP neural network has better prediction performance in the test stage. This method can accurately predict fatigue crack growth rate and propagation path. And it is a reference of damage tolerance analysis and determination of inspection intervals for structures.