When the deep space probe approaches the target planet, due to the rapid increase of the gravity of the target planet, the orbital dynamics model will have a rapid acceleration change. Because the noise covariance is not completely known, the traditional filtering algorithm cannot obtain the optimal estimation of navigation parameters, which is difficult to meet the performance requirements of the approach navigation system. In order to meet the requirements of high stability and accuracy of the system, a sliding window adaptive nonlinear filtering algorithm based on system noise covariance is proposed. By constructing the system noise covariance update function and using the sliding window to smooth the noise covariance, the errors caused by velocity noise and position noise are separated, the filter parameter information used is updated in real time, and the system noise covariance is adjusted adaptively. Taking the Mars probe as an example, the simulation results show that, compared with the traditional unscented Kalman filtering method, the position accuracy and velocity accuracy of the proposed method are improved by 90.97% and 66.17% respectively, which suppresses the rapidly changing integral error on the system model, and solves the divergence problem of the traditional filtering method. In addition, the influence of filtering period and window size on navigation performance is analyzed, which provides a feasible new adaptive filtering method for autonomous navigation of deep space exploration.