As multistage gear transmission systems are complex and precise, the flexibility of shaft can influence the dynamic response of system. In order to study dynamic response of the system, we build the rigid model of gear system and the finite element model of the gear shaft. The rigid-flex coupling model is established with the virtual prototype technology, and a contrast between rigid model and rigid-flex coupling model is constructed. With these methods, the dynamic responses with different rotation speeds and different loading magnitudes are examined. We also analyze the influence of shaft flexibility, rotation speeds and loading magnitudes on the vibration characteristics of gear transmission systems. © 2016, Shanghai Jiaotong University and Springer-Verlag Berlin Heidelberg.