Abstract: In order to study the stability of the gear transmission system in high-speed trains, a dynamic model describing the torsional vibration behaviors of the gear system was developed. In this model, the time-varying mesh stiffness of meshing teeth pairs was calculated through finite element analysis, and the mesh stiffness and transmission error were expanded using the technique of Fourier series. Based on this model, the multiple scales method was used to solve the nonlinear differential equations of gear systems, and the approximate analytical solution and transition curves that separate stable from unstable regions were obtained. In addition, the main factors that influence the stability were discussed. The results show that the unstable regions decrease with the decrease of the train's running speed, but an unstable region always exists at the speed where parametric resonance occurs; increasing the damping is effective to reduce the unstable regions: as the damping increases from 0.01 to 0.05, the amplitude of mesh stiffness fluctuation in stable regions increases from 5% to 20%; and, an increase in the contact ratio can help suppress the harmonic characteristics of mesh stiffness so as to improve the stability of system.