Abstract: This paper carries out research on the highly optimized mode decision algorithm of x264 software and accurately pins out its redundant points. The mode selection process is reorganized hierarchically. Auxiliary inter modes’ invalidating condition and main inter modes’ validating condition are proposed.New mode selection algorithm based on them is designed. According to the experimental results, the new algorithm brings remarkable speed gain for both P and B slices.