Abstract: A novel spacetime speeded up robust features (SURF) descriptor and its application to human action recognition by combining with a bag of video words approach were presented. The new descriptor can better represent the spatiotemporal nature of the video data in the application of action recognition. A bag of words approach is used to represent videos, and a soft weighting strategy is exploited. Experiment is done in the KTH’s action recognition dataset. In the experiment a voting system containing second pass prediction is employed in classifying actions as well as the traditional classification framework. The results of experiment show this approach is able to outperform the previously proposed schema both in speed and accuracy, while the new voting schema works better than the traditional one in some actions.