A novel method toward color image segmentation is proposed based on edge linking and region grouping. Firstly, the edges extracted by the Canny detector are linked to form regions. Each of the end points of edges is connected by a direct line to the nearest pixel on another edge segment within a sub-window. A new distance is defined based on the feature that the edge tends to preserve its original direction. By sampling the lines to the image, the image is over-segmented to labeled regions. Secondly, the labeled regions are grouped both locally and globally. A decision tree is constructed to decide the importance of properties that affect the merging procedure. Finally, the result is refined by user's selection of regions that compose the desired object. Experiments show that the method can effectively segment the object and is much faster than the state-of-the-art color image segmentation methods. © Shanghai Jiaotong University and Springer-Verlag Berlin Heidelberg 2011.