Grouping is an important tool in computer vision for improving both matching and recognition. Most papers consider grouping as a segmentation problem and a hard decision is made about which pixels in the image belong to the same object. In this work we instead focus on soft pairwise grouping, that is computing affinities between pairs of pixels that reflect how likely that pair is to belong to the same object. The approach is based on a simple but effective method to group pixels based on color statistics. By using only color information and no prior higher level knowledge about objects and scenes we develop an efficient classifier that can separate the pixels that belong to the same object from those that do not. In the context of segmentation where color is also used only nearby pixels are generally considered, and very simple color information is taken into account. Based on an existing initial implementation, the objectives of the project are to: 1) develop efficient algorithms based on the color-based grouping approach; 2) test the algorithm on standard datasets; and 3) apply to resulting algorithms to recognition and segmentation tasks.