Abstract: To study the problem of detecting and tracking fast-moving vehicles in complex circumstances, an adaptive algorithm for image segmentation and filtering was proposed using intelligent Agent based on knowledge database, and an adaptive background model was built through the dynamic matrix of accumulated frame differences. In the tracking process, an improved SSD (sum of squared differences) algorithm was designed to forecast the initial iteration points. According to the Jensen inequality, a MeanShift algorithm for iterative updating of the adaptive kernel-bandwidth was derived to achieve the adaptive intelligent tracking of moving vehicles in videos. The Experimental results shows that the algorithm can track the moving targets in videos effectively and accurately, and has a strong adaptive ability. Compared with other existing algorithms, the tracking error is reduced by 54.4%, and the average tracking time is extended by 41.3%.