In this paper, a fast pixel-level adapting background detection algorithm is presented. The proposed background model records not only each pixel's historical background values, but also estimates the efficacies of these values, based on the occurrence statistics. It is therefore capable of removing the least useful background values from the background model, selectively adapting to background changes with different timescales, and restraining the generation of ghosts. A further control process adjusts the individual decision threshold for each pixel, and reduces high frequency temporal noise, based on a measure of classification uncertainty in each pixel. Evaluation results based on the ChangeDetection.net database are presented in this paper. The results indicate that the proposed algorithm outperforms the majority of earlier state-of-the-art algorithms not only in terms of accuracy, but also in terms of processing speed. ?? 2014 IEEE.