The purpose of this study is to quantify and optimize the performance of an automatic portal image analysis procedure under clinical conditions and to compare the performance with that of human operators. A new method, based on analysis of variance, is introduced to quantify the clinical performance of portal image analysis tools in terms of systematic and random variations. The automatic portal image analysis procedure is based on chamfer matching. Two image enhancement techniques have been investigated in the automatic procedure: morphological top-hat (MTH) transformation and multiscale medial axis (MMA) transformation. The performance of these enhancements was quantified and optimized as a function of filter size using images obtained from clinical treatment. All images used for this study were obtained from pelvic treatment fields by means of an electronic portal imaging device. The random variations in the alignment of AP fields are typically 0.5 mm and 0.5 degrees (1 SD) for both the human operators and the optimized automatic analysis procedure. Random variations in the alignment of lateral pelvic fields are typically twice as large for all operators. MMA enhancement yields smaller random variations than MTH enhancement for lateral fields, but the differences are marginal for AP fields. The optimized automatic analysis procedure has a success rate ranging from 99% for AP large fields to 96% for lateral fields and 85% for AP boost fields. The accuracy of the method is comparable with the accuracy of the human operators for most investigated fields. For lateral boost fields and simultaneous boost fields, the random variations of the automatic analysis are typically two times larger than the variations of the human operators. Automatic analysis is 4 to 20 times faster than human operators yielding a large reduction in work load.