At the end of my last post, I promised to describe the evaluation and rating process. Although I have a document which describes the various ratings and what they mean, it doesn't appear to be available on-line anywhere else, hence I won't be posting it here. There must be a reason why this is not a publicly available document per the USTA, ITF, ATP, WTA or Grand Slam tennis organizations. In any case, I will describe it.
There are 7 rating levels in the current evaluation system; previously, it had 4 with sub-levels (a 2003 document with this information can be found on-line). The highest rating is 7, for a 'perfect' rotation, and the lowest rating is 1, for an unacceptable rotation. The wording of what constitutes a 1, 2, 3, 4, 5, 6, and 7 rating is articulated in four categories: technique, accuracy, communication, and preparation & professionalism. A sheet detailing this information is posted in the umpire room at the tournament; I have seen at least two different versions of this sheet, either because it has been modified or there are different sheets for different events (e.g. lower level vs. ATP events), though it's probably the former.
In general, an average rating greater than 4 is expected (a '4' is considered the minimum acceptable rating for any rotation); any line umpire who fails to consistently achieve an average higher than 4.0 should probably look for another profession. Better line umpires receive 5s, 6s and the occasional 7, though - because of the human element - there is an inherent inconsistency in the evaluation system.
One factor that leads to an inconsistent application of ratings (despite the evaluation guidelines sheet) involves professional disagreements among the chair umpires about what kind of performance should receive a given rating. For instance, one chair umpire may want to see a given line umpire make at least (e.g.) 5 close calls successfully with no other errors during their rotation before giving them a 5 rating, while another may require a different number. Also, if there weren't enough opportunities in a given rotation to make said calls, one chair might give the line umpire a 4 rating while another would give "no rating" (which potentially protects the umpire's higher average) for the same performance.
In the long run, the ratings should average out to indicate which line umpires are better than others but, when one is a new line umpire, the evaluation system can seem unfair. When one begins to work at the ATP level, only designated chair umpires' ratings count in the average, so there are even fewer opportunities to show one's ability, making each grade more precious (it has much higher value).
Obviously the higher your rating (or GPA) as a line umpire the more you should be hired, however there are also other factors. The number of chief umpires (who do the hiring) is fairly small and it's a somewhat close knit community. This means that a given line umpire's attributes and characteristics - e.g. team player or not - are also taken into consideration when the hiring is done, like any other job. In the end, the system is as fair as any other, given the human factors involved.
