SUBJECTIVITY IN CONCOURS JUDGING

From the beginnings of the Ferrari Club of America, concours judging of our Ferraris has been a major part of the club’s activities.

    In earlier decades, the judging was largely subjective; how clean, well-presented and authentic the car appeared was good enough. But a way to more critically evaluate these fascinating cars was needed.  

    Taking the lead in making this change, Ed Gilbertson established the International Advisory Council for the Preservation of the Ferrari Automobile in 1984.  (IAC/PFA)

    Gilbertson enlisted his many like-minded and Ferrari-knowledgeable friends in drafting a set of criteria under which Ferraris could be more equitably judged.

    From the outset, this effort was welcomed by the Ferrari community and began to be employed in judged events worldwide.

    Full adoption of IAC/PFA standards naturally took some time, but the benefits were quickly recognized.  Owners wanted their cars evaluated fairly and annual events cultivating prestigious image and reputations were pleased to have empirical guidelines under which the cars would be judged consistently.

    The heart of the IAC/PFA is a twenty-member council composed of individuals who carry serious weight in Ferrari knowledge and experience, and includes practicing top-level restorers and historians.

    The group truly represents Ed’s goal of perfecting Ferrari judging. The Council meets every year at Cavallino to discuss ideas to improve judging, listen to experts on various Ferrari minutia and fine-tune the process—always aiming for the most objectivity and fairness possible.

    The stated goal is to recognize automobiles that correctly reflect how they were built in Maranello decades ago.  In this way the mission of ‘preserving’ these autos is served by rewarding correct restorations, not just the shiniest.

    The IAC/PFA also recognizes that they have another role, which is to educate Ferrari owners so they may more accurately restore their cars.

    To fast-forward, let’s acknowledge that for the past two decades IAC/PFA guidelines have been applied to most of the world’s premier Ferrari concours events.  Let’s stipulate, too, that great effort has been invested in selecting and training the judges who hold IAC/PFA credentials and are committed to reducing subjectivity on the concours field. Given that, how, then, do individual cars sometimes receive different scores at successive events?

    That is actually the topic here, as the question is invariably brought up after any event’s results are posted. No matter the guidelines, diligence and expertise striving for consistency, apparent inconsistency sometimes pops up.

    Simply stated, the judges are human, and their opinions are subjective—informed, no doubt, but still subjective.  Anticipating this, the Chief Judge assigns individual judges to a given class who he (or she; congratulations, Leslie!) knows to have expertise in those particular cars.

    Three judges are assigned to each class to further balance the task and specific protocols are observed to minimize variables. But still, a given car may have been judged at 97 points at Cavallino, but then ten months later at the FCA National receive 95. Or 96. Or 98.

    Assuming that the car has remained absolutely the same in the meantime, how is this possible…if we are judging by specific guidelines and rules?  As knowledgeable as they may be, each judge has a little different depth of knowledge of a particular Ferrari detail, or a little different background, and these variances can account for a point or two.

    As examples, let’s say that the judge assigned to evaluate the exterior (one does exterior, another interior and the third does engine and chassis) happens to have a lot of body shop experience and can tell that a door’s fit is off—but most lay people would never see it.

    Ditto if the interior judge has upholstered Ferraris for years and knows that the texture of leather in a particular car is incorrect—but looks fine to you and me.

    If the engine/chassis guy has been doing Ferrari V12s for decades and sees that a car has incorrect carburetors, say, few others would ever know and maybe whoever judged the car last didn’t know, and did not deduct the penalty for those carburetors.

    In a perfect world, we would always have such absolute experts judging our cars, but such expertise is rare and the Chief Judge uses the best talent available that day, and those judges do the best they can, which has actually proven to be pretty darn consistent over the years.

    When a team is asked to evaluate ten cars at 100 possible points each and managed at the end of the day to only award or deduct two or three of those thousand points inconsistently, that’s amazing!  But, if one of those points was the difference between a Platinum or not…that’s not so philosophically accepted.

    OK, besides some human variability down at the real minutia level, what else could account for an inconsistent score for the same car between events?

    The number of cars in the class to be judged is a big factor.  There is a finite amount of time available to complete judging and get the results turned in to the scorekeeper—usually about three hours.

    One of the stipulations of IAC/PFA judging is to allot the same amount of time to each car in the class, so all entrants are treated equally. If there are ten cars in a class, the judges have to hustle along to stick to 15 minutes on each. More cars will mean less time for each, but if the class is small, maybe twenty minutes will be available.

    What that really means is those cars will be under scrutiny 30% longer, and the longer you look, the more you see. If a car was judged once against eleven others, and the next time there were only six cars in the class, it’s almost guaranteed the second score will be more critical.

    It rained at Cavallino in 2019, so owners were unable to display their car’s paper documentation, and interior judges tried to keep cars’ doors closed as much as possible. (And even dull paint looks good when wet!)

    While not insignificant, these factors could have contributed to a slightly different score, up or down, for a particular car than it achieved another day. Or high winds could make opening hoods and doors risky, so judges on that day have to look quickly.

    Bottom line, while the IAC/PFA goes to great lengths to provide consistent, objective judging, some subjectivity creeps into a volunteer corps of judges who have varying levels of knowledge and philosophy.

    I have judged with some who will note an insignificant flaw on the judging sheet—but deduct no points for it—preferring it to be a teaching experience for the owner. But other judges will exact a half-point toll for the same item, believing they have been charged with objectively evaluating the car, not providing charity.

    As judges, most of us love to meet car owners who are eager to improve their cars, and who view judging as a means to gauge their progress, shooting for a little better score at each event.  But as we have seen, sometimes other factors may inhibit a point progression and seem like a step back has been taken.

    What’s our take-away? Having a car judged in several events will result in a good overall appraisal of its correctness—assuming all events were IAC/PFA-judged. If your (unchanged) car was judged four times by multiple judges and scored 95, 98, 94 and 97, that’s a pretty consistent and objective evaluation, with the variances attributed to judges’ expertise, external factors like rain and especially the time available to judge each car.

    Dedicated owners value having their cars expertly evaluated and will use this year’s criticisms to direct their efforts toward a better score next time.