For him aerobatics is “a hugely interesting mix of motor sport and flying, full of the most stimulating people trying to be the best at something that is so difficult that they will just never get close to perfection …”
And judging is “... The most fantastic opportunity to mix with and judge the best bunch of people I have ever met, not just in the UK but all over the world.”
What made you decide to become an aerobatic judge?
When I joined the British Aerobatic Association in 1991 I had just started flying a 160hp Pitts-S1C but I didn’t really have a clue what I was supposed to be doing. I was asked to help Bill Hood, an American pilot and IAC judge who was then working in the UK on the US Marines Harrier programme, to help him to run a judging school for the BAeA – and this quickly taught me that it is quite important to know exactly what the judges are looking for before you can fly the aeroplane properly and hope and do what is expected of you. You might be surprised how many competition pilots seem to have a very poor understanding of precisely what is required of them!
What do you like most about judging? Why?
I find it enormously interesting and rewarding to watch clever and capable pilots that I know driving their brain to its deepest ability to fly intricate aerobatic figures, and at the same time cope with the wind, the complex figure designs, use their imagination to solve difficult flight challenges, making awful mistakes, all this in a hugely dynamic and fast moving situation. I know how nearly impossible this is too …
Can you describe the usual path one would take to become an international judge?
Judging actually isn’t very difficult – but it is a bit like flying; once you start you can’t stop, the pilot keeps going and you just have to keep up with the job. When you become reasonably competent you can graduate to judging at Advanced or Unlimited in your national championships, through which you should be able to store some judging data for reference and – if the right software is used for scoring – also obtain a Ranking Index. If you are good enough this can open the door to contacting CIVA and talking to the judging co-ordinator, who is always looking for new judges each year to work at World and European championships.
What is the most difficult part of being Contest Chief Judge of the International Aerobatic Championship?
The Chief Judges’ job is to manage and control all aspects of the judging panel’s work so that it conforms correctly to the published CIVA rules. Most judges are pretty experienced and know the rules well enough, but interpreting these rules is not always simple and this can lead to strong and diverse opinions being expressed regarding whether for example a Hard Zero should be given for a figure – then the CJ’s job is to determine the “correct” result by conference with all the judges. We use the video to determine these matters of fact, and always it is the majority view that decides the result – if the opinions remain 50/50 then the pilot must get the benefit of any doubt.
Of all the international contests you have judged, which do you remember being the most impressive or liking the most? Why?
Although the standard of flying at international championships is generally quite high throughout, there is inevitably a “top group” of pilots who are the ones most likely to be the front runners – and of course they are the interesting ones to watch. I remember especially the Touzim EAC in 2010, when Renaud Ecalle was just incredible to watch … his ability to keep the French Team Extra-330SC accurately in front of us throughout each sequence without seeming to hurry or make any conscious effort to stay in exactly the right place was supreme and so impressive. Pure skill of the highest order, very rare, great to watch and extremely memorable. I was lucky to be at the US Nationals in Texas last year, and the 4-minute Freestyle flight from Rob Holland was awesome … lots of “How do you do that?” moments, quite superb.
Which figures and flights have you judged that have impressed you the most?
It doesn’t have to be an Unlimited ace – even a Sport sequence can demonstrate great skill and consistency in the flying. In the UK David Thompson, who is now flying a CAP-232 at Advanced, used a 180hp Citabria at Intermediate level a few years ago in some very impressive and accurate sequence flying, and watching that big rectangular wing accurately meet the angles and attitudes of even the simple figures required was a real joy.
As a software developer, you designed the ACRO aerobatic scoring software that is widely used competitively and utilizes the Fair Play System (FPS). FPS has revolutionised the way that pilots receive marks, resulting in a more objective scoring system. Could you give us an overview of FPS?
Hmmm – not a simple subject! Judges quite often give slightly different marks when they see the same errors, and that’s OK – this happens because their “style” can be harsh or lenient, with a smaller or larger spread from the highs to the lows … but sometimes the differences between one judge and the others can be significant, and then to be fair to the pilot we must try to see if it should be included unchanged … or if the mark should be replaced by something closer to the majority view. FPS works within small groups of marks that should ideally be the same, normally for all the pilots on a figure-by-figure basis. First the judges “styles” are statistically balanced so they can be compared to each other mark for mark / pilot for pilot. Then the system looks at each mark to see if any fall outside the central 97.5% of statistically acceptable grades; any “outliers” thus identified are rejected, and subsequently replaced by a calculated or “fitted” value that is derived for that judge / that pilot based of course upon only the remaining acceptable marks. After this stage has been completed for all the figures an initial table of scores is calculated (marks multiplied by the figure K’s) and a similar process seeks to detect judging bias; where necessary a further two-step procedure provides unbiased replacement “fitted” scores. Overall this process ensures that every pilot gets a fair result, with no unacceptable figure grades or pilot scores remaining in the results. In the computer this is very fast, but the whole thing does involve zillions of double-precision calculations from the start to the finish.
Why was it necessary to develop a system like this?
History records that after the 1964 World Championship in Bilbao it was all too clear that some judges had given an impressive degree of favouritism to their own countries’ pilots, and the results were inevitably viewed with a degree of suspicion. Since then a number of increasingly complex numeric and statistical systems have attempted to identify and resolve this unacceptable national bias. In Olympic competitions it is common to discard the highest and lowest grades and form the result from the remainder, but this is very wasteful and in our view it is far better to work from the whole data set and carefully resolve the anomalies that we find there. In turn we have had raw averaging, Bauerisation, TBL, TBLP – and now FPS. The FairPlay System is a major improvement over all the previous systems, and although probably not perfect it is certainly a great advance over those previous attempts to tackle poor grading and unintentional (or intentional …) bias between the different nations’ judges. Obviously it is important that the best pilot wins, not the best judge!
Do you think that FPS needs any improvement?
An unfortunate – but seemingly unavoidable – aspect of FPS (and TBLP before it) is that due to its inherent complexity it is just too difficult for most people to understand, and thus the truth of the resulting output must be taken on trust. Some believe that returning to averaged raw grades would make for clearer and more obvious results, but it is imperative that we have a reliable way to resolve errors and bias in judging – these undesirable influences, whether intended or not, are present to some degree all the time, and clearly is of the utmost importance at international events to make sure that their effect is removed or at least minimised. It is imperative to CIVA that all the interim computations and the final results are completely exposed to anyone who wishes to check them, and the open availability of the ACRO software and all the contest files on the CIVA Results website ensures that this is so. At the moment we just don’t know of a better way.
How does FPS facilitate judges’ scoring and how does this result in more accurate results for pilots?
If I may be pedantic it is the judges that give marks and the computer calculates the scores. In ACRO I have used FPS to make available a comprehensive analysis for the Chief Judge and each individual judge that shows how closely their marks for each figure and each pilot match – or are different to – those of the other judges; from this they can each reflect on how they have derived their marks by comparison with their colleagues, and by how much and hopefully why they are different. I must say again – little differences are OK, it is the big differences that matter. Without this sort of feedback every judge might think that he or she is perfect, but of course we are all human and in our world nothing is ever that good. The FairPlay system ensures that individual marks and/or scores that are “unusual” or statistically unacceptable don’t get to affect the results, and this is definitely fairer to the pilots who have all made fantastic personal commitments just to secure their places at these great events.
Within FPS, you have the “Judges Ranking” concept. What is the aim of this index?
The fundamental aim for every judge is to rank the competitors in the correct results order – from the best flying at the event down to, well, the least good. The detailed results are really there to provide all the supporting evidence, it is the overall rank order we require. Thus if a judge gets every pilot ranked in the correct order we can say that he or she has in effect done a perfect job; this judge would get a Ranking Index (RI) of zero. The more a judge puts the pilots in the “wrong” order and the larger the differences between his or her pilot scores and those in the final results, then clearly the less accurately he or she has contributed to the results. The Ranking Index for each judge is raised from zero by a combination of both the rank and the score differences for each pilot. At a major event with perhaps 10 judges and 50+ pilots we would expect a judges overall RI (i.e. an average of the RI’s from all the sequences) that is less than about 15 to be pretty good, from 15 to perhaps 25 the RI is becoming less acceptable, and anything over that value would infer that the judge was significantly different to the majority view of the other judges and might thus not be giving acceptably reliable marks. This is not an exact science for sure, but the system does seem to give us a pretty good indication of how well each judge fits into the overall pattern of output from the whole panel.
Are the judges Ranking Index used to select judges for international championships?
Yes – CIVA uses a “rolling three year average” of judges recorded RI’s to guide selection of the primary seven judges for each championship, then there is a more open system that encourages judges that are not already in the CIVA Ranking Index database to be promoted by their NAC into the remaining three positions. In this way CIVA works to get some new judges into the system each year, to replace those that may have ceased judging or whose RI is not sufficiently near the top of the list.
You and your colleague Graham Hill had been holding judging courses in Spain, Belgium, Germany and other countries for pilots and other people who were interested in learning the aerobatic judging system and its rules. Now your course has developed into a formal CIVA international seminar. Could you explain what this course consists of and whom it is designed for?
While Graham and I have been successful roaming around Europe for quite a few years giving our very British judging seminar, I have always thought that CIVA should take the lead in setting the very best standards for international judge training – and thus should create a Judging Seminar package that provides a first class example of how we should be training international judges according to the current rules etc.. This package now exists in an initial format, and I have been extremely pleased to present it for the first few times. Although we do not yet have an official procedure for access to it by others, the material can now be made available to selected presenters on application to CIVA. The key change for me has been to realise that just grinding through the CIVA Judging Rules is a very boring and generally unproductive process, in fact this is much better done by the aspiring judge in the comfort of his or her own home. Far more important is the training and insight required to take this learned material and apply it within the tense and very public atmosphere of the judging line, where performance stress, relentless speed and noise, and of course peer pressure all make for a hostile environment where it can be extremely difficult to work to a good and reliable standard. At WAC-2011 our excellent video operator Javier Marquerie helped us to create a superb record of a warm-up flight by Anton Berkutov in one of the works Sukhoi-26M3’s, with a clear soundtrack of me calling the figures and Graham giving his marks. I have since stabilised and annotated this video to create an excellent training aid, and this clear example of calling a complex 14-figure unlimited Free Unknown sequence – with some inadvertent major errors included – forms the centre-piece of the CIVA presentation. I have added various other interesting and amusing brain-teasing items that keep attendees more enlightened and engaged than we have in the past, and overall I am very pleased with this first step. There is much more to do – certainly some good interactive online judging see-and-tell material would be a valuable next step, and we need more “real” and immersive videos like the excellent one I now have. We can always improve and do things better!
As an experienced aerobatic Contest Chief judge, you have seen the evolution of the aerobatic sports world over the last several years. In which ways has judging changed? How has the flight level of the pilots changed?
Two questions to answer here: First the changes to judging; domestically (in the UK at least) the level of technical understanding and knowledge of the rules is broader, but more importantly the self-management techniques for applying this knowledge are becoming better understood. Make no mistake – judging is a challenging and demanding task, and must be learned and practiced time and again to reach an effective standard that pilots can rely upon. Undoubtedly the internet has provided a much richer environment for this learning process, and helped us to keep everything more up-to-date in each successive year. Internationally the judge selection process has benefitted enormously from the use of results based RI’s and the FPS output from ACRO, which allows us all to see everything that has been committed to the score sheets – no more hiding of poor decisions! As far as the standard of flying is concerned, I do think that the increase in tough training regimes coupled to use of the finest new aeroplanes and from most pilots a relentless desire to compete and be seen as the best of the best has definitely raised the standard at the head of the field; even the flying we see at the other end of the results is getting better too. However … there is always the possibility to do even better still and this is almost certainly the main attraction.
Do you think aerobatics has been affected by the global economic crisis?
I would say yes for some areas, but not all. The incredible rise of the Advanced power category – to an astonishing 84 pilots with all bar one flying monoplanes at WAAC in Hungary last year – seems to have been completely unaffected, but paradoxically the Unlimited class is struggling to provide sufficient entries. Perhaps the former has drawn too many from the latter, and the balance may forever be changed. With the 330-SC, the XA-42 series and (in the US) the MXS the physical demands from flying an Unlimited sequence are ever more violent and extreme; competing at this level is certainly no longer a part-time exercise … and more than a little expensive too. Next year we should see the first international combined Yak-52 and Intermediate international championship in South Africa, the Intermediates running to identical rules with that of the Russian class, and I look forward very much to see how the popularity of this category rises toward that of Advanced. Who knows where this significant enlargement of the pilot class base will lead?
Do you think aerobatics and aerobatic competitions are promoted enough by the events’ organizers, particularly as it pertains to recruiting sponsors for the events?
This is such a hard subject … of course we would all like to see more publicity and some really worthwhile sponsorship, but inescapably it seems that, just like the formal parts of Olympic ice figure-skating, it is very hard to make the flying of formal Aresti sequences sufficiently interesting for Joe Public to flock to our events. To be fair to them they can’t possibly understand the detail of what is really going on, and the typical absence of a first class commentator and a score-board with the details of each performance doesn’t help. Unlimited Freestyle flying may well provide the essential level of excitement and entertainment that this requires, but the number of pilots who are good enough at this is small and a whole succession of Free sequences certainly cannot provide that level of attraction. The recent tie-up between FAI and their FAME marketing offshoot may offer the possibility of a better relationship between headline sponsors, organisers and aerobatic pilots, but so far the vibes are not earth shaking … We worked very hard in preparation for WAC-09 at Silverstone to engage some major names in other sports and many international business regimes, but in the final analysis we achieved little real success. We must also remember that this search requires its own experienced staff and dedicated time and commitment, perhaps more wisely allocated elsewhere … it is indeed a hard subject.
What do you think aerobatics should be and/or will be in the future?
Aaaahh … deep in my crystal ball, what can I see? As pilots our demographic base is inescapably locked into rather less than 1% of the worlds’ population, although I would expect that this is likely to represent a high quality niche of relatively intelligent, successful and interesting self-starters, thus we should in the main be a tight band of high achievers and practical doers. The design and capability of aeroplanes for our sport continues to move forward, at least in power – for gliders the prospect is not quite so bright. Technology will provide more and more ways for us to more easily measure things like box outs, flight-path and attitude, figure accuracy and/or errors; the judging line operation will be slicker and swifter; results will be more immediate, and so on. However – the key element surely is us pilots and our poor old human brains; pulling more G, rolling and flicking faster, keeping the sequence picture alive and true in our heads … now there’s a whole area that is unlikely to change at all. Perhaps we should look further outside the rigid Aresti flight envelope toward more inventive formats and less medically demanding environments? Soon the aeroplanes and the information systems will surpass the natural human performance envelope, and then how we develop this wonderful sport will be largely limited and defined by our own bodies and our own imagination. Perhaps we can make it safer, like the motor racing Grand Prix engineers have? Otherwise I think we are quite close to a natural plateau, and what will be in 25 years’ time is not that far away from where we are right now. I do not mean this to be a negative view, simply one of practical reality. I truly hope that I am still here to see it!