Ham radio exams, algorithms, and scaling on the fly

This year, Caroline and I had an opportunity to attend BSides Las Vegas and DEFCON (fondly known as Hacker Summer Camp). I had a blast at both conferences, where I was introduced to a ton of new ideas, hacked on projects new and old, and made fantastic new friends. I was also privileged to help with the DEFCON Amateur Radio license exam session where we set a conference record for most exams taken (205 exams in 3 hours). After we finished the exams, we had dinner and drinks with a group of our friends and spent quite a bit of time dicussing the exams, especially in terms of algorithmic efficiency and scaling. I’ve been thinking about it ever since, and I could really use some new content over here, so here goes…

How to run a ham license exam session in 270 simple steps

First, some background. The basic process for conducting an amateur radio license exam for the ARRL Volunteer Examination Council is as follows:

Alan, a wanna-be ham, arrives at his friendly neighborhood testing location with a photo ID and $15. Bernice and Carl, two volunteer examiners assigned to handle intake, check his photo ID, collect his money, and record his name on the exam roster. They then walk him through the process of filling out VEC Form 605. Form 605 is used to collect the date, the testing location, Alan’s full name, address, phone number, social security number or FCC FRN number, and information related to the exam he wishes to take (the license grade for which he is testing, whether or not he is applying for a new license, etc.). Once Alan has filled out Form 605, both Bernice and Carl double-check it to ensure he’s filled it out correctly, then give him an exam sheet and test booklet.

Alan writes his name, the date, the testing location, his FRN or SSN, the exam series number on the test booklet (more about this in a minute), his FCC-assigned callsign if he has one, and his signature on the answer sheet. Alan is then free to begin the exam. For the Technician class license exam, he will have to answer 35 questions on various topics: FCC rules, electronics, radio frequency properties, best practices, etc. While Alan takes the exam, another VE – let’s call her Dianne – acts as a proctor to ensure that Alan does not cheat or disturb other examinees.

Once Alan finishes his exam, he raises his hand. Dianne comes over to verify he has filled out the answer sheet correctly and checks that he has not written in the exam booklet (the booklets are reused), and then takes Alan’s completed exam over to Bernice and Carl. Dianne uses a pre-punched answer key that corresponds to the exam series number to grade Alan’s exam, then marks the number of correct answers on the answer sheet and hands it to Bernice. Bernice uses the answer key to verify that Dianne has graded the exam correctly, as does Carl. Once all three VEs have verified that the exam has been graded properly, they indicate whether or not Alan has passed based on his score (for Technician Class, Alan needs 26 correct answers or better to pass).

If Alan fails the initial exam, he must pay another $15, fill out a new Form 605, and take the test again. (Most exam sessions prefer to have you study and come back another time, but there is no rule requiring this.) He receives no proof whatsoever that he took the exam, though proof does exist (more about this below). If Alan haspassed his Technician exam, he is given the option to take the subsequent General class exam at no additional cost. This is referred to as “going on” in ham parlance. If he fails the General exam in this exam session, he still gets credit for the Technician exam.

Once Alan is finished taking exams for the day (either by deciding not to take the next test or by failing the next test), he is issued a Certificate of Successful Candidate Examination, or CSCE. The CSCE is effectively a triplicate carbon-copy “receipt” proving that Alan passed one or more exams during that exam session, and is the only proof given to Alan that he has done so. The CSCE lists the test date and time, Alan’s full name, his address, his existing callsign (if he has one), the exam elements he passed during that exam session, his signature, and the signatures and callsigns of the three VEs who graded his exam. Alan receives the top copy of his CSCE, is (hopefully) congratulated, and told of when he should expect to receive his license information.

In addition, the VEs fill in Alan’s Form 605 with some additional information – whether or not he passed his exams and which license classes he obtained – along with their signatures and callsigns.

At the end of the exam session, the VEs compile the exam roster, all copies of Form 605 (regardless of pass or fail status), all marked answer sheets, and one of the two remaining carbon copies of the CSCEs, as well as any voided paperwork, and send it back to the ARRL VEC. As this is not part of our current examination of algorithm design, we shall leave the details of this process for the reader to discover.

Pros and cons

This process was designed in the 1980s when the FCC transitioned the amateur radio community to self-accreditation and, other than paperwork design and changes to requirements (such as dropping Morse code), has changed little since. At first glance, this process has some clear benefits. First, you only need 3 VEs to administer a full exam session. This allows license exams to be conducted in the middle of nowhere, or where transportation is difficult (see: Alaska). Second, by triple verifying exam results, both the FCC and the ARRL can be reasonably assured hat the operator understands the basic principles of RF operation and thus probably won’t jack up the airwaves. Finally, with regards to the CSCE, there is an established paper trail that ensures that even if something goes horribly wrong – such as a fire, or loss of the exam materials in the mail – an examinee can still obtain license credit for successfully completing an exam.

Under most circumstances, this process serves well enough. However, a typical exam session has between 5 and 15 examinees. However, it does not scale in a way most of us would consider clean. (Remember the early days of Twitter? Yeah. Kinda like that.)

So many hackers, so few chairs

DEFCON 22’s exam session was run by 25 VEs. Of those, 12 served as graders, 3 handled intake, 5 served as exam proctors, 2 served as “barkers”, and 3 served as “routers” . Routers were responsible for moving completed exams between grading teams and proctors, as well as filing of completed paperwork and managing the exam keys. Barkers were responsible for answering questions for people waiting to take the exam, as well as handing out copies of Form 605 and helping examinees fill them out correctly. Proctors administered the exams and returned the results. Graders were organized in “pods” of 3 and, as you can imagine, were responsible for grading exams and completing CSCEs and copies of Form 605. Intake collected money, took Form 605, and handed out exam booklets and answer sheets.

At the beginning of the exam session, the process worked smoothly enough. However, as time went on and the exam room filled up, we discovered several problems.

• There was no defined order for how exams got graded. For a normal exam session, this isn’t a big deal because the volume of exams is so low as to essentially guarantee a FIFO (first in, first out) grading order. However, as examinees completed their exams, the proctors placed them on top of the router’s “to be graded” pile. This essentially created a FILO (first in, last out) grading order, which meant that some examinees received their exam results after other examinees who started before them.
• There were large variations in the amount of time it took for a grader to verify her colleagues’ work. For some, this meant a quick looking-over of an already graded exam; for others, this meant essentially grading the exam a second or third time. This also lead to some examinees receiving their exam results and/or CSCE well before others who had finished earlier.
• While allowing examinees to “go on” to the next element is normally a huge time saver, in this case it contributed to even more delays. Rather than receiving a CSCE and allowed to clear the room, examinees had to wait to find out whether or not they passed (which took non-trivial time, see above) and then essentially repeat the process all over again. Only once they passed all 3 exams, or failed the subsequent exam, were they issued a CSCE and allowed to clear the room. We attempted to streamline this process by having examinees indicate whether or not they wanted to “go on” at the bottom of their answer sheet, but it made little difference in the end.
• Once an examinee entered the exam room, they were essentially stuck there until they received their results. Due to the aforementioned problems, that lead to some examinees waiting upwards of two hours to receive their results and leave the room. This is normally not a problem; however, the exam room only had space for 65 examinees at a time, which lead to a very crowded exam space.

Thankfully, almost all of our examinees were very understanding of the situation, and some were incredibly helpful with suggestions that allowed us to free up space. For example, someone suggested that people who were waiting for results and not planning to “go on” should get up and move to the back of the room, so that new examinees could sit down. This helped us free up substantial amounts of space and gave more people an opportunity to take their test.

Learning and improving

That being said, there are a number of things we’ve already identified that would improve the throughput of the system and make the overall process more efficient and scaleable.

• assign a dedicated VE to the routing team as a “scheduler”. This person is in charge of making sure that the grading queue remains FIFO, and has the ability to escalate priority for some exams (such as for people needing to make a flight).
• increase the number of grading pods.
• reduce cache size by issuing 3 exams at a time to grading pods. That prevents huge backlogs and gives the routing team a better idea of which grading pods are idle.
• move to a bigger room. This, obviously, is up to DEFCON CnE, but hopefully our results can convince them of the need.
• have a dedicated waiting area. This is for both those waiting for results, and for those waiting to “go on”. That frees up more seats for people waiting to take their first exam, but also keeps people in the room until their exam processing is complete.
• have separate entries and exits, and put all paperwork collection near the exit. That keeps the routing team in charge of the exam area from drowning in paperwork, and prevents a huge bottleneck at the front of the room.
• assign dedicated VEs to returning exam results and new exams for those “going on”. This frees up the proctoring staff to manage table space and return ungraded exams to the routing staff.

It’s a bit difficult to test these ideas when you only run the exam session once a year, but you could write some code to simulate the flow of people through the process, and also observe the process at other large exam sessions. In any case, as someone who does a lot of ham radio evangelism, I will gladly play a part in making sure next year’s license exams can get as many people on the air as possible.