Ham radio exams, algorithms, and scaling on the fly

This year, Caroline and I had an opportunity to attend BSides Las Vegas and DEFCON (fondly known as Hacker Summer Camp). I had a blast at both conferences, where I was introduced to a ton of new ideas, hacked on projects new and old, and made fantastic new friends. I was also privileged to help with the DEFCON Amateur Radio license exam session where we set a conference record for most exams taken (205 exams in 3 hours). After we finished the exams, we had dinner and drinks with a group of our friends and spent quite a bit of time dicussing the exams, especially in terms of algorithmic efficiency and scaling. I’ve been thinking about it ever since, and I could really use some new content over here, so here goes…

How to run a ham license exam session in 270 simple steps

First, some background. The basic process for conducting an amateur radio license exam for the ARRL Volunteer Examination Council is as follows:

Alan, a wanna-be ham, arrives at his friendly neighborhood testing location with a photo ID and $15. Bernice and Carl, two volunteer examiners assigned to handle intake, check his photo ID, collect his money, and record his name on the exam roster. They then walk him through the process of filling out VEC Form 605. Form 605 is used to collect the date, the testing location, Alan’s full name, address, phone number, social security number or FCC FRN number, and information related to the exam he wishes to take (the license grade for which he is testing, whether or not he is applying for a new license, etc.). Once Alan has filled out Form 605, both Bernice and Carl double-check it to ensure he’s filled it out correctly, then give him an exam sheet and test booklet.

Alan writes his name, the date, the testing location, his FRN or SSN, the exam series number on the test booklet (more about this in a minute), his FCC-assigned callsign if he has one, and his signature on the answer sheet. Alan is then free to begin the exam. For the Technician class license exam, he will have to answer 35 questions on various topics: FCC rules, electronics, radio frequency properties, best practices, etc. While Alan takes the exam, another VE – let’s call her Dianne – acts as a proctor to ensure that Alan does not cheat or disturb other examinees.

Once Alan finishes his exam, he raises his hand. Dianne comes over to verify he has filled out the answer sheet correctly and checks that he has not written in the exam booklet (the booklets are reused), and then takes Alan’s completed exam over to Bernice and Carl. Dianne uses a pre-punched answer key that corresponds to the exam series number to grade Alan’s exam, then marks the number of correct answers on the answer sheet and hands it to Bernice. Bernice uses the answer key to verify that Dianne has graded the exam correctly, as does Carl. Once all three VEs have verified that the exam has been graded properly, they indicate whether or not Alan has passed based on his score (for Technician Class, Alan needs 26 correct answers or better to pass).

If Alan fails the initial exam, he must pay another $15, fill out a new Form 605, and take the test again. (Most exam sessions prefer to have you study and come back another time, but there is no rule requiring this.) He receives no proof whatsoever that he took the exam, though proof does exist (more about this below). If Alan haspassed his Technician exam, he is given the option to take the subsequent General class exam at no additional cost. This is referred to as “going on” in ham parlance. If he fails the General exam in this exam session, he still gets credit for the Technician exam.

Once Alan is finished taking exams for the day (either by deciding not to take the next test or by failing the next test), he is issued a Certificate of Successful Candidate Examination, or CSCE. The CSCE is effectively a triplicate carbon-copy “receipt” proving that Alan passed one or more exams during that exam session, and is the only proof given to Alan that he has done so. The CSCE lists the test date and time, Alan’s full name, his address, his existing callsign (if he has one), the exam elements he passed during that exam session, his signature, and the signatures and callsigns of the three VEs who graded his exam. Alan receives the top copy of his CSCE, is (hopefully) congratulated, and told of when he should expect to receive his license information.

In addition, the VEs fill in Alan’s Form 605 with some additional information – whether or not he passed his exams and which license classes he obtained – along with their signatures and callsigns.

At the end of the exam session, the VEs compile the exam roster, all copies of Form 605 (regardless of pass or fail status), all marked answer sheets, and one of the two remaining carbon copies of the CSCEs, as well as any voided paperwork, and send it back to the ARRL VEC. As this is not part of our current examination of algorithm design, we shall leave the details of this process for the reader to discover.

Pros and cons

This process was designed in the 1980s when the FCC transitioned the amateur radio community to self-accreditation and, other than paperwork design and changes to requirements (such as dropping Morse code), has changed little since. At first glance, this process has some clear benefits. First, you only need 3 VEs to administer a full exam session. This allows license exams to be conducted in the middle of nowhere, or where transportation is difficult (see: Alaska). Second, by triple verifying exam results, both the FCC and the ARRL can be reasonably assured hat the operator understands the basic principles of RF operation and thus probably won’t jack up the airwaves. Finally, with regards to the CSCE, there is an established paper trail that ensures that even if something goes horribly wrong – such as a fire, or loss of the exam materials in the mail – an examinee can still obtain license credit for successfully completing an exam.

Under most circumstances, this process serves well enough. However, a typical exam session has between 5 and 15 examinees. However, it does not scale in a way most of us would consider clean. (Remember the early days of Twitter? Yeah. Kinda like that.)

So many hackers, so few chairs

DEFCON 22’s exam session was run by 25 VEs. Of those, 12 served as graders, 3 handled intake, 5 served as exam proctors, 2 served as “barkers”, and 3 served as “routers” . Routers were responsible for moving completed exams between grading teams and proctors, as well as filing of completed paperwork and managing the exam keys. Barkers were responsible for answering questions for people waiting to take the exam, as well as handing out copies of Form 605 and helping examinees fill them out correctly. Proctors administered the exams and returned the results. Graders were organized in “pods” of 3 and, as you can imagine, were responsible for grading exams and completing CSCEs and copies of Form 605. Intake collected money, took Form 605, and handed out exam booklets and answer sheets.

At the beginning of the exam session, the process worked smoothly enough. However, as time went on and the exam room filled up, we discovered several problems.

  • There was no defined order for how exams got graded. For a normal exam session, this isn’t a big deal because the volume of exams is so low as to essentially guarantee a FIFO (first in, first out) grading order. However, as examinees completed their exams, the proctors placed them on top of the router’s “to be graded” pile. This essentially created a FILO (first in, last out) grading order, which meant that some examinees received their exam results after other examinees who started before them.
  • There were large variations in the amount of time it took for a grader to verify her colleagues’ work. For some, this meant a quick looking-over of an already graded exam; for others, this meant essentially grading the exam a second or third time. This also lead to some examinees receiving their exam results and/or CSCE well before others who had finished earlier.
  • While allowing examinees to “go on” to the next element is normally a huge time saver, in this case it contributed to even more delays. Rather than receiving a CSCE and allowed to clear the room, examinees had to wait to find out whether or not they passed (which took non-trivial time, see above) and then essentially repeat the process all over again. Only once they passed all 3 exams, or failed the subsequent exam, were they issued a CSCE and allowed to clear the room. We attempted to streamline this process by having examinees indicate whether or not they wanted to “go on” at the bottom of their answer sheet, but it made little difference in the end.
  • Once an examinee entered the exam room, they were essentially stuck there until they received their results. Due to the aforementioned problems, that lead to some examinees waiting upwards of two hours to receive their results and leave the room. This is normally not a problem; however, the exam room only had space for 65 examinees at a time, which lead to a very crowded exam space.

Thankfully, almost all of our examinees were very understanding of the situation, and some were incredibly helpful with suggestions that allowed us to free up space. For example, someone suggested that people who were waiting for results and not planning to “go on” should get up and move to the back of the room, so that new examinees could sit down. This helped us free up substantial amounts of space and gave more people an opportunity to take their test.

Learning and improving

That being said, there are a number of things we’ve already identified that would improve the throughput of the system and make the overall process more efficient and scaleable.

  • assign a dedicated VE to the routing team as a “scheduler”. This person is in charge of making sure that the grading queue remains FIFO, and has the ability to escalate priority for some exams (such as for people needing to make a flight).
  • increase the number of grading pods.
  • reduce cache size by issuing 3 exams at a time to grading pods. That prevents huge backlogs and gives the routing team a better idea of which grading pods are idle.
  • move to a bigger room. This, obviously, is up to DEFCON CnE, but hopefully our results can convince them of the need.
  • have a dedicated waiting area. This is for both those waiting for results, and for those waiting to “go on”. That frees up more seats for people waiting to take their first exam, but also keeps people in the room until their exam processing is complete.
  • have separate entries and exits, and put all paperwork collection near the exit. That keeps the routing team in charge of the exam area from drowning in paperwork, and prevents a huge bottleneck at the front of the room.
  • assign dedicated VEs to returning exam results and new exams for those “going on”. This frees up the proctoring staff to manage table space and return ungraded exams to the routing staff.

It’s a bit difficult to test these ideas when you only run the exam session once a year, but you could write some code to simulate the flow of people through the process, and also observe the process at other large exam sessions. In any case, as someone who does a lot of ham radio evangelism, I will gladly play a part in making sure next year’s license exams can get as many people on the air as possible.

Caroline is in the hospital: part deux

Hello out there, sports fans. Here’s the latest:

We got the results of Caroline’s MRI back yesterday. Everything’s normal there: no lesions, no swelling, and no tumors (though they did discover a very over-developed pun center, in a portion of the brain shaped like a kitten). Her blood tests continue to come back normal.

This morning, the attending physician told us that the results of Caroline’s cerebrospinal fluid tests for viral meningitis came back negative. That means she no longer has to be on IV acyclovir, and that if everything else checks out, she can most likely go home this afternoon. They are still waiting on Lymes tests in the CSF, but those cultures can take days and the treatment is the antibiotic she’s already on. I’m hoping we can setup Camp Caroline on the recliner couch in our living room.

The good news here is that all the scary stuff has been ruled out: no brain cancer, no infections in the brain, no lesions to indicate stroke, aneurysm, or epilepsy, no bacterial meningitis, no parasites or complications from malaria, etc. Unfortunately, we can’t be positive it’s Lyme’s encephalitis, either, though she has been responding to treatment for that. The episodes of confusion and difficulty speaking have been getting shorter and less severe, and she is feeling much better.

I would like to thank everyone who came to visit her in the hospital, especially while I was dealing with stuff for work. It did a lot to raise her spirits, and it meant the world to both of us.

Caroline is in the hospital: Part one

*blows dust off site*

(In the interests of keeping things easy to follow, I’m going to use my blog for updating everyone on Caroline’s condition. There may be some duplicate information from emails and Facebook, so keep that in mind.)

 My wife is sick. Herein lies the tale.

Caroline has been getting treatment for Lyme disease for about a week.  She’s had headaches, sensitivity to light and sound, body aches, and  a general sense of malaise, but no fever or respiratory problems. She’s also been experiencing what I’ve taken to describing them as “Flowers for Algernon” periods. Her headache will intensify, she’ll start to experience trouble speaking, moving, or thinking, and she’ll get very easily confused. That will last for about 1-3 hours, and then it’ll just go away. Those have been intensifying, and on Monday afternoon (10/15/2012) her speech started to slur during the episodes and she got confused to the point of having trouble opening a car door. We called her primary care physician, and they told us to go to the ER.

We went to the ER at UW hospital, where they did a CT scan and a lumbar puncture, as well as some other lab work. The doctors were not able to diagnose her, so they decided to admit her. After they admitted her, they did an MRI and a series of chest x-rays.

Right now, they suspect either encephalitis as a result of Lyme disease, or viral meningitis. They won’t know for a few days until the cultures on the lumbar puncture come back, so she’ll be in the hospital for a few days. They started her on IV anti-viral medication in case her symptoms are caused by viral meningitis, and they’ve continued to treat her for Lyme disease with doxycycline.

Right now she’s fine, considering her circumstances. She hadn’t eaten much last night before we got here, but then she devoured a BLT and a piece of cheesecake from the hospital cafeteria. She’s in good spirits, but worried and slightly perturbed when the residents do rounds. I spent the night at the hospital last night, and I’ll head home in a bit to take a shower, gather some of her things, and catch a few more hours of sleep.

Caroline has her tablet, and the hospital has WiFi, so she’ll be able to do Facebook and some email. I’m also going to bring her laptop to her, so that she’ll be able  to watch movies or do more interesting stuff. She’s also just insisted that she would be happy to have visitors and phone calls. Her room number is F6/570 and her phone number is 608-267-8566.

Stay tuned to this blog for more information. I’ll try to post again this evening after more lab work comes back.

The Letter R

This is not my last message from Ghana. It is, however, my last message as a Peace Corps Volunteer.

I moved out of my house on Saturday morning, and said goodbye to Seikwa for who knows how long. I left saddened, but satisfied. I know that in some small way, I left the place better than I found it. My students were better off for my having been there, as was the school and, to a lesser extent, the community. And now, I’m in Accra, going through the Close-of-Service Process.

Accra is — as it always has been — surreal. I took a tro-tro to the office this morning, and was passed by at least 3 BMVs on the way. It is a harbinger, of sorts — it’s as close to America as I can get at the moment, and I’m not exactly sure how I feel about that. It’ll grow on me in time, I suppose, but I’ll never again look at a used car or a bathtub the same way ever again.

My Close of Service has gone as smoothly as I could expect it to go. My bags are packed. My paperwork is in order. All I’m waiting for is one last interview and some signatures tomorrow, and I’ll be a Returned Peace Corps Volunteer.


There’s a lot of significance in that letter R. It means that despite the frustrations, the low points, the longing for home, the illnesses, the delays, and everything else that seems to work in consort to make you give up, I was successful. That letter R is a testament to my service. That R means I am a full-fledged dirty hippie. That R symbolizes my commitment to my work, and my desire to have a positive impact on the world. That R is my entire service boiled down to one
letter, one word: Returned.

It also means that my service is never over. As I’ve said before, I’ll spend the rest of my life committed to sharing Ghana with other Americans. I’ll spend the rest of my life searching for banku and making waakye for people, and talking to classrooms full of students who, like me, will some day hopefully wonder what they can do to make the world a better place.

I will spend the rest of my life remembering this experience, and recounting it time and time again to anyone who will do me the kindness of listening. I will spend countless words describing what I saw, what I did, and how it affected me. Right now, I’m savoring that R.

The Other 90%

There’s a saying in software development: the first 90% of the work takes 90% of the time, and the remaining 10% of the work takes the other 90%.
Well, that’s where I’m at right now. I have about 4 weeks left before I travel to Accra and go through my Completion of Service process, and I feel like I’ve got about another year’s work ahead of me. I have lessons to give, papers to mark, exams to write, computer labs to fix, documentation to write so that other people can keep the lab running, paperwork to fill out, Peace Corps books and property to inventory and return, reports to write, people to say goodbye to, bank accounts to close, and a house full of stuff that I have to decide whether to keep, give away, or try to sell.
On top of that, there’s everything related to returning to the United States. Caroline and I have been working on updating our resumes, looking for jobs (though that’s pretty low key at the moment), trying to arrange for housing in Madison, figuring out how to budget our combined re-adjustment allowances, and beginning to talk about what kind of wedding we want, and when and where we should have it. I’m also at the point in my marathon training where I’m running about 30 miles a week and hungry much more often than I used to be.
So, there’s that.
My last day here in Seikwa is August 1st. My last day as a PCV will be August 6th. As I attempt to wind down my life here, and at the same time try to prepare for a new life in America, I find myself reminiscing about the good times, as well as reflecting on the bad times and what I could have done better. I’ve learned a lot of things here — patience and persistance pays off (case in point: my computer lab); don’t believe anything until you see it for yourself (again, the computer lab, but also travel and other things); no plan survives its first contact with reality; and never be afraid to ask questions, no matter how trivial or ridiculous they may seem to you (Ghanaian culture at large).
The most important thing I’ve learned, though, is this: as long as you are willing to learn, and to try new things, and to embrace your sense of curiousity, anything is possible. I’ve traveled and met people and experienced life here for two years, and every day it becomes more strikingly clear just how little I actually know. I’ve learned a lot about administering Linux boxes, and maintaining computers, and squeezing the absolute maximum performance out of a computer; and as I’ve learned to do that, I’ve realized how much more I didn’t know about using computers, and teaching people how to use them, and teaching people how to maintain them. I’ve learned a lot about myself; and in doing so, I’ve realized just how little I actually knew, and still know about who I am and who I want to be.
I am absolutely fascinated by the world around me, and how it works, and I know so very little about it. But as long as I am willing to be curious, and ask questions, and accept that inevitably I will not know or understand something, then my ignorance will not be an obstacle, and the world will be open to me. That’s true of the Peace Corps and experiencing Ghanaian (or any other) culture, but that’s also true of the American culture I’m returning to. This is work — hard work — that I will likely never finish, but, then again, I’m not sure I want it to be. That’s part of the fun.
And, in the end, that’s the other 90% of what I have to do. Peace Corps has 3 goals; the first one is to train people in technical skills; the second is to help people of other countries get a better understanding of America. But the third one is to help Americans to get a better understanding of other countries. I spent two years here, teaching people about computers and computer skills, learning about and experiencing Ghana first-hand, and now I get to come back and spend the rest of my life trying to explain Ghana to you.
So, the next time you see me in America, remember this: I may look at appliances with varying degrees of bewilderment and confusion. I may ask questions that seem ridiculous, such as “So, who’s this Lady Gaga that I keep hearing about?”, and I might cry a little if you offer me a turkey sandwich on wheat bread with a glass of lemonade. But if you ask me a question about Ghana or the Peace Corps, I’ll try to answer it as best I can, and I will ask questions of my own in return. Together, we can figure out how the world works a little bit better, and take care of that other 90%.