What AI alignment can learn from Aviation.

As I’ve outlined previously, AI alignment as a noun is the process undertaken to align AI models values with human ones. As an adjective, it is the degree to which an AI based system conforms to said human values.

Before I go any further, alignment generally requires data scientists or some sort of engineering chops in your organisation. Most of what I write about is more to do with aviation safety management systems applying to the AI world, in a safety sense, rather than an alignment sense. But there’s still a lot that AI alignment can learn from aviation. Especially how to think about it.

What does aviation say about alignment?

Well if AI alignment is the degree to which AI systems exhibit the values we want them to exhibit, then human alignment is the degree to which we want humans to exhibit the values us as an organisation want them to exhibit in order to reach our organisational goals.

So how does aviation do that?

Well we take a person off the street, and subject them to multiple layers of screening and training. Screening to ensure the worst tendencies of humans are not present, and training to embed the good tendencies, and the patterns of behaviour that enable someone to be effective at the tasks they need to accomplish.

Now 99.9% of organisations out there won’t be training or even fine tuning their own AI models. They will be selecting publicly available models to perform the work they want in their organisation - or choosing technologies from third parties that use publicly available, pre made models. So the main thing that you will need to do is to screen, rather than train your AI models.

In aviation, we put everyone through a security screen, in Australia our equivalent of the FBI and CIA get a look in, and they make sure you’re not suss. That’s the basic screening for every single person that gets access to walk into or around a plane or airport in Australia. In AI this would the equivalent of ensuring your model has been built by a decent company or reputable organisation that does not have clear or concealed ulterior motives. Vendor selection becomes important and it is tied to use case. When you ask about Tiananmen Square to Chinese open models, they don’t know what you’re talking about. This may be ok for a model that takes workers hours from excel and places it into the payroll system, but probably not appropriate for providing grading assistance for university history lecturers.

Another level of screening that goes on is with the most safety critical roles in aviation - Air Traffic Controllers and Pilots - they undergo physical health screening and basic mental health screening to ensure they are physically and mentally stable people. Now what’s the parallel in AI? I would posit that models that are operating at the edge of their capabilities, e.g. continuously at the edge of their context window limits, in domains that have only been invented subsequent to their training cutoff dates or in domains very weakly represented in their training data/distilling process/postraining process represent this kind of need for screening. This is a much harder screen to make, but it is possible to get a feeling for the distribution of capabilities through publicly available benchmarks (I’m a fan of artificial analysis, go Aussies). An example is that some of the Chinese open source models score highly on general intelligence, but this overall intelligence is disproportionately because of their ability to code, and not their ability to mark a history essay.

So these are two kinds of screening for alignment that can and should be made on models prior to incorporating them into the work that you need to do. The basic “are they a terrorist” or security screen and the tighter “are they fit” screen.

Next, actual alignment:

Training a person and training an AI model have so many similarities it’s not funny. But this is because we both utilise neural networks. Now the training analogy can work in two way - both in the alignment of the actual AI models themselves, or in how nan overarching safety system builds trust in the models that are being used whether specifically aligned or not. I’m going to talk about the former, the model level.

So when a pilot is taken off the street, given the security screen then given the fitness screen the next thing they do is go into training for eighteen months or so to become qualified to fly an aircraft. The culmination of this is an award of a pilot’s license. In contrast, an airline manager may graduate from an aviation management degree, taken off the street, given the security screen (these days they are at least) and although they have a qualification they spent four years attaining they aren’t licensed to do anything. The trust they have to perform their tasks is much more implicit than explicit. Why? Because they can’t directly mistakenly crash a plane, only a pilot can do that.

I think we see these two levels of model training in AI. We have reached the “qualification” level where modes are very good at tasks, they know A LOT (basically have a bachelors degree in everything) and they are also very intelligent. But they aren’t trusted to autonomously perform dangerous actions. The next level is where it gets super technical - being able to develop so much trust in a model, that you trust it to autonomously perform potentially dangerous actions. I.e. training an AI model to the extent you would give them a license. Perhaps unsurprisingly, the only people that are pursuing this seriously are the European Aviation Safety Agency. They see the benefit of allowing neural network based systems to autonomously make decisions that have the potential to result in loss of life. So they have drawn up a very comprehensive, very technical methodology in how they would like to see deep neural networks architected, trained and verified in order to apportion such trust. Now this is also assuming that they are placed into a wider system that manages erroneous output even from such well made and aligned AI models.

Remember how before I said that 99.9% of organisation aren’t going to train their own models? Well on top of this, I’d say 99.999% of organisations aren’t going to build custom neural networks that are designed and verified in a certifiable manner so they can be apportioned a license level of trust in their outputs. That is why having an AI safety management system is so much more useful and practical for everyone else. It allows the use of generally available models to do risky things so you can capture all of the upside and value of these activities while minimising the danger and downside.

There is certainly more to this comparison between human training and model training and I could have said a lot more. But I’ll leave this post where it is for now. It will be helpful next to discuss how this training analogy can be applied to the systems level of safety rather than model alignment in future posts.