This is the first in a series of blogs detailing the work of Sortition Foundation’s Sortition Services team.
Our Sortition Services team provides recruitment services for citizens' assemblies (CAs) and other deliberative democracy events. Typically a local council or other governing body will decide to hold a CA and they will need to buy in expertise on two distinct fronts: (a) they need an organisation which can run the deliberative process (the meetings etc that make up the CA itself); (b) they need an organisation which can recruit participants to the CA. The SF Sortition Services team are experts in this second component - CA recruitment - and this is the service which we provide.
What makes CA recruitment challenging is that the participants in a CA should form a "mini-public". That is to say, they should be representative of the community in which the CA is being held. So, for instance, if Hobbitshire council wants to organise a CA with 40 people on a certain topic, then they may ask SF to recruit 40 people who reflect the community of Hobbitshire as far as age, gender, ethnicity and education are concerned: it may be that, say, 25% of the population of Hobbitshire have a university degree and so the organisers may want 25% of those 40 people to have a degree (so 10 people in all).
In formal terms, the process we have just described is called stratified selection (or stratified sampling): our aim is to recruit participants to the CA in a fair and random fashion… but subject to the constraint of being representative of the community.
This sounds fairly straightforward but, delve a little deeper, and one quickly encounters some interesting political, mathematical and technical conundra. The aim of this post is to describe some of the complexities involved in recruiting such a mini-public. In the descriptions below these complexities are framed as questions and some brief clarifying remarks are also included. In the coming weeks we will post a series of blogs addressing these questions and explaining how the SF Sortition Services team has responded to each issue.
The questions we consider are concrete and practical. Still, it is important that our responses are informed by theory and by the growing literature around the legitimacy of CAs and other deliberative processes. The guiding principle here is that if we want people to trust the legitimacy of the CA, and the decisions that come out of it, then all aspects of the CA, including recruitment, must be demonstrably fair and transparent. We will return to this theme at the end.
- What are the right socio-economic and demographic categories to choose for the event? We mentioned gender, ethnicity and education above but who is to say that these are the best categories to use? There are an unlimited number of possible categories and we have to have some kind of limit on the number or it will be impossible to fill all the quotas.... So which categories should we choose?
A brief aside on terminology: We’re going to use the word "category" to describe demographic markers like gender and ethnicity as well as, potentially, attitudinal and behavioural markers (e.g. “who did you vote for?” or “are you concerned about climate change?”). Each of these categories has a set of possible values (e.g. gender can be male / female / non-binary and so on) and we will refer to these as the category-values.
- What are the right category-values? Or, put another way, what is the best way to divide a category? For many categories there are many different ways to set up an array of category-values: ethnicity and gender are two good examples of categories for which there are a myriad of different possible category-values. We'd like people to be able to identify their own category-value as accurately as possible but we need to limit the number of possibilities or we can't fill our quotas. And this leads naturally to...
- What do we do about small category-values? Suppose we are recruiting 40 people to a CA from a population in which only 1% of the population is Asian; this would correspond to 0.4 of a person. Do we round up or down? And if we round up and so have 1 Asian person in the room, then a number of questions occur: Is the single Asian participant going to feel comfortable to speak up if they feel that they are somehow "the odd one out"? Will they feel that their inclusion is “tokenistic”? And, to what extent, does any individual participant in a CA represent any given community of people? What does representation mean in this context?
- How do we deal with category-value intersections? Suppose that age and gender are 2 of our categories for a particular CA. Our aim is to recruit a representative number of male / female / other-gender participants, and also a representative number of participants aged 18-25 / 26-40 / etc. Ideally we would also want to check the proportions of Males aged 18-25 / Females aged 18-25, and so on. However keeping track of all intersections is typically too demanding from a computational point of view, and so we may have to focus on getting the numbers right for only a selection of intersections.
- What do we do if our census data is imperfect? In the examples given so far we've assumed that when we come to recruiting for our CA we know, say, what percentage of our population are disabled. But what if this information is not easily available? A good current example pertains to the UK census data on gender which in 2011 allowed just two possible answers: male and female. In the 10 years since 2011 a lot has changed with the way people identify their own gender -- how do we allocate a representative array of participants that includes people who identify as non-binary?
For the next couple of conundra we need a little background on SF's usual method of recruitment. Say SF has been tasked to recruit for a 40-person CA in Hobbitshire. The recruitment process has two stages:
- Around 10 000 letters are sent out to a random selection of addresses in Hobbitshire. (We usually recommend sending between 200 and 300 letters for each prospective participant, depending on the nature of the specific CA in question.) These addresses invite people to register themselves (online or by phone) as possible participants for the CA and to give us the necessary data on gender / ethnicity / etc that will allow us to make a selection. We usually hope to get at least 300 registrations this way.
- Once the registration period has ended we use our selection software to randomly select 40 participants out of these 300 respondents. Our software does this random selection in such a way that we get a representative profile for all of our chosen categories. To be confident of hitting our quotas for all categories it’s important that the number of respondents is significantly larger than the number of participants.
So, now, we have more questions:
- What do we do about large category-values? Suppose in our pool of 500 respondents we end up with a disproportionately high number of older people and a disproportionately high number of well-educated people. If we are not careful, then this could mean that a well-educated older respondent, say, is much less likely to be selected as a participant, as compared to the complete pool of respondents. This observation represents a threat to the perceived legitimacy of our CA: a fair selection process must not exclude any individual respondent as a consequence of their demographic profile. How can we make sure that our selection algorithm satisfies this measure of fairness? It turns out that this question can be answered in a very precise way, as we shall explain in a future blog post.
- What do we do about excluded voices? A key principle of CAs is that all voices should be heard, including (especially) those voices that aren't usually in the conversation. Sortition explicitly seeks to correct this by selecting a representative array of participants, however for this to work effectively we need our pool of respondents to include people from all walks of life. But now exclusion works on many levels: SF have noticed, for instance, that the rate of response (i.e. the percentage of people who register out of all of the people who are sent a letter) amongst people in poor neighbourhoods tends to be considerably lower than the rate of response among people from more affluent neighbourhoods. Could it be that people in poorer neighbourhoods are so used to their voice being ignored in the national conversation that they simply assume the same will be true in a CA? How do we correct for that? What about other groups of people corresponding to other category-values?
The discussion above has focused on some of the subtleties of recruiting a mini-public for a CA. However one cannot entirely divorce the recruitment process from the other things that make up a CA including, in particular, the substantive issue(s) to be considered by the CA as well as the deliberative processes that will take place once the CA is underway. For instance, at our final bullet point above we considered how to deal with excluded voices in the recruitment process of a CA, and we should ask the same question when considering the deliberative process.
With regard to recruitment specifically, we have framed our discussion around the problem of ensuring that the participant group is demographically, socio-economically and (perhaps) attitudinally representative. However other factors may also be important in recruitment, for instance, ensuring that the group includes voices who are more affected by the substantive issue under consideration.
All this notwithstanding, it is probably still fair to say that the most essential outcome of the recruitment process is that it results in a wide cross-section of voices: that participants bring a wide array of experiences and perspectives to the table. In light of this we might perhaps regard some of the questions above as being excessively pernickety: no particular CA is going to stand or fall depending on whether we round up or down for a particular category-value.
More profoundly, it is obviously nonsensical to think that all poor, able-bodied, white people with a degree (say) will have the same opinion on any given topic. Thus calculating the probability of choosing a particular person with any precise given profile of category-values does not give us any meaningful indication of how effective a CA is going to be.
On the other hand, given that we really do want CAs to allow for participation from people from a wide range of backgrounds, it seems sensible to use the data we have available about our population and the people in it. And the way we use this data is important: our recruitment processes and selection algorithms need to be demonstrably as fair as they can be so that people can trust the legitimacy of the CA and the decisions that come out of it.
This is where the SF Sortition Services team comes in: we spend (some of) our time thinking about the minutiae of these processes so that others don't have to! Not everyone who organises a CA has the time and wherewithal to work through all of the factors that may affect the validity of their recruitment process. Instead the SF Sortition Services team have done this, as we will describe in future blog posts. We will see how the questions above (and many more) have been considered and addressed in the way that we recruit for CAs. Ultimately we aim to demonstrate that it is possible to recruit participants to a CA in a fair and open fashion.