Who you ask changes the number you will act on

Ask one person how a process runs and you have an account. Ask a second person, in a different position, and you often have a different account of the same process. Who Answers Changes the Number treats that gap as a property of organizations, not as noise to be removed by a larger sample of the same kind of respondent.

It shows up in official statistics. The job title of the person who fills out a U.S. Census survey moves the reported rate of AI use by ten points after controlling for firm size, industry, and state. It shows up inside a single site. In the largest process-safety culture survey conducted at U.S. refineries, operators and operations managers at the same refinery, answering the same item, were more than twenty points apart. The pattern is not that one form was filled out carelessly. It is that position changes what a person can see and what they are willing to say.

Averaging informants is justified when they are estimating the same quantity and their errors are independent. A process owner and an operator meet neither condition. They are not estimating one quantity. The owner is often describing the process as designed. The operator is describing the process as carried out, including the exception that never re-entered the system. Their errors are not independent, because each is produced by the role.

A diagnostic that interviews only the people who own the process will recover the designed process and call it the operation. A diagnostic that interviews only the people on the shift will recover the exceptions and miss the constraint the design was trying to enforce. Both are incomplete. The error is to add them together and report the mean as “how the work is done.” The mean is a third description, and it is not one that anyone in the organization recognizes.

The paper’s working rule is to assign authority by the type of claim. The person accountable for the design is the authority on the process as intended. The people who execute it are the authority on the process as done, including the workaround, the delay, and the channel that is not the system of record. When those accounts conflict, the conflict is a finding. It is evidence that the intended process and the executed process have come apart, which is usually the fact a later automation decision needs.

Who to interview, then, is not “the expert” and not “as many people as possible.” It is at least one person who owns the process and the people who run it when it leaves the plan, asked about specific recent episodes rather than about how things generally work. The paper does not claim that an interview conducted outside the line of command, including one conducted by a model, leaves those answers unchanged. It lists that as unmeasured. What it does claim is that a number produced by averaging across positions is the wrong input to a decision about where work actually breaks.