Quantitative epistemology: conceiving a new human-machine partnership

I'm very excited to announce a major new research direction for our lab: quantitative epistemology. We believe this is an area of study that can help us define a new human-machine partnership—in healthcare and beyond.

For a long time, I've envisioned a strand of machine learning aimed at understanding, supporting, and improving human decision-making. We've recently been working on this in piecemeal fashion and it's something I talk about a lot, but I haven't been able to link it together in writing—until now.

What is quantitative epistemology?

Quantitative epistemology involves building machine learning models that can capture how humans acquire knowledge and turn that knowledge into beliefs, and those beliefs into actions.

We can then use these models to identify potential suboptimalities in beliefs and decision processes (e.g., cognitive biases, selective attention, imperfect retention of past experience) and their implications for learning and decision-making.

In doing so, our goal is to construct decision-support systems that provide people with information pertinent to their intended actions, their possible alternatives and counterfactual outcomes, as well as other evidence to empower better decision-making.

This is distinct from imitation learning (replicating expert actions) and apprenticeship learning (matching expert returns): rather than constructing autonomous agents to mimic/replace human demonstrators, we want to use machine learning to help humans become better decision-makers.

The AI/ML community has spent a lot of time developing methods to imitate or compete with humans, but there's so much potential for methods that can help us study and improve ourselves. Again, the focus here is on assistance, not replacement—a new human-machine partnership.

So, why "quantitative epistemology"? Because our approach involves the study of knowledge through observational data, as well as using machine learning methods to support and improve knowledge acquisition and its impact on decision-making.

Potential applications of quantitative epistemology

Broadly, we currently see four potential areas of application for quantitative epistemology, none of which are limited to healthcare:

1. Decision support

This is arguably the most intuitive and straightforward application of understanding human decision-making. In medicine, for example, we can combine a meaningful understanding of the basis on which decisions are made with normative standards for optimal decision-making in areas such as diagnosis, treatment, and resource allocation.

Furthermore, we can apply quantitative epistemology in a single-agent or multi-agent setting, using our understanding of decision-making to optimize decision-making across multiple individuals or groups, whether in a co-operative or a competitive setting.

2. Analysis of variation

In many fields such as healthcare, there is often remarkable regional, institutional, and subgroup-level variability in practice. This variability renders detection and quantification of biases crucial.

Quantitative epistemology can yield powerful tools to audit clinical decision-making to investigate variation in practice, biases, and sub-optimal decision-making, and understand where improvements can be made.

3. (Re-)definition of normative standards

There are many areas in which normative standards have not been defined, or may need to be continually redefined. Through the application of quantitative epistemology, we can determine whether normative standards are realistic and effective representations of desired outcomes, enabling policy-makers to design better policies going forward.

4. Education and training

Quantitative epistemology aims to produce a data-driven, quantitative—and most importantly interpretable—description of the process by which humans form and adapt their beliefs and understanding of the world. This could yield enormous benefit in education and training: both the content and instructional methods employed in courses could be extensively tailored to specific individuals, taking into account their learning styles, biases, and preferences.

While the benefits for healthcare are obvious (decision support, improved training, clinical audit, consistency in practice, new clinical guidelines), this field of study is relevant wherever important decisions are made (so, everywhere)!

Intersection with other areas of research

Our lab has already committed substantial long-term resources to quantitative epistemology, and will continue to do so. Our work here will complement and build on projects across the lab’s research areas, including decision support systems, predictive analytics, AutoML, ITE inference, interpretability, synthetic data, etc.

Find out more!

This post represents an extremely truncated introduction to quantitative epistemology. More information, including real-world examples, projects conducted so far, and plans for further work, can be found on our website (via the URL below).

We are also planning an engagement session on this topic in June/July, so please visit our site and sign up if you're interested!