Adopted Experiment

[PROPOSED EXPERIMENT]: Health Malawi - Research Protocol Stage 1A

Research protocol (RP) stage 1 is divided into two smaller components: RP1a and 1b. This is because they examine and assess different aspects of patients’ ability when communicating symptoms. They are included together in stage 1 because the methodologies are performed simultaneously. Statiscally, 246 data points are required to obtain a reliable result from our correlation tests.


*RP1a and RP1b are conducted in conjunction with one another*


Lean Phase: Solution

Assumption:  246 patients are able to communicate their symptoms to Project Everest research assistants (PERA) in the same manner as they would to clinicians. In addition, RP1b assumes that patients’ health literacy are sufficient enough to make full use of the self-triage system.

Time Frame: 18th March to 29th April 2019

Success Metric

Success Metric Point: a kappa score above 0.4 and p value > 0.05 are observed for all symptoms


Green Light: Proceed to the next stage of research protocol.


Orange Light Point: a kappa score of 0.4 or higher and p value < 0.05 are observed for some/all symptoms


Orange Light: Assess the degree of correlation (using kappa score) qualitatively. This would determine whether the statistical significance is valid in a health setting.


Red Light Point: a kappa score below 0.4 is observed for some/all symptoms and p values < 0.05 are also observed for these symptoms.


Red Light: Identify why a significant difference in response was observed for the symptom(s). Determine way(s) how the research protocol can be re-designed to improve the correlation.


Experiment Build

To assess correlation between how patients communicate their symptoms to PERA and clinicians, we provide patients with paper surveys before their consultation with clinicians. In the survey, they will be asked to identify their symptoms with the visual aid of a symptom description sheet, and the duration of each symptom. During or after each consultation, clinicians are asked to record what symptoms patients have described to them, regardless of their importance and relevance in diagnosis. Ideally, to establish as much fairness in the experimental design as possible, translators will be available to allow patients to verbally describe their symptoms, which will be then recorded by PERA. This is so that their written ability in completing paper surveys would be eliminated as a variable.


Correlation Assessment

Despite using the same methodology, correlation assessment is performed separately for RP1a and 1b.


Results from both RP1a and 1b are sorted separately into contingency tables and subsequently analysed using kappa-score test. The kappa score for each symptom describes the degree of correlation based on criteria previously designed in other studies in medical research and health.

However, kappa scores are often associated with systemic bias in that a sample pool with high correlation may generate a low kappa score due to the nature of the sample pool. To account for this possibility, we employ a second form of assessment - the McNemar Test. McNemar Test measures statistical significance of correlation between two categorical data sets that have binary outcomes. Thus, by using these two assessments, we can quantify similarities as well as differences between patient and clinician survey responses with minimal bias.

edited on 25th February 2019, 19:02 by Lucy Noble

Lucy Noble 3 months ago

Status label added: Experiment adopted

Reply 0