Experiment adopted

[EXPERIMENT ADOPTED] Health Malawi - Validating Accuracy of Symptom Based Triage - [1/3] - Medical Accuracy

[EXPERIMENT ADOPTED]: Malawi - Symptom based triage compared to face-to-face consultations - December 2018

Lean Phase: Solution

Assumption: Symptoms based triage can be accurately similar to face-to-face triage.

Time Period: Week 3 December 2018 Projects to Week 2 of February Projects 2019. 

Metric: Percentage of symptom only triages that are the same as face-to-face consultations.


Green Light: Proceed to test #2, found at:

Success Point: 95% accuracy of symptoms only triage in comparison to face-to-face consultations. Due to the nature of the application and the implications of those using if the application is not accurate, it is necessary to ensure that a symptom only triage is as accurate as a face-to-face consultation with a high degree of confidence.

Orange Light: Optimise the accuracy by adding more symptoms into the list of those provided to ensure that anything patients may not think is relevant can be included.

Failure Point: 70% or less accuracy between symptoms only triage and face-to-face consultations.

Red Light: Identify why there is such a discrepancy between the symptom only triage and the face-to-face consultations. Change the symptoms provided on the surveys. From this, the point at which the clinician completes the symptom only triage surveys may need to be changed.


Experiment build:

In order to ensure that a patient being triaged based only on their symptoms (i.e. without vitals results) is as accurate as a face-to-face consultation (i.e. with vitals results), a survey will be created [see attached file], whereby a patient fills out what symptoms they have and the symptom duration before seeing a clinician. At the end of the day, the clinician will then triage all the patient symptom surveys to ensure that they are not providing the same diagnosis simply because they know what it is.

A secondary survey will be created [see attached file] that the clinician fills out during the face-to-face consultations to ensure that the patient is providing the application with the same symptoms that they are providing to the clinician.

A total of 385 data points are required to ensure that they results are statistically relevant. These responses will be sourced from a variety of clinics within the Blantyre region, namely the Zingwangwa Clinic and Gateway Clinic. Ideally, a secondary testing area will be identified and the experiment carried out to ensure that the desired results can be achieved in a number of locations.

PROPOSED LEARNING: To validate that symptom only based triage (i.e. without vitals results) is as accurate as a face-to-face clinician consultation (i.e. with vitals results) AND to ensure that patients would provide the current ‘solution’ with the same symptoms as they would provide to their clinician.

Our MEDICAL GOAL for the month is to “Validate that the current MVP will provide the same result as clinicians at least 90% of the time”.

After receiving COMREC approval we will be able to test our current MVP with real patient data to test, validate, and improve on the current system. We plan to partner with clinicians to allow us to get patients to complete their surveys, with the assistance of a translator, and then have the practitioner state whether their visit was necessary and what the patient's diagnosis was. The patient will not see the result or the practitioners comments to reduce interference with the practitioner's diagnosis.

DATA Collection Method -

BEFORE visiting clinic:

  1. Modify surveys to minimise bias and ensure we are getting the best and most effective data possible in order to execute this program.

  2. Organise our execution by ensuring that we have knowledge of where each survey is going and who is required to fill them out.

 START of clinic visit:

  1. Get all practitioners involved to read and sign the consent document.

  2. Show practitioners how to complete the patients identification form, ensuring they know how to assist patients filling out their own forms and how to fill out the clinician form.

  3. Explain/show translators how the questions work and how to input the data.

DURING the clinic visit:

  1. Ask patients if they would be willing to participate in the experiment - Get those who most recently arrived and have not yet registered (this is to avoid confusion or logistical concerns, and remove any risk of them missing their appointment).

  2. Explain the research and their role-requirements to patients in their native language - Patient reads and signs consent form.

  3. Patient is assigned an ID number and given an ID form - The ID form will also include entry and exit times of clinic.

  4. Translator guides patient through the user symptom survey.

  5. Patient begins consultation as usual, bringing with them their ID number.

  6. Clinician retrieves patient user symptom survey, to compare during consultation.

  7. Clinician fills out ‘clinician triage survey’ in order to compare symptoms and vitals with the triage.

  8. Clinician fills out clinician survey at the end of the day after all surveys from patients have been collected.

AFTER the clinic visit:

  1. Compare the symptoms only triage  to the results of the face-to-face consultations.

ANALYSING The Results:

A spreadsheet has been established to be able to input what symptoms patients say that they are experiencing [see attached file]. On the same spreadsheet, a way to collate whether or not the the patient is telling us the same symptoms (on the pre-consultation survey) as they are during the consultation with the symptoms. The final purpose of the spreadsheet is to determine if symptom only triage is the same as face-to-face triage. A formula has been created so that the accuracy progress can be traced so that modifications can be made during the experiment if needed. 

MEDICAL results:

Data will be inputted in an excel spreadsheet using the four sheets developed for each survey. Averaging out the answers and then converting them into percentages for a simpler read.

The recommendation of the medical professionals will be assumed correct, however it is important to note that their response is not going to be 100% accurate. At this stage we will not be differentiating between the results given by the different occupations (nurses, clinicians, doctors) and will be unable to take into consideration the different experience levels of individual clinicians.

Quantitative data may include information such as regularly visited locations, mode of transport, access to phones, and willingness to pay a small fee to use the service.

If the results reach the green light number of 95%, this will deem success and will mean that our experiment has worked and we can move to the next stage of the product execution.

edited on 21st January 2019, 04:01 by Gabriel Raubenheimer

Gabriel Raubenheimer 2 months ago

Status label added: Proposed Experiment

Reply 0

Samantha McKechnie 2 months ago

What a great post Gabe really enjoyed reading it

Reply 2

Ella Grier 2 months ago

Status label added: Experiment adopted

Reply 0

Ella Grier 2 months ago

Status label removed: Proposed Experiment

Reply 0

Andrew Vild 2 months ago

This is good, it just needs a goal or number of data points required for it to be an accurate or significant data set.

Reply 0