Experiment adopted

[ADOPTED EXPERIMENT]: Health Malawi - Validating Systems Ability to Act Independently of PEV - [3/3]


Lean Phase: Solution


Time Box: February to July (6 months)


Assumption: To ensure the system is reliable and accurate without Project Everest intervention.


Success Metric:

Percentage of symptom only triages that match face-to-face consultations after the system is used without direct PEV engagement with patients.


Green light: Integrate algorithm into product


Success point: 95%, As both experiment 1 and 2 ran at a 95% confidence interval, so should the system in experiment 3.


Orange light:  Ensure instructions were clear for both clinicians and patients and check poster for design flaws. Run experiment for another month.


Failure point: The failure point for this design is 80% as a high accuracy is crucial for the system to go forward. Anything below 80% will involve research into advertising, communication methods, user interface and cultural differences.


Red light: Look at the channels used including advertisement, communication methods, user interface and cultural differences and/or sensitivity. Follow this with a redesign of any flaws located within the experiment. After a rebuild, the experiment should be conducted again with the same success metric.  


Experiment Build:



For this stage of medical testing, the system will run without direct interference from PEV in all clinics that are leads. Both the clinicians and patients will be using our USSD systems to record their results which can then be compared over a six month period. The result being compared is the triages recommendation for clinic visitation as well as the doctors. The goal of this is to not only test the system long term usability but to collect enough data to input into the machine learning algorithm.


Proposed learning

To validate that the system is functional  in clinics without interference from PEV


Medical goals

The medical goal for this experiment is to validate the MVP’s usability without interference or the use of translators. After receiving COMREC approval we will be able to test our current MVP with real patient data to test, validate, and improve on the current system.


Methodology and data collection


Before clinic visit

  1. Get approval to run experiments

  2. Organise time to set up and run experiment in clinics

  3. Brief clinicians on use of system

  4. Organise/ explain how data will be collected

  5. Create sign for clinic - number and outline of project

  6. Establish contact with head clinician liaison and organise contact schedule


During protocol

  1. Patients enter the clinic and are prompted by signage to start the system on their mobile phones

  2. Patients enter code on wall to start up the system

  3. Patients sign consent agreement within the system

  4. Enter age

  5. Between ages 16-18 parents will sign consent form for them

  6. Patient ID number sent to patient by text

  7. Patients use system then visit clinician

  8. Patients give clinicians their ID number

  9. Clinicians complete their own USSD system survey about their patients and their need to come into the clinic

  10. Contact PEV every two weeks to make inquiries and give updates



All data will be sent to the online database ‘Africa’s talking’ to be stored and processed. It will be used to determine whether the system is medically accurate and user friendly without PEV intervention. The information collected will include a comparison of the systems result versus the clinicians recommendation (the necessity of  treatment). Along with the patients inputted data, no identifying details will be stored. From this data we will then look at the different trends within to identify issues present. Once this experiment is given a green light we will integrate the data points and algorithm within a machine learning process.

Medical results

Data will be collected within the online database. This will average out answers and convert them to percentages for a simpler read. The recommendation of the medical professionals will be assumed correct, however it is important to note that their response is not going to be 100% accurate. At this stage we will not be differentiating between the results given by the different occupations (nurses, clinicians, doctors) and will be unable to take into consideration the different experience levels of individual clinicians. If the results reach the green light number of above 95 %, this will be deemed a success.

Tagged users
edited on 27th January 2019, 06:01 by Ella Grier

Gabriel Raubenheimer 1 month ago

Status label added: Proposed Experiment

Reply 1

Ella Grier 3 weeks ago

Status label added: Experiment adopted

Reply 0

Ella Grier 3 weeks ago

Status label removed: Proposed Experiment

Reply 0