Project Everest

NAYO Clinic Ada Application Study Design

by
Josh Marshall
Josh Marshall | Nov 30, 2017 | in Health Consulting

The Health team in Malawi have made substantial progress in the opening days of their December project. Specifically, the team has re-engaged with contacts at the NAYO clinic, a local health clinic providing various health services within rural areas of Malawi. The clinic has been very interested in the the team’s idea of a pre-diagnostic health application, and have agreed to allow for the testing of one such application called Ada in their clinic within the following weeks. The team first outlined the parameters, process and methodology internally, before further discussing these with the staff of the NAYO clinic to gain insight to their opinions of the project.

 

In terms of parameters, the team decided to focus on accuracy, efficiency and usability. The team then defined each of these parameters, and then evaluated how these would be measured. The team defined accuracy as the ability for application to correctly diagnose patients, which will be assessed through comparison of the Ada diagnosis with clinician/nurse diagnosis, and then conducting hypothesis testing using a paired t-test. The team defined efficiency as the ability for Ada to reduce time in the diagnosis process, and plan to measure this by comparing the time spent in the clinic with the time using the application. For time spent during the clinic, the team will take measurements of both the entry and exit time to the clinic, and for the Ada application, recording start and finish time of using the application. Although these times will overlap, the team does not expect them to impact each other due to the long lines identified at the NAYO clinic during the health team’s initial empathising stage. For usability, the team has implemented a questionnaire at the end of the entire process, to understand the ease in navigating the Ada application. This questionnaire is still to be finalised, with a following post discussing this in greater detail.

 

When testing the Ada application, the team wanted to make the process as effective and efficient as possible for the NAYO clinic and its patients. The team were weary of the possibility of creating a greater workload for the staff at the NAYO Clinic, and therefore committed to the least disruptive process applicable. Patients will enter the NAYO Clinic and proceed to registration, where they will be asked to participate with the trial of the Ada application. We acknowledge that taking patients from within the queue may cause confusion and greater congestion, and therefore will question patients from the back of the line to minimise this disruption. Patients will receive an ID Form and lead to a separate room where the Project Everest team will be waiting. The ID Form enables privacy and confidentiality as no names will be recorded during data collection. The form will also include four designated time slots, as explained above. An explanation of the application process will be provided and patients will be guided through each question. At completion of the questions, a screenshot will be taken of the suspected diagnosis for our data records, however we will not provide this information to the patient. This is to ensure that the patient does not carry a bias when seeing the nurse or clinician and therefore impact their judgement of the overall diagnosis. Patients will have the option to access to their Ada diagnosis following completion of their consultation with the nurse or clinician.

 

An obvious issue to the entire process is the language barrier in the local community, with the majority of people speaking the local language Chichewa and not English. As none of the team speaks Chichewa, the team requires translators to translate Ada for the patients. Currently, the team has at least one volunteer from the NAYO clinic to help with this, and potentially another depending on available resources at the clinic. Another idea the team had to overcome this issue was to develop a partnership with the local College of Medicine and gain access to Chichewa and English speaking students to help with translation. This would be beneficial for both parties, as the team would then be able to increase total data collected, while students at the college gain invaluable education, understanding and experience associated with working in rural areas. The team is currently in the process of contacting the college to develop this relationship.

 

The December health team is very excited about the current avenue they are exploring in terms of using the Ada application as a pre-diagnostic tool. The team believes that if successful, the tool can help reduce the many barriers to healthcare that have previously been identified in the greater Blantyre region of Malawi. These include a reduction in patient waiting times, a reduction in long patient travelling times as well as the potential to even improve accuracy of clinic diagnoses, as suggested by a nurse at the NAYO clinic. Hopefully through testing Ada at the NAYO clinic in the coming weeks, the team will gain a better understanding of the viability of both Ada and a pre-diagnostic tool in the future.

Liam Donovan Nov 30, 2017

What a tremendous idea. Who helped you with this???

Reply 2

Alessanda Oliveri Nov 30, 2017

So excited to see where this goes!

Reply 1

William Lee Dec 2, 2017

Hey guys,

Love the progress being made on this, and the goals/objectives look awesome.

How many people come through the NAYO clinic each day? i.e. what's the sample size you guys can get over a day/week? Are you guys planning to have just one version of the questionnarie?

The reason why I ask this is because the while the length of the questionnaire will help understand the patients' issues in greater depth, it would also add to the time and efficiency of the diagnosis system.

So, given this tradeoff, (and if you were to have enough of a sample size), you could test out questionnaires with differing lengths to see how many questions you would actually need to get a pre-diagnosis that is of adequate accuracy.

So instead of testing one version for a week, iterating on it and putting out a revised version the next week and so on, you could have multiple versions of the survey at once and see which version has the best tradeoff outcome. This would speed up the process of finding that optimal balance.

Let me know what you think.

Reply 0

Edan Baker Dec 4, 2017

Hi Will!

Love the critical thinking!

It's a bit hard to understand what you're referring to when you say 'the questionnaire'. Ada is a complex system that asks many many different questions based on inputs from the user. I'd super suggest downloading it and having a look. So, Ada is not just a questionnaire. But I do understand what you're getting at when you're talking about comparing different questions being asked to get the right balance between time and accuracy. So on that point, firstly it is important that accuracy is key here when dealing with people's lives. Secondly, it is significant to point out that if Ada is receiving inputs that require more questions to be asked, then the condition is most likely more complex and would require a 'real life physician' to ask more questions anyway, HENCE the impact on timing would be null, UNLESS the accuracy is off and the diagnosis from Ada took an unnecessary amount of time than it actually needed... cause it was wrong. Hopefully that makes sense.

The only section of the project that involves a questionnaire is the usability section but that comes right at the end of the whole process, therefore not impacting the time and efficiency measurements because we put the END TIME of the usual process as ending at when a diagnosis from the 'real life physician' is made, and not when the patient walks completely out of the clinic.

Edit: the sample size for NAYO clinic is approx 100 patient's per day. They are open from 09.00-13.00 on Monday-Thursday. Monday and Tuesday are essentially 'GP clinic' days, in that they see any and all patients. Wednesday is an out-patient palliative care day where patients who are mobile and require palliative care are seen. Thursday is an out-patient community palliative care day, where they go out to patients in need of palliative care who are immobile. This therefore reduces the weekly maximum sample size to 200 a week since interacting with palliative care patient's seems unsuitable both from a data standpoint and ethical standpoint.
The team used a different clinic today called the Pensulo clinic, which has the same patient numbers per day but longer days and 5 days a week. The team will be focusing on that clinic until NAYO clinic can be used again (they are currently "closed" as they have no medication).

Reply 2

William Lee Dec 4, 2017

Thanks for all this Edan! Just downloaded Ada, and looks like an awesome app.

How closely are you guys engaged with the developers behind Ada, so that in the future you might roll out a version in Chichewa?

Also, out of interest, how does the Ada app make money? Would it/does it charge PE a fee for using their service on a larger scale?

Reply 0

Edan Baker Dec 12, 2017

Hi Will, sorry for the late reply, there have been some internal developments regarding our partnerships and future with various applications (namely, Ada & Your.MD).

We have had email contact with Ada in the lead up to project and a bit during project in terms of showing them our study design, sharing survey tools and exploring future relationships. We've identified that they are very attached to their own vision of the product and relatively inflexible with adapting to proposed changes. After some deliberation as to what the Malawi market will need in terms of what these types of applications will need to deliver, we've decided to switch over to testing Your.MD. A large part of that decision has to do with the flexible nature of the company and our ability to make changes in the future, and potentially develop our own app.

Reply 1

William Lee Dec 12, 2017

Fair choice; was their anything about Ada that made it your first choice? or, to put it another way, what can Ada do that Your.MD cannot? and how will those limitations affect its use in Malawi?

Reply 0

Edan Baker Dec 17, 2017

Ada is a better product in terms of usability and has a massive team of expertise behind it. The accuracy between the two is similar but usability plays a big role in the symptom assessment process. Imagine having two doctors, one that you have a good rapport with and find it easy to understand and answer their questions, and one that is hard to communicate with. You would undoubtedly want to see the one who you would find easier to communicate with.

The unfortunate thing about Your.MD is that it would be harder for people to use, especially when English is not a first language and when they are not always used to interacting with these kinds of applications. The benefit about testing a worse product however, is that you have a lot of room to improve it! lol. We'll be posting soon about how we can counter-act the barrier of it not being in Chichewa and having people who may not know how to use it. The idea actually compliments a business model to generate what we think should be the revenue stream for the end-product.

I understand it may be a bit counter-intuitive to go with what can be seen as the worse product but the potential in development is greater, easier to obtain and more feasible to tailor the product to the Malawi.

Reply 0

View all replies (5)

Samantha Fisher 1 month ago

Wishing you the best of luck for all your blogging efforts.This is my first opportunity to chat this website. I found some interesting things and I will apply to the development of my blog. Link Building Service

Reply 0