projecteverest

Adopted Experiment

[JAN 19] [COLLECTION] offer testing to [Large hotels(a), locally owned restaurants/hotels (BagPay customer segment)(b), schools(c), Organisations with subsidiaries (govt, supermarkets, stores)(d)]

Lean Phase: problem

Assumption: [hotels, locally owned restaurants/hotels (BagPay customer segment) schools, organisations with subsidiaries] in the Dili area and surrounds would be interested in a solution to address the waste management problem in their community

Time frame:2 week time frame

Success metric: success will be measured by the number of people who are interested in or actively looking for a waste management solution.

Post-experiment actions:

Success point: if 15% of the responses are positive, that they are interested in a solution to these problems, the experiment is considered a success. 

Green light Proceed: Move to currency testing with this segment to determine willingness to pay for a solution.

Orange light optimise: Speak to the customer segment about what kind of problems they have around the waste management issue and re-evaluate the problem definition. Repeat offer testing with revised problem set and possibly new potential solutions if promising. If they are not actively seeking a waste management solution it will focused on an understanding of pain points around waste management

Orange light point: If less than 15% but more than 7% of responses are positive then the experiment needs to be optimised.

Red light failure: Re-evaluate the problem definition and look at whether the core issue is not waste or waste burning but something related to but separate from those.

Red light failure: If less than 7% of the responses are positive the experiment will be considered a failure.

The experiment

Ask questions to validate whether the problem is understood well by us or not. That Timorese people have issues surrounding waste that can be solved by offering a recycling service.

The script would run as follows:

  1.  What do you do with your waste?

  2. Do you separate your waste? Do you know why you would separate your waste?

  3. What different types of waste do you have

  4. What is your most common form of waste? (glass, paper, plastic bottles, cans)

At the beginning of the conversation, we assume of one of these 3 pain points groups will arise:

Burning of waste

Waste on the streets

Plastic in the ocean

How do you get rid of your waste?

If they burn their waste

  • Where do you burn the waste?

  • Do you burn the waste with
     Others?

  • What is the smoke like? [do people get sick, how does burning the waste make them feel etc)

  • Have you ever considered doing something else? What other options do you have?

If someone collects their waste

  • Where do they collect the waste?

  • Do they collect all your
     Waste?

  • How much do you pay to have your waste collected?

  • How often do you get it
    collected? How reliable?

With their current solution mentioned above

  • Are you happy with this solution?

  • If you could change 3 things about x collection method

  • Have you heard of the
     concept of recycling? [explain recycling and foster a discussion]

-        How do you feel about the waste on the streets?

-        What do you think should be done about this?

-        Do you think there is another solution to this? What does it look like?

-        If there were 3 things you would change about the rubbish on the streets, what would they be?

-        How do you feel about rubbish on the streets and in the ocean?

Did you know that a result of plastic in the ocean, majority of the fish caught in Dili have plastic in their flesh?

 

*show photo*

- Were you aware this was a result of waste on the beaches?

- How does that make you feel now about plastics on the beach?

 

 After these are asked, OFFER to solve these problems. “We are looking at a solution to these problems, would you be interested?” It is important not to provide too much specific detail, only to gauge if they are interested in a solution in general, to understand if they are aware of the problem or not.

If yes, get some kind of call to action, could be a phone number, email address, or simply names and village for them to display some level of quantitative interest. If they are unwilling then that is okay and we might need to adjust our problem definition through more empathising.

If responses are positive and they are asking questions about what the solution would like, you can ask what kind of solution do they think would solve their problem. Allow them to come up with ideas first. Can then prompt them with the idea of a collection. Responses to that and also asking questions is regarded as positive results.

edited on 17th January 2019, 04:01 by Rose Gooding

Rose Gooding 5 months ago

Completed script accessible through https://docs.google.com/document/d/1tScOqwWow...asDU0zJWUU/edit

Reply 0

Harry Telford 5 months ago

Status label added: Experiment adopted

Reply 0

James Balzer 5 months ago

Good post and I'm a massive fan of the detail, especially in the Experiment Design itself.

I'd be curious to see if you could leverage Offer Testing via online platforms such as Facebook or do blast texts to businesses? I'm sure there'd be online groups that agglomerate businesses together in a digital space that you could appeal to.

Reply 0

James Balzer 5 months ago

Hi again sorry,

I'm just struggling to get to sleep with a storm raging outside my bedroom, so I decided to be productive and do another round of Crowdicity commenting on various projects, this time taking a more in-depth look at things.

Upon reading the December Handover document, there are a few noticeable problems that I think needs to be brought to the forefront when assessing the nature of this experiment. This is done with the best of intentions, and it's all done in a constructive mindset that is open to criticism on the below points.

In short, I'm not sure what the above experiment is seeking to test that hasn't been tested already. Likewise, based on previous ERS teams' efforts, I'm not entirely convinced that the assumption for the experiment is validated.

According to the handover document, ERS currently have 6 premium contracts and 3 BagPay contracts on board (9 in total). Throughout December 2018 alone, the ERS team contacted over 70 businesses, 41 of which actively engaged with them and only 6 of which actually signed contracts (15%) with 13 others still considering (32%). It's worth mentioning that these percentage figures are based off the 41 that actually engaged in empathy assessments with PEV, instead of the 70 that were contacted, meaning that the 29 that didn't reply aren't included in that percentage. It could be that they just didn't read their E-mails, but it could also be that they weren't interested in engaging with the service. By all means I would encourage the ERS team to re-contact these people who didn't reply, but overall the numbers don't seem to be that strong. This is especially if you consider the 70 businesses contacted in December also don't include all of the businesses that have been contacted by the ERS team in previous months, meaning that the number of businesses engaged and asked if they would like to engage with the ERS service is maybe around the 90-100 mark (although maybe I'm wrong, feel free to correct me). Out of those amount, the fact that only 9 have signed in total is a little worrying in my opinion.

This is especially considering that there aren't a massive amount of businesses in Dili, and less so that have a big enough waste problem that they are actively seeking to fix and also pay for. Ultimately, considering that there aren't a huge amount of businesses in Dili, the current amount of engaged/interested businesses along with the 15% green light goal seem like rather small. I'm just concerned that as a raw number, it won't cut it in terms of trying to run ERS at a profit, let alone scaling it in the long term.

This isn't to mention the fact that quite a few customers have dropped out of their initial contract. Admittedly, the experiment that seeks to get feedback from Bagpay customers may re-ignite the interest of some of these customers if you're able to re-develop the Bagpay system to suit them better.

In short, I'm not entirely sure what the purpose of the above experiment would be given all of the above that I've said, which is all based on information from the December handover. Feel free to correct me, but I feel as though you may find it more productive to do a few things based on the current situation. These include:

1) Re-contacting those businesses that didn't engage with you when you contacted them.
2) Re-contacting the 32% that said that they were still considering purchasing ERS' waste management service.
3) Engaging with educational institutions such as UNTL and other schooling places to see if you can get more socially/environmentally conscious younger people to pass on their institution's recycling to your business as a key customer segment.

I know this is a lot of information, and not all of it is necessarily cheerful. However, honestly stating the problems at hand is the catalyst for figuring out solutions, which is ultimately more beneficial in the long term.

Reply 1

Fiona Aaron 5 months ago

Great post Jimmy, and you raise some interesting points. I'm very keen to see an answer from the team?

Users tagged:

Reply 0

Rose Gooding 5 months ago

- The idea of fb and blast texts is an idea for channels we are considering and looking into, however we want to validate the other areas of the canvas before doing so, or we we will run this as another experiment
- All in all the December handover deals with a different angle of ERS, and one more geared to the old sales-centric approach. The main problem with this approach was that we were constantly signing on new customers, as old ones dropped off, clearly indicating that we weren’t properly understanding their UVP.
- The approach for January is to go back and validate the first 4 points of the lean canvas for 4 different customer segments. ERS started 2 years ago and this was never completed, something I believe is massively holding us back.
- The current stats from December are 7 customers, 5 of which are ‘on call’ which essentially means they call us when they need a collection. From my experience remotely managing ERS for 11 months, this has only effectively worked for one ‘on call’ customer we have signed with the rest never eventuating to a collection. This clearly indicates we are not delivering on their value prop.
- The strategy for doing this offer test is to understand how they view their problem and seeing how we can deliver on their UVP. I am open to the fact this is not likely to look like the service we are currently offering, and ultimately the outcome would be improving the conversion rate.
- From our market research (internal and external) we gave found 182 restaurants, over 90 hotels and over 140 schools (market segments black label post coming). For example, out of the restaurants, ERS has only contacted 36 EVER. This is a HUGE market opportunity if we can effectively understand their problems and deliver them value.
- The purpose of the experiment is to just see how these segments view waste management. Historically we have always viewed it as our own problem and our own assumptions and feelings (that burning plastic is bad) rather than areas related to their own problems such as how it makes them feel to see their kids walk over piles of rubbish when they walk to school.
- The 15% green light is based on the fact that we are looking at using relatively large sample sizes and the general percentage of early adopters being 13.5%.
- I can really see where you have derived these productive tasks, however with the new approach we are taking, the above makes sense.
- I hope this clarifies things, and I’m keen to see your thoughts after this explanation

Reply 0

James Balzer 5 months ago

Thanks for the reply :)

Based on that reply, here are some points of consideration. I've split it up into a "point" and "considerations" format, where I first quote one of your points above and then state considerations for those points.

Point: "The current stats from December are 7 customers, 5 of which are ‘on call’ which essentially means they call us when they need a collection. From my experience remotely managing ERS for 11 months, this has only effectively worked for one ‘on call’ customer we have signed with the rest never eventuating to a collection. This clearly indicates we are not delivering on their value prop."

Considerations: I would be careful about the 'on call' model of things. It appears to be a business practice that instigates a lack of consistency in the use of the service by your customers. This makes cash flow disjointed and unpredictable, making it a weak pillar of the business. Despite this, it currently forms 5/7 customers for ERS.

Likewise, it currently appears to be an invalidated assumption that "we are not delivering on their value prop." I understand that the above experiment seeks to test this, but be open minded about considering that multivariate reasons why businesses mightn't be continuing their involvement with ERS. They may understand all of the problems associated with poor waste management, but that doesn't necessarily mean that they're looking for a solution and even if they are, it doesn't necessarily mean that they're willing to pay for it, and even if they want to pay for it, they mightn't like your prices. Even if they like your prices, maybe they find the collection to be irregular, the service not up to scratch or maybe they had shortcomings in their business inflows that meant that they needed to cut back on their outflows. There are so many variables that need to be tested in order to understand why businesses mightn't engage with the service.

Even if you could validate all of that and fix the business model to keep your customers on board, how can you expand the LTV of the relevant businesses? Having cashflow inconsistency and insecurity due to unreliable LTV seems to be a flaw that the ERS business model has been victim to in the period between July and December. Just be aware of that when re-establishing customers or establishing new ones.


Point: "The strategy for doing this offer test is to understand how they view their problem and seeing how we can deliver on their UVP. I am open to the fact this is not likely to look like the service we are currently offering, and ultimately the outcome would be improving the conversion rate."

Considerations: Once again, as per my point above, I would be cautious about assuming that trying to change the way in which they see their problem is the means through which to re-establish customers. The customers that you're looking for as early adopters can't just perceive that there is a problem, but have to also be actively looking for a solution, and they must have the desire to pay for that solution in a way that is conducive with good ROI for ERS. I understand that you'll be doing currency testing if the offer testing is successful, but just understand that finding a customer with all the right characteristics will filter down your potential customers to quite a small segment.

Point: "From our market research (internal and external) we gave found 182 restaurants, over 90 hotels and over 140 schools (market segments black label post coming). For example, out of the restaurants, ERS has only contacted 36 EVER. This is a HUGE market opportunity if we can effectively understand their problems and deliver them value."

Considerations: This is actually a really good point that I didn't realise and if you can manage to connect with these business types, then you'll place yourself in a good position. However, the question is a matter of economies of scale. From looking at the financial projections document, ERS is running at a loss that increases quite notably each month. That projection was based on how the business was going in July, when you were collecting aluminium and glass as well, and when your warehouse wasn't almost full. If you were to expand out to all these businesses, then you'd have to increase financial outflows associated with collection. This would indicate that you would need to figure out the equilibrium between having a certain amount of businesses signed on and investment into expanding collection/storage services. You would have to hope that the businesses that you connect with deliver enough waste to you in a consistent, secure manner in order to have consistently have positive ROI and therefore profit in a relatively predictable manner.

I'm not sure if this is correct, but from what I understand ERS gets paid based on the amount of material they receive from businesses, instead of a flat, standard rate (please correct me if I'm wrong). If this is the case, but you still invest the same amount of money into expanding collection services regardless of how much waste a business gives you, then it would be a business model set up to suffer from diseconomies of scale.

The other thing is that the numbers of businesses that you listed above, while great, are kind of a block in the long term. This is as you can't really go anywhere beyond Dili businesses given the current timeframes/places in which PEV operates. In the absolute best case scenario where you sign on 100's of restaurants/businesses, where to from there? This is thinking really long term admittedly, but I still think it's a worthwhile consideration.

All of the above being said, if you can find some way to address these issues then perhaps ERS could do super well.

Point: "The 15% green light is based on the fact that we are looking at using relatively large sample sizes and the general percentage of early adopters being 13.5%."

Considerations: I still feel as though this is too low, given that typically speaking the green light for Early Adopters is 60%+ of validation for those that you did offer/currency testing on (as we were taught at BDT). Don't forget that there are 3 things Early Adopter's need to demonstrate. These are:

1. They have a problem
2. They know they have it
3. Are actively looking for a solution that they are willing to pay for.

The above offer testing only tests the first 2 of these points, yet you're Green Light is 15%. In reality, you should aim for 60%+ of those that you do Offer/Currency Testing on to provide you with validation on all 3 of the above points before attempting to re-establishing market penetration with your customer segments.

Reply 0

View all replies (3)
Share