BOTEC Estimates of Washington Cannabis Market are UNDER-Estimates

Executive Summary:  (for the TL;DR crowd)

BOTEC’s recent estimates of the size of Washington’s Cannabis market are terrible, horrible, and rather gnarly.  One might think that BOTEC would be embarrassed by the work they sold to the Washington State Liquor and Cannabis Board.  Of more immediate importance, these BOTEC-generated magical estimates are, simply,  way too small.  Systematically way too small.  This is important in that these under-estimates have already contributed to the too-small allocation of only 222 new retail access points to support the medical cannabis market.  If you agree with my assessment of BOTEC’s work, I would suggest that you write to let both your state legislators and the WSLCB know that you do not believe they should be using (or repeating) the errors that BOTEC has supplied as finished work product.  That finished work product is published on the WSLCB site  here, in case you wish to follow some of the specific page references below.

This is my first HI-Blog post in the last 4 weeks.  I’ve been busy transitioning HI-Blog so that it is now  hosted under my new commercial website www. straightlineanalytics.biz.  Attempts to go to the old HI-Blog site should re-direct to the blog page on SLA.  The SLA site is basically complete, but note that I have turned off the  “Shop” until later this week.  This is  where documents, service packages and tools will be made available for purchase.  I just have to do some banking things in order to set that up, and I did not want to hold off any longer on blogging …. my lack of a productive outlet for my words here has resulted in me getting way too verbose on Danielle’s forum.

I expect to be posting at least 2-3 articles per week on HI-Blog  going forward.  With luck the pressure of production will lead to less verbosity (on average) going forward.

On to BOTEC – or how to make a living producing meaningless numbers.

The following is the distillation of the top 9 things I found horribly wrong with BOTEC’s work – coupled with a bit of related commentary.

My thanks to the colleagues that provided input on my initial drafts of this work.

.

Points of Critique on BOTEC’s Estimates of the Size of the Medical Market in Washington State

Released by the Washington State Liquor and Cannabis Board Dec 2015

Point #1) – Lack of Accuracy and Quality Control in BOTEC’s work

• The tables summarizing the primary results of BOTEC’s study are full of errors (tables on pages 2 and 30) … to the point that 2/3 of the rows do not even add up to the expected total (note … this is not a rounding problem, it is a blatant error and makes the results of the study virtually impossible to understand).

Point #2) – The use of DOLLARS to define the size of each market component is problematic

• Given the different price-points (per gram) in the three markets, it is difficult to see how money corresponds to either the “consumer count” or “product volume” by which markets are traditionally sized.  BOTEC does appear to have pulled in external estimates relating to what the price (per gram) should be in the medical and illicit markets.  Setting the average price of Medical Cannabis in Washington at $7.11 per gram does, however, seem to be low (by a factor of approximately 60%).  Setting the I-502 Retail price per gram at between $5.26 and $7.36 per gram is, simply, ridiculous (Fig 11, page 21) (and compounds their systematic under-estimation)

Point #3) – Multiplying the level of recreational sales that occurred in October 2015 by 12 is not a reasonable way to estimate the expected annual size of a market that is still in it’s infancy

• aside from the fact that Oct’15 saw diminished sales in the face of Oregon beginning to offer tax-free Cannabis on Oct 1, our recreational market is far from saturation  in it’s 19^th month of existence.

Point #4) – The use of CURRENTLY OPEN DISPENSARIES as the fundamental proxy by which to “size” the medical Cannabis market is inappropriate in an environment where dispensaries have been the focus of increased enforcement and closure over the past year.

• This almost seems designed to low-ball the estimated size of the medical market.  I believe the use of this proxy is indefensible.  Taken to an extreme, had the state and local authorities been effective in closing ALL operating dispensaries by last October, BOTEC’s methodology would have declared the size of the medical market to be ZERO (dollars).  Should BOTEC wish to defend this aspect of their methodology, might I suggest a public debate on the topic.  I volunteer myself to be the “con” party in that debate.  This is, IMHO, one of the key logical flaws to how BOTEC chose to approach the initial question.  It will be interesting to see BOTEC’s defense of this methodology – even if not in the form of a debate.

Point #5) – Having grossly underestimated the sizes of both the recreational and medical components of Washington’s Cannabis marketplace, BOTEC then estimated the size of the Illicit market.  Given the apparently systematic underestimates of the other two market components (recreational and medical), the method by which BOTEC has chosen to estimate the size (in dollars) of the illicit market is largely irrelevant.

• I’m being a bit hard on them here.  Illicit market estimates are tough at the best of times.  Even if done perfectly by BOTEC, their estimates regarding the illicit market are rendered less useful given the flaws with their estimates concerning the two more visible components.  If the tables mentioned in Point #1 above actually totaled to 100%, (or the Low/Best/High overall market size dollar amounts) there might be more value in what they produced for their illicit market estimates.  As it is, I am not quite sure what the final estimates are for the sizes of this market (Low, Best and/or High).  Is anyone?  Are they?

 Point #6) – BOTEC seems to assume that “no relevant data exist” upon which to make certain key estimates (such as the price per gram at Retail for the recreational market) and thereby justifies less-than-optimal proxies with which to fill in the blanks.

• This methodology is acceptable if there are no relevant data.  BOTEC appears to be unaware of  the richness of data contained in the LCB Traceability system (on, for example, retail pricing, market growth, transactional volume, product turnover rates, Etcetera).  It is a little hard to see how this omission got by WSLCB Staff while reviewing earlier drafts of this work, but I guess they don’t spend as much time with their data as do I.

Point #7) – BOTEC appears to use biased language in describing the “significance” of certain key intermediate results.

• Calling correlations of .47 and .44 “Strong Relationships” in relatively low-sample-size analyses seems odd (see near the top of page 15).  It is highly likely that these two correlations were not statistically significant.  BOTEC does not follow the standard of reporting on the p-value associated with these statistics.  They should be required to do so.  Should they have called statistically non-significant correlations “Strong Relationships”, their work would be verging on fraudulent reporting.  At a minimum, it would be misleading.

Point #8 –BOTEC appears to use Monte Carlo simulation in both appropriate and inappropriate ways   They may believe that Monte Carlo will shed light that cuts through the uncertainties inherent in their methodology.  If so, they would be wrong.

• This is a technical point, but it basically reduces into a “garbage in / garbage out” argument.  At least a portion of how they appear to have used Monte Carlo appears to be garbage.  It is, however, difficult to tell with certainty, given how few specific model details are supplied.  While I appreciate the transparency afforded by  Figure 10 on page 18, that transparency is tarnished by the apparent error in the reported parameters for the  “uniform distribution” reported in the last row of the table.  Seeing the full model that they used would be helpful in fully diagnosing just how bad (or good) their Monte Carlo work actually is.  Regardless, using Monte Carlo simulations to illustrate the uncertainty implicit in the underlying model is OK, but it is circular and adds no incremental value to their analysis.  But it’s “fancy” and “complicated” and does “10s of thousands of iterations”, so it must be OK — and worth lots of money in fees.

Point #9) –How can the work of an entity that apparently has no shame be trusted?

• This work appears to have been designed with little consideration of the original mandate to provide “fair and balanced” (my words) estimates of the size of the medical Cannabis market in Washington.  I was not aware that the standards for work provided to Public Sector clients had become so low as to enable work of this quality to be put forward as finished work-product.  That BOTEC felt the need to place a disclaimer in their work that it should, effectively, not be used for anything may help them sleep better at night.  It does not lessen the disservice that they have done to their Client and to the entity and people that it represents.
• That it was subsequently, published by a Government Agency is unfortunate.
• That it has, apparently, informed the under-allocation of only 222 additional retail access points to service the Patient Protection Act is potentially tragic.

I found BOTEC’s work disturbing.  I thought they were one of the “experts” in this field.  I expected more of them.

I would imagine that a reasonable person might well expect them to be ashamed of their work and the negative impact it is poised to  have on public health and safety in Washington..

If allowed to stand, the allocation of only 222 additional retail access points for Cannabis (be it recreational or medical) will cause harm to public health and safety and it will do so BEFORE July 1 of this year.  Ironically, the emergency rule that proposed this allocation is accelerating down a path that will, predictably, in itself create a serious emergency that will have negative outcomes on human health.

I’d like to offer one specific and crucial piece of advice regarding any efforts that might arise to mitigate and/or avert this fully predictable impending (and real) emergency.  That is that the way in which the Board chose to allocate the new access points in December REINFORCES the lack of access in areas suffering the impact of bans or moratoria.  A good rule – a good process would, hopefully, mitigate or (at least) remain neutral regarding the impact of local obstructionism on the implementation of the Patient Protection Act.

Note to the WSLCB:  Medicine is not beer is not wine is not whiskey is not tobacco.

It is my belief that it is neither the role nor the intention of the WSLCB to double-down on local obstructions to access.  Particularly for stores with medical endorsements. That the allocation method used for the Dec 16 announcement did, effectively, double-down on obstructionism, is unfortunate.  It may well have been a well-intentioned attempt to ensure that the much-needed new access points would at least exist SOMEWHERE in the state.  Ban/Moratoria areas, after all DO have stores allocated — just not open or licensed, for the most part.

If the LCB reinforces local obstructionism today, that will contribute to ongoing compromise to medical access for the populace of affected areas.  I hope they reconsider and allow for additional store allocations in affected areas.  The amount that the local people need —- not the amount that the local leaders desire.

At a minimum, how about ensuring that the number of stores allocated is AT LEAST half (rounded up) of the number of retail pharmacies that exist in the area?  That is an easy and quick empirically-based  allocation that does not cost the people of Washington any consultant fees and reflects local pharmaceutical demand.  It’s a way of letting the “market” tell you what it needs.

Regardless, the LCB and other Governmental agencies should stop using BOTEC’s recent analysis to inform their rule-making and policy-making efforts relating to Washington’s Cannabis marketplaces.

This, of course, suggests that the WSLCB should be trying to get some good estimates of the size of Washington’s Cannabis market(s).

Me stating this may come across as self-serving, as I have produced such estimates, and know how it SHOULD be done.

CASP might be able to help if the WSLCB finds they need help with this.   They can certainly do a better job than BOTEC has done on this question.

Perhaps the WSLCB might think of dusting off the work that a group of us (CASP, myself & the CCSE) supplied to them (free of charge) last year which estimated how many incremental access points were needed to support the forced migration of Washington’s medical Cannabis infrastructure to the regulated and controlled world.

I would like to think, given how horribly bad BOTEC’s recent work is, that the WSLCB might assign some critical eyes to look over the prior work BOTEC has submitted to the State of Washington.  Work, remember, that the State has used to design and implement the I-502 market and much of their legislative and regulatory decision-making regarding State-Legal Cannabis.

To summarize, it is my strong belief that the results of the BOTEC study are, essentially, meaningless.

They systematically under-estimate values commonly known to be higher.

Their results should not be used to inform or guide policy or business decisions.

Let’s try not to hurt people in 2016 if we can avoid it.  Using BOTEC’s results (and even assuming them to be useable) will hurt people in 2016.  Doing so will also hurt businesses — lots of businesses.

Underestimating the size of the medical market guides policy that causes patients harm.

Underestimating the size of the recreational market guides policy that harms the development of the recreational market.

Underestimating the size of the illicit market is something I expect will generally happen (for lots of reasons).  Regardless, state policy should be to diminish the illicit market.  I think most agree on that.  To measure progress on illicit-market-reduction efforts, good estimates of the true (not dollar-based) sizes of the three market components is key – as is an understanding of the flows of patient (and product) between the three.

It is unfortunate that, in attempting to fulfill whatever terms were placed on their contract to estimate the sizes of this State’s Cannabis markets, BOTEC has failed to deliver meaningful information on any of these things.

One wonders how many of our tax-payer dollars they received for this work.  Dollars that could have gone to help pay for education (or to get a better traceability solution).

Back Of The Envelope Calculations, indeed!

Last I saw, computers were better than envelopes for doing quantitative analytic work.