Note: Shortly after I post this, I’m going to e-mail each of the labs certified for Cannabis testing under I-502 with a link to this blog. I will solicit their feedback, and am going to try to set up a “behind the scenes dialogue” with them in order to reduce the likelihood that some weird pattern I see in the numbers is due to a completely normal, expected artifact of how the testing and/or reporting of testing results is conducted.
I do not want to be doing a disservice to either the lab testing sector or to individual players within that sector. I am not an expert in testing for moisture or foreign material or bugs & germs or fungi & molds or residual solvent or cannabinoid levels (I guess, truthfully, I have some experience with the last one). I am, however, reasonably adept at detecting odd patterns in data.
It seems only fair to give the Labs a chance to set me straight if I’m seeing things as bad that are really OK and explainable once one knows the inner secrets of the lab testing kingdom.
I’ve taken a close look at the lab results data for all tests that have occurred through August of this year, and will report some observations regarding moisture testing in this post.
I intend to publish my LAB RESULT work in a series of postings over the next couple of weeks, and feel that the time is ripe for such work, as we move into the final phases of both the LCB and DOH rule-making that will impact the industry, and the tests that will and/or will not be required of industry participants in the future.
I have been crystal clear for at least the past year that it is my firm belief that Proficient/Reproducible/Accurate/Good/Unbiased/Empirical (PRAGUE) lab testing is crucial to the health and development of the State-legal Cannabis industry.
It is an issue of “fair weights and measures”, and it is an issue of customer assurance that the “quality” of the product in the regulated market will be better known (and, hopefully, “better”) than that from the other channel of Cannabis acquisition (hereinafter referred to as “The Darkside”).
At its core, this is the ONLY advantage that the regulated market has over The Darkside (other than relative levels of legality and “Revenuer-focus”).
In the short term, costs of production will be higher in the regulated market. Tax rates are higher in the regulated market. One must show ID to get product in the regulated market. One must be entered in a database to get medical product in the regulated market. One must also subject oneself to being filmed in order to get product in the regulated market. One can’t even reliably SMELL the product in the regulated market before purchase.
Aside from a high level of assurance that retail store employees are not packing firearms during transactions, I see little advantage to the regulated market other than its labelling and quality assurance testing.
Hence, I worry more than I likely should about the importance of having PRAGUE labs and lab results servicing the needs of this industry and its consumers and patients.
To that end, it is my null hypothesis going into this work that the labs are all PRAGUE and the labs will all show similar patterns of results. Their results will reflect reality, and they will (when appropriate) follow the rules of statistical probability that tend to govern non-biased measurement systems.
Enough touchy-feely opinionated intro — on to the initial results of my investigation into lab results.
To answer the question posed in the title of this post, I’d be willing to bet that the regulated bud you are currently (or recently have been, or might soon be) smoking/vaping contains less than 15% moisture.
That’s an easy one to answer, because anything above 15% FAILS the test for moisture and hence, does not make it to market (unless it passes a re-test, I’d imagine).
It is also easy to answer, because the current RATE of tests failing the moisture test is less than .05%.
There is nothing wrong with that, but looking at the trend across the 15 months of testing (broken out into 5 consecutive 3-month “Quarters”), it’s fairly clear that the initial failure rate of 2-3% has rapidly and reliably declined.
On one hand, it looks very much as if Washington farmers are learning how to adequately dry their flower.
On the other hand … and this is the part where knowing EXACTLY what the testing protocols entail would be handy … there is an odd aspect to the overall distribution of reported moisture levels that I’ll get to in a moment.
Without knowing EXACTLY how this testing is done, I’m going to imagine that a grower (or processor, or producer/processor) pretty much dries their bud down to a level that makes sense to them (given their experience, and/or what “the books” say).
For the most part, this will result in bud that is somewhere around X% moisture (where “X” is a theoretical “ideal”, and values above and below X will tend to occur with less frequency the further one gets from “X”). Some will be a bit wetter and some will be a bit drier. A small number will be much drier and a small number will be much wetter.
The fact that being TOO wet (over 15% moisture) causes a failure would lead me to expect a biasing of the distribution of testing results to be BELOW 15% and, indeed, to avoid the 15% level (and above) like the plague that a failed test result should be viewed as being.
There will likely be some very low moistures and some very high ones, but not too many of either (I’d imagine very few people want completely dried flower and that no-one wants moldy goop drizzling out of their mylar package upon opening).
Before I looked at the data, I actually expected to see a fairly sparse distribution of results above about 13% … on the assumption that most cures were likely to be shooting for the 8-12% range of ultimate moisture (just a guess …the key point being that I assume most growers would shoot for a reading of at least a few points BELOW the 15% cut-off).
During the most recent quarter (June, July and August, there were 8,511 moisture tests reported in the State Traceability system. Only 4 of these were reported as exceeding 15.0% moisture content (for a failure rate of less than .05% (or less than 1 in every 2,000 tests resulting in a failure).
That’s a very small number, and not something that is generally amenable to powerful statistical examination around “expected rates of occurrence” and related concepts.
However, the symmetry (and density) of a distribution around an important threshold (like a 15% failure level) IS amenable to investigation using all kinds of tools (including my favorite of flipping a coin – but that is just because I like money so very much).
Here is a scatterplot of the moisture test results reported over the past 3 months:
The two things about this chart that I find the most interesting are the “spike” observed around 6% moisture, and the “cliff” that occurs at exactly 15.0% moisture.
The “spike” is suggestive of the “normal-like” curve that one might reasonable expect to see in an efficient system where there is, truly, an “optimal” moisture level that folks are shooting for.
Based on these data, I’d now guess that this “optimal” level is probably somewhere around 6-7% . I’m assuming it is a bit above 6%, as sensible growers would bias their results down a bit in order to avoid a “too-wet and failed” result.
The remaining question is why there appear to be so many moisture results coming in between (say) the 10-15% range, particularly when there are virtually none showing up ABOVE 15%.
This issue becomes more clear, when the distribution of values between 14.1% and 20% is summarized
Given that the grower DOES NOT KNOW the moisture level of the product before submitting it for testing, why would there be 91 observed values falling in the range of 14.91% – 15.0% moisture, but only 4 results with values over 15%? (update note from Jim: I have been challenged on this point by a grower I trust … namely that some growers DO, actually know their moisture levels before submitting samples … not sure if they know it down to 1/10% precision, but if this is true it tempers my concern a wee little bit).
It’s kind of like flipping a coin 95 times and getting a head (or passing the test) 91 times.
The chances of that happening if the coin was fair (i.e., if being below or above 15% were equally likely to occur when looking at values between the ranges of 14.91% and 20%), are really small (the proverbial “p” is less than .00000001).
In this example, the “coin” is clearly not fair. Noone wants results over 15%, and virtually no one is getting results over 15% any more. The “luck of the draw”, however, for those that are REALLY REALLY Close to 15% is very very very strongly favoring a passing grade for such samples. If each test REALLY were a coin toss and we believed the coin to be one that would normally come up “heads” 9 times out of 10 (90% of the time), we would STILL see 91 “heads” out of 95 flips only 2.18% of the time. …. Still a statistically unlikely pattern of results to see … even with a coin biased 90/10 in favor of seeing such biased results.
There should, IMRO (in my resulting opinion), be more bud testing at above 15.0% than is currently being reported.
I’m thinking about writing lyrics for a song to be titled “Where has all the wet bud gone? (sung to “Where have all the flowers gone?” … I’m glad Pete lived to see Legal Cannabis in the US of A)
The first verse would likely go something like:
“Where has all the wet bud gone?
No Doubt Oregon
Where has all the wet bud gone?
Moldering away”
My personal belief – wet bud is out there. You are smoking it or have smoked it or, likely, will smoke it at some point in the future if nothing changes in how we hold growers accountable for their product and labs accountable for the results they report in exchange for being paid to run the quality assurance tests.
You just don’t KNOW that you are / have /may be smoking it.
I picked moisture to start with, as it seems pretty much innocuous. It is not like high THC levels driving price, no sirree. It’s pretty innocuous. Except that failed bud cannot be sold as flower.
It is also important to remember that moist bud coupled with extended shelf time and/or the right levels of heat WILL promote the growth of mold. Mold can hurt. Mold can kill. Mold is bad (except when on cheese or producing penicillin).
In the world that existed between June and August of this year, virtually no flower samples failed their moisture tests, yet the reported test results themselves almost cry out for there to have been more failures. When one sees no failures, one cannot help but wonder about the value of even conducting the tests (much like I feel about airport security protocols in the USA).
Everyone in this industry … or at least anyone wishing this industry to be all that it can be should stand up and take note of ANYTHING that compromises trust in the quality assurance results being reported for State-legal Cannabis in the regulated market.
If the regulated Cannabis market in Washington does not have meaningful PRAGUE Lab results, then it does not have a meaningful advantage over The Darkside.
I’d summarize the moisture results as being strike one against Washington’s State-legal Cannabis Lab Testing Industry. I’m not reporting lab-specific results here, but I AM looking closely at them and am accumulating a report-card for each test against which I am scoring the labs. When I have finished my analyses of all of the quality testing results, I’ll collate full report cards for the individual labs. I don’t know if I’ll publish those here — I guess that depends on what they look like.
While it is clearly good business practice to produce good results for one’s clients, the goal here is not a passing grade. THE GOAL HERE IS SAFE PRODUCT OF KNOWN QUALITY.
There is a downside to systematically under-reporting high moisture results, and that is having people (including patients) more likely to be inhaling or otherwise ingesting molds and mildews and related nasties.
Killing for profit is so un-Cannabis. If that is what is happening here, let’s see a stop to it now.
Then again, if I have missed something about how testing is done or if my logic is otherwise faulty, and these results are, actually PRAGUE, then my apologies in advance to the Labs.
If I got something wrong, please post comments on this posting to that effect. Let me know what I got wrong. If I buy it, I’ll re-do the analysis and report on the “correct” results.
If I did not get it wrong (or if I pig-headedly adhere to the belief that I did not get it wrong), then I hope you all look forward to the remaining posts in this series (and possible mainstream articles that flow from this work).
Over the next few weeks, I expect to be reporting on Foreign Matter, Residual Solvents, Microbial Screening (Bugs & Germs & Feces), and (my personal favorite) Potency. I’ll leave potency for last. I suspect that it will be hard to keep quiet about which lab is tied to which results when I get to potency.
Go Hawks?
Jim-
I totally agree with your thinking and applaud your efforts to crack the code of the frequency of these statistical aberrations via the data from the traceability program. I know you have been at this for a while; we talked about it after the meeting at the city council in downtown Spokane a while back.
Simply the data don’t make sense. I reviewed and crunched the numbers from the CTP website and was shocked at how you couldn’t even tie results statistically between labs in a similar geography (i.e. microbiology results). This obviously indicates at least one metric that points to the BS that mentioned in your post.
Simply, you can’t possibly have a total aerobic count average as zero (or close to it). You can’t only a few reports of THC-A; it should be measured for every 502 test. Clearly there are labs operating out of 502 compliance and every producer knows where to take their tests to “pass” and get the “high” THC number (see what I did there with the pun?).
I have been harping on reproducibility between labs and the lack thereof. I have lobbied DoH, LCB, and DofAg. All have been remarkably responsive and have asked for more input on how we can collectively fix the system.
In closing, I just wanted to say thank you for putting in all of this effort. Your Moisture Content assessment associated with 502 testing compliance tells the whole story. Labs don’t want to fail people and therefore lose their clients. A fail sends the producers to the lab, literally down the street or across the sound that they know will pass them. Look for advertisements that advertise “Never Fail Microbial Again”. The environment is simply “Lie or Die” at the labs. Two labs have already folded and I’ll guess that many of the survivors are on the ropes. That’s what $60 502 testing does to the landscape. You can’t do real science to complete a true 502 test for $60.
I own part of a lab as a transparent disclaimer. Mistakes happen but lies are deliberate. Call out the labs. Keep a score card. We could use the oversight. What’s next? How about total cannabinoid content? Have you seen those figures which eclipse 60%? What was the term you used? Bullshit, that’s right. The problem with BS reporting and testing is solved simply. Look at microbial turn times through Biotrack, if they are three days or less see what platform they claim to be using for the testing and then audit their reagent purchases. There’s your scoop.
Thank-you, Jason.
I’m now able to tie all lab results “together” and to both the labs conducting the tests and the business entity supplying the product to be tested (and what the product is, etc).
I’m generally stripping out the values that are clearly in error (percents above 100, some negative values, etc). However, the “theoretically possible” but highly unbelievable points are being left in, as I believe they are relevant to the ultimate question of PRAGUE-ness in the lab results being enjoyed by the industry.
I’ll be leaving “potency” (Cannabinoid) results ’till the end of my lab series. One thing I noticed already, though is the calculation of “TOTAL” seems to differ across labs, yet I thought the 1X + .877Y formula was going to be applied going forward, per the LCB note earlier this year (May?)
I appreciate your comment.
The answer here is likely a simple one. Moisture content is typically the first assessment of the product. If a customer brings a sample to the lab that exceeds 15% moisture, it is reasonable to conclude an incomplete drying/curing cycle on the part of the producer/processor. Labs are vested with the authority to reject samples for a variety of reasons, and so the labs may very well be rejecting the sample and telling the producer to complete their cure and submit another sample at that time. I don’t think this approach is unreasonable.
Analytically, I’m most suspect of the number of samples testing between 14% and 15%. Without the graphs being interactive, I had to “ball park” the total number of samples in that group at 367 samples. This represents 4.3% of the total dataset and those testing precisely 15% moisture are about 1.1% of it. Does this indicate lying? Perhaps. It does seem, in this context, the simpler explanation is rejection of samples not ready for sale, with potential for a small percentage of shady reporting.
It is important for labs to build the kind of relationship with customers that allow them to educate on such matters, providing tools for the producers and processors to ensure their product is ready prior to testing. Moisture measurement devices can vary greatly in analytical precision, but high quality tools for this are being developed by laboratories. In the new year, our lab will be introducing a small handheld analyzer to our customers that can accurately measure moisture on site at the production facilities as well as at the lab, helping to eliminate such ambiguities. Our current development projects also include simple devices that can ball park cannabinoid profiles in situ as well. And these tools are going to be priced at points easily accessible to producers, processors, retailers, laboratories, regulators, and even the consuming public.
In regard to the comment section’s ”You can’t do real science to complete a true 502 test for $60,” I find this to be grossly misinformed. The price tag, on its own, is an exceedingly poor indicator of scientific quality. In reality, high scientific precision can be done at this price, and the problem here is more a comment on a lab’s business model. Sample volumes and lean business practices in forms such as high throughput efficiency and excellence in data modelling are better metrics for designing pricing models that make sense for an operation. Orders of magnitude of analytical equipment, hiring highly qualified scientists, negotiating winning vendor contracts with suppliers, and forming smart partnerships with labs and manufacturers who’ve illustrated high standards of accreditation to produce a high-value test at this price has been demonstrated right here in Washington State. A $60 test can be a poor quality test, or a high quality test, it just depends on the quality of the lab’s business model.
All that said, we really like the review you’re conducting on this blog. Your candor and position on the project is very much appreciated. Could you add a bit more to your text and graphs describing your dataset? It’d be great for readers to be able to run hard numbers as they evaluate the points you make and questions you pose. My guess is your review of residual solvent tests will show a similar pattern. Likewise, the WSLCB has expressly told labs that residual solvent failures may be handled by reporting back to the producer/processor, having them conduct additional purging, and then resubmit a new sample.
Thank-you, Bobby.
Sorry about my delay in response … I have to become more disciplined in checking my wordpress “spam” filter … it is catching lots of comments inappropriately.
You hit it on the head … it is PRECISELY the asymmetry around 15.000% that caught my eye (coupled with the increasing concentration of results just below 15%). If there are so few over 15.0, there should be “so few” just below 15.0, as well.
Regardless, the “resubmission of samples for re-testing” certainly adds another layer of complexity to the results.
Thanks again.
Jim,
Looking at your graphs it appears you have a normal distribution of moisture values in 502 compliance tests. Great! Normal distribution is what we’d expect a priori, and since it looks that way it’s safe to assume that it is. You point out that the right side tail of the distribution is missing. It does appear to be missing. In fact, it appears that your dataset has been excluded of samples above the moisture failure threshold. That’s what you’re on about. I like how Bobby of confidenceanalytics.com was able to provide some explanation for these missing values. It seems that some — perhaps most — of the missing samples haven’t been “fudged and passed”, to quote no-one-in-particular, but rather they’ve been “rejected and retested” after drying. Makes sense. I’d hate to throw my good bud to the extractor because it failed on something so simple as moisture. Not to mention, I typically test my buds before packaging (I think most of us do) and so the flower lot dries even further before packaging anyway. Shoot, bending the bud and squeezing it between your fingers is definitely sufficient for knowing if it’ll pass the moisture test. Trust me I’m a doctor.
Dr. Chris. Being a doctor myself, I have learned not to trust doctors by default. All of those years in the halls of academe have a tendency to breed a high degree of skepticism, questioning and (occasionally) pig-headedness on the part of doctoral candidates. The ones that survive and matriculate are a special subset of that gang …. and we are often the most pig-headed.
.
If, however, you mean you are a physician, then that is another story. If you are … thank-you for dedicating yourself to human health and, no doubt, helping many people in your life.
I spent most of my career applying the skills I learned in the rat-infested world of experimental psychology to (using loose language) assist the efforts of pharmaceutical folks to effectively educate physicians about the benefits of whatever was the product-of-the-moment that we were pushing. Effectiveness can have lots of definitions, but I often viewed it as when the little rats responded to the stimuli we supplied and subsequently showed a statistically- (and financially-) significant increase in the choice of our cheese vs the other cheeses alleged to treat the disease in question.
Spending years honing our ability to get our cheese eaten changed my opinion of physicians and their place on the hierarchy of humanity. Very honorable profession. Easy, however, to manipulate. In a way, I suppose that puts this type of “doc” close to politicians on the hierarchy of “easy-ness to influence” (consumers, of course, are at the top of this hierarchy). Politicians need good meals, wealth, and the promise of future benefits. Docs need good information, well (ahem) “packaged”. Consumers just need pretty pictures and distractions and good sound-bites or ditties or other tricks to associate brands with their own well-being. Child consumers just need Joe Camel.
OK … ’nuff said on that. There are lots of medical docs for whom I hold the highest respect. I just don’t hold the default assumption that they (or their diagnostic or clinical skills) are trustworthy as a group.
I appreciate your comments on the moisture post. I actually DO NOT see a normal distribution in the data.
I see TWO superimposed distributions. One centered at about 6% moisture that is very “peaky”, and one that is not as “peaky” (sounds better than “kurtosisy”) centered around 10-12% moisture.
I’m guessing the 6% threshold has something to do with when a common test shows the bud is close to getting too dry (e.g., when the stemlets first start breaking cleanly or when the pressed bud crushes and turns to powder or whatever).
I’m guessing that the larger “hump” is, in large part, an artifact of the test/re-test process, coupled with the fact that approximately 36.2% of our farmers seem to know what they are doing with respect to moisture evaluation and apply that expertise before the labs ever see the bud.
There still are not enough failures being reported.
Of course, that is just my opinion … .and I AM a Doctor.
Thanks again for your comment, Doc.