soupsperiment
So: can you taste a bay leaf in a soup bowl?
My estimable colleagues in this empirically minded mystery cult take time on occasion to ritually assert their affiliations to empirical study. Here in this publication dedicated to this most sacred of sacraments – Let us look at the world! Let us categorize it into parts and determine their relations, their correlations! Let us attempt to perceive the joints of the world itself, and when we have perceived them let us take out our knives and carve!!!! – I humbly submit the record of my own novitiate endeavours, in a domain I hold quite close to my heart, indeed quite close.
My stomach is, after all, right there, on the other side of my diaphragm. And while in the map of my body my nose and tongue may seem more distant, consider after all that on astronomical scales my entire body is smaller than our error bars for our state-of-the-art instrumentation.
Let us, in the spirit of philia and ludus, of ethos and pathos and logos in turn,
test each other!
experimental design
We've written and submitted this whole portion of the whole thing to The Loop's preregistration committee before the first subject is ever tested.
Now, bay leaf aromachemicals, I'm told, are supposed to be fat-soluble, and so we must have a fatty solvent. This doesn't stop me from suspecting its efficacy can translate well to a water-based broth, because, well, one man's ingredient is another friend's recipe, and all it takes to emulsify an oil in a water broth is a little protein or starch to give it a li'l gel matrix to anchor to.
In these lights I construct our categories and reify them as CONCOCTIONS:
- coconut milk, neutral oil, salt
- coconut-milk-based soup
- water-broth soup
- water, cornstarch, neutral oil, and salt
A COOK will prepare each of the above concoctions, at first without bay leaves, and portion them out into individual servings. Then the cook will add bay leaves (first bloomed in the neutral oil) to the pots, heat for five more minutes, and portion them out once more into individual servings.
A SECRET-KEEPER will create a MARKER-BASED LEGEND, and mark each of these serving cups on the bottom rim said legend, such that it is invisible to the cook; and array them next to the pots in groups. They will also NUMBER THE CUPS.
The legend will work as follows:
- red-green-blue-black for each of the concoctions
- orange-purple for with vs without bay leaf.
After double-checking that the serving has occurred as intended, the secret-keeper will give the READY signal, and we will proceed into our waitng pit of hungry, sniffly, shivering TEST SUBJECTS where they lay unsuspecting in our TEST SETTING – that is, a room of friends and colleagues who have noticed that it is a rainy day and would perhaps be willing to fill out a SURVEY FORM if it comes bundled with some soup.
Said subjects will answer TWO SURVEY QUESTIONS on their respective SLIPS OF PAPER that they then fold up and submit to us. They will also hopefully refrain from identifying themselves in said slips of paper.
The questions are as follows:
- Rate everything you tried!
- 9 entries with fields for cup number and score.
- What's the experiment about?
Results
Find the raw data in this gsheet.
We received 12 total entries; of these, two self-reported their disqualification in question 2, as requested, because they'd already heard about the experiment because they were in the room during its deisgn phase. (Though this is conjecture; they did not reveal their identities on the survey form.)
Analytical methodology
Id' like to formally apologize to Alexander Wales for running my mouth last night about whtehr analysis counts as empricism. This whole exercise is accursed and useless without a hefty layer of principled interpretation to even begin to make sense of what's happening. As it stands, getting anything I'd not laugh at calling a conclusion from these apltry numerical offerings is like unto squeezing blood from a stone. And yet, having gone out there and collected the data, I feel so strongly convinced, in my heart of hearts, that it ought to be able to tell us something. Anything. I have touched the bones of the world! I have stared at the heart of a flower and rubbed the yellow dust of its propagation between my fingers! I thought I smelled God in the perfume of a frying bay leaf! Shouldn't I have learned something?
Ahem.
Anyway.
I tried a groupby and a mean of each group, aware all the while that each axis of grouping available to me (8 buckets, 3 axes, 12 – TWELVE – datapoints) was already definitionally near to about as riddled with confound as it was possible to be and still hope to learn anything at all. (This shows that people really like coconut milk, and not much else.)
Claude has informed me that the instrument I seek is not a linear regression, but a mixed linear model; machinery that GSheets in its wisdom has not seen fit to provide me. Why do they always lock up the power tools in places I'd have to vibe up properly if I wanted to get anywhere properly with them.
Conclusions and conjectures
Coconut milk is tasty. Vegans everywhere (those without coconut allergies, anyway) agree.
The rest will take a moment to confirm.
Qualitative analysis (i.e. my own opinions, which I get to call science because I'm consecrated into the clan of the qualified opinion havers now)
Folks conjectured wildly about what it was we were actually trying to test; absolutely nobody guessed bay leaf to be among the differentiators. This despite a faint greenish tinge in both the cornstarch and coconut milk concoctions, which I thought was a dead giveaway.
The closest guess was someone who asked if this was a leave-some-out ingredient attribution (not in those words), without specifying which ingredient.
Two people had guesses that it was about coconut milk. Seeing the data, I'm not surprised; the single biggest single-factor positive effect on flavor was coconut milk. Tracks, and fair. Nature's emulsions usually beat ours, when we're not trying very hard to beat them. In practice, I used the same mirepoix base for both broths; coconut was indeed a little something extra.
Problems with the design and execution
Methodological hiccups
- Our sample size was very very small.
- Respondent 6 rated one thing 6? Unclear what exactly was up there.
- Respondent 11 rated on a scale of 10, and asked me to normalize by dividing by 2. This will result in off-by-one-ish effects. Fortunately (?), they started from 0, which I've already decided to normalize to 1.
- Speaking of zeros –several other people rated 0s for some things. Fair. Some things feel like a zero. I should have given you all Likert bubbles. This was a fault of the form.
- I had to cut the experiment short because I'd forgotten I had an appointment. One of my gracious editors helpfully offered to clean up the mess we'd left on the island. I thanked him and dashed off, having entirely forgotten that I hadn't taken any note of the key on the bottom. An hour and a half later, I could be found fishing the cups out of the trash, dictating the number-color triads to a friend I'd flagged down.
Regrets in the late night
- Should we have been more clear about what was being tested? Or was this the correct level of obfuscation of intent? I'm torn. Perhaps I just dislike withholding information I think is cool; perhaps in subjective experimentation you only ever really get anywhere when you lead the human to the funky-tasting mostly-water with a few more steps.
- I found myself aesthetically displeased with the amount of wastage involved in setting this experiment up in a hygienic and safe manner. Individual tasitng cups for everybody, plus tasting spoons to taste, plus fresh utensils and cups for folks worried about flavour cross-contamination…
- Speaking of which, I would have liked to have a policy about flavour cross-contamination. I believe that this problem was adequately mitigated by letting people take their bepoke best shots at tasting.
- These are not the conditions under which one enjoys a bowl of soup. To do this right, I'd have to have hosted a dinner party – one in which, in fact, I provided a variety of soups, whose similarity to each other would only be suspiciously remarked upon by those not in the know. We'd have an army of helpful, near-invisible service staff, checking in on the diners' satisfaction with the meal as a whole, making close note of expressions, speed and quantity of consumption, unforced commentary. This whole affair would of course need a large and diverse enough sample size that we must of course host at a sutiable venue; and of course to gather good data about food, we cannot study it in isolation. We'd have to hold these monthly, inviting a great variety of people, on the thinnest of external excuses. Was this what hosting was, once upon a time? A/B testing on your undoubtedly clued-in guests?
- Actually, no, wait, what I'm actually realizing is that I already know who gets to do this in the modern day: !this sort of thing is a huge part of what makes working at an ad-tech shop fun.
Resource constraints
- It is currently 23:34 and I have to hit publish. Given more time I'd have written a shorter, tastier, better designed study.
Acknolwedgments
- Natalia, for doing the broth grocery run while I schemed and planned and underwent a preregistration review round and printed things out – and then doing another when it turned out that I'd overestimated the Lighthaven kitchens and assumed they just had cans of coconut milk on hand.
- Lighthaven, for the justified esteem I hold its kitchen in. This whole thing should not have been this easy to put together in an afternoon.
- wadapan, for acting as the secret keeper, creating the legend, and going back and forth marking up cups.
- Vishal, for chopping my onion (and stirring my soups while I cried at the onion he had chopped).
- The good residents of the building I was cooking these soups in, for tolerating a lot of logistical hullaballoo.
- And finally, the good editors of The Loop, for inspiring this entire exercise through an alchemical combination of nerd-sniping, peer pressure, environmental structure, and timely logistical and editorial support.