Search This Blog

Friday, August 23, 2019

Qualitative Data is Data Too (Part 2)

In Part 1, I contrasted traditional data analysis and qualitative analysis. In this part, I tell a story that combines them.

Unfortunately, I don't have a source for the story. At one point, I thought I read it in the work of Russ Ackoff but I haven't been able to find it in his writing. Then I thought it might be from Gene Woolsey. I was fortunate enough to spend some time on the phone with Dr. Woolsey toward the end of his career. He agreed that the story sounded like something that either he or Ackoff would have written but it wasn't his and he didn't recognize it from Ackoff's work. Therefore, even though I know that I read it or heard it somewhere, it remains apocryphal.

Since I can't find the source, I can't double-check the details. If anyone can confirm a source, please let me know.

Here's the story...

Back around 1970, a quantitative analyst was hired as a consultant by a Fortune 500 firm in Los Angeles. They were considering moving their operation out of the city to the suburbs and they wanted him to evaluate the long-range cost/savings of expanding where they were versus relocating. The consultant gathered the necessary data, did the analysis, and concluded that there would be significant savings if the company moved. He reported his results, got paid for his time, and went on his way. Nothing changed at the company.

About 18 months later, he ran into the CEO who had hired him. He brought up the project and apologized that his work hadn't been useful. The CEO seemed surprised and assured him that his analysis had been crucial to their decision.

"But you're not moving" the consultant said.

"No, we're not" admitted the CEO. "We wanted to stay in the city. We thought it was better for our employees and we believed that we had a positive impact on our neighborhood. Our corporate mission seemed to fit with the city location. There was a lot of pressure to move and save money but we didn't know what the savings would be or, conversely, what the cost would be to stay put. Your analysis made the numbers clear and we decided that would afford to not move."

The moral of the story? The analyst did his job with a traditional quantitative analysis. The company made their decision on a combination of quantitative and qualitative factors. On the numbers alone, one could argue that the company reached the wrong conclusion. On the qualitative analysis, it wasn't clearly a right or wrong conclusion but it was a decision that the executives and the board were comfortable with.

In Part 3, I'll give an example of my own qualitative analysis. By necessity, my conclusions will be "fuzzy" and you might disagree with them but you should still be able to see the logic connecting my observations and conclusions.




Wednesday, August 14, 2019

Qualitative Data is Data Too (Part 1)

To data professionals, "data" implies some sort of structure with defined records and variables. In this context, quantitative variables are numbers (such as price, income, age) and qualitative variables are non-numbers (color, country, gender, payment method). Depending on the lingo in your particular world, qualitative data could also be called nominal data or categorical data but it's still structured data.

However, some fields use the term "qualitative data" differently. Their text or observations aren't easily codified into records and variables but they still examine large segments of data to discern patterns and themes. Philosophers might read Plato, Hobbes, Smith, and Marx to generate theories and find text passages to support, or refute, those theories.

Modern technology can be used to bridge these approaches (see the digital humanities). There are tools available to process online comments or customer reviews and determine how many are "positive" but some questions and some data simply don't lend themselves to any sort of structured data methods.

Let's use the Bible as an example. One could ask "How many times is the word money in the Bible?" Since the Bible wasn't written in English, we'd first have to agree on which translation we're going to use. Then it wouldn't be difficult to process every single word and count the number of times "money" occurs. Of course, this has already been done. I suppose it's somewhat interesting that the King James version uses "money" 140 times, but I'm not sure that this mini-fact is particularly informative.

There are other words for "money". The Bible might mention payment, wages, debt, inheritance, silver, gold, ...  This source tells us that there are over 2300 verses in Bible that mention "money, wealth, or possessions".

But are "possessions" and "money" really the same thing? This analysis requires another step. As before, we'd have to agree on which translation to use but we'd also have to agree on a list of synonyms for "money". We might even come up with an ordinal scale for whether a word is a true equivalent or simply related. Then, as with the previous question, a program could process every word in the text and count how many times "money" and each synonym occurred.

In both of these examples, it's possible to process the Biblical text as more or less traditional data but the results aren't all that useful. A much more interesting question is "What does the Bible teach about money?" and neither example answers that. Many have attempted to answer that question, but none of them were able to do so with traditional statistical or data analysis tools. Sure, the first two examples could be modified to "flag" text segments that might be useful in answering the question, but (so far*) a person still needs to read through those segments and evaluate them.

Data professionals often aren't comfortable with this type of qualitative analysis, It's too fuzzy or too touchy feely. However, it's likely that the people data professionals report to are using all kinds of "fuzzy" analysis so it might be wise to study some fields where qualitative analysis is commonly used.

In Part 2, I'll tell a story of qualitative analysis and decision making. In Part 3, I'll do my own qualitative analysis.

*AI tools are advancing rapidly. If you know of tools that can do this without any human intervention, then please tell me about them.