Defining Data Intuition
My summary:
For a field that has both ‘data’ and ‘science’ in its name, it may seem out-of-place to bring in ‘intuition’. But, as these blog posts point out, there are times when intuition inevitably comes into play. So, it’s important to think about where we rely on intuition and how much we (should) rely on intuition and in which contexts.
The blog post from Harvard Business is looking, not surprisingly, from the business side of things and how to make decisions when the data is not always clear-cut or does not point clearly in one direction or another.
The post from Ryan Harter is focused more specifically on how to look at the results of a specific analysis. He gives the following definition for data intuition:
Data Intuition is a resilience to misleading data and analyses.
Essentially, we perform a certain analysis, but based on experience, have a sense of whether something does not seem right and then double-check the steps of, or the logic behind, the analysis.
To bring these ideas together, here is a bonus third blog post from Brandon Cosley. He breaks down intuition in different ways. For example, while data scientists do not necessarily know how to derive from scratch all the algorthms that they work with, they still have an intuition into how they work.
To recap, intuition comes into play throughout the data science process:
At the beginning of an analysis / project, one cannot realistically try every statistical test or every ML model, so one must rely on some intuition to decide where to start and to focus one’s efforts.
Once you have results of an analysis or ML model, how do you determine when you are done and have reliable results?
While the decision-making on the business side might come from a level above the data science team, in any case, some intuition is likely to be relied upon alongside the data analysis performed.