Visualizing frscore results, part 1

This post is the first in a series of short texts about the frscore package, in which I describe upcoming changes in the package itself, and try to give some tips about how to make working with the package more convenient. This post and the next one are about visualizing the output of frscore() and frscored_cna().

Discussing with users of cna, I get the impression that the method is fairly often used specifically to discover or generate hypotheses about interactions between known causes of some outcome of interest. That is, the primary purpose of the analysis is not to discover which individual factors are causally relevant for the outcome as such, but to study how the causes of the outcome could be grouped into distinct, causally sufficient mechanisms for the outcome.

If there is reason to believe that the causes indeed do interact as components of such mechanisms, then the analyst is probably not interested in very simple models, since such models can at best confirm what is already known: that this or that factor is causally relevant for the outcome. At the same time, at least some very simple models quite often end up with high fr-scores, because they are submodels of numerous more complex models found at higher consistency and coverage thresholds. Nonetheless, frscore functions can be, and apparently are being used to carry out the analyses also in these kinds of settings, and the results can (I believe…) be useful. The point of the concept of fit-robustness is to gauge how idiosyncratic the causal claims made by a model are, when compared to models obtained using different consistency and coverage thresholds, when the data contain noise. The underlying assumption is that the models that erroneously causally interpret the noise (e.g. overfit) will only be returned at specific consistency and coverage settings, and can be “outed” by the idiosyncratic causal attributions they make. I see no reason why this idea wouldn’t apply when the primary interest is to discover or theorize about causal complexity, rather than discovering causal relevance, but in such a case the analyst will likely want to ignore the very simplest models even if they top the fr-score rankings.

The challenge then is to decide which models are complex enough to be interesting, but not too brittle and idiosyncratic. I have no take on how exactly these decisions should be made, but I was alerted to the fact that doing this often requires combing through the frscore results with some care, and this alone can be a fairly annoying task. One thing that may help here is that the results often tend to cluster: the models returned by frscored_cna() form “families”, the members of which have roughly equal scores. This is not always easy to see in the printed output of the functions, especially if there are more than twenty-ish unique model types. But it is immediately noticable if one were to visualize the scores: the distribution of the scores from the top to the bottom features some relatively flat plateaus separated by steeper downward drops, like a kinked staircase handrail. Looking closer at these plateaus one by one could then be a reasonable first step heuristic for finding the complex-enough but not-too-weird models. In any case, eyeballing the results cannot hurt.

However, plotting the scores per model type often results in a completely unreadable figure; there are just too many unique model types. Fortunately, it is easy to make a regular ggplot figure interactive, so that one can zoom into regions of the plot, and retrieve information about the data points by clicking or hovering the mouse over them. Below is a short example demonstrating how this can be done using the plotly package.

library(frscore)
library(forcats)
library(ggplot2)
library(plotly)


# generate some frscore results with frscored_cna()
res <- frscored_cna(d.error, fit.range = c(.9, .6),
                  granularity = .05,
                  maxsols = Inf)

# grab the frscore() results object to be plotted
res_frscore <- res[[1]]

# plot the results. we'll use the fct_inorder() function
# from the forcats package to prevent ggplot from
# messing up the order of the models.
# here we are plotting the unadjusted score --
# change the aes(y =) argument to norm.score to
# plot normalized scores instead
res_ggplot <- ggplot(
  res_frscore,
  aes(x = fct_inorder(condition),
      y = score,
      text = paste('model:', fct_inorder(condition),
                   'score:', score))
  ) +
  geom_point(size = 0.6) +
  theme(axis.text.x = element_text(size = 5, angle = 52)) +
  labs(x = "model")

# turn the ggplot object into an interactive plotly plot
# where hovering over the plotted scores shows the score
# and the model
res_plotly <- ggplotly(res_ggplot, tooltip = "text")
res_plotly

The figure generated by this code is shown below. I have used the d.error artificial data set that comes with frscore in order to keep the runtime of the analysis itself short, in case you want to run the code as-is. Hence this particular plot is almost readable also as a static figure, but the point here is merely to illustrate the basic functionality of the ggplot2+plotly combo for this purpose. When the number of unique model types gets to the hundreds, you’d start to actually benefit from the ability to zoom into specific regions of the figure. Hover over the figure to view the controls (zoom etc.) in the upper right corner. Hovering over the points displays the corresponding model, and the exact score.

This plot is obviously not very pretty, but it is something that I would imagine quickly doing to help inspect the results of frscore() and frscored_cna() when the number of unique model types is large. Please have a look at the documentation of ggplot2 and plotly packages to learn how to customize your own figures.

The next post will be about inspecting and visualizing networks of submodel relations between CNA models. Doing that will require a few tweaks to the frscore() function, such that it returns an appropriate network representation of the results in addition to the results table that it currently returns. These changes will be introduced to the development version of the package as soon as I manage to get them done, followed by the blog post.