frscore 0.3.1

A new version of frscore is now on CRAN. Alas, this version does not deliver the features I promised in an earlier post. Other issues were simply deemed more urgent at this time. The NEWS file gives a bullet-point summary of changes. This post is an attempt to give an overview of a conceptual issue that necessitated the biggest change in the package. Due to this change, some old results generated with frscored_cna() and frscore() may not be reproducible with the new default behaviors, hence this post. Other developments are explained at the end of the post.

Measuring agreement in causal relevance ascriptions

In Parkkinen & Baumgartner (2021) we describe the conceptual background of the procedure implemented in the frscore functions. The idea is that for a cna or other CCM model, robustness against changes in consistency and coverage settings need not mean that the very same model is inferred at many different settings. Rather, what matters is whether or not the causal ascriptions made by a model are in agreement with (many) other models inferred at different settings. In practice, the frscore functions implement this so that models inferred at different con/cov settings are checked against each other with cna::is.submodel(), counting the number of sub- and supermodels each model has in the whole lot. This number is each model’s unadjusted fit-robustness score.

The aforementioned paper and the manual page for cna::is.submodel() both give wordy definitions of the submodel relation, but in short, each atomic component model (“asf”) of a submodel must have a counterpart asf in the supermodel such that the former could be generated by eliminating symbols from the latter. E.g. the single-asf model A+B<->C is a submodel of A*Y+B<->C, as we can get to the former by removing *Y from the latter. To be specific, we might call this relation a syntactic submodel relation, as it involves no interpretation, causal or otherwise, of the meaning of the symbols that appear in the models. When we interpret the models causally, a syntactic submodel relation means that all claims about direct causal relevance made by the submodel are contained in the supermodel. This is because claims of direct causation are explicitly represented by the asf syntax: anything on the left hand side of the equivalence sign "<->" is a direct cause of the outcome on the right hand side.

In the context of frscore, the problem with this is twofold. First, complex models comprising more than one asf can make indirect causal relevance ascriptions that cannot be read off the overt syntax of the model. Second, the concept of direct causation is relative to (the set of factors included in) a model. For example, (A+B<->C)*(C+D<->E) claims that A and B are indirectly causally relevant for E in virtue of causing C. Since every causal relevance claim corresponds to a claim about difference-making, this means that the model claims that there are circumstances in which wiggling A while suppressing B, or the other way around, makes a difference to E (via making a difference to C). Now think of A+B<->E. A+B<->E describes those same causal relations between {A,B,E}, entailing the exact same difference-making relations between those three factors, but does so at a coarser granularity, omitting the mediating factor C. While (A+B<->C)*(C+D<->E) gives a more detailed description of the causal process than A+B<->E, the latter is clearly not making claims about causal relevance that conflict the more detailed model. Hence, if these were models inferred from the same data set, the frscore functions should recognize that they agree in their causal relevance ascriptions. But when models are compared using cna::is.submodel(), this is not recognized: A+B<->E is not a syntactic submodel of (A+B<->C)*(C+D<->E), because there is no suitable counterpart asf in the latter for A+B<->E.

Consider another pair of models: (A+B*D<->C)*(C<->G) and (A+B*D<->C)*(C+D<->G). Here, the former model is a syntactic submodel of the latter. Yet, the former makes a causal relevance claim that disagrees with the latter. (A+B*D<->C)*(C<->G) claims that A, B and D are indirectly relevant for G in virtue of causing C. The latter model claims that A is indirectly relevant for G, D directly so, and B not causally relevant for G at all. It may be easier to make this conflict obvious by considering the associated difference-making claims. (A+B*D<->C)*(C<->G) claims that wiggling B makes a difference to G when D is present and A is absent. By contrast, (A+B*D<->C)*(C+D<->G) claims that this is impossible: According to (A+B*D<->C)*(C+D<->G), G is always present whenever D is. Yet, the old frscore functions that relied on cna::is.submodel() would judge this pair of models to be free of conflicting causal relevance ascriptions.

Causal submodels

To fix this problem, frscore 0.3.1 introduces a new function causal_submodel() that is like cna::is.submodel(), except that it checks whether all causal relevance ascriptions, direct or indirect, of one model have a suitable[1] counterpart in another model. If yes, one is a causal submodel of the other. The syntax of causal_submodel() resembles cna::is.submodel(): the candidate submodel to be tested is given as the first argument, and the target supermodel as the second. A notable difference is that unlike cna::is.submodel(), you cannot (for now) give causal_submodel() a vector of more than one candidate models at a time. For processing multi-valued models, causal_submodel() will also need to know the range of admissible factor values. This is provided via the argument dat, and would typically be the data set from which the models were inferred.

By default, frscored_cna() and frscore() will now use causal_submodel() to compare models when calculating fr-scores. This induces a slight performance penalty compared to the older versions of these functions. In case performance is crucial, or one wishes to replicate old results, frscored_cna() and frscore() functions have a new argument comp.method that allows the user to switch to the old behavior. comp.method takes values "causal_submodel" (default) or "is.submodel". With the default value, fr-scores are counted using causal_submodel(), and with comp.method = "is.submodel", cna::is.submodel()[2] is used instead. According to our tests, the differences in the results are small between these two options, so using comp.method = "is.submodel" might still be reasonable for some users. We nonetheless recommend using the default value when possible, as this more closely tracks the kind of robustness we intended.

When processing multi-valued models with frscore() using comp.method = "causal_submodel", the admissible factor value range must be provided as the (new) argument dat, similarly to causal_submodel(). In frscored_cna(), the user-supplied data set is automatically used for this purpose.

Below are some examples of causal_submodel() output compared to cna::is.submodel().

if(compareVersion(as.character(packageVersion("frscore")), "0.3.1") > 0){
  install.packages("frscore") # update package if needed
}
library(frscore)

## Loading required package: cna

## Registered S3 method overwritten by 'cna':
##   method          from
##   some.data.frame car

## Loading required package: lifecycle

target <- "(A+B<->C)*(C+D<->E)"
candidate <- "A+B<->E"
is.submodel(x = candidate, y = target)

## A+B<->E
##   FALSE
## attr(,"target")
## [1] "(A+B<->C)*(C+D<->E)"

causal_submodel(x = candidate, y = target)

## A+B<->E
##    TRUE
## attr(,"target")
## [1] "(A+B<->C)*(C+D<->E)"

target2 <- "(A+B*D<->C)*(C+D<->G)"
candidate2 <- "(A+B*D<->C)*(C<->G)"
is.submodel(candidate2, target2)

## (A+B*D<->C)*(C<->G)
##                TRUE
## attr(,"target")
## [1] "(A+B*D<->C)*(C+D<->G)"

causal_submodel(candidate2, target2)

## (A+B*D<->C)*(C<->G)
##               FALSE
## attr(,"target")
## [1] "(A+B*D<->C)*(C+D<->G)"

# two models inferred from the `d.pban` example data set
mv_target <- "(C=1 + C=2 + F=2 <-> PB=1)*(F=1 + F=2 <-> T=2)"
mv_candidate <- "(T=1 + V=0 <-> PB=1)*(F=1 + F=2 <-> T=2)"

# causal_submodel() needs the data to process mv models
causal_submodel(x = mv_candidate, y = mv_target, dat = d.pban)

## (T=1+V=0<->PB=1)*(F=1+F=2<->T=2)
##                            FALSE
## attr(,"target")
## [1] "(C=1+C=2+F=2<->PB=1)*(F=1+F=2<->T=2)"

causal_submodel() is an approximation of a strictly speaking valid test for containment of causal relevance claims between models, even though a closer approximation than cna::is.submodel(). Internally, the inputted models need to be manipulated in various ways to explicate all indirect causal ascriptions, which involves minimizing the manipulated models to remove redundancies. To speed things up, the minimization relies on cna::rreduce(), which randomly chooses a single minimization path whenever faced with ambiguous models. In practice, this may result in the output of causal_submodel() changing between subsequent calls on the same pair of models, if those models happen to be ambiguous. This however happens rarely enought to not affect the frscore functions in a meaningful way. Models that describe causal feedback cycles present another unique problem. Such models claim that some factors are causally relevant for themselves, as in (A+B<->C)*(C+D<->A). It is not conceptually entirely clear how comparisons between cyclic and non-cyclic models should be conducted. Here causal_submodel() takes the least costly option and simply checks whether a syntactic submodel relation is present between the candidate and the target models, and returns the result.

Other changes

The scoretype argument of frscore() and frscored_cna() is now deprecated, and will be removed in the next version of the package. The reason for this is simple: If you changed the scoretype value from its default, thereby forcing the scores to be counted based on either sub- or supermodel relations only, then the scores would not reflect the robustness of the models as we defined it in Parkkinen & Baumgartner (2021). We can offer no guidance as to how such results should be interpreted, so having the option to change how the scores are calculated can only lead to confusion. If one is mostly interested in complex (resp. simple) models, one should explicitly argue for one’s choice of focus, rather than changing the fr-scoring routine to arbitrarily favor one’s favorite model(s). After running the whole fr-scoring routine with frscored_cna(), one can always view all the models returned in a reanalysis series, and the score composition for every model type, by inspecting the respective elements in the return object. From there, one can calculate any other statistics about the returned models one wishes.

The ncsf argument in rean_cna() is deprecated in favor of n.init, which does exactly the same thing: its value determines the maximum number of complex models (“csf”s) computed. Note that this only applies when output = "csf" (the default). In practice, rean_cna() is primarily a helper function called by frscored_cna() for performing a reanalysis series; I’d imagine it to be very rare for a regular user to need it directly. Hence, n.init is also added as a new explicit argument to frscored_cna(), to make it obvious to the user that one can control the maximum number of csfs if needed, and to make eventual warnings about n.init value that actually arise from cna::csf() less confusing. This change brings the syntax of both of these functions in line with the cna functions that they rely on anyway. The default value of n.init is also the same as in the relevant cna functions. As the cna functions have become more efficient, imposing a lower limit is usually no longer needed.

frscore() gains an argument dat, which was covered above.

[1] See the manual page for causal_submodel() for an explanation of what counts as a “suitable” counterpart.

[2] To be precise, the package now uses a faster, internal implementation of cna::is.submodel, thanks to generous help of cna maintainer Mathias Ambühl.