How do you think we can notice those people who may be false negatives (ex. young people)?

Continuing discussion from Towards Data Science article, Scott Ogden asks:

“It appears that, even controlling for risk factors, the outcomes of this disease are bimodal and may depend on viral load. E.g. some young people get super sick. How do you think we can notice those people who may be false negatives? I was thinking myself about how to capture ‘even more severe risk’ and thought about creating a proxy model for patients most likely to die from a respiratory event. Is that something we could work on?”

Yes, the CV19 Index was built on a proxy outcome of complications from respiratory events. one of the improvements we’d like to make is to understand how to refine that proxy as we start getting information about the results of real COVID-19 cases. Even before we can build a predictor just off of COVID-19 outcomes, we can get a good idea of what are the most relevant proxies.

Can you explain how the xgboost predictor was trained? What were the targets? I am wondering if we could train an institution specific model.

we are training institution specific models daily via the ClosedLoop hosted version. Medical Home Network was the firs and there is a press release. There are multiple others in testing at the moment. What specifically did you have in mind?

At Strata, we are trying to help our clients model case mix for both financial forecasting and capacity planning. I was just wondering how the ClosedLoop pre-trained model would generalize to our clients’ claims data and whether it would be worthwhile to train on our institution-specific data.

The model we have today is built with Medicare data and is not really appropriate for a younger, generally healthy working age population. If you have a commercially insured population I’d strongly recommend retraining on your own population. If you can sign a BAA or deidentify your data we can do the retraining on our platform. If not, there is a description of how we defined the surrogate endpoint to train the model in our technical report, available at: https://arxiv.org/abs/2003.07347 .

You can use the same features in the open source model and the open source version of XGBoost to train.

Hi Dave, thanks for the insights and link to the technical report. We will try to retrain the predictor on our own data using a few types of targets. Thanks for your assistance!

I am retraining an xgboost model using our data on a different target (ICU stay- true/false). How do I save this trained model in a format that can be used by do_run?

Are you using the same inputs that our model uses?

It’s not quite built for this, but here is what you can do:

  • unpickle the original model file. That will be a dictionary, let’s call it “d”.
  • Replace d["model’] in the dictionary with your xgboost model.
  • update the d[‘hyper_params’][‘scale_pos_weight’] = 1.0
  • Compute quantiles for your test distribution (for example using np.quantile) and replace d[“prediction_quantiles”] with your computed quantiles. If is optional but if you don’t do it your “percentile” values will not be correct.
  • Pickle the updated dictionary and save in the same place

I know that was a lot. Let me know if any of that was confusing.

If you are using different inputs, then it gets trickier.

Hi Dave,

Thanks for those pointers. I think that all made sense…why do you set “scale_pos_weight” to 1 instead of the inverse ratio of “rare events?” I’m still trying to make sense of the risk scores we are seeing. I have an issue with getting the neg/pos factors to show up for each individual though. I am wondering if it would depend on the size of the training data. We currently have about 400k total (600 rare, prevalence of 0.15%).

Honestly, on the “scale_pos_weight” I was trying to keep the explanation simple. In practice I would recommend setting scale_pos_weight to the inverse ratio of “rare events”. Whatever value you train XGBoost with you should set in the model package. What the model package does is apply a correction factor to the predicted probabilties. If you overweight the rare events in training, you will end up with predicted probabilities that are much to high, and so this correction fixes that.

I’ll answer the factor question ina bit

On the factors, just adding in that scaling might fix the problem. If it doesn’t, try passing shap_cutoff=0.0 to the do_run function. Normally we have a cutoff to filter out insignificant SHAP values, but if your predictions are scaled differently then you might need a different value than our default here.

Actually, I got that wrong on the scale_pos_weight answer. You need to update the “train_data_stats” with an object that has ‘total_events’ and ‘rare_events’. This is what is used to do the rescaling.

Hi Dave, thanks for those suggestions. I had been passing along the “train_data_stats” dictionary so perhaps changing the shap_cutoff will help. I truly appreciate your support!

1 Like