Beware of False Positive Accumulation
My first gig as a data scientist, at IMVU, I was tasked with building a random forest model to prevent credit card chargebacks. The model was pretty simple. It would build features about the transaction and the user and use those to estimate the probability of a chargeback. We had a large, beautiful dataset of previous chargebacks, because unfortunatly at the time we were getting a lot of them. The model didn’t need to be very complex to do a decent job, that was not the interesting part of the project.