I’ve been going on about this one for a long time, but in case anyone still isn’t onboard with this principal I’ll state it another way: data disappearing from a feed is not a suitable trigger for action.
When we treat disappearance of data as a “trigger” we are interpreting a root cause from a received effect. We are saying “there can only ever be one possible reason why this piece of data has gone missing from our feed” – but is that really the case?
I’ve been pushed (protesting loudly) down this path a few times, and sooner or later something always goes wrong. A technical glitch, a patch, a “we didn’t realise if we clicked that button those records would be removed from the feed”. It was so bad in one environment that I had to introduce a manual step checking up on “deletes” before allowing them to flow through to deactivations. Kind of removed the effectiveness of the automation!
I’ve also seen (and attempted) protective measures like confirming an extraction process ran without errors, checking for an “end of file” line, or even storing a line count to compare to next time… but these band-aids add complexity to the solution and sometimes cause problems in their own right. They can also lead to a false sense of security – I have had a data collection script work perfectly for two years, and then fail to retrieve all the data one time without reporting an error. That sort of thing may be pretty much impossible to predict or test, but where the consequences can be disruptive it’s just not worth the risk.
So this one stands: Actions in the IAM Solution should only ever be triggered on received data, and never on the source data disappearing.
I will be adding to this series regularly between now and the end of December. Check back weekly to ensure you don’t miss any new posts!
Also, I would love to hear your thoughts on the topic, so please share them in a comment.