Fixed Two Weeks of Silent Email Failures in EpsteinScan
Well, this is embarrassing. My DOJ monitor alerts have been silently failing since March 6th, and I just noticed today when I realized I hadn't gotten any notifications in a while.
Turns out it was a classic case of multiple things breaking at once. The main culprit was a SendGrid 403 error — I was trying to send emails FROM monitor@epsteinscan.org, but only cwaffiliateinvestments.com is actually verified as a sender domain in my SendGrid account. Rookie mistake.
But wait, there's more! While digging into this, I discovered the SendGrid API key in my crontab was completely expired. I had a perfectly good key sitting in /home/epsteinscan/.env, but the cron job was trying to use some old revoked key. No wonder nothing was working.
Here's what I fixed:
Changed the FROM addresses in both doj_monitor.py and monitor_sources.py from monitor@epsteinscan.org to alerts@cwaffiliateinvestments.com. While I was at it, I figured I'd fix the monitor_sources script too since it had the same broken FROM address.
Updated the crontab on my Hetzner server with the correct API key. Found it was referenced in two places, so I fixed both.
Tested everything with a couple of test emails to make sure SendGrid was happy with the new setup. Success!
The whole thing is deployed and committed (91fa92e). Now I just need to wait for the next cron run (every 6 hours) to confirm everything works in production.
Lesson learned: I really should set up some kind of health check for these monitoring scripts. It's pretty ironic when your monitoring system fails silently. Maybe I'll add a daily "I'm alive" ping or something.
Next on my TODO is consolidating these SendGrid API keys — having them scattered between .env files and crontabs is asking for trouble.