Slack notifications from backends without the noise

The failure mode

You add Slack notifications. The team reads them for a week. Then a few false-positive alerts fire, then a few duplicate alerts. By month two, the channel is muted and nobody reads it.

You did not build alerting. You built noise.

The rules

One channel per severity, not per service

• #alerts-page — wakes someone up at night. Must be real.
• #alerts-warn — daytime attention. Looked at within an hour.
• #alerts-info — diagnostic, low priority. Skimmed once a day.

If everything goes into one channel, everything gets ignored equally.

Deduplicate at the source

Five errors of the same type in a minute should be one Slack message that says "this is happening 5x." Not five messages.

Most alert systems support this (Sentry's "issue" model does it well). If you are rolling your own, debounce or use a tiny Redis cache to track recent alerts.

Include the link

Every alert should have a link to the dashboard, log, or Sentry issue. "Payment failed" with no link is useless. "Payment failed: <link to Sentry>" is actionable in 5 seconds.

No alerts for known noise

If you know a third-party fires false errors on Tuesdays, do not page on Tuesday errors. Filter them. Annotate them.

If you cannot filter automatically, add a manual "this is known noise" Slack workflow and let the on-call mute it.

Implementation

For Python:

import httpx

async def notify_slack(channel: str, text: str, link: str | None = None):
    payload = {"text": text}
    if link:
        payload["blocks"] = [
            {"type": "section", "text": {"type": "mrkdwn", "text": text}},
            {"type": "actions", "elements": [
                {"type": "button", "text": {"type": "plain_text", "text": "Open"}, "url": link}
            ]},
        ]
    await httpx.AsyncClient().post(SLACK_WEBHOOK_URL, json=payload)

One function. Used from anywhere. Wrap it with a rate limiter (~5 messages per minute per key) so a runaway loop does not flood Slack.

What we will not do

We will not Slack-notify on successful runs. "Payments processed: 47" every five minutes is noise. Build a dashboard for that.

Slack is for things that need human attention. Anything else belongs in a metric.