System wide notifications with proper Slack integration

[bimtauer]

Flyte currently only allows sending notifications by email and requires users to explicitly define notification rules in their launchplans.

A better notifications setup would enable

• configuring system-wide alerts (no need to explicitly enable notifications on a per-launchplan basis)
• limiting alerts to a subset of available states, i.e. receive notification for all workflow failures and aborts
• limit for certain domains/projects/workflows
• message templating for Slack

As an inspiration I would point to ArgoCD Notifications, I think they implemented a very nice solution with triggers, templates and subscriptions.

[davidmirror-ops]

@bimtauer Sorry for the delay
Watch here the discussion around this idea. The general consensus is that it's a great one and it fits in the scope of an RFC

See the RFC process here and let us know any question/comment

Thank you!

[davidmirror-ops]

2023-11-09 Contributor's meetup notes: this entry needs a champion to make it an RFC. It's dependent on the development of Webhooks: #4147