Alertmanager prometheus, Sending to invalid slack channel
up vote
1
down vote
favorite
We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.
So we have config that includes this:
slack_configs:
- send_resolved: true
channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '
(The %.11s ensures the channel name stays within the 21 character limit)
This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).
Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:
alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"
Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels
slack prometheus-alertmanager
New contributor
add a comment |
up vote
1
down vote
favorite
We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.
So we have config that includes this:
slack_configs:
- send_resolved: true
channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '
(The %.11s ensures the channel name stays within the 21 character limit)
This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).
Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:
alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"
Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels
slack prometheus-alertmanager
New contributor
Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16
This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find anychannel_not_found
errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19
Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago
Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago
Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.
So we have config that includes this:
slack_configs:
- send_resolved: true
channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '
(The %.11s ensures the channel name stays within the 21 character limit)
This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).
Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:
alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"
Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels
slack prometheus-alertmanager
New contributor
We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.
So we have config that includes this:
slack_configs:
- send_resolved: true
channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '
(The %.11s ensures the channel name stays within the 21 character limit)
This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).
Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:
alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"
Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels
slack prometheus-alertmanager
slack prometheus-alertmanager
New contributor
New contributor
New contributor
asked Nov 9 at 15:10
Steve Goodenough
62
62
New contributor
New contributor
Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16
This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find anychannel_not_found
errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19
Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago
Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago
Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday
add a comment |
Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16
This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find anychannel_not_found
errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19
Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago
Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago
Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday
Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16
Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16
This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any
channel_not_found
errors from Slack Incoming Webhook calls.– Adil B
Nov 9 at 18:19
This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any
channel_not_found
errors from Slack Incoming Webhook calls.– Adil B
Nov 9 at 18:19
Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago
Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago
Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago
Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago
Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday
Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.
Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.
Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.
Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53228336%2falertmanager-prometheus-sending-to-invalid-slack-channel%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16
This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any
channel_not_found
errors from Slack Incoming Webhook calls.– Adil B
Nov 9 at 18:19
Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago
Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago
Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday