Alertmanager prometheus, Sending to invalid slack channel

up vote
1
down vote

favorite

We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.

So we have config that includes this:

slack_configs:
 - send_resolved: true
 channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '

(The %.11s ensures the channel name stays within the 21 character limit)

This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).

Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:

alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"

Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels

asked Nov 9 at 15:10

Steve Goodenough

New contributor

Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16

This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any channel_not_found errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19

Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago

Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago

Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday

add a comment |

up vote
1
down vote

favorite

We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.

So we have config that includes this:

slack_configs:
 - send_resolved: true
 channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '

(The %.11s ensures the channel name stays within the 21 character limit)

This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).

Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:

alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"

Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels

asked Nov 9 at 15:10

Steve Goodenough

New contributor

Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16

This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any channel_not_found errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19

Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago

Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago

Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday

add a comment |

up vote
1
down vote

favorite

We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.

So we have config that includes this:

slack_configs:
 - send_resolved: true
 channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '

(The %.11s ensures the channel name stays within the 21 character limit)

This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).

Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:

alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"

Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels

asked Nov 9 at 15:10

Steve Goodenough

New contributor

We have a multi tenant Kubernetes cluster and using Prometheus Alertmanager to send alerts to those tenants via slack.

So we have config that includes this:

slack_configs:
 - send_resolved: true
 channel: ' printf "topic-svc-%.11s" (index (index .Alerts 0).Labels "namespace") '

(The %.11s ensures the channel name stays within the 21 character limit)

This works great if the slack channel exists, but if the channel doesn't exist the alert ends up in the ether (not good for an alert!).

Alertmanager logs are pretty limited in what they tell you, e.g. it's a generic error with no user data:

alertmanager-k8s-0 alertmanager level=error ts=2018-11-09T15:01:52.134984182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for "slack" due to unrecoverable error: unexpected status code 404"

Tried all sorts of options, checked StackOverflow but seems all examples have simple fixed name for slack channels

slack prometheus-alertmanager

asked Nov 9 at 15:10

Steve Goodenough

New contributor

asked Nov 9 at 15:10

Steve Goodenough

New contributor

asked Nov 9 at 15:10

Steve Goodenough

New contributor

asked Nov 9 at 15:10

Steve Goodenough

asked Nov 9 at 15:10

Steve Goodenough

New contributor

Steve Goodenough is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16

This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any channel_not_found errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19

Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago

Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago

Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday

add a comment |

Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16

This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any channel_not_found errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19

Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago

Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago

Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday

Just to add, yes we know all our tenants :) but 1. in case adding a new tenant you forget to create the slack channel (or misspell it) and 2. We found out we were getting alerts for kube-system and obviously don't have a channel for that.
– Steve Goodenough
Nov 9 at 15:16

This is an interesting problem. You'd probably want to take a look at the logs for Alertmanager itself and find any channel_not_found errors from Slack Incoming Webhook calls.
– Adil B
Nov 9 at 18:19

Hey @AdilB, I'm guessing you mean slack logs (as I've already looked at Altermanager)? Going to be difficult as that app belongs to another team but I'll reach out to them.
– Steve Goodenough
2 days ago

Try out this tool: prometheus.io/webtools/alerting/routing-tree-editor Maybe it will help to find a routing problem.
– mibrl12
2 days ago

Thanks @mibrl12, yes we use that tool to check all our routing (slack is only one of the routes we use), but it just shows the route as going to slack not anything to do with what the slack channel name is.
– Steve Goodenough
yesterday

add a comment |

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53228336%2falertmanager-prometheus-sending-to-invalid-slack-channel%23new-answer', 'question_page');

);

Post as a guest

Name

active

oldest

votes

Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Steve Goodenough is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Post as a guest

Name

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Pfthb