OpenNMS SNMP Traps Stopped Working - How to Further Troubleshoot
About 5 days ago, OpenNMS Horizon 22.02 on Ubuntu 18.04.1 LTS stopped accepting traps from network elements. No changes were made to configuration or underlying operating system to my knowledge.
There are about 125 network elements, all Cisco, sending traps.
So far I have checked the following:
- tcpdump shows the traps coming into the interface on port 162
- Turned on Debug for trapd.log and incoming traps from network elements do not create any log entries
- Traps sent with send-trap.pl from the localhost create traps that flow all the way to events
- Traps sent with snmptrap either on localhost or another host create log entries that flow all the way to events. The other host is using the same interface that the network elements are using.
- ss -lnpu sport = :162 shows an open UPD "UNCONN"
- sudo lsof -i :162 shows a single listener java process
- Startup of trapd does not seem to show any warnings in the log
- I have verified that the ufw and iptables are off
- I have updated OpenNMS to 22.04 and updated Ubunutu with no relief
- Restarted OpenNMS many many times...
- I moved Trapd startup after Asterisk in service-configuration.xml based on this
All of this seems similar to this. I think the last commenter on that thread asked about comparing the successful and unsuccessful traps in Wireshark which I have not done but all of the traps that are being sent have worked hundreds if not thousands of times until November 6th.
Is there anywhere else to look for errors as to why Trapd is not accepting traps? I think I have ruled out network issues.
I created a new Ubuntu 18.04 VM, updated it and then installed Horizon 23.01 fresh. I pointed my stream of traps at it and it behaves the exactly the same way, none of the traps create any log entries on the trapd.log with the level set to debug. Tcpdump shows the traps coming to the interface.
opennms
asked Nov 11 '18 at 20:02
Brian Kassa
12
add a comment |
About 5 days ago, OpenNMS Horizon 22.02 on Ubuntu 18.04.1 LTS stopped accepting traps from network elements. No changes were made to configuration or underlying operating system to my knowledge.
There are about 125 network elements, all Cisco, sending traps.
So far I have checked the following:
- tcpdump shows the traps coming into the interface on port 162
- Turned on Debug for trapd.log and incoming traps from network elements do not create any log entries
- Traps sent with send-trap.pl from the localhost create traps that flow all the way to events
- Traps sent with snmptrap either on localhost or another host create log entries that flow all the way to events. The other host is using the same interface that the network elements are using.
- ss -lnpu sport = :162 shows an open UPD "UNCONN"
- sudo lsof -i :162 shows a single listener java process
- Startup of trapd does not seem to show any warnings in the log
- I have verified that the ufw and iptables are off
- I have updated OpenNMS to 22.04 and updated Ubunutu with no relief
- Restarted OpenNMS many many times...
- I moved Trapd startup after Asterisk in service-configuration.xml based on this
All of this seems similar to this. I think the last commenter on that thread asked about comparing the successful and unsuccessful traps in Wireshark which I have not done but all of the traps that are being sent have worked hundreds if not thousands of times until November 6th.
Is there anywhere else to look for errors as to why Trapd is not accepting traps? I think I have ruled out network issues.
I created a new Ubuntu 18.04 VM, updated it and then installed Horizon 23.01 fresh. I pointed my stream of traps at it and it behaves the exactly the same way, none of the traps create any log entries on the trapd.log with the level set to debug. Tcpdump shows the traps coming to the interface.
opennms
asked Nov 11 '18 at 20:02
Brian Kassa
12
add a comment |
About 5 days ago, OpenNMS Horizon 22.02 on Ubuntu 18.04.1 LTS stopped accepting traps from network elements. No changes were made to configuration or underlying operating system to my knowledge.
There are about 125 network elements, all Cisco, sending traps.
So far I have checked the following:
- tcpdump shows the traps coming into the interface on port 162
- Turned on Debug for trapd.log and incoming traps from network elements do not create any log entries
- Traps sent with send-trap.pl from the localhost create traps that flow all the way to events
- Traps sent with snmptrap either on localhost or another host create log entries that flow all the way to events. The other host is using the same interface that the network elements are using.
- ss -lnpu sport = :162 shows an open UPD "UNCONN"
- sudo lsof -i :162 shows a single listener java process
- Startup of trapd does not seem to show any warnings in the log
- I have verified that the ufw and iptables are off
- I have updated OpenNMS to 22.04 and updated Ubunutu with no relief
- Restarted OpenNMS many many times...
- I moved Trapd startup after Asterisk in service-configuration.xml based on this
All of this seems similar to this. I think the last commenter on that thread asked about comparing the successful and unsuccessful traps in Wireshark which I have not done but all of the traps that are being sent have worked hundreds if not thousands of times until November 6th.
Is there anywhere else to look for errors as to why Trapd is not accepting traps? I think I have ruled out network issues.
I created a new Ubuntu 18.04 VM, updated it and then installed Horizon 23.01 fresh. I pointed my stream of traps at it and it behaves the exactly the same way, none of the traps create any log entries on the trapd.log with the level set to debug. Tcpdump shows the traps coming to the interface.
opennms
asked Nov 11 '18 at 20:02
Brian Kassa
12
About 5 days ago, OpenNMS Horizon 22.02 on Ubuntu 18.04.1 LTS stopped accepting traps from network elements. No changes were made to configuration or underlying operating system to my knowledge.
There are about 125 network elements, all Cisco, sending traps.
So far I have checked the following:
- tcpdump shows the traps coming into the interface on port 162
- Turned on Debug for trapd.log and incoming traps from network elements do not create any log entries
- Traps sent with send-trap.pl from the localhost create traps that flow all the way to events
- Traps sent with snmptrap either on localhost or another host create log entries that flow all the way to events. The other host is using the same interface that the network elements are using.
- ss -lnpu sport = :162 shows an open UPD "UNCONN"
- sudo lsof -i :162 shows a single listener java process
- Startup of trapd does not seem to show any warnings in the log
- I have verified that the ufw and iptables are off
- I have updated OpenNMS to 22.04 and updated Ubunutu with no relief
- Restarted OpenNMS many many times...
- I moved Trapd startup after Asterisk in service-configuration.xml based on this
All of this seems similar to this. I think the last commenter on that thread asked about comparing the successful and unsuccessful traps in Wireshark which I have not done but all of the traps that are being sent have worked hundreds if not thousands of times until November 6th.
Is there anywhere else to look for errors as to why Trapd is not accepting traps? I think I have ruled out network issues.
I created a new Ubuntu 18.04 VM, updated it and then installed Horizon 23.01 fresh. I pointed my stream of traps at it and it behaves the exactly the same way, none of the traps create any log entries on the trapd.log with the level set to debug. Tcpdump shows the traps coming to the interface.
opennms
opennms
asked Nov 11 '18 at 20:02
Brian Kassa
12
asked Nov 11 '18 at 20:02
Brian Kassa
12
edited Nov 11 '18 at 22:38
asked Nov 11 '18 at 20:02
Brian Kassa
12
asked Nov 11 '18 at 20:02
Brian Kassa
12
asked Nov 11 '18 at 20:02
Brian Kassa
12
12
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Issue Resolved.
The underlying operating system lost its static route for the subnet that the traps were coming from. OpenNMS had a route back to the subnet but not via the path that the traps were coming in from. Once the static route was restored, traps started working again and were flowing all the way to events.
answered Nov 12 '18 at 16:49
Brian Kassa
12
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252701%2fopennms-snmp-traps-stopped-working-how-to-further-troubleshoot%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Issue Resolved.
The underlying operating system lost its static route for the subnet that the traps were coming from. OpenNMS had a route back to the subnet but not via the path that the traps were coming in from. Once the static route was restored, traps started working again and were flowing all the way to events.
answered Nov 12 '18 at 16:49
Brian Kassa
12
add a comment |
Issue Resolved.
The underlying operating system lost its static route for the subnet that the traps were coming from. OpenNMS had a route back to the subnet but not via the path that the traps were coming in from. Once the static route was restored, traps started working again and were flowing all the way to events.
answered Nov 12 '18 at 16:49
Brian Kassa
12
add a comment |
Issue Resolved.
The underlying operating system lost its static route for the subnet that the traps were coming from. OpenNMS had a route back to the subnet but not via the path that the traps were coming in from. Once the static route was restored, traps started working again and were flowing all the way to events.
answered Nov 12 '18 at 16:49
Brian Kassa
12
Issue Resolved.
The underlying operating system lost its static route for the subnet that the traps were coming from. OpenNMS had a route back to the subnet but not via the path that the traps were coming in from. Once the static route was restored, traps started working again and were flowing all the way to events.
answered Nov 12 '18 at 16:49
Brian Kassa
12
answered Nov 12 '18 at 16:49
Brian Kassa
12
answered Nov 12 '18 at 16:49
Brian Kassa
12
answered Nov 12 '18 at 16:49
Brian Kassa
12
12
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252701%2fopennms-snmp-traps-stopped-working-how-to-further-troubleshoot%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
