Hortonworks HDFS Name Node tryLock issue on startup
We're using HDP HDFS module version 2.7.3.2.6.5.0-292.
The server was stuck and had to be hard-reseted - now the Name Node service throws an error upon start up.
After successfully acquiring a lock file, it instantaneously fails by trying to acquire it again, even through it's for the same process (presumably the same thread) - it fails.
How should we start the name node with the data in tact?
18/11/14 20:19:24 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/11/14 20:19:24 INFO util.GSet: VM type = 64-bit
18/11/14 20:19:24 INFO util.GSet: 0.029999999329447746% max memory 1011.3 MB = 310.7 KB
18/11/14 20:19:25 INFO util.GSet: capacity = 2^15 = 32768 entries
18/11/14 20:19:25 INFO common.Storage: Lock on /mnt/pd1/hadoop/hdfs/namenode/in_use.lock acquired by nodename 10635@hadoop-327
18/11/14 20:19:25 ERROR common.Storage: It appears that another node 10635@hadoop-327 has already locked the storage directory: /mnt/pd1/hadoop/hdfs/namenode
java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1113)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:770)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:738)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:551)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:502)
java hadoop hdfs hortonworks-data-platform namenode
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
add a comment |
We're using HDP HDFS module version 2.7.3.2.6.5.0-292.
The server was stuck and had to be hard-reseted - now the Name Node service throws an error upon start up.
After successfully acquiring a lock file, it instantaneously fails by trying to acquire it again, even through it's for the same process (presumably the same thread) - it fails.
How should we start the name node with the data in tact?
18/11/14 20:19:24 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/11/14 20:19:24 INFO util.GSet: VM type = 64-bit
18/11/14 20:19:24 INFO util.GSet: 0.029999999329447746% max memory 1011.3 MB = 310.7 KB
18/11/14 20:19:25 INFO util.GSet: capacity = 2^15 = 32768 entries
18/11/14 20:19:25 INFO common.Storage: Lock on /mnt/pd1/hadoop/hdfs/namenode/in_use.lock acquired by nodename 10635@hadoop-327
18/11/14 20:19:25 ERROR common.Storage: It appears that another node 10635@hadoop-327 has already locked the storage directory: /mnt/pd1/hadoop/hdfs/namenode
java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1113)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:770)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:738)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:551)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:502)
java hadoop hdfs hortonworks-data-platform namenode
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
add a comment |
We're using HDP HDFS module version 2.7.3.2.6.5.0-292.
The server was stuck and had to be hard-reseted - now the Name Node service throws an error upon start up.
After successfully acquiring a lock file, it instantaneously fails by trying to acquire it again, even through it's for the same process (presumably the same thread) - it fails.
How should we start the name node with the data in tact?
18/11/14 20:19:24 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/11/14 20:19:24 INFO util.GSet: VM type = 64-bit
18/11/14 20:19:24 INFO util.GSet: 0.029999999329447746% max memory 1011.3 MB = 310.7 KB
18/11/14 20:19:25 INFO util.GSet: capacity = 2^15 = 32768 entries
18/11/14 20:19:25 INFO common.Storage: Lock on /mnt/pd1/hadoop/hdfs/namenode/in_use.lock acquired by nodename 10635@hadoop-327
18/11/14 20:19:25 ERROR common.Storage: It appears that another node 10635@hadoop-327 has already locked the storage directory: /mnt/pd1/hadoop/hdfs/namenode
java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1113)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:770)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:738)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:551)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:502)
java hadoop hdfs hortonworks-data-platform namenode
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
We're using HDP HDFS module version 2.7.3.2.6.5.0-292.
The server was stuck and had to be hard-reseted - now the Name Node service throws an error upon start up.
After successfully acquiring a lock file, it instantaneously fails by trying to acquire it again, even through it's for the same process (presumably the same thread) - it fails.
How should we start the name node with the data in tact?
18/11/14 20:19:24 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/11/14 20:19:24 INFO util.GSet: VM type = 64-bit
18/11/14 20:19:24 INFO util.GSet: 0.029999999329447746% max memory 1011.3 MB = 310.7 KB
18/11/14 20:19:25 INFO util.GSet: capacity = 2^15 = 32768 entries
18/11/14 20:19:25 INFO common.Storage: Lock on /mnt/pd1/hadoop/hdfs/namenode/in_use.lock acquired by nodename 10635@hadoop-327
18/11/14 20:19:25 ERROR common.Storage: It appears that another node 10635@hadoop-327 has already locked the storage directory: /mnt/pd1/hadoop/hdfs/namenode
java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1113)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:770)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:738)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:551)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:502)
java hadoop hdfs hortonworks-data-platform namenode
java hadoop hdfs hortonworks-data-platform namenode
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
asked Nov 14 '18 at 20:36
Eran BetzalelEran Betzalel
2,86122962
2,86122962
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
So it appears that we had 2 paths that target the same directory at dfs.namenode.name.dir, which caused the double lock. Once we used a single path, everything was back to order.
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53308351%2fhortonworks-hdfs-name-node-trylock-issue-on-startup%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
So it appears that we had 2 paths that target the same directory at dfs.namenode.name.dir, which caused the double lock. Once we used a single path, everything was back to order.
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
add a comment |
So it appears that we had 2 paths that target the same directory at dfs.namenode.name.dir, which caused the double lock. Once we used a single path, everything was back to order.
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
add a comment |
So it appears that we had 2 paths that target the same directory at dfs.namenode.name.dir, which caused the double lock. Once we used a single path, everything was back to order.
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
So it appears that we had 2 paths that target the same directory at dfs.namenode.name.dir, which caused the double lock. Once we used a single path, everything was back to order.
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
answered Nov 18 '18 at 9:51
Eran BetzalelEran Betzalel
2,86122962
2,86122962
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53308351%2fhortonworks-hdfs-name-node-trylock-issue-on-startup%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
