Writing data in hive warehouse directory in two separate tables using flume
0
I want to write data in hive warehouse directory, in two separate tables called flumemaleemployee and flumefemaleemployee. Last 3 records should be inserted in the female table and upper 3 records should be inserted in male table.Below is my data:
1,alok,mumbai
1,jatin,chennai
1,yogesh,kolkata
2,ragini,delhi
2,jyotsana,pune
1,valmiki,banglore
Below is my flume conf code:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = stdl std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/Desktop/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.il.regex = A(\d}
agent.sources.tailsrc. interceptors. M.serializers = t1
agent.sources.tailsrc. interceptors, i1.serializers.t1. name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.stdl.channel = mem1
agent.sinks.stdl.batchSize = 1
agent.sinks.std1.hdfs.path = /user/hive/warehouse/aisehibanayatp.db/flumemaleemployee
agent.sinks.stdl.rolllnterval = 0
agent.sinks.stdl.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = /user/hi ve/warehouse/aisehibanayatp.db/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
I am not getting any error but when I start flume service with below command it just stucks at something I don't know how to deal with that since I am not getting any error
flume-ng agent --name agent -conf-file /home/cloudera/Desktop/flume1.config
and it stucks at below step:
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: mem2. channel.event.take.success == 0
18/11/13 08:03:00 INFO node.Application: Starting new configuration: sourceRunners: sinkRunners:std2=SinkRunner: policy:org.apache.flume.sink.DefaultSinkProcessor@17ade71c counterGroup: name:null counters: channels:mem2=org.apache.flume.channel.MemoryChannelname: mem2
18/11/13 08:03:00 INFO node.Application: Starting Channel mem2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: mem2 started
18/11/13 08:03:00 INFO node.Application: Starting Sink std2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: std2: Successfully registered new MBean.
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: std2 started
So how can I achieve this ??
apache hadoop bigdata flume flume-ng
asked Nov 13 '18 at 16:12
RushHourRushHour
969
|
show 4 more comments
0
I want to write data in hive warehouse directory, in two separate tables called flumemaleemployee and flumefemaleemployee. Last 3 records should be inserted in the female table and upper 3 records should be inserted in male table.Below is my data:
1,alok,mumbai
1,jatin,chennai
1,yogesh,kolkata
2,ragini,delhi
2,jyotsana,pune
1,valmiki,banglore
Below is my flume conf code:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = stdl std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/Desktop/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.il.regex = A(\d}
agent.sources.tailsrc. interceptors. M.serializers = t1
agent.sources.tailsrc. interceptors, i1.serializers.t1. name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.stdl.channel = mem1
agent.sinks.stdl.batchSize = 1
agent.sinks.std1.hdfs.path = /user/hive/warehouse/aisehibanayatp.db/flumemaleemployee
agent.sinks.stdl.rolllnterval = 0
agent.sinks.stdl.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = /user/hi ve/warehouse/aisehibanayatp.db/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
I am not getting any error but when I start flume service with below command it just stucks at something I don't know how to deal with that since I am not getting any error
flume-ng agent --name agent -conf-file /home/cloudera/Desktop/flume1.config
and it stucks at below step:
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: mem2. channel.event.take.success == 0
18/11/13 08:03:00 INFO node.Application: Starting new configuration: sourceRunners: sinkRunners:std2=SinkRunner: policy:org.apache.flume.sink.DefaultSinkProcessor@17ade71c counterGroup: name:null counters: channels:mem2=org.apache.flume.channel.MemoryChannelname: mem2
18/11/13 08:03:00 INFO node.Application: Starting Channel mem2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: mem2 started
18/11/13 08:03:00 INFO node.Application: Starting Sink std2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: std2: Successfully registered new MBean.
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: std2 started
So how can I achieve this ??
apache hadoop bigdata flume flume-ng
asked Nov 13 '18 at 16:12
RushHourRushHour
969
Not sure I understand the regex. Should it not be based on value of first col, 1 or 2?
– thebluephantom
Nov 13 '18 at 18:01
looks ok at first glance
– thebluephantom
Nov 13 '18 at 18:19
yeah it doesn't throw any error
– RushHour
Nov 13 '18 at 18:20
my first comment? can you find output files?
– thebluephantom
Nov 13 '18 at 18:23
I do not think there is an error
– thebluephantom
Nov 13 '18 at 18:25
|
show 4 more comments
0
0
0
I want to write data in hive warehouse directory, in two separate tables called flumemaleemployee and flumefemaleemployee. Last 3 records should be inserted in the female table and upper 3 records should be inserted in male table.Below is my data:
1,alok,mumbai
1,jatin,chennai
1,yogesh,kolkata
2,ragini,delhi
2,jyotsana,pune
1,valmiki,banglore
Below is my flume conf code:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = stdl std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/Desktop/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.il.regex = A(\d}
agent.sources.tailsrc. interceptors. M.serializers = t1
agent.sources.tailsrc. interceptors, i1.serializers.t1. name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.stdl.channel = mem1
agent.sinks.stdl.batchSize = 1
agent.sinks.std1.hdfs.path = /user/hive/warehouse/aisehibanayatp.db/flumemaleemployee
agent.sinks.stdl.rolllnterval = 0
agent.sinks.stdl.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = /user/hi ve/warehouse/aisehibanayatp.db/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
I am not getting any error but when I start flume service with below command it just stucks at something I don't know how to deal with that since I am not getting any error
flume-ng agent --name agent -conf-file /home/cloudera/Desktop/flume1.config
and it stucks at below step:
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: mem2. channel.event.take.success == 0
18/11/13 08:03:00 INFO node.Application: Starting new configuration: sourceRunners: sinkRunners:std2=SinkRunner: policy:org.apache.flume.sink.DefaultSinkProcessor@17ade71c counterGroup: name:null counters: channels:mem2=org.apache.flume.channel.MemoryChannelname: mem2
18/11/13 08:03:00 INFO node.Application: Starting Channel mem2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: mem2 started
18/11/13 08:03:00 INFO node.Application: Starting Sink std2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: std2: Successfully registered new MBean.
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: std2 started
So how can I achieve this ??
apache hadoop bigdata flume flume-ng
asked Nov 13 '18 at 16:12
RushHourRushHour
969
I want to write data in hive warehouse directory, in two separate tables called flumemaleemployee and flumefemaleemployee. Last 3 records should be inserted in the female table and upper 3 records should be inserted in male table.Below is my data:
1,alok,mumbai
1,jatin,chennai
1,yogesh,kolkata
2,ragini,delhi
2,jyotsana,pune
1,valmiki,banglore
Below is my flume conf code:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = stdl std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/Desktop/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.il.regex = A(\d}
agent.sources.tailsrc. interceptors. M.serializers = t1
agent.sources.tailsrc. interceptors, i1.serializers.t1. name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.stdl.channel = mem1
agent.sinks.stdl.batchSize = 1
agent.sinks.std1.hdfs.path = /user/hive/warehouse/aisehibanayatp.db/flumemaleemployee
agent.sinks.stdl.rolllnterval = 0
agent.sinks.stdl.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = /user/hi ve/warehouse/aisehibanayatp.db/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
I am not getting any error but when I start flume service with below command it just stucks at something I don't know how to deal with that since I am not getting any error
flume-ng agent --name agent -conf-file /home/cloudera/Desktop/flume1.config
and it stucks at below step:
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: mem2. channel.event.take.success == 0
18/11/13 08:03:00 INFO node.Application: Starting new configuration: sourceRunners: sinkRunners:std2=SinkRunner: policy:org.apache.flume.sink.DefaultSinkProcessor@17ade71c counterGroup: name:null counters: channels:mem2=org.apache.flume.channel.MemoryChannelname: mem2
18/11/13 08:03:00 INFO node.Application: Starting Channel mem2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: mem2 started
18/11/13 08:03:00 INFO node.Application: Starting Sink std2
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: std2: Successfully registered new MBean.
18/11/13 08:03:00 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: std2 started
So how can I achieve this ??
apache hadoop bigdata flume flume-ng
apache hadoop bigdata flume flume-ng
asked Nov 13 '18 at 16:12
RushHourRushHour
969
asked Nov 13 '18 at 16:12
RushHourRushHour
969
asked Nov 13 '18 at 16:12
RushHourRushHour
969
asked Nov 13 '18 at 16:12
RushHourRushHour
969
asked Nov 13 '18 at 16:12
RushHourRushHour
969
969
Not sure I understand the regex. Should it not be based on value of first col, 1 or 2?
– thebluephantom
Nov 13 '18 at 18:01
looks ok at first glance
– thebluephantom
Nov 13 '18 at 18:19
yeah it doesn't throw any error
– RushHour
Nov 13 '18 at 18:20
my first comment? can you find output files?
– thebluephantom
Nov 13 '18 at 18:23
I do not think there is an error
– thebluephantom
Nov 13 '18 at 18:25
|
show 4 more comments
Not sure I understand the regex. Should it not be based on value of first col, 1 or 2?
– thebluephantom
Nov 13 '18 at 18:01
looks ok at first glance
– thebluephantom
Nov 13 '18 at 18:19
yeah it doesn't throw any error
– RushHour
Nov 13 '18 at 18:20
my first comment? can you find output files?
– thebluephantom
Nov 13 '18 at 18:23
I do not think there is an error
– thebluephantom
Nov 13 '18 at 18:25
Not sure I understand the regex. Should it not be based on value of first col, 1 or 2?
– thebluephantom
Nov 13 '18 at 18:01
Not sure I understand the regex. Should it not be based on value of first col, 1 or 2?
– thebluephantom
Nov 13 '18 at 18:01
looks ok at first glance
– thebluephantom
Nov 13 '18 at 18:19
looks ok at first glance
– thebluephantom
Nov 13 '18 at 18:19
yeah it doesn't throw any error
– RushHour
Nov 13 '18 at 18:20
yeah it doesn't throw any error
– RushHour
Nov 13 '18 at 18:20
my first comment? can you find output files?
– thebluephantom
Nov 13 '18 at 18:23
my first comment? can you find output files?
– thebluephantom
Nov 13 '18 at 18:23
I do not think there is an error
– thebluephantom
Nov 13 '18 at 18:25
I do not think there is an error
– thebluephantom
Nov 13 '18 at 18:25
|
show 4 more comments
1 Answer
1
active
oldest
votes
0
The problem is typos and lack of formatting and spaces and l instead of 1. I managed to fix these and it ran, I altered your regex, you can tailor that, but most of it is an accuracy issue. Use the file as follows and it works, your own HDFS and settings of course:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = std1 std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.i1.regex = ^.*(1|2)
agent.sources.tailsrc.interceptors.i1.serializers = t1
agent.sources.tailsrc.interceptors.i1.serializers.t1.name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.std1.channel = mem1
agent.sinks.std1.batchSize = 1
agent.sinks.std1.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumemaleemployee
agent.sinks.std1.rolllnterval = 0
agent.sinks.std1.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
|
show 5 more comments
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53285112%2fwriting-data-in-hive-warehouse-directory-in-two-separate-tables-using-flume%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
The problem is typos and lack of formatting and spaces and l instead of 1. I managed to fix these and it ran, I altered your regex, you can tailor that, but most of it is an accuracy issue. Use the file as follows and it works, your own HDFS and settings of course:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = std1 std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.i1.regex = ^.*(1|2)
agent.sources.tailsrc.interceptors.i1.serializers = t1
agent.sources.tailsrc.interceptors.i1.serializers.t1.name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.std1.channel = mem1
agent.sinks.std1.batchSize = 1
agent.sinks.std1.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumemaleemployee
agent.sinks.std1.rolllnterval = 0
agent.sinks.std1.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
|
show 5 more comments
0
The problem is typos and lack of formatting and spaces and l instead of 1. I managed to fix these and it ran, I altered your regex, you can tailor that, but most of it is an accuracy issue. Use the file as follows and it works, your own HDFS and settings of course:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = std1 std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.i1.regex = ^.*(1|2)
agent.sources.tailsrc.interceptors.i1.serializers = t1
agent.sources.tailsrc.interceptors.i1.serializers.t1.name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.std1.channel = mem1
agent.sinks.std1.batchSize = 1
agent.sinks.std1.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumemaleemployee
agent.sinks.std1.rolllnterval = 0
agent.sinks.std1.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
|
show 5 more comments
0
0
0
The problem is typos and lack of formatting and spaces and l instead of 1. I managed to fix these and it ran, I altered your regex, you can tailor that, but most of it is an accuracy issue. Use the file as follows and it works, your own HDFS and settings of course:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = std1 std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.i1.regex = ^.*(1|2)
agent.sources.tailsrc.interceptors.i1.serializers = t1
agent.sources.tailsrc.interceptors.i1.serializers.t1.name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.std1.channel = mem1
agent.sinks.std1.batchSize = 1
agent.sinks.std1.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumemaleemployee
agent.sinks.std1.rolllnterval = 0
agent.sinks.std1.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
The problem is typos and lack of formatting and spaces and l instead of 1. I managed to fix these and it ran, I altered your regex, you can tailor that, but most of it is an accuracy issue. Use the file as follows and it works, your own HDFS and settings of course:
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = std1 std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/cloudera/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.i1.regex = ^.*(1|2)
agent.sources.tailsrc.interceptors.i1.serializers = t1
agent.sources.tailsrc.interceptors.i1.serializers.t1.name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = hdfs
agent.sinks.std1.channel = mem1
agent.sinks.std1.batchSize = 1
agent.sinks.std1.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumemaleemployee
agent.sinks.std1.rolllnterval = 0
agent.sinks.std1.hdfs.fileType = DataStream
agent.sinks.std2.type = hdfs
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.hdfs.path = hdfs://quickstart.cloudera:8020/user/hive/warehouse/flumefemaleemployee
agent.sinks.std2.rolllnterval = 0
agent.sinks.std2.hdfs.fileType = DataStream
agent.channels.mem1.type = memory
agent.channels.meml.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
agent.sources.tailsrc.channels = mem1 mem2
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
edited Nov 14 '18 at 6:48
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
answered Nov 13 '18 at 22:53
thebluephantomthebluephantom
2,8633927
2,8633927
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
|
show 5 more comments
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
If I follow documentation and If I use only mandatory properties for source. Like for "exec" the only mandatory property is type and command. So will it work with these two properties also ?? Coz I haven't understood the purpose of other properties over here
– RushHour
Nov 14 '18 at 7:17
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
Yeah it worked but it would be very helpful if you can explain me those properties in layman terms specially the regex part so that in future I can tackle such problems
– RushHour
Nov 14 '18 at 7:48
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
It ran and for me it’s over. Work to do
– thebluephantom
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Okay whenever you get time will you help me with understanding these properties ?
– RushHour
Nov 14 '18 at 7:49
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
Ok will do but the example is reading from a file that gets the stuff and from start of line looks for 1 or 2 ans splits to 2 files. I will post you a link to read on regex
– thebluephantom
Nov 14 '18 at 8:14
|
show 5 more comments
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53285112%2fwriting-data-in-hive-warehouse-directory-in-two-separate-tables-using-flume%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Not sure I understand the regex. Should it not be based on value of first col, 1 or 2?
– thebluephantom
Nov 13 '18 at 18:01
looks ok at first glance
– thebluephantom
Nov 13 '18 at 18:19
yeah it doesn't throw any error
– RushHour
Nov 13 '18 at 18:20
my first comment? can you find output files?
– thebluephantom
Nov 13 '18 at 18:23
I do not think there is an error
– thebluephantom
Nov 13 '18 at 18:25