Pfthb

I want to index a topic from kafka in avro to elasticsearch format but
I have problems with my timestamp field to be recognized by
elasticsearch as date format field.

I have used the following configuration for the connector.

 
 "name": "es-sink-barchart-10",
 "config": 
 "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
 "value.converter": "io.confluent.connect.avro.AvroConverter",
 "key.converter": "io.confluent.connect.avro.AvroConverter",
 "key.converter.schema.registry.url": "http://localhost:8081",
 "value.converter.schema.registry.url": "http://localhost:8081",

 "connection.url": "http://localhost:9200",

 "type.name":"type.name=kafka-connect",

 "topics": "exchange_avro_01",

 "topic.index.map": "exchange_avro_01:exchange_barchart",

 "key.ignore": "true"
 
 

The original field is bigint type and I want the target field to be date type with any valid format with elasticsearch. I have defined a dynamic template to try to solve it in the following way:

curl -XPUT "http://localhost:9200/_template/kafkaconnect/" -H 'Content-Type: application/json' -d'

 "index_patterns": "exchange*",
 "settings": 
 "number_of_shards": 1,
 "number_of_replicas": 0
 ,
 "mappings": 
 "kafka-connect": 
 "dynamic_templates": [
 
 "dates": 
 "match_mapping_type": "long",
 "match": "TIME",
 "mapping": 
 "type": "date",
 "format": "yyyy-MM-dd HH:mm:ss"
 
 
 
 ]
 ,
 "properties": 
 "CLOSE": 
 "type": "double"
 ,
 .
 .
 .
 
 

 
 
}'

When I load the connector described above nothing is indexed to elasticsearch.

Any help?

If your source is a bigint then presumably it's an epoch. If it's an epoch, then this won't work:

"mapping": 
 "type": "date",
 "format": "yyyy-MM-dd HH:mm:ss"
 

because you're telling Elasticsearch that the date format is yyyy-MM-dd HH:mm:ss (which it isn't).

So instead, try this (omitting your custom mapping for the moment; get this working first and then add that back in):

 "index_patterns": "exchange*",
 "settings": 
 "number_of_shards": 1,
 "number_of_replicas": 0
 ,
 "mappings": 
 "kafka-connect": 
 "dynamic_templates": [
 
 "dates": 
 "match": "TIME",
 "mapping": 
 "type": "date"
 ] 

Also ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html#date-detection

nothing is indexed to elasticsearch.

Check the Kafka Connect worker log and the Elasticsearch log for any errors.