Flow Activity Monitoring : Logging, Analyzing and Monitoring Data of our camel integration flows using ElasticSearch, Logstash and Kibana.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Many times we receive requests like: Where’s my message? Are you sure that my result has been routed to the good destination ? how many orders has been received from our partner? 


All monitoring solutions gives only part of answer. We have experienced the use of wireTap in our camel routes, but we have to design a module that consumes these messages and stock them into Database.


The Deal for us was : how we can design our flows without thinking to the monitoring ? how we can audit our integration flows already in production without any impact?


I think that the solution will be : Flow Activity Monitoring

1. Each camel flow deployed in JBoss Fuse are audited by Fuse BAI [REF-1], You have to develop a flow Backend BAI that consumes from endpoint vm:audit and log the content of the event using camel mdc logging. 

2. You have to setup a custom logging appender in JBoss Fuse in order to create a specific log for each camelContext deployed: 

e.g:

# Appender App 

log4j.appender.app = org.apache.log4j.sift.MDCSiftingAppender 

log4j.appender.app.key = camel.contextId 

log4j.appender.app.default = unknown 

log4j.appender.app.appender = org.apache.log4j.RollingFileAppender 

log4j.appender.app.appender.layout = org.apache.log4j.PatternLayout 

log4j.appender.app.appender.layout.ConversionPattern =% d {ISO8601} |%-5.5p |% X {camel.contextId} |% {X} camel.routeId |% X {camel.exchangeId} |% m% n 

log4j.appender.app.appender.file karaf.home = $ {} / log/apps/mediation- $ \ \ {camel.contextId \ \}. log 

log4j.appender.app.appender.append = true 

log4j.appender.app.appender.maxFileSize = 1MB 

log4j.appender.app.appender.maxBackupIndex = 10 


3. logstash [REF-2] listening to all logs in the follwing directory ${karaf.home}/log/apps/  ,parse and filter the message to header fields, Properties fields, Body, breadcrumb..etc, after we sent the output to ElasticSearch [REF-3].

4. Kibana for good graphic representation 😉 


Here’s the schema for the solution :

 

Kibana dashboard :

 

Conclusion:

– Your mission-critical projects need management and monitoring. Today, is just possible using open source products: logstash, Elasticsearch and Kibana.

– The future previous version of the JBoss Fuse 6.1 allow to send automatically all information of the message to Elasticsearch by enabling insight-camel, Though the downside is this assumes fabric and 6.1;


Best regards,

Abdellatif BOUCHAMA (@a_bouchama)

Linkedin : fr.linkedin.com/in/abouchama/

 

References

[REF-1] – https://github.com/jboss-fuse/fuse/tree/master/bai

[REF-2] – http://logstash.net/

[REF-3] – http://www.elasticsearch.org/

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Abdellatif Bouchama

Abdellatif's main area of expertise lies within the fields of SOA, ESB, BigData and IoT.

You may also like...

1 Response

  1. gnanagurus says:

    Crisp, A nice article with cool pictures. You have taken log4j to a new level :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>