The home page of my blog posts, articles, presentations and open source projects.
In this blog I will talk about two of my recent open source projects called log-parse-agent and log-parse-ui, the reason why I wrote that and how it can help perform grep and tail operations on remote logs.
We had various applications (distributed in nature) deployed across dozens of virtual machines (mostly Linux). For any new feature development or debugging of production issues we had to login (via SSH) to many of these machines and look at respective logs. It was very time consuming to login to all machines and then look at respective logs, and we wanted something like a centralized user interface where we could quickly search logs on all remote machines.
While there exist solutions like ELK, Splunk, etc. which provides many great features on log parsing, analysis, visualization and alerting, our requirement was simple to quickly identify patterns in remote logs for troubleshooting and also not have to setup big stacks like ELK, Splunk where we have to maintain central server and allocate enough storage to persist remote logs for analysis.
I wrote log-parse-agent project using Golang programming language which can be deployed in remote machines and configured to allow grep and tail operations on logs remotely via REST API endpoints.
The log-parse-ui is another project which can provide a centralized UI for all remote log-parse-agents to list their supported logs, perform grep and tail operations on them and view results in the UI itself.
For installation and usage details please refer to log-parse-agent and log-parse-ui projects.
If you have any feedback or suggestions please raise issue in log-parse-agent and log-parse-ui projects or email me at mchopker@gmail.com
Thank you!