logging of important events within the system

[ralidousti]

While changing the log level from the default info to debug, it looks like the state changes/detections (crit, warn, ...) are not being logged. Tt makes sense to have some logging indicative of important events within the system.

[AlexGrs]

Indeed, event like join results etc should be visible in logs. It would help a lot when trying to create alerts.

[nathanielc]

@ralidousti @AlexGrs Can you give me an example of the kind of logs you would like to see? And how you would like to control turning them on and off?

From my perspective logging things like every state change/detection in an AlertNode or the results of every JoinNode would be too noisy and not helpful.

One thought I had to make debugging tasks easier, would be to add a LogNode that you could put anywhere in your task and then it would log all the data that it sees. This way you can easily control what is logged and remove it for production running tasks.

For example:

var data0 = stream
   .from()....

var data1 = stream
   .from()....

// Log all data in stream data0
data0.log()
// Log all data in stream data1
data1.log()

data0.join(data1)...
      // Log all joined data
      .log()

Thoughts?

[ralidousti]

What you are suggesting is prudent. Basically there needs to be a way to tell whether a given condition holds within the system through the application logging. But in case of crit/warn conditions I'd still think that it needs to be logged without any prior/explicit tick statement, since they are critical/warning conditions for the user anyway. I think a join/union/... is different and an explicit tick statement makes more sense in their case.

[AlexGrs]

I agree. For some cases such as expression evaluation for crit and warn, a direct logging without specifying it in tick would be really appreciated (only if kapacitor is set in debug mode).

For union & join, a LogNode is great for me, from the moment I can access to the internal data of the node to understand what is happening with my joins.

[nathanielc]

Sounds like we are on the same page. I'll get something up soon to improve logging.