Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FastISOTimestampFormatter as a fast alternative to Java's DateTimeFormatter for standard ISO formats #711

Merged
merged 7 commits into from
Dec 20, 2021
Merged

FastISOTimestampFormatter as a fast alternative to Java's DateTimeFormatter for standard ISO formats #711

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3dc67c7
Introduce FastISOTimestampFormatter: a fast alternative to Java's Dat…
brenuart Nov 15, 2021
0a88f75
Update testcase to use a standard format not supported by the FastISO…
brenuart Dec 8, 2021
8903316
Mostly cosmetic changes after review
brenuart Dec 11, 2021
79ac562
Cosmetic changes as per the review
brenuart Dec 12, 2021
532e098
Add some words about the fasted timestamp formats in README.md
brenuart Dec 15, 2021
6e80a13
Remove obsolete test case
brenuart Dec 16, 2021
c65e2fc
Fix typo in comments
brenuart Dec 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1643,8 +1643,16 @@ The value of the `timestampPattern` can be any of the following:
* `[` _`constant`_ `]` - (e.g. `[ISO_OFFSET_DATE_TIME]`) timestamp written using the given `DateTimeFormatter` constant
* any other value - (e.g. `yyyy-MM-dd'T'HH:mm:ss.SSS`) timestamp written using a `DateTimeFormatter` created from the given pattern

The provider uses a standard Java DateTimeFormatter under the hood. However, special optimisations are applied when using one of the following standard ISO formats that make it nearly 7x faster:

You can change the timezone like this:
* `[ISO_OFFSET_DATE_TIME]`
* `[ISO_ZONED_DATE_TIME`]
* `[ISO_LOCAL_DATE_TIME`]
* `[ISO_DATE_TIME`]
* `[ISO_INSTANT`]


The formatter uses the default TimeZone of the host Java platform by default. You can change it like this:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
Expand All @@ -1657,6 +1665,7 @@ For example `America/Los_Angeles`, `GMT+10` or `UTC`.
Use the special value `[DEFAULT]` to use the default TimeZone of the system.



## Customizing LoggingEvent Message

By default, LoggingEvent messages are written as JSON strings. Any characters not allowed in a JSON string, such as newlines, are escaped.
Expand Down
151 changes: 96 additions & 55 deletions src/main/java/net/logstash/logback/composite/AbstractFormattedTimestampJsonProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,11 @@
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.Objects;
import java.util.TimeZone;
import java.util.function.Function;

import net.logstash.logback.fieldnames.LogstashCommonFieldNames;
import net.logstash.logback.util.TimeZoneUtils;
Expand Down Expand Up @@ -86,43 +89,21 @@ public abstract class AbstractFormattedTimestampJsonProvider<Event extends Defer
* Writes the timestamp to the JsonGenerator.
*/
private TimestampWriter timestampWriter;

/**
* Writes the timestamp to the JsonGenerator
*/
private interface TimestampWriter {
protected interface TimestampWriter {
void writeTo(JsonGenerator generator, String fieldName, long timestampInMillis) throws IOException;

String getTimestampAsString(long timestampInMillis);
}

/**
* Writes the timestamp to the JsonGenerator as a string formatted by the pattern.
*/
private static class PatternTimestampWriter implements TimestampWriter {

private final DateTimeFormatter formatter;

PatternTimestampWriter(DateTimeFormatter formatter) {
this.formatter = formatter;
}


@Override
public void writeTo(JsonGenerator generator, String fieldName, long timestampInMillis) throws IOException {
JsonWritingUtils.writeStringField(generator, fieldName, getTimestampAsString(timestampInMillis));
}

@Override
public String getTimestampAsString(long timestampInMillis) {
return formatter.format(Instant.ofEpochMilli(timestampInMillis));
}
}

/**
* Writes the timestamp to the JsonGenerator as a number of milliseconds since unix epoch.
*/
private static class NumberTimestampWriter implements TimestampWriter {
protected static class NumberTimestampWriter implements TimestampWriter {

@Override
public void writeTo(JsonGenerator generator, String fieldName, long timestampInMillis) throws IOException {
Expand All @@ -136,22 +117,38 @@ public String getTimestampAsString(long timestampInMillis) {
}

/**
* Writes the timestamp to the JsonGenerator as a string representation of the of milliseconds since unix epoch.
* Writes the timestamp to the JsonGenerator as a string, converting the timestamp millis into a
* String using the supplied Function.
*/
private static class StringTimestampWriter implements TimestampWriter {

protected static class StringFormatterWriter implements TimestampWriter {
private final Function<Long, String> provider;

StringFormatterWriter(Function<Long, String> provider) {
this.provider = Objects.requireNonNull(provider);
}

@Override
public void writeTo(JsonGenerator generator, String fieldName, long timestampInMillis) throws IOException {
JsonWritingUtils.writeStringField(generator, fieldName, getTimestampAsString(timestampInMillis));
}

@Override
public String getTimestampAsString(long timestampInMillis) {
return Long.toString(timestampInMillis);
return provider.apply(timestampInMillis);
}

static StringFormatterWriter with(DateTimeFormatter formatter) {
return new StringFormatterWriter(tstamp -> formatter.format(Instant.ofEpochMilli(tstamp)));
}
static StringFormatterWriter with(FastISOTimestampFormatter formatter) {
return new StringFormatterWriter(formatter::format);
}
static StringFormatterWriter with(Function<Long, String> formatter) {
return new StringFormatterWriter(formatter);
}

}



public AbstractFormattedTimestampJsonProvider() {
setFieldName(FIELD_TIMESTAMP);
updateTimestampWriter();
Expand All @@ -177,34 +174,78 @@ protected String getFormattedTimestamp(Event event) {
* Updates the {@link #timestampWriter} value based on the current pattern and timeZone.
*/
private void updateTimestampWriter() {
timestampWriter = createTimestampWriter();
}

private TimestampWriter createTimestampWriter() {
if (UNIX_TIMESTAMP_AS_NUMBER.equals(pattern)) {
timestampWriter = new NumberTimestampWriter();
} else if (UNIX_TIMESTAMP_AS_STRING.equals(pattern)) {
timestampWriter = new StringTimestampWriter();
} else if (pattern.startsWith("[") && pattern.endsWith("]")) {
String constant = pattern.substring("[".length(), pattern.length() - "]".length());
try {
Field field = DateTimeFormatter.class.getField(constant);
if (Modifier.isStatic(field.getModifiers())
&& Modifier.isFinal(field.getModifiers())
&& field.getType().equals(DateTimeFormatter.class)) {
try {
DateTimeFormatter formatter = (DateTimeFormatter) field.get(null);
timestampWriter = new PatternTimestampWriter(formatter.withZone(timeZone.toZoneId()));
} catch (IllegalAccessException e) {
throw new IllegalArgumentException(String.format("Unable to get value of constant named %s in %s", constant, DateTimeFormatter.class), e);
}
} else {
throw new IllegalArgumentException(String.format("Field named %s in %s is not a constant %s", constant, DateTimeFormatter.class, DateTimeFormatter.class));
}
} catch (NoSuchFieldException e) {
throw new IllegalArgumentException(String.format("No constant named %s found in %s", constant, DateTimeFormatter.class), e);
return new NumberTimestampWriter();
}

if (UNIX_TIMESTAMP_AS_STRING.equals(pattern)) {
return StringFormatterWriter.with(tstamp -> Long.toString(tstamp));
}

if (pattern.startsWith("[") && pattern.endsWith("]")) {
// Get the standard formatter by name...
//
String constant = pattern.substring(1, pattern.length() - 1);

// Use our fast FastISOTimestampFormatter if suitable...
//
ZoneId zone = timeZone.toZoneId();
if ("ISO_OFFSET_DATE_TIME".equals(constant)) {
return StringFormatterWriter.with(FastISOTimestampFormatter.isoOffsetDateTime(zone));
}
} else {
timestampWriter = new PatternTimestampWriter(DateTimeFormatter.ofPattern(pattern).withZone(timeZone.toZoneId()));
if ("ISO_ZONED_DATE_TIME".equals(constant)) {
return StringFormatterWriter.with(FastISOTimestampFormatter.isoZonedDateTime(zone));
}
if ("ISO_LOCAL_DATE_TIME".equals(constant)) {
return StringFormatterWriter.with(FastISOTimestampFormatter.isoLocalDateTime(zone));
}
if ("ISO_DATE_TIME".equals(constant)) {
return StringFormatterWriter.with(FastISOTimestampFormatter.isoDateTime(zone));
}
if ("ISO_INSTANT".equals(constant)) {
return StringFormatterWriter.with(FastISOTimestampFormatter.isoInstant(zone));
}


// Otherwise try one of the default formatters...
//
DateTimeFormatter formatter = getStandardDateTimeFormatter(constant).withZone(zone);
return StringFormatterWriter.with(formatter);
}


// Construct using a pattern
//
DateTimeFormatter formatter = DateTimeFormatter.ofPattern(pattern).withZone(timeZone.toZoneId());
return StringFormatterWriter.with(formatter);
}



private DateTimeFormatter getStandardDateTimeFormatter(String name) {
try {
Field field = DateTimeFormatter.class.getField(name);
if (Modifier.isStatic(field.getModifiers())
&& Modifier.isFinal(field.getModifiers())
&& field.getType().equals(DateTimeFormatter.class)) {
return (DateTimeFormatter) field.get(null);
}
else {
throw new IllegalArgumentException(String.format("Field named %s in %s is not a constant %s", name, DateTimeFormatter.class, DateTimeFormatter.class));
}
}
catch (IllegalAccessException e) {
throw new IllegalArgumentException(String.format("Unable to get value of constant named %s in %s", name, DateTimeFormatter.class), e);
}
catch (NoSuchFieldException e) {
throw new IllegalArgumentException(String.format("No constant named %s found in %s", name, DateTimeFormatter.class), e);
}
}


public String getPattern() {
return pattern;
}
Expand Down
Loading