Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs API to be user-facing #4225

Closed
wants to merge 24 commits into from
Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ release.
([#4203](https://github.com/open-telemetry/opentelemetry-specification/pull/4203))
- Make all fields as identifying for Logger. Previously attributes were omitted from being identifying.
([#4161](https://github.com/open-telemetry/opentelemetry-specification/pull/4161))
- Remove Events API and SDK. Rename Logs Bridge API to Logs API and do not discourage from direct usage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge Events API and SDK

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to simply bring back the Events API and SDK. We can address it later as a separate PR.

([#4225](https://github.com/open-telemetry/opentelemetry-specification/pull/4225))

### Events

Expand Down
20 changes: 0 additions & 20 deletions spec-compliance-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,26 +203,6 @@ Disclaimer: this list of features is still a work in progress, please refer to t
| Can plug custom LogRecordExporter | | | + | | + | | | + | | + | | |
| Trace Context Injection | | | + | | + | | | + | | + | + | |

## Events

Features for the [Events API](specification/logs/event-api.md) and the [Events SDK](specification/logs/event-sdk.md).
Disclaimer: Events are currently in Development status - work in progress.

| Feature | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table is (was) incomplete, several languages already have this capability and users of this API.

JS, Java (Android) and recently Python

|----------------------------------------------------------------------------|----------|----|------|----|--------|------|--------|-----|------|-----|------|-------|
| [EventLoggerProvider](specification/logs/event-api.md#eventloggerprovider) | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
| Get EventLogger | | | | | | | | | | | | |
| Get EventLogger accepts version | X | | | | | | | | | | | |
| Get EventLogger accepts schema_url | X | | | | | | | | | | | |
| Get EventLogger accepts attributes | X | | | | | | | | | | | |
| [EventLogger](specification/logs/event-api.md#eventlogger) | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
| Emit event accepts name | | | | | | | | | | | | |
| Emit event accepts AnyValue body | X | | | | | | | | | | | |
| Emit event accepts severity | X | | | | | | | | | | | |
| Emit event accepts timestamp | X | | | | | | | | | | | |
| Emit event accepts attributes | X | | | | | | | | | | | |
| Emit event accepts context | X | | | | | | | | | | | |

## Resource

| Feature | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
Expand Down
4 changes: 1 addition & 3 deletions specification/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ path_base_for_github_subdir:
- [Baggage](baggage/api.md)
- [Tracing](trace/api.md)
- [Metrics](metrics/api.md)
- [Logs](logs/README.md)
- [Bridge API](logs/bridge-api.md)
- [Event API](logs/event-api.md)
- [Logs](logs/api.md)
- SDK Specification
- [Tracing](trace/sdk.md)
- [Metrics](metrics/sdk.md)
Expand Down
4 changes: 2 additions & 2 deletions specification/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ concept the attributes of the scope cannot change at runtime.

This refers to the `name` and (optional) `version` arguments specified when
creating a new `Tracer` or `Meter` (see
[Obtaining a Tracer](trace/api.md#tracerprovider)/[Obtaining a Meter](metrics/api.md#meterprovider))/[Obtaining a Logger](logs/bridge-api.md#loggerprovider).
[Obtaining a Tracer](trace/api.md#tracerprovider)/[Obtaining a Meter](metrics/api.md#meterprovider))/[Obtaining a Logger](logs/api.md#loggerprovider).
The name/version pair identifies the
[Instrumentation Scope](#instrumentation-scope), for example the
[Instrumentation Library](#instrumentation-library) or another unit of
Expand Down Expand Up @@ -250,7 +250,7 @@ recommended to call it out specifically.
### Log Appender / Bridge

A log appender or bridge is a component which bridges logs from an existing log
API into OpenTelemetry using the [Log Bridge API](./logs/bridge-api.md). The
API into OpenTelemetry using the [Logs (Bridge) API](./logs/api.md). The
terms "log bridge" and "log appender" are used interchangeably, reflecting that
these components bridge data into OpenTelemetry, but are often called appenders
in the logging domain.
34 changes: 17 additions & 17 deletions specification/logs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,16 +140,18 @@ Given the above state of the logging space we took the following approach:
OpenTelemetry log data model. OpenTelemetry Collector can read such logs and
translate them to OpenTelemetry log data model.

- OpenTelemetry defines a Logs Bridge API
for [emitting LogRecords](./bridge-api.md#emit-a-logrecord). Application developers are
NOT encouraged to call this API directly. It is provided for library authors
to build [log appender](../glossary.md#log-appender--bridge), which use
the API to bridge between existing logging libraries and the OpenTelemetry log
data model. Existing logging libraries generally provide a much richer set of
features than what is defined in OpenTelemetry. It is NOT a goal of
OpenTelemetry to ship a feature-rich logging library.

- OpenTelemetry defines an [SDK](./sdk.md) implementation of the [Bridge API](./bridge-api.md),
- OpenTelemetry defines a Logs API for [emitting LogRecords](./api.md#emit-a-logrecord).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This NEEDS to include comments about also emitting events as that is the intention

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is described in API and Data Model. Do we need to repeat it? Can you suggest a change?

It is provided for library authors to build [log appenders](../glossary.md#log-appender--bridge),
which use the API to bridge between existing logging libraries and the
OpenTelemetry log data model.
The [API](./api.md) is also provided for instrumentation libraries to emit log
records following semantic conventions (also known as OpenTelemetry Events).
pellared marked this conversation as resolved.
Show resolved Hide resolved
Existing logging libraries generally provide a much richer set of features
than what is defined in OpenTelemetry. It is NOT a goal of OpenTelemetry to
ship a feature-rich logging library. However, nothing prevents application
cijothomas marked this conversation as resolved.
Show resolved Hide resolved
developers to use it directly.

- OpenTelemetry defines an [SDK](./sdk.md) implementation of the [API](./api.md),
which enables configuration of [processing](./sdk.md#logrecordprocessor)
and [exporting](./sdk.md#logrecordexporter) LogRecords.

Expand Down Expand Up @@ -230,10 +232,9 @@ processor.
### Infrastructure Logs

These are logs generated by various infrastructure components, such as
Kubernetes events (if you are wondering why events are discussed in the context
of logs see [Event API Data model](./event-api.md#event-data-model)). Like system logs, the
infrastructure logs lack a trace context and can be enriched by the resource
context - information about the node, pod, container, etc.
Kubernetes events. Like system logs, the infrastructure logs lack a trace
context and can be enriched by the resource context - information about
the node, pod, container, etc.

OpenTelemetry Collector or other agents can be used to query logs from most
common infrastructure controllers.
Expand Down Expand Up @@ -359,7 +360,7 @@ as parsers, log tailing and rotation. It also enables the possibility to send
logs directly to the logging backend without using a log collection agent.

To facilitate both approaches described above OpenTelemetry provides
a [Bridge API](./bridge-api.md) and [SDK](./sdk.md), which can be used together with existing
a [API](./api.md) and [SDK](./sdk.md), which can be used together with existing
logging libraries to automatically inject the trace context in the emitted logs,
and provide an easy way to send the logs via OTLP. Instead of
modifying each logging statement, [log appenders](../glossary.md#log-appender--bridge)
Expand Down Expand Up @@ -444,10 +445,9 @@ standard output.

## Specifications

* [Logs Bridge API](./bridge-api.md)
* [Logs API](./api.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about having links to the old API's stating that the this new API will encompass the existing functionality.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not want to create yet another API.

* [Logs SDK](./sdk.md)
* [Logs Data Model](./data-model.md)
* [Event API](./event-api.md)
* [Trace Context in non-OTLP Log Formats](../compatibility/logging_trace_context.md)

## References
Expand Down
40 changes: 22 additions & 18 deletions specification/logs/bridge-api.md → specification/logs/api.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Logs Bridge API
# Logs API

**Status**: [Stable](../document-status.md), except where otherwise specified

Expand All @@ -9,6 +9,7 @@

<!-- toc -->

- [Overview](#overview)
- [LoggerProvider](#loggerprovider)
* [LoggerProvider operations](#loggerprovider-operations)
+ [Get a Logger](#get-a-logger)
Expand All @@ -18,21 +19,33 @@
+ [Enabled](#enabled)
- [Optional and required parameters](#optional-and-required-parameters)
- [Concurrency requirements](#concurrency-requirements)
- [Artifact Naming](#artifact-naming)
- [References](#references)

<!-- tocstop -->

</details>

<b>Note: this document defines a log *backend* API. The API is not intended
to be called by application developers directly. It is provided for logging
library authors to build
[log appenders](./supplementary-guidelines.md#how-to-create-a-log4j-log-appender),
## Overview

This document defines a log API.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As part of OTep #265 it was determined that there SHOULD be a public facing "Logging" API to enable the emitting of "Events" (as a type of OpenTelemetry Log) as well as other general "Logs" <- I'm implicitly inferring based on discussions that occurred as part of #265 (ie. while all events can be represented as Logs, not all Logs are events)

One of the key words here is type, there are many types of Logs and each type fundamentally defines the "intent" (or the "why") the log is being emitted.

These can include "Event", "Audit", "Security", "Transaction", "Message", "Server", "File" and many, many more, and just as "Events are a type of Log", several of these can also be sub-classified as a type of event.

As part of any Logging API we SHOULD provide a front-end API and guidance for all of the CORE fundamental concepts to "try" to make it clear about what will and won't occur when you do or don't use the specific API.
(Right NOW we don't have this clearly defined (because we are always trying to take small steps) which always lands us into the argument of "But it's just a Log", which is not a productive argument as it ignores the requirements by different teams / use cases that there are different needs and different types of Logs that should have some uniform structure (Semantic Conventions / processing) around them.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the PR so that Logs API can be used to emit events but I have not removed the Events API.

The API serves following use cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many additional features / requirements that exist and are yet to be defined as we move this API forward (some of which was starting to occur in the Event API) as it was moving (or trying to move) towards stability.
Some of these will include different processing requirements for the different types and as it moves towards a stabilized state it may cause implementation changes, so for now (as a small step) I'll try to define a starting point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should rather require refining the SDK and not the API.

It is provided for instrumentation libraries to emit events (log records following
OpenTelemetry Semantic Conventions).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should include an emitEvent function which MUST include the name as a first-class concept as the specific value of the name defines the Semantic-Definition of any provided body for the specific event.

As the name is a required argument / property for this function it MUST also enforce that the name conforms to the OpenTelemetry general semantic conventions (ie. it is fully qualified -- just like an Attribute name (this doesn't mean a complete list of valid "registered" events), e.g. it just needs to have a leading domain and then name domain.eventName where the domain may have multiple separators.

And it should also

  • Be used only to emit a well-structured, well defined event as per the definitions from an OpenTelemetry and application perspective.
  • These events will include the fundamental log types like "Application", "User", "Client", "Device", "Security".
  • Which "It signals that "this thing happened at this time" and provides additional specifics about the occurrence"
  • Should be used by Applications (Adopters) and Instrumentations only.
  • Because the exact structure and semantic meaning of the body of the event is driven and defined by the name, this allows for SDK's (or applications) to extend and provide (if required)
    • explicit validation of the content
    • explicit sampling / rules governing that specific "instance"
    • explicit delivery mechanism (Send NOW, Don't Drop, Create Metric, as some possible examples)
    • The exact location of the "where" and "when" of any rules is up to the specific application / implementation (eg. it "MAY" occur on the client, in the SDK, at the aggregation point (Collector) or at rest and queried as required.

And (if) future versions of OpenTelemetry evolve to change the internal representation of the event as a LogRecord all of the "changes" can be limited to the SDK implementation/version of the API, so that all existing users of the API have zero change when it's just an internal "representation" within the LogRecord (not-withstanding that this would have downstream implementations)

It is provided for logging library authors to build
[log appenders/bridges](./supplementary-guidelines.md#how-to-create-a-log4j-log-appender),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • emitBridgeLog (just making the name different from above -- needs a better one), just here to have the distinction explicit.
    • Exists to "bridge" into the OpenTelemetry Logs pipeline only
    • Used by log Appenders ONLY to create OpenTelemetry LogRecords and have them processed
    • NO special processing (beyond existing Log Processors ??)*
      • (*)There may be some special use cases where some Log Processors may be required to be skipped (like a processor that links OTel Logs -> Native Logging)
      • Should we change this and have "explicit" Appender/Bridge Log Processors
    • IF an existing language / logging library supports the creation of "Events" or specific "Structured" Logs then this API MAY be used to "Simulate" (generate) a LogRecord that looks like those created by the above API's.
      • HOWEVER, this WILL NOT re-apply / perform any special processing that may be configured by the SDK instance for the other API methods
      • So it's entirely 100% up to the appender (caller of this API) to ensure that it is complying with any necessary semantic conventions to avoid any downstream complications around deterministically identifying / processing the resulting LogRecord.
      • It also MUST NOT just convert everything into an Event UNLESS
        • it can EXPLICITLY identify that the intent of the LogRecord was to identify (signal) that the specifically named event occurred vs general Detailed Logging
        • or it create a general semantic convention that identifies that this LogRecord that looks like an event was created by or follows the semantic conventions of X. ie. it has a specific name like log.*, applog.* or event.ilogger again so the intent (type) can be deterministically evaluated AND it would avoid any "clashes" with defined semantic conventions.
        • To phrase another way as long as the Appender "Knows" that the user emitting the LogRecord WANTS this specific log to be called XXXXX then this is 100% acceptable as long as the control (setting) of this name is explicitly defined by the user. eg. Application has their own logger for their own events, but they want to explicitly emit an OpenTelemetry defined event with a Specific name via their own logger and not via the OpenTelemetry emitEvent with the understanding (implicit or otherwise) that any configured OpenTelemetry "Event" processing will NOT be performed via this path.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MSNev can you highlight any specific blocking concerns you have about this PR beyond #4225 (comment), and open tracking issues for the rest? Once we get this initial restructuring PR landed then we'll be able to start sending PRs to try to address those. Thanks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is up to the bridge implementation how the logging library would be mapped to OTel Logs Data Model. I think we should rather create some "guidance" / "recommendations" for log bridge authors. We cannot control how the bridges are going to be implemented.

For sure this should be out of scope of this PR.

which use this API to bridge between existing logging libraries and the
OpenTelemetry log data model.</b>
OpenTelemetry log data model.

The Logs Bridge API consist of these main components:
It is provided for application developers to emit structured log records
(including OpenTelemetry Events).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should call out the "difference" between the processing / handling / semantics of an event vs a structured log

  • emitLog
    • Used to emit a structured Log (without (necessarily) any implicit meaning
    • These logs will include type "Server", "Transaction", "Message", "Audit", "Debug", "Diagnostic" etc.
    • To be used to provide "Detailed Information" which may be used to understand operational issues, often used for post mortem analysis rather than operational dashboards or monitoring.
      • Should be used by Applications, Instrumentations and SDK implementations.
    • If there is any explicit event processing, this API won't perform any explicit processing beyond general common Log Processors

Problem Statement and confusion

  • As "structured" logs generally also need some form of "identifier" about the "type", so these will also need some form of name, schema or identifying attributes as these (vs general unstructured logging) generally have some semantic conventions on their structure and beyond just have "OTLP" AnyValues processors will likely want a simple mechanism for identifying them.
  • The MUST however, NOT be identified in the same manner as events, as then there is no deterministic mechanism for identifying "How" the type of LogRecord was produced (as an Event or as a Detailed Information record).
  • As such should this include or be restricted to have a general (optional) name
  • If a Language / runtime has a standard logging library with a supported log appender then there will be confusion between using this API vs their specific (preferred) logging library.
    • The difference should be isolated around configured specific Log Processing (if any) -- which of course doesn't exist today.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should call out the "difference" between the processing / handling / semantics of an event vs a structured log

I find it as out of scope. We need such mechanism for all log records. From last SIG meeting agenda:

We need filtering and separate log processing pipelines. E.g. distinct processing for different exporters or different processing for events and other log records. Also needed for Specify how Logs SDK implements Enabled #4207

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I stating anywhere that we don't -- NO.
But I am stating that we ALSO need this for Events (The Event API has this for that exact purpose), we could have "just" ignored it and said hey go just use a genera Log processor on your Log definition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on the current version of spec, it's totally valid to emit events using log bridge. We don't have a requirement to support separate pipelines today and this PR does not change status quo

If a logging library is capable of creating logs which correctly map
to the Event data model, logging in this manner is also an acceptable way to
create Events.


Languages MAY provide a separate Logs Bridge API if they need different
ergonomics for consumers that are building log appenders/bridges.

The Logs API consists of these main components:

* [LoggerProvider](#loggerprovider) is the entry point of the API. It provides access to `Logger`s.
* [Logger](#logger) is responsible for emitting logs as
Expand Down Expand Up @@ -167,22 +180,13 @@ provide it.

## Concurrency requirements

For languages which support concurrent execution the Logs Bridge APIs provide
For languages which support concurrent execution the Logs APIs provide
specific guarantees and safeties.

**LoggerProvider** - all methods are safe to be called concurrently.

**Logger** - all methods are safe to be called concurrently.

## Artifact Naming

The Logs Bridge API is not intended to be called by application developers
directly, and SHOULD include documentation that discourages direct use. However,
in the event OpenTelemetry were to add a user facing API, the Logs Bridge API would
be a natural starting point. Therefore, Log Bridge API artifact, package, and class
names MUST NOT include the terms "bridge", "appender", or any other qualifier
that would prevent evolution into a user facing API.

## References

- [OTEP0150 Logging Library SDK Prototype Specification](https://github.com/open-telemetry/oteps/blob/main/text/logs/0150-logging-library-sdk.md)
162 changes: 0 additions & 162 deletions specification/logs/event-api.md

This file was deleted.

Loading
Loading