Java Consultant Tip #2: Use OpenTelemetry Java Agent to learn a new application
As Java consultants, when we land on a new gig with an unknown application, we need to come up to speed as quickly as possible. Going through codes is inefficient and can easily miss an important interaction. In my experience, one of the best ways to learn a new Java application is to add automatic instrumentation using the OpenTelemetry’s Java Agent. This article will show up step-by-step how to use it.
Getting Started with OpenTelemetry Java Agent
Downloading Java Agent
First, we need to download the latest version of the Java Agent from GitHub: https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases
Startup Parameters
Once we have the JAR file downloaded, all we need to do is to add it as a java agent into your Java application’s startup parameter, e.g.
java -javaagent:path/to/opentelemetry-javaagent.jar \
-jar myapp.jarIf you are using Maven and Spring Boot plugin, you can achieve the above using a couple of plugins, e.g.
<!-- First, use the dependency plugin to download and unpack the Java Agent into the build directory -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>copy</goal>
</goals>
<phase>process-resources</phase>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.splunk</groupId>
<artifactId>splunk-otel-javaagent</artifactId>
<version>1.6.0</version>
<type>jar</type>
<outputDirectory>${project.build.directory}</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin><!-- Add the Java agent into the Spring Boot plugin's program argument -->
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>com.example.Application</mainClass>
<agents>
<agent>${project.build.directory}/splunk-otel-javaagent-1.6.0.jar</agent>
</agents>
<systemProperties>
....
Tip: Grouping the above plugins into a Maven profile would let you conditionally turn on/off tracing
Additional System Properties
The OpenTelemetry Java Agent is configured using System properties. Most of them have reasonable defaults. Some of the important ones are:
otel.resource.attributes This is used to set some internal attributes, e.g. deployment.environment and service.name . For example, otel.resource.attributes=deployment.environment=dev,service.name=MyApp
otel.traces.exporter This specifies the trace exporter. In this article, we use Jaeger. So, otel.traces.exporter=jaeger
otel.exporter.jaeger.endpoint This specifies the endpoint of your Jaeger installation. If you use a standard Jaeger installation, set otel.exporter.jaeger.endpoint=http://localhost:14250/api/traces
otel.traces.sampler This changes the frequency of emitting traces. Set it to always_on if your purpose is to learn about the application.
otel.instrumentation.common.peer-service-mapping This property allows you to map a remote service’s hostname to a service name. For example, if the Java application makes a REST call to service1.yourdomain.com and you want to call this service PdfGenerator , you can say: otel.instrumentation.common.peer-service-mapping=service1.yourdomain.com=PdfGenerator . Multiple service mapping can be separated by , . This property is extremely useful because it enables Jaeger to build an architectural diagram.
NOTE: Unfortunately, remote services with the same hostnames but different port numbers are mapped to the same service name.
You can find more configuration options in the following page: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/agent-config.md
Visualizing traces with Jaeger
Once we have included the Java agent in your startup parameter, our application will start sending traces into the endpoint we specified in the System properties.
Follow the excellent instructions from the Jaeger UI website to get it up and running: https://www.jaegertracing.io/docs/1.29/getting-started/.
After Jaeger is up and running, we can start our Java application with the OpenTelemetry Java Agent and run through a few use cases/scenarios.
Go to the Jaeger UI http://localhost:16686/search
The Java agent should automatically add traces to any REST endpoint invocation and any database statements sent to the database. (A complete list can be found here: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks)
In addition, if you have configured the otel.instrumentation.common.peer-service-mapping property, you should see a dependency diagram in the System Architecture tab. If you are missing anything in the diagram, simply add the service into the aforementioned property.
With these detailed traces and diagrams, we are much better off in understanding the application than just going through the code.
Screenshots
Here are a few screenshots to help you navigate the UI:
On the left-hand side, you can choose the filters for your traces. If you have configured the mapping property, your services should show up in the “Service” menu as soon as their traces are found.
On the right-hand side, you have the actual traces. Each trace is composed of one or more spans. A span is a traced operation inside a trace, e.g. a database call. For example, in the screenshot, the /dispatch endpoint involves 50 other traced operations.
Clicking on each trace exposes the individual spans and their relationships. Sometimes, a trace can span multiple services. In the above scenario, “frontend” calls into “customer” and “driver”.
Each span can further be expanded to see the details. For example, the SQL that was invoked.
Finally, the “System Architecture” tab shows us a dependency diagram between different services of your running application.
Security Manager
As a side note, OpenTelemetry’s Java Agent does not work with Java’s Security Manager. In this case, your only option is to add traces manually into the codebase.
Conclusion
Learning a new application can be a daunting task, especially if it interacts with multiple external services. Observing application behavior through traces and interaction diagram can greatly reduce the learning curve. The OpenTelemetry Java Agent can automatically instrument the codebase with the minimal development effort. The Jaeger UI can visualize the traces and build an interaction digram.
