Streamline Java Application Deployment: Pack, Ship, and Unlock Distributed Tracing with Elastic APM on Kubernetes

In my last article, I dug into Distributed Tracing and exposed how to enable it in Java applications. We didn’t see yet how to deploy an application on Kubernetes and get distributed tracing insights. Several strategies can be considered, but the main point is how to minimize the impact of deploying APM agents on the whole delivery process.

In this article, I will expose how to ship APM agents for instrumenting Java applications deployed on top of Kubernetes through Docker containers.

To make it clearer, I will illustrate this setup by the following use case:

• We have an API “My wonderful API” which is instrumented through an Elastic APM agent.
• The data is then sent to the Elastic APM.

Now, if we dive into the “Wonderful System”, we can see the Wonderful Java application and the agent:

We can basically implement this architecture in two different ways:

1. Deploying the agent in all of our Docker images
2. Deploying the agent asides from the Docker images and using initContainers to bring the agent at the startup of our applications

We will then see how to lose couple application docker images to the apm agent one.

1 Why not bringing APM agents in our Docker images?

It could be really tempting to put the APM agents in the application’s Docker image.

Why? Because you just have to add the following lines of code in our Docker images definition:

RUN mkdir /opt/agent
COPY ./javaagent.jar /opt/agent/javaagent.jar

Nonetheless, if you want to upgrade your agent, you will have to repackage it and redeploy all your Docker images.

For regular upgrades, it will not bother you, but, if you encounter a bug or a vulnerability, it will be tricky and annoying to do that.

What is why I prefer loose coupling the “business” applications Docker images to technical tools such as APM agents.

2 Deploy an APM agent through initContainers

While looking around how to achieve this, I came across to the Kubernetes initContainers.

This kind of container is run only once during the startup of every pod. A bunch of commands is ran then on top of it. For our current use case, it will copy the javaagent into a volume such as an empty directory volume.

2.1 Impacts in the “Wonderful Java Application Docker image

The main impact is to declare a volume in your Docker image:

VOLUME /opt/agent

It will be used by both the Docker container and the initContainer. We can consider it as a “bridge” between these two ones.

We also have to declare one environment variable: JAVA_OPTS.

For instance:

ENV JAVA_OPTS=$JAVA_OPTS
[...]
ENTRYPOINT ["sh", "-c", "java ${JAVA_OPTS} org.springframework.boot.loader.JarLauncher"]

Il will be used during the deployment to set up our Wonderful Java Application.

Now, let’s build our initContainer’s Docker image.

2.2 InitContainer Docker Image creation

It is really straightforward. We can use for example, the following configuration:

FROM alpine:latest
RUN mkdir -p /opt/agent_setup
RUN mkdir /opt/agent
COPY ./javaagent.jar /opt/agent_setup/javaagent.jar
VOLUME /opt/agent

2.3 Kubernetes configuration

We can now set up our Kubernetes Deployment to start the corresponding container and copy the Java agent.

kind: Deployment
spec:
  containers:
  - name: java-app
    image: repo/my-wonderful-java-app:v1
    volumeMounts:
    - mountPath: /opt/agent
      name: apm-agent-volume
  initContainers:
  - command:
    - cp
    - /opt/agent_setup/javaagent.jar
    - /opt/agent
    name: apm-agent-init
	image: repo/apm-agent:v1
    volumeMounts:
    - mountPath: /opt/agent
      name: appd-agent-volume
  volumes:
    - name: appd-agent-volume
      emptyDir: {}

3 Start the Java Application with the agent

Last but not least, we can now configure the pods where we run our Java applications.

We will use the JAVA_OPTS environment variable to configure the location of the Java agent, and the Elastic APM Java system properties.

For instance:

JAVA_OPTS=-javaagent:/opt/agent/javaagent.jar -Delastic.apm.service_name=my-wonderful-application -Delastic.apm.application_packages=org.mywonderfulapp -Delastic.apm.server_url=http://apm:8200

You can then configure your Kubernetes deployment as:

spec:
  containers:
  - name: java-app
    env:
    - name: JAVA_OPTS
      value: -javaagent:/opt/agent/javaagent.jar -Delastic.apm.service_name=my-wonderful-application -Delastic.apm.application_packages=org.mywonderfulapp -Delastic.apm.server_url=http://apm:8200

Et voila!

4 Conclusion

We have seen how to pack and deploy Distributed Tracing java agents and Java Applications built on top of Docker images. Obviously, my technical choice of using an InitContainer can be challenged regarding your technical context and how you are confortable with your delivery practices. You probably noticed I use an emptyDir to deploy the Java agent. Normally it will not be a big deal, but I advise you to check this usage with your Kubernetes SRE/Ops/Administrator first.

Anyway, I think it is worth it and the tradeoffs are more than acceptable because this approach are, in my opinion, more flexible than the first one.

Hope this helps!