SC-23051 perf: integrate pyroscope for continuous profiling

[akshatagarwl]

Summary

Integrate Pyroscope for continuous profiling of JVM applications monitored by App Agent.

Changes

• Added Pyroscope Java agent dependency (v0.13.0)
• Created PyroscopeInitializer class to initialize Pyroscope profiling
• Initialize Pyroscope during agent startup in MonitorAgent
• Added default Pyroscope properties in agent.default.properties

Configuration

Pyroscope is configured via environment variables (passed by STA):

Variable	Description	Default
PYROSCOPE_ENABLED	Enable profiling	false
PYROSCOPE_SERVER_ADDRESS	Pyroscope server URL	http://localhost:4040
PYROSCOPE_APPLICATION_NAME	App name in Pyroscope	sematext-agent
PYROSCOPE_LABELS	Static labels (key=val,key2=val2)	empty
PYROSCOPE_PROFILER_ALLOC	Alloc threshold	512k
PYROSCOPE_PROFILER_LOCK	Lock threshold	10ms

Profile Types

With JFR format enabled:

• CPU profiling (default)
• Allocation profiling (if threshold set)
• Lock profiling (if threshold set)

Related

• Companion PR for STA: https://github.com/sematext/agent/pull/1280

Testing Without Merging

Option 1: Local Testing (Java Agent Only)

# Build the Java agent
mvn clean package -DskipTests

# Run with a simple Java app
export PYROSCOPE_ENABLED=true
export PYROSCOPE_SERVER_ADDRESS=http://localhost:4040
export PYROSCOPE_APPLICATION_NAME=test-app
java -javaagent:spm-monitor/target/spm-monitor-*-withdeps.jar \
  -jar your-app.jar

Option 2: Test Cluster with Dev Images

1. Build STA dev image (from sematext/agent repo):

git checkout aks_SC-23051_pyroscope-aa-passthrough
lima make AGENT_VERSION=dev-pyroscope-$(git rev-parse --short HEAD) image-dev

2. Build App Agent dev image (trigger Jenkins or build locally):

mvn clean package -DskipTests
docker build -t sematext/app-agent:dev-pyroscope-$(git rev-parse --short HEAD) .
docker push sematext/app-agent:dev-pyroscope-$(git rev-parse --short HEAD)

3. Update deployment config:

# In deployment/configs/sematext-agent-test.yaml
CONTAINER_APP_AGENT_IMAGE: sematext/app-agent:dev-pyroscope-<sha>
AUTODISCO_PYROSCOPE_ENABLED: true
AUTODISCO_PYROSCOPE_SERVER_ADDRESS: http://pyroscope.pyroscope.svc.cluster.local:4040
AUTODISCO_PYROSCOPE_APPLICATION_NAME: sematext-aa

4. Verify profiles in Pyroscope:

kubectl port-forward -n pyroscope svc/pyroscope 4040:4040
# Open http://localhost:4040 and search for sematext-aa

Test Plan

• Build Java agent with Pyroscope integration
• Verify Pyroscope agent starts when PYROSCOPE_ENABLED=true
• Verify profiles are sent to Pyroscope server
• Test with allocation profiling enabled
• Test with lock profiling enabled
• Integration test with STA passing config

[otisg]

@akshatagarwl did you want to add a reviewer here?

[akshatagarwl]

@otisg not yet. need to do

we also need to make STA configurable so that it spawns AA with correct pyroscope values when initializing

first