feat(llm-o): connect Session replay to LLM Observability #29224

k11kirky · 2025-02-25T22:37:59Z

Problem

Not easy to connect LLM events and replys

Changes

Enhance UI for LLM events in session reply and add link to LLM trace.
If session id exisits in LLM event add link in trace back to replay

greptile-apps

PR Summary

Enhanced UI for LLM events in session recordings with bidirectional linking between session replays and LLM traces, providing better context for AI-related activities.

Added AIEventItems.tsx with AIEventExpanded and AIAIEventSummary components to display AI event details and error states
Modified ItemEvent.tsx to support AI events ($ai_generation, $ai_span, $ai_trace) with a dedicated "Conversation" tab and "View LLM Trace" button
Added hasSessionID and getSessionID utility functions in utils.ts to enable navigation between LLM traces and session recordings
Updated cost definitions for new AI models including 'claude-3.7-sonnet:thinking', 'deepseek-chat', and 'gemini-2.0-flash-lite-001'

_{7 file(s) reviewed, 5 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

frontend/src/scenes/session-recordings/player/inspector/components/AIEventItems.tsx

greptile-apps · 2025-02-25T22:39:09Z

frontend/src/scenes/session-recordings/player/inspector/components/AIEventItems.tsx

+    let input = event.properties.$ai_input_state
+    let output = event.properties.$ai_output_state ?? event.properties.$ai_error
+    let raisedError = event.properties.$ai_is_error
+    if (event.event === '$ai_generation') {
+        input = event.properties.$ai_input
+        output = event.properties.$ai_output_choices ?? event.properties.$ai_output
+        raisedError = event.properties.$ai_is_error


style: Redundant reassignment of raisedError on line 15 since it's already set on line 11 with the same value

Suggested change

let input = event.properties.$ai_input_state

let output = event.properties.$ai_output_state ?? event.properties.$ai_error

let raisedError = event.properties.$ai_is_error

if (event.event === '$ai_generation') {

input = event.properties.$ai_input

output = event.properties.$ai_output_choices ?? event.properties.$ai_output

raisedError = event.properties.$ai_is_error

let input = event.properties.$ai_input_state

let output = event.properties.$ai_output_state ?? event.properties.$ai_error

let raisedError = event.properties.$ai_is_error

if (event.event === '$ai_generation') {

input = event.properties.$ai_input

output = event.properties.$ai_output_choices ?? event.properties.$ai_output

greptile-apps · 2025-02-25T22:39:10Z

frontend/src/scenes/session-recordings/player/inspector/components/ItemEvent.tsx

@@ -92,9 +97,20 @@ export function ItemEvent({ item }: ItemEventProps): JSX.Element {
 }

 export function ItemEventDetail({ item }: ItemEventProps): JSX.Element {
+    // // Check if this is an LLM-related event


syntax: There's an extra comment slash here that should be removed

Suggested change

// // Check if this is an LLM-related event

// Check if this is an LLM-related event

greptile-apps · 2025-02-25T22:39:34Z

products/llm_observability/frontend/LLMObservabilityTraceScene.tsx

+                                <Link to={urls.replay(undefined, undefined, getSessionID(event) ?? '')}>
+                                    <IconOpenInNew />
+                                    View session recording
+                                </Link>


style: The icon and text should be wrapped in a flex container with proper alignment and spacing for better visual consistency with other links in the UI.

Could be a and would get pretty much the same styling but with gap between icon and text

greptile-apps · 2025-02-25T22:39:44Z

products/llm_observability/frontend/utils.ts

+    }
+    return '$session_id' in event


style: Redundant check for 'properties' in event since isLLMTraceEvent already confirms this property exists

Suggested change

}

return '$session_id' in event

return typeof event.properties.$session_id === 'string'

return '$session_id' in event

github-actions · 2025-02-25T22:50:02Z

Size Change: +2.48 kB (+0.03%)

Total Size: 9.72 MB

Filename	Size	Change
`frontend/dist/toolbar.js`	9.72 MB	+2.48 kB (+0.03%)

_{compressed-size-action}

…ents/AIEventItems.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

pauldambra · 2025-02-26T08:59:24Z

frontend/src/scenes/activity/explore/EventDetails.tsx

@@ -145,6 +147,14 @@ export function EventDetails({ event, tableProps }: EventDetailsProps): JSX.Elem
            label: 'Conversation',
            content: (
                <div className="mx-3 -mt-2 mb-2 space-y-2">
+                    {event.properties.$session_id ? (
+                        <div className="flex flex-row items-center gap-2">
+                            <Link to={urls.replay(undefined, undefined, event.properties.$session_id)}>


commented elsewhere this could be a lemon button

pauldambra · 2025-02-26T09:00:18Z

frontend/src/scenes/activity/explore/EventDetails.tsx

@@ -145,6 +147,14 @@ export function EventDetails({ event, tableProps }: EventDetailsProps): JSX.Elem
            label: 'Conversation',
            content: (
                <div className="mx-3 -mt-2 mb-2 space-y-2">
+                    {event.properties.$session_id ? (


if you know it's a web event you can gate on recording being active

in error tracking

issueProperties['$session_id'] && issueProperties['$recording_status'] === 'active'

but that's not essential - we just can't guarantee that the presence of a session id means there will be a recording.

pauldambra · 2025-02-26T09:01:56Z

frontend/src/scenes/session-recordings/player/inspector/components/ItemEvent.tsx

+    const isAIEvent =
+        item.data.event === '$ai_generation' || item.data.event === '$ai_span' || item.data.event === '$ai_trace'


nit-picking we could put this in taxonomy.ts but am only imagineering

pauldambra · 2025-02-26T09:06:00Z

frontend/src/scenes/session-recordings/player/inspector/components/ItemEvent.tsx

+        ) : item.data.event === '$ai_generation' ||
+          item.data.event === '$ai_span' ||
+          item.data.event === '$ai_trace' ? (
+            <AIEventSummary event={item.data} />


ah, i think you need eager loading of the properties for the error flag in here to work...

here we subscribe to web vitals events and preload their event properties

posthog/frontend/src/scenes/session-recordings/player/sessionRecordingDataLogic.ts

Line 1165 in 3f0512f

webVitalsEvents: (value: RecordingEventType[]) => {

you can copy that

we delay loading properties in the inspector list until expanded otherwise

pauldambra

a note on preloading but that could even be a follow-up

this is great!

daibhin · 2025-02-26T09:48:38Z

frontend/src/scenes/activity/explore/EventDetails.tsx

+                    {event.properties.$session_id ? (
+                        <div className="flex flex-row items-center gap-2">
+                            <Link to={urls.replay(undefined, undefined, event.properties.$session_id)}>
+                                <IconOpenInNew />
+                                View session recording
+                            </Link>
+                        </div>
+                    ) : null}


What Paul is suggesting would basically look like this:

Suggested change

{event.properties.$session_id ? (

<div className="flex flex-row items-center gap-2">

<Link to={urls.replay(undefined, undefined, event.properties.$session_id)}>

<IconOpenInNew />

View session recording

</Link>

</div>

) : null}

{mightHaveRecording(event.properties) && (

<ViewRecordingButton sessionId={event.properties.$session_id} />

)}

daibhin · 2025-02-26T09:49:48Z

products/llm_observability/frontend/LLMObservabilityTraceScene.tsx

+                        {hasSessionID(event) && (
+                            <div className="flex flex-row items-center gap-2">
+                                <Link to={urls.replay(undefined, undefined, getSessionID(event) ?? '')}>
+                                    <IconOpenInNew />
+                                    View session recording
+                                </Link>
+                            </div>
+                        )}


Same as my comment above

k11kirky added 4 commits February 25, 2025 10:05

itemEvent

42e096b

feat: conversation view by default on ai events

39d6ad2

Merge remote-tracking branch 'origin' into feat/llm-replay

7b10d5d

feat: trace to session

3718578

k11kirky requested a review from pauldambra February 25, 2025 22:37

k11kirky changed the title ~~itemEvent~~ feat(llm-o): connect Session replay to LLM Observability Feb 25, 2025

greptile-apps bot reviewed Feb 25, 2025

View reviewed changes

k11kirky and others added 2 commits February 25, 2025 22:57

Update frontend/src/scenes/session-recordings/player/inspector/compon…

bcca87f

…ents/AIEventItems.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

addlink to generations tab

d91c701

pauldambra reviewed Feb 26, 2025

View reviewed changes

pauldambra approved these changes Feb 26, 2025

View reviewed changes

daibhin reviewed Feb 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(llm-o): connect Session replay to LLM Observability #29224

feat(llm-o): connect Session replay to LLM Observability #29224

k11kirky commented Feb 25, 2025

greptile-apps bot left a comment

greptile-apps bot Feb 25, 2025

greptile-apps bot Feb 25, 2025

greptile-apps bot Feb 25, 2025

pauldambra Feb 26, 2025

greptile-apps bot Feb 25, 2025

github-actions bot commented Feb 25, 2025 •

edited

Loading

pauldambra Feb 26, 2025

pauldambra Feb 26, 2025

pauldambra Feb 26, 2025

pauldambra Feb 26, 2025

pauldambra left a comment

daibhin Feb 26, 2025

daibhin Feb 26, 2025

	// // Check if this is an LLM-related event
	// Check if this is an LLM-related event

		const isAIEvent =
		item.data.event === '$ai_generation' \|\| item.data.event === '$ai_span' \|\| item.data.event === '$ai_trace'

feat(llm-o): connect Session replay to LLM Observability #29224

Are you sure you want to change the base?

feat(llm-o): connect Session replay to LLM Observability #29224

Conversation

k11kirky commented Feb 25, 2025

Problem

Changes

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Feb 25, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 25, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 25, 2025

Choose a reason for hiding this comment

pauldambra Feb 26, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 25, 2025

Choose a reason for hiding this comment

github-actions bot commented Feb 25, 2025 • edited Loading

pauldambra Feb 26, 2025

Choose a reason for hiding this comment

pauldambra Feb 26, 2025

Choose a reason for hiding this comment

pauldambra Feb 26, 2025

Choose a reason for hiding this comment

pauldambra Feb 26, 2025

Choose a reason for hiding this comment

pauldambra left a comment

Choose a reason for hiding this comment

daibhin Feb 26, 2025

Choose a reason for hiding this comment

daibhin Feb 26, 2025

Choose a reason for hiding this comment

github-actions bot commented Feb 25, 2025 •

edited

Loading