Conversion Tracking & Cohort Analytics

Measure the business impact of your AI interactions by correlating LLM usage with downstream outcomes. Demeterics enables you to group interactions into cohorts and then submit outcomes (conversions, engagement metrics, revenue) to analyze which prompts, variants, or models drive real results.

How It Works

1. Tag interactions with a cohort ID using the /// COHORT metadata header
2. Collect business outcomes (likes, purchases, CSAT scores, etc.)
3. Submit outcomes via API to correlate with the cohort's interactions
4. Analyze in BigQuery to find which AI strategies perform best

This enables true A/B testing of AI experiences, ROI measurement, and data-driven prompt optimization.

1) Tagging Interactions with Cohorts

Add the /// COHORT header to your prompts to group interactions. Common patterns:

/// APP VideoRecommender
/// FLOW generate_caption
/// COHORT video-abc123
/// VARIANT control

Generate an engaging caption for this video about cooking...

The cohort ID can be any identifier meaningful to your business:

• Content ID (video, article, product)
• Session/user ID for engagement tracking
• Campaign ID for marketing attribution
• Support ticket ID for CSAT correlation

Supported Aliases

The following are all equivalent:

• /// COHORT video-123
• /// COHORT_ID video-123
• /// ID video-123
• /// EXPERIMENT_ID video-123
• /// BATCH_ID video-123

Cohort ID Requirements

• Max length: 128 characters
• Allowed characters: Alphanumeric, hyphens (-), underscores (_)
• Examples: video-abc123, user_session_456, campaign-2025-q1

2) Submitting Outcomes via API

After collecting your business metrics, submit them to correlate with the cohort:

POST /api/v1/cohort/outcome

curl -X POST https://api.demeterics.com/api/v1/cohort/outcome \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "cohort_id": "video-abc123",
    "outcome": 1523.0,
    "outcome_v2": 42.5,
    "label": "24h engagement",
    "event_date": "2025-11-28"
  }'

Request Fields

Response

{
  "success": true,
  "cohort_id": "video-abc123",
  "outcome": 1523.0,
  "outcome_v2": 42.5,
  "event_date": "2025-11-28",
  "interaction_count": 47,
  "message": "Outcome recorded for cohort video-abc123"
}

The interaction_count shows how many LLM interactions are associated with this cohort.

Upsert Behavior

Submitting an outcome for the same cohort_id + event_date combination will update the existing record. This allows you to:

• Update metrics as they change (e.g., likes accumulating over time)
• Correct mistaken submissions
• Track the same cohort across multiple days with different event_date values

3) Retrieving Outcomes

GET /api/v1/cohort/outcome

curl -X GET "https://api.demeterics.com/api/v1/cohort/outcome?cohort_id=video-abc123" \
  -H "Authorization: Bearer dmt_your_api_key"

Response

{
  "success": true,
  "cohort_id": "video-abc123",
  "outcome": 1523.0,
  "outcome_v2": 42.5,
  "interaction_count": 47
}

4) Use Cases by Industry

Content & Media

Video platforms:

• outcome: Views, watch time, or likes
• outcome_v2: Shares or comments
• label: "7d engagement" or "viral score"

# Video caption A/B test
curl -X POST https://api.demeterics.com/api/v1/cohort/outcome \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "cohort_id": "video-cooking-tips-v2",
    "outcome": 45230,
    "outcome_v2": 1247,
    "label": "views and likes after 48h"
  }'

Content recommendation:

• outcome: Click-through rate or dwell time
• outcome_v2: Bounce rate
• label: "recommendation CTR"

E-commerce

Product descriptions:

• outcome: Conversion rate or add-to-cart count
• outcome_v2: Revenue per view
• label: "product page conversion"

Search & discovery:

• outcome: Search-to-purchase rate
• outcome_v2: Average order value
• label: "search intent match"

Customer Support

Chatbot effectiveness:

• outcome: CSAT score (1-5 or 1-10)
• outcome_v2: Resolution time in minutes
• label: "support ticket CSAT"

# Customer support satisfaction
curl -X POST https://api.demeterics.com/api/v1/cohort/outcome \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "cohort_id": "ticket-12345",
    "outcome": 4.5,
    "outcome_v2": 3.2,
    "label": "CSAT and resolution time"
  }'

Agent assist quality:

• outcome: Agent acceptance rate of AI suggestions
• outcome_v2: Time saved per interaction
• label: "agent assist adoption"

Marketing & Personalization

Email/notification copy:

• outcome: Open rate or click rate
• outcome_v2: Unsubscribe rate
• label: "email campaign performance"

Ad creative:

• outcome: CTR or ROAS
• outcome_v2: Cost per acquisition
• label: "ad variant performance"

Healthcare & Life Sciences

Clinical decision support:

• outcome: Clinician acceptance rate
• outcome_v2: Time to decision
• label: "CDS adoption"

Patient engagement:

• outcome: Adherence score
• outcome_v2: Follow-up completion rate
• label: "patient engagement"

Finance & Insurance

Underwriting assistance:

• outcome: Approval accuracy
• outcome_v2: Processing time reduction
• label: "underwriting efficiency"

Financial advice:

• outcome: Client satisfaction score
• outcome_v2: Assets under management change
• label: "advisor assist quality"

Education

Tutoring effectiveness:

• outcome: Quiz score improvement
• outcome_v2: Time to mastery
• label: "learning outcome"

Content generation:

• outcome: Student engagement score
• outcome_v2: Completion rate
• label: "educational content quality"

5) Analyzing Results in BigQuery

Once you have outcomes and interactions tagged, join them for analysis:

Basic Cohort Analysis

SELECT
  o.cohort_id,
  o.outcome AS business_metric,
  o.label AS metric_type,
  COUNT(i.transaction_id) AS interaction_count,
  AVG(i.tokens.total) AS avg_tokens,
  AVG(i.timing.latency_ms) AS avg_latency_ms,
  SUM(i.total_cost) AS total_cost
FROM `demeterics.demeterics.cohort_outcomes` o
LEFT JOIN `demeterics.demeterics.interactions` i
  ON o.cohort_id = i.meta.cohort_id
  AND o.user_id = i.user_id
WHERE o.event_dt >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY o.cohort_id, o.outcome, o.label
ORDER BY o.outcome DESC;

A/B Test Analysis by Variant

SELECT
  i.meta.variant,
  AVG(o.outcome) AS avg_outcome,
  COUNT(DISTINCT o.cohort_id) AS cohort_count,
  COUNT(i.transaction_id) AS total_interactions,
  AVG(i.total_cost) AS avg_cost_per_interaction
FROM `demeterics.demeterics.cohort_outcomes` o
JOIN `demeterics.demeterics.interactions` i
  ON o.cohort_id = i.meta.cohort_id
  AND o.user_id = i.user_id
WHERE i.meta.app = 'VideoRecommender'
  AND i.meta.flow = 'generate_caption'
GROUP BY i.meta.variant
ORDER BY avg_outcome DESC;

ROI by Model

SELECT
  i.model,
  AVG(o.outcome) AS avg_outcome,
  SUM(i.total_cost) AS total_cost,
  AVG(o.outcome) / NULLIF(SUM(i.total_cost), 0) AS roi_ratio
FROM `demeterics.demeterics.cohort_outcomes` o
JOIN `demeterics.demeterics.interactions` i
  ON o.cohort_id = i.meta.cohort_id
  AND o.user_id = i.user_id
WHERE o.event_dt >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
GROUP BY i.model
HAVING total_cost > 0
ORDER BY roi_ratio DESC;

6) Best Practices

Cohort ID Design

• Be consistent: Use the same ID format across all interactions for a cohort
• Be specific: video-abc123 is better than just abc123
• Be meaningful: Include context like campaign-spring2025-variant-a

Outcome Timing

• Immediate signals: Thumbs up/down, click-through (submit within minutes/hours)
• Short-term metrics: 24h likes, daily engagement (submit next day)
• Long-term outcomes: Revenue, retention (submit after attribution window closes)

Use the event_date field to track metrics at different time horizons for the same cohort.

Multi-dimensional Metrics

Use outcome and outcome_v2 for related metrics:

• outcome: Primary success metric (revenue, CSAT)
• outcome_v2: Secondary diagnostic (cost, time)

Or submit multiple outcomes with different event_date values to track evolution.

Security & Isolation

• Outcomes are scoped to your API key's user_id and api_key_id
• You cannot read or modify other users' cohorts
• Cohort IDs are sanitized to prevent injection attacks

7) Complete Example: Video Caption A/B Test

Step 1: Generate captions with cohort tagging

# Control variant
curl -X POST https://api.demeterics.com/groq/v1/chat/completions \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{
      "role": "system",
      "content": "/// APP VideoApp\n/// FLOW caption\n/// VARIANT control\n/// COHORT video-abc123\n\nGenerate a brief, professional caption for this video."
    }, {
      "role": "user",
      "content": "Video about making sourdough bread"
    }]
  }'

# Treatment variant
curl -X POST https://api.demeterics.com/groq/v1/chat/completions \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{
      "role": "system",
      "content": "/// APP VideoApp\n/// FLOW caption\n/// VARIANT emoji-style\n/// COHORT video-def456\n\nGenerate an engaging caption with emojis for this video."
    }, {
      "role": "user",
      "content": "Video about making sourdough bread"
    }]
  }'

Step 2: Submit outcomes after 24 hours

# Control cohort outcome
curl -X POST https://api.demeterics.com/api/v1/cohort/outcome \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "cohort_id": "video-abc123",
    "outcome": 1250,
    "outcome_v2": 45,
    "label": "24h views and likes"
  }'

# Treatment cohort outcome
curl -X POST https://api.demeterics.com/api/v1/cohort/outcome \
  -H "Authorization: Bearer dmt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "cohort_id": "video-def456",
    "outcome": 2340,
    "outcome_v2": 127,
    "label": "24h views and likes"
  }'

Step 3: Analyze in BigQuery

SELECT
  i.meta.variant,
  AVG(o.outcome) AS avg_views,
  AVG(o.outcome_v2) AS avg_likes,
  AVG(o.outcome_v2) / NULLIF(AVG(o.outcome), 0) * 100 AS like_rate_pct,
  COUNT(DISTINCT o.cohort_id) AS videos_tested
FROM `demeterics.demeterics.cohort_outcomes` o
JOIN `demeterics.demeterics.interactions` i
  ON o.cohort_id = i.meta.cohort_id
  AND o.user_id = i.user_id
WHERE i.meta.app = 'VideoApp'
  AND i.meta.flow = 'caption'
GROUP BY i.meta.variant;

Result: The emoji-style variant achieved 87% more views and 182% more likes, demonstrating a clear winner for this use case.

Next Steps

• Prompt Engineering Guide - Learn about metadata headers and A/B testing
• API Reference - Full endpoint documentation
• BigQuery Schema - Understand the data model