Details

Added NLTK-based sentiment analyzer metric that provides positive, negative, neutral, and compound sentiment scores for text evaluation.

Issues

Resolves #1151

Testing

• Added unit tests for the Sentiment metric that verify:
  • Positive, negative, and neutral sentiment classification
  • Empty text handling
  • Import error handling when nltk is not available

Manually tested the metrics with this script

# test_sentiment.py
import opik
from opik.evaluation.metrics.heuristics.sentiment import Sentiment

opik.configure(use_local=True)

from opik import Opik
opik_client = Opik(project_name="sentiment-test-project")
project_name = "sentiment-test-project"
print(f"Using project: {project_name}")

sentiment = Sentiment()

test_texts = [
   "I absolutely love this product! It's amazing and exceeded all my expectations.",
   "This is terrible. I'm very disappointed with the quality and service.",
   "The product arrived on time. It works as described."
]

trace = opik_client.trace(name="sentiment-analysis-test")

for i, text in enumerate(test_texts):
   span = trace.span(
       name=f"text-{i+1}",
       input={"text": text}
   )
   
   result = sentiment.score(text)
   
   span.update(
       metadata={
           "sentiment_score": result.value,
           "sentiment_metadata": result.metadata,
           "sentiment_reason": result.reason
       },
       output={"sentiment_score": result.value}
   )
   span.end()
   
   print(f"\nText {i+1}: {text}")
   print(f"Sentiment score: {result.value}")
   print(f"Metadata: {result.metadata}")
   print(f"Reason: {result.reason}")

trace.end()

print("\nTest complete! Check your Opik UI at http://localhost:5173")
print(f"Navigate to project '{project_name}' to see the results")

see the log

(venv) ❯ python3 test_sentiment.py
Found local Opik instance on: http://localhost:5173/, do you want to use it? (Y/n)y
OPIK: Configuration saved to file: /home/mintu-ubuntu/.opik.config
Using project: sentiment-test-project
OPIK: Started logging traces to the "sentiment-test-project" project at http://localhost:5173/api/v1/session/redirect/projects/?trace_id=0197075d-18c1-7f39-8c74-1c8aff42ef90&path=aHR0cDovL2xvY2FsaG9zdDo1MTczL2FwaS8=.
OPIK: Started logging traces to the "Default Project" project at http://localhost:5173/api/v1/session/redirect/projects/?trace_id=0197075d-18c3-779b-972d-ebc808416e3d&path=aHR0cDovL2xvY2FsaG9zdDo1MTczL2FwaS8=.

Text 1: I absolutely love this product! It's amazing and exceeded all my expectations.
Sentiment score: 0.862
Metadata: {'neg': 0.0, 'neu': 0.512, 'pos': 0.488, 'compound': 0.862}
Reason: Text sentiment analysis: positive (compound score: 0.8620)

Text 2: This is terrible. I'm very disappointed with the quality and service.
Sentiment score: -0.7574
Metadata: {'neg': 0.419, 'neu': 0.581, 'pos': 0.0, 'compound': -0.7574}
Reason: Text sentiment analysis: negative (compound score: -0.7574)

Text 3: The product arrived on time. It works as described.
Sentiment score: 0.0
Metadata: {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0}
Reason: Text sentiment analysis: neutral (compound score: 0.0000)

Test complete! Check your Opik UI at http://localhost:5173
Navigate to project 'sentiment-test-project' to see the results

in the UI

https://github.com/user-attachments/assets/5afa810d-751a-450a-a1ff-faf8ca255865

Documentation

Updated in apps/opik-documentation/documentation/fern/docs/evaluation/metrics/heuristic_metrics.mdx

/claim #1151