Sr. AI Observability Engineer (Remote)
Tealium
When applying for roles at Tealium, please use our official careers page or LinkedIn company profile. All other sites where Tealium careers may appear may not be legitimate.
WHO WE ARE
Tealium is the trusted leader in real-time Customer Data Platforms (CDP), helping organizations unify their customer data to deliver more personalized, privacy-conscious experiences. As the demand for connected, intelligent customer engagement grows, Tealium’s leadership in CDP is translating directly into leadership in enabling enterprise AI strategies. By providing clean, consented, and actionable data, Tealium empowers its customers to accelerate the adoption of AI and machine learning, fueling smarter personalization, predictive insights, and business outcomes at scale.
More than 800 leading global brands trust Tealium to power their customer data strategies and deliver real-time, personalized experiences at scale.
Team Tealium has team members present in nearly 20 countries worldwide, serving customers across more than 30 countries. We win together with respect and appreciation for the talents required of all positions and the people who contribute to each of these. We are intentional about our WOWs (Ways of Work) culture, our investment in our team members, and how we care and connect.
With an extraordinary portfolio of investors (including Georgian, Silver Lake Waterman, Battery, and others) and deep industry experience, Tealium has the financial backing, profitability, and expertise to continue to outpace competitors and lead the way in innovation. Today, Tealium holds over 50 patents, and a few of the recent industry recognitions include:
A Leader in the 2025 Gartner® Magic Quadrant™ for Customer Data Platforms
2025 TrustRadius Award Winner: Buyer’s Choice
2024 Invoca Partner Collaboration Award
2024 G2 Leader in Tag Management & Enterprise Data Governance
Tealium Customer Data Hub achieved the Top Rated Award by TrustRadius (2024)
Named on Destination CRM’s 2024 Top 100 Technologies List for Sales
Named on the 2024 Best and Brightest in the Nation list
BuiltIn’s 2024 Best Place to Work
WHAT WE ARE LOOKING FOR
We are seeking a Senior AI Observability Engineer to lead the observability strategy for Tealium’s AI/ML systems and AI-powered features. This role blends advanced observability engineering with a strong understanding of AI/ML lifecycles, ensuring visibility, reliability, performance, and responsible usage of both off-the-shelf and custom AI models across our products and internal platforms.
You’ll join a team of 3 observability engineers, working cross-functionally with SRE, MLOps, data engineering, security, and product teams to deliver instrumentation for model quality, latency, drift, cost, user experience, and ethical safeguards.
YOUR DAY TO DAY
Lead end-to-end observability design for AI/ML features in production and internal usage (e.g., RAG, Copilots, LLM-enhanced customer experiences).
Instrument AI features in Tealium products (e.g., ML-powered segmentation, decisioning, or predictions) for latency, accuracy, drift, usage, and cost.
Implement monitoring and cost tracking for third-party AI services (OpenAI, Anthropic Claude, Amazon Q, etc.), including rate limiting, quota management, and failover strategies
Build telemetry pipelines to track LLM request/response metrics, prompt engineering observability, token usage, hallucination detection, and failover.
Collaborate with data science and product teams to define and automate quality SLIs/SLOs for models.
Implement AI-aware tracing (e.g., OpenTelemetry + LangChain/LLM traces) into the broader observability stack.
Participate in on-call rotations and help triage AI-specific incidents related to model regressions, latency spikes, or API failures.
Automate validation pipelines to ensure AI features are robust across environments.
Establish dashboards and alerts for AI observability using tools like Datadog, Sumologic, Prometheus, OpenTelemetry, and Grafana.
Contribute to ethical AI monitoring practices: PII exposure detection, prompt abuse, fairness, and content compliance.
Help guide Tealium’s use of Generative AI developer tools (e.g., GitHub Copilot, Amazon Q Developer, Cursor) for coding efficiency and ensure telemetry around their use is captured appropriately.
Initial goal within 6 months: Establish baseline AI observability across 3+ production ML features.
WHAT YOU BRING TO TEALIUM
6+ years in Site Reliability Engineering, Observability Engineering, or ML Ops with a focus on production-grade AI/ML systems.
Deep experience in instrumenting AI pipelines (e.g., LLMs, recommender systems, ML APIs) for observability, including drift detection and cost tracking.
Familiarity with prompt engineering, embeddings, vector DBs (Neptune), and RAG-style architectures.
Hands-on experience with OpenTelemetry, Datadog, Sumologic, Prometheus, or similar.
Experience integrating observability into AI platforms: e.g., Bedrock, Neptune, LangChain, LlamaIndex, HuggingFace, SageMaker, etc.
Proficiency with Python, Go, or similar languages used in backend and ML infrastructure.
Familiarity with AWS services (especially those relevant to AI: SageMaker, Bedrock, Lambda, DynamoDB, etc.).
Experience deploying and observing third-party LLM APIs (OpenAI, Claude, Amazon Q).
Strong background in Infrastructure-as-Code (Terraform, ArgoCD) and CI/CD tooling (Jenkins, GitHub Actions).
Understanding of Kubernetes and container orchestration.
Experience with FinOps/cost optimization for AI workloads
Strong understanding of ethical AI practices and responsible telemetry instrumentation. Additionally, Data Privacy and compliance experience
Excellent collaboration skills and comfort leading across SRE, Data Engineering, and Product/ML teams.
Experience mentoring or leading technical initiatives
Communication skills for explaining complex AI concepts to non-technical stakeholders
WAGE TRANSPARENCY
In many U.S. states, employers are required to include a pay range for posted positions. Although this isn't a requirement in every state, communicating transparently is a cornerstone of our operations at Tealium, and we believe in making this information available to all applicants.
The U.S. pay range for this full-time position is listed below, however, base pay offered may vary depending on job-related knowledge, skills, and experience. In addition to a competitive base salary, this position is eligible for a robust benefits package that includes the following:
Employees are eligible to receive an annual bonus and stock options.
Employees and their families are eligible for medical, dental, vision, life, and disability insurance.
Employees have the option to enroll in our 401k plan and are eligible to receive contributions for company matching.
Employees are eligible for flexible paid time-off and extended paid parental leave.
We offer 11 paid holidays annually.
We offer 15 hours of paid work time for volunteer activities and programs.
-
Our sick leave accrual is the following for our employees:
Exempt CA employees (not including San Francisco) including NY : accrue 40 hours each year. Unused sick leave carries over into the next year. Employees cannot exceed 80 hours in a given year.
Exempt Non - CA employees (not including NY) including SF: Accrue 1 hour every 30 hours worked. Cannot exceed 180 hours in the calendar year.
Non-Exempt: accrue 1 hour every 30 hours worked. Unused carries over to the next year. Not to exceed 108 hours in a calendar year.
An overview of our benefits and perks can be found on our careers page, https://tealium.com/careers/. Additional details regarding the benefits package will be provided during your interview process.
Compensation Range: $165,000 - $200,000 Base + Variable
This position will earn commission pursuant to Tealium’s commission policy, the details of which will be provided upon request.
#LI-KK1
#LI-Remote
WHY YOU WANT TO WORK HERE
At Tealium, we don’t just offer the ordinary, we provide the extraordinary:
- Tealium WOWs (Ways of Work), our award winning culture is how with think, act and connect together at Tealium
- Mosaic, our commitment to diversity, equity and inclusion is grounded in our mosaic of diverse perspectives and shared belonging as we live in work across the US and in nearly 20 countries
- Tealium Cares, to promote caring in our communities, 15 hours of paid work time for volunteer activities and programs is offered annually
- Tealium Connects (remote-first working), enabling many of us to choose where we do our best work and offering new hire stipends to assist with purchasing things we need to support a successful home office environment
- Tealium Ownership, share in the success of Tealium by becoming an owner of Tealium beginning with new hire equity grants
- Tealium Time, paid time-off policy to offer flexibility to take time when needed and robust leave programs, including extended paid parental leave and company holidays
- Healium, health and wellness programs to help us be our best selves in the experiences of health, physical, mental, social, and even financial well-being and wellness
- Tealium LIFT (Learning is Facilitated at Tealium), offering a myriad of professional development opportunities with over 6,000 courses available on demand to best-in-class manager and leadership development programs
- Health and Related Benefits Programs, offering market competitive benefits programs