Principal Observability Engineer



Multiple locations
Posted on Wednesday, June 26, 2024

To get the best candidate experience, please consider applying for a maximum of 3 roles within 12 months to ensure you are not duplicating efforts.

Job Category

Software Engineering

Job Details

About Salesforce

We’re Salesforce, the Customer Company, inspiring the future of business with AI+ Data +CRM. Leading with our core values, we help companies across every industry blaze new trails and connect with customers in a whole new way. And, we empower you to be a Trailblazer, too — driving your performance and career growth, charting new paths, and improving the state of the world. If you believe in business as the greatest platform for change and in companies doing well and doing good – you’ve come to the right place.

Principal Observability Engineer

Business Technology connects people and technology to transform the future of work at Salesforce. Guided by our core values of Trust, Customer Success, Equality, Innovation, and Sustainability, we deliver business outcomes that fuel growth, drive competitive advantage, and empower our employees and customers globally. BT’s scope stretches beyond traditional IT: We are also strategic partners, advocating for the best outcomes for our customers, always innovating, and helping to shape the future of work.

We oversee technology strategy, Salesforce on Salesforce, applications engineering, infrastructure, collaboration, enterprise operations, architecture, and program enablement. We are Customer Zero, the best example of Salesforce products delivered globally, at scale, and sustainably.

The Salesforce Business Technology (BT) Technology Services team seeks a Principal Observation Engineer. This role contributes to the overall success of the Business Technology (BT) organization by providing vision, oversight, and improvement of the Observation platform and focusing on enabling automation and efficiency. This role partners daily with BT partners, Infrastructure, Enterprise Applications, and Product Management teams, working on assignments and delivering services across infrastructure and application domains.

Areas of ownership include deployment oversight, service improvement, and data analysis of the BT Observability domain. Thought leadership and innovation are expected across Security, Integrations (M&A), End-user Experience, AI, and Scalability. An in-depth understanding of technology trends and innovation opportunities, including AI-powered observability, OpenTelemetry, standardized data formats, and more. The Principal Observability Engineer will be accountable as the delivery lead and is expected to lead the design and delivery of the technology architecture in the most complex and high-priority projects or work within BT Observability.


  • Serve as an Evangelist for the BT Observability domain across the enterprise and influence engineers, business partners, and stakeholders to apply observability best practices.

  • Design, develop, and maintain automation solutions to support observability and BT operations, focusing on improving system monitoring, alerting, and reporting capabilities.

  • Evaluate, design, and deliver AIOps solutions tailored to our organization’s needs.

  • Partner with architects, business partners, colleagues, and technology areas to design and document a technical structure based on standards and processes to meet the strategic and integration needs of the business.

  • Provide technology and/or process solutions to high-impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards.

  • Own, develop, and be accountable for BT Observability policies, processes, and architectural decisions.

  • Responsible for ensuring operational methods, procedures, facilities, and tools are established, reviewed, and maintained.

  • Evaluate vendor software releases to determine functional requirements and provide detailed information on how changes will affect end-users.

  • Monitor and research emerging observability trends and technologies with the potential to improve efficiency, security, and business capabilities. Develop and execute proof-of-concept projects to evaluate new solutions for potential adoption.

  • Develops documentation (e.g., including data flow diagrams, logical diagrams, and physical diagrams) and training in compliance with standards.

  • Apply enterprise design principles and best practices for implementing and supporting BT Observability services.

  • Operate with a limited level of direct supervision and exercise independence of judgment and autonomy.

  • Serve as advisor and coach to less senior team members, allocating work as necessary.

Required Skills/Experience

  • 8+ years of dynamic experience in infrastructure/software engineering, with at least 5 years in an observability role leading large-scale, distributed systems/applications.

  • Excellent communication (visual, verbal, and written) and interpersonal skills, ability to collaborate with internal customers and establish solid, collaborative partnerships with engineers, non-technical team members, and executives.

  • Outstanding leadership capabilities, with the ability to build a positive, collaborative, and technically-focused team environment.

  • Strong project management and organizational skills, prioritizing multiple priorities and achieving results.

  • Advanced knowledge, experience, and understanding of observability concepts, principles, practices, and their components, including SDKs and APIs.

  • Proficiency coding in a language (e.g., Python, Terraform, PowerShell, JavaScript, Typescript, NodeJS, etc.).

  • Experience with cloud automation/orchestration technologies (Puppet, Ansible, Bigfix, AWS SSM, etc.)

  • Experience with compute technologies/platforms (VMware, AWS, Docker, Kubernetes, Podman, AWS ECS, AWS EKS, etc.)

  • Maintain revision control, automated testing workflow, and continuous delivery experience (i.e., Git, Jenkins)

  • Demonstrated understanding and respect of IT service management practices (e.g., change, release, incident, problem management).

  • Develop and maintain standards for specific domains of responsibility.

  • The ability to influence and complete projects spanningmulti-functionall domains high in complexity and risk.

  • Proactively review and research trends in technology and recommend new paths forward from a strategic and execution perspective.

  • Highly analytical with a proven track record of solving sophisticated problems, bringing creative and technical approaches to bear.

  • Experience creating long-range plans, technical visions, and roadmaps


If you require assistance due to a disability applying for open positions please submit a request via this Accommodations Request Form.

Posting Statement

At Salesforce we believe that the business of business is to improve the state of our world. Each of us has a responsibility to drive Equality in our communities and workplaces. We are committed to creating a workforce that reflects society through inclusive programs and initiatives such as equal pay, employee resource groups, inclusive benefits, and more. Learn more about Equality at and explore our company benefits at

Salesforce is an Equal Employment Opportunity and Affirmative Action Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status. Salesforce does not accept unsolicited headhunter and agency resumes. Salesforce will not pay any third-party agency or company that does not have a signed agreement with Salesforce.

Salesforce welcomes all.