Senior Data Engineer-Databricks

Versant•Universal City, CA

6h•$140,000 - $160,000

About The Position

The Data Engineering team is seeking a Senior Data Engineer to help design, build, and scale the modern data platform that powers analytics, data science, and data products across Versant brands. In this role, you’ll collaborate closely with Data Product, Data Science, Analytics, and Engineering teams to deliver reliable, high-impact data solutions used by hundreds of internal users. We're looking for engineers who enjoy hands-on development, take ownership of production systems, and influence implementation through collaboration and technical expertise.

Requirements

Bachelor’s degree in Computer Science, Data Engineering, or equivalent practical experience
5+ years of experience building production data solutions in cloud environments (e.g., Databricks, AWS)
Expertise in SQL and Python, with strong proficiency in Spark-based data processing
Hands-on experience building batch or streaming production data pipelines using distributed processing frameworks (e.g., Spark, Flink) and query engines such as Presto
Proficiency with orchestration tools such as Apache Airflow or Dagster, and experience designing systems with monitoring, logging, and alerting for proactive issue detection
Experience working with modern data architectures, including event-driven and distributed systems
Proficiency with Git and collaborative development workflows
Solid understanding of infrastructure, networking, and data security fundamentals
Passion for clean, reliable, and scalable data systems
Strong communication and collaboration skills
Ability to balance near-term delivery with thoughtful technical design
Curiosity and a growth mindset
A team-first attitude who supports and uplifts others

Nice To Haves

Experience building Lakehouse platforms and medallion pipelines in Databricks
Familiarity with Unity Catalog, data governance, and compliance frameworks (e.g., PCI)
Hands-on experience with CI/CD pipelines, orchestration tools, and infrastructure-as-code
Experience with Lakeflow Spark Declarative Pipelines (SDP), MLflow, feature stores, and MLOps practices
Background in media and entertainment data (e.g., video metadata, ad tech, audience analytics)
Experience building data platforms within the media industry, with a strong understanding of audience analytics. Hands-on experience with audience research tools and methodologies (e.g., Nielsen), clickstream data processing, and identity graph or identity spine solutions for identity resolution and personalization
Comfortable using AI-assisted development tools (e.g., ChatGPT)
Databricks or cloud certifications (e.g., Databricks Certified Data Engineer)

Responsibilities

Design and implement lakehouse architecture using Delta Lake, including medallion pipeline patterns (Bronze/Silver/Gold), schema enforcement, and time travel
Build and operate batch and real-time ingestion pipelines leveraging Databricks Auto Loader, Structured Streaming, and Change Data Capture patterns
Implement data governance and security using Unity Catalog, RBAC, and compliance-driven practices for sensitive environments
Optimize performance and manage costs through FinOps strategies, including cluster sizing, workload tagging, Spark tuning, and Photon acceleration
Design, implement, and maintain CI/CD pipelines and orchestration workflows using Databricks Workflows, Delta Live Tables, and tools such as Airflow
Collaborate with Data Science teams on ML workflows, including MLflow, feature store integration, and model lifecycle management
Ensure data quality, observability, and lineage across media-specific datasets such as streaming logs, ad impressions, and audience metrics
Provide technical mentorship through code reviews, pairing and knowledge sharing