About the position
We are seeking a highly skilled Senior Data Engineer to help evaluate and design robust data integration solutions for large-scale, disparate datasets spanning multiple platforms and infrastructure types, including cloud-based and potentially undefined or evolving environments. This role is critical in identifying optimal data ingestion, normalization, and transformation strategies while collaborating with cross-functional teams to ensure data accessibility, reliability, and security across systems. This position is contingent upon award.
Responsibilities
- Develop, expand and optimize our data and data pipeline architecture.
- Optimize data flow and collection for cross functional teams.
- Support software developers, database architects, data analysts and data scientists on data initiatives.
- Ensure optimal data delivery architecture is consistent throughout ongoing projects.
- Create new pipeline and maintain existing pipeline.
- Update Extract, Transform, Load (ETL) process and create new ETL features.
- Build PoCs with Redshift Spectrum, Databricks, AWS EMR, SageMaker, etc.
- Implement large dataset engineering: data augmentation, data quality analysis, data analytics, data profiling, data algorithms, and develop data strategy recommendations.
- Operate large-scale data processing pipelines and resolve business and technical issues.
- Assemble large, complex sets of data that meet non-functional and functional business requirements.
- Identify, design, and implement internal process improvements.
- Build required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies.
- Build analytical tools to utilize the data pipeline, providing actionable insight into key business performance metrics.
- Work with stakeholders including data, design, product and government stakeholders.
- Write unit and integration tests for all data processing code.
- Work with DevOps engineers on CI, CD, and IaC.
- Read specs and translate them into code and design documents.
- Perform code reviews and develop processes for improving code quality.
- Perform other duties as assigned.
Requirements
- All candidates must pass public trust clearance through the U.S. Federal Government.
- Bachelor's degree in Computer Science, Software Engineering, Data Science, Statistics, or related technical field.
- 10+ years of experience in software/data engineering, including data pipelines, data modeling, data integration, and data management.
- Expertise in data lakes, data warehouses, data meshes, data modeling and data schemas (star, snowflake…).
- Strong expertise in SQL, Python, and/or R, with applied experience in Apache Spark and large-scale processing using PySpark or Sparklyr.
- Experience with Databricks in a production environment.
- Strong experience with AWS cloud-native data services, including S3, Glue, Athena, and Lambda.
- Strong proficiency with GitHub and GitHub Actions, including test-driven development.
- Proven ability to work with incomplete or ambiguous data infrastructure and design integration strategies.
- Excellent analytical, organizational, and problem-solving skills.
- Strong communication skills, with the ability to translate complex concepts across technical and business teams.
- Proven experience working with petabyte-level data systems.
Nice-to-haves
- Experience working with healthcare data, especially CMS (Centers for Medicare & Medicaid Services) datasets.
- In-depth knowledge of CMS regulations and experience with complex healthcare projects.
- Demonstrated success providing support within the CMS OIT environment.
- Demonstrated experience and familiarity with CMS OIT data systems (e.g. IDR-C, CCW, EDM, etc.).
- Experience with cloud platform services: AWS and Azure.
- Experience with streaming data (Kafka, Kinesis, Pub/Sub).
- Familiarity with data governance, metadata management, and data quality practices.
Benefits
- Highly competitive salaries.
- Full healthcare benefits.
