Senior Software Engineer - Data Delivery

About the position

The Data Delivery team is seeking a Senior Software Engineer with a proven track record building large scale data streaming platforms. The ideal candidate will play a crucial role in developing and maintaining data-related systems comprising high throughput data streams, data curation and data storage / file system architectures, data security and governance, as well as inline inferencing and AI algorithms.

Responsibilities

• Develop a scalable data platform to support continuing increases in data volume and complexity.
• Develop data pipelines to ingest, process, and curate large volumes of data using Spark Structured Streaming or similar technologies.
• Support the infrastructure required for optimal ingestion, transformation, and delivery of data.
• Design and implement rigorous data validation, governance, and quality checks to uphold data integrity.
• Mentor & Support junior engineers by providing guidance, coaching and educational opportunities.

Requirements

• Bachelor's degree in computer science or a related field
• At least 5 years of hands on experience with Big Data Tools: Hadoop, Spark, Kafka, etc.
• Mastery with databases - Advanced SQL and NoSQL databases, including Postgres and Cassandra
• Proven cloud experience and strong familiarity with at least one cloud platform (Microsoft Azure - preferred, AWS, GCP).
• Experience in building distributed data processing micro-services and high-quality, scalable data products.
• Experience with stream-processing systems: Storm, Spark-Streaming, etc.
• 5 or more years with object-oriented/object function scripting languages: Python, Java, Scala, etc.
• Ability to tackle problems quickly and completely
• Ability to identify tasks which require automation and automate them
• A demonstrable understanding of networking/distributed computing environment concepts
• Ability to multi-task and stay organized in a dynamic work environment
• Good understanding and experience with CI/CD practices.

Nice-to-haves

• Data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
• Azure cloud services

Benefits

• Medical, dental, vision, Health Savings Account, Flexible Spending Accounts
• Retirement savings plan
• Sickness and accident benefits
• Life insurance
• Paid vacation & holidays
• Tuition assistance programs
• Employee assistance program
• GM vehicle discounts