Data & AI Architect / Lead Data Engineer

Gen II Fund Services•New York, NY

7d•$220,000 - $260,000•Hybrid

About The Position

Lead in-house/ partner technical teams to design and build the technical foundation that transforms fragmented fund data into a coherent, scalable, and secure data platform. This hybrid role combines architectural vision with hands-on technical leadership, integrating AI/ML capabilities to deliver the data proposition and associated roadmap.

Requirements

8+ years data architecture and engineering experience in financial services
Bachelor’s Degree in Computer Science, Software Engineering or related filed.
Proven experience designing and implementing enterprise data platforms from scratch
Strong background in fund administration, asset management, or alternative investments data
Hands-on experience with modern data platforms (Snowflake, Databricks, Azure Synapse)
Experience leading technical teams and working with partners
Expert-level SQL and strong programming skills in Python (required), with knowledge of Scala/Java (preferred)
Deep expertise in data modelling: dimensional modelling, data vault, graph databases
Hands-on with data orchestration tools and transformation frameworks
Strong API design experience and microservices architecture
Cloud platform expertise, particularly Azure
Experience with AI/ML frameworks and integration (LLMs, document processing, predictive models)
DevOps practices
Deep understanding of fund administration data flows: NAV, capital calls, distributions, waterfall calculations
Familiarity with fund accounting systems and their data structures
Knowledge of regulatory reporting requirements and data governance in financial services
Understanding of private equity fund structures, investment strategies, and performance metrics

Responsibilities

Design and build enterprise data model for fund data across all sources to support the data proposition, leveraging existing solutions where appropriate
Define integration architecture connecting source systems necessary for the data proposition
Create technical roadmap aligned with proposition roadmap & strategy
Evaluate and select technology stack (data warehouse, ETL tools, API gateway, analytics platforms)
Design cloud architecture (Azure preferred) optimized for performance, cost, and scalability
Establish architectural standards, design patterns, and technical documentation
Design and build AI/ML integration points leveraging Gen II's existing AI capabilities and solutions
Architect solutions for unstructured data extraction from PDFs, LPA documents, and fund agreements
Develop intelligent data enrichment and validation capabilities using AI/ML models
Explore advanced analytics use cases: predictive modelling, anomaly detection, natural language processing
Partner with Virtusa and other technology vendors on multi-agent AI architecture development
Lead design and development of production data pipelines
Build and maintain data APIs for product consumption with proper versioning and documentation
Implement data transformation layer with business rules engine for calculation and enrichment logic
Optimize pipeline performance and cost efficiency
Establish monitoring, logging, and alerting infrastructure for proactive issue detection
Implement CI/CD practices for automated testing and deployment of data pipelines
Establish data governance framework including lineage, metadata management, and quality standards
Design security and access control framework ensuring data privacy and compliance (GDPR, SOC 2)
Define data retention, archival, and disaster recovery policies
Ensure compliance with regulatory requirements and industry best practices
Collaborate with Data Quality Lead on monitoring and validation frameworks
Lead in house / partner team of data engineers on technical excellence and best practices
Collaborate with Data Product Manager on feature requirements and technical feasibility
Partner with Gen II's existing technology team and external vendors (Virtusa) on implementation
Participate in agile ceremonies and provide technical guidance to cross-functional teams
Foster engineering culture emphasizing code quality, automation, and continuous learning