AI Data Analyst
About The Position
The Research Data Science team is responsible for pushing the boundaries of how Spring Venture Group utilizes unstructured data. While traditional analytics focuses on what happened, our team focuses on why it happened, using Large Language Models (LLMs) to extract structured insights from conversation transcripts. You will play a pivotal role in adapting the organization to the rapidly evolving future of AI, helping us stay ahead of the curve in a landscape that changes daily. As a Research Data Analyst, you will act as the bridge between human intuition and machine precision. You will not simply be "writing prompts"; you will be engineering structured data extraction pipelines and performing rigorous statistical analysis on the results. You will translate abstract business concepts—like customer sentiment or negotiation friction—into rigorous, testable prompt definitions, and then analyze how those new metrics correlate with downstream business outcomes like revenue and retention. This is a role for a scientist at heart. You will own the lifecycle of conversational metrics: from Discovery (finding the signal), to Definition (codifying the prompt), to Validation (proving it works via statistics), and finally to Experimentation (using the metric to judge AB tests). You should be passionate about finding the truth in data and have the technical discipline to verify that truth. 3 months into the job, you will: Understand the "Shared Ideas" of the business: Learn the nuances of our sales conversations and how different departments define success. Master our existing Prompt Library: specific syntax and schema enforcement Audit the Machine: Execute stratified sampling strategies to manually validate model outputs, calculating precision and recall to ensure our metrics are trustworthy. Analyze the Impact: Visualize LLM-derived data to explain trends in conversation quality and identify the statistical drivers of performance to stakeholders. 6 months into the job, you will: Lead Metric Discovery: Use LLMs to cluster topics and propose new schemas for unstructured data features. Drive Experimentation: Design and measure AB experiments. You will be responsible for verifying that the metrics used to judge "winners" are semantically aligned with the business goals. Strategic Analysis: Suggest new variants for treatments based on deep-dive analysis of conversation logs, connecting qualitative insights to quantitative financial results. Guard the Truth: Act as the gatekeeper for "Prompt Definitions," ensuring that we do not deploy metrics that rely on hallucination or weak signals.
Requirements
- Understanding of how Large Language Models work (context windows, temperature, hallucination risks) and AI.
- Fluency in at least one program language, preferably Python.
- Understanding of A/B testing frameworks, statistical significance, and metric definition.
- Strong knowledge of SQL and Relational Database systems.
- Experience with Snowflake
- A strong understanding of programming and a willingness to learn even more.
Responsibilities
- Designing or writing prompts to support specific AI outcomes
- Analyzing AI performance and making process and algorithm updates for performance improvement
- Continued research and education on the ever-changing AI field to ensure updated best practices
- Go beyond assumptions to scientifically validate prompt performance
- Understand assumptions and their pitfalls; challenge them when appropriate
- Apply sound logic and mathematics to database queries and reports
- Create simple and elegant solutions to seemingly complex problems
- Maintain and update Streamlit/SQL reports as necessary
- Perform other duties and responsibilities as assigned
Benefits
- Competitive compensation
- Medical, Dental and Vision benefits after a short waiting period
- 401k matching program
- Generous paid time off program with an additional company break during the holidays
- Annual Volunteer Time Off (VTO) and a donation matching program
- Life Insurance
- Employee Assistance Program (health & well-being on and off the job)
- Rewards and Recognition
- Maternity and Parental Leave
- Training program and ongoing support throughout your entire Spring Venture Group career
- Diverse, inclusive & welcoming culture
- Optional enrollment options include HSA/FSA, AD&D, Spousal/Dependent life insurance, Short Term/Long Term Disability, Travel Assist, and Legal plans
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Entry Level
