Cloud Architect - Observability (Splunk)

Samtek Inc

85d•Remote

About The Position

We are seeking an experienced Splunk Cloud Engineer to join our team supporting federal agencies in optimizing security monitoring, log management, and observability solutions. In this role, you will design, implement, and maintain Splunk Cloud environments, including architecting multi-organization setups and leading the migration of a high-volume 20TB/day Splunk Enterprise cluster to Splunk Cloud. The ideal candidate will have deep expertise in Splunk administration, cloud infrastructure, and federal compliance standards, with the ability to collaborate across teams to deliver scalable, secure solutions. This position offers the opportunity to drive high-impact federal IT initiatives while working remotely with a supportive, innovative team.

Requirements

5+ years of experience in Splunk administration, with at least 2 years focused on Splunk Cloud environments.
Proven experience architecting multi-organization Splunk Cloud deployments, including tenant isolation and cross-org data sharing.
Hands-on experience migrating large-scale Splunk Enterprise clusters (e.g., 20TB/day) to Splunk Cloud, with expertise in data optimization and performance tuning.
Proficiency in Splunk Core, Splunk Cloud, and related tools (e.g., Splunk ES, ITSI, Phantom for SOAR).
Hands-on experience with data ingestion, search processing language (SPL), dashboard development, and machine learning toolkit (MLTK).
Strong knowledge of cloud platforms (AWS, Azure, or GCP) and integration with Splunk (e.g., AWS Lambda, Azure Event Hubs).
Experience with scripting languages (Python, Bash) and automation tools (Ansible, Terraform) for Splunk deployments.
Familiarity with federal IT security standards (e.g., FISMA, NIST, RMF) and log management in regulated environments.
Excellent problem-solving skills, with the ability to troubleshoot complex issues in distributed, high-volume systems.
Strong communication skills for technical and non-technical audiences.
Must have resided in the U.S. for at least 3 of the last 5 years.
Must be eligible for Public Trust clearance (ability to obtain and maintain).
Visa sponsorship is not available for this role.

Nice To Haves

Splunk certifications (e.g., Splunk Certified Architect, Splunk Cloud Certified Admin, Splunk Enterprise Security Certified Admin).
Experience supporting federal agencies (e.g., CMS, DoD, DHS) or healthcare environments with Splunk for SIEM and compliance.
Knowledge of SIEM integrations with tools like Microsoft Sentinel, ELK Stack, or ArcSight.
Bachelor's degree in Computer Science, Information Technology, or a related field.
Experience with containerization (Docker, Kubernetes) and microservices architectures in cloud environments.

Responsibilities

Design, deploy, and manage Splunk Cloud instances, including data onboarding from diverse sources (e.g., logs, metrics, network data) using forwarders, HTTP Event Collectors, and APIs.
Architect multi-organization, multi-CSP Splunk Cloud setups, configuring tenant isolation, shared services, and cross-org data access while maintaining security and performance.
Configure and optimize Splunk indexes, data models, and knowledge objects (e.g., searches, dashboards, reports) to support security operations, compliance reporting, and performance monitoring.
Implement advanced features such as Splunk Enterprise Security (ES), IT Service Intelligence (ITSI), and User Behavior Analytics (UBA) for threat detection and incident response.
Lead the migration of a 20TB/day Splunk Enterprise cluster to Splunk Cloud, including planning, data validation, and optimization of indexing and search performance.
Assess and map on-premises configurations (e.g., indexes, apps, lookups) to Splunk Cloud, addressing incompatibilities and ensuring minimal downtime.
Develop migration strategies for high-volume data ingestion, including compression, batch processing, and prioritization of critical data sources.
Integrate Splunk Cloud with cloud platforms (e.g., AWS, Azure) and on-premises systems, ensuring seamless data flow and compliance with federal security protocols.
Develop and maintain Splunk apps, add-ons, and custom scripts (e.g., using Python, Splunk SDK) to automate workflows, alerting, and remediation processes.
Collaborate with DevOps and security teams to build CI/CD pipelines for Splunk configurations and ensure high availability, scalability, and disaster recovery.
Enforce security best practices, including role-based access controls (RBAC), data encryption, and audit logging in Splunk Cloud environments, with specific focus on multi-org security models.
Conduct performance tuning, capacity planning, and troubleshooting to maintain 99.9%+ uptime and optimize resource utilization for high-volume data environments.
Support federal compliance requirements (e.g., FISMA, NIST 800-53, HIPAA) by generating reports, conducting audits, and implementing controls for sensitive data handling.
Work closely with stakeholders, including cybersecurity analysts, system architects, and agency leadership, to gather requirements and deliver tailored Splunk solutions.
Document configurations, migration plans, multi-org architectures, and best practices, while providing training and knowledge transfer to team members.
Monitor emerging Splunk features, cloud trends, and federal regulations to recommend improvements and enhancements.