Principal System Engineering
About The Position
Join AT&T and help shape the future of communications and technology that connect the world. We value innovators who seek to explore the unknown and challenge the status quo. Bring your bold ideas and fearless spirit to redefine connectivity and transform how people share stories and experiences. At AT&T, you won’t just imagine the future—you’ll build it. What you’ll do: As a Principal System Engineering, you will be responsible for driving the automation, reliability, and scalability of enterprise-grade storage infrastructure. This role sits at the intersection of traditional storage engineering and modern software-driven operations, with a strong emphasis on building and owning the automation that keeps production environments running at scale with zero-touch human intervention. The successful candidate will bring deep expertise across Fibre Channel SAN, multipath I/O, and NAS technologies, paired with hands-on experience administering enterprise storage arrays from vendors such as NetApp, Pure Storage, and HPE. Equally important is a strong engineering mindset — this role requires proficiency in scripting and automation (Python, Bash, PowerShell), infrastructure-as-code practices (Terraform, Ansible, GitOps), and the confidence to own the full lifecycle of the tooling and automation you deliver, from development through production support. You will be expected to stand behind your work, ensuring the solutions you build are resilient, observable, and maintainable. Solid foundation in the Linux block layer, virtualization platforms (KVM, CloudStack, or OpenStack), and modern monitoring stacks (Prometheus, Grafana) will be essential. The ideal candidate is someone who takes pride in engineering solutions that reduce toil, improve reliability, and operate with minimal intervention — and who steps up to diagnose and resolve issues when they arise. Working in a high-velocity environment where execution and communication matters. You must be equally effective collaborating with fellow engineers and operations teams in the trenches as you are communicating status, risk, and strategy to executive leadership. We value engineers who can translate complex technical challenges into clear, actionable narratives for any audience. Success in this role requires a collaborative mindset, a bias toward action, and the ability to thrive in a fast-moving team where shared ownership and continuous improvement are the norm.
Requirements
- 8+ years’ Linux and Windows infrastructure engineering experience with a primary focus on enterprise-scale storage in production environments
- Proven expertise with Linux DM-Multipath (configuration, failover behavior, troubleshooting path instability): Linux OS (RockyLinux, Ubuntu, RHEL, etc.), Linux block layer fundamentals: devices, partitions, LVM, filesystems (ext4/xfs), udev rules, etc., and Multipath configuration and troubleshooting: alua, etc.
- Experience supporting NAS (NFS) storage for virtualization or large-scale Linux environments (NFS v3 and v4)
- Hands-on experience with enterprise storage arrays including provisioning and performance troubleshooting: NetApp, Pure, HPE (Primera and 3PAR), Concepts: RAID, thin provisioning, overprovisioning, etc., Storage monitoring using vendor-based tooling and open-source solutions like Prometheus, Grafana
- Scripting/automation and software engineering: Python, Bash and PowerShell
- Strong Fibre Channel SAN experience including host-side troubleshooting (HBAs, discovery, LUN mapping, zoning/masking coordination): SCSI/iSCSI and NVMe
- Proven Infrastructure as Code and automation experience: Terraform, Git, Metal as a Service (MAAS)
- Virtualization and IaaS integrations: KVM
Nice To Haves
- Experience with distributed software storage like Ceph, OpenEBS
- Scripting/automation and software engineering: Java, Go, Rust, and C
- Proven Infrastructure as Code and automation experience: GitOps and Ansible
- Virtualization and IaaS integrations: Apache CloudStack/OpenStack and Kubernetes CSI
- 24x7 environments with incident response, RCA, and change control
Benefits
- Medical/Dental/Vision coverage
- 401(k) plan
- Tuition reimbursement program
- Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
- Paid Parental Leave
- Paid Caregiver Leave
- Additional sick leave beyond what state and local law require may be available but is unprotected
- Adoption Reimbursement
- Disability Benefits (short term and long term)
- Life and Accidental Death Insurance
- Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
- Employee Assistance Programs (EAP)
- Extensive employee wellness programs
- Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
