Backend Engineer, Otter POS - Los Angeles, CA

Otter•Los Angeles, CA

13h•Onsite

About The Position

We're looking for a Backend Engineer to help architect and scale Otter POS's real-time order management, payment processing, and multi-channel orchestration platforms. This is a high-impact role for someone who enjoys distributed systems, thrives on solving complex synchronization challenges, and gets excited about building performance-critical systems that push the boundaries of what's possible. Here's the kind of problems we solve: We built a custom CRDT (Conflict-Free Replicated Data Type) implementation that outperforms industry-standard libraries like Ditto, Automerge, and Y-CRDT by 90% in latency and uses 4x less memory (read our deep dive). Why? Because restaurant environments are hostile—devices drop from WiFi constantly, tablets overheat, someone spills liquid near a power supply—yet burger joints cannot stop selling burgers just because the internet goes down. We need to sync state across 10+ devices per restaurant without a leader, on low-end Android hardware, handling millions of orders daily. You'll work across backend services that support Android POS devices, web ordering surfaces, and mobile applications. You'll partner closely with Android engineers who deal with the realities of low-end tablets, product teams defining the next generation of restaurant workflows, data engineers building real-time analytics pipelines, and operations teams supporting tens of thousands of restaurants. You'll help define how we design, build, and scale backend systems that are both theoretically sound and practically deployable in the messiest real-world conditions.

Requirements

Strong proficiency in backend programming with Java and/or Kotlin (or willingness to learn if coming from similar JVM languages)
Experience with distributed databases such as CockroachDB, PostgreSQL, or similar SQL databases, with deep understanding of transactions, consistency models, and distributed systems challenges
Experience with event streaming platforms like Kafka, understanding of event-driven architectures, and familiarity with stream processing frameworks (Flink, Kafka Streams, etc.)
Understanding of cloud platforms like AWS, Google Cloud, or Azure, including container orchestration, service mesh, and cloud-native architectures
Deep knowledge of backend system design: RESTful APIs, gRPC, microservices, distributed systems patterns (eventual consistency, CQRS, saga patterns), concurrency, and fault tolerance
Experience with offline-first or local-first architectures is a plus—understanding CRDTs, sync protocols, or conflict resolution strategies
Hands-on experience with observability tools: metrics (Prometheus, Grafana), distributed tracing (Jaeger, Zipkin), logging platforms, and building production-grade monitoring
A high-performance, hands-on engineer who sets a high bar for code quality, reliability, and operational excellence—you understand production is where your work matters most
Strong collaboration skills: experience working closely with Android/mobile engineers, frontend teams, product managers, data scientists, and business stakeholders
Product mindset: passion for building systems that directly impact restaurant operations, staff efficiency, and customer experiences—you care about the "why" behind the code

Nice To Haves

Experience with Bazel build system for large monorepo codebases
Knowledge of payment processing systems, PCI compliance, and financial transaction processing
Experience with Android development or understanding mobile constraints and offline operation requirements
Background in restaurant technology, retail POS systems, or logistics/fulfillment platforms
Experience with real-time synchronization protocols and multi-device coordination
Familiarity with loyalty platforms, discount engines, or promotional systems

Responsibilities

Write high-quality, well-tested code in Java and Kotlin, set a strong bar for engineering excellence, and own the full software lifecycle from design to production monitoring
Lead backend initiatives, ensuring scalability, reliability, and low-latency performance across critical domains such as order orchestration, payment processing, menu management, and real-time device synchronization
Design and implement distributed systems using CockroachDB, Kafka Event Queues (KEQ), and Flink stream processing that handle millions of transactions with strong consistency guarantees
Build APIs and services that support offline-first operation using CRDT (Conflict-Free Replicated Data Types), enabling POS devices to function reliably without network connectivity
Drive development with a strong focus on reliability-by-design, observability, and testability, including comprehensive metrics, distributed tracing, and alerting
Design scalable databases and data models; write complex SQL queries for analytics and operational insights using Trino and Superset
Work with Protocol Buffers for inter-service communication and event schemas
Build and maintain integrations with external systems including payment processors (Mercury), kitchen display systems (KDS), third-party delivery platforms, and loyalty services
Collaborate on the architecture of event-driven systems that process order lifecycle events, payment transactions, and device synchronization events in real-time
Mentor and uplevel engineers through technical leadership, code reviews, design reviews, and hands-on collaboration