Note: The job is a remote job and is open to candidates in USA. INSPYR Solutions is a national expert in delivering flexible technology and talent solutions. They are seeking a Lead Data Engineer to architect and optimize data pipelines, maintain Lakehouse architecture, and enable machine-learning workflows while collaborating with various teams to deliver scalable data solutions.
Responsibilities
- Architect, build, and optimize distributed data pipelines using Apache Spark in a high‐volume, mission‐critical environment
- Design and maintain enterprise Lakehouse architecture with Delta Lake, ensuring ACID compliance, lineage, auditability, and data governance
- Develop automated ingestion frameworks (batch, streaming, and event‐driven) across multiple cloud services and integration points
- Enable machine‐learning workflows by preparing feature‐ready datasets and establishing reproducible ML deployment patterns
- Lead platform‐wide data quality, access control, and cataloging frameworks
- Implement advanced cost‐optimization, cluster tuning, and performance engineering strategies
- Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions
- Own production reliability, troubleshooting, and root‐cause analysis for data and ML pipelines
Skills
- 7+ years of experience in advanced data engineering with distributed compute technologies
- Expert-level Spark engineering (performance tuning, cluster configuration, partition strategies, optimization of large datasets)
- Hands-on experience with Lakehouse architectures including ACID transactions, schema evolution, and governance frameworks
- Deep proficiency in Python and SQL for large-scale data transformation
- Experience supporting machine-learning pipelines or model operationalization
- Proven experience architecting cloud-native data platforms (Azure, AWS, or GCP)
- Strong background integrating diverse, complex data sources at enterprise scale
- Demonstrated ability to own mission-critical production systems
- Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment
- Design and maintain enterprise Lakehouse architecture with Delta Lake, ensuring ACID compliance, lineage, auditability, and data governance
- Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points
- Enable machine-learning workflows by preparing feature-ready datasets and establishing reproducible ML deployment patterns
- Lead platform-wide data quality, access control, and cataloging frameworks
- Implement advanced cost-optimization, cluster tuning, and performance engineering strategies
- Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions
- Own production reliability, troubleshooting, and root-cause analysis for data and ML pipelines
- Experience with distributed streaming frameworks (Kafka, Event Hubs, or similar)
- Experience building or supporting ML platforms, feature stores, or experiment-tracking systems
- Background in data security, compliance controls, or audit-ready governance
- Experience automating data operations with CI/CD and infrastructure-as-code
- Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S
Benefits
- Comprehensive medical benefits
- Competitive pay
- 401(k) retirement plan
- 63;and much more!
Company Overview