Title: Data Engineer/Architect - Cloudera NiFi
Location: Remote
Duration: 6 Months Contract
Key Responsibilities
NiFi Platform Assessment & Optimization
• Review existing Cloudera NiFi flows and architectures for efficiency, scalability, and maintainability.
• Assess current use cases to ensure NiFi is being applied appropriately and effectively.
• Identify opportunities to simplify flow designs, reduce operational overhead, and improve performance.
• Provide recommendations on best practices, standards, and reusable patterns.
Data Integration & Pipeline Support
• Support and enhance ETL and ELT pipelines built on NiFi.
• Integrate data from diverse sources including databases, APIs, filesystems, streaming platforms, and cloud storage.
• Troubleshoot data flow failures, latency issues, and throughput constraints.
• Implement robust error handling, monitoring, and recovery mechanisms.
Python & Custom Development
• Use Python for transformations, scripting, automation, and custom processor logic (where applicable).
• Support integration between NiFi and external systems using custom scripts or utilities.
Governance, Security & Operations
• Apply best practices for secure data movement, authentication, authorization, and encryption.
• Assist with flow versioning, deployment processes, and operational runbooks.
• Collaborate with internal teams to ensure platform changes align with operational and compliance requirements.
Advisory & Collaboration
• Act as a technical advisor to internal data and platform teams.
• Clearly document findings, recommendations, and implemented improvements.
• Provide knowledge transfer and guidance to ensure long‑term sustainability.
Required Experience & Skills
• Proven, hands‑on experience with Cloudera NiFi in production environments.
• Strong knowledge of ETL/ELT concepts and modern data pipeline architectures.
• Proficiency in Python for data processing, scripting, and automation.
• Experience working with Linux‑based systems and distributed environments.
• Ability to analyze existing implementations and deliver actionable optimization recommendations.
Preferred Experience
• Experience within the Cloudera ecosystem (e.g., Kafka, HDFS, Hive, Iceberg).
• Familiarity with CI/CD practices for data pipelines.
• Experience with monitoring and tuning distributed data platforms.
• Exposure to cloud and hybrid data architectures.
Engagement Expectations
• Deliverables‑focused, hands‑on consulting role.
• Ability to work independently while collaborating with internal stakeholders.
• Clear written and verbal communication of technical findings and recommendations.