Job Title: Senior Data Engineer – Apache NiFi
Location: Remote - Candidates can be based on a 100% Remote basis in either the USA or Canada
Duration: 3 months (possible extension)
Rate: $70-80 USD/hour
Please Note:
Apache NiFi experience is Essential for the position.
The ideal candidate will have their own corporated company.
US Citizens or Canadian citizen/Permanent residency required. No sponsorship or w2 avaialble for particular position.
Purpose:
We are seeking a Senior Data Engineer with strong Apache NiFi expertise to design, build, and maintain scalable, reusable data pipelines.
The role involves integrating multiple source systems, transforming data to predefined formats, and enabling downstream analytics, including Power BI.
Key Responsibilities:
Location: Remote - Candidates can be based on a 100% Remote basis in either the USA or Canada
Duration: 3 months (possible extension)
Rate: $70-80 USD/hour
Please Note:
Apache NiFi experience is Essential for the position.
The ideal candidate will have their own corporated company.
US Citizens or Canadian citizen/Permanent residency required. No sponsorship or w2 avaialble for particular position.
Purpose:
We are seeking a Senior Data Engineer with strong Apache NiFi expertise to design, build, and maintain scalable, reusable data pipelines.
The role involves integrating multiple source systems, transforming data to predefined formats, and enabling downstream analytics, including Power BI.
Key Responsibilities:
- Build and maintain NiFi flows for ingestion, transformation, and loading across various sources.
- Implement parameterized, version-controlled pipelines using NiFi Registry.
- Design and optimize data models and SQL queries in Azure SQL for analytics consumption.
- Ensure data quality, error handling, and auditability across pipelines.
- Containerize and deploy NiFi using Docker/Kubernetes; automate deployments with CI/CD pipelines.
- Secure credentials and flows using Azure Key Vault and TLS.
- Collaborate with functional teams on mapping, cutover planning, and testing.
- Strong experience building robust flows using processors such as Fetch/PutSFTP, QueryDatabaseTable, PutSQL, ConvertRecord, UpdateRecord, ExecuteScript, RouteOnAttribute, MergeRecord, HandleHttpRequest/Response.
- Record-oriented processing using Avro, JSON, CSV; configuring Controller Services like RecordReader/Writer, DBCPConnectionPool, SSLContextService.
- Expertise with Parameter Contexts, Variable Registry, FlowFile attributes, back pressure, prioritizers, and retry patterns.
- NiFi Registry for flow versioning, promoting flows between environments, and flow comparison.
- Implementing error handling & dead-letter queue patterns (failure, retry, provenance-based replay).
- Performance tuning (concurrent tasks, run schedule, batch sizes, repository tuning).
- Groovy, Python, or Java for ExecuteScript and custom processors.
- REST API integration, JSON/XML transformations.
- Strong SQL skills, including query optimization and bulk load strategies.
- Experience designing for Power BI consumption (star schema, fact/dimension design, incremental refresh).
- Docker for local/dev environments (including mounting JDBC drivers, persisting repositories/logs).
- Docker Compose for NiFi + Registry + dependencies orchestration.
- Experience integrating SAP ECC or S/4HANA sources.
- Familiarity with SAP connectors in NiFi, IDoc/BAPI structures, and authentication mechanisms (Basic, OAuth, SAP-specific).
- Minimum 5+ years of experience in Data Engineering with hands-on NiFi.
- Proven experience with Azure cloud services (ADF, Data Lake, SQL).
- Experience with CI/CD pipelines, Docker, and containerized deployments.
- Strong collaboration and communication skills to work with cross-functional teams.
