Lead Data Engineer
Data Engineering
India
10+ Years
Full-time
Remote
Client Project Opportunity
Join Intelicore Systems and contribute to a high-impact enterprise data and artificial intelligence programme.
You will build the data foundation for revenue reconciliation, account prioritisation, agreement analysis and operational reporting. The solution integrates payment feeds, bank receipts, claims and document-extraction outputs using Microsoft Azure, Azure Databricks and Microsoft Fabric.
About the Role
We are looking for an experienced Lead Data Engineer to build and operate reliable enterprise data pipelines, lakehouse structures and curated datasets.
You will integrate payment feeds, bank receipts, claims and document-extraction outputs into standardized datasets that support reconciliation, predictive models, agreement analysis and operational reporting. You will also own production deployment, monitoring, recovery, performance and operational documentation for the data platform.
Key Responsibilities
- Integrate scheme payment feeds, scheduled files, bank receipts, claims and document-extraction outputs.
- Transform source data into reliable and standardized enterprise datasets.
- Preserve bank receipts as the authoritative source of cash received.
- Support the identification and investigation of unmatched financial transactions.
- Design and develop ingestion and transformation pipelines using Microsoft Azure, Azure Databricks and Microsoft Fabric.
- Build and maintain scalable lakehouse structures.
- Create reusable data models for reconciliation engines, account scoring, approved agreement terms and operational dashboards.
- Prepare reliable datasets for predictive models and downstream applications.
- Implement automated data validation and data-quality controls.
- Implement data lineage, access controls and appropriate data-governance practices.
- Build reliable incremental-data processing.
- Implement reconciliation and balancing controls.
- Own the production deployment of data pipelines and platform components.
- Monitor production pipelines and resolve failures.
- Design and maintain appropriate failure-recovery processes.
- Optimize pipeline reliability, scalability and performance.
- Create and maintain technical and operational documentation.
- Collaborate with machine learning and application engineering teams to provide dependable data interfaces.
Must-Have Skills & Experience
- 10+ years of relevant experience in data engineering, enterprise data integration and data-platform development.
- Strong hands-on experience with Azure Databricks.
- Strong hands-on experience with Microsoft Fabric.
- Advanced Python programming skills.
- Advanced Structured Query Language skills.
- Strong PySpark development experience.
- Experience designing and implementing Delta Lake solutions.
- Experience with Azure Data Factory.
- Strong data-modelling and lakehouse-design skills.
- Experience building enterprise ingestion and transformation pipelines.
- Experience implementing data-quality, validation and reconciliation controls.
- Understanding of data lineage, security and access-control requirements.
- Experience building incremental-data processing.
- Experience with automated build and deployment practices for data pipelines.
- Proven ownership of production data pipelines.
- Strong production troubleshooting and failure-recovery skills.
- Experience managing data-platform reliability and performance.
- Ability to collaborate effectively with machine learning and application-development teams.
Good to Have
- Experience working with financial payment, banking, claims or reconciliation data.
- Experience supporting machine learning and artificial intelligence workloads.
- Experience integrating document-extraction or document-processing outputs.
- Experience with Azure DevOps or GitHub Actions.
- Knowledge of Microsoft Purview or similar data-governance platforms.
- Experience implementing observability and cost optimization for Azure data platforms.
- Relevant Microsoft Azure, Databricks or Fabric certification.
Role Details
- Experience - 10+ Years
- Location - India
- Work Mode - Remote
- Employment - Full-time
- Joining Preference - Immediate Joining
- Domain - Enterprise Data Engineering / Azure Lakehouse and Analytics
Apply for this Job
* indicates a required field