Senior Data Engineer – Microsoft Fabric

Company: Finarb Analytics Consulting Pvt. Ltd.

Job Title: Senior Data Engineer – Microsoft Fabric

Location: Hyderabad or Kolkata

Experience: 6–10 Years

Employment Type: Full-Time

Number of Openings: 1

About FinarbFinarb Analytics Consulting is a data, analytics, and technology consulting firm helping organizations transform complex data into actionable business insights through advanced analytics, business intelligence, artificial intelligence, and modern data platforms.

Website: https://www.finarb.ai

About the RoleWe are seeking an experienced, hands-on Senior Data Engineer specializing in Microsoft Fabric to lead the development, migration, optimization, and maintenance of enterprise data engineering solutions.

The ideal candidate will possess strong expertise in Microsoft Fabric, PySpark, SQL, data warehousing, ETL/ELT pipelines, and cloud-based data engineering.

This role will play a critical part in migrating existing SQL Server and Azure Data Factory workloads to Microsoft Fabric while ensuring data accuracy, performance, reliability, and minimal disruption to business operations.

The successful candidate will work closely with the Principal Fabric Data Architect, data quality engineers, BI developers, and business stakeholders to implement scalable Fabric solutions and establish robust production engineering practices.

We are looking for someone who can independently deliver complex engineering solutions, troubleshoot technical challenges, and take ownership of critical data pipelines throughout migration and ongoing production operations.

Key Responsibilities1. Microsoft Fabric Data Engineering

• Design, develop, and maintain scalable data engineering solutions using Microsoft Fabric.
• Build and optimize Fabric Lakehouse and Warehouse solutions.
• Develop data ingestion, transformation, and orchestration pipelines using Fabric Data Factory.
• Implement complex data transformations using PySpark notebooks and Spark SQL.
• Work with OneLake, Delta Lake tables, and medallion architecture (Bronze, Silver, Gold).
• Develop reusable data engineering frameworks and components.
• Implement incremental loading, change data capture (CDC), and efficient data processing patterns.
• Ensure data engineering solutions meet scalability, reliability, and performance requirements.2. Data Migration and Modernization
• Execute migration of existing SQL Server, Azure Data Factory, and legacy data workloads to Microsoft Fabric.
• Analyze existing ETL pipelines, stored procedures, dependencies, and data transformations.
• Translate legacy transformation logic into Fabric-compatible implementations.
• Develop migration scripts, notebooks, pipelines, and validation processes.
• Implement full and incremental data loading strategies.
• Perform source-to-target reconciliation and ensure data consistency during migration.
• Support parallel runs, cutover planning, and post-migration stabilization.
• Identify and resolve migration-related performance, compatibility, and integration issues.3. PySpark and Advanced Data Transformations
• Develop, test, and optimize PySpark notebooks for large-scale data processing.
• Implement complex transformation logic using Spark DataFrames and Spark SQL.
• Optimize Spark jobs through partitioning, caching, efficient joins, and resource management.
• Handle schema evolution, data skew, incremental processing, and large datasets.
• Build modular, reusable, and maintainable notebook-based processing frameworks.
• Implement logging, error handling, and monitoring for notebook execution.4. SQL and Data Warehousing
• Develop and optimize complex SQL queries and data transformations.
• Design and implement efficient data structures in Fabric Warehouse and Lakehouse.
• Work with dimensional data models, fact tables, dimension tables, and star schemas.
• Implement data transformation logic supporting downstream Power BI reporting.
• Optimize query performance and data processing workloads.
• Ensure consistency of business rules and transformations across data layers.5. Data Quality and Reconciliation
• Implement automated data validation and reconciliation processes.
• Validate data completeness, accuracy, consistency, and integrity across source and target systems.
• Develop reusable reconciliation scripts using SQL and Python/PySpark.
• Collaborate with data quality engineers to investigate discrepancies and resolve defects.
• Support migration testing, regression testing, and production validation.
• Ensure traceability and auditability of critical data transformations.6. Production Engineering and Performance Optimization
• Monitor, maintain, and troubleshoot production Fabric pipelines, notebooks, and data workloads.
• Investigate pipeline failures, Spark execution errors, and performance bottlenecks.
• Optimize Spark processing, SQL workloads, and Fabric capacity utilization.
• Implement appropriate logging, alerting, retry mechanisms, and error handling.
• Perform root cause analysis and implement permanent corrective actions.
• Ensure data pipelines meet agreed performance and reliability requirements.
• Support production deployments, releases, and incident resolution.7. Technical Collaboration and Engineering Standards
• Collaborate with the Principal Fabric Data Architect to implement architectural designs and engineering standards.
• Guide junior data engineers on Fabric development practices and troubleshooting.
• Participate in technical design discussions and code reviews.
• Establish reusable coding patterns, documentation, and testing practices.
• Support Git integration, CI/CD pipelines, and controlled deployments.
• Work closely with Power BI developers to ensure data models and pipelines support reporting requirements.
• Identify opportunities to improve engineering efficiency, automation, and operational reliability.Required Technical SkillsTechnology / SkillExpected ProficiencyMicrosoft FabricStrong hands-on production experienceFabric Lakehouse & WarehouseAdvanced development experienceFabric Data FactoryPipeline development and orchestrationPySparkAdvanced transformations and optimizationSQL / T-SQL / Spark SQLAdvanced querying and transformationsOneLake & Delta LakeStrong practical understandingETL / ELTComplex pipeline design and developmentAzure Data FactoryDevelopment and migration experienceSQL ServerStrong understanding of legacy workloadsData WarehousingDimensional modeling and medallion architectureData MigrationSource-to-target migration and reconciliationPerformance OptimizationSpark, SQL, pipeline, and capacity tuningProduction SupportAdvanced troubleshooting and root cause analysis

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• 6–10 years of professional experience in data engineering or data platform development.
• Strong hands-on experience developing solutions using Microsoft Fabric.
• Proven experience with PySpark and distributed data processing.
• Advanced SQL programming and data transformation skills.
• Experience designing and implementing Lakehouse and Warehouse solutions.
• Strong understanding of Delta Lake, medallion architecture, and modern data engineering patterns.
• Experience building production-grade ETL/ELT pipelines.
• Practical experience with SQL Server and Azure Data Factory.
• Experience migrating or modernizing enterprise data workloads.
• Strong understanding of data reconciliation, validation, and migration testing.
• Demonstrated ability to independently troubleshoot complex production issues.
• Experience collaborating with architects, engineers, and cross-functional technical teams.Preferred / Good-to-Have Skills
• Experience with Azure Data Lake Storage Gen2 and other Azure data services.
• Experience with Azure Databricks or Apache Spark environments.
• Familiarity with Power BI semantic models and Direct Lake integration.
• Knowledge of Fabric capacity management and workload optimization.
• Experience implementing CI/CD using Azure DevOps or GitHub.
• Familiarity with data governance, security, and access control in Fabric.
• Experience with Microsoft Purview and data lineage.
• Knowledge of data observability and automated data quality frameworks.
• Experience implementing CDC, streaming ingestion, or near-real-time processing.
• Microsoft Certified: Fabric Data Engineer Associate (DP-700) certification is an advantage.Key Competencies
• Strong technical ownership and independent execution.
• Excellent analytical and complex problem-solving capabilities.
• Ability to troubleshoot performance and reliability issues across multiple data layers.
• Strong understanding of enterprise data engineering practices.
• Attention to data quality, accuracy, and maintainability.
• Ability to mentor and support other engineers.
• Effective communication with technical and business stakeholders.
• Ability to balance migration delivery with production stability.What Success Looks LikeThe successful candidate will be expected to:
• Deliver reliable, scalable, and maintainable Microsoft Fabric data engineering solutions.
• Successfully migrate complex SQL Server and Azure Data Factory workloads into Fabric.
• Develop high-performance PySpark notebooks and Fabric data pipelines.
• Ensure accurate reconciliation between legacy and migrated data environments.
• Maintain stable production operations during and after migration.
• Optimize processing performance, reliability, and Fabric resource utilization.
• Establish reusable engineering practices and support the development of the broader data engineering team.
• Contribute to a successful transition toward a modern Microsoft Fabric data platform.Why Join Finarb?
• Work on enterprise-scale data engineering and modernization initiatives.
• Gain hands-on experience with Microsoft Fabric and advanced cloud data technologies.
• Work alongside experienced data architects and engineering professionals.
• Participate in complex migration and data platform transformation programs.
• Develop expertise in modern Lakehouse architecture, PySpark, and cloud-native engineering.
• Contribute to technically challenging projects with significant business impact.