Data and AI Knowledge Engineer - Associate

Position Summary

The Data and AI Knowledge Engineer will prepare, organize, govern, and operate the structured and unstructured information required by the firm’s AI applications. The role is accountable for creating reliable pipelines, retrieval structures, metadata, quality controls, and lineage so AI solutions operate from current, permission-aware, and authoritative information.

This is not primarily a reporting or data-analyst role. The successful candidate must be capable of engineering data and knowledge systems that support enterprise AI, automation, analytics, and traceable decision support.

Primary Responsibilities

Data and knowledge architecture

• Inventory and profile structured databases, SharePoint repositories, document libraries, APIs, Dataverse tables, and other approved data sources.
• Establish authoritative-source hierarchies and identify duplicate, conflicting, outdated, or incomplete content.
• Design data models, taxonomies, metadata, indexes, and knowledge structures.
• Work with practice SMEs to define content labels, business terminology, document types, effective dates, service categories, industries, and other contextual metadata.

Data pipelines and integration

• Develop ingestion, transformation, enrichment, synchronization, and archival pipelines.
• Build data-quality checks, reconciliation routines, and exception monitoring.
• Integrate data across Microsoft Fabric, Azure data services, Dataverse, SQL, SharePoint, and external APIs.
• Maintain lineage between source information, transformed content, retrieval indexes, and downstream AI responses.
• Establish refresh schedules and stale-content detection.

Retrieval engineering

• Configure Azure AI Search and other approved knowledge-retrieval services.
• Develop chunking, segmentation, embedding, metadata-filtering, hybrid-search, and reranking approaches.
• Test retrieval precision, completeness, ranking, citation traceability, and permission behavior.
• Optimize indexes for quality, speed, cost, and maintainability.
• Support AI engineers with representative evaluation and regression datasets.

Data governance and operations

• Implement technical controls for data classification, sensitivity, permission trimming, retention, deletion, and access.
• Support source-owner approvals and periodic content revalidation.
• Monitor data-pipeline health, index freshness, source availability, and quality exceptions.
• Develop Power BI reporting for data quality, solution usage, value, cost, and operational performance.
• Document schemas, metadata, lineage, data ownership, business rules, and known quality limitations.

Required Technical Skills

• Advanced SQL and relational data modeling.
• Python for data processing, automation, and quality validation.
• Microsoft Fabric, Azure Data Factory, or comparable pipeline technologies.
• Azure AI Search, vector search, semantic search, hybrid search, embeddings, metadata filtering, and reranking.
• Dataverse, SharePoint, Microsoft Graph, Azure Data Lake, and SQL platforms.
• APIs, file ingestion, ETL and ELT patterns, and event-based synchronization.
• Power BI semantic modeling and operational reporting.
• Data lineage, data quality, master data, taxonomy, metadata, retention, and security concepts.
• Row-level and object-level security, sensitivity handling, and access-control testing.

Preferred Experience

• 1-2 or more years in data engineering, information architecture, search engineering, analytics engineering, or related disciplines.
• Experience preparing unstructured content for AI or enterprise search.
• Experience supporting retrieval-augmented generation applications.
• Experience with Microsoft Fabric, Azure, SharePoint, Dataverse, and Power BI.
• Experience in regulated, confidential, or professional-services environments.
• Familiarity with document-management, records-management, and knowledge-management practices.

Success Measures

• Data-source availability and pipeline reliability.
• Retrieval precision and completeness.
• Index freshness and stale-content rate.
• Data-quality exception volume and resolution.
• Source-to-response traceability.
• Permission and access-control test performance.
• Reuse of common data models, taxonomies, and ingestion patterns.
• Reduction in duplicate or conflicting knowledge.
• Completeness of lineage, metadata, and ownership documentation.

Role Boundaries

This role ensures information is technically ready for AI use. Business and practice data owners remain accountable for determining authoritative content, approving permitted uses, resolving substantive conflicts, and maintaining the source information.

#WeaverIndia