Case Studies

Challenge

L1 and L2 operations support staff were spending significant time querying multiple disconnected systems and software tools to diagnose and resolve operational issues. The manual process of cross-referencing logs, monitoring dashboards, and system data slowed incident response and created a steep learning curve for new team members.

What IData Did

IData designed and built OpsChat — an AI-powered conversational interface that allows operations staff to ask natural language questions about operational systems instead of manually querying multiple tools. The platform ingests operational data into a RAG vector database at over 140,000 log messages per second, powered by a multi-server GPU server farm. Support staff can now ask questions like "What's causing the latency spike on the payment service?" and get instant, context-rich answers grounded in real-time operational data.

Results

• L1 and L2 support staff resolve issues faster by querying a single conversational interface instead of multiple systems
• Ingestion pipeline processes 140,000+ log messages per second into the RAG vector database
• Multi-server GPU farm ensures low-latency responses even under heavy operational load
• Reduced time-to-diagnosis for operational incidents
• Accelerated onboarding for new support staff — institutional knowledge is now queryable

Background

IData founded Datris.ai to solve a problem we kept seeing across every client engagement: enterprise data engineering is too complex, too fragile, and too dependent on specialized skills. Organizations spend months building and maintaining data pipelines that break when requirements change — and none of them were designed for the AI-native world we're entering.

What We Built

Datris is an open-source, AI agent-native data platform that handles the complete data lifecycle — ingestion, validation, transformation, storage, and retrieval — all driven by configuration and natural language instead of code. The platform treats AI agents as first-class pipeline operators via a built-in Model Context Protocol (MCP) server, allowing Claude, Cursor, and other AI agents to register datasets, trigger jobs, profile data, and query databases through conversation.

The platform replaces traditional hand-coded data quality rules with natural language AI rules — teams describe validation logic in plain English and Datris enforces it. Schema generation, data profiling, row-level transformations, and failure root cause analysis are all AI-powered. A complete RAG pipeline handles document extraction (PDF, Word, PowerPoint, Excel, HTML, email), chunking, embedding, and storage across five vector databases.

Architecture

Datris is entirely self-hosted with no vendor lock-in, built on proven open-source infrastructure: MinIO for S3-compatible object storage, MongoDB for metadata, Apache Kafka for streaming, HashiCorp Vault for secrets management, and Apache Spark for columnar file processing. It supports Anthropic Claude, OpenAI, and local Ollama models — swappable without configuration changes. Ingestion spans file uploads, bucket events, database polling (PostgreSQL, MySQL, MSSQL), and Kafka topics, with parallel output to Parquet, ORC, PostgreSQL, MongoDB, Kafka, REST endpoints, and vector databases including Qdrant, Weaviate, Milvus, Chroma, and pgvector.

Results

• Open-sourced on GitHub — community-driven with full documentation
• AI agents can operate data pipelines end-to-end via MCP, eliminating the need for specialized data engineering for many use cases
• Natural language data quality rules replace hand-coded validation logic
• Supports 12+ input formats and 7+ output destinations in a single pipeline
• Complete RAG pipeline from raw documents to vector search — out of the box
• Zero vendor lock-in — runs on any cloud, on-premise, or locally via Docker

Client

Large foreign pension fund managing $200B+ in public equities, private equities, real estate, infrastructure, and fixed income.

Challenge

The fund's data was scattered across dozens of on-premise databases and siloed teams. Each investment group had built independent, fragmented solutions. There was no enterprise visibility into available data, no single source of truth, and no foundation to support the analytics-driven investment strategies the fund wanted to pursue.

What IData Did

IData was engaged to architect and build a complete cloud-based data ecosystem on AWS. This included designing the multi-account security architecture, network infrastructure (Transit VPC, PrivateLink), infrastructure-as-code deployment (Terraform), serverless compute and storage, a full ETL ingestion pipeline, an enterprise data lake, a data catalog for discovery and governance, and API-based access for downstream analytics and applications. A sandbox environment was also created to support experimentation with new datasets and technologies.

Results

• Investment teams gained access to hundreds of datasets through a searchable data catalog
• Analysts could pull data directly via API into their own environments (Python, Scala, Spark)
• New software features deployed in hours or days instead of months via CI/CD
• Eliminated redundant data reconciliation across teams
• Created the data foundation required for AI/ML-driven investment analytics
• Sandbox environment enabled rapid experimentation with new data sources

Client

Large U.S. government-sponsored enterprise (GSE) in the mortgage industry.

Challenge

The GSE held vast amounts of mortgage, investment, and personal data spread across numerous legacy data warehouses and Hadoop environments. The firm maintained 40–50 staff in duplicated teams managing each siloed environment — teams with identical skill sets replicated many times over. Costs were high, agility was low, and there was no path to leverage modern analytics or AI.

What IData Did

IData led the migration and centralization of the GSE's data into AWS S3, integrating modern cloud technologies including Snowflake and MongoDB. The engagement involved consolidating decades of fragmented data infrastructure into a unified, cost-efficient cloud platform with modern governance and access patterns.

Results

• Consolidated siloed data warehouses into a single cloud-based data platform
• Dramatically reduced infrastructure costs (object storage vs. legacy warehouse licensing)
• Freed skilled personnel from maintenance roles to work on higher-value projects
• Now in its fourth year of implementation and expansion
• Created the unified data layer needed to support future AI/ML initiatives