From upkeep PDFs to actionable insights with AI brokers
Operations and upkeep suppliers for photo voltaic and wind crops usually ship studies as PDFs, with key info unfold throughout free textual content, tables and pictures. This format is accessible however not scalable: groups should manually learn every doc to know faults, developments or recurring points, making cross‑plant comparisons sluggish and inconsistent because the variety of belongings grows.
Plenitude and Databricks constructed an agent‑based mostly system that converts these PDF upkeep studies into structured knowledge. The core thought is easy: remodel paperwork into knowledge, then use an AI agent to derive actionable insights from that knowledge. Customers can now ask questions in pure language, analyze developments over time, examine crops and export structured outputs, as a substitute of navigating studies one after the other.
Agent‑based mostly structure for PDF to knowledge analytics
The answer begins with event-driven ingestion of plant-level PDF studies. Every new report triggers a Databricks Job that parses the doc and applies LLM‑based mostly extraction. Extracted components are serialized as JSON and saved in Delta Lake, which maintains full model historical past for audit and replay.
To deal with the basic downside that upkeep info lives nearly solely in unstructured PDFs, Plenitude makes use of Databricks Doc Intelligence AI Capabilities — particularly ai_parse_document, to extract a number of sorts of components from every web page, together with textual content blocks, tables, figures and metadata. Every ingredient is enriched with attributes corresponding to plant, reporting interval, web page quantity and content material sort, and each document maintains a direct hyperlink again to the unique report for traceability.
This construction unlocks highly effective capabilities:
- Filtering by time, class and geography.
- Figuring out content material varieties and utilizing spatial coordinates.
- Tracing each perception again to the unique PDF.
- Integrating with BI instruments and digital brokers with out altering the underlying paperwork.
As an alternative of static information, upkeep studies develop into a persistent knowledge layer prepared for superior analytics and agent reasoning.
Knowledge processing on Databricks: from PDF to Delta Lake
The structure is organized into three major layers: ingestion and parsing, knowledge structuring and agent-based interplay.
Step 1: parsing
Utilizing ai_parse_document, the pipeline extracts textual content, tables and metadata from every web page and serializes them as structured JSON objects. Even advanced tables are captured with full context, together with their location on the web page and HTML illustration.
Step 2: normalization and storage
For each web page (page_id) and object (id), the system creates one row in a Delta Lake desk. Every row comprises:
- The extracted JSON content material.
- Web page and object identifiers.
- Coordinates (coords) representing the bounding field on the web page.
- Content material sort (for instance, textual content or desk).
- Excessive-value metadata corresponding to month, yr, file identify, class and nation.
This normalized mannequin turns PDFs right into a unified, queryable dataset that’s clear and simple to affix with different sources, whereas preserving full traceability again to the unique paperwork.
Step 3: Genie area and Agent mode
On high of this curated knowledge layer, Plenitude builds a devoted Genie area after which leverages Genie’s Agent mode to carry out Deep Analysis on the info. Genie makes use of the structured Delta Lake tables as its major context and lets customers work together with upkeep knowledge utilizing pure language.
When a consumer asks a query, Genie:
- Makes use of semantic metadata in Unity Catalog to establish obtainable tables and columns.
- Leverages detailed column descriptions, a curated data retailer and SQL samples to information question era.
- Generates and executes SQL in opposition to the structured layer.
- Returns solutions, visualizations and optionally exportable outcomes.
This design allows Genie to know each the enterprise semantics of upkeep knowledge and its underlying construction, leading to correct, context-aware solutions.
Why metadata and directions matter for Genie
To acquire dependable outcomes from advanced PDF-derived datasets, context alone isn’t ample. Plenitude discovered that two design patterns are crucial: wealthy metadata and specific directions for the Genie area.
Metadata as a contract with the agent
Effectively-defined desk and column descriptions inform Genie what every subject means and the way it needs to be used. For instance, page_id identifies the supply web page within the unique report, sort signifies whether or not the ingredient is textual content or a desk, coords encodes the spatial location, and content material comprises the extracted textual content or desk illustration. This metadata turns uncooked JSON into comprehensible data that Genie can motive over.
Basic directions as operational grounding
When knowledge is fragmented or spans a number of pages, domain-specific directions added to the Genie area’s native data retailer develop into important. Plenitude encodes guidelines for dealing with multipage tables, ignoring HTML artifacts, excluding header rows and making use of plant-specific filters.
A sensible instance: even with full metadata, Genie may compute an incorrect quarterly whole if it sums YTD columns or ignores lacking months. By including clear directions corresponding to “solely use month-level columns, by no means YTD fields” and “validate that every one required months are current earlier than summing,” the workforce gives Genie with operational guardrails that guarantee constant outcomes.
These Genie space-specific directions, mixed with metadata from Unity Catalog, assist Genie apply the fitting logic to interpret knowledge appropriately.
Utilizing Genie and Agent Bricks for scalable agent workflows
Whereas Genie gives a strong analysis agent expertise on high of the structured upkeep layer, Plenitude additionally wants repeatable workflows and orchestration to help a rising set of use instances. Agent Bricks is the subsequent step in that evolution.
With Agent Bricks, Plenitude can transfer from “LLM plus immediate” patterns to agentic workflows that execute sequences of actions on behalf of upkeep analysts and engineers. The identical curated Delta Lake tables, metadata and directions that energy Genie might be reused by Supervisor-style brokers constructed with Agent Bricks to:
- Decompose advanced questions into smaller analytical duties.
- Name the Genie instrument flows to generate and execute SQL.
- Set off downstream actions, corresponding to report era or alert creation.
What used to require guide wiring of prompts, instruments and validation logic can now be centralized in Agent Bricks, on the identical Databricks Platform that manages the info.
Optimizing efficiency with automated liquid clustering
As a result of agent-driven queries are exploratory and dynamic, conventional Z-ORDER-based tuning isn’t at all times preferrred. Plenitude noticed that entry patterns evolve as new studies, customers and questions seem, which makes guide clustering exhausting to keep up.
Computerized liquid clustering, in contrast, learns how tables are literally used and adapts the structure accordingly. This reduces the necessity for upfront index design and ongoing tuning, which is particularly vital through the proof-of-concept and early go-live phases. On this context, auto clustering is the popular alternative for agent and LLM-driven workloads on Delta tables.
Securing knowledge entry for Genie Rooms
Upkeep knowledge usually has nation or region-specific entry necessities. To implement these guidelines constantly, Plenitude makes use of row-level safety together with Unity Catalog and tables.
A Unity Catalog perform determines which international locations the present consumer can entry and returns an inventory or the key phrase ALL if they’ve full visibility. A desk then filters rows based mostly on that perform, so every consumer solely sees knowledge for his or her approved international locations.
When customers work together by way of the Genie Room, all queries run on the filtered desk, so row-level safety is utilized mechanically. This implies customers can ask questions in pure language, however solely obtain outcomes from the info they’re allowed to see. The identical dataset powers Genie, brokers and BI instruments, whereas visibility adjusts per consumer.
Future enhancements: towards predictive upkeep
As a result of upkeep studies include open incidents and fault particulars, the structured knowledge mannequin is a powerful basis for predictive upkeep. Inverters are an excellent instance: failures can result in the lack of a number of megawatts-hours per unit, and recurring points usually seem first in upkeep notes.
By analyzing fault patterns over time, Plenitude can:
- Determine potential recording points.
- Detect early warning indicators.
- Prioritize crops that want deeper investigation.
- Feed predictive fashions with higher-quality incident histories.
The agent-based system turns these indicators into accessible analytics, developments and visualizations so groups can anticipate points somewhat than merely react to them.
Key advantages and capabilities
Within the earlier strategy, evaluation was restricted to studying studies individually, which made it exhausting to construct historic developments, examine crops or generate structured outputs. Creating charts, exporting outcomes or combining insights from a number of studies was guide at finest and sometimes not possible.
With the Genie Agent Mode on Databricks and an agent-friendly knowledge mannequin, Plenitude can:
- Discover upkeep knowledge throughout time and throughout crops.
- Generate visualizations and export outcomes, together with PDF outputs.
- Detect early indicators and recurring patterns.
- Scale evaluation with out scaling guide effort.
By combining structured knowledge, enterprise metadata and AI reasoning, the system generates analytics, developments and visualizations that help early detection and anticipation of issues, not simply retrospective reporting.
Study extra about Databricks Genie and Agent Bricks.
