AI forecasting models are only as good as the interval data they train on. Before you can get accurate feeder-level forecasts out of a machine learning system, the data pipeline feeding it needs to meet certain minimum quality thresholds. Here is what we look for when evaluating a new utility's data readiness, and how to prioritize the gaps we commonly find.
The minimum viable data profile
To train a feeder-level load forecasting model that produces reliable 4-hour ahead predictions, we need at minimum 12 months of interval data at 15-minute resolution or better for each target feeder. The data needs to cover at least one full annual load cycle, including both summer and winter peak periods. Anything shorter than 12 months produces models that generalize poorly to seasonal conditions they have not seen.
Completeness matters more than length. A 14-month dataset with 15 percent missing intervals is often less useful than a 12-month dataset with 98 percent completeness. Missing data is not just a gap. It is a structural distortion in the training set if it correlates with specific conditions, such as sensor dropouts that are more likely during high-load events.
Data quality issues we find most often
The most common issue is timestamp inconsistency. Interval data from multiple SCADA nodes or substations often uses different time zone handling, DST transition behavior, or timestamp granularity conventions. When you join data from multiple sources, these inconsistencies produce phantom peaks and impossible transitions that corrupt training data. Resolving this requires a clear timestamp normalization layer that is applied consistently before data enters the training pipeline.
The second most common issue is sensor range validation. Raw SCADA data often contains readings that are technically valid from the sensor's perspective (the sensor reported a number and the number was transmitted correctly) but physically implausible given the feeder's known characteristics. A feeder that routinely peaks at 800 amps does not have 15-minute intervals showing 3,200 amps unless something has changed structurally. These spikes need to be identified and flagged before training.
Third is systematic calibration drift. Metering equipment calibrates at installation and then drifts. If a feeder's apparent load trend is driven partly by meter drift rather than actual load growth, a model trained on that data will have a systematic bias in its future projections. We look for evidence of calibration events in the historian and adjust training windows accordingly.
Prioritizing fixes
If your EMS data pipeline has multiple quality issues, the priority order for fixing them is: timestamp normalization first (because it affects every downstream analysis), then sensor validation (because spikes directly corrupt model training), then completeness (because missing data gaps can often be handled with interpolation if the gap rate is under 5 to 8 percent), then drift correction (because it requires the most manual investigation).
Utilities that have invested in a centralized data historian with standardized ingestion often have much cleaner data than those still relying on ad hoc SCADA exports. If you are at early stages of a data infrastructure build, designing for standardized timestamping and automated range validation at the ingestion layer will pay dividends for any AI application, not just load forecasting.
Weather data integration requirements
Load forecasting models depend on weather data as heavily as they depend on load data. The weather data pipeline has its own quality requirements. The most important is geographic resolution: a single statewide or regional NWP temperature series is not adequate for feeder-level forecasting when feeders cover diverse terrain or when significant elevation variation exists across the service territory.
Ampgrove pulls NWP data at hourly resolution for each feeder's geographic centroid. The ingestion pipeline normalizes timestamps to UTC, validates that values are within physically plausible ranges for the location and time of year, and handles gaps using linear interpolation for periods under 3 hours. Gaps longer than 3 hours revert to climatological means for the hour and season, with a quality flag set on the affected forecasts to indicate reduced confidence.
Handling data from multiple SCADA vintages
Many utilities operate SCADA infrastructure that spans multiple generations of vendor hardware and software. A single service territory might have substations running different historian versions, different interval cadences, or different data encoding conventions. Joining data from these systems into a consistent training dataset is one of the more time-consuming parts of a typical integration project.
The most common challenge we encounter is interval cadence mismatch. One historian exports 15-minute intervals, another exports 5-minute intervals, and the load research meters on some feeders run at 1-minute intervals. Aggregating these to a consistent 15-minute resolution sounds straightforward but produces subtle artifacts if the aggregation handles DST transition periods incorrectly. A 15-minute interval that crosses a DST boundary contains either 14 or 16 minutes of actual data, and naive averaging produces incorrect energy values for those intervals.
We address this by requiring all interval data to be stored and processed in UTC before any aggregation is performed. Converting to local time for display and reporting is done as a final step. This approach eliminates the DST-transition aggregation error completely, at the cost of requiring the data engineering team to work consistently in UTC from the start.
Ongoing pipeline monitoring in production
Data quality issues that are manageable in a one-time historical extraction become more operationally significant in a live production pipeline. A sensor that drops out for 4 hours affects one block of historical training data. A sensor that drops out recurrently during high-load events affects the model's ability to train on exactly the conditions it most needs to handle well.
Ampgrove's production pipeline includes continuous data quality monitoring with alerts sent to our operations team when feeder data quality metrics drop below threshold. We track completeness rate (percentage of expected intervals received), range validity rate (percentage of intervals within feeder-specific physical bounds), and latency (time between data generation at the substation and arrival in the forecast system). Each metric has a threshold below which we flag the feeder as data-degraded and display reduced confidence indicators on the forecast outputs for that feeder.
Providing this quality monitoring as an output to utility operations teams is on our product roadmap. Distribution operations teams often do not have automated data quality monitoring on their EMS historian today, and the byproduct visibility from a forecasting system that depends on clean data is a useful operational tool independent of the forecasting application itself.