Session 13-F

A Rule-Based Solution for the Automation of Anomaly Detection in Hydro-Meteorological Data

← Back to Agenda

This paper addresses the challenges faced by the water resource management industry in managing its extensive water resource infrastructure. Over the past fifteen years, the proliferation of field sensors has added complexity to data management processes, particularly in water level monitoring. Manual methods, reliant on substantial manpower, which is labor intensive, i.e., costly, introduce challenges such as data review delays and susceptibility to human errors. To overcome these limitations, this paper proposes a rule-based automated anomaly detection system to efficiently process the increasing volume of real-time data. Focusing on upstream and downstream water level sensors, the proposed system integrates thirteen business rules and specific algorithms to identify deviations from established quality standards. These rules encompass criteria such as sensor value limits, historical standard deviation, physical limits, and the rate of change. The methodology ensures sustained effectiveness in upholding data integrity and facilitating timely decision-making. Implemented using the R statistics package and GIS, the automated processes occur seasonally, annually, and multi-yearly. The near real-time process tags provisional fine-resolution data based on business rules, while a nightly process estimates tagged data through linear regression. The comprehensive three-phase approach involves data analyst review, manual and automated processes running in parallel, and a final hybrid process for data repair. This methodology not only addresses current challenges but also aligns with the water resource management industry’s commitment to providing better data quality in near real-time. By enhancing anomaly detection, the proposed approach supports a sustainable future for water resource management in the region. This paper marks a significant step toward ensuring data integrity and informed decision-making in the dynamic field of water resource management.

Speaker

John Raymond

Section Lead (retired), South Florida Water Management District

Rutambara Sonawane

Scientist 4, South Florida Water Management District

Sarah Noorjahan

Section Lead, South Florida Water Management District

THE CDOIQ SYMPOSIUM HAS BEEN SUPPORTED BY THE SYMPOSIUM SPONSORS, CHIEF DATA OFFICERS AND DATA LEADERS.

Welcome to the CDOIQ Symposium, where innovation meets data excellence! Our symposium sponsors play a crucial role in shaping the future of data and information quality. As champions of cutting-edge technologies, thought leadership, and industry advancements, these forward-thinking organizations contribute to a dynamic ecosystem dedicated to harnessing the power of data for transformative business outcomes. Join us in recognizing and celebrating the invaluable support of our sponsors, who drive the conversation and inspire breakthroughs in the rapidly evolving landscape of Chief Data and Information Quality. Together, we pave the way for a data-driven future that propels organizations to unprecedented heights of success.

TIER-1 SPONSORS

TIER-2 SPONSORS

TIER-3 SPONSORS