Near Real-Time Data Analysis
We are now to the point where we have gigabytes worth of data on soil moisture, plant canopy processes, precipitation, wind speed, and temperature, but the amount of data is so overwhelming that we are having a difficult time dealing with it. The cost of measurement data is dropping so quickly, people are forced to change from a historical mindset where they analyzed individual data points to the mindset of turning gigabytes of data into knowledge.
One approach suggested by my colleague Rick Gill, a BYU Ecologist, is to collaborate with bioinformatics students. Because they are used to working with DNA data, these students understand how to write computer programs that analyze large amounts of data in near real-time. Rick came up with the idea to tap these students’ expertise in order to analyze the considerable information he anticipates collecting in our Desert FMP Project, an experiment which will use TEROS 21 and SRS sensors to determine the role of varying environmental and biological factors involved in rangeland fire recovery.
Rick and I are predicting that near real-time data analysis will give us several advantages. First, we need readily available information so we can tell that sensors and systems are working at the remote site. Large gaps in data are common for sites that aren’t visited often, and sensor failures are missed when data are collected but never analyzed. With our new approach, all data are databased instantly, and the results are visualized as we go. Not only that, we’ll be able to control what’s being analyzed as we see what’s happening. We can tell the bioinformatics students what we need as we begin to see the results come in. If we see important trends, we can assign them to analyze new data that may be relevant right away.
These techniques have the potential to help scientists from all disciplines become more efficient at collection and analysis of large data streams. Although we’ve started the process, we have yet to determine its effectiveness. I will post more information as we see how well it is working and as new developments arise.
Watch Dr. Gill’s data analysis webinar: Finding Insights in Big Data Sets
Get more information on applied environmental research in our