Cleaning Up Noisy Data with a Software Filter
Electronic sensors are frequently used to measure physical properties and send the information out as an electrical signal (voltage or current) that can be read by a computer or other electronic instrument. Physical properties such as temperature, distance, pressure, electrical resistance, or optical transmission are not always easy to read due to the inconsistencies of the real world. Surfaces are not smooth, temperature is not stable, glass is not perfectly clear, electrical connections are not ideal, etc.
This results in sensor values that are higher or lower than the actual value being read and the erroneous data is called noise. Another kind of noise comes from electrical interference on the cables and connectors that the data are used to communicate the data from the sensor to the microprocessor. It is best to reduce both types of noise at the source, and read in good data, but when that is not possible a software filter can be used to “clean up” or smooth out the data.
Moving average filter
A moving average filter is fairly easy to implement in software, and typically requires past data to be saved and averaged in with the current data. After enough cycles have passed to get an average, the underlying trends in noisy data can be seen. In the code the filter is regularly updated with new data and the oldest data is purged.
An example of a moving average filter that averages the last 5 readings is shown here.
Relative changes vs. absolute changes in data
The moving average filter is particularly useful when it is important to find relative changes in the data, or trends up and down, as is the case with data from the Robot Whisker. In this sensor the conductive grease is encased in stretchable silicone rubber. In its undisturbed state, the grease has a baseline resistance that is stable. When the whisker is deflected the shape of the grease path changes and its resistance changes by a significant amount. However, the when the sensor goes back to its original position, the resistance does not go back to its baseline value, rather it takes several seconds to return the baseline value. In other words, the data trends up and down relatively slowly compared to jumping up and down along with the touch events.
This creates two problems, the first one being that the sensor readings will have to be taken several seconds apart, which might miss some whisker hits. The second problem comes from choosing to take sensor readings quickly; since the sensor resistance would not have gone back to its baseline level, there would be false sensor readings.
The moving average filter solves these problems by allowing for many readings per second, and removing the slowly changing resistance of the undisturbed resistor but still passing the quick changes. See the example in this graph.