6.8: Vision-Based Sensing

Learning Objectives

After completing this section, you will be able to:

1. Use the IR camera to detect agricultural spray system fault.
2. Explain the automotive image-based lane detection process.

In Section 6.1, we reviewed an image-based in-cylinder temperature sensing method. We used an example related to internal combustion engine research for auto-ignition control of the combustion process for a homogenous charge compression ignition (HCCI) [14], [15]. In this section, we will provide two more examples for image-based diagnostics and detection.

The first example is a solid-set canopy delivery system (SSCDS). It was developed to deliver water and chemical solutions to high-density fruit trees. For the purpose of detecting SSCDS faults, we propose using the thermal image-based detection method; see Chapter 5 of reference [12]. In the experimental setup, a FLIR thermal camera was mounted on a modified commercial unmanned aerial vehicle (UAV). Thus, any gusher fault, or an emitter nozzle, whether partially plugged or plugged, can be captured by the thermal infrared (IR) camera during the water and chemical spray processes. Due to the nature of the SSCDS, the fault detection mechanism is necessary. Over time, the emitters are easy to clog (plug) or fall off (gusher). To help growers locate faulty emitters in the field without checking emitters one-by-one in person, the proposed thermal fault detection mechanism is based on a thermal IR camera mounted on a UAV. Live thermal videos of the faulty sections can be sent to the user with a video transmitter mounted on the drone.

Because the camera is mounted on a drone, the thermal IR camera should be light in weight and small in size with low operational power. These requirements led to selection the FLIR VUE Pro R, a thermal IR camera for professional drones. This thermal camera, along with a conventional camera, is used for recording the spray process. The thermal IR videos are broken into frames and stitched into panorama pictures for post analysis.

For experiments conducted during a summer morning, the water temperature is usually lower than the atmosphere. Due to the transpiration effect, the trees are slightly cooler than the atmosphere prior to the spray. During the spray, the sprayed area temperature may be even lower than the area not being sprayed. When the outside temperature is low (below 50\(\mathtt{^\circ\!{F}}\)), the water is hotter than the atmosphere; the tree temperature condition is reversed prior to and during spraying. Note that the thermal camera has a resolution of \(\pm 0.2\mathtt{^\circ\!{C}}\) in distinguishing the temperature differences between objects. The thermal camera is used to pick up the hot or cold spots on trees. These spots will be identified as fault regions. The corresponding thermal footage is then referred to as feasible thermal footage. The flight and spray patterns are the key for a successful fault detection.

For the thermal footage in a black and white pattern, temperature differences are indicated by the level of darkness: darker color means lower temperature and lighter color higher temperature. In this case, experiments were conducted on a cold day with almost no wind. Therefore, the water was hotter than the atmosphere. In this orchard, faults were also manually created on purpose; Table 6.3 lists the locations.

Table 6.3: Emitter faults created.

Fault location in Figure 6.45	Emitter fault type
B1	Plugged
B2	Gusher
B3	Plugged
B4	Gusher

Because these emitters were too close to each other, only the faulted emitters were marked with small pieces of white stripes on top of the irrigation line, as seen in Figure 6.45 (e). The faulted emitters are also marked from B1 to B4 at the corresponding locations in Figure 6.45 (d); see Table 6.3 for the type of fault. In this experiment, thermal plate fusion was used; the associated results are shown on Figure 6.45 (a) and (b) associated with the spray with and without created faults, respectively. A black-white thermal plate was also used for fault and no-fault spray situations. These are shown in Figure 6.45 (c) and (d), respectively. The fusion video was taken first. The black-white video was taken two hours later after all the sprayed solution was evaporated and the canopy temperature recovered back to normal. The experiment parameters were exactly the same for these two thermal plates.

Comparing Figure 6.45 (a) with Figure 6.45 (c), we can see that even though the experiment parameters were exactly the same for the two thermal plate runs, the figures indicate a huge difference in these two thermal plates. The black-white footage shows a clearer result, with a larger faulted area than the thermal fusion one, indicating that for this thermal camera, the black-white plate has a better sensitivity than the fusion plate.

From Figure 6.45, we can easily detect the gusher effects using the thermal camera by observing the hot areas due to higher flow rate of hot solution (water) with normal emitter. However, the plug effects are not even visible in the thermal footage. But with known locations for the plugged emitters, these plugged (missing) emitters can be detected.

The second image-based sensing is for road lane detection based on forward video images. Lane centering is also known as lane centering assist, lane assist, auto steer, or autosteer. As a feature in automated driving, it is an advanced driver-assistance system used to keep a moving vehicle remaining at the center of the lane, relieving the driver of the task of steering. Lane centering keeps the car centered in the lane. It is similar to lane-departure warning and lane-keeping assistance that warn the driver and keep the car away from the lane edge, respectively. Together with adaptive cruise control (ACC), keeping a safe distance from the vehicle in front, this feature could enable unassisted driving for some length of time. It is also part of automated lane keeping system.

We will use the framework of the minimalistic approach described in [13] to explain the process of lane detection shown in Figure 6.46. In reference [13], color segmentation is used to detect objects or parts of the image that are of a particular color. The algorithm processes only sections of the image that have the specified colors. This approach is particularly useful for extracting information for lane markings; around the world, lane markings are dominantly painted in either yellow or white. The image-processing involves loading the image, then converting it from its original RGB color space to a hue, saturation, and lightness (HLS) color space. Note that RGB is an additive color model, meaning it adds red, green, and blue light together to create different colors. The process further splits the space into its color bands of hue, lightness, and saturation. Canny edge detector [56] is then used to produce the best edge image. Considering noise contained in any real-world image, smoothing filter is used to reduce image noise. Then, the Hough transform, a popular computer vision technique, is used for detecting lines, and linear regression is employed to obtain smooth lines (detected lane); see [13] for a detailed approach.

---