ML for crop prediction models

[Audio] Hi all, in this presentation, we are going to talk about machine learning applications for crop prediction models. My name is Sebastiaan Verbesselt, researcher in precision agriculture at ILVO, Flanders Research Institute for Agriculture, Fisheries and Food. The text of my presentation was converted to audio for this video. Hopefully, it remains easily understandable, and you will have an instructive presentation..

[Audio] Here is a quick overview of the topics we are going to discuss today. First a general introduction of artificial intelligence, machine learning and deep learning. Then we are going to explain how different sensors and platforms can be used to collect data for crop monitoring. We will touch on some of the computer vision algorithms that exist to analyse these data, some examples of research cases we have at ILVO whereby we use machine learning to monitor crops. Last, we will discuss some of the challenges of machine learning for agricultural research..

[Audio] Let's start on what is Artificial intelligence? It is a concept invented by The English mathematician Alan Turing, farther of the first "computer" that decoded the secret codes of the Germans in World War 2. He defined (A-I ) as "AI is the ability of a digital computer or computer-controlled robot to perform tasks commonly associated with intelligent beings"..

[Audio] But what is "intelligence"? Even for living organisms, Biologists have trouble to find a correct definition (or the right characteristics) to say if an organism is intelligent or not..

[Audio] This is because all living organisms have a level of "intelligent" behaviour. From bacteria to plants, animals and humans. All these organisms have strategies to survive, like finding food, escaping from predators, finding mates and so on. And we don't necessarily need "brains" to do this, like plants, bacteria and even some animals..

[Audio] The same goes for computers, or machines steered by computers. Even calculators which most people would not consider as being very "intelligent", can calculate much faster complex equations compared to humans..

[Audio] What is easier to do, is to rank organisms and computers from less intelligent to more intelligent. For example, we could rank humans being more intelligent by chimpanzees, chimpanzees more than mice, mice more than snails and snails more than plants. We can also rank self driving cars being more intelligent than a deep learning object detection algorithm, an object detection algorithm more than a simple machine learning classification algorithm that classifies "pets" into cats and dogs, based on snouth length and ear geometry. The classification algorithm can be ranked being more intelligent than computations in spreadsheets or Excel, and Excel more than a simple calculator..

[Audio] So, there are number to "categorise" (A-I). One way is by level on intelligence (like we did in previous slide). All processes whereby a human has defined the rule set for the data analysis (like for example in spreadsheets) can be considered as "automation". Most machine learning and deep learning algorithms, which we will discuss later, can be considered as "weak (A-I )". They are good for doing one or a limited number of tasks. We see ourself (humans) as being General intelligent, so we are "intelligent" for multiple tasks. Nowadays, we see that first "General Artificial Intelligence" is emerging, like for example Chat-G-P-T. Chat-G-P-T can do multiple 'tasks' at the same time, like translating texts, summarising text, generating new text and images and sentiment analysis. Strong (A-I ), whereby the (A-I ) is more intelligent than humans is for now a theoretical concept..

[Audio] We can also categorise (A-I) based on functionality. Reactive machines are algorithms that don't have any memory. They have predetermined, fixed actions and solutions for every new incoming data. For example, a chess playing computer. Reactive machines can not handle uncertainty or learn from memory. Limited memory (A-I) include most machine and deep learning algorithms of today. They use memory to learn from data in order to make predictions on new, unseen data. Theory of mind (A-I ) is still in a research stage. It includes (A-I) that understand the needs of other intelligent entities (like humans), like complex emotions, thought processes, ideas. Self aware (A-I) is, like the name says, self aware. This is a popular concept in science fiction films where (A-I) algorithms take over the world. This is of course a hypothetical form of (A-I) and is not yet achieved..

[Audio] So, most of the (A-I) that is already present in our daily live are reactive machines and limited memory (A-I)..

[Audio] You can also divide (A-I ) based on application purpose. For (A-I ) that understand and interpret human language is often referred to Natural Language Processing, for images, films and other visual information we call it computer vision, and for audio fragments and human speech, we call it speech recognition. Machine agents that interact with the physical world, whereby they often collect information of there surrounding and use (A-I ) for the interpretation, are called robots..

[Audio] Most (A-I ) experts will however categorize (A-I ) based on technical approach or learning method (explained in the next slide). The general term for intelligent behaviour of machines and computers is called (A-I ). A subdivision of (A-I ) is machine learning, whereby algorithms detect patterns within the data and learn form memory. Data scientists and engineers will help these algorithms by already selecting interesting features within the data to give to the algorithm. For deep learning, a subsection of machine learning, the algorithm will mostly find these features by itself. Last, we have generative (A-I ) (which is a subsection of deep learning), where algorithms can generate new content based on patterns a memory of previously seen data. For example, Chat-G-P-T and Dall-E.

[Audio] Last, we can divide (A-I ) (in this case, machine learning) based on learning method. we have: Supervised Learning: whereby the model is trained on labelled data, meaning each input has a corresponding correct output. For example: Image classification with labelled categories. Second, we have unsupervised Learning: whereby the model is trained on unlabelled data and whereby it must find hidden patterns or structure on its own. For example: Clustering characteristics of customer groups in marketing. You also have semi–Supervised Learning whereby the model is trained on a mix of labelled and a large amount of unlabelled data. It learns patterns from the unlabelled data while being guided by the labelled data. For example: Speech recognition with limited transcribed audio. And last, there is reinforcement learning. The model learns an agent which interacts with an environment and receiving rewards or penalties based on its actions. For example: Training a robot to walk..

[Audio] We are going to illustrate the difference between machine learning and deep learning with the metaphor of learning a child to cycle. In this metaphor, the child is the computer which can use its brain, or algorithms, to learn. The data it must analyse is the bicycle. The parent is in this example a data analyst, that can indicate the interesting features or properties of the data to learn. He or her can indicate on how to steer the bicycle, how to brake, sit on the saddle and step and the pedals. The child or computer will optimize the problem by learning. The less the child falls, the better he or her can cycle. It will also store the information in memory and use it when he or her must bike again..

[Audio] For deep learning, the role of the data analyst or parent will disappear. So (almost) no features already pre-defined. The child or computer will just try to learn by itself and will still optimize the problem. It only takes more training time and data..

[Audio] There is an important downside to most machine learning models, how it learns is often a 'black box'. The computer can sometimes have 'original' solutions. If this works, it is not a problem, but if it eventually goes wrong somewhere, it requires a lot of effort to open and analyse the black box..

[Audio] Wat is essential high-quality conduct machine learning? First, data input. High volumes of relevant and high-quality data that 'represents' the problem to be solved. Data is analysed by algorithms, which are mathematical formulas/architectures that can analyse the data and recognise patterns (and pass this along as insight). For deep learning, these are often artificial neural networks who looks in build like the neuron cells in our brain. Computers with enough processing power are required to do the calculations and optimalisations of the algorithms. This can be sometimes very energy consuming. Both bad for the environment and costly for the end user. To correctly collect and label data and find relevant machine learning applications, the developers' team or data engineers themselves need to have sufficient domain knowledge. Collecting and annotating or labelling large datasets is time consuming and often costly. Agriculture is a very complex sector, both business wise as application wise. Markets are volatile and interactions between living organisms (like the crops and plants) with their environment are complex, which creates variation in data and uncertainty within the machine learning models, lowering their accuracy and precision. The last thing to consider are the ethics of your (A-I) or machine learning application. Are their potential risks or drawbacks of your application? Do you use private, personal or sensitive data? Are you enough transparent of your application to end users and so on..

[Audio] For crop monitoring, you need to understand the complex processes and interactions of plants with their environment. Weather, soil, management, genetics and other organisms can play a role on the plant's phenotype and how the grow. Some useful dataset you can collect to better understand and monitor your field are mentioned on the slide..

[Audio] For the direct monitoring of crops and other relevant organisms as weeds, diseases and pests, we are more and more using remote and proximal sensing technology to do the monitoring. Sometimes combined with a limited number of field observations. This technology uses different types of sensors, which can be divided into active and passive sensors. Active sensors sent out a signal to an object. In this domain, often the crops. The signal will hit the target will be reflected to the camera or sensor, providing it with useful information like biomass, 3D structure, canopy information, water content and so on. Passive sensors measure the reflected electromagnetic spectrum (like visual light, near infrared, thermal infrared and microwaves) of the sun by the plants and terrain. This gives information of the chlorophyll content, biomass, cell structure, water and chemical content of plants. The signals can be used to monitor the plants health, detect diseases, damage by pests and to distinguish weeds from crops..

[Audio] Sensors measure information of plants and soils in an indirect may, without making contact. They can be mounted on aerial platforms like satellites, helicopters, hot air balloons, planes and drones. We often refer to this technology as "remote sensing". Sensors and cameras can also be handheld (like digital cameras or cameras in your cell phone), mounted on carts, robots and agricultural machines or mounted on stationary poles and platforms. We often refer to this technology as proximal sensing..

[Audio] Satellite remote sensing for civilian use are often are open source. Their image rasters have low to medium spatial resolution. Famous examples are the NASA's Landsat-8 satellite with spatial resolutions between 15 to 100 meter per pixel and ESA's Sentinel-2 satellites with spatial resolution between 10 to 60 meter per pixel. Commercial satellites can have higher resolution, like PlanetScope with 3 meter per pixel resolution and WorldView 3 with 0.31 to 30 meter per pixel. Satellites can have high temporal resolution, which can lead to interesting time series data if atmospheric conditions are optimal, so not too much clouds. Pixels are however often larger than the objects we are interested in. In this case, the crops. This is why the machine learning models operate on pixel level. The data can be used for plot boundary detection, anomaly detection (for example, change in management, crop type, disturbances or damage by fire, floods, animals and so on). The data can be used to predict yield on a plot or subplot level or for landcover classification. Satellites collect vast amount of data due to their large coverage. One satellite image of Sentinel-2 has a width of 290 kilometre..

[Audio] Other aerial platforms have medium to very high spatial resolution, depending on the flight altitude, the camera and lens type. Data collection is often costly and not freely available. One flight can however cover a large area, often faster than measurements on the ground. Machine learning algorithms can both function on a pixel level (if the pixels are larger than the plants) or on a object level, where multiple small pixels can represent a plant, insect, soil and so on. Typical computer vision models that work on an object level will use image regression, image classification or recognition, object detection, semantic segmentation, instance segmentation or panoptic segmentation..

[Audio] Proximal sensing has both mobile and stationary platforms. The data is collected at very high spatial resolution but is costly to collect. Machine learning models will again work at object level. Sensors on agricultural machinery and robots for example will use machine learning to detect crop rows for automatic steering, weeds to target them with local treatment or for the evaluation of fruits for harvest..

[Audio] (A-I) and Machine learning models that work on pixel level will collect information on pixel colour, hue, intensity and texture to make predictions for regression, ordination, clustering or classification. Some of the typical examples are given in this slide..

[Audio] (A-I ) and Machine learning models that work on object level will beside collecting information on pixel colour, hue, intensity and texture also look at the form and structure of these pixels within the object. Often deep learning algorithms like convolutional neural networks are used instead of traditional machine learning architectures to analyse these images..

[Audio] Let's discuss some examples on how machine learning computer vision is used for proximal and remote sensing research at ILVO..

[Audio] Machine learning can be used in combination with crop growth simulation models to predict fertilization strategies for leek crops in Flanders. Within the Wikileeks project, soil samples together with soil scan data where collect before the growing season while crops where monitored with satellites and drones during the growing season. This data were used to simulate fertilization strategies for homogeneous zones within the field..

[Audio] Leeks are very Nitrogen demanding plants. In order to minimize the effect to the environment, lower input costs and maximize yield, these managements zones where simulated. Machine learning models where especially used to predict leek biomass and nitrogen uptake from multispectral data collected by the satellites and drones..

[Audio] Another project is the Flaxense 2 project, where Inagro, ILVO and VITO are working together on a digital visual monitoring tool for flax growers based on satellite imagery as input. This tool will provide them with remote information about the crop condition of their flax and advise them about the optimal sowing time, possible re sowing and the use of inhibitors for a homogeneous crop growth..

[Audio] An earlier study (leaded by Inagro and ILVO) in close cooperation with a handful of flax growers showed that satellite images provide a good indication of crop growth in the field. To be able to link useful cultivation advice to this, ILVO will calibrate and validate an existing flax growth model in this new project. To this end, Inagro and ILVO will annually draw data from some twenty practical plots that they will monitor both from the ground and with satellite images. In addition, Inagro and ILVO will set up specific tests to investigate the difference in growth between the varieties and to measure the impact of a growth regulator on the growth of the flax. Those trials will also be monitored with drones. With that data, we will set up a decision support tool around the braking of the flax, so that flax growers can make informed decisions about whether braking is necessary or not (spot on). Through the use of soil moisture maps and satellite images, in combination with weather forecasts, we also want to draw up advice on the ideal sowing time. This data is shown in the WatchItGrow platform of partner VITO..

[Audio] damage but imagery can also be successfully implemented to classify or detect Colorado beetles and their larvae within potato fields. Especially the larvae are clustered within certain areas within the field. They do most damage but can also be treated better than the adults with pesticides. Detected locations of these larvae can be converted into pest maps and task maps for location specific treatment or spot spraying. Again, this reduces the total amount of chemical input, which is both good for the environment and the costs for the farmers. This concept is tested (still ongoing) by ILVO within the koda2030 project: towards a more sustainable cultivation of potato by 2030..

[Audio] Within the same project, ILVO also test if drone and satellite imagery can be used for location specific potato haulm killing. We evaluated the haulm biomass after 0, 1, 2 and 3 treatments of herbicide, whereby we used 2 different potato cultivars, two fertilization treatments and three herbicide volumes. The indices can be converted from a vegetation index to a binary map with living plant or not. The percentage of living biomass per subplot can then be evaluated. The goals is to test if these platforms can be successful tools for monitoring and how vegetation indices can be used to advice farmers where to spray herbicides and by what volume of herbicide in the form of task maps..

[Audio] Computer vision models can also be used to detect weeds within crops. This was showcased at ILVO in 2021, where a drone sent its images out to a deep learning model within the cloud during its flight. For this a 5G antenna was installed by partner Proximus next to the field. This cloud computing ensured a semi real-time site-specific herbicide application in maize. 5 minutes after flight, the results of the (A-I ) model where already converted into a task map for spot spraying and given to the sprayer. This demonstration is showed in the video in the next slide..

Remote and proximal sensing – research examples @ILVO.

[Audio] One year later, we did a similar demonstration. This case, the weeds were not chemically controlled by spot spraying. Instead, the task map was given to our biggest robot, the CIMAT robot. The robot did thermal weed control by only burning away the weed plants..

[Audio] Another research topic is the detection of the leaf disease early blight, caused by the fungus Alternaria solani in potato crops. Alternaria attacks and disrupts the leaf and stem cells and can be recognized by brown and black spots, called lesions, on the plant. At later stages, it can also attach the tubers of the plant. The disease can severely lower the quality and quantity of the potato harvest. The disease can be detected at an earlier stage with specialized camera's that can detect the near infrared, before it is visible for us. This result is shown by looking at a hyperspectral camera, that goes from the blue visible light to green, red and eventually to the near infrared. At the wavelength 730 nanometres, you will see the lesions appearing on the leaves. By modifying a normal digital R-G-B camera, we could look in the near infrared region for better detection of the disease. The plants are no longer green but appear red or orange due to the near infrared filter..

[Audio] With this specialized camera, we performed flight at low altitude, 10 meters above the ground. A zoom lens provided ultra-were images of 0,3 millimetre per pixel. The drone images were cropped to smaller tiles and given to a convolutional neural network for classification of the tiles into healthy and infected patches. From these predictions, we can make infection maps and application maps for location specific fungicide treatment..

[Audio] Sensors can also be mounted on robot platform. The data in collected and real time processed via edge computing for direct application. In this example, we used a multispectral camera mounted on our smallest robot. Via N-D-V-I calculation of the images and thresholding, which is a simple case of green on brown detection, a prescription map was made for the spraying boom. Via adaptable nozzles, only locations where green vegetation (in this case, the weeds) was detected, herbicide was applied under the soil strips of blue berries bushes. This is demonstrated in the video of the next slide. It is however in Dutch, but the images should be clear..

Remote and proximal sensing – research examples @ILVO.

[Audio] We can also train robots to autonomously inspect vineyard by reinforcement learning. For this, we first made a 3D representation of a vineyard were multiple sensor information. R-G-B cameras, lidar, stereo and depth cameras, real time kinematic G-P-S information and odometry data were given to the model..

[Audio] A complete 3D map was created and used for a simulation environment. The model could learn the agent (robot) in this environment to correct navigate via feedback loops. In the end, we want the robot to navigate with these sensor information in an adaptive environment. We cannot let the robot navigate based on a fixed 3D map since the plants will grow and change in structure. Small errors of the G-P-S readings could also hamper fixed navigation paths, while robots with a reinforcement model can correct its path in a smart manner..

[Audio] After the model for navigation was finished, the robot could be used to detect the position of grape bunches via a real sense depth camera. A hyperspectral camera mounted on a robot arm was brought before the grapes to monitor the acidity and sugar content of the grapes, measuring thereby the quality and timing for yield. This is also demonstrated in the next slide. The video is unfortunately in Dutch..

Remote and proximal sensing – research examples @ILVO.

[Audio] Finally, we want to talk about the challenges of machine learning for agriculture. We have already talked about it in the previous slides, but here is a quick summary. Agricultural data is characterised as making highly variable but models uncertain. So be aware of the limits are of your models and re-evaluate in time. Be aware that sufficient domain knowledge is needed to make really good models. Involve agricultural experts in the creation and application of your models. Data is not always readily available. Collecting and labelling them can be time consuming and costly. Use simple models if you have less data available and if they are already sufficiently performant. Some models require a lot of energy and computing power. Check there are sometimes alternatives to your problem. Test new code with limited data and limit your training time. Only use lots of data and long training time after your exploration, when you really want to start training validating your model. Some models are real black boxes. Try to use explainable (A-I ) where possible. Think about the possible ethical implications of your machine learning applications..

[Audio] Our video ends here. Thank you for your attention, hopefully you found it interesting..