Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
1
MUSHROOM HARVESTING VISION SYSTEM AND METHOD OF HARVESTING
MUSHROOMS USING SAID SYSTEM
FIELD OF THE INVENTION
[001] The present invention relates to a mushroom harvesting vision system, as
well as a method of harvesting mushrooms using said system.
BACKGROUND OF THE INVENTION
[002] Vision systems that can detect mushrooms are known, such as that
disclosed in Canadian Patent No. 3,073,861C and W02020097727A1 (Inventor:
GOOD; Applicant: MYCIONICS INC.). Such vision systems can detect the
properties of mushrooms (e.g., position, size, shapes, orientations, growth
rates,
volumes, mass, stem size, pivot point, maturity, and surrounding space), which
can
be used to determine strategies required for instructing a picking system for
autonomous mushroom harvesting. Such vision systems comprise 3D scanners,
each having a pair of camera apertures for capturing images and a laser slot
for
permitting a laser line to project from the vision system rail onto the
mushroom
below. The vision system rail can have its multiple 3D scanners aligned in one
straight line to effectively form a combined line scanner within tightly
constrained
vertical spaces, while achieving sub-millimeter accuracy and high data
throughput.
Such vision system rails can also generate color information that is
overplayed on
a 3D point cloud allowing for real-time disease detection, mushroom quality
and
type identification.
[003] The above documents also describe conventional automated harvesters
comprising vision systems supported by a rail at one end of a frame, the
vision
system configured to scan a growing bed under the frame; and a picking system
moveable within a working area defined by the frame.
[004] Other known vision systems include those disclosed in US Patent No.
9,730,394 and Canadian Patent application No. 2,943,302A1 (Inventor: VAN DE
VEGTE; Applicant VINELAND RESEARCH AND INNOVATIONS CENTRE INC.),
which disclose a method and system for controlling harvesting of mushrooms
from
a bed during a mushroom graze harvest operation accounting for both mushroom
separation and stagger, thereby providing for automatic selection of which
mushrooms in the bed are to be harvested in a given shift.
Date Recue/Date Received 2021-03-12
2
[005] Such documents disclose a system for harvesting mushrooms from a bed,
the system comprising:(a) one or more mushroom harvesters configured to pick
mushrooms from the bed; (b) one or more cameras for locating mushrooms in the
bed and measuring cap diameters of the mushrooms; and (c) a control apparatus
operatively linked to the one or more cameras and operatively associated with
the
one or more mushroom harvesters, wherein the control apparatus is configured
to
receive image data from the one or more cameras and from the image data to
determine cap diameters of the mushrooms, locate centroid positions of
mushrooms having a cap diameter greater than a pre-determined value, and for
mushrooms for which the centroid position was located calculate centroid-to-
centroid distances to each neighboring mushroom, compare the centroid-to-
centroid distances for sets of two mushrooms to sum of radii for the two
mushrooms, count the number of interfering mushrooms to identify clumps of
mushrooms to be thinned and determine which mushrooms to pick from the
identified clumps of mushrooms, and wherein the control apparatus is
configured
to aid or operate the one or more mushroom harvesters to pick mushrooms having
cap diameters equal to or greater than a pre-set value and pick the mushrooms
determined to be picked from the identified clumps of mushrooms.
[006] US Patent No. 5,471,827 (Inventor: JANSSEN; Applicant: CCM BEHEER
B.V.) also discloses a device for the automatic selective harvesting of
mushrooms
grown on a growing bed that includes: at least one camera for observing the
mushrooms on the growing bed; a carrier which is movable above the growing bed
relative thereto, and is provided with apparatus for positioning one or more
picking
heads for picking mushrooms on the basis of information coming from each
camera.
[007] US Patent No. 5,058,368 (inventor: WHEELER) discloses an apparatus for
harvesting delicate produce, but particularly mushrooms. Said apparatus
includes
a picking head which is controlled to be positioned over an item of produce to
be
harvested, by a camera which scans a tray of said items and a control unit
operating on a camera output to determine the co-ordinates of those items
found
to be suitable for picking, for example from the size of those items.
Date Recue/Date Received 2021-03-12
3
SUMMARY OF THE INVENTION
[008] In accordance with the present invention, there is provided mushroom
harvesting vision system comprising: at least one camera; a processor for
running
an algorithm in a computer-operated algorithm, the computer-operated algorithm
comprising: an input module for selecting an ellipse diameter range, a
sensitivity
value, and an accuracy value; an image input module for inputting at least one
image into the computer-operated algorithm, said at least one image depicting
one
or more mushrooms and having been taken by the at least one camera; a
preprocessing module for preprocessing each image and converting each image
into greyscale; and a detection module for detecting coordinates of at least
one
mushroom, the detection module comprising an image processor in which each
image is processed to produce at least one image output matrix; wherein each
row
of each image output matrix comprises x, y, z coordinates of one of said at
least
one mushroom, a corresponding gripper orientation, and a corresponding
mushroom orientation; and a robot comprising a gripper arm, wherein the robot
utilizes the output matrix to automatedly pick the at least one mushroom with
the
gripper arm.
BRIEF DESCRIPTION OF THE DRAWINGS
[009] Figure 1 is a schematic of an embodiment of the system according to an
embodiment of the present invention.
[010] Figure 2 is a picture of a mushroom bed taken using a camera of an
embodiment of a system of the present invention.
DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS
[011] Referring first to Figure 1, in a first aspect of the present invention,
a
mushroom harvesting vision system 10 is provided. The system 10 includes at
least one camera 12 and a processor 14 for running a computer-operated
algorithm
16. The computer-operated subsystem 16 includes an input module 18, wherein
an ellipse diameter range, a sensitivity value, and an accuracy value are
input
parameters 19 that may be selected. The computer-operated subsystem 16
includes an image input module 20 for inputting at least one image into the
computer-operated algorithm 16. The at least one image depicting one or more
Date Recue/Date Received 2021-03-12
4
mushrooms and having been taken by the at least one camera 12. The computer-
operated algorithm 16 includes a preprocessing module 22 for preprocessing
each
image and converting each image into greyscale; and a detection module 24 for
detecting coordinates of at least one mushroom. The detection module 24
includes
an image processor in which each image is processed to produce at least one
image output matrix. Each row of each image output matrix comprises x, y, z
coordinates of one of said at least one mushroom, a corresponding gripper
orientation, and a corresponding mushroom orientation. The system includes a
robot 26 comprising a gripper arm 28. The robot 26 utilizes the output matrix
to
automatedly pick the at least one mushroom with the gripper arm 26.
[012] The mushroom harvesting vision system can be used for harvesting a
variety of mushrooms, preferably Agricus Bisporous mushrooms.
[013] The at least one camera used by the system of the present invention can
be any camera known for use in such mushroom harvesting vision systems. In
preferred embodiments, each camera is preferably an Intel D435i camera. In
preferred embodiments, only a single camera is used. The camera is for taking
pictures of the mushroom bed where harvesting will take place; an example of
such
a picture is shown at Figure 2. In embodiments, for example in the case of a
rather
large mushroom bed, multiple images are taken, each image representing an area
of the mushroom bed to be harvested. In alternative, preferred embodiments, a
single image is taken, said image being of the entire mushroom bed to be
harvested.
[014] The processor used in the present invention can be any processor that is
capable of running the computer-operated algorithm. In addition, the computer-
operated algorithm can be written in any computer language. In preferred
embodiments, the computer algorithm is written in Matlab.
[015] What follows is a description of the various modules of the computer-
operated algorithm.
[016] Input module: With the input module, an ellipse diameter range, a
sensitivity
value, and an accuracy value are selected. In embodiments, these input
parameters are automatically provided as an argument to the function (the
algorithm can be run as a function) after being selected.
Date Recue/Date Received 2021-03-12
5
[017] The ellipse diameter range will determine the range of mushroom
diameters
(using the major axis length of each mushroom) that the mushroom harvesting
vision
system will include in the at least one output matrix for the robot to
harvest. If a
mushroom diameter is outside the ellipse diameter range, the mushroom will not
be
harvested. Naturally, as this value is a range, upper and lower limits can be
selected, although it is also possible to select mushrooms of a specific size.
[018] The sensitivity value is another input parameter; the value of
sensitivity
depends on the type of room and its ambient lighting. The lighting of a room
can
be adjusted inside of the algorithm, in order to compensate for lighting that
is 'too
bright' or 'too dull'. The sensitivity value can preferably range from 0.9 to
0.99,
going from very dull lighting to bright lighting. However, the skilled person
would
understand that different ranges can be used.
[019] The accuracy value is another input parameter, referring to the accuracy
threshold, also referred to as the edge threshold. The accuracy threshold can
be
controlled inside of the algorithm in order to work with specific algorithmic
accuracy.
This accuracy parameter is provided as an input argument in order to perform
the
detection as per the accuracy requirement for harvesting mushrooms.
[020] The accuracy threshold is an important parameter for quality control, as
some users may require mushrooms to be precisely of a certain size range.
Thus,
in that case, the accuracy parameter (function argument) needs to be set high.
[021] On the other hand, it is occasionally necessary (e.g. during cleaning),
to
harvest all size ranges of mushrooms. In such cases, the accuracy threshold
may
be set to a low value.
[022] Image input module: Each image input into the algorithm will have been
taken using the at least one camera. The image input module is configured so
as
to receive the at least one image automatically after having been taken. Each
image input into the algorithm can be an RGB image of size 1280*720 pixels,
although different sizes and color schemes can be used, depending on the
camera
used and the needs of the user. As with the ellipse diameter range, the
sensitivity
value, and the accuracy value, each image is provided as an argument to the
function. The image input module and the input module can be the same module,
or separate modules, depending on the needs of the user.
Date Recue/Date Received 2021-03-12
6
[023] Preprocessing module: with the preprocessing module, each image is
preprocessed using Gaussian and Prewitt filters. Each image is then further
converted into a GrayScale image for further processing. It is to be
understood that
if the image in question is already in greyscale (e.g. the camera takes
greyscale
images), it is not necessary for the preprocessing module to convert the image
into
greyscale.
[024] Detection module: with the detection module, the coordinates of the
mushrooms are automatedly detected based on each pre-processed image and the
input parameters. In embodiments, the detection module is configured to
process
each image using an ellipse detection method, whereby the image is processed
using a pixel density and a density characteristic of the image at all regions
of the
image based on each region's pixel coordinates, such that certain areas of the
image are eliminated, and the detection of the coordinates of the at least one
mushroom is a result of mathematical computation on all the regions.
[025] In embodiments, each image is treated as a parent matrix and each of a
plurality of subsections on the image are treated as a child matrix, wherein
the child
matrices are used to process characteristics at each specific region and
processing
results for each child matrix are plotted back at the parent matrix.
[026] The coordinates and the processing results of each child matrix are
interlinked using a graphing system from 0,0 up to 720,1280 (although this
value
may vary depending on the image used) with each adjacent child matrix, such
that
the coordinates from the child matrices do not overlap with adjacent child
matrices.
[027] Furthermore, outside of the child matrix, the whole image is also
processed
as a parent matrix.
[028] As a result of such processing, the detection module produces the at
least
one image output matrix comprising x, y, z coordinates of one of said at least
one
mushroom, a corresponding gripper orientation, and a corresponding mushroom
orientation (defined in more detail below).
[029] Image output matrix: Each output from the algorithm is a matrix of size
n*5
(although larger sizes than "5" may be used if the user wishes to have
additional
output parameters), where n represents the number of detected mushrooms that
Date Recue/Date Received 2021-03-12
7
the robot will harvest. The first three columns represent the x,y,z
coordinates of a
specific mushroom; the fourth column represents the gripper orientation for
said
mushroom, and the fifth column represents the mushroom orientation of said
mushroom. The skilled person would understand these five values could occupy a
different order in the matrix depending on the preferences of the user, so
long as
said matrix can be properly used with the robot.
[030] As mentioned, each row in the image output matrix corresponds to a
different detected mushroom that the robot will pick. Once the robot has
harvested
a mushroom from one row, it can then proceed to the next row. In embodiments,
multiple image output matrices can be produced, each matrix comprising
information on one or more mushrooms that the system will harvest. For
example,
the detection module can output an image output matrix for each image that is
input
into the image input module.
[031] In preferred embodiments, each output matrix is converted into CSV
(comma separated value) format in order to be used in conjunction with the
robot,
although any format can be used, as long as it can be used in conjunction with
the
robot.
[032] X,v,z coordinates of the mushrooms: The output of coordinates of
mushrooms (comprising x, y, and z coordinates) are based on the major axis
length
of the mushrooms.
[033] Gripper Orientation: The gripper orientation refers to the angle the
gripper
needs to be rotated in order to find the space between clusters, thus
facilitating the
picking process, such that the picking action does not harm either mushroom to
be
picked or neighboring mushrooms. As mentioned, the gripper orientation is an
output of the algorithmic processing. In preferred embodiments, the value of
the
gripper orientation ranges from 1-4, although other ranges may be used
depending
on the preferences of the user. What follows is a more detailed explanation of
the
gripper orientation using various scenarios, taking into consideration the
following
matrix image:
Date Recue/Date Received 2021-03-12
8
1 ,
446601 6)
t.)
_4 =
p....
.1 6
_
(
-.)1
.01.
[034] Scenario 1: In the above image, the figure represents a 3*3 matrix, the
values range from 1:9. If '5' represents a mushroom that the system intends to
pick,
and the system knows that there are mushrooms at '1', '2', '4','6' and '9',
then the
only places where the robot may be able to fit the gripper fingers would be at
'3'
and '7'. Thus, by doing the respective computational analysis, the algorithm
will
give the result as index '3'.
[035] Scenario 2: once again, '5' represents a mushroom that the system
intends
to pick, and the system knows that there are mushrooms at '1', '2',
'3','7','8' and '9'.
Now the only place where the robot may be able to fit the gripper fingers
would be
at '4' and '6'. Thus, by doing the respective computational analysis, the
algorithm
will give the result as index '4'.
[036] Scenario 3. Once again, '5' represents a mushroom that the system
intends
to pick, and the system knows that there are mushrooms at '1','3','4','6','7'
and '9'.
Now the only place where the robot may be able to fit the gripper fingers
would be
at '2' and '8'. Thus, by doing the respective computational analysis, the
algorithm
will give the result as index '2'.
[037] The algorithm would perform a similar computation and give an output of
'1', if there is space at '1' and '9'.
[038] In cases where there is no space, the algorithm will skip that mushroom,
and will analyse the mushrooms at that mushroom's respective neighbor, such
that
the algorithm will find a way inwards and outwards to harvest and find the
best
possible path and ranking for harvesting mushrooms.
[039] The reason for solving such a problem can be thought of by analysing the
image shown in Figure 2. In Figure 2, it can be seen that the mushrooms grow
in
Date Recue/Date Received 2021-03-12
9
clusters, and if a user were to simply keep a standard/fixed orientation of
the
gripper, said gripper will keep on hitting/bruising the neighbouring mushrooms
of
the mushroom that is required to be picked.
[040] Mushroom Orientation: The mushroom orientation refers to the angle of
the
mushrooms with respect to its respective ground (typically compost). This is
useful
in order to analyse the approach of the end effector (specifically the
gripper) to
harvest mushrooms. In preferred embodiments, the angle output of the mushroom
orientation varies from about 45 degrees to about 135 degrees, although
different
ranges of angles may be used, depending on the preferences of the user.
[041] The robot used by the system of the present invention is configured to
automatedly receive the image output matrix and to harvest mushrooms having
regard to said image output matrix. By receiving the image output matrix, the
robot
can automatedly harvest mushrooms in such a manner as to harvest the
mushrooms as desired by the user (e.g. having a certain ellipse diameter) in
an
efficient manner that minimizes (or preferably entirely avoids) hitting,
damaging, or
bruising any mushrooms (both the harvested mushrooms or any neighbouring
mushrooms).
[042] As mentioned, the robot comprises a gripper that is capable of
harvesting
the mushrooms. Furthermore, the robot is capable of automatedly rotating and
moving the gripper (which is preferably one and the same as the end effector)
in
order to harvest the mushrooms. The gripper can be any gripper used for such
harvesting systems as known in the prior art.
[043] In embodiments, the robot may comprise more than one gripper. Moreover,
in embodiments, the system of the present invention may comprise more than one
robot, each robot having one or more grippers.
[044] The system of the present invention can be used in a mushroom harvester.
METHOD OF HARVESTING MUSHROOMS USING MUSHROOM HARVESTING
VISION SYSTEM
[045] In a second aspect of the present invention, a method of harvesting
mushrooms using the mushroom harvesting vision system is provided, the method
comprising:
Date Recue/Date Received 2021-03-12
10
= inputting an ellipse diameter range, a sensitivity value, and an accuracy
value into an input module of a computer-operated algorithm, the computer-
operated algorithm being run with a processor;
= taking at least one image using at least one camera of an area comprising
one or more mushrooms;
= inputting each image into an image input module of the computer-operated
algorithm;
= preprocessing each image using a preprocessing module of the computer-
operated algorithm, thereby converting each image into greyscale;
= detecting the coordinates of at least one mushroom using a detection
module of the computer-operated algorithm, said detection comprising
processing each image using an image processor to produce at least one
image output matrix; wherein each image output matrix comprises x, y, z
coordinates of one of said at least one mushroom, a corresponding gripper
orientation, and a corresponding mushroom orientation; and
= harvesting said at least one mushroom using a robot comprising a gripper
arm, wherein the robot utilizes the at least one output matrix to pick the at
least one mushroom with the gripper arm.
[046] The method of the present invention is performed using the above-defined
system. Accordingly, the ellipse diameter range, the sensitivity value, the
accuracy
value, the input module, the computer-operated algorithm, the at least one
image,
the at least one camera, the image input module, the preprocessing module the
detection module, the image processor, the at least one image output matrix;
the
x, y, z coordinates of the mushrooms, the gripper orientation, the mushroom
orientation, the robot, and the gripper arm are as defined in the previous
section.
[047] For clarity, in embodiments, the detecting step comprises processing the
image using an ellipse detection method, whereby each image is processed using
a pixel density and a density characteristic of the image at all regions of
the image
based on each region's pixel coordinates, such that certain areas of the image
are
eliminated, and the detection of the coordinates of the at least one mushroom
is a
result of mathematical computation on all the regions.
Date Recue/Date Received 2021-03-12
11
[048] Each image is treated as a parent matrix and each of a plurality of
subsections on the image are treated as a child matrix, wherein the child
matrices
are used to process characteristics at each specific region and processing
results
for each child matrix are plotted back at the parent matrix.
[049] The coordinates and the processing results of each child matrix are
interlinked using a graphing system from 0,0 up to 720,1280 (although this
value
may vary depending on the image used) with each adjacent child matrix, such
that
the coordinates from the child matrices do not overlap with adjacent child
matrices.
Outside of the child matrix, the whole image is also processed as a parent
matrix,
thereby producing the at least one image output matrix.
[050] As mentioned in the system section, the method of harvesting mushrooms
of the present invention is almost entirely automated, as the only step that
the user
would have to perform themselves is inputting the ellipse diameter range, the
sensitivity value, and the accuracy value (and even this step can be performed
automatedly), while the remaining steps are performed automatedly once the
user
decides to activate the system to harvest mushrooms.
[051] The scope of the claims should not be limited by the preferred
embodiments
set forth in the examples, but should be given the broadest interpretation
consistent
with the description as a whole.
DEFINITIONS
[052] The use of the terms "a" and "an" and "the" and similar referents in the
context of describing the invention (especially in the context of the
following claims)
are to be construed to cover both the singular and the plural, unless
otherwise
indicated herein or clearly contradicted by context.
[053] The terms "comprising", "having", "including", and "containing" are to
be
construed as open-ended terms (i.e., meaning "including, but not limited to")
unless
otherwise noted.
[054] Recitation of ranges of values herein are merely intended to serve as a
shorthand method of referring individually to each separate value falling
within the
range, unless otherwise indicated herein, and each separate value is
incorporated
into the specification as if it were individually recited herein. All subsets
of values
Date Recue/Date Received 2021-03-12
12
within the ranges are also incorporated into the specification as if they were
individually recited herein.
[055] All methods described herein can be performed in any suitable order
unless
otherwise indicated herein or otherwise clearly contradicted by context.
[056] The use of any and all examples, or exemplary language (e.g., "such as")
provided herein, is intended merely to better illuminate the invention and
does not
pose a limitation on the scope of the invention unless otherwise claimed.
[057] No language in the specification should be construed as indicating any
non-
claimed element as essential to the practice of the invention.
[058] Herein, the term "about" has its ordinary meaning. In embodiments, it
may
mean plus or minus 10% or plus or minus 5% of the numerical value qualified.
[059] Unless otherwise defined, all technical and scientific terms used herein
have the same meaning as commonly understood by one of ordinary skill in the
art
to which this invention belongs.
[060] The scope of the claims should not be limited by the preferred
embodiments
set forth in the examples but should be given the broadest interpretation
consistent
with the description as a whole
Date Recue/Date Received 2021-03-12
