Note: Descriptions are shown in the official language in which they were submitted.
Doc. No. 374-1 CA/PCT 1
Patent
SYSTEM FOR CONTROLLING CONSTRUCTION PROCESSES
The invention is related to the field of construction and assembly of
structures,
namely to a system for controlling the processes of constructing and
assembling structures
using BIM technology, and can be applied to the construction and assembly of
buildings,
bridges, ships, airplanes, furniture, and other objects.
BIM technology (Building Information Model or Modeling ¨ information modeling
of buildings and structures) covers the processes of design, construction, and
operation of
various structures using a single coordinated system of three-dimensional
models. The main
BIM element is the information that is designed in the project, as well as the
exchange
process of this information between various participants.
Hereinafter, the use of BIM technology implies the availability of an
information
model of the structure and the ability to work with the design three-
dimensional model of
the structure.
A construction process control system is known from the prior art (CN
110335341
A, October 15, 2019), containing a remote server and at least one computing
device
connected with a memory module, a data exchange module for communication with
the
remote server, a visualization module, and an interface module, wherein the
remote server
is capable of storing an information model of the structure in which
information about its
elements is recorded. The computing device is capable of receiving from the
remote server
the information model by means of the data exchange module, storing it in the
memory
module, and issuing a command to the visualization module to display the three-
dimensional
model of the structure and information about its elements. In this case, the
interface module
allows the user to enter a label of the structure defect, identified during
the survey, on the
corresponding section of the three-dimensional model. The computing device
saves the
changes in the memory module and sends the corrected three-dimensional model
to the
remote server via the data exchange module.
In this way, the construction process is monitored. This method does not allow
detecting defects in automatic mode.
A system for construction process control is known (KR 101897434 B1, September
10, 2018), containing at least one computing device connected with one or more
video
cameras for performing marker scanning, a memory module, a data exchange
module, and
a visualization module, wherein the computing device is capable of obtaining
an information
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 2
Patent
model by means of the data exchange module and storing it in the memory
module,
recognizing markers applied in the form of barcodes or QR codes on structural
elements,
when scanning them with a video camera, uploading information about the
corresponding
elements, forming a two-dimensional model of the scanned construction part,
selecting the
corresponding two-dimensional construction part from the information model,
and
comparing these parts. After the comparison, the visualization module displays
the
discrepancies ¨this is how construction control is performed.
The main disadvantageous features of this solution are the impossibility of
the remote
monitoring of construction or assembly processes and the low information
content of the
model.
As a prototype, the system for monitoring construction processes was selected
(US
10739590 B2, August 11, 2020), containing a remote server and at least one
computing
device connected with one or more video cameras for scanning markers, memory
module, a
data exchange module for communication with the remote server, and a
visualization
module, wherein the remote server is capable of storing the information model
of the
construction and data containing information about the elements of the
construction and the
places coordinates on them for markers. The computing device is capable of
receiving from
the remote server the information model and data by means of the exchange
module and
storing them in the memory module, recognizing markers when performing
scanning after
applying them to the elements of the structure in accordance with the marked
places in the
information model. Also, the computing device at the recognition of the
markers applied on
structure elements unloads the information model and data for displaying the
model by
means of the visualization module in the augmented reality mode. The markers
serve as
reference points for determining the position of the operator, and the
operator can compare
the location of the markers in the model with the current one in real time.
The disadvantageous features of the prototype are the lack of the automated
comparison of the structure model with the current location of its elements,
the lack of the
accuracy of matching markers when scanning them, as well as the lack of the
possibility to
remotely control the correctness of the structure assembly.
The task of the invention is to create a solution integrated with BIM
technology,
which makes it possible to control the construction and assembly of structures
from the
moment of production to assembly from anywhere in the world, to compare the
position of
the structure elements with the design position clearly and automatically.
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 3
Patent
The technical result is to reduce the time of construction and assembly of
structures
due to the capabilities of the computing device.
The specified result is achieved by a system for monitoring construction
processes,
comprising a remote server and at least one computing device connected with
one or more
video cameras for marker scanning, a memory module, a data exchange module for
communication with the remote server, and a visualization module.
In this case, the remote server is capable of storing the three-dimensional
information
model of the structure and the data containing information about the elements
of the
structure, coordinates of the design places on them for applying markers, and
identification
codes of the elements making it possible to reveal information about them, and
also capable
of saving the transmitted data.
The computing device is capable of receiving from the remote server the
information
model and data by means of the exchange module and storing them in the memory
module,
while carrying out the scanning of their recognition markers and the
calculation of distances
to the recognized markers, as well as carrying out the following:
- unloading information about the design elements and their identification
codes after
receiving the data from the remote server, linking the identification codes of
the elements
with the recognized markers before or after applying them to the elements in
accordance
with the coordinates of the design locations, issuing a command to the
visualization module
to display the unloaded information about the elements whose identification
codes are linked
to the markers, storing the data about the performed linkage in the memory
module, and
transmitting them to the remote server via the exchange module;
- when the markers are scanned and data on their connection with
identification codes
and data from the remote server are available, the computing device unloads
information on
the structural elements on which the markers are recognized, with coordinates
of the design
locations for them on the elements, determines the mutual position between the
recognized
markers, compares them with the mutual position of the markers according to
the coordinates
of the design locations, and issues a command to the visualization module to
display the
unloaded information, the results of the comparison, as well as the distances
to the markers
and/or the distances between the markers, storing data about the recognized
markers and
distances to them, as well as about the results of the comparison in the
memory module and
transmitting them to a remote server through the data exchange module.
In a particular case, the computing device, when scanning one or more markers
and
having data about their connection with identification codes and data from a
remote server,
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 4
Patent
can unload information about design elements on which markers are recognized
and create
a three-dimensional model reflecting the current location of design elements
with recognized
markers, issue commands to the visualization module to display the unloaded
information
about elements and the created three-dimensional model, store it in the memory
module, and
transfer it to the remote server via the data exchange module.
Preferably, the computing device, when scanning one or more markers and having
data on their association with identification codes and data from the remote
server, may
unload information about the elements of the structure on which the markers
are recognized,
issue commands to the visualization module to display the distances to the
markers and the
unloaded information, store the data on recognition of one or more markers and
the distances
to them in the memory module, and transmit them to the remote server via the
data exchange
module.
Also, the computing device, when scanning one or more markers and having data
about their association with identification codes and storing the three-
dimensional design
information model in the memory module, may issue commands to the
visualization module
to display at least a portion of the three-dimensional design model showing
the elements on
which the markers are recognized.
More specifically, the computing device, when scanning one or more markers and
having data about their association with identification codes and storing the
three-
dimensional construction information model in the memory module, may issue
commands
to the visualization module to display at least a portion of the three-
dimensional construction
model showing elements on which the markers are recognized in an augmented
reality mode.
Additionally, the system includes a GPS module connected with the computing
device, which is designed to, when scanning one or more markers with the
availability of
data on their relationship to identification codes, calculate distances to
recognized markers
with the assignment of GPS coordinates, store the corresponding data in the
memory module,
and transmit them to a remote server via the data exchange module.
In the case of storing in the memory module the results of recognition of
markers
with the calculation of distances to them after the first scanning, the
computing device may
be able to issue a command to the visualization module when the markers are
scanned again
and a part of them is recognized to display the distances to the locations of
unrecognized or
hidden markers.
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 5
Patent
At least one video camera may be provided in a smartphone, a tablet computer,
a
laptop computer, a surveillance system, virtual reality goggles, augmented
reality goggles,
on a worker's helmet, and/or on a quadcopter.
The use of data containing information about structural elements with
coordinates of
design locations on them for markers makes it possible to know design
distances and angular
positions between the markers placed on these places and elements in the
assembly,
respectively, and the possibility to calculate the distances to them and,
respectively, to the
structure elements when recognizing the markers makes it possible to compare
the design
position of the elements marked in the information model with the actual
location with an
acceptable error. Displaying information on compliance or rejection, as well
as recording
and transmitting such information to a remote server for remote monitoring,
can substantially
reduce the time to find and correct violations, thereby reducing construction
and construction
assembly time.
The present invention is explained with reference to Figs. 1-5.
In Figs. 1a-1b, an example image of a three-dimensional information model of a
structure with marked locations for applying markers according to the data is
given.
Figs. 2a-2b show examples of displaying by the visualization module when
scanning
markers information about the elements on which the markers are recognized,
and the results
of comparing the current mutual position of the markers with the mutual
position according
to the coordinates of the design locations, as well as on the display of
distances to the markers
and between them.
Figs. 3a-3c show examples of how, when markers are scanned, the imaging module
can display distances to markers, information about elements on which markers
are
recognized, and the locations of unrecognized or hidden markers when
rescanning.
Figs. 4a-4b show an example of the ability of the computing device, when
scanning
markers, to generate a three-dimensional model showing the location of
structural elements
relative to each other.
Figs. 5a-5b are photographs showing examples of the visualization module
displaying at least a portion of a three-dimensional model of a structure when
scanning
markers, showing elements on which markers are recognized.
The system according to the present invention can be applied to buildings,
bridges,
ships, airplanes, furniture, and other structures of various sizes and
purposes.
The proposed system for controlling the construction and structural assembly
processes using BIM technology includes a remote server and at least one
computing device
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 6
Patent
connected with one or more video cameras for performing marker scanning, a
memory
module, a communication module for communicating with the remote server, and a
visualization module. The principle of the system operation and the methods of
implementing the system are obvious to a person skilled in the art, as
evidenced by the
description below.
The video camera, memory module, data exchange module, and visualization
module
may be combined with a computing device, which is typically a smartphone, a
laptop
computer, or a tablet computer. A combination of two or more devices is
possible, such as a
laptop and an external camera such as an IP camera, or a camera mounted on a
worker's
helmet or quadcopter.
Accordingly, the video camera may be installed in a smartphone, a tablet
computer,
a laptop, a surveillance system, virtual reality goggles, augmented reality
goggles, on a
worker's helmet, or on a quadcopter. The computing device necessarily contains
a processor
that executes the program code. A memory module is a built-in or external data
storage
device, a data communication module is also a built-in or external modem in
the computing
device that performs mostly wireless communication with the remote server, and
a
visualization module is usually a display built into the computing device,
which is made with
a separate input device, such as a keyboard, or with "touch screen"
technology.
The remote server is designed to store the three-dimensional information model
of
the structure and related data, to store new transferred data, as well as to
provide access to
them to users. Initially, the three-dimensional information model is recorded
on the server
(Fig. la), for example, in fbx format, and connected data, for example, in
json format,
containing information about the elements of the structure, coordinates of
design locations
on them for markers, wherein the coordinates can be viewed on the model (Fig.
lb), as well
as identification codes of the elements, making it possible to identify
information about the
elements.
The information about elements, such as structural beams, includes their names
and
may additionally include at least one of the following: information about
element types,
element sizes, element weights, names of neighboring elements with which the
current
element is in connection, their size, weight, and other things that may be
entered during the
design phase of the three-dimensional model using BIM technology. The data on
the
coordinates of design locations on elements for marker application is
connected with the
information on the dimensions of future markers, which is used to increase the
accuracy of
measuring the distance to them; however, distance measurement is possible
without
information on dimensions.
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 7
Patent
The markers can be applied at any stage, such as the production of elements,
their
storage, transportation, or assembly/construction. The markers are
predominantly graphic
images, usually black and white, of simple shape in the form of a rectangle or
a square with
an identifier-image inscribed inside, but other images may also be used. The
use of similar
images in fiduciary markers is known from the prior art.
With the help of the computing device and visualization module, actions are
performed with the application of all possibilities of the mentioned device or
only part of
them, depending on the need for specific capabilities at the current stage of
the construction
or assembly process, for example, the stages may be storage, transportation,
assembly or
installation, whereby different system capabilities may also be applied at
each said stage, as
will be apparent from the description of the work. It is essential that the
system is in principle
capable of performing all of the functions claimed in the independent claim.
In this case, a
particular user can at this stage apply only part of the capabilities, for
example, used in
linking markers to the identification codes of elements, and another user in
another period
of time ¨ scan the applied markers on the connected elements of the structure
and identify
discrepancies with the design (Figs. 2a-2b).
The computing device is capable of receiving, that is downloading/uploading,
the
information model and the said data from the remote server by means of the
data exchange
module and storing them in the memory module. Any receive-transfer of data by
means of
the exchange module takes place in the presence of communication with the
remote server.
Also, the computing device is able, when scanning markers by means of a video
camera, to carry out their recognition and calculation of distances to the
recognized markers,
including the determination of their angles, which makes it possible to
correctly identify the
position of structural elements. Scanning means the processes of pointing the
video camera
at the marker and processing the video stream in real-time. When recognizing
markers and
calculating distances to them, algorithms known in the art are used, which are
often used in
augmented reality implementation.
Calculation of distances to markers and determination of the mutual position
between
them comes at the expense of using the characteristics of the video camera
sensor. Data on
the focal length and the location on the frame of the point indicating the
offset of the depth
axis of the frame are used. The calculations use marker size data and the
known size of the
marker on the frame in pixels to improve accuracy. Using the above techniques,
which are
input data, by calculating the radius distance from the video camera to the
center of the
marker and calculating the X and Y axis distances in the frame plane from the
center of the
frame to the center of the marker, three-dimensional coordinates of the center
of the marker
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 8
Patent
relative to the video camera are compiled, which are used to calculate the
distance between
the markers.
The system is used when markers are applied to structural elements, before or
after
any marker is directly attached to the element. For this purpose, the
computing device is
capable of unloading the information on structure elements and their
identification codes
after receiving data from the remote server, which means definition of these
data in the
loaded one or several files and granting access to them to the user; then
connecting the
identification codes of the elements with recognized markers before or after
their application
on elements according to coordinates of places and issuing a command to the
visualization
module to display the unloaded information on elements, identification codes
of those are
bound with markers, which is implemented by enabling the user to search and
select an
element from the catalog, for example, by name, which is included in the
concept of element
information, and enabling the user to bind the selected element with the
recognized marker,
which is already attached or will be attached; after which the computing
device stores the
data on the performed binding in the memory module and transmits them to the
remote server
via the exchange module.
Adhesive stencils on which the markers are printed may be used to increase the
accuracy of applying the markers to the elements, with dimensions
corresponding to the
dimensions of the elements. For example, a marker should be applied at a
distance of 0.3 m
from the edge of a construction beam. For this purpose, a stencil is made with
a marker
whose center is at a distance of 0.3 m from the edge of the stencil. The
height of the stencil
also corresponds to the height of the beam. Another option to increase the
accuracy of marker
placement is possible.
Once the markers have been placed, the proposed system can be used to directly
control the construction or assembly processes. At this stage, the remote
server additionally
contains data on the binding of the markers to the element identification
codes.
After downloading by means of the exchange module from the remote server the
data
containing information about the structure elements, data on the performed
linking of
markers with identification codes, and coordinates of design places on the
elements for
markers, scanning of markers with the recognition and calculation of distances
to them is
carried out. The computing device unloads from the data stored in the memory
module the
information about the structure elements on which the markers are recognized,
determines
the mutual position between the recognized markers by calculating the distance
to them, and
compares it with the mutual position of the markers according to the
coordinates of the
design places on the elements.
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 9
Patent
After that, the computing device issues commands to the visualization module
to
display the uploaded information about the elements, for example, their names,
the results
of comparison, for example, by color, where red or yellow ¨ no match, green ¨
corresponds
to the design position (Figs. 2a-2b), as well as to display the distances to
the markers (Figs.
2a-2b ¨ the distance is shown in white numbers in meters next to the markers)
and/or the
distances between them (Figs.2a-2b ¨ the distance is shown between the
markers).
Depending on the implementation of the system, the color of the readings,
style and font
may vary, also only the distances to the markers, only the distances between
the markers, or
both distances may be displayed. The device then stores the recognized markers
and the
calculated distances to them, as well as the comparison results, in the memory
module and
transmits them to the remote server via the data exchange module. This data is
used to control
the construction and structure collection processes by the remote user.
Obviously, the listed basic capabilities of this system allow reducing the
time of
construction/assembly of structures due to the realized principle of instant
control of
conformity of the assembled structure to the project with the output of the
results to the
checker and the remote user.
The computing device can optionally be capable of performing additional
actions
listed below. They extend the functionality of the system, but are not
necessary to achieve
the technical result.
When one or more markers on structural elements are scanned and there are pre-
loaded data containing information about the elements, data on markers'
association with
identification codes and coordinates of design locations on the elements for
markers, the
device is capable of unloading information about the elements on which markers
are
recognized and creating a three-dimensional model that reflects the current
location of
structural elements on which markers are recognized, according to the
calculated distances
to them, issuing a command to the visualization module to display the uploaded
information
about the elements and the created three-dimensional model (Figs.4a-4b),
storing it in a
memory module, and transferring it to a remote server via a data exchange
module. This
mode is mainly used to monitor structural elements during storage and/or
transportation.
When one or more markers are scanned and there is pre-loaded data containing
information about elements, data on markers' connection with identification
codes and
coordinates of design locations on elements for markers, the computing device
can perform
unloading of information about structural elements, on which markers are
recognized with
calculation of distances to them, issuing a command to the visualization
module to display
distances to markers and unloaded information (Figs. 3b, 3c), saving data on
recognition of
one or more markers (Figs. 3b, 3c). This mode is mainly used for monitoring
structure
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 10
Patent
elements during storage and/or transportation, viewing information about
elements, which
in addition to their names can contain data on weight, dimensions, and names
of neighboring
elements, in connection with which the current element is located, their size
and weight,
which will help to store, for example, construction beams, which will be in
connection, next
to each other, to determine the maximum load of the vehicle by weight
information and
many other things.
When one or more markers are scanned and a preloaded three-dimensional
construction information model and data containing, among other things, marker
association
data with identification codes are available, the computing device may upload
the three-
dimensional model and command the visualization module to display at least a
portion of
the three-dimensional model showing elements on which one or more markers are
recognized (Figs. la, 5a, 5b). This mode may be used to view the model to
identify
neighboring elements, their design location, and others. In addition,
visualization in this
mode can be performed in augmented reality, that is, the user, for example, in
the appropriate
glasses scans markers and at the same time sees the design three-dimensional
model of the
structure, superimposed on the actually located elements. This mode is
convenient at the
stage of construction installation.
The claimed system may include a GPS module connected with a computing device,
which is additionally designed to, when scanning one or more markers and
having pre-
loaded data containing, among other things, data on the markers' association
with
identification codes, to calculate distances to the recognized markers and
assign GPS
coordinates to them, store the corresponding data in a memory module, and
transmit them
to a remote server via a data exchange module. GPS coordinates are assigned to
markers
with correction for the distance to them from the device with a GPS module.
This mode can
be used for monitoring the structural elements during storage and/or
transportation, their
territorial position is checked. It is preferably combined with other modes,
in particular, with
the mode in which distances to recognized markers are calculated and
displayed.
In the case of storing in the memory module the results of the recognition of
markers
with the calculation of distances to them after the first scan, the device can
perform the
issuance of commands to the visualization module in the second scanning of
markers and
recognizing part of them to display distances to the locations of unrecognized
or hidden
markers (Fig. 3a ¨ blue location indicators in the upper part). This feature
will speed up the
search for the necessary elements.
The proposed system works as follows.
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 11
Patent
Using BIM technology, a three-dimensional information model of a structure,
such
as a building frame, is developed. Each element, for example, each structural
beam, is
assigned an identification code, by which it is possible to identify
information about the
corresponding element ¨ its name and in some cases may be at least one of the
following:
type, size, weight, names of neighboring elements, in connection with which
the current
element is located, their size and weight. The information model and data
comprising the
said information, identification codes, and coordinates of locations for
markers are stored on
a remote server.
For example, in the manufacturing stage of the structural elements, a graphic
marker
is applied to each beam by gluing or otherwise according to the design
coordinates, which
are preloaded as part of the data by the computing device and viewed by the
user. To increase
the accuracy of marker application, the markers may be produced on a stencil
whose
dimensions, when applied to the edge of the design element, allow the marker
to be
accurately applied. Markers can also be applied at the stage of
assembly/assembly of the
structure.
Before or after the markers are applied, the markers are recognized by
scanning, and
each marker is linked to the corresponding element identification code.
Binding is performed
by selecting the element according to the information displayed by the
visualization module
about structural elements, for example, by searching by the name of the beam.
The binding
data is saved and sent to a remote server if communication is available.
Further, at the stage of construction installation, the worker performs
recognition of
markers by scanning, for example, with a video camera of a smartphone, from
the
downloaded data containing information about binding of markers, information
about the
corresponding elements on which markers are recognized is automatically
uploaded,
distances to markers in the field of view of a video camera, their angular
positions are
calculated, the mutual position between them is determined, comparison with
the mutual
position of markers according to coordinates of design locations is carried
out and shown on
the display, for example, a smartphone, information about the elements ¨ the
name, for
example, "B3-1" (Figs. 2a-2b, 3a-3b, 4a), and the results of comparison with
the position of
markers in the information model, for example, by highlighting correctly
installed beams in
green, and red ¨ wrong (Figs. 2a-2b), since the distance between the markers
does not match
the distance according to the project, taking into account the allowable
deviation. Distances
to markers and/or distances between them are also displayed, depending on the
installation
system.
CA 03211568 2023- 9-8
Doc. No. 374-1 CA/PCT 12
Patent
The marker recognition and comparison results are then stored in a memory
module
and transmitted to a remote server via a data exchange module.
Additionally, the worker can observe the assembly process and, when scanning
and
recognizing markers, view at least parts of a three-dimensional model of the
structure
showing the elements on which one or more markers are recognized, that is, the
design three-
dimensional model (Figs. la, 5a-5b). For this purpose, when a marker is
recognized on a
design element and information about it is displayed, the user marks the
element as an anchor
element. After that, at least a part of the design model of the structure is
displayed on a
display, for example, a smartphone. Alternatively, the design model may be
displayed in a
real-time augmented reality mode.
The advantage of the proposed solution is the possibility of the checker to
address at
any moment the information model and all the data and results of recognition
with distance
calculation stored on the remote server, and to see the stage of construction
or assembly of
the structure, the degree of completion, detected violations and deviations
from the design
in the process of assembly, the position of elements, and when using the
possibility of linking
to GPS coordinates also the territorial location of the elements of the
structure on the map.
Thus, the use of the system according to the present invention will allow
recognizing
errors in the assembly of structures at an early stage and promptly making a
decision on the
method of elimination, including through remote monitoring, to track the
processes of
storage and transportation, to place the elements depending on the order of
assembly of the
structure, to minimize the risks of incorrect assembly due to the output of
information when
recognizing markers in real-time, which significantly reduces construction
time.
CA 03211568 2023- 9-8
