Note: Descriptions are shown in the official language in which they were submitted.
SYSTEM AND METHOD OF SYNCHRONIZING DATA IN NEO4J TO SEARCH SERVER
Field
[0001] The present disclosure relates to technical field of image database
field, particularly to method and
system of synchronizing data in Neo4j to search server.
Background
[0002] Neo4j image database is a very popular image database that provides
powerful and stable image
storage and image query functions, which is widely used in knowledge graph
application, and search server
is often used to provide support for search service of application system.
[0003] Graph visualization is an application field of knowledge graph, which
usually includes services
such as search query and display of image data. In the visualization
application scenario of the knowledge
graph, the search server can be used to assist in the construction of query
services for image data, and the
image data can be synchronized to the search server cluster, and then the
image data can be efficiently
searched from the search server to avoid directly scanning the full amount of
data from Neo4j library,
thereby improving the performance of the search service for image data.
Therefore, it is a problem that must
be solved to synchronize the image data in Neo4j to search server in real
time.
Invention Content
[0004] The purpose of the present invention is to provide a method and system
of synchronizing data in
Neo4j to a search server and solve the problem of how to synchronize image
data in Neo4j to search server
in real time.
[0005] To achieve the above-mentioned purpose, the present invention provides
following solutions:
1
Date Recue/Date Received 2021-12-24
[0006] A method of synchronizing data in Neo4j to search server, wherein, the
method comprises:
[0007] Starting Neo4j image database, creating a search sever client,
constructing a transaction event
processor, and registering the transaction event processor to Neo4j service;
[0008] Using the transaction event processor to respectively construct image
data node and image data
relationship into document request object of search server, then storing in
Map;
[0009] Using the transaction event processor to construct a batch of document
request objects of the search
server based on the document request object, using search server client to
synchronize a batch of document
request objects to search server cluster.
[0010] Preferably, the search server adopts Elasticsearch search server.
[0011] Preferably, creating a search server client, constructing a transaction
event handler, and registering
the transaction event handler to the Neo4j service, the method comprises:
[0012] Starting Neo4j image database, reading search server cluster address
and search server index name
in Neo4j image database configuration file;
[0013] Creating a search server client according to the search server cluster
address;
[0014] Using the search server client to check whether the search server index
name in the Neo4j image
database configuration file exists, if not exists, creating search server
index name;
[0015] Constructing a transaction event processor according to the Neo4j image
database configuration
2
Date Recue/Date Received 2021-12-24
file and registering the transaction event processor to Neo4j service.
[0016] Preferably, using the transaction event processor to respectively
construct image data node and
image data relationship into document request object of search server, then
storing in Map, the method
comprises:
[0017] Creating a Map object in the transaction event processor, and the Map
object adopts a key-value set
mapping method;
[0018] Respectively constructing image data node and image data relationship
in Neo4j image database
into document request objects of search server;
[0019] Storing object constructed by entity id and index type of the image
data node as the key to Map
object, and/or
[0020] Storing the object constructed by entity id and index type of the image
data relationship as the key
to Map object;
[0021] Storing the document request object as value corresponding to Map
object.
[0022] Specifically, the method of separately constructing image data nodes in
Neo4j image database into
document request objects of search server, comprising:
[0023] Determining whether image data nodes need to be synchronized according
to Neo4j image database
configuration file, if image data nodes need to be synchronized, perform one
or more of following
operations to synchronize image data nodes:
3
Date Recue/Date Received 2021-12-24
[0024] Constructing the image data nodes created in the transaction object of
Neo4j image database into
index request document objects of search server;
[0025] Constructing the image data nodes deleted from the transaction object
of Neo4j image database into
delete request document object of search server;
[0026] Constructing newly-labeled image data nodes in the transaction object
of Neo4j image database
into index request document object of search server;
[0027] Constructing the image data nodes with delete label in the transaction
object of Neo4j image
database into delete request document object of search server;
[0028] Constructing the image data nodes with newly added attribute in the
transaction object of Neo4j
image database into index request document object of search server;
[0029] Constructing the image data nodes with delete attribute in the
transaction object of Neo4j image
database into the delete request document object of search server.
[0030] Specifically, the method of separately constructing image data
relationships, wherein the image
data relationships need to be synchronized in Neo4j image database into
document request objects of search
server, comprising:
[0031] Determining whether the image data relationships need to be
synchronized according to Neo4j
image database the configuration file, if image relationship nodes need to be
synchronized, perform one or
more of the following operations to synchronize image data relationships:
[0032] Constructing the image data relationships created in the transaction
object of Neo4j image database
4
Date Recue/Date Received 2021-12-24
into index request document objects of search server;
[0033] Constructing the image data relationships deleted from the transaction
object of Neo4j image
database into delete request document objects of search server;
[0034] Constructing newly-labeled image data relationships in the transaction
object of Neo4j image
database into index request document objects of search server;
[0035] Constructing the image data relationships with delete attribute in the
transaction object of Neo4j
image database into the delete request document objects of search server.
[0036] Furthermore, using the transaction event processor to construct a batch
of document request objects
of search server based on the document request object, the method of using the
search server client to
synchronize a batch of document request objects to search server, comprising:
[0037] Using the transaction event processor to construct a batch if document
request objects of search
server based on the document request object;
[0038] Search server client uses an asynchronous request mode or a synchronous
request mode to perform
image data synchronization operations and synchronizes a batch of document
request objects to search
server cluster.
[0039] Preferably, according to Neo4j's configuration file, determining of
search server client adopts
asynchronous request mode or synchronous request mode to perform image data
synchronization
operations.
[0040] Preferably, a batch of document request objects are a Bulk request
object, the asynchronous request
Date Recue/Date Received 2021-12-24
mode is an asynchronous Bulk request mode, and the synchronous request mode is
synchronous Bulk
request method.
[0041] A system of synchronizing data in Neo4j to a search server, the system
comprises an initialization
module, a mapping module and a synchronizing module, wherein,
[0042] The initialization module configured to start Neo4j image database,
create a search sever client,
constructing a transaction event processor, and register the transaction event
processor to Neo4j service;
[0043] The mapping module configured to use the transaction event processor to
respectively construct
image data node and image data relationship into document request object of
search server, then store in
Map;
[0044] The synchronizing module configured to use the transaction event
processor to construct a batch of
document request objects of the search server based on the document request
object and use search server
client to synchronize a batch of document request objects to search server
cluster.
[0045] Compared with the prior art, the method and system for synchronizing
data in Neo4j to search
server provided by the present invention has the following beneficial effects:
[0046] The method of synchronizing data in Neo4j to a search server provided
by the present invention,
the search server client and transaction event processor are first created in
Neo4j image database, then using
the transaction event processor to construct the image data node and the image
data relationship into the
document request object of the search server and storing in Map, finally,
using the transaction event
processor to construct a batch of document request objects of search server,
and using the search server
client to synchronize a batch of document request objects to search server
cluster. This allows Neo4j to
automatically synchronize changed image data after submitted data is written
into the transaction to search
6
Date Recue/Date Received 2021-12-24
server cluster when performing writing operation, thereby helping developers
to use Neo4j image database
and search server to build high-quality graph visualization application.
[0047] The system of synchronizing data in Neo4j to a search server provided
by the present invention
adopts the above-mentioned method of synchronizing data in Neo4j to a search
server and solves the
problem of how to synchronize image data in Neo4j to a search server in real
time.
Drawing Description
[0048] The drawings described here are used to provide a further understanding
of the present invention
and constitute a part of the present invention. The exemplary implementations
of the present invention and
their descriptions are used to explain the present invention, and do not
constitute an improper limitation of
the present invention. In the attached figures:
[0049] Figure 1 is a process diagram of a method of synchronizing data in
Neo4j to search server in
implementation of the present application;
[0050] Figure 2 is a process diagram of starting Neo4j image database in
implementation of the present
application.
[0051] Figure 3 is a process diagram of image data synchronizing in
implementation of the present
application;
[0052] Figure 4 is a system module diagram of a method of synchronizing data
in Neo4j to search server
in implementation of the present application.
Specific implementation methods
7
Date Recue/Date Received 2021-12-24
[0053] To make clearer purpose, characteristic and benefit of the above-
mentioned invention, the following
will describe the technical solutions of the implementations in the present
application with accompanying
drawings, obviously the described implementations are only a part of the
implementations in the present
application. Based on the implementations in the present application, all
other implementations obtained
by those of ordinary skilled in the art will fall in the protection scope of
the present application.
[0054] Implementation one
[0055] Please refer to Figure 1, a method of synchronizing data in Neo4j to a
search server, comprising:
[0056] Starting Neo4j image database, creating a search sever client,
constructing a transaction event
processor, and registering the transaction event processor to Neo4j service;
[0057] Using the transaction event processor to respectively construct image
data node and image data
relationship into document request object of search server, then storing in
Map;
[0058] Using the transaction event processor to construct a batch of document
request objects of the search
server based on the document request object, using search server client to
synchronize a batch of document
request objects to search server cluster.
[0059] The method of synchronizing data in Neo4j to a search server provided
in this implementation
enables Neo4j to automatically convert changed image data into a batch
document request objects after
submitted data is written into the transaction when performing a write
operation, furthermore, using search
server client to synchronize a batch of document request objects to search
server cluster, thereby helping
developers to use the Neo4j image database and search server to build high-
quality graph visualization
applications.
8
Date Recue/Date Received 2021-12-24
[0060] In specific implementation, the search server can use the Elasticsearch
search server. The
Elasticsearch search server provides an open-source distributed document
storage and analysis search
engine, which can provide efficient index query capabilities, and has a wide
range of applications in the
field of big data search, which is often used to provide support for the
search service of the application
system.
[0061] In addition, the method in this implementation can be developed based
on Neo4j image database
community edition (hereinafter referred to as Neo4j) and implemented using the
Java programming
language. Neo4j has a core extension factory interface KernelExtensionFactory,
this interface will create a
Neo4j life cycle object, this Neo4j life cycle object can perform custom
extension operations when Neo4j
is started and closed. In addition, Neo4j will trigger a data writing
transaction when performing a writing
operation, the transaction event processor executes corresponding methods in
the three phases before the
start of the data writing transaction, after the transaction is committed, and
after the transaction is rolled
back, to synchronize the Neo4j middle map data to the search server. Neo4j
saves transaction data in a
transaction object (TransactionData) which is passed as a parameter to the
method of the transaction event
handler interface.
[0062] The method of synchronizing data in Neo4j to a search server provided
by an implementation of
the present invention uses these two mechanisms of Neo4j to read the search
server cluster and search in
the neo4j configuration file (neo4j.conf file) during server index
configuration, initialize the search server
client (for example: Elasticsearch client) and create index. When Neo4j
performs writing operation, in the
method of before data writing transaction is executed (beforeCommit() method),
extracting image data in
transaction object, and in the method after the data write transaction is
committed (afterCommit() method)
to synchronize extracted image data to search server cluster. Among them, the
method of creating a search
server client, constructing a transaction event handler, and registering
transaction event handler to Neo4j
service comprises:
9
Date Recue/Date Received 2021-12-24
[0063] Starting Neo4j image database, reading search server cluster address
and search server index name
in the configuration file of Neo4j image database;
[0064] Creating search server client according to the search server cluster
address;
[0065] Using search server client to check whether search server index name in
the configuration file of
Neo4j image database exists, and if not exists, create index name of search
server;
[0066]
Constructing a transaction event handler (ElasticSearchEventHandler) according
to the
configuration file of Neo4j image database and registering the transaction
event handler in Neo4j service.
[0067] The process of constructing a transaction event processor specifically
includes: configuring a cluster
of search servers (a plurality of nodes are separated by commas), configuring
index name of search server
(synchronized image data is saved in this index), configuring number of index
shards, configuring number
of index copies, configuring propagation mode of Elasticsearch, configuring
whether to synchronize node
data, whether to synchronize relational data, and whether to configure to
synchronize data in an
asynchronous manner, etc.
[0068] After search server client and transaction event processor are created,
all data packaging and
synchronization actions are performed in transaction event processor. The
method of using transaction
event processor to construct image data node and image data relationship into
document request object of
search server and storing in Map, comprising:
[0069] Creating a Map object in the transaction event processor, and the Map
object adopts a key-value set
mapping method;
[0070] Respectively constructing image data node and image data relationship
in Neo4j image database
Date Recue/Date Received 2021-12-24
into document request objects of search server;
[0071] Storing object constructed by entity id and index type of the image
data node as the key to Map
object, and/or
[0072] Storing the object constructed by entity id and index type of the image
data relationship as the key
to Map object;
[0073] Storing the document request object as value corresponding to Map
object.
[0074] It is possible that a certain Neo4j transaction event processor only
processes image data nodes or
only image data relationships, accordingly, it is only necessary to store
object constructed by entity id and
index type of image data node as a Map object, or storing object constructed
by entity id of image data
relationship and index type as key of the Map object, and then storing the
document request object as the
value corresponding to the key, so as to avoid establishing too many key
values, causing resource waste
and improving index complexity, etc.
[0075] Since the data structure of image data includes image data nodes and
image data relationships, it is
necessary to construct image data nodes and image data relationships into
document request objects of
search server respectively. In the actual process of synchronizing image data,
first constructing image data
nodes in Neo4j image database into the document request objects of search
server. The specific method
includes:
[0076] Determining whether image data nodes need to be synchronized according
to Neo4j image database
configuration file, if image data nodes need to be synchronized, perform one
or more of following
operations to synchronize image data nodes:
11
Date Recue/Date Received 2021-12-24
[0077] Constructing the image data nodes created in the transaction object of
Neo4j image database into
index request document objects of search server;
[0078] Constructing the image data nodes deleted from the transaction object
of Neo4j image database into
delete request document object of search server;
[0079] Constructing newly-labeled image data nodes in the transaction object
of Neo4j image database
into index request document object of search server;
[0080] Constructing the image data nodes with delete label in the transaction
object of Neo4j image
database into delete request document object of search server;
[0081] Constructing the image data nodes with newly added attribute in the
transaction object of Neo4j
image database into index request document object of search server;
[0082] Constructing the image data nodes with delete attribute in the
transaction object of Neo4j image
database into the delete request document object of search server.
[0083] After image data node synchronization is completed, the image data
relationships that need to be
synchronized in the Neo4j image database are respectively constructed into
document request objects of
search server. The specific method includes:
[0084] Determining whether the image data relationships need to be
synchronized according to Neo4j
image database the configuration file, if image relationship nodes need to be
synchronized, perform one or
more of the following operations to synchronize image data relationships:
[0085] Constructing the image data relationships created in the transaction
object of Neo4j image database
12
Date Recue/Date Received 2021-12-24
into index request document objects of search server;
[0086] Constructing the image data relationships deleted from the transaction
object of Neo4j image
database into delete request document objects of search server;
[0087] Constructing newly-labeled image data relationships in the transaction
object of Neo4j image
database into index request document objects of search server;
[0088] Constructing the image data relationships with delete attribute in the
transaction object of Neo4j
image database into the delete request document objects of search server.
[0089] In specific implementation process, the image data relationships that
need to be synchronized in
Neo4j image database can also be constructed as the document request objects
of search server, and then
the image data nodes in Neo4j image database can be separately constructed as
document request objects
of search server, the requested document objects and indexes are saved in Map
object in real time, and the
corresponding value is returned according to key value index when it needs to
be called, which is
convenient, fast and not easy to make mistakes.
[0090] After obtaining document request objects, using the transaction event
processor to construct a batch
of document request objects of search server based on the document request
object, the method of using the
search server client to synchronize a batch of document request objects to
search server, comprising:
[0091] Using the transaction event processor to construct a batch if document
request objects of search
server based on the document request object;
[0092] Search server client uses an asynchronous request mode or a synchronous
request mode to perform
image data synchronization operations and synchronizes a batch of document
request objects to search
13
Date Recue/Date Received 2021-12-24
server cluster.
[0093] Finally, according to Neo4j configuration file, it is determined that
search server client adopts an
asynchronous request mode or a synchronous request mode to perform image data
synchronization
operation. A batch of document request objects are Bulk request objects, the
asynchronous request mode is
an asynchronous Bulk request mode, and the synchronous request mode is
synchronous Bulk request
method. For example, according to Neo4j configuration file to determine
whether to use asynchronous
request method, if yes, the search server client uses asynchronous Bulk
request method to synchronize a
batch of document request objects to search server cluster, if not, search
server client uses synchronous
Bulk request to synchronize a batch of document request objects to search
server cluster. In specific
implementation, the Setting<T> class provided by Neo4j is used to read the
required configuration values
from neo4j.conf file according to configuration item name. The method for
synchronizing data in Neo4j to
a search server provided by the present invention solves the problem from the
field of graph visualization
applications, when Neo4j and search server are used to construct image search
service, the image data
cannot be automatically synchronized to the search server cluster in real time
and help development people
to use Neo4j and search server to build high-quality graph visualization
applications.
[0094] Implementation two
[0095] Please refer to Figure 4, the present implementation provides a system
of synchronizing data in
Neo4j to a search server, the system comprises an initialization module, a
mapping module and a
synchronizing module, wherein, the initialization module configured to start
Neo4j image database, create
a search sever client, constructing a transaction event processor, and
register the transaction event processor
to Neo4j service; the mapping module configured to use the transaction event
processor to respectively
construct image data node and image data relationship into document request
object of search server, then
store in Map; the synchronizing module configured to use the transaction event
processor to construct a
batch of document request objects of the search server based on the document
request object and use search
14
Date Recue/Date Received 2021-12-24
server client to synchronize a batch of document request objects to search
server cluster. The system of
synchronizing data in Neo4j to search server can be integrated into the Neo4j
image database in form of a
plug-in, so that when Neo4j performs a writing operation, it can automatically
convert the changed image
data after the submitted data is written into the transaction into a batch of
document request objects, and
then use the search server client to synchronize a batch of document request
objects to search server cluster.
[0096] The system of synchronizing data in Neo4j to the search server provided
by the present invention
adopts the method of synchronizing data in Neo4j to search server in the
implementation one above, which
solves the problem of how to synchronize the image data in Neo4j to search
server in real time. Compared
with the prior art, the beneficial effects of the system of synchronizing data
in Neo4j to search server
provided by the implementation of the present invention are the same as the
beneficial effects of the method
of synchronizing data in Neo4j to search server provided in the implementation
one, in addition, other
technical features in the system are the same as those disclosed in the method
of the previous
implementations and will not be repeated here.
[0097] In the description of the foregoing implementations, specific features,
structures, materials, or
characteristics can be combined in any one or more implementations or examples
in a suitable manner.
[0098] The above-mentioned are only specific implementations of the present
invention, but the protection
scope of the present invention is not limited thereto. Any skilled in the art
can easily think of changes or
changes within the technical scope disclosed by the present invention, all
replacements shall be covered
within the protection scope of the present invention. Therefore, the
protection scope of the present invention
should be subject to the protection scope of the claims.
Date Recue/Date Received 2021-12-24
