Note: Descriptions are shown in the official language in which they were submitted.
LOG STORAGE METHOD, APPARATUS, AND COMPUTER SYSTEM
Field
[0001] The present disclosure relates to computer technology filed,
particularly to a log storage method,
apparatus, and system.
Background
[0002] During system operation, a system log is generated. In the period of
small-scale system, the number
of generated system logs is small, the original text of the generated logs can
be stored directly, and the
operation and maintenance of the system logs can be manually performed by the
operation and maintenance
people. However, with the rapid expansion of business requirements and the
diversified development of
services, the system has become more and more complex, a system can include a
plurality of devices and
deploy a plurality of instances, resulting in inconsistent log data
structures, when system has problems,
operation and maintenance people are required to manually locate the cause of
the problem, which is not
only very difficult but also inefficient.
Invention Content
[0003] To solve the present technical problems, the present application
provides a log storage method,
apparatus and system. The technical solutions are as following:
[0004] To achieve the above-mentioned purpose, the first aspect is providing a
log storage method,
comprising:
[0005] Obtaining a to-be-processed log in a preset format generated by a
preset service platform and
configuration data corresponding to the preset service platform, the to-be-
processed log includes a key-
value pair and a preset identifier, and the key-value pair is used to store
historical operation records of the
1
Date Recue/Date Received 2021-12-24
preset service platform, the configuration data include meaning corresponding
to each data key,
replacement rule corresponding to preset data key, and index library address
corresponding to each preset
identifier;
[0006] Parsing the to-be-processed log, recognizing data key and data value
corresponding to the data
keys included in the to-be-parsed log;
[0007] When included data key pair includes preset data key, modifying data
value corresponding to the
included preset data key according to the corresponding replacement rule;
[0008] Generating historical operation record according to both meaning and
data value corresponding to
each data key;
[0009] Storing historical operation record in index library address
corresponding to the preset identifier.
[0010]
In some implementations, storing historical operation record in index library
address
corresponding to the preset identifier, comprising:
[0011] Sending data storage request to preset system corresponding to the
index library address, so that
the preset system returns data access authority according to the data storage
request;
[0012] When receiving the returned data access authority, storing the
historical operation record in the
index library address.
[0013] In some implementations, the method also comprises:
[0014] Receiving log reading request sent by user;
2
Date Recue/Date Received 2021-12-24
[0015] Generating visual log according to historical operation record
corresponding to the log reading
request;
[0016] Returning the visual log to the user.
[0017] In some implementations, the to-be-processed log is in Json format.
[0018] In some implementations, obtaining a to-be-processed log in a preset
format generated by a preset
service platform and configuration data corresponding to the preset service
platform, comprising:
[0019] Using Flume log collection system to obtain the to-be-processed log in
the preset format generated
by the preset service platform and the configuration data corresponding to the
preset service platform;
[0020] Parsing the to-be-processed log, recognizing data key and data value
corresponding to the data key
included in the to-be-parsed log, comprising:
[0021] Using Milk streaming computing engine to parse the to-be-processed log,
recognizing the data key
and the data value corresponding to the data key included in the to-be-parsed
log.
[0022] The second aspect, the present invention provides a log storage method,
the method comprises:
[0023] Sending generated to-be-processed log to preset processing terminal, so
that the preset processing
terminal receives the to-be-processed log and obtains corresponding
configuration data, the to-be-processed
log includes a key-value pair and a preset identifier, and the key-value pair
is used to store the historical
operation record of the preset service platform, the configuration data
includes the meaning corresponding
to each data key, the replacement rule corresponding to the preset data key,
and the index library address
3
Date Recue/Date Received 2021-12-24
corresponding to each preset identifier; parsing the to-be-processed log,
recognizing data key and data value
corresponding to the data key included in the to-be-parsed log; when the
included data key pair includes
preset data key, modifying data value corresponding to the included preset
data key according to the
corresponding replacement rule; storing the historical operation record to the
index library address
corresponding to the preset identifier.
[0024] The third aspect, the present invention provides a log storage
apparatus, the apparatus comprises:
[0025] An obtaining module configured to obtain a to-be-processed log in a
preset format generated by a
preset service platform and configuration data corresponding to the preset
service platform, the to-be-
processed log includes a key-value pair and a preset identifier, and the key-
value pair is used to store
historical operation record of the preset service platform, the configuration
data include meaning
corresponding to each data key, replacement rule corresponding to preset data
key, and index library address
corresponding to each preset identifier;
[0026] A parsing module configured to parse the to-be-processed log,
recognizing data key and data value
corresponding to the data key included in the to-be-parsed log; when included
data key pair include preset
data key, modifying data value corresponding to the included preset data key
according to the corresponding
replacement rule; generating historical operation record according to both
meaning and data value
corresponding to each data key;
[0027] A storage module configured to store historical operation record in
index library address
corresponding to the preset identifier.
[0028] The fourth aspect, the present invention provides a computer system,
the system comprises:
[0029] One or a plurality of processors;
4
Date Recue/Date Received 2021-12-24
[0030] A memory associated with one or a plurality of processors, the memory
is configured to store
program commands, if the program commands are executed by one or a plurality
of processors, executing
following operations:
[0031] Obtaining a to-be-processed log in a preset format generated by a
preset service platform and
configuration data corresponding to the preset service platform, the to-be-
processed log includes a key-
value pair and a preset identifier, and the key-value pair is used to store
historical operation record of the
preset service platform, the configuration data include meaning corresponding
to each data key,
replacement rule corresponding to preset data key, and index library address
corresponding to each preset
identifier;
[0032] Parsing the to-be-processed log, recognizing data key and data value
corresponding to the data key
included in the to-be-parsed log;
[0033] When included data key pair include preset data key, modifying data
value corresponding to the
included preset data key according to the corresponding replacement rule;
[0034] Generating historical operation record according to both meaning and
data value corresponding to
each data key;
[0035] Storing historical operation record in index library address
corresponding to the preset identifier.
[0036] The beneficial effects achieved by the present invention are:
[0037] The present invention discloses to a log storage method, including
obtaining to-be-processed log
in preset format generated by preset service platform and configuration data
corresponding to preset service
Date Recue/Date Received 2021-12-24
platform, to-be-processed log includes key-value pair and preset identifier,
key-value pair is used to store
historical operation record, the configuration data include meaning
corresponding to each data key,
replacement rule corresponding to preset data key, and index library address
corresponding to each preset
identifier; the configuration data include meaning corresponding to each data
key, replacement rule
corresponding to preset data key, and index library address corresponding to
each preset identifier; parsing
to-be-processed log, recognizing data key and data value corresponding to the
data key included in to-be-
parsed log; when included data key pair includes preset data key, modifying
data value corresponding to
included preset data key according to corresponding replacement rule;
generating historical operation
record according to both meaning and data value corresponding to each data
key; storing historical
operation record in index library address corresponding to preset identifier;
storing historical operation
record in index library address corresponding to preset identifier improves
log processing efficiency, and
when need to consult the log later, people can directly obtain the stored
parsed log without re-analyzing the
log, non-operations and maintenance people can also understand the parsed log,
which is convenient for
non-operations and maintenance people to deal with problems caused by the
system and improve system
problem-solving efficiencies.
Drawing Description
[0038] In order to describe the technical solutions clearer in the
implementations of the present
application or the prior art, the following are drawings that need to be used
are briefly introduced.
Obviously, the drawings in the following description are only some
implementations of the application, for
those of ordinary skill in the art , without creative work, they can also
obtain other drawings based on these
drawings.
[0039] Figure 1 is a service workflow diagram of a log storage system in
implementation of the present
application;
6
Date Recue/Date Received 2021-12-24
[0040] Figure 2 is a method process diagram in implementation of the present
application;
[0041] Figure 3 is a structural diagram of apparatus in implementation of the
present application.
[0042] Figure 4 is a structural diagram of computer system in implementation
of the present application.
Specific implementation methods
[0043] The following will describe the technical solutions of the
implementations in the present
application with accompanying drawings, obviously the described
implementations are only a part of the
implementations in the present application. Based on the implementations in
the present application, all
other implementations obtained by those of ordinary skilled in the art will
fall in the protection scope of the
present application.
[0044] As described in the background art, in order to solve the above-
mentioned technical problems, this
application proposes a log storage system, which includes a service platform
for generating logs and a
server for storing logs. The system can include a plurality of service
platforms, and each service platform
is preset with the same log format. Preferably, the log can be in Json format.
Implementation one
[0046] As shown in Figure 1, the storage system performs process of log
storage and query comprising:
[0047] Step 1, the service platform generates a log in a preset format and
sends it to server;
[0048] The generated log can include prefix and Json string. Each prefix
uniquely corresponds to an index
library. The Json string stores historical operating record of the platform in
the format of key-value pair.
7
Date Recue/Date Received 2021-12-24
The historical operation record can include the service platform's operation
process parameter, executed
data operation, etc., which are used to record the platform's operation
situation and usage which is not
limited in this application.
[0049] Step 2, the service terminal uses Flume log collection system to obtain
the configuration data
corresponding to the service platform;
[0050] Flume is a system for highly available, highly reliable, distributed
massive log collection,
aggregation and transmission, Flume supports customizing various data senders
in the log system to collect
data; at the same time, Flume provides ability to simple process data and
write to various data recipients.
[0051] Each service platform has corresponding configuration data, the
configuration data includes the
meaning of each data key contained in the log generated by the service
platform, the field type of each key-
value pair, whether each data key is displayed, the enumeration value
corresponding to the preset data key
and the meaning corresponding to the enumeration value.
[0052] The configuration data may include the meaning of the enumerated value
corresponding to the
preset data key. For example, the configuration data can specify that the
value corresponding to a certain
data key can be 1 or 0, when the value corresponding to the data key is 0,
which means the purchase has
failed, and the value 1 means the purchase is successful, etc., this
application does not add to this limited.
The configuration data can also include filtering rules, and the data values
corresponding to the data keys
contained in the filtering rules are not stored.
[0053] Step 3, the server uses Flink streaming to obtain the data value
corresponding to each data key
contained in the log and determines the effects of the data value
corresponding to each data key according
to the meaning of each data key.
8
Date Recue/Date Received 2021-12-24
[0054] Flink is an open-source stream processing framework whose core is a
distributed stream data
stream engine written in Java and Scala. Flink executes arbitrary stream data
programs in data parallel and
pipeline mode, Flink's pipeline nmtime system can execute batch and stream
processing programs.
[0055] According to the effects of the data value corresponding to each data
key, the server can obtain
the historical operation record of the service platform contained in the to-be-
stored log. For example, the
data value corresponding to a certain data key is used to indicate the user's
purchase amount on the service
platform, the server can determine the user's purchase amount according to the
data value corresponding to
the data key in the log and generate a purchase record. The historical
operation record includes the purchase
record.
[0056] Preferably, the historical operation record can be generated in any
format, which is not limited in
this application.
[0057] Step 4: The server determines the index library corresponding to the
log according to the prefix;
[0058] Since each prefix uniquely corresponds to an index library, the server
can determine the
corresponding to-be-stored index library according to the prefix. The index
library can be deployed on any
storage system or device such as ElasticSearch, which is not limited in this
application.
[0059] Step 5, the server requests data access permission from the index
library, when obtaining the data
access permission, the historical operation record is stored in the
corresponding index library.
[0060] Step 6, the server receives the user's log query request;
[0061] The server determines to-be-read log according to user's log query
request and locates the index
library where the to-be-read log is located.
9
Date Recue/Date Received 2021-12-24
[0062] Step 7, the server generates a corresponding visual log according to
the obtained corresponding
historical operation record and the preset template, then returns the visual
log to the user.
[0063] The server requests data access permissions from the index library, and
when obtaining the data
access permissions, obtaining the corresponding historical operation record
from the index library.
Implementation two
[0065] For the above-mentioned implementation, the present invention provides
a log storage method, as
shown in figure 2, the method includes:
[0066] 210, obtaining a to-be-processed log in a preset format generated by a
preset service platform and
configuration data corresponding to the preset service platform, the to-be-
processed log includes a key-
value pair and a preset identifier, and the key-value pair is used to store
historical operation records of the
preset service platform, the configuration data include meaning corresponding
to each data key,
replacement rule corresponding to preset data key, and index library address
corresponding to each preset
identifier;
[0067] 220, parsing the to-be-processed log, recognizing data key and data
value corresponding to the
data keys included in the to-be-parsed log;
[0068] Preferably, obtaining a to-be-processed log in a preset format
generated by a preset service
platform and configuration data corresponding to the preset service platform,
comprising:
[0069] 221, using Flume log collection system to obtain the to-be-processed
log in the preset format
generated by the preset service platform and the configuration data
corresponding to the preset service
Date Recue/Date Received 2021-12-24
platform;
[0070] Parsing the to-be-processed log, recognizing data key and data value
corresponding to the data key
included in the to-be-parsed log, comprising:
[0071] 222, using Flink streaming computing engine to parse the to-be-
processed log, recognizing the data
key and the data value corresponding to the data key included in the to-be-
parsed log.
[0072] 230, when included data key pair includes preset data key, modifying
data value corresponding to
the included preset data key according to the corresponding replacement rule;
[0073] 240, generating historical operation record according to both meaning
and data value
corresponding to each data key;
[0074] 250, storing historical operation record in index library address
corresponding to the preset
identifier.
[0075] Preferably, storing historical operation record in index library
address corresponding to the preset
identifier, comprising:
[0076] 251, Sending data storage request to preset system corresponding to the
index library address, so
that the preset system returns data access authority according to the data
storage request;
[0077] 252, when receiving the returned data access authority, storing the
historical operation record in
the index library address.
[0078] Preferably, the to-be-processed log is in Json format.
11
Date Recue/Date Received 2021-12-24
Implementation three
[0080] For the above-mentioned implementation, as shown in Figure 3, the
present application provides
a log storage apparatus, the apparatus comprises:
[0081] An obtaining module 310 configured to obtain a to-be-processed log in a
preset format generated
by a preset service platform and configuration data corresponding to the
preset service platform, the to-be-
processed log includes a key-value pair and a preset identifier, and the key-
value pair is used to store
historical operation record of the preset service platform, the configuration
data include meaning
corresponding to each data key, replacement rule corresponding to preset data
key, and index library address
corresponding to each preset identifier;
[0082] A parsing module 320 configured to parse the to-be-processed log,
recognizing data key and data
value corresponding to the data key included in the to-be-parsed log; when
included data key pair include
preset data key, modifying data value corresponding to the included preset
data key according to the
corresponding replacement rule; generating historical operation record
according to both meaning and data
value corresponding to each data key;
[0083] A storage module 330 configured to store historical operation record in
index library address
corresponding to the preset identifier.
[0084] Preferably, the storage module 330 is also for sending data storage
request to preset system
corresponding to the index library address, so that the preset system returns
data access authority according
to the data storage request; when receiving the returned data access
authority, storing the historical
operation record in the index library address.
12
Date Recue/Date Received 2021-12-24
[0085] Preferably, the apparatus also comprises a receiving module 340
configured to receive log reading
request sent by user; generate visual log according to historical operation
record corresponding to the log
reading request; return the visual log to the user.
[0086] Preferably, the to-be-processed log is in Json format.
[0087] Preferably, the obtaining module 310 is also configured to use Flume
log collection system to
obtain the to-be-processed log in the preset format generated by the preset
service platform and the
configuration data corresponding to the preset service platform, the parsing
module 320 is also configured
to use Flink streaming computing engine to parse the to-be-processed log,
recognize the data key and the
data value corresponding to the data key included in the to-be-parsed log.
Implementation four
[0089] For the above-mentioned method, the present application provides a log
storage method, the
method comprise:
[0090] Sending generated to-be-processed log to preset processing terminal, so
that the preset processing
terminal receives the to-be-processed log and obtains corresponding
configuration data, the to-be-processed
log includes a key-value pair and a preset identifier, and the key-value pair
is used to store the historical
operation record of the preset service platform, the configuration data
includes the meaning corresponding
to each data key, the replacement rule corresponding to the preset data key,
and the index library address
corresponding to each preset identifier; parsing the to-be-processed log,
recognizing data key and data value
corresponding to the data key included in the to-be-parsed log; when the
included data key pair includes
preset data key, modifying data value corresponding to the included preset
data key according to the
corresponding replacement rule; storing the historical operation record to the
index library address
corresponding to the preset identifier.
13
Date Recue/Date Received 2021-12-24
Implementation five
[0092] The implementation five of the present application provides a computer
system, comprising: one
or a plurality of processors; A memory associated with one or a plurality of
processors, the memory is
configured to store program commands, if the program commands are executed by
one or plurality of
processors, executing the following operations:
[0093] Obtaining a to-be-processed log in a preset format generated by a
preset service platform and
configuration data corresponding to the preset service platform, the to-be-
processed log includes a key-
value pair and a preset identifier, and the key-value pair is used to store
historical operation records of the
preset service platform, the configuration data include meaning corresponding
to each data key,
replacement rule corresponding to preset data key, and index library address
corresponding to each preset
identifier;
[0094] Parsing the to-be-processed log, recognizing data key and data value
corresponding to the data
keys included in the to-be-parsed log;
[0095] When included data key pair includes preset data key, modifying data
value corresponding to the
included preset data key according to the corresponding replacement rule;
[0096] Generating historical operation record according to both meaning and
data value corresponding to
each data key;
[0097] Storing historical operation record in index library address
corresponding to the preset identifier.
Wherein, Figure 4 exemplarily shows the architecture of the computer system ,
which can specifically
include a processor 1510, video
display adapter 1511, disk driver 1512, input/output
14
Date Recue/Date Received 2021-12-24
interface 1513, network interface 1514, and memory 1520. The above-mentioned
processor 1510, video
display adapter 1511, disk driver 1512, input/output interface 1513, network
interface 1514 and
memory 1520 can be connected through a communication bus 1530.
[0098] Wherein, the processor 1510 can be achieved by using a general CPU
(Central Processing Unit),
Microprocessor, Application Specific Integrated Circuit (ASIC) , or one or
more integrated circuits, which
are used to execute some relative program to achieve the technical solutions
provided in this application.
[0099]
The memory 1520 can adopt ROM (Read Only Memory), RAM (Random Access
Memory), static storage devices and dynamic storage devices to achieve. The
memory 1520 can store
operate system 1521 used to control the running of the computer system 1500,
used to control the low-level
operation of the computer system 1500's Basic Input Output System (BIOS) 1522.
In addition, storing a
web browser 1523, data storage management 1524, and device identity
information processing
system 1525 and so on. The above-mentioned device identity information
processing system 1525 can be
the specific application that implements the above-mentioned steps. To sum up,
when achieving the
technical solutions provided by this application through software or firmware,
related program codes are
stored in the memory 1520 and executed by a processor 1510. Input / output
interface 1513 is used for
connecting input / output modules to achieve the information input and output.
Input / output module can
be configured in the device as a component ( not shown in the figure ), or it
can be connected to the device
to provide corresponding functions. Wherein, Input devices can include
keyboards, mice, touch screens,
microphones, various sensors, etc., and output devices can include monitors,
speakers, vibrators, lights and
so on.
[0100] The network interface 1514 is used to connect a communication module
(not shown in the
figure) to achieve the communication interaction between this device and other
devices. Wherein, the
communication module can achieve communication through wired means (such as
USB , network cable,
etc. ) , or through wireless methods ( such as mobile network, WIFI,
Bluetooth, etc.) to achieve
Date Recue/Date Received 2021-12-24
communication.
[0101] The bus 1530 includes a path and transmits information among various
components of the device
(such as the processor 1510, the video display adapter 1511, the disk driver
1512, the input/output
interface 1513, the network interface 1514, and the memory 1520).
[0102] In addition, the electronic device 1500 also can obtain information
with specific receiving
conditions from the virtual resource object's receiving condition information
database 1541 for condition
judgement and son on.
[0103] What should be noted is that although the above device only shows the
processor 1510 , the video
display adapter 1511, the disk driver 1512, input/output
interface 1513, network
interface 1514, mcmory 1520, bus 1530, etc., but in the process of the
specific implementation, the device
may also include other essential components for normal operation. In addition,
those skilled in the art can
understand that the above apparatus can comprise only the essential components
of the present application
to achieve the implementation, but there is no need to contain all the
components as shown in figure.
[0104] Known from the description of the above implementations that those
skilled in the art can clearly
understand that the application can be achieved with the help of software and
essential general hardware
platform. Based on this understanding, the essence of the technical solution
of this application, or in other
words, the part that contributes to the existing technology can be implemented
in the form of a software
product, the computer software product can be stored in storage media, such as
ROM/RAM, magnetic
disks, optical disks, etc., including several commands to make a computer
device (can be a personal
computer, a cloud server, or a network device, etc.) to execute the methods
described in each
implementation or some of the implementations of the present application.
[0105] The various implementations in this description are described in a
progressive manner, the same
16
Date Recue/Date Received 2021-12-24
and similar parts among the various implementations can be referred to each
other separately, and each
implementation focuses on the differences compared with the other
implementations. Especially for the
concern of the system or the system implementations, since it is basically
similar to the implementation
method, the description is relatively simple. For related details, please
refer to the implementation
method. The system and system implementations described in the above are only
illustrative, and the units
described by separate parts may or may not be physically separate, and the
parts displayed as units may or
may not be physical units, which means, it can be in one place, or it may be
distributed to a plurality of
network units. Some or all the modules are selected according to actual needs
to achieve the
implementation's solution purpose. The ordinary skill in the art can
understand and implement without
creative work.
[0106] The above-mentioned are only preferred implementations of the present
invention, but not used to
restrict the present invention, anything in the spirits and the principles of
the present invention, then any
modifications, equivalent replacements, improvements shall be included in the
protection scope of the
present invention.
17
Date Recue/Date Received 2021-12-24
