Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
RESOURCE ADJUSTING METHOD, DEVICE, EQUIPMENT AND STORAGE
MEDIUM FOR FLINK TASK PROCESSING
Technical Field
[0001] The present disclosure relates to computer technical field,
particularly to a resource
adjustment for Flink task processing method, apparatus, computer device and
storage medium.
Background
[0002] At present, each TaskManager of the underling Flink framework is set to
IC and 2G
by default, user can adjust quantity of Slots (task slots) and memory of the
TaskManager
through advanced parameters. For example, the current user sets concurrency,
mainly by
setting a plurality of TaskManagers or setting a plurality of slots through
advanced parameters.
The main problem is when user configures concurrency, due to the lack of
experience in
recourse allocation, a small amount of concurrency is required to solve the
problem but user
configures too much concurrency, CPU and memory utilization is low, resulting
in waste of
resources. Therefore, how to rationally adjust resources has become a problem
to be solved.
[0003] In traditional technical scheme, when Flink framework performs task
processing, the
utilization of resources is generally evaluated by the highest value and
average value of
QPS(Throughput), and resources are adjusted according to evaluation result,
which easily leads
to evaluation inaccurate, unreasonable adjustment issues.
Summary
[0004] Based on this, it is necessary to provide a source adjustment for Flink
task processing
method, apparatus, computer device and storage medium.
[0005] A source adjustment method for Flink task processing, the method
comprises:
[0006] Obtaining actual usage of each preset resource usage indicator during
the execution of
current Flink task;
1
Date Regue/Date Received 2023-01-24
[0007] Generating resource usage total score of current Flint( task according
to the actual
usage of each resource usage indicator and pre-configured indicator parameter
of each resource
usage indicator;
[0008] Obtaining adjustment strategy of current Flink task according to the
resource usage
total score of current Flink task and the mapping relationship between pre-
configured resource
usage total score and adjustment strategy;
[0009] Adjusting resource allocation of current Flink task according to
adjustment strategy of
current Flink task.
[0010] In an embodiment, the above-mentioned indicator parameter includes the
maximum
value and the indicator highest score, wherein generating resource usage total
score of current
Flink task according to the actual usage of each resource usage indicator and
pre-configured
indicator parameter of each resource usage indicator, comprising:
[0011] =When the evaluation factor of the resource usage indicator belonging
is first factor,
generating usage ratio of the resource usage indicator of each first factor
according to the actual
usage amount and the maximum value of the resource usage indicator of each
first factor;
[0012] Generating first score according to the usage ratio of the resource
usage indicator of
each first factor and the indicator highest score;
[0013] When the evaluation factor of the resource usage indicator belonging is
second factor,
the resource usage indicator includes first indicator and second indicator,
generating
consumption ratio based on the actual usage of first indicator and the actual
usage of second
indicator;
[0014] Generating second score according to the consumption ratio and the
highest indicator
2
Date Regue/Date Received 2023-01-24
score of the resource usage indicator of second factor;
[0015] Generating resource usage total score of current Flink task according
to the second
score and the first score.
[0016] In an embodiment, the above-mentioned obtaining adjustment strategy of
current
Ffink task according to the resource usage total score of current Ffink task
and the mapping
relationship between pre-configured resource usage total score and adjustment
strategy,
comprising:
[0017] Obtaining adjustment strategy matching the range of the resource usage
total score of
current Flink task according to the mapping relationship between the resource
usage total score
and the adjustment strategy, then obtaining adjustment strategy of current
Flink task.
[0018] In an embodiment, wherein the adjustment strategy of current Flink task
includes
adjustment value corresponding to resource usage indicator to be adjusted,
wherein adjusting
resource allocation of current Ffink task according to adjustment strategy of
current Flink task,
comprising:
[0019] Adjusting the resource allocation amount corresponding to each resource
usage
indicator of current Flink task according to adjustment value corresponding to
resource usage
indicator to be adjusted.
[0020] In an embodiment, the above-mentioned method also comprises:
[0021] When receiving resource adjustment request sent by terminal, executing
the step of
obtaining actual usage of each preset resource usage indicator during
execution of current Flink
task;
[0022] Alternatively, when detecting of the current Flink task starts to run,
executing the step
3
Date Regue/Date Received 2023-01-24
of obtaining actual usage of each preset resource usage indicator during
execution of current
Ffink task.
[0023] In an embodiment, the method also comprises:
[0024] Comparing actual usage corresponding to each resource usage indicator
with preset
threshold to obtain comparison result;
[0025] Generating detailed resource usage information corresponding to each
resource usage
indicator according to the comparison result;
[0026] Sending the detailed resource usage information to the terminal for
display.
[0027] In an embodiment, the method also comprises:
[0028] Receiving configuration request for the mapping relationship between
resource usage
total score and adjustment strategy;
[0029] Creating the mapping relationship between the newly added adjustment
strategy and
the resource usage total score corresponding to the newly added adjustment
strategy according
to the newly added adjustment strategy in the configuration request and the
resource usage total
score corresponding to the newly added adjustment strategy.
[0030] A resource adjustment apparatus for Flint( task processing, wherein,
the apparatus
comprises:
[0031] An obtaining module configured to obtain actual usage of each preset
resource usage
indicator during the execution of current Flink task;
[0032] A generating module configured to generate resource usage total score
of current Flink
4
Date Recue/Date Received 2023-01-24
task according to the actual usage of each resource usage indicator and pre-
configured indicator
parameter of each resource usage indicator.
[0033] A processing module configured to obtain adjustment strategy of current
FlMk task
according to the resource usage total score of current Mink task and the
mapping relationship
between pre-configured resource usage total score and adjustment strategy;
[0034] An adjusting module configured to adjust resource allocation of current
Flink task
according to adjustment strategy of current Flink task.
[0035] A computer device, including a memory, a processor and a computer
program stored
in the memory and run on the processor configured to achieve following steps
when the
processor executes the computer program:
[0036] Obtaining actual usage of each preset resource usage indicator during
the execution of
current Flink task;
[0037] Generating resource usage total score of current Mink task according to
the actual
usage of each resource usage indicator and pre-configured indicator parameter
of each resource
usage indicator;
[0038] Obtaining adjustment strategy of current FlMk task according to the
resource usage
total score of current Rink task and the mapping relationship between pre-
configured resource
usage total score and adjustment strategy;
[0039] Adjusting resource allocation of current Flink task according to
adjustment strategy of
current Flink task.
[0040] A computer readable storage medium stored with a computer program
configured to
achieve following steps when the processor executes the computer program:
Date Regue/Date Received 2023-01-24
[0041] Obtaining actual usage of each preset resource usage indicator during
the execution of
current Flink task;
[0042] Generating resource usage total score of current Ffink task according
to the actual
usage of each resource usage indicator and pre-configured indicator parameter
of each resource
usage indicator;
[043] Obtaining adjustment strategy of current Flink task according to the
resource usage total
score of current Flink task and the mapping relationship between pre-
configured resource usage
total score and adjustment strategy;
[0044] Adjusting resource allocation of current Flink task according to
adjustment strategy of
current Flink task.
[0045] The above-mentioned to resource adjustment for Flink task processing
method,
apparatus, computer device and storage medium, comprising: obtaining actual
usage of each
preset resource usage indicator during current Flink task; generating resource
usage total score
of current Flink task according to actual usage and pre-configured indicator
parameter of each
resource usage indicator; obtaining adjustment strategy of current Flink task
according to the
resource usage total score and the mapping relationship between pre-configured
resource usage
total score and adjustment strategy; adjusting resource allocation of current
Flink task
according to the adjustment strategy. This application sets plural resource
usage indicators,
scores resource usage of current Flink task from plural dimensions, and
accurately locates
adjustment strategy that matches current Flink task based on the mapping
relationship between
resource usage total score and adjustment strategy, making the obtaining of
adjustment strategy
more accurate, and adjusting resource based on the adjustment strategy, making
resource
adjustment more accurate.
6
Date Recue/Date Received 2023-01-24
Drawing Description
Detailed Description
[0052] In order to make clearer application purposes, technical solutions, and
advantages, the
present disclosure is further explained in detail with a particular embodiment
thereof, and with
reference to the drawings. It shall be appreciated that these descriptions are
only intended to be
illustrative, but not to limit the scope of the disclosure thereto.
[0053] The present application provides source adjustment for Flink task
processing method
applied in the application environment from Figure 1. As shown in Figure 1,
the application
environment includes server 100 and terminal end 101, server 100 connects with
terminal 101
through network 102 to achieve the source adjustment method for Flink task
processing.
[0054] The server 100 is configured to obtain actual usage of each preset
resource usage
indicator during the execution of current Flink task when receiving the
resource adjustment
request sent by terminal 101; generate resource usage total score of current
Flink task according
to the actual usage of each resource usage indicator and pre-configured
indicator parameter of
each resource usage indicator; obtain adjustment strategy of current Flink
task according to the
resource usage total score of current Flink task and the mapping relationship
between pre-
configured resource usage total score and adjustment strategy; adjust resource
allocation of
current Flink task according to adjustment strategy of current Flink task, the
server is also
configured to generate detailed resource usage information corresponding to
each resource
usage indicator according to the actual usage amount and indicator
corresponding to each
resource usage indicator; and send resource usage detailed information to
terminal 101 for
display. Wherein, server 100 can be implemented by an independent server or a
server cluster
composed of a plurality of servers.
[0055] Terminal 101 is configured to send resource adjustment request to
server 100 and
receive resource usage details corresponding to each resource usage indicator
sent by server,
display source usage details corresponding to each resource usage indicator on
user interface.
7
Date Regue/Date Received 2023-01-24
Terminal 101 can be but is not limited to various personal computers, laptops,
smartphones,
tablet computers and portable wearable device.
[0056] Network 102 is configured to achieve the network connection between
terminal 101
and server 100, specifically, the network 102 can include various types of
wired or wireless
networks.
[0057] In an embodiment, as shown in Figure 2, a resource adjustment method
for Flink task
processing is provided, the method is applied to the server in Figure 1 as an
example for
illustration, comprising following steps:
[0058] Si!, obtaining actual usage of each preset resource usage indicator
during the
execution of current Flink task.
[0059] In the present application, the above-mentioned current Flink task can
be processing
task of the real-time data flow of Flink framework. The above resource usage
indicator can
include the highest memory usage, the average memory usage, the quantity of
FullGC (Full
Garbage Collection), FullGC time, the highest TaskManager CPU usage, the
average
TaskManager CPU usage, task sender Buffer usage, committedOffsets(commit
offset(compensation) to a specific location), currentOffsets(current offset
location).
[0060] Furthermore, the actual usage of the above-mentioned resource usage
indicator is the
actual value of each resource usage indicator during the execution of current
Flink task. For
example, the highest memory usage can be obtained during the execution of
current Flink task.
Specifically, the present application can preset a preset time period, obtain
the actual usage
amount of each resource usage indicator within the preset time period. For
example, if the
preset time period is one hour, the actual usage of each resource usage
indicator within one
hour is obtained. A certain period of time on a certain day can also be set to
obtain the actual
usage of each resource usage indicator. The specific situation is set
according to actual
requirements, and no specific limitation is made here.
8
Date Regue/Date Received 2023-01-24
[0061] S12, generating resource usage total score of current Flink task
according to the actual
usage of each resource usage indicator and pre-configured indicator parameter
of each resource
usage indicator.
[0062] In the present application, different indicator parameters are set for
different resource
usage indicators. For example, for the average memory usage, setting the
indicator parameter
as the maximum memory value 16, and the indicator maximum score value is 50.
[0063] In the present application, obtaining score of each resource usage
indicator according
to the actual usage of each resource usage indicator and pre-configured
indicator parameter of
each resource usage indicator, further, obtaining resource usage total score
of current Ffink task
according to the score of each resource usage indicator.
[0064] In an embodiment, the above-mentioned indicator parameter includes the
maximum
value and the indicator highest score, as shown in Figure 3, a process diagram
of generating
resource usage total score of current Flinlc task according to the actual
usage of each resource
usage indicator and pre-configured indicator parameter of each resource usage
indicator is
provided in an embodiment, comprising following steps:
[0065] S31, when the evaluation factor of the resource usage indicator
belonging is first
factor, generating usage ratio of the resource usage indicator of each first
factor according to
the actual usage amount and the maximum value of the resource usage indicator
of each first
factor;
[0066] S32, generating first score according to the usage ratio of the
resource usage indicator
of each first factor and the indicator highest score;
[0067] S33, when the evaluation factor of the resource usage indicator
belonging is second
factor, the resource usage indicator includes first indicator and second
indicator, generating
9
Date Regue/Date Received 2023-01-24
consumption ratio based on the actual usage of first indicator and the actual
usage of second
indicator;
[0068] S34, generating second score according to the consumption ratio and the
highest
indicator score of the resource usage indicator of second factor;
[0069] S35, generating resource usage total score of current Rink task
according to the
second score and the first score.
[0070] In the present application, a plurality of evaluation factors are
preset. The evaluation
factors can include memory, CPU, FullGC, Kafka consumption capacity, and back
pressure,
data skew, etc. One or more resource usage indicators can be set for each
evaluation factor. For
example, corresponding resource usage indicator can be set for memory,
including the highest
memory usage and the average memory usage. Corresponding resource usage
indicator can be
set for CPU, including the highest TaskManager CPU usage ang the average
TaskManager
CPU usage. Corresponding resource usage indicator can be set for FulIGC,
including the
number of FullGC and the time of FullGC. For Kafka consumption capacity,
corresponding
resource usage indicator can be set, including committedOffsets and
currentOffsets. For back
pressure and data skew, corresponding recourse usage indicator can be set,
including the usage
of Buffer at the task sending end and the usage of Buffer at the task
receiving end.
[0071] Furthermore, the above-mentioned first factor includes memory, CPU,
FullGC, back
pressure, and data skew. The above-mentioned second factor includes Kafka
consumption
capacity. The above-mentioned first indicator is committedOffsets. The above-
mentioned
second indicator is currentOffsets.
[0072] Specifically, for the memory evaluation factor, assuming that indicator
parameter
corresponding to the memory is set to include a maximum memory value 16, a
highest indicator
score 50, obtaining the maximum memory usage M under the memory evaluation
factor, then
the score of memory evaluation factor = M/16*50.
Date Regue/Date Received 2023-01-24
[0073] In the same way, for the CPU evaluation factor, assuming that indicator
parameter
corresponding to the CPU is set to include a CPU maximum value 100, a highest
indicator
score 20, obtaining the average TaskManager CPU usage N under the CPU
evaluation factor,
then the score of CPU evaluation factor = N/100*20.
[0074] For the FullGC evaluation factor, assuming that indicator parameter
corresponding to
the FullGC is set to include a maximum value of FullGC number 20, a highest
indicator score
5, obtaining actual value of FullGC number S, then the score of FullGC
evaluation factor =
S/20*5.
[0075] For the back pressure and data skew evaluation factor, assuming that
corresponding
indicator parameter is set to include a highest score of back pressure 5, a
maximum value 100,
obtaining the actual value of the Buffer usage at the task sending end Y, then
the score of back
pressure and data skew evaluation factor = Y/100*5.
[0076] For the Kafka consumption capacity evaluation factor, assuming that
corresponding
indicator parameter is set to a highest score of offset 15, obtaining the
actual value of
committedOffsets E and the actual value of currentOffsets F, then the score of
Kafka
consumption capacity evaluation factor = F/E*15.
[0077] Therefore, the above-mentioned first score comprises the score of
memory evaluation
factor = M/16*50, the score of CPU evaluation factor = N/100*20, the score of
FullGC
evaluation factor = S/20*5, the score of back pressure and data skew
evaluation factor =
Y/100 5, the score of Kafka consumption capacity evaluation factor = F/E*15.
Then the
calculation formula of total resource usage score of current Ffink task is as
follows:
[0078] Resource usage total score = M/16*50+N/100*20+S/20*5+Y/100*5+F/E*15.
11
Date Regue/Date Received 2023-01-24
[0079] In the present application, by setting a plurality of resource usage
indicators and
obtaining the actual usage of each resource usage indicator in any time
period, based on the
actual usage, generating the resource usage total score of current Flink task
to achieve resource
utilization from plural dimensions, making the evaluation of resource
utilization more accurate
and scientific, further making resource utilization adjustment more reasonable
and solving the
problem of resource waste caused by unreasonable resource allocation in the
existing
technology, improving resource utilization.
[0080] What should be noted is that a plurality of resource usage indicators
under one
evaluation factor can exist. In this case, when calculating the resource usage
total score of
current Flink task, the actual usage of one of the resource usage indicators
can be selected from
a plurality of resource usage indicators for calculating. For example, for the
resource usage
indicators of the highest memory usage and the average memory usage under the
memory
evaluation factor, when calculating the resource usage total score of current
Flink task, the
actual value of the highest memory usage can be selected for calculating.
[0081] In another embodiment, all resource usage indicators under each
evaluation factor can
also be involved in the calculating. For example, the first score
corresponding to the highest
memory usage and the average memory usage can be used to calculate the
resource usage total
score of current Flink task.
[0082] S13, obtaining adjustment strategy of current Flink task according to
the resource
usage total score of current Flink task and the mapping relationship between
pre-configured
resource usage total score and adjustment strategy.
[0083] In the present application, the above-mentioned mapping relationship
between the
resource usage total score and the adjustment strategy includes the
corresponding relationship
between the resource usage total score and the adjustment strategy. Wherein,
the above-
mentioned adjustment strategy can include the resource usage indicator to be
adjusted and the
corresponding adjustment value, can also include the reasons for resource
shortage or resource
12
Date Regue/Date Received 2023-01-24
excess, and can also include usage details of other resource usage indicators.
[0084] Specifically, the present application can obtain adjustment strategy
according to
resource usage total score of current Flink task and can obtain the
corresponding adjustment
strategy according to the score of each evaluation factor of current Flink
task at the same time,
so that the obtained adjustment strategy is more accurate.
[0085] For example, the resource usage total score of current Flink task is A,
the adjustment
strategy is matched according to the mapping relationship between the resource
usage total
score and the adjustment strategy: the CPU is recommended to reduce to 2C and
the memory
is reduced to 4G.
[0086] In an embodiment, the above-mentioned obtaining adjustment strategy of
current
Flink task according to the resource usage total score of current Flint( task
and the mapping
relationship between pre-configured resource usage total score and adjustment
strategy,
comprising:
[0087] Obtaining adjustment strategy matching the range of the resource usage
total score of
current Flink task according to the mapping relationship between the resource
usage total score
and the adjustment strategy, then obtaining adjustment strategy of current
Flink task.
[0088] In the present application, the above-mentioned mapping relationship
between the
resource usage total score and the adjustment strategy includes the
corresponding relationship
between the resource usage total score and the adjustment strategy. Wherein,
the relationship
can be one-to-one correspondence between the resource total score and the
adjustment strategy,
or one-to-one correspondence between the numerical range of the resource usage
total score
belonging and the adjustment strategy. The former is a score corresponding to
an adjustment
strategy, and the latter is a score range corresponding to an adjustment
strategy, the specific
settings can be set according to actual requirements.
13
Date Regue/Date Received 2023-01-24
[0089] The present application can locate the adjustment strategy more
accurately through
the present embodiment, making the resource adjustment more accurate.
[0090] S14, adjusting resource allocation of current Flink task according to
adjustment
strategy of current Flink task.
[0091] In the present application, the above-mentioned adjustment strategy can
include the
resource usage indicator to be adjusted and the corresponding adjustment
value, can also
include the reasons for resource shortage or resource excess, and can also
include usage details
of other resource usage indicators.
[0092] In an embodiment, the above-mentioned adjustment strategy of current
Flink task
includes adjustment value corresponding to resource usage indicator to be
adjusted, wherein
adjusting resource allocation of current Flink task according to adjustment
strategy of current
Flink task, comprising:
[0093] Adjusting the resource allocation amount corresponding to each resource
usage
indicator of current Rink task according to adjustment value corresponding to
resource usage
indicator to be adjusted.
[0094] In the present application, the adjustment strategy of current Flink
task includes the
adjustment value corresponding to the resource usage indicator to be adjusted
and the resource
usage details information of each resource usage indicator.
[0095] For example, the above-mentioned adjustment strategy can be:
[0096] The resource currently applied for task B is 4C CPU and 8G memory,
during the
statistical period (2021-11-8:10 to 2021-11-8:11), the total score is C,
according to the mapping
relationship between the total usage score and the adjustment strategy, the
matched adjustment
strategy is: the actual resource usage rate is low, and resource is
recommended to reduce to
14
Date Recue/Date Received 2023-01-24
2C(CPU)4G(Memory).
[0097] Wherein, resource usage detailed information is as follows:
[0098] (1) Memory: applied memory is 8G, the average used memory during
statistical
period is actually 3G.
[0099] (2) CPU: applied CPU is 4C, actual use during statistical period is 2.
[0100] (3) FullGC: during statistical period, the number of FullGC is 2, the
total time length
is 5 minutes, not frequent.
[0101] (4) Back pressure, data skew: the average usage rate of outPoolU sage
and
inPoolUsage is lower than 30% during the statistical period, no back pressure
occurred.
[0102] (5) Consumption capacity: the offset consumption capacity is normal, no
backlog
occurred.
[0103] According to the above-mentioned adjustment strategy, the adjustment
strategy
includes the adjustment value of resource usage indicator to be adjusted and
the detailed
resource usage information. Wherein, the resource usage detailed information
describes
information such as actual usage information of each resource usage indicator
and whether the
usage is within a reasonable range. According to the above-mentioned
adjustment strategy, the
memory can be reduced to 4G, the CPU can be reduced to 2C. This application
can achieve
self-defined resource adjustment according to the adjustment strategy with
high flexibility.
[0104] In an embodiment, the method also comprises:
[0105] When receiving resource adjustment request sent by terminal, executing
the step of
obtaining actual usage of each preset resource usage indicator during
execution of current Flink
Date Regue/Date Received 2023-01-24
task;
[0106] Alternatively, when detecting of the current Flink task starts to run,
executing the step
of obtaining actual usage of each preset resource usage indicator during
execution of current
Flink task.
[0107] After the current Flink task is created, the present application can
trigger the resource
adjustment method of the above-mentioned Flink task processing through the
adjustment
request submitted by client. Also, the present application can automatically
trigger the resource
adjustment method of Flink task processing when the current Flink task starts
to execute.
Generally, each Plink task can execute the above-mentioned resource adjustment
method for
Flink task processing once during execution. Specifically, when obtaining the
actual usage of
each resource usage indicator, the actual usage of any time period during task
execution or the
actual usage of the entire process can be obtained.
[0108] The present application obtains actual usage amount of each resource
usage indicator
in any time period, generates the adjustment strategy based on the actual
usage of each resource
usage indicator in any time period, makes the adjustment strategy more
accurate and also
improves the flexibility of the solution.
[0109] In an embodiment, as shown in Figure 4, the above-mentioned method also
comprises:
[0110] S41, comparing actual usage corresponding to each resource usage
indicator with
preset threshold to obtain comparison result;
[0111] S42, generating detailed resource usage information corresponding to
each resource
usage indicator according to the comparison result;
[0112] S43, sending the detailed resource usage information to the terminal
for display.
16
Date Regue/Date Received 2023-01-24
[0113] Wherein, resource usage detailed information comprises the actual usage
amount of
each resource usage indicator, whether each actual usage is within a
reasonable range and
corresponding reasons analysis.
[0114] The above-mentioned generating detailed resource usage information
corresponding
to each resource usage indicator according to the actual usage amount
corresponding to each
resource usage indicator and indicator score, can comprise:
[0115] Comparing actual usage corresponding to each resource usage indicator
with preset
threshold and generating detailed resource usage information according to the
comparison
result.
[0116] The present application compares the actual usage corresponding to each
resource
usage indicator with the preset threshold generating detailed resource usage
information
according to the comparison result. In another embodiment, the actual usage
corresponding to
each resource usage indicator can be converted into usage rate, comparing the
obtained usage
rate with a preset threshold.
[0117] Specifically, when the resource usage indicator is the highest memory
usage, obtaining
the highest memory usage rate according to the highest memory usage, comparing
the highest
memory usage rate with the corresponding preset threshold. Determining whether
the above-
mentioned highest memory usage rate is lower than the preset threshold, if
yes, the memory
usage rate of the entire task is low, the TaskManager memory or the quantity
of TaskManager
is recommended to be reduced. The above-mentioned preset threshold can be 30%.
[0118] When the resource usage indicator is the number of FullGC, the
corresponding preset
threshold can be 50%. Specifically, counting the number of FullGC within a
preset time period,
such as the number of FullGC within an hour and the duration of each FullGC,
multiplying the
number of FullGC and the duration of each FullGC to compare with statistical
period, if
multiplication result exceeds 50% of the statistical period, then generating
over frequent during
17
Date Regue/Date Received 2023-01-24
the execution of current Ffink task, considering of improving the resource
usage details of the
TaskManager memory.
[0119] When the resource usage indicator is the highest task TaskManager CPU
usage, the
corresponding preset threshold can be 5%. Specifically, the highest
TaskManager CPU usage
rate can be obtained according to the obtained highest TaskManager CPU usage,
comparing
the highest TaskManager CPU usage rate with 5%, if lower than 5%, then
generating the
highest detailed resource usage information with low TaskManager CPU usage
rate.
[0120] When the resource usage indicator is the usage amount of task sending
end Buffer and
the usage amount of the task receiving end Buffer, generating resource usage
detailed
information, can comprise following steps:
[0121] Obtaining the usage rate of corresponding task sending end Buffer and
the usage rate
of task receiving end Buffer respectively according to the usage amount of
task sending end
Buffer and the usage amount of task receiving end Buffer.
[0122] =When the usage rate of ExclusiveBuffer(private Buffer) is lower than
10% and the
usage rate of FloatingBuffer(shared Buffer) is higher than 90%, the generated
reason analysis
information can be data skew.
[0123] When the Buffer occupancy rate of the receiving end inPoolUsage(input
capacity) and
the sending end outPoolUsage(output capacity) of a Subtask are both lower than
10%,
indicating normal.
[0124] When the Buffer occupancy rate of the receiving end inPoolUsage of a
Subtask is
more than 90%, the Buffer occupancy rate of the sending end
outPoolUsage(output capacity)
of a Subtask is lower than 10%, then the generated reason analysis information
is that
conducting the back pressure upstream can be the source of back pressure.
18
Date Regue/Date Received 2023-01-24
[0125] When the Buffer occupancy rate of the receiving end inPoolUsage and
outPoolUsage
of a Subtask are both higher than 90%, then generated reason analysis
information is under
downstream back pressure.
[0126] When the resource usage indicator is committedOffsets and
currentOffsets, if each
partition is successfully submitted to the current submission offset position
of ICafka partition,
much lower than the position of offset in the current loaded partition, and
indicating that there
is backlog, then generating reason analysis information such as whether the
resource allocation
is reasonable and whether the business data flow has increased.
[0127] Through the present embodiment, the present application generates
detailed resource
usage information based on the actual usage of each resource usage indicator,
sends the detailed
information to terminal for display, so that relevant person can analyze data
and improve
solution flexibility.
[0128] In an embodiment, the above-mentioned method also comprises:
[0129] Receiving configuration request for the mapping relationship between
resource usage
total score and adjustment strategy;
[0130] Creating the mapping relationship between the newly added adjustment
strategy and
the resource usage total score corresponding to the newly added adjustment
strategy according
to the newly added adjustment strategy in the configuration request and the
resource usage total
score corresponding to the newly added adjustment strategy.
[0131] In the present application, when an adjustment strategy requires to be
added, a
configuration request for the mapping relationship between the resource usage
total score and
the adjustment strategy can be sent through terminal. The configuration
request carries the new
adjustment strategy and the resource usage total score corresponding to the
new adjustment
strategy, according to the configuration request, creating a mapping
relationship of the resource
19
Date Recue/Date Received 2023-01-24
usage total score corresponding to the newly added adjustment strategy, and
the mapping
relationship is stored in a preset database. The adjustment strategy of the
present application
supports medication and addition, and the scheme flexibility is improved.
[0132] In an embodiment, as shown in Figure 5, a resource adjustment apparatus
for Flink
task processing is provided, comprising: obtaining module II, generating
module 12,
processing module 13 and adjusting module 14, wherein:
[0133] An obtaining module configured to obtain actual usage of each preset
resource usage
indicator during the execution of current Flink task;
[0134] A generating module configured to generate resource usage total score
of current Flink
task according to the actual usage of each resource usage indicator and pre-
configured indicator
parameter of each resource usage indicator;
[0135] A processing module configured to obtain adjustment strategy of current
Flink task
according to the resource usage total score of current Flink task and the
mapping relationship
between pre-configured resource usage total score and adjustment strategy;
[0136] An adjusting module configured to adjust resource allocation of current
Flink task
according to adjustment strategy of current Flink task.
[0137] In an embodiment, the above-mentioned indicator parameter includes the
maximum
value and the indicator highest score, the above-mentioned generating module
12 generates
usage ratio of the resource usage indicator of each first factor according to
the actual usage
amount and the maximum value of the resource usage indicator of each first
factor when the
evaluation factor of the resource usage indicator belonging is first factor,
generates first score
according to the usage ratio of the resource usage indicator of each first
factor and the indicator
highest score, when the evaluation factor of the resource usage indicator
belonging is second
factor, the resource usage indicator includes first indicator and second
indicator, generating
Date Recue/Date Received 2023-01-24
consumption ratio based on the actual usage of first indicator and the actual
usage of second
indicator, generates second score according to the consumption ratio and the
highest indicator
score of the resource usage indicator of second factor, and generates resource
usage total score
of current Flink task according to the second score and the first score.
[0138] In an embodiment, the above-mentioned processing module 13 can obtain
adjustment
strategy matching the range of the resource usage total score of current Flink
task according to
the mapping relationship between the resource usage total score and the
adjustment strategy,
then obtaining adjustment strategy of current FIMk task.
[0139] In an embodiment, the above-mentioned adjustment strategy of current
Flink task
includes adjustment value corresponding to resource usage indicator to be
adjusted, the above-
mentioned adjusting module 14 can adjust the resource allocation amount
corresponding to
each resource usage indicator of current Flink task according to adjustment
value
corresponding to resource usage indicator to be adjusted.
[0140] In an embodiment, the above-mentioned apparatus also comprises
receiving module
(not shown in Figure), when receiving resource adjustment request sent by
terminal, executing
the step of obtaining actual usage of each preset resource usage indicator
during execution of
current Flink task, alternatively, when detecting of the current Flink task
starts to run, executing
the step of obtaining actual usage of each preset resource usage indicator
during execution of
current Flink task.
[0141] In an embodiment, the above-mentioned apparatus also comprises
comparing module
(not shown in Figure), the present comparing module can compare actual usage
corresponding
to each resource usage indicator with preset threshold to obtain comparison
result, generate
detailed resource usage information corresponding to each resource usage
indicator according
to the comparison result, sends the detailed resource usage information to the
terminal for
display.
21
Date Regue/Date Received 2023-01-24
[0142] In an embodiment, the above-mentioned apparatus also comprises newly
addition
module (not shown in Figure), the newly addition module can receive
configuration request for
the mapping relationship between resource usage total score and adjustment
strategy, creating
the mapping relationship between the newly added adjustment strategy and the
resource usage
total score corresponding to the newly added adjustment strategy according to
the newly added
adjustment strategy in the configuration request and the resource usage total
score
corresponding to the newly added adjustment strategy.
[0143] In an embodiment, a computer device is provided, the computer device
can be a server
whose internal structure diagram is shown in Figure 6. The computer device
includes a
processor, a memory, a network interface, and a database connected through a
system bus. The
processor of the computer device is configured to provide calculating and
control capabilities.
The memory of the computer device includes non-volatile storage medium and
internal
memory. The memory of non-volatile storage medium has an operation system,
computer
programs and database. The internal memory provides an environment for the
operation system
and computer program running in a non-volatile storage medium. The network
interface of the
computer device is used to communicate with an external terminal through a
network
connection. The computer program is executed by the processor to implement a
resource
allocation method for compiling a virtual machine.
[0144] In an embodiment, a computer device is provided, including a memory, a
processor
and a computer program stored in the memory and ran on the processor
configured to achieve
the following steps when the processor executes the computer program:
obtaining actual usage
of each preset resource usage indicator during the execution of current Ffink
task; generating
resource usage total score of current Flink task according to the actual usage
of each resource
usage indicator and pre-configured indicator parameter of each resource usage
indicator;
obtaining adjustment strategy of current Flink task according to the resource
usage total score
of current Mink task and the mapping relationship between pre-configured
resource usage total
score and adjustment strategy;
22
Date Regue/Date Received 2023-01-24
[0145] Adjusting resource allocation of current Flink task according to
adjustment strategy of
current Flink task.
[0146] In an embodiment, the above-mentioned indicator parameter includes the
maximum
value and the indicator highest score, the processor executes the computer
program to achieve
the above-mentioned generating resource usage total score of current Rink task
according to
the actual usage of each resource usage indicator and pre-configured indicator
parameter of
each resource usage indicator, specifically achieving following steps:
[0147] When the evaluation factor of the resource usage indicator belonging is
first factor,
generating usage ratio of the resource usage indicator of each first factor
according to the actual
usage amount and the maximum value of the resource usage indicator of each
first factor;
[0148] Generating first score according to the usage ratio of the resource
usage indicator of
each first factor and the indicator highest score;
[0149] When the evaluation factor of the resource usage indicator belonging is
second factor,
the resource usage indicator includes first indicator and second indicator,
generating
consumption ratio based on the actual usage of first indicator and the actual
usage of second
indicator;
[0150] Generating second score according to the consumption ratio and the
highest indicator
score of the resource usage indicator of second factor;
[0151] Generating resource usage total score of current Flink task according
to the second
score and the first score.
[0152] In an embodiment, the processor executes the computer program to
achieve the above-
mentioned obtaining adjustment strategy of current Flink task according to the
resource usage
total score of current Flink task and the mapping relationship between pre-
configured resource
23
Date Recue/Date Received 2023-01-24
usage total score and adjustment strategy, specifically achieving following
steps:
[0153] Obtaining adjustment strategy matching the range of the resource usage
total score of
current Flink task according to the mapping relationship between the resource
usage total score
and the adjustment strategy, then obtaining adjustment strategy of current
Ffink task.
[0154] In an embodiment, the adjustment strategy of current Flink task
includes adjustment
value corresponding to resource usage indicator to be adjusted, the processor
executes the
computer program to achieve the above-mentioned adjusting resource allocation
of current
Flink task according to adjustment strategy of current Flink task,
specifically achieving
following steps:
[0155] Adjusting the resource allocation amount corresponding to each resource
usage
indicator of cm-rent Flink task according to adjustment value corresponding to
resource usage
indicator to be adjusted.
[0156] In an embodiment, the processor executes the computer program to
achieve following
steps:
[0157] When receiving resource adjustment request sent by terminal, executing
the step of
obtaining actual usage of each preset resource usage indicator during
execution of current Flink
task;
[0158] Alternatively, when detecting of the current Flink task starts to run,
executing the step
of obtaining actual usage of each preset resource usage indicator during
execution of current
Flink task.
[0159] In an embodiment, the processor executes the computer program to
achieve following
steps:
24
Date Regue/Date Received 2023-01-24
[0160] Comparing actual usage corresponding to each resource usage indicator
with preset
threshold to obtain comparison result;
[0161] Generating detailed resource usage information corresponding to each
resource usage
indicator according to the comparison result;
[0162] Sending the detailed resource usage information to the terminal for
display.
[0163] In an embodiment, the processor executes the computer program to
achieve following
steps:
[0164] Receiving configuration request for the mapping relationship between
resource usage
total score and adjustment strategy;
[0165] Creating the mapping relationship between the newly added adjustment
strategy and
the resource usage total score corresponding to the newly added adjustment
strategy according
to the newly added adjustment strategy in the configuration request and the
resource usage total
score corresponding to the newly added adjustment strategy.
[0166] In an embodiment, a computer readable storage medium is provided, the
medium
stored with computer program and the processor performs the following steps
when executing
the computer program: obtaining actual usage of each preset resource usage
indicator during
the execution of current Flink task; generating resource usage total score of
current Mink task
according to the actual usage of each resource usage indicator and pre-
configured indicator
parameter of each resource usage indicator; obtaining adjustment strategy of
current Flink task
according to the resource usage total score of current Mink task and the
mapping relationship
between pre-configured resource usage total score and adjustment strategy;
adjusting resource
allocation of current Flink task according to adjustment strategy of current
Flink task.
[0167] In an embodiment, the above-mentioned indicator parameter includes the
maximum
Date Regue/Date Received 2023-01-24
value and the indicator highest score, the processor executes the computer
program to achieve
the above-mentioned generating resource usage total score of current Flink
task according to
the actual usage of each resource usage indicator and pre-configured indicator
parameter of
each resource usage indicator, specifically achieving following steps:
[0168] When the evaluation factor of the resource usage indicator belonging is
first factor,
generating usage ratio of the resource usage indicator of each first factor
according to the actual
usage amount and the maximum value of the resource usage indicator of each
first factor;
[0169] Generating first score according to the usage ratio of the resource
usage indicator of
each first factor and the indicator highest score;
[0170] When the evaluation factor of the resource usage indicator belonging is
second factor,
the resource usage indicator includes first indicator and second indicator,
generating
consumption ratio based on the actual usage of first indicator and the actual
usage of second
indicator;
[0171] Generating second score according to the consumption ratio and the
highest indicator
score of the resource usage indicator of second factor;
[0172] Generating resource usage total score of current Flink task according
to the second
score and the first score.
[0173] In an embodiment, the processor executes the computer program to
achieve the above-
mentioned obtaining adjustment strategy of current Flink task according to the
resource usage
total score of current Flink task and the mapping relationship between pre-
configured resource
usage total score and adjustment strategy, specifically achieving following
steps:
[0174] Obtaining adjustment strategy matching the range of the resource usage
total score of
current Flink task according to the mapping relationship between the resource
usage total score
26
Date Regue/Date Received 2023-01-24
and the adjustment strategy, then obtaining adjustment strategy of current
Ffink task.
[0175] In an embodiment, the adjustment strategy of current Flink task
includes adjustment
value corresponding to resource usage indicator to be adjusted, the processor
executes the
computer program to achieve the above-mentioned adjusting resource allocation
of current
Flink task according to adjustment strategy of current Flink task,
specifically achieving
following steps:
[0176] Adjusting the resource allocation amount corresponding to each resource
usage
indicator of current Flink task according to adjustment value corresponding to
resource usage
indicator to be adjusted.
[0177] In an embodiment, the processor executes the computer program to
achieve following
steps:
[0178] When receiving resource adjustment request sent by terminal, executing
the step of
obtaining actual usage of each preset resource usage indicator during
execution of current Ffink
task;
[0179] Alternatively, when detecting of the current Flink task starts to run,
executing the step
of obtaining actual usage of each preset resource usage indicator during
execution of current
Flink task.
[0180] In an embodiment, the processor executes the computer program to
achieve following
steps:
[0181] Comparing actual usage corresponding to each resource usage indicator
with preset
threshold to obtain comparison result;
[0182] Generating detailed resource usage information corresponding to each
resource usage
27
Date Recue/Date Received 2023-01-24
indicator according to the comparison result;
[0183] Sending the detailed resource usage information to the terminal for
display.
[0184] In an embodiment, the processor executes the computer program to
achieve following
steps:
[0185] Receiving configuration request for the mapping relationship between
resource usage
total score and adjustment strategy;
[0186] Creating the mapping relationship between the newly added adjustment
strategy and
the resource usage total score corresponding to the newly added adjustment
strategy according
to the newly added adjustment strategy in the configuration request and the
resource usage total
score corresponding to the newly added adjustment strategy.
[0187] The skilled in the art can understand that all or partial of procedures
from the above-
mentioned methods can be performed by computer program instructions through
related
hardware, the mentioned computer program can be stored in a non-volatile
material computer
readable storage medium, this computer can include various embodiment
procedures from the
abovementioned methods when execution. Any reference to the memory, the
storage, the
database, or the other media used in each embodiment provided in current
application can
include non-volatile and/or volatile memory. Non-volatile memory can include
read-only
memory (ROM), programable ROM (PROM), electrically programmable ROM (EPRPMD),
electrically erasable programmable ROM (EEPROM) or flash memory. Volatile
memory can
include random access memory (RAM) or external cache memory. As an instruction
but not
limited to, RAM is available in many forms such as static RAM (SRAM), dynamic
RAM
(DRAMD), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM),
enhanced SRAM (ESDRAM), synchronal link (Synchlink) DRAM (SLDRAM), memory bus
(Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and
memory bus dynamic RAM (RDRAM), etc.
28
Date Regue/Date Received 2023-01-24
[0188] The technical features of the above-mentioned embodiments can be
randomly
combined, for concisely statement, not all possible combinations of technical
feature in the
abovementioned embodiments are described. However, if there are no conflicts
in the
combinations of these technical features, it shall be within the scope of this
description.
[0189] The above-mentioned embodiments are only several embodiments in this
disclosure
and the description is more specific and detailed but cannot be understood as
the limitation of
the scope of the invention patent. Evidently those ordinary skilled in the art
can make various
modifications and variations to the disclosure without departing from the
spirit and scope of
the disclosure. Therefore, the appended claims are intended to be construed as
encompassing
the described embodiment and all the modifications and variations coming into
the scope of
the disclosure.
29
Date Regue/Date Received 2023-01-24
