Note: Descriptions are shown in the official language in which they were submitted.
CA 03070860 2020-01-22
WO 2019/027458 PCMJS2017/045156
INTEGRATING REAL-TIME PERSONALIZED DOCUMENTS INTO A VIDEO
CONFERENCE SESSION
BACKGROUND
Field
[0001] Embodiments of the present invention generally relate to video
conferencing, and more specifically, to integrating real-time, personalized
documents into a video conference session.
Description of the Related Art
[0002] Often,
users engage in video conferencing sessions with professionals,
such as accountants, attorneys, and financial advisors. In many
cases, a
professional requires additional information (such as documentation) from the
users
so that the professional may provide different services to the users. One
object of
the video conferencing session is for the professional to identify the
necessary
additional information. However, the video conferencing session imparts
limitations
on such scenarios. For example, the video conference often occupies an entire
screen, making it difficult to create and share lists of necessary
information. Instead,
the professional often prepares such a list outside of the video conference
session,
and emails the list to the user after the video conference session. However,
such a
scenario leads to questions, missed information, and incomplete or
incomprehensive
lists.
SUM MARY
[0003] In one
embodiment, a method comprises establishing a video conference
between a first application instance and a second application instance, and
during
the video conference: receiving, by an application server from the first
application
instance, an indication of a first data element required to prepare a tax
return,
storing, by a notification server, the indication of the first data element,
and
publishing, by a notification server, the indication of the first data element
to the
second application instance, wherein the second application instance is
configured
to output the indication of the data element for display.
1
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
[0004] In another embodiment, a system comprises a processor and a memory
containing a program which when executed by the processor performs an
operation
comprising establishing a video conference between a first application
instance and
a second application instance, and during the video conference: receiving, by
an
application server from the first application instance, an indication of a
first data
element required to prepare a tax return, storing, by a notification server,
the
indication of the first data element, and publishing, by a notification
server, the
indication of the first data element to the second application instance,
wherein the
second application instance is configured to output the indication of the data
element
for display.
[0005] In another embodiment, a non-transitory computer readable medium
stores instructions, which, when executed by a processor, performs an
operation
comprising establishing a video conference between a first application
instance and
a second application instance, and during the video conference: receiving, by
an
application server from the first application instance, an indication of a
first data
element required to prepare a tax return, storing, by a notification server,
the
indication of the first data element, and publishing, by a notification
server, the
indication of the first data element to the second application instance,
wherein the
second application instance is configured to output the indication of the data
element
for display.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] So that the manner in which the above recited features of the
present
disclosure can be understood in detail, a more particular description of the
disclosure, briefly summarized above, may be had by reference to embodiments,
some of which are illustrated in the appended drawings. It is to be noted,
however,
that the appended drawings illustrate only exemplary embodiments and are
therefore not to be considered limiting of its scope, may admit to other
equally
effective embodiments.
[0007] Figure 1 illustrates an example system for integrating real-time
personalized documents into a video conference session, according to one
embodiment.
2
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
[0008] Figures 2A-2C illustrate example graphical user interfaces for
integrating
real-time personalized documents into a video conference session, according to
various embodiments.
[0009] Figure 3 is a flow chart illustrating a method to integrate real-
time
personalized documents into a video conference session, according to one
embodiment.
[0010] Figure 4 is a flow chart illustrating a method to establish a video
conference session, according to one embodiment.
[0011] Figure 5 is a flow chart illustrating a method to generate a list of
real-time
personalized documents, according to one embodiment.
[0012] Figure 6 is a block diagram illustrating a system configured to
integrate
real-time personalized documents into a video conference session, according to
one
embodiment.
[0013] To facilitate understanding, identical reference numerals have been
used,
where possible, to designate identical elements that are common to the
figures. It is
contemplated that elements and features of one embodiment may be beneficially
incorporated in other embodiments without further recitation.
DETAILED DESCRIPTION
[0014] Embodiments disclosed herein provide techniques to integrate real-
time,
personalized documents in video conferencing sessions. Generally, one or more
client users may participate in a networked video conferencing session with a
professional user via respective instances of a common application. During the
video conference, embodiments disclosed herein provide graphical user
interfaces
(GUIs) that allow the professional user to generate indications of required
documentation (or other data). A notification server orchestrates the receipt
of the
generated indications from the professional user, and the transmission of the
generated indications to the client user (e.g., in real-time as they are
generated by
the professional user). Furthermore, embodiments disclosed herein provide GUIs
to
allow the client user to view the indications in real-time.
3
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
[0015] Figure 1 illustrates an example system 100 for integrating real-time
personalized documents into a video conference session, according to one
embodiment. As shown, the system 100 includes a client system 101, one or more
servers 1021_N, and a professional system 103. The client system 101 and
professional system 103 are representative of any type of physical or virtual
computing system, such as a laptop, desktop, tablet, and/or mobile device. As
shown, the client system 101 and the professional system 103 include
respective
instances of a tax application 104, namely instances 104A and 104C. The tax
application 104 is generally configured to facilitate preparation of a tax
return for one
or more users. As shown, the instances of the tax application 104A,C include a
video conferencing interface 111 and data stores of client data 112. The video
conferencing interface 111 allows users of the client system 101 and
professional
system 103 to engage in a real-time audio/video conference. For example, a
user of
the client system 101 may initiate a video conference with a tax professional.
The
user may be paired with an available tax professional, e.g., a user of the
professional system 103, and a video conference is established.
[0016] During the video conference, the parties may discuss the information
needed for the professional to prepare a tax return for the client. The tax
professional may then identify data elements necessary to prepare the tax
return,
such as income statements, receipts, and the like. As shown, the instance of
the tax
application 104C on the professional system 103 provides a generation
interface
113 that allows the professional to enter indications of the necessary data
elements.
Once entered by the professional, the indications may be stored in the
instance of
the client data 112 on the professional system 103 (and/or on the servers
1021_N).
The instance of the tax application 104A on the client system 101 includes a
client
data interface 115 for viewing the indications of data elements entered by the
tax
professional.
[0017] As shown, the servers 1021_N include a plurality of application
servers
1051_N, a plurality of notification servers 1061_N, and a data store of client
data 112.
The servers 102, 105, and 106 are representative of physical and virtualized
servers. The application servers 1051-N execute instances of the tax
application 104,
namely instances 104B, which support the functionality (including video
4
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
conferencing) provided by the tax application 104. The notification servers
1061-N
are generally configured to receive indications of data elements entered by
the
professional via the generation interface 113, and store the indications in
the client
data cache 114. The
notification servers 1061-N may then generate push
notifications of the data elements entered by the professional, and transmit
the
notifications to any subscribing application server 1051_N and/or subscribing
instances of the client data interface 115.
[0018]
Generally, a given instance of the tax application 104 executing on the
client system 101 and/or professional system 103 establishes a socket
connection
with an application server 1051-N when the respective instance of the tax
application
104 is launched. For example, the instance of the tax application 104A
executing on
the client system 101 may establish a socket with application server 1051,
while the
instance of the tax application 104C executing on the professional system 103
may
establish a socket connection with the application server 1052. In other
embodiments, the instances of the tax application 104A,C on the systems 101,
103
may establish socket connections with the same application server 105N.
[0019] A user
of the client system 101 may then use the video conferencing
interface 111 to initiate a video conference with a professional (e.g., a user
of the
professional system 103). The instances of the tax application 104B on the
application servers 1051_N may then establish a connection to initiate the
video
conference between the client and the tax professional. Furthermore, a socket
connection is established between the associated application servers 1051_N
and the
notification servers 1061_N. For example, the instances of the tax application
104B
and/or the respective application servers 1051.2 may each establish a socket
connection with the notification server 1061. The client data interface 115
and/or the
tax application 104A on the client system 101 may then subscribe to receive
push
notifications from the notification server 1061. In at least one embodiment,
the
respective instance of the tax application 104B on the application server 1051
subscribes to receive push notifications from the notification server 1061,
which then
pushes applicable notifications to the client data interface 115. Therefore,
in one
embodiment, when the tax professional enters indications of needed data
elements
via the generation interface 113, the generation interface 113 transmits an
indication
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
of the needed data element to the application server 1052. The instance of the
tax
application 104B executing on the application server 1052 then transmits an
indication of the needed data element to the notification server 1061, which
stores
the indication in the client data cache 114. In another embodiment, the
instance of
the tax application 104C on the professional system 103 transmits the
indication
directly to the notification server 1061.
[0020] The notification server 1061 then generates a push notification to
all
subscribing client data interfaces 115. Continuing with this example, the push
notification is sent from the notification server 1061 to the application
server 1051,
where the instance of the tax application 104B transmits the push notification
to the
client data interface 115 on the client system 101. The client data interface
115 then
outputs an indication of the push notification to the user, and optionally
stores an
indication of the needed data in the client data 112.
[0021] Figure 2A illustrates an example GUI 200 used by a professional user
of
the professional system 103, according to one embodiment. As shown, the GUI
200
includes a video 205 and an example generation interface 113. The video 205
depicts the video data of the user of the client system 101. The generation
interface
113 allows the professional user to enter indications of needed data elements.
As
shown, the generation interface 113 includes a plurality of fields 210
corresponding
to different types of tax forms. The generation interface 113 also includes a
plurality
of fields 211 allowing the tax professional to enter additional information
related to
the corresponding type of tax form 210 (e.g., the professional may indicate a
particular employer from whom a particular type of tax form is needed).
Therefore,
as shown, the tax professional has entered two example W2 forms and an example
form 1099-INT via the generation interface 113 of the GUI 200. As the tax
professional enters each entry, an indication of the entry is transmitted to
the
corresponding application server 105N, which then forwards the indication to
the
corresponding notification server 106N. The indication may include metadata
describing the required data element, such as the type of data element, and a
source of the data element (e.g., a W2 from employer "example1"). The
indication
may further include a copy of the data element, or a template thereof (e.g., a
blank
W2 form, a W2 form pre-populated with some information describing the user,
etc.).
6
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
The notification server 106N may then store the indication in the client data
cache
114. In some embodiments, the notification server 106N stores a copy of the
data
element. However, if the size of the data element exceeds a threshold, the
notification server 106N may store a link to the data element in the cache
114, and
store the actual data element at a different network location to minimize the
size of
the cache 114.
[0022] The notification server 106N then generates a push notification
describing
the data element, which is transmitted to all subscribing client data
interfaces 115.
Figure 2B illustrates an example GUI 201 which includes a video 205 and an
example client data interface 115, according to one embodiment. The video 205
is
the video data from the tax professional using the professional system 103 in
the
video conference. As shown, the client data interface 115 includes a list of
data
elements that correspond to the data elements entered by the tax professional
via
the generation interface 113. Generally, the client data interface 115
displays new
data elements in real-time, as they are added by the tax professional via the
generation interface 113.
[0023] Figure 2C depicts a GUI 202 which includes a list 220 of data
elements,
according to one embodiment. In at least one embodiment, the GUI 202 is
accessible to the user of the client device 101 outside of a video
conferencing
session (e.g., after a tax consultation via video conferencing has completed).
As
shown, the list 220 includes example data elements (e.g., tax forms, address
information, user metadata) that the tax professional entered via the
generation
interface 113. As shown, the GUI 202 further includes elements 221 that allow
the
user to enter the corresponding information. For example, the user may upload
a
W2 document, enter address information, and the like, via the GUI elements
221.
Once the user provides the necessary information, the information is stored in
the
client data 112 (e.g., on one or more of the client device 101, the servers
1021_N,
and/or the professional device 103). Furthermore, the user provides the
necessary
information, the GUIs 200-202 may be updated to reflect the user submission.
For
example, the professional may use the GUI 200 to view the information (e.g.,
forms,
metadata, etc.) submitted by the user. In at least one embodiment, as the user
provides the information, the application server 105N and/or the notification
server
7
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
106N generates and sends a notification to the professional. For example, if a
user
uploads their W2 document, the application server 105N and/or notification
server
106N may generate and send an email to the associated tax professional.
[0024] Figure 3
is a flow chart illustrating a method 300 to integrate real-time
personalized documents into a video conference session, according to one
embodiment. As shown, the method 300 begins at block 310, where an instance of
the tax application 104A executing on the client device 101 establishes a
socket
connection with an application server 1051-N. At block
320, the connected
application server 1051-N receives a request from the instance of the tax
application
104A executing on the client device 101 to initiate a video conference with a
tax
professional. At block 330, described in greater detail with reference to
Figure 4, a
video conference session is established between the requesting user and a tax
professional. At block 340, described in greater detail with reference to
Figure 5, the
tax professional generates a real-time list during the video conferencing
session.
The real-time list includes one or more required data elements necessary to
allow
the tax professional to prepare the requesting user's taxes. At block 350, the
real-
time list and associated objects are integrated into the video conference. For
example, as the tax professional enters indications of the data elements via
the GUI
200, the user of the client device 101 may view the indications of the data
elements
in real-time via the GUI 201. At block 360, the tax application 104A may
optionally
provide the list generated at block 350 to the user (e.g., via email and/or
the GUI
202) after the video conference is completed.
[0025] Figure 4
is a flow chart illustrating a method 400 corresponding to block
330 to establish a video conference session, according to one embodiment. As
shown, the method 400 begins at block 410, where an available tax professional
is
identified. Generally, the instances of the tax application 10413 on the
application
servers 1051-N maintain indications of tax professionals available for video
conferences. The instances of the tax application 10413 on the application
servers
1051-N may select an available tax professional based on any number of
factors,
such as an expertise of the tax professional matching a need or business
requirement of the client, a favorability rating of the tax professional, an
amount of
time since the last video conference the tax professional participated in, and
the like.
8
[0026] At block 420, a socket connection is established between the
application
servers 1051-N connected to the instances of the tax application 104A,C
executing
on the client device 101 and the professional device 103 (e.g., the devices
participating in the video conference). At block 430, a socket connection is
established between a notification server 1061-N and the application server
1051-N to
which the instance of the tax application 104C of the professional device 103
is
connected. At block 440, a socket connection is established between a
notification
server 1061-N and the application server 1051-N to which the instance of the
tax
application 104A executing on the client device 101 is connected. At block
450, the
application server 1051-N to which the instance of the tax application 104A
executing
on the client device 101 is connected subscribes to push updates provided by
the
notification server 1061-N. In at least one embodiment, the instance of the
tax
application 104A executing on the client device 101 subscribes to receive push
updates from the notification server 1061-N. At block 460, the video
conference is
established, and the parties may begin communicating via the video conference.
[0027] Figure 5 is a flow chart illustrating a method 500 corresponding
to block
340 to generate a list of real-time personalized documents, according to one
embodiment. As shown, the method 500 begins at block 510, where the instance
of
the tax application 104C executing on the professional device 103 receives
indications of needed data elements from the tax professional. Generally, the
tax
professional enters the indications via the video conferencing interface 111
during
the video conference. At block 520, a loop including blocks 530-590 is
executed for each received indication. At block 530, the instance of the tax
application 104C executing on the professional device 103 determines whether
to publish the current data element or a link to the data element, based on
the size
of the current data element. Generally, to reduce the size of the client data
cache
114, data elements having a size that exceeds a threshold are not stored in
the
client data cache 114. Instead, larger data elements are stored at a storage
location in the cloud (e.g., on a server 1021-N), and a uniform resource
locator
(URL) to the storage location is stored in the client data cache 114.
[0028] At block 540, the notification server 1061-N receives an
indication of the
current data element via the socket connection between the notification server
1061_
9
Date Recue/Date Received 2021-08-10
N and the application server 1051-N to which the instance of the tax
application 104C
of the professional device 103 is connected. At block 550, the notification
server
stores an indication of the current data element in the client data cache 114.
Generally, the indication includes metadata describing the current data
element,
such as a type (e.g., tax form, personal metadata, income information, etc.),
an
associated party (e.g., the client, a spouse, a dependent, etc.), and the
like. As
previously indicated, if the size of the current data element does not exceed
the
threshold, the current data element is also stored in the client data cache
114.
However, if the size of the current data element exceeds the threshold, the
current
data element is stored at a storage location in a server 1021-N (e.g., in the
client data
112), and a URL of the storage location is stored in the client data cache
114.
[0029] At block 560, the notification server 1061_N generates a push
notification
for the current data element, and pushes the notification to any subscribed
application servers 1051_N. At block 570, the subscribing application servers
1051_N
transmit the received push notification to the instance of the tax application
104A
executing on the client device 101. At block 580, the instance of the tax
application
104A executing on the client device 101 outputs an indication of the current
data
element for display during the video conference. Doing so allows the client to
view
the information in real-time and during the video conference without having to
switch
screens or manually save the list of required data elements. At block 520, if
more
data elements remain, the method returns to block 520. Otherwise, the method
500
ends.
[0030] Figure 6 is a block diagram illustrating a system 600 configured
to
integrate real-time personalized documents into a video conference session,
according to one embodiment. The networked system 600 includes a computer
602. The computer 602 may also be connected to other computers via a network
630. In general, the network 630 may be a telecommunications network and/or a
wide area network (WAN). In a particular embodiment, the network 630 is the
Internet.
[0031] The computer 602 generally includes a processor 604 which obtains
instructions and data via a bus from a memory 606 and/or a storage 608. The
computer 602 may also include one or more network interface devices 618, input
Date Recue/Date Received 2021-08-10
devices 622, and output devices 624 connected to the bus. The computer 602 is
generally under the control of an operating system (not shown). Examples of
operating systems include the UNIX operating system, versions of the Microsoft
Windows operating system, and distributions of the Linux operating system.
(UNIX
is a registered trademark of The Open Group in the United States and other
countries. Microsoft and Windows are trademarks of Microsoft Corporation in
the
United States, other countries, or both. Linux is a registered trademark of
Linus
Torvalds in the United States, other countries, or both.) More generally, any
operating system supporting the functions disclosed herein may be used. The
processor 604 is a programmable logic device that performs instruction, logic,
and
mathematical processing, and may be representative of one or more CPUs. The
network interface device 618 may be any type of network communications device
allowing the computer 602 to communicate with other computers via the network
630.
[0032] The storage 608 is representative of hard-disk drives, solid
state drives,
flash memory devices, optical media and the like. Generally, the storage 608
stores
application programs and data for use by the computer 602. In addition, the
memory 606 and the storage 608 may be considered to include memory physically
located elsewhere; for example, on another computer coupled to the computer
602
via the bus.
[0033] The input device 622 may be any device for providing input to the
computer 602. For example, a keyboard and/or a mouse may be used. The input
device 622 represents a wide variety of input devices, including keyboards,
mice,
controllers, and so on. Furthermore, the input device 622 may include a set of
buttons, switches or other physical device mechanisms for controlling the
computer
602. The output device 624 may include output devices such as monitors, touch
screen displays, and so on.
[0034] As shown, the memory 606 contains the application servers 1051-N
and
the notification servers 1061-N, each described in greater detail above. As
shown,
the storage 608 contains the client data 112, which stores user profile data.
For
example, the profile data stored in the client data 112 may include data
elements
identified by a tax professional as being necessary to prepare a tax return
for the
11
Date Recue/Date Received 2021-08-10
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
user. As described herein, the data elements may be identified during a video
conference leveraging the video cameras 650 of the client system 101 and the
professional system. Generally, the system 600 is configured to implement all
systems, methods, and functionality described above with reference to Figures
1-5.
[0035] It may
be noted that, descriptions of embodiments of the present
disclosure are presented above for purposes of illustration, but embodiments
of the
present disclosure are not intended to be limited to any of the disclosed
embodiments. Many modifications and variations will be apparent to those of
ordinary skill in the art without departing from the scope and spirit of the
described
embodiments. The terminology used herein was chosen to best explain the
principles of the embodiments, the practical application or technical
improvement
over technologies found in the marketplace, or to enable others of ordinary
skill in
the art to understand the embodiments disclosed herein.
[0036] In the
preceding, reference is made to embodiments presented in this
disclosure. However, the scope of the present disclosure is not limited to
specific
described embodiments. Instead, any combination of the recitd features and
elements, whether related to different embodiments or not, is contemplated to
implement and practice contemplated embodiments.
Furthermore, although
embodiments disclosed herein may achieve advantages over other possible
solutions or over the prior art, whether or not a particular advantage is
achieved by a
given embodiment is not limiting of the scope of the present disclosure. Thus,
the
recited aspects, features, embodiments and advantages are merely illustrative
and
are not considered elements or limitations of the appended claims except where
explicitly recited in a claim(s). Likewise, reference to the invention" shall
not be
construed as a generalization of any inventive subject matter disclosed herein
and
shall not be considered to be an element or limitation of the appended claims
except
where explicitly recited in a claim(s).
[0037] Aspects
of the present disclosure may take the form of an entirely
hardware embodiment, an entirely software embodiment (including firmware,
resident software, micro-code, etc.) or an embodiment combining software and
hardware aspects that may all generally be referred to herein as a
"component,"
"circuit," "module" or "system." Furthermore, aspects of the present
disclosure may
12
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
take the form of a computer program product embodied in one or more computer
readable medium(s) having computer readable program code embodied thereon.
[0038] Any combination of one or more computer readable medium(s) may be
utilized. The computer readable medium may be a computer readable signal
medium or a computer readable storage medium. A computer readable storage
medium may be, for example, but not limited to, an electronic, magnetic,
optical,
electromagnetic, infrared, or semiconductor system, apparatus, or device, or
any
suitable combination of the foregoing. More specific examples a computer
readable
storage medium include: an electrical connection having one or more wires, a
hard
disk, a random access memory (RAM), a read-only memory (ROM), an erasable
programmable read-only memory (EPROM or Flash memory), an optical fiber, a
portable compact disc read-only memory (CD-ROM), an optical storage device, a
magnetic storage device, or any suitable combination of the foregoing. In the
current context, a computer readable storage medium may be any tangible medium
that can contain, or store a program.
[0039] The flowcharts and block diagrams in the Figures illustrate the
architecture, functionality and operation of possible implementations of
systems,
methods and computer program products according to various embodiments of the
present invention. In this regard, each block in the flowchart or block
diagrams may
represent a module, segment or portion of code, which comprises one or more
executable instructions for implementing the specified logical function(s). In
some
alternative implementations the functions noted in the block may occur out of
the
order noted in the figures. For example, two blocks shown in succession may,
in
fact, be executed substantially concurrently, or the blocks may sometimes be
executed in the reverse order, depending upon the functionality involved. Each
block of the block diagrams and/or flowchart illustrations, and combinations
of blocks
in the block diagrams and/or flowchart illustrations can be implemented by
special-
purpose hardware-based systems that perform the specified functions or acts,
or
combinations of special purpose hardware and computer instructions.
[0040] While the foregoing is directed to embodiments of the present
disclosure,
other and further embodiments of the disclosure may be devised without
departing
13
CA 03070860 2020-01-22
WO 2019/027458 PCT/US2017/045156
from the basic scope thereof, and the scope thereof is determined by the
claims that
follow.
14
