Note: Descriptions are shown in the official language in which they were submitted.
CA 02306524 2000-04-25
A SYSTEM AND METHOD FOR ENHANCED STREAMING MEDIA VIEWING
The present invention relates to networked multimedia systems, and more
particularly to a
system and method for improving the perceived quality of multimedia
information delivered to
one or more clients over a limited bandwidth network.
BACKGROUND OF THE INVENTION
The explosive growth of the Internet has resulted in a growing need to provide
multimedia
o material such as video and audio to users. Interactive television, movies on
demand and other
multimedia technologies are becoming more popular. This popularity coupled
with an ever-
increasing demand for a high quality of service (QoS) has placed significant
demands on
bandwidth and hardware requirements of the Internet.
15 The Internet, like other networks, is based on an IP protocol. Packets of
data are routed around
the network between a sender and recipient based on the address of the
intended recipient
contained in the packet. Thus, a specific connection between the sender and
recipient is not
required. Generally, packets may be sent via different routes in an effort to
increase the
likelihood of delivery, which means that data packets do not inherently have a
guaranteed arnval
2o time. In fact, packets corresponding to a signal message may even be
received out of order.
This fact significantly affects how certain multimedia data are delivered.
Since multimedia data
requires real time delivery, a particular media stream needs to be delivered
in the proper time
sequence, to allow the user to play back the audio/video selection as real
motion or real audio.
25 In order to support the delivery of realtime content over the Internet, a
realtime protocol (RTP)
was introduced to replace the conventional transmission control protocol
(TCP). The RTP
provides a framework that real time applications can use directly for data
transport. Currently,
the RTP standard supports a particular type of message, namely one for
carrying the media
content data for streaming data. A separate protocol known as the realtime
control protocol
30 (RTCP) is used with the RTP to pass control message for session management,
rate adaptation
and the like. Along with the realtime aspect of multimedia, as described
above, multimedia data
typically require an enormous number of bytes to represent the information
contained therein.
CA 02306524 2000-04-25
Consequently, compression techniques are utilized to reduce the amount of data
transmitted.
Generally, compression requires, an encoder to compress source data and a
decoder to
decompress the compressed data.
The quality of multimedia transmitted between a server and a client computer
over a network
depends largely on the bandwidth of the network. Bandwidth refers to the
maximum number of
bits that a network can transmit from one node coupled to the network to
another node coupled
to the network. If insufficient bandwidth is available, the number of bits
transmitted is reduced
and consequently, the image or sound of the multimedia is distorted or not
fully represented. In
to many computer networks, the primary element which limits the bandwidth is
the network
interface device which interfaces the server and client computers to the
network. In the Internet
realm, such interface devices include modems and Ethernet controllers.
A solution around this problem is the introduction and adoption of broadband
delivery systems
such as asymmetric digital subscriber lines (ADSLs), cable modems, and ATMs,
to name only a
few. These broadband delivery systems are now capable of offering the consumer
a relatively
inexpensive access to the Internet. While these broadband delivery systems
have made the
reception of the streaming media more satisfying to the user, there is still a
limitation in the
ultimate content that can be provided. Currently, the streaming media content
is still limited to
2o the reception of simulcasts of radio broadcasts over the Internet, video
clips, and very low
quality or low definition television.
Even though improved protocols, hardware, compression and decompression
techniques and
increased network bandwidth are being introduced, the number of users
requiring or requesting
such streaming media is increasing proportionally. Thus, users are still
experiencing fluctuations
in bandwidth in the network causing choking and picture freezing, making
streaming media still
annoying to the end user.
Accordingly, there is a need for a system and method that is capable of
allowing streaming
media such as television and movies to be transmitted over the Internet while
providing the user
CA 02306524 2000-04-25
with a perceived transmission improvement over current transmissions for
realtime television,
audio transmission or the like.
SUMMARY OF THE INVENTION
It is an advantage of the present invention that streaming media provided to a
user appears to
flow seamlessly.
According to an embodiment of the present invention there is provided a method
for transmitting
streamed digital data from a server, the server being configured for coupling
to a client computer
1o via a computer network. The method comprises the following steps. A
modified data stream is
provided to the media player, the modified data stream including a plurality
of markers located at
predetermined positions for indicating new content insertion points. The
modified data stream is
transmitted to a requesting client which monitors the data rate of the
received media. The client
inserts locally stored content at one or more marker positions while buffering
the received data
IS upon the data rate being below a predetermined level. Playback of the
streaming data is resumed
upon the data rate returning to the predetermined level.
BRIEF DESCRIPTION OF THE DRAWINGS
An embodiment of the invention will now be described by way of example only
with reference
2o to the following drawings in which:
Figure 1 is a schematic block diagram of a multimedia delivery system;
Figure 2 is a schematic block diagram of a media player according to an
embodiment of
the present invention;
Figure 3 is a flow chart depicting a method for multimedia display according
to the present
25 invention;
Figure 4(a) and 4(b) are schematic diagrams of a data buffer in a first state;
and
Figure 5(a) and 5(b) are schematic diagrams of a data buffer in a second
state.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
CA 02306524 2000-04-25
In the following description, like numerals refer to like structures in the
diagrams. Referring to
figure 1, a streaming media system is shown generally by numeral 100. The
system 100 includes
a streaming media server 112 for serving media files 114 over a network 116 to
a requesting
client 117. The client 117 is typically a personal computer, hand-held
computer or similar
device, which includes a memory, a processor and software for providing
control instructions to
the media server and for displaying the received media information on a
display device 119.
Typically, this client software is termed a media player and executes within a
web browser such
as Windows Internet ExplorerTM or Netscape NavigatorTM. Alternately, the media
player may be
1o executable without the need for a web browser, such as RealAudioTM. The
client computer also
includes a storage device 118, such as a hard disk or RAM memory or similar.
On the server
side 112, there is included a specialized piece of software known as the media
server which may
perform one or more functions of compressing the media data, multicasting and
multistreaming.
Most commercially available media servers stream the media information using
one or more
15 protocols over the Internet standard TCP protocol. For example, a media
server by VivoTM
offers an HTTP based streaming that enables almost any web site to offer
streaming video clips,
other vendors like VDONet, offer a UDP (user datagram protocol) based server.
In general
however, both the media server and the client player are matched to
communicate via a mutually
recognized protocol. The latter media player is downloaded by the user when
attempting to
2o access the particular media server 112. Content for the media server 112
may be derived from
prestored video, television, audio or similar. Alternatively, the content may
be provided by a
live feed 120 such as a radio broadcast or television transmission.
While the above describes the basic components of a multimedia system, the
present invention
25 provides an improvement to this system in a manner to be described below.
The improvement
allows the existing multimedia system architecture as described above to be
utilized in order to
provide the user with a perceived higher quality display of multimedia
information. For
example, real motion, high resolution video should be optimally displayed at
24 to 30 frames per
second. An advantage of the present invention is to achieve such a perceived
quality of service.
3o The present invention attempts to replicate a user's familiarity with the
format of current
television broadcast programs to create the impression of smooth real time
real motion video in a
4
CA 02306524 2000-04-25
multicasting or multistreaming environment. More specifically, television
viewers are
accustomed to television programs which typically in five to ten minute
segments followed by
advertising or commercials which run for two to three minutes. These
commercials are inserted
in the broadcast video by local distributors. These commercials are generally
known in advance
of the broadcast.
Refernng now to figure 2, a schematic diagram of a media player according to
an embodiment of
the present invention is shown generally by numeral 200. The player includes
an executive 202
for controlling and conducting operation of the media player and a user input
interface 204 for
t o receiving user commands; a module for determining the incoming data rate
206; a cache 208 for
buffering an incoming data stream under control of the executive 202, a
storage for prestored
information 210, and a display module 219 for transferring the received media
stream to the
display 119 as determined by a scheduling algorithm.
15 The operation of the system may be explained with reference to figure 3.
Initially, television
commercials or other predetermined information is transferred to the client
computer for storage
in the prestored information memory 210 at one or more of a plurality of
predetermined time
periods. These commercials will be used in lieu of streaming on-demand
commercials from a
server, and will be inserted automatically by the media player 200 when
triggered by a marker as
2o discussed with reference to figure 4.
Commercial clips can be downloaded using several methods. The clips can be
downloaded
when the user switches the computers on. They can also be scheduled to
download on a different
connection to the streaming video connection in which case priority should be
given to the video
25 streaming. Commercials can also be downloaded during the normal commercial
breaks that are
provided by the broadcasters. While local prerecorded commercials are playing,
additional
downloaded new commercial clips could be downloaded through a dedicated URL.
Furthermore, new commercials can be transmitted to a user's computer during
off peak hours,
while the computer is still connected to the Internet.
CA 02306524 2000-04-25
In order to facilitate the invention, the media to be delivered from the media
server is modified
by the addition of markers to the streaming content. Markers may be inserted
in the data stream
broadcast by a broadcasting station indicating the positions in the streaming
media wherein the
prestored information of commercials 210 may be inserted. These markers may be
of various
forms. For example, a first type of marker termed "a broadcast marker" may
instruct the media
player as to the positions for inserting commercials or a second type of
marker termed a
"supplementary marker" may be inserted in the video stream to indicate
positions in the stream
where the streaming may be interrupted without affecting the smoothness or the
viewing of audio
content as perceived by the user. The insertion of the markers maybe done
manually on
1o prerecorded material or by using a rule-based image analysis algorithm. The
rule-based image
analysis algorithm may be used to determine critical points, such as a scene
change in a video
stream or a pause in an audio stream for insertion of the markers.
Referring back to figure 3, at start up, the media player checks if the media
cache buffer 208 as
shown in figure 2 has sufficient content for display. If not, the prestored
sequence of
commercials is sent to the display from the storage 210. The media player
continues to check the
buffer 208. Upon the buffer being at a sufficient level, the media content is
played. In the case
of video, the video images are displayed or in the case of audio, the audio is
played. The media
player 200 continues to stream data into the buffer 208 while playing. As
described earlier, the
2o streaming data rate will depend on the bandwidth of the network. If the
average bandwidth, as
determined by the media player, is lower than the bandwidth for real time
continuous playback
then the buffered data is used for playback. Consequently, the buffered data
begins to decrease.
When the buffer size reaches a critical point where choking is predicted, the
playback is faded
out and local commercials are shown from the storage 210, while the buffer 208
to rebuild itself
to a sufficient capacity.
In order to avoid interruptions in video sequences which might be annoying to
the user, the
scheduling of the local content done at the appropriate marker position.
3o If the buffer 208 is depleted below the critical level, the scheduling
algorithm is initiated.
Refernng to figure 4 (a) a schematic diagram of the buffer 208 is shown
wherein the buffer
CA 02306524 2000-04-25
contains a "broadcast marker" 402. The marker 402 notifies the user that a
commercial is to be
embedded. The buffer is allowed to deplete beyond the critical level 404, and
the local
commercial is switched in at the broadcast marker 402.
Refernng to figure 4(b), the buffer 208 does not contain a "broadcast marker"
but does contain
"supplementary markers", which is provided by a distributor server? The local
commercial will
be shown at the supplementary marker that is the last one in the buffer.
The algorithm for displaying local content is a dynamic algorithm. For
example, if there are two
1o supplementary markers in the buffer at detection time, and while the buffer
depletes, a third
marker arnves, the scheduling algorithm will wait for this latest arnval
before playing the local
information thereby allowing the maximum playback time without interruption.
During local information display, the buffer 208 fills up to sustainable level
at the end of the
~ 5 current local information playback at which point normal video streaming
resumes.
If, due to excess choking, a large number of commercials were played, the
scheduling algorithm
adjusts the local content playback where possible and resumes normal media
streaming as soon
as possible, perhaps even skipping scheduled local information playback. A
total number of
2o commercial broadcasts are stored in an accumulator. The accumulator is
increased for excessive
numbers of commercials. Referring to figure 5(a), there is shown a buffer
having a broadcast
marker 402, which is encountered with a relatively full buffer. In this
particular circumstance,
other broadcast markers are present in the buffer and the commercial quota has
been exceeded.
Therefore, the local content playback is skipped until the next broadcast
marker. Refernng to
25 figure 5(b), there are no other broadcast markers 402, but there are
supplementary markers and
the commercial quota has been exceeded. The local content playback showing is
delayed as long
as there are markers in the buffer.
During the periods illustrated in figures 5(a) and 5(b), the streaming media
server transmits
3o markers while the buffered stream is being played. If the markers can be
skipped as previously
CA 02306524 2000-04-25
described, the accumulator is decreased appropriately. When the accumulator is
empty, normal
local content synchronization can start again at the next available marker.
The size of the buffer 208 is calculated with respect to the statistical
measurement of average
bandwidth on the channel, and the critical level is determined as a percentage
of a full buffer.
In addition to the above scheduling scheme, the case of sustained low
bandwidth is addressed. If
the average bandwidth drops below the required streaming needs for an extended
period, and the
local commercial scheduling has been exhausted, an auxiliary connection with
the server may be
1o established to instruct the server to decrease the frame rate of the
transmission. In that case the
server will drop, for example, to 25 or 24 frames per second, or even to 15
frames per second as
required. When bandwidth quality resumes to an acceptable level, the server
will increase again
the frame rate to a higher level, such as 30 frames per second. The
synchronization between the
client and the server is controlled so that the content is played back.
Another feature for improving noticeable quality is having the media player
check for the
existence of other connections or routing having a higher bandwidth. When this
is detected, the
client connection can seamlessly switch to the better connection (similar to
telephone channel
switching over satellite for overseas routing).
In summary, the present invention provides a new media player and a modified
content which is
used by the media player to improve a user's perceived quality of service of
the playback media.
As described earlier, streaming media servers use a client/server arrangement
to allow insertion
of streaming media advertisements (commercials). The client end begins a new
stream for each
inserted commercial in association with a downloaded script. In an alternate
embodiment, the
streaming media server uses a server push model to insert streaming media
advertisements
directly into a live broadcast media stream or pre-recorded stream. The server
inserts the
commercial at a pre-determined time and location in the streaming media, and
does not require
3o any feedback or communication from the client.
CA 02306524 2000-04-25
In a further application of an embodiment of this invention, an on demand
marketing and on
demand advertising system may be implemented. In this instance, the client
computer receives
instructions from a user or viewer via a keyboard, mouse or any other suitable
input means. The
instructions received by the client is forwarded to the server for requesting
certain types of
advertising content. The content may be specified by the viewer or user in
order to receive
specifically targeted marketing information during a streaming media session.
More specifically,
during a streaming media session, the client may during the playback of local
content such as a
commercial, interact with the client computer to connect to an advertiser
streaming media site.
These sites may offer such things as motor vehicles and may also include a
live video interactive
to session. In this instance, the client is provided with a web camera or
similar for communication
directly with a live person. Thus, the client may include a teleconferencing
camera with an audio
input to enable a user view to talk live to a customer service representative
of the advertiser or
the like. During this interaction, the player may continue to stream video
into the buffer for
resuming playback once the user has terminated the interactive session.
A further embodiment of the invention provides for a plurality of channels
which are available to
the client. Each channel having a unique set of commercials. The server
inserts markers
corresponding to an appropriate set of commercials or locally stored media in
the appropriate
channel indicated provision of the allowable content. These in markers may be
read by the
2o client, inform the client as to whether the appropriate group of
commercials is to be played or
not. Generally, the group of commercials may be a predetermined sequence of
advertising clips
which are thematically related.
The inserted commercial can reside on the same server as the pre-recorded clip
or broadcast, or
reside on a remote commercial server, and be automatically inserted at the
media server end.
In yet another embodiment, a client pull streaming media model allows for
communication
between the streaming media server and client for determining when commercials
are to be
inserted. The server generates a script that is read by the client. The client
then makes a request
for content based on this script. The script can also be modified by the
client based on a set of
rules or information that it has gathered from the end user.
CA 02306524 2000-04-25
Once the client has made the request for insertion of content, the server
responds by modifying
it's current stream of information, or launches another stream for the
commercial. After the
commercial has completed playing, the client issues a request to resume the
original content
stream.
In yet an alternate embodiment, a targeted streaming media ad insertion model
allows the
streaming media server to gather user information from client applications
installed on a user's
computer.
The server collects information from the client application based on the
following criteria: The
user's responses to forms and surveys; the user's viewing habits and
timelines; the length of
viewing; the number of commercials served; the type of content viewed; the
user's "hot click"
actions; the user clicking on a commercial to launch a separate
HTML/DHTML/XML/JAVA
window; purchases based on commercial content; the user's geographic location;
and any other
information gathered as statistics or usage logs
The server then sorts and analyzes the information into a user profile, and
allows for narrow
target marketing. All of the data collection for this model is based upon a
permissions system
2o that requires user permission to use and analyze collected data.
Further, a streaming media portal site is provided as a human indexed
directory of video and
audio on the Internet. The portal is being provided as a preferred location on
the Internet for
showcasing and indexing rich media. Users are able to search for video and
audio clips that are
hosted on other companies' servers, and find scheduled live events from around
the world.
The use of human indexing allows for a better quality Internet experience
versus sites such as
YahooTM, InfoseekTM, and AltavistaTM that feature computer indexed content.
Video and audio
are best experienced and described by humans.
10
CA 02306524 2000-04-25
The streaming media portal site is positioned extremely well. Current sites
indexing video and
audio are fragmented and developed by companies with an interest in featuring
their own media
formats. By offering all video formats, the streaming media portal builds
customer loyalty, and
can become the one stop for all media on the Internet.
The site will be expanded with weekly programming guides delivered directly to
users,
chatrooms, sponsored live events, and other community building features.
Although the invention has been described with reference to certain specific
embodiments,
1o various modifications thereof will be apparent to those skilled in the art.
11
