Note: Descriptions are shown in the official language in which they were submitted.
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
SYSTEM, APPARATUS AND METHOD FOR PROVIDING
IMPROVED PERFORMANCE OF AGGREGATED/BONDED NETWORK
CONNECTIONS BETWEEN REMOTE SITES
FIELD OF THE INVENTION
[0001] The present invention relates generally to network
communications and, in
particular, to aggregating or bonding communications links so as to improve
network
performance such as Cable, DSL, T1, or Metro Ethernet, so as to improve
network
performance or quality of services in regards to a variety of different
networks including
wired and wireless networks, and including Wide Area Networks ("WAN").
BACKGROUND
[0002] While the capacity of network connections has increased since
the
introduction of dial up, high speed connectivity is not ubiquitous in all
regions. Also,
bandwidth is not an unlimited resource and there is a need for solutions that
improve the
utilization bandwidth and also that address network performance issues.
[0003] Various solutions exist for improving network performance such
as load
balancing, bonding of links to increase throughput, as well as aggregation of
links. In
regards to bonding/aggregation various different technologies exist that allow
two or more
diverse links (which in this disclosure refers to links associated with
different types of
networks and/or different network carriers) are associated with one another
for carrying
network traffic (such as a set of packets) across such associated links to
improve network
performance in relation for such packets.
[0004] Examples of such technologies include load balancing, WAN
optimization, or
ANATM technology of TELolP TM, as well as WAN aggregation technologies.
[0005] Many of such technologies for improving network performance are used
to
increase network performance between two or more locations (for example
Location A,
Location B, Location N or the "Locations"), where bonding/aggregation of links
is provided at
one or more of such locations. While the bonded/aggregated links provide
significant
network performance improvement over the connections available to carry
network traffic for
example from Location A to an access point to the backbone of a network
(whether an
Internet access point, or access point to another data network such as a
private data
1
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
network or high performance wireless network) ("network backbone"), the
bonded/aggregated links are generally slower than the network backbone.
[0006]
Prior art technologies including bonding/aggregation generally result in what
is often referred to as "long haul" bonding/aggregation, which means that the
bonded/aggregated links are maintained for example from Location A and
Location 13,
including across the network backbone, which in many cases results in network
impedance.
As a result, while bonding/aggregation provides improved network performance
for example
from Location A to the network backbone, network performance across the entire
network
path for example from Location A to Location 13, may be less than optimal
because the
technology in this case does not take full advantage of the network
performance of the
network backbone.
[0007]
There is a need for a system and method that addresses at least some of
these problems.
SUMMARY OF THE INVENTION
[0008] In one aspect, a networking system is provided for improving network
communication performance between at least a first client site and a second
client site,
wherein the first client site and the second client site are at a distance
from one another that
would usually require long haul network communication, the system comprising:
(A) at least
one network bonding/aggregation computer system that includes: (i) at least
one client site
network component that is implemented at least at a first client site, the
client site network
component bonding or aggregating one or more diverse network connections so as
to
configure a bonded/aggregated connection that has increased throughput; and
(ii) at least
one network server component, configured to interoperate with the client site
network
component, the network server component including a server/concentrator that
is
implemented at an access point to a high performing network; wherein the
client site network
component and the network server component are configured to interoperate so
as to create
and maintain a network overlay for managing network communications between the
at least
first client site and the access point, wherein between the client site
network component and
the network server component data traffic is carried over the
bonded/aggregated connection
and between the access point and the second client site the network server
component
automatically terminates the bonded/aggregated connection and passes the data
traffic to
the associated network backbone, while maintaining management of data traffic
so as to
provide a managed network path that incorporates both at least the
bonded/aggregated
2
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
connection and at least one network path carried over the high performing
network, and
thereby providing improved performance in long haul network connections
between the first
client site and the second client site.
[0009] In another aspect, the distance between the first client site
and the second
client site is such that data traffic over a bonded/aggregated connection
between the first
client site and the second client site is subject to long haul effects.
[0010] In another aspect, the managed network path is maintained
between at least
a first client site and a second client site without routing of network
communications through
a central server that increases long haul effects.
[0011] In another aspect, the plurality of network server components are
implemented in a geographic area so as to provide a Point-of-Presence (PoP),
which are
made available to proximate client site network components.
[0012] In yet another aspect, one or more client site network
components and one or
more associated network server components include peer-to-peer programming and
based
on such peer-to-peer programming operate on a peer to peer basis.
[0013] In yet another aspect, the network server component is
disposed at a distance
from an access point that does not result in long haul effects between the
network server
component and the access point.
[0014] In a still another aspect, the two or more Points-of-Presence
may be
accessible to the at least one client site network component, and the client
site network
component: collects network performance information; and initiates the
configuration of the
network overlay to include one or more network server components so as to
improve
network communication performance.
[0015] In one aspect, each network server component may be accessible
to a
plurality of client site network components, each client site network
component being
associated with a client site.
[0016] In another aspect, the system may include a network of Points-
of-Presence,
distributed geographically so as to serve a plurality of client locations each
associated with at
least one client site network component.
3
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0017] In
yet another aspect, the system may further include: a client site network
component at each of the first client site, and at the second client site; a
network server
component proximal to each of the first client site and proximal to the second
client site;
wherein communications between the first client site's client site network
component and
associated network server component are bonded or aggregated, then terminated
by the
network server component associated with the first client site's client site
network
component, and passed to the high performing network; data traffic is received
by the
network server component associated with the second client site and
transferred on a
bonded or aggregated connection between the network server component
associated with
the second client site and the client site network component associated with
the second
client site.
[0018] In
still another aspect, the bonding/aggregation computer system includes a
network aggregation device that: (A) configures a plurality of dissimilar
network connections
or network connections provided by a plurality of diverse network carriers
("diverse network
connections") as one or more aggregated groups, each aggregated group creating
an
aggregated network connection that is a logical connection of the plurality of
diverse
connections; and (B) routes and handles bi-directional transmissions over the
aggregated
network connection; wherein two or more of the diverse network connections
have dissimilar
network characteristics including variable path bidirectional transfer rates
and latencies;
wherein the logical connection is utilized for a transfer of communication
traffic bidirectionally
on any of the diverse network connections without any configuration for the
dissimilar
network connections or by the diverse network carriers; and wherein the
network
aggregation engine includes or is linked to a network aggregation policy
database that
includes one or more network aggregation policies for configuring the
aggregated groups
within accepted tolerances so as to configure and maintain the aggregated
network
connection so that the logical connection has a total communication traffic
throughput that is
a sum of available communication traffic throughputs of the aggregated group
of diverse
network connections.
[0019] In
one aspect, a computer implemented method for improving network
communication performance between at least two sites, is provided where the
two sites are
at a distance from one another that would usually require long haul network
communication,
comprising: (A) using a client site network component associated with a first
client site to
connect to a proximal network server component, the network server component
being
connected to an access point to a high performing network, forming thereby a
network
4
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
overlay that provides a bonded or aggregated connections for carrying data
packets; (B) the
network server component terminating the bonded or aggregated connection; and
(C) the
network server component transferring the data packets to the high performing
network, for
delivery to a second client site, while maintaining management of data traffic
so as to
provide a managed network path that incorporates at least the
bonded/aggregated
connection and at least one network path carried over the high performing
network, thereby
reducing long haul effects.
[0020] In
another aspect, the method may include receiving the data traffic at the
second client site.
[0021] In another aspect, the method may include maintaining management of
data
traffic so as to provide a managed network path that includes the bonded or
aggregated
connection and one or more network paths of the high performing network.
[0022] In
yet another aspect, the method may include receiving the data traffic at a
network server component associated with the second client site, the network
server
component initiating a bonded or aggregated connection to a client site
network component
associated with the second client site.
[0023] In
still another aspect, a plurality of network server components form a Point-
of-Presence, and the client site network component selects one or more of the
network
server components of the Point-of-Presence for establishing a network overlay
so as to
improve network performance.
[0024] In
this respect, before explaining at least one embodiment of the invention in
detail, it is to be understood that the invention is not limited in its
application to the details of
construction and to the arrangements of the components set forth in the
following description
or illustrated in the drawings. The invention is capable of other embodiments
and of being
practiced and carried out in various ways. Also, it is to be understood that
the phraseology
and terminology employed herein are for the purpose of description and should
not be
regarded as limiting.
BRIEF DESCRIPTION OF THE DRAWINGS
[0025]
Examples of embodiments of the invention will now be described in greater
detail with reference to the accompanying drawings, in which:
5
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0026] FIG. 1 a illustrates a prior art network configuration that
includes a
bonded/aggregated network connection. FIG. I a illustrates the problem of long
haul
aggregation/bonding.
[0027] FIG. lb also illustrates a prior art network configuration
that includes central
management of bonded/aggregated network connections.
[0028] FIG. 2a shows a network solution in accordance with the
present invention,
with bonding/aggregation implemented at both Site A and Site B, while
minimizing long haul
effects based on the technology of the present invention.
[0029] FIG. 2b shows another network solution in accordance with the
present
invention, in which bonded/aggregated network service exists at Site A but not
at Site B.
[0030] FIG. 2c shows a still other network solution in accordance
with the present
invention, in which bonding/aggregation is implemented as between Site A, Site
B, and Site
C.
[0031] FIG. 2d shows a further implementation of the network
architecture of the
present invention, in which a plurality of servers/concentrators are
implemented as part of a
Point-of-Presence.
[0032] FIG. 3 is a block diagram of a communication device
incorporating a
particular embodiment of the invention, demonstrating the device as an
aggregation means
on the client/CPE side of a network connection.
[0033] FIG. 4 is a block diagram of a communication device incorporating a
particular embodiment of the invention, demonstrating the device as an
aggregation means
on the server/concentrator side of a network connection.
[0034] FIG. 5 is a block diagram of a communication network
incorporating a
particular embodiment of the invention, demonstrating the device as an
aggregation means
on both the client/CPE side and server/concentrator side of a network
connection.
[0035] FIG. 6 is a flow diagram of a method of providing redundancy
and increased
throughput through a plurality of network connections in an aggregated network
connection.
[0036] FIG. 7a illustrates a prior art network architecture where
long haul effects
apply, and presents network performance based on download speed.
6
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0037] FIG. 7b
illustrates, in similar network conditions as in FIG. 7a but
implementing the present invention in order to reduce long haul
bonding/aggregation,
improved network performance based on faster download speed.
DETAILED DESCRIPTION
[0038] In one
aspect, a novel and innovative system, network architecture and
networking method is provided.
[0039] In another
aspect of the present invention, a network solution is provided for
improving network communication performance between at least two sites, where
the two
sites are at a distance from one another that would usually require long haul
network
communication. The network solutions includes at least one network
bonding/aggregation
system that includes (A) at least one first network component that is
implemented at a first
service site, the first network component being configured to bond or
aggregate one or more
diverse network connections so as to configure a bonded/aggregated connection
that has
increased throughput; and (B) a second network component, configured to
interoperate with
the first network component, the second network component including a
server/concentrator
(also referred to as network server component) that is implemented at an
access point to a
high performing network backbone.
[0040] In one
aspect, the first network component, may be implemented using what
is called in this disclosure a "CPE" or customer premises equipment (also
referred to as
client site network component). The CPE and a server/concentrator component
(more fully
described below) interoperate to configure the bonded/aggregated connections
in order to
provide improve network connections at a site associated with the CPE.
[0041] In one
aspect of the present invention, the server/concentrator is
implemented at an access point, with access to the network backbone so as to
avoid long-
haul bonded/aggregated network communications. As set out in the Example in
Operation
cited below, network architectures that involve long-haul bonded/aggregated
network
communication result in less than optimal performance, thereby minimizing the
advantages
of the bonding/aggregation technology. In other words, while the
bonding/aggregation
technology may improve service to Site A associated with for example a CPE (or
equivalent), based on bonding/aggregation between the CPE and an associated
server/concentrator (or equivalent), overall performance may be less than
desired and in fact
may be less than what would be available without bonding/aggregation because
of the long
7
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
haul effects of carrying the bonded/aggregated from Site A, to at least Site
B. These long
haul effects will present wherever Site A and at least Site B are at a
substantial distance
from one another.
[0042] The Example
in Operation below illustrates the decrease in performance that
results from the long haul effects.
[0043] FIG. la
illustrates the problem of long haul aggregation/bonding generally. In
the prior art bonded/aggregated network communication shown in FIG. 1 a,
packets are
carried over the Internet through an extension of the bonded/aggregated
connection across
the Internet (102), rather than the high performing Internet. The
bonded/aggregated
connection, across a distance that is subject to long haul effects, will not
perform as well as
the Internet, thereby providing less than ideal performance.
[0044] The Example
in Operation reflects another problem with prior art
bonding/aggregation solutions, namely that they generally require control or
management by
a central server. Depending on the location of the central server, this can
result in
multiplying of the long haul effects because traffic between a Site A and Site
B may need to
also be transferred to a Site C that is associated with the central server.
This aspect of the
invention is illustrated for example in FIG. lb. Central server (104) manages
network
communications, and in fact routes network communications between Site A and
Site C. To
the extent that the distance between central server (104) is substantial from
either of Site A
or Site C, long haul effects will present. If central server (104) is at a
substantial distance
from each of Site A and Site C, then there will be a multiplying of the long
haul effects, as
network traffic will pass from Site A to the central server (104) to Site C,
and from Site C to
the central server (104) to Site A.
[0045] As
illustrated in the Example of Operation, long haul effects have a negative
impact on speed (slowing traffic) and also on latency. Conversely, the present
invention
provides significant improvements in regards to both speed and latency.
[0046] The present invention provides a novel and innovative network
solution,
including a networking system and architecture and associated networking
method, that
addresses the aforesaid long haul effects that have a negative effect on
performance.
[0047] As shown in
FIG. 2a, in one aspect of the invention, the server/concentrator
side of a bonding/aggregation network solution for Site A (120a) is
implemented such that
(A) the location of the server/concentrator is implemented with access to the
network
8
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
backbone of Internet (112), and (B) the server/concentrator (110a) includes
functionality for
(i) receiving packets by means of the bonded/aggregated connection (116a),
(ii) interrupting
the bonded/aggregated connection (116a) using an interruptor (118), and (iii)
directing the
packets (114) to the Internet (112) for delivery to a Site B (120b). If Site B
also has
bonded/aggregated network service, then the packets are delivered to a Site B
side
server/concentrator (110b).
Server/concentrator (110b) established a further
bonded/aggregated connection (116b) and directs the packets (114) via the
bonded/aggregated connection (116b) to a CPE(B) (124b) at Site B.
[0048]
FIG. 2b illustrates a configuration where bonded/aggregated network service
exists at Site A but not at Site B.
[0049] A
skilled reader will understand that more than two sites are possible, where
the network system of the present invention improves network performance for
network
communications between for example Site A, Site B, and Site C where one or
more sites will
include bonded/aggregated service. In one implementation of the invention as
shown in
FIG. 2c, bonded/aggregated service is present for each of Site A, Site B and
Site C. FIG. 2c
illustrates one possible implementation of the invention, where the network
system is based
on a distributed network architecture where server/concentrators (110a) (110b)
(110c) and
corresponding CPEs (124a) (124b) (124c) are configured to provide improved
network
communications, including interruption of network communications at the
network backbone
so as to reduce long haul effects, dynamically and on a peer to peer basis
without the need
for a persistent central manager. In one implementation, each of the network
components of
the network system included functionality to operate on a peer to peer basis.
[0050] A
CPE (124) initiates network communications on a bonded/aggregated
basis, cooperating with a server/concentrator (110), with packets destined for
a remote
location. Each server/concentrator (110) receives dynamic updates including a
location and
identifier associated with other server/concentrators (110). Packets are
dynamically sent to
a server/concentrator (110) at the remote location, if available, and from the
server/concentrator (110) at the remote location to its CPE (124). The CPEs
(124) and their
server/concentrators (110) use bi-directional control of network
communications to establish
a network overlay to provide improved network performance. The network overlay
for
example provides desirable quality of service despite underlying network
conditions that may
otherwise resulted in a decrease in network performance.
9
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0051] In
accordance with the present invention, the network system of the present
invention establishes and manages two or more network overlays. Referring for
example to
FIG. 2a a first network overlay (126) is established between the CPE(A) (124a)
and
server/concentrator (110a); then, communications are transferred over the
Internet (112)
without a network overlay; then, a second network overlay (129) is established
between
server/concentrator (110b) and CPE(B) (124b). As a result, IP transport is
provided between
Site A and Site B where this will provide better performance than the
aggregated/bonded
network connections. Bonding/aggregation in effect is distributed across the
locations,
rather than attempting to span the distance between the locations with end to
end
bonding/aggregation.
[0052] The
present invention therefore provided distributed bonding/aggregation.
The present invention also provides a network system that automatically
provides distributed
bonding/aggregation in a way that bonding/aggregation is proximal, and beyond
proximal
connections IP transport is used, with proximal bonded/aggregated connections
and fast
Internet being used as part of end-to-end improved service.
[0053] In
another aspect of the invention, and as shown in FIG. 2d, one or more
server/concentrators can be implemented at a physical location, as part of a
Point-of
Presence (PoP) (130). In one aspect, in the context of the present invention,
a PoP (130)
can define a relatively high concentration of servers/concentrators within an
area. In another
aspect, a plurality of PoPs (130) may be available in a geographic location. A
plurality of
PoPs (130) may be established based on network topology or service
requirements in a
given area.
[0054] In
one aspect, each PoP (130) has one or more network backbone
connections (132), because in some locations different network backbones may
be
available. The PoP (130) may be implemented so that it dynamically
interoperates with
surrounding networks. The PoP (130) is a collection of network components,
established at
the periphery of the network backbone (112), associated with a plurality of
networks, and
cumulatively providing network communication service to a one or more clients
in a defined
geographic area. In one possible implementation, the server/concentrators
(110) located
within the PoP (130) functions as a network access server for connecting to
the Internet
(112). The network access server (110) acts as the access point to the
Internet (112) for a
plurality of CPE devices (124) that are connected to the PoP (130). The
servers/concentrators (110) may be configured to communicate with one another
to share
information regarding network conditions. Servers/concentrators (110) provide
connectivity
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
to CPEs (124) and may also run a networking protocol such as BGP to route
servers and
other network backbone connections (112).
[0055] In one aspect, servers/concentrators (110) are configured to
detect changes
in their network environment.
[0056] The CPE (124) may be configured to collect information from network
components in its vicinity including from one or more available PoPs (130) and
their
servers/concentrators (110). The CPE (124) for example connects to a closest
available
server/concentrator (124), implemented as part of a PoP (130), and thereby
having access
to a connection to the network backbone (112). Whether the connection to the
network
backbone (112) is direct or indirect, the network connections are established
so as to
minimize long haul effects.
[0057] In one implementation, each CPE (124) wanting to establish a
connection
dynamically advertises its IP address, and receives replies from associated
servers/concentrators (110) along with their current network performance
information. The
CPE (124) initiates a bonded/aggregated connection with a server/concentrator
(110) that is
both proximal (to minimize long haul effects between the CPE (124) to the
network
backbone (112)), and also that based on network conditions relevant to the
particular
server/concentrator, is performing well.
[0058] In one implementation, a network device is deployed that bonds
or aggregate
multiple, diverse links. The network device may be WAN aggregator or a link
aggregator.
[0059] Once the network overlay is established, various other network
optimization
and quality of services ("OOS") techniques may be applied.
[0060] One or more CPEs and one or more concentrators can create
various
different network configurations that improve network performance in relation
to network
communications between them. The CPEs and concentrators are designed to be
self-
configuring, and to interoperate with one another to manage traffic in a more
effective way.
[0061] "Proximal" means a distance such that based on relevant
network conditions,
long haul network communication and associated effects are avoided. The
distance
between the CPE and the server/concentrator is proximal, thereby enabling good
network
service.
11
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0062] A
skilled reader will also understand that in order to take advantage of the
network architecture of the present invention, the server/concentrator (110)
should be
located at an access point to the network backbone (112) or in some other way
to minimize
the long haul effect, for example, by the server/concentrator being located
proximal to an
access point so as to further avoid long haul network communication.
[0063] In
another aspect, the bonded/aggregated connection at Site A and the
bonded/aggregated connection at Site B may be different, in the sense that
each may
include different types of network connections and that may be associated with
different
carriers. In
one aspect of the invention, the network overlay provided operates
notwithstanding such diversity.
[0064] The
more sites that have the CPE/concentrators associated with them the
better network performance between them. Representative performance details
are
included below.
[0065] The
network backbone (112) could be any high performance network
including for example a private WAN, the Internet, or an MPLS network.
Network Overlay
[0066] In
one aspect of the invention, one or more network overlays are established
in accordance with the present invention, thereby in one aspect providing a
multi-POP
network that exploits multiple points of presence so as to provide a
persistent,
configurable/reconfigurable network configuration that provides substantial
network
performance improvements over prior art methods.
[0067] In
one aspect of the invention, the CPEs/concentrators may monitor network
performance, including in the areas proximate to their position, and may
reconfigure the
network overlay dynamically, across multiple locations (including multiple
PoPs) based on
changes in network performance while providing continuity of service.
[0068] In
one aspect, the network components of the present invention are
intelligent, and iteratively collect network performance information.
Significantly, in one
aspect each CPE is able to direct associated concentrator(s) and any CPE to in
aggregate
re-configure the network overlay.
12
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0069] Significantly, in the network overlay created by the present
invention
management of the network may be centralized or decentralized, depending on
the
configuration that provides the best overall performance. This is in contrast
to prior art
solutions that generally require central management for example of termination
of
connection which results in traffic being carrier over bonded/aggregated
connection that
involve long haul transmission that fail to take advantage of network paths
that may provide
inherently better performance than the bonded/aggregated connection paths.
[0070] In one aspect, decentralized managed is made possible by peer-
to-peer
functionality implemented to the network components of the present invention.
[0071] In another aspect of the invention, a plurality of
servers/concentrators may be
established in multiple locations covering a plurality of different access
points. Each
server/concentrator may be used for multiple clients associated with different
CPEs to
improve network performance for such multiple clients by providing termination
of their
bonded/aggregated connection and transfer of communications to the network
backbone.
The network solution of the present invention therefore may include multiple
Points-of-
Presence, distributed geographically including for example in areas requiring
network
service, and through the network architecture of the present invention
bridging
geographically disparate areas with improved network communication
therebetween.
Additional Implementation Detail
[0072] As previously stated, the present invention may be implemented in
connection with any technology for bonding or aggregating links, and thereby
reduce long
haul effects.
[0073] What follows is additional detail regarding link aggregation,
which is one form
of bonding/aggregation that may be used as part of the overall network
solution and network
architecture disclosed in this invention.
[0074] In one aspect of the invention, the system, method and network
architecture
may be implemented such that the aggregated/bonded network connections
described are
implemented using the link aggregation technology described in Patent No.
8,155,158.
What follows is further discussion of possible embodiments of the CPE and the
server/concentrator (or concentrator) components previously described,
emphasizing their
creation and management of the bonded/aggregated connections between them,
which in
13
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
the network configuration of the present invention form a part of the overall
network overlay
that incorporates the one or more portions that are carried over the network
backbone.
[0075] Diverse network connections may be aggregated into a virtual
(logical)
connections that provide higher throughput as well as independence of the
network
characteristics of the constituent (physical) network. Aggregation may be
performed to a
given CPE.
[0076] For instance, in one example of the implementation of the
present invention a
Metro Ethernet 10Mbps (E10) link and a T1 (DS1) link are aggregated in
accordance with
the invention as described below, in order to provide higher fault tolerance
and improved
access speeds. The aggregation of diverse carriers in accordance may extend to
any
broadband network connection including Digital Subscriber Line (DSL)
communications
links, Data over Cable Service Interface Specification (DOCSIS), Integrated
Services Digital
Network, Multi protocol Label Switching, Asynchronous Transfer Mode (ATM), and
Ethernet,
etc.. The network connections may also include a WAN.
[0077] According to one aspect of the invention, an apparatus is provided
for
managing transfer of communication traffic over diverse network connections
aggregated
into a single autonomous connection, independent of the various underlying
network
connections. The apparatus may include a network aggregation device and an
aggregation
engine. The network aggregation device may be adapted to configure a plurality
of network
connections, which transfers communication traffic between a further network
connection
and the plurality of network connections, as an aggregated group for providing
a transfer rate
on the further communication link, and to allocate to the aggregate group a
rate of transfer
equal to the total available transfer rate of the underlying networks. The
aggregation engine
may be adapted to manage the distribution of communication traffic received
both to and
from a plurality of network connections, establishing newly formed aggregated
network
connections. The aggregation engine may be implemented in software for
execution by a
processor, or in hardware, in a manner that is known to those skilled in the
art.
[0078] In accordance with this aspect of the present invention, a
plurality of diverse
network connections may be aggregated to create an aggregated network
connection. The
diversity of the network connections may be a result of diversity in provider
networks due to
the usage of different equipment vendors, network architectures/topologies,
internal routing
protocols, transmission media and even routing policies. These diversities may
lead to
different network connections with different latencies and/or jitter on the
network connection.
14
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
Also, variation within transmission paths in a single provider network may
lead to latency
and/or jitter variations within a network connection.
[0079] Latency and jitter typically affect all data communication
across the network
connection. Latency, as is known to those skilled in the art, is the round-
trip time for a
transmission occurring end-to-end on a network connection. Jitter, as is known
to those
skilled in the art, is the variance in latency on a network connection for the
same data flow.
High latency and jitter typically have a direct and significant impact on
application
performance and bandwidth. Applications such as VOIP, and video delivery are
typically
highly sensitive to jitter and latency increases and can degrade as they
increase.
[0080] Transparent aggregation of a plurality of network connections in an
aggregated network connection requires the management of data transmitted over
the
aggregated connection by the aggregation engine and received from the
aggregation traffic
termination engine. In one aspect of the present invention, transparent
aggregation does not
require any configuration by a network provider. The aggregation engine and
the
aggregation traffic termination engine may manage data transmission such that
the variable
path speeds and latencies on the plurality of network connections do not
affect the
application data transmitted over the aggregated network connection. The
network
aggregation engine and the aggregation traffic termination engine may handle
sequencing
and segmentation of the data transmitted through the aggregated connection to
transparently deliver application data through the aggregated connection with
minimal
possible delay while ensuring the ordered delivery of application data.
[0081] In one aspect of the invention, the network aggregation engine
provides a
newly aggregated network connection with a capacity equal to the sum of the
configured
maximum throughput of the network connections.
[0082] The aggregation engine and an aggregation traffic termination engine
(further
explained below) handle the segmentation of packets as required in
confirmation with
architectural specifications such as Maximum Segment Size (MSS) and Maximum
Transmission Unit of the underlying network connections. The network
aggregation device
is operable to handle assignment of sequence identifiers to packets
transmitted through the
aggregated network connection for the purpose of maintaining the ordering of
transmitted
data units over the aggregated network connection.
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0083] In a further aspect of the invention, the network connection
device includes or
is linked to a connection termination device, and a plurality of fixed or hot
swappable
transceivers for transmitting communication traffic on respective sets of
network
connections, for the purpose of configuring a plurality of network connections
as an
aggregated connection or the management of multiple aggregated network
connections and
providing access to the aggregated network connection for any network
communications
traversing the device.
[0084] In the present specification, routing protocols or route
selection mechanisms
described are intended only to provide an example but not to limit the scope
of the invention
in any manner.
[0085] FIG. 3 is a block diagram of a communication device
incorporating a
particular embodiment of the invention, demonstrating the device acting as a
client.
[0086] As shown in FIG. 3, the network element/network aggregation
device (also
referred to in this disclosure simply as the "device" or the "network
aggregation device") 23
includes (in this particular embodiment shown for illustration) a network
connection
termination module 25 that includes representative transceiver interfaces 14,
15 and 16.
Each transceiver interface 14, 15 and 16 represents an interface to a physical
communication medium through which communications may be established to
network
connections.
[0087] A possible implementation of the network aggregation device may use
a
single or multiple chassis with slots for multiple network connection
termination modules and
multiple network aggregation engine modules. The multiple network connection
termination
modules may be grouped by protocol specific or medium specific
transceiver/interfaces.
[0088] The network aggregation engine 11 may handle the configuration
of the
network aggregation device and all related interactions with external inputs.
A device
configuration store 24 may provide persistent data storage for device
configuration
information such as a network aggregation policy.
[0089] The network aggregation engine 11 may handle queries from
external
sources, such as configuration parameters a network management protocol such
as Simple
Network Management Protocol, for example. The interface 10 may be a protocol
agent and
may provide for communication with a Network Management System (NMS) or
operator
system for configuration of the aggregation engine by the definition of an
aggregation policy.
16
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
Control and management information may be transferred between the network
aggregation
device 23 and the NMS or operator system through the interface 10 via any
available or
specifically designated network connection 19, 20, 21 and 17 through any
transceiver
interface 14, 15 and 16.
[0090] In accordance with an aspect of the present invention, multiple
network
connections may be combined to form an aggregated network connection 22, as
disclosed in
further detail herein. Each individual network connection may be configured
with a
maximum communication traffic rate, which could be expressed as a bit rate in
bits per
second.
[0091] The network aggregation engine 11 may be implemented in software for
execution by a processor in the network aggregation device 23, or in hardware
such as by
means of a Field Programmable Gate Array (FPGA) or other integrated circuit,
or some
combination thereof. The network aggregation engine 11 may be implemented in a
distributed manner by distributing aggregation engine intelligence to the
network connection
termination module 25, in a manner that is known.
[0092] The network aggregation engine 11 may receive traffic from
client network
connection device 18 through a network connection 17 provided through a
transceiver
interface 16. The client network connection device 18 may be any device
including, without
limitation, a router, switch, or media converter that is capable of providing
termination for a
single or multiple client nodes, where nodes are any devices capable of
connecting to a
network irrespective of protocol or interface specificity. In various
embodiments, traffic may
be received over multiple network connections through a single or multiple
transceiver
interfaces. The network aggregation engine 11 may accept all traffic from the
client network
connection, may provide encapsulation and segmentation services for the
traffic for
transmission through the aggregated network connection 22, and may transmit it
over any of
the network connections 19, 20 and 21 through any of the transceiver
interfaces 14, 15 and
16. The network aggregation engine 11 may handle segmentation in a manner that
avoids
the fragmentation of aggregated communication traffic received through the
client network
connection device 18, when transmission occurs over the aggregated network
connection 22
through any of the network connections 19, 20 and 21, by ensuring that the
length of a
packet/frame transmitted over any of the network connections 19, 20 and 21 is
less than or
equal to the configured or detected frame length for the respective
connections in the
aggregated network connection 22.
17
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0093] The network aggregation engine 11 may poll the state of
network connections
19, 20 and 21, for example as per configured intervals stored in the device
configuration
store 24, to ensure that all network connections configured in an aggregated
group are
within configured acceptable tolerances. If a network connection 19, 20, and
21 exceeds
acceptable tolerance values for any of the polled parameters, the network
aggregation
engine 11 may remove the network connection 19, 20, and 21 from within the
aggregated
network connection 22 without removing it from the polled network connections
list. By
leaving the removed network connection 19, 20, and 21 in the polled network
connection list,
the network aggregation engine 11 may aggregate the network connection into
the
aggregated network connection 22 once it has come back within acceptable
tolerance
values. This may ensure that a network connection may change states between
residing in
an aggregated network connection 22 or not, without the intervention of an
external system
or input. The network aggregation engine 11 may handle notifications to all
end points
configured within the device configuration store 24 with internal events such
as changes in
network connection state, threshold violations on configured thresholds for
any number of
configurable variables for any object within or connected to the network
aggregation device
23. The network aggregation engine 12 may also handle events such as changes
in the
state of a network connection 19, 20, and 21 included in the aggregated
connection,
changes in latency of a network connection included in the aggregated network
connection
22, scheduling changes, event logging, and other events.
[0094] FIG. 4 is a block diagram of a communication device
incorporating a
particular embodiment of the invention, demonstrating the device acting as a
server/concentrator.
[0095] The network aggregation engine 11 may provide access to a
network
aggregation policy database 36 which stores configuration information related
to the various
aggregated network connections that terminate on the aggregated network
connection
device 28. The network aggregation termination device 28 may be implemented in
such a
manner that each aggregated network connection defined in the network
aggregation policy
database 36 is handled by its own virtual instance, the use of which enables
termination of
each aggregated network connection from multiple customer premises equipment
(CPE).
[0096] FIG. 5 is a block diagram of a communication network
incorporating a
particular embodiment of the invention, demonstrating the function of the
device acting as a
client/CPE and server/concentrator.
18
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[0097] In accordance with a particular embodiment of the invention,
aggregated
network connections 70, 71 and 72 may be built by network aggregation devices
63, 64 and
65, which terminate to a single aggregated network connection termination
device 61
through network connections 66 and 68 as their endpoint. The aggregated
network
connection termination device 61 may access external communications networks
through
network connections 66 and 68 to access external/remote network resource 69.
Access to
external communications networks may be provided by the aggregated network
connection
termination device 61 by using either network connection 66 or 68 through the
use of a
routing protocol, such as Border Gateway Protocol (BGP), Open Shortest Path
(OSPF), or
through the use of simpler mechanisms such as load sharing over multiple
static routes
within the communication network 74 that acts as the valid next-hop for the
aggregated
network connection termination device 61.
[0098] Aggregated network connections 70, 71 and 72 may provide
access to client
network nodes 67 connected to the network aggregation devices 63, 64 and 65
through the
aggregated network connections 70, 71 and 72 to communications networks 74
accessible
by the aggregated network connection termination device 61.
[0099] A client network node 67 may request data provided by an
external/remote
network resource 69 accessible through a communication network 74. This
request for the
external/remote network resource may be routed over the network connection 73
providing
access from the client network node 67 over the aggregated network connection
70 to its
end point which is the aggregated network connection termination device 61.
This may be
done through the communication network 74 through the network connection 66
into the
aggregated network connection termination device 61. Any data sent by the
external/remote
network resource 69 may be routed back through the aggregated network
connection
termination device.
[00100] A particular embodiment of this invention may use the Internet
as the
communication network 74 referenced in FIG 5. However, the communication
network 74
may alternatively be built by multiple sub-networks created through the use of
multiple
network aggregation devices 63, 64 and 65 with aggregated network connection
termination
device 61 end points through multiple network connections 66 and 68.
[00101] A further aspect of the invention relates to the provisioning
of high availability
over the aggregated network connection by the network aggregation engine 11.
FIG 6
illustrates a method of providing redundancy and increased throughput through
a plurality of
19
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
network connections in an aggregated network connection. The method 90 may
begin with
a step of configuring a plurality of network connections 91 through the
creation of a network
aggregation policy to form 92 the aggregated network connection. The
aggregated network
connection may be initialized as per the network aggregation policy. Control
connections
may be created 93 for the plurality of network connections configured as part
of the
aggregated connection to allow the aggregation engine 11 to manage the
membership of a
network connection within the aggregated connection. The network aggregation
engine 11
may accept packets for transmission 94 over the aggregated network connection
22. The
network aggregation engine 11 may choose a network connection 95 among the
group of
network connections configured 91 in the aggregate in the stored aggregation
policy for
transmission of the current packet being transmitted. The choice of network
connection for
transmission of the current packet may be specified within the aggregation
policy and may
take into account data provided by the control connection built at 94.
[00102] According to one embodiment of the invention, a non-responsive
network
connection may be easily detected when using latency and packet loss as a
measure. The
mechanism for detecting 96 and adapting to 97 the network connection change
within an
aggregated network connection may be implemented within the data transmission
routine in
the aggregation engine 11 or as a separate process in parallel to the
transmission routine in
the aggregation engine 11 to allow for further flexibility in provisioning
redundancy within the
aggregated network connection.
[00103] Since this may occur on a per packet basis as opposed to on a
per stream
basis, a single non-responsive network connection may not affect the
aggregated network
connection and may allow data transmission to continue regardless of the
individual states of
network connections so long as a single network connection within the
aggregated network
connection is available for data transmission.
Example In Operation
[00104] In one possible implementation of the present invention, 3
locations are
provided namely Site A, Site B, and Site C, and Site D. FIGS. 7a and 7b
illustrate network
performance as discussed herein. FIG. 7a illustrates performance with long
haul effects.
FIG. 7b illustrates performance with reduction of long haul effects, based on
the present
invention in network conditions otherwise similar to those on which FIG. 7a is
based.
CA 02893548 2015-06-08
WO 2015/013805
PCT/CA2014/000595
[00105] FIG. 7b shows an improvement in performance over FIG. 7a,
based on
reduction of long haul effects in relatively long distance network
communications are
implemented using the network architecture.
[00106] The present invention therefore provides improved network
performance
relative to speed. A skilled reader will appreciate that the improvement in
performance
shown in significant. Other aspects of network performance are also improved,
based on
the present invention, for example latency.
Advantages and Use Case
[00107] The present invention significantly improves network
performance between
disparate locations by leveraging network bonding/aggregation technology, but
by
implementing a system, method and network configuration that provides
intervening network
components disposed adjacent to access points so as to manage traffic between
two or
more sites such that bonded/aggregated connections are terminated and traffic
is directed to
a network backbone, and optionally passed to one or more further
bonded/aggregated
connections associated with a remote additional site.
[00108] The network solutions of the present invention is flexible,
responsive, scalable
and easy to implement. New sites, optionally having their own CPE/concentrator
can be
easily added, and the network solution supports various types of multi-point
network
communications, and various network performance improvement strategies
including various
QoS techniques.
[00109] The network solution is easily updated with new programming or
logic that is
automatically distributed on a peer to peer basis based on the interoperation
of network
components that is inherent to their design, as previously described.
[00110] Network performance is significantly improved over prior art
solutions as
illustrated in the Example In Operation provided above.
21
