Note: Descriptions are shown in the official language in which they were submitted.
CA 02919182 2016-01-28
1 System and method for initializing and maintaining a series of virtual
local area networks
2 contained in a clustered computer system
3 Background of the Invention
4 Traditionally, clients of a data center are required to buy or rent
physical servers,
switches, and storage arrays to put into data centers to house items such as
web applications,
6 databases, voip servers, data servers, etc. This can be extremely costly
for small businesses
7 which may only need to run a small web application such as a storefront
or a payroll application.
8 Alternatively, the same client can rent web space on a database and web
server, but is often
9 limited to what can be done with it, number of users or databases that
can be contained within, or
how much traffic it can receive.
11 What is needed is a system where a client may purchase CPU cycles,
storage, and
12 network resources "a la carte," being able to obtain only what is
required by their business, no
13 more, no less. It would be beneficial to the client to be able to
purchase these resources on the
14 fly, as needed, without having to leave the comfort of the office and
having them work
automatically. There would be nothing to hookup, nor anything to configure so
that hardware
16 works with one another. In addition to fully-functional servers, clients
may lease shared
17 resources and have them integrate with existing infrastructures
seamlessly.
18 In the field metropolitan area networks (MANs), a system is used to
isolate users into
19 virtual local area networks, or VLANs. Recently, the idea of
encapsulating a VLAN inside
another VLAN has been introduced simply to be able to house more users. While
before
1
CA 02919182 2016-01-28
21 network engineers were limited to 256 VLANs on most equipment, they may
now be able to use
22 256x256 separate VLANs.
23 What is described herein is using the concepts of VLANs and
virtualization on a large
24 pooled system to be able to dynamically allocate network resources to
users, as well as bridge
25 and share network resources.
26 Herein, the term "computing device" refers to any electronic device
with a processor and
27 means for data storage. Used herein, the term "network connection"
refers to any means to allow
28 a plurality of computing devices to communicate. Further, the term
"trunked" used herein refers
29 to programmatically relating multiple network connections to each other
to create redundancy
30 and greater bandwidth in a single logical connection. The term "network
packets" refers to a
31 formatted message transmitted over a network. The term "hardware
resource" refers to a
32 networkable computing device. The term "virtual resource" refers to an
allocation on a
33 networkable computing device which refers to a virtual representation of
a computing device or a
34 software application, such as a database. Used herein, the term
"management local area
35 network", sometimes referred to as a "MLAN", refers to a LAN containing
hardware or virtual
36 resources used exclusively for the initialization, configuration, and
maintenance of other LANs.
37 Used herein, the term "data center" refers to a central storage complex
containing a multitude of
38 servers and network routing hardware. A "traditional data center" is a
data center absent of
39 virtualization. The term "virtual firewall" refers to a virtual
implementation of a firewall with a
40 virtual ethernet port. Used herein, the term, "maintaining" refers to
keeping a network resource
41 functioning.
42 Brief Summary
2
CA 02919182 2016-01-28
43 Disclosed herein is a system, method and computer program product
for initializing and
44 maintaining a series of virtual local area networks (VLANs) contained in
a clustered computer
45 system to replace a traditional data center. A physical network contains
a management local area
46 network (MLAN) and numerous client VLANs nested within a top-level VLAN.
The MLAN
47 contains at least a physical or virtual firewall. Each client VLAN
contains a virtual firewall as
48 well as a number of physical hardware machines and virtual machines
maintained by the
49 clustered system. The client VLAN appears as a normal subnet to the
user. A network
50 administrator is able to create, change, move, and delete virtual
resources contained in a client
51 VLAN dynamically and remotely.
52 The system itself connects a plurality of computer systems as a
clustered system through
53 a switched fabric communications link, such as a switch fabric
communications link sold under
54 the name INFINIBANDO. All storage devices in the system are clustered to
create a distributed
55 file system, which makes the drives appear to be a giant pool of space
in which any particular
56 virtual machine may be contained anywhere within.
57 Also described herein is a method for sharing a network resource,
physical or virtual,
58 between a plurality of client VLANs. The shared resource may be
contained in one of the client
59 VLANs, or in a separate top-level VLAN.
60 Brief Description of the Drawings
61 Figure 1 is a block diagram of the hardware used in the system.
62 Figure 2 is a block diagram of the VLAN structure.
3
CA 02919182 2016-01-28
63 Figure 3 demonstrates reaching shared resources through a single
port of a shared
64 resource firewall.
65 Figure 4 demonstrates reaching shared resources through multiple
ports of a shared
66 resource firewall.
67 Figure 5 demonstrates reaching shared resources through private, non-
editable client
68 firewalls.
69 Detailed Description
70 Referring to Figure 1, system 100 comprises a plurality of redundant
array of inexpensive
71 storage nodes (RAIDS) 101a-101f, a plurality of non-redundant storage
nodes 102a-102c, a
72 plurality of processing nodes 103a-103-g, a plurality of network
connections 104a-104g, and a
73 plurality of network switches 105a-105b. Storage nodes 101a -101f are
redundant high level
74 storage. Each node is mirrored for a redundant distributed fault
tolerant file system. In the
75 embodiment presented in Figure 1, storage nodes 101a and 101b make a
pair, 101c and 101d
76 make a pair, and 101e and 101f make a pair. Non-redundant storage nodes
102a through 102c
77 contain 48 different disk drives with no cross-server redundancy for
customers who don't need
78 the added security of redundancy. Each processing node 103a through 103g
contains 2, 4, 8, or
79 more dual processors. In the preferred embodiment, network connections
104a through 104g
80 may either be 6 trunked 1 Gbps ethernet connections, or 2 trunked 4x 2.5
Gbps INFINIBANDS
81 connections. In additional embodiments, network connections 104 may use
more or less
82 connections and use other protocols. Network switch 105a may be a switch
such as an ethernet
83 switch or an INFINIBAND switch depending on what protocol network
connections 104 use;
4
CA 02919182 2016-01-28
84 network switch 105b is may be a switch such as an ethernet switch used
to communicate outside
85 the network. INFINIBANDO switches use IP-over-INFINIBANDS. The switches
are able to
86 add VLANs on a granular level. The switches may natively support Q-in-Q
double tagged
87 VLANs, which allow for nested client VLANs out of the box. In other
embodiments, all nested
88 client VLAN tags are handled by processing nodes 103. One of ordinary
skill in the pertinent art
89 will recognize that the number of components shown in Figure 1 is simply
for illustration and
90 may be more or less in actual implementations.
91 Referring to Figure 2, VLAN 2100 is a top-level VLAN used as a
management LAN, or
92 MLAN, containing the firewall 2101 initialized by the storage server.
MLAN 2100 is
93 responsible for the initialization, configuration, and maintenance of
all client VLANs in system
94 100, as well as shared resource networks and physical networks on the
system. Firewall 2101
95 has 3 ports, one connected to MLAN 2100, one connected to the untagged
ethernet port "VLAN
96 0", and one connected to VLAN 2200, the shared resources VLAN. In some
embodiments,
97 firewall 2101 is mirrored several times and referred to as a firewall
cluster. The firewall cluster
98 is spread across several multiple processing nodes 103 for faster
routing. Top-layer VLAN 2300
99 contains multiple client VLANs 2310, all with their own firewalls, 2311.
One of ordinary skill
100 in the pertinent art will recognize that the numbers of elements
depicted in Figure 2 are only
101 exemplary. For instance, each top-layer VLAN may contain up to 255
client VLANs.
102 On bootup, each storage node 101 contacts each of the other storage
nodes to discover
103 whether or not any of them has started the boot process of creating a
management firewall 2101
104 of Figure 2, a boot server and a management console 2102. If none of
the other nodes has started
105 the process yet, the pinging node begins the process. Initially the
management firewall 2101 or a
CA 02919182 2016-01-28
106 management firewall cluster is started. If the MLAN 2100 is routed by a
virtual firewall, the
107 storage nodes 101 will need to initially run the process that starts
the management firewall
108 cluster. This does not preclude a hardware firewall for the MLAN 2100,
but in the preferred
109 embodiment only servers and switches are needed and the same underlying
structures that
110 provide redundancy and availability to servers can give high
availability to firewalls and routers
111 in a virtual environment. In this preferred embodiment, a group of
storage servers can start
112 redundant copies of the firewall/router 2101. Each instance of the
firewall will have the same
113 MAC address and VLAN assignment for any attached ethernet ports. Using
normal routing
114 schemes, this may cause a bank of switches to route packets to
differing firewalls depending on
115 the source of a connection, but this will have no ill effects if the
network devices in question
116 continue to have the same settings and routing information.
117 The management console 2102 has many of the same properties as the
firewall in system
118 100. While in the preferred embodiment it is run on the storage nodes
101 as a virtual machine, it
119 can likewise be a physical machine. It is started up at the same time
as the firewall/router cluster
120 and can also be deployed in a cluster format.
121 In the preferred embodiment, the boot server contains a tftp server,
an NFS server, a PXE
122 boot service and a preconfigured kernel image. This image will have a
runtime environment for
123 the local interconnect (INFINIBANDS, trunked ethernet or other similar
high speed
124 interconnect) and the ability to mount the clustered file system that
exists across the storage
125 nodes 101. The processing nodes 103 then contact the management console
2102 for initial
126 settings such as an IP address and host name, for example. The
clustered file system is mounted
127 and the processing nodes 103 boot in a normal fashion. Once startup is
complete the processing
6
CA 02919182 2016-01-28
128 nodes 103 contact the management console 2102 and indicate that they
are ready to take a load
129 of virtual machines to host for clients.
130 Once the processing nodes 103 have begun to activate, the management
console 2102
131 gets a list of virtual machines that need to be started up by the
processing nodes 103 from its
132 datasource. The management console 2102 then begins to start virtual
machines on processing
133 nodes 103 in a weighted round robin fashion. Processing nodes 103 are
assigned to groups based
134 on their capabilities and architecture; for example, 64-bit processing
nodes would be associated
135 as a group. There is a server mask for each virtual machine that
assigns it to a particular
136 processing node group. This is both to comply with per-processor
licensing issues and to ensure
137 that virtual servers with particular hardware, redundancy or
connectivity requirements can be
138 met by the appropriate physical machine. During the startup process
management console 2102
139 may even initiate a delay if more virtual machines exist than the bank
of processing nodes 103
140 can run. After a predetermined interval, if this imbalance is not
corrected, a warning system will
141 be started to alert human operators of the lack of server resources. As
the virtual machines are
142 assigned to physical servers, each physical server reports CPU and
memory usage to the
143 management console 2102 and these figures are used as selection
mechanisms to ensure that
144 processor and memory loads are evenly distributed across all physical
nodes. Even after the
145 physical layer is booted, the processing nodes 103 continue to report
CPU and memory usage to
146 the management console 2102 at regular intervals.
147 The virtual servers undergo a normal startup process themselves.
Once a command to
148 start a virtual server is issued, (either by a system-wide startup,
client start command or other
149 system need) the management console 2102 takes the start request and
queries the datasource for
7
CA 02919182 2016-01-28
150 available processing nodes. Once one is selected by the mechanism
mentioned above; that virtual
151 server creates an ethernet device that is attached to either the top
layer VLAN or the Q-in-Q
152 nested VLAN 2310 that the virtual server connects to. Unlike normal
ethernet devices, this
153 VLAN device is not given an IP address or any routing information. The
physical server itself
154 does not respond and actually does not see any packets it receives from
this interface. The
155 physical device is instead mapped directly to a virtual one, giving the
virtual machine access to a
156 completely separate network than the physical machine exists on. After
the appropriate network
157 devices are added to a processing node, the management console 2102
then queries its
158 datasource and connects to the client's hidden firewall. This firewall,
as described later in
159 reference to Figure 5, is for routing console and virtual screen
information from the MLAN 2100
160 back to the client's network and represents a NAT mapping from the MLAN
2100 to the clients
161 subnet. In the current embodiment, a virtual serial port is used to add
rules to this virtual routing
162 device to keep the methodology consistent with non-addressable
firewalls that clients may want
163 to add rules and configurations to. This is not necessary, however
since this translating firewall
164 has an IP address that exists in the MLAN 2100 directly. On startup of
the virtual machine a rule
165 is added to provide the client with console access to a web interface
to the management console
166 2102. This gives the clients the ability to access virtual servers as
if they were at the keyboard of
167 a physical machine. From the clients secure management console web
interface they are able to
168 control the screen, keyboard and mouse inputs of their virtual servers.
In the current embodiment
169 VNC is used as a remote console but other protocols are available.
During this process the virtual
170 server itself is issued a start command and is then accessible to the
client.
171 When a new client is added, they are given a number of external IPs
and a unique subnet
172 of their network. Every possible IP of the subnet is statically
assigned to a MAC address that
8
CA 02919182 2016-01-28
173 may or may not be used. A client VLAN 2310 is created and the first
address of the subnet is
174 assigned to the client VLAN's firewall 2311. The firewall contains a
DHCP table that is created
175 when the firewall is initialized to hold the mappings of the
preregistered MAC addresses to IPs
176 so that the IP is known as machines are added. The client is given a
gateway 2001 configured to
177 deliver the client's network packets directly to the virtual firewall
2311 through an IPSEC
178 tunnel. In addition, network packets of all external traffic are routed
directly to the client's
179 virtual firewall 2311. Virtual firewall 2311 has one port connected to
external port 2317 which
180 receives external traffic through network switch 205b, which is
equivalent to network switch
181 105b. Traffic from the client through the IPSEC tunnel to the client's
personal VLAN 5310a is
182 shown as a dotted line in Figure 2., Virtual firewall 2311 further has
one port connected to their
183 personal client VLAN 2318, and in some embodiments, an optional port
for connecting to shared
184 resources 2319, such as those contained in VLAN 2200, or in another
client VLAN.
185 The last address of the subnet is assigned as the management console
2102. The
186 management console 2102 is connected to main firewall 2101 in MLAN 2100
and, in some
187 embodiments, is reached through the optional port of the client
firewall. From there, the client
188 may view network settings and add machines 2312-2315. The client is
able to create and be
189 charged for virtual machines on their client VLAN through the
management console 2102
190 remotely. The client is capable of adding 253 virtual machines. The
virtual machines may be
191 just about any kind of machine, such as a Windows or Linux web server,
a voice-over-IP server,
192 etc. After a machine is chosen, a MAC address is assigned from the
client firewall 2311 and a
193 template image corresponding to the machine from a storage node 101 is
taken and initialized in
194 storage depending on the kind of storage system the client has chosen
(redundant storage nodes
195 101, or non-redundant storage nodes 102). From there, the management
console 2102 adds the
9
CA 02919182 2016-01-28
196 machine to the list of machines that need to be ran. The next
processing node 103 that inquires
197 on tasks that need to be run is assigned the machine. If it is the
first machine run on that
198 particular client VLAN, it starts up a virtual listening port for that
VLAN. Once the virtual
199 machine is connected to the VLAN, the firewall looks at its MAC address
and assigns it its
200 preconfigured IP address from the DHCP table.
201 The client is able to use VNC or remote desktop to login to the
newly created virtual
202 machine and see the user API/GUI as if they were sitting in front of a
physical machine with the
203 same image. From there the user is allowed to do anything that can be
normally done on a
204 physical machine, completely abstracted from the virtualization of the
machine or the fact that it
205 is contained in a VLAN ran on system 100 in a distant data center. To
the user, virtual machines
206 2312-2315 appear to be like any other machine contained on a
traditional network subnet.
207 The client is also able to add a physical machine to their subnet.
In the preferred
208 embodiment, the switches natively support Q-in-Q double tagging, which
allows for routing
209 double tagged network packets to physical machines out of the box. In
other embodiments, the
210 nested client VLAN is turned into another top-layer VLAN to allow for
physical machines on the
211 VLAN.
212 Clients are able to share resources either between their client
VLANs, or in a shared
213 resources network such as resources 2202-2205 in VLAN 2200. In some
embodiments, clients
214 are able to connect to these resources by setting up the optional port
on their client firewall 2311
215 to connect to the IP of the selected shared resource. An empty VLAN is
created between the
216 ports of both firewalls on both sides as a "virtual wire". Rules are
set up on the firewalls on both
217 ends to handle the new traffic. On the client VLAN side, firewall
2311dynamically adds a
CA 02919182 2016-01-28
218 virtual port to itself and maps the port in a network address table
within client firewall 2311. If
219 a client wishes to share resources from more than one location,
multiple optional ports may be
220 added. In this situation, the firewall must be temporarily shutdown to
make the configuration.
221 Figure 3, Figure 4, and Figure 5 show alternate embodiments for
routing data through
222 system 100. Referring to Figure 3, shared resource VLAN 3200 and client
VLANs 3310 are
223 identical to shared resource network 2200 and client VLANs 2310,
respectively. Shared
224 resource firewall 3201 has one port for incoming resource requests. The
connection is
225 essentially a "virtual switch", labeled as 3206, that filters traffic
based on incoming IPs. Using
226 the "virtual switch", client VLANs 3310 are able to reach their
designated shared resources,
227 residing within 3202-3205. Referring now to Figure 4, shared resource
VLAN 4200 and client
228 VLANs 4310 are identical to shared resource network 2200 and client
VLANs 2310,
229 respectively. Figure 4 shows an alternate embodiment that has a
separate port on shared resource
230 firewall 4201 for each incoming connection from client VLANs 4310
attempting to use a shared
231 resource 4202-4205. A firewall rule is designed for each individual
port.
232 Figure 5 illustrates the preferred embodiment of handling shared
resources. The system
233 of Figure 5 is identical to that of Figure 2 with the addition of each
client VLAN 5310 containing
234 a second firewall, private firewall 5316. Private firewall 5316 is not
editable by the client and
235 contains predefined rules to reach shared resources within shared
resource VLAN 5200 or within
236 another client VLAN, VNC connections to physical machines on the
client's subnet, and the
237 management console 5102. Using this non-editable private firewall
ensures that a user does not
238 inadvertently change routing rules that hinder routing throughout
system 100.
11
