Note: Descriptions are shown in the official language in which they were submitted.
:~L2~3~0
REAL TIME DATA TRAXSFO~TION AND TR~`:S~1ISSIO~ OVERLAPPING DEVICE
BACKGP~O"~D OF THE IDEATION
Field of the Invention
This invention relates to data transformation and transmission, and in
particular to a device for compressing data from a fast access storage
devils and for transmitting compressed data to a slower medium.
Description of the Prior Art
Compression or transformation of data usually results in an irregular
flow of compressed data which is not conducive to efficient recording on
magnetic tape or transmitting over communication channels with reasonable
efficiency. Computers with large, volatile direct access main storage
devices usually require that the data contained therein be saved on nonvol-
anile, removable media such as tape for archival and backup purposes. The
amount OX data to be store in conjunction with the relatively slow
sequential access speed o' tape storage devices compared to the fast access
speed of direct access storage devices has led to significant efforts to
compress data and to increase the speed of tape storage devices. However,
there is still a disparity in their relative speeds which is usually dealt
with by first storing blocks of data from the direct access storage device
Jo 20 unto a buffer which is dedicated to providing data to tape during save
operations. I,
~984-017
;~Z3~
In U. S. Patent Jo. 4,360,840 to i~olfrum et at, data is compressed in
real tile as it us produced by a facsimile raster canter and stored in a
buffer for transmission. Data is not transmitted until a full pave of text
has been compressed and stored. While this procedure reduces the amount of
buffer space required to store â pave of data, it does not address the
problem of transmitting the data from the buffer while it is being come
pressed. This can result in loss of valuable transmission time and
possibly wrier larger storage devices to store a full pave of text.
U. S. Patent Jo. 3,490,690 to Apple et at relates to compression of an
instruction trace and recording the compressed data to tape. Compressed
data is written to tape and sections of data are dropped and not recorded
when the tape gets behind or "no data" characters are recorder on the tape
when the tape jets ahead. This results in lost data and does not optimize
the data sty capability of the tape.
Seymour of the Invention
A real time gala transformation and transformed data transmission
device is provided which compresses data provided from a first data medium
arid provides the compressed data to a second data medium which accepts data
at a rate stower than thy rate at which tile first data medium provides
swept. The compressed date is provided to eke second data medium as
required ho top second data medium to operate in an efficient manner. A
tran3fcrm.ltion means rccciveC 2 first Blake of data from the first data
medium and compresses the data into a corresponding block of compressed
data which is preferably smaller than the uncompressed block of data. A
R~8~-nl7
-~23~l4~(~
-3- I
buffer means receives the compressed data and stores blocks of compressed
data for provision to the second data medium. Jo control jeans us ccu?iec
to the transformation means and to the buffer neons for controlling the
buffer means to provide a continuous flow of compressed data to the second
data medium as a function of the data acceptance rate of the second data
medium. Overlapped with the provision of compressed data to the seco.,G
data medium, the control means also causes the transformation means to
compress a second bloc or data into the buffer jeans as a function of the
amount of usable free space in the buffer means.
In the preferred embodiment, the first data medium is a direct access
storage device of a computer itch periodically has its data save onto the
second medium which is a tape storage device. The tape storage device
receives data at a rate compatible iota constant operation of the tare
storage device so that the tape need not be stopped and restarted which
considerably slows the save operation. Data which is stored Jo the tape is
compressed if. accordance with a desired compression scheme to increase data
density on the tape which reduces the number of tapes needed for the save
operation and reduces the tire required to store data on tape.
The first block of data prom tile direct access storage device is
compressed and stored in the buffer means. Once the entire firs bloc'.; of
data is stored in the buffer means, write to tape operation begins.
Twill the first block of data is being written to tape, the control means
coordinates the writing to the buffer means of the second block of data.
Since the data transfer rate of the direct access storage device is much
greater than the data receiving rate of the tape device, the second block
Ripley
I
-L-
of data is preferably written into the buffer means before the first block
of data is completely written to the tape. The second hock no data is
then written to tape, while a third block of data is compressed to the
buffer means in whatever free space exists. The control means assures what
kowtow in the buffer is not lost while blocks are being stored and WriteNow at
the same Tao; the control means also allocates buffer cycles between
storing and writing data, with writing data to the tape having priority
over the storing of compressed data from the direct access storage device.
The Sterno and writing of compressed blocks of data continues until
lo all the desired data is saved on tape. The tape is run in a continuous or
streaming mode unless there is not a complete block of compressed data
available from the buffer means to prevent under running of the tape.
In the event that there is not a complete block of data available from
the buffer means, the tape will stop in an interlock gap until a complete
block is available. The storage capacity of the buffer means is chosen to
be large enough to hold the largest possible block of data which has been
compressed In some instances this may be larger than the block of date
to be compressed if the block of data does not lend itself to compression.
with that size buffer means, and consider no the data transfer rate of the
direct access storage device and the tape device, and the transror~atioll or
compression characteristics of the data to be saved, the availability of a
complete block of compressed data from the suffer means when required by
the tare device is virtually assured.
ROY
~23~6'~
I,
In en. alternate embodiment, data is compressed or transformed arc
stoned in the buffer means. Coordination of the overlc.?pin~ o'
~ransforning and storing, data in the buffer means with writing to and from
tape is based on the amount of compressed data stored in the buffer means.
In this embodiment, the size of the buffer means is determine as a
function of the data transfer rates of the direct access storage device and
the tape device, and the transformation characteristics of tile data to be
saved such that transformed data is always available when requested by the
tape device.
The present invention has the advantage of compressing data in real
time as defined by the tape device requirements for data. This permits the
data to be compressed to its limit in accordance with the selected
compression technique and be written to tape US fast as the tape accepts
the data. Fewer tapes are required for a save operation because the data
is compressed to its limit. Because the compression and writing to tape
are overlapped a desired amount as a function of the predetermined size of
the blocks of data, the tape operates in a continuous or streaming rode
thus reduce no the lime required for the save operation.
B of Description or the Drawing_
JIG. I is a si~plifled block diagram of a real time data transformation
and transmission o~erlappin~. device in accordance with the present
involution; and
~9~4-01,
3~l~60
--6--
Figs PA and 2B, the left side ox I muting with the right side of us,
are a schematic 'lock diagram of the data transformation and transmission
overlapping device of Fig. 1.
Detailed Description of the Preferred Embodiments
.,, ., . . .,,, _
real time data transformation and transmission device is indicated
generally at 10 in Fig. 1. A first data medium 12 such as a direct access
storage device of a computer is coupled by a line 14 to a data
transformation means 16 (also referred to as data compressor 16~ which
transforms data as by compression, encryption or other forms of data
transformation. The lines referred to herein comprise busses, coaxial
cable, optical fibers or other appropriate communicative means. In the
preferred embodiment, data compressor 16 receives a block of data from
first data medium I and compresses the block of data by removing redundant
data bytes or characters itch are typically randomly scattered throughout
the block of data.
i
; sty compressor 16 provides the compressed data to n buffer lo 210ng a
line I Buffer I comprises a dynamic random access memory and provides
compressed data to a second data medium 24 on a line 26. Second data
medium US has a data transfer rate which is usually slower than the data
reinsurer rate of first data medium 12.
In one functional embodiment, second data medium 24 is a magnetic tape
unit such as an I'M model 3430, having known start and stop times and first
data medium 12 is a random access storage device such as an IBM model 3370.
* P~gistered trade Mark
?~09~ 17
.,
....
I
-7- Jo
ennui a section of rlata contains a series of redundant boles, a zap during
itch rho data is provided to buffer 18 occur. If data err written row
the buffer 18 directly to the second data medium 24 as the data is being
compressed, the gap old cause second data edema 24 to stop and roared
the tape to the correct porn. to start receiving data again.
To solve this problem, a control means or controller 78 is coupled
between compressor 16 by a line 30 and buffer lo by a line 3 Controller
I determines Lyon a Blake of data has been compressed ho compressor 16 and
7~rlttelt to buffer 18. Controller 28 then ir.-tiates the writing of that
block of data to the second data medium 24 at a rate determined by second
data medium 24 requests for bytes of data. Lyle the dote it being
transmitted to the sPconfi data tedium 24, a Nat bloc of data so
compressed by compressor 16 as controlled by controller I. The next block
of data is compressed into buffer 18, which is a first in, first out data
huller, before the first block of coF~pre~seci data is co~letely transr;titted
to the second data moderate 24 such that the sickness data medium 24 operates
in a streaming mode and it not recaptured to stop arid start during a bloc or
bitterly data blocks.
In rigs I and ZB, a hardware implementation of a Syste7~ts ~etvork.
Architecture (Sty) compression algorithm used, and the apparatus of the
present inventlolt t S shown. The block of data is received from a storage
device 34 one byte at a time on a line 36 and is rotten into a first
resister 38 and then into a second resister 40 via a line 62 while the knockout
byte ox data is written into first register 38 such that the registers
keynoter. sequen;lal bites ox the block of data.
~09l~ 7
~23~60
Jo first comparator 44 receives the first byte of data in second
l-~gister 40 on a line 46, the second byte of data in first resister ~-~, and
a prime or preselected byte on a line 48 from a memory device 52. Frost
comparator 44 compares the first and second bytes of data with the prize
byte to determine if the data contains a series of at least two prime
bytes. A second comparator 54 receives the first data byte on line 46, the
second data hype on line 42 and a third data byte on line 36 to determine
if there is a series of at least three identical bytes. A series of at
least three identical bytes of data which are not the same as the pyre
byte, detected in this manner are referred to as a nor prime series. If no
prime or non prime series are detected, the condition is referred to as
mixed data.
Ed data bytes are written into a buffer 53 which is coupled by a
line 60 through a selector 62 to line 46. Selector 62 is controller by a
kippers sequencer 64 which receives information identifying the type of
data series prom first comparator 44 and second comparator 54 on lines 68
and 70 respectively. From this information, compress sequencer 64 controls
formation of a string control byte (SOB) ho a SOB coder 72. SOB coder 72
forms the SOB as a function of information provided from cypress sequencer
64 011 a line Al The information and hence the SKYE is representable o.
the type of series ox bytes and indicates the number of bytes in the
particular series o' bytes it represents.
my controlling selector 62 via a line US, compress sequencer 64
controls the content and order in which data provided to selector 62 by
sucker wrester 40 via line by and Subs provided to selector 62 Lo SOB
I nl7
9,.;~3~
q
; coder 72 on a line 79, are written into buffer 58. in the case of a Roy
series, the SKYE is not fulgid by data b-t-- because the prize Tao is
redefined in memory device 52. An SKYE indicating a norpri~e series to
followed by a data byte of the repeated character. on SKYE ind'ca~i~~ mixed
dawn is fulgid by all bytes of the mixed gala.
Using the above SUE compression scheme, data is usual LO compressed o'er
than 50 percent. on the preferred e~Dodi~ent, an SKYE s a bite of dât2 'I
having 2 bits defining the type of series it represents, and 6 bits itch
represent the number of bytes of data in the series up to 'ox. suffer f~8 is
capable of storing 32~768 bytes. I. a bloc is 211 let data, the date
ill expand by one byte, the SOB, for each 63 b toes of data. Wherefore,
tile size o. a Blake of data to be cc~pressed ~'25 predeter~-'ned to go 3-,2;6
bytes 90 that it hill always fit if. borer 58 when compressed.
Use of other compression or encryption routines is within the scope 07' the
present invention. The particular routine described above is implemented
; in hardware to obtain a desired high steed of compression,
Compress sequencer 64 is coupled to first register 38 and second
register 40 by a line I Jo shift data from line TV into first east. US
and to shift data from first register 33 to second register 40. Compress
sequencer 64 also indicates to a compress ~edcless resister lo in I' h. -8
by a line if., the correct buffer 58 address for data to be written irrupt.
Compress sequence 64 is coupled to 2 SOB address register 114 by a lyre
116 and to a previous address resister lo by a line lo SKYE address
~0984-017
ill Z 3 Al Lo I
--10--
register 114 contains the buffer 58 address to the SOB indicating the type
of data run it precedes.
Subs are written to buffer I at a time after a run of mixed data
occurs because the SOB indicates the length of the run which is not Nemo
until the run of mixed data is finished. However, the address of the SOB
precedes the data whether it is mixed or ncnprime. Previous address
resister 118 contains the address of the last byte of the last pre~iiouslv
compressed block in buffer 58. An address selector 124 which is coupled to
compress address register 110 and SOB address register 114 by lines 120 and
10 128 respectively provides buffer a with the appropriate address for data
and Subs on a line 129 as they are written into buffer 58. SOB address
register 114 receives an address from congress address register !10 on fire
126 when initiated by compress sequencer 64 on line 116.
k buffer eon oiler 132 lo coupled to address swallowtail 124 by a line 134
15 arid initiates selection and provision of the address to buffer 58 by
address selector 124. Buffer controller 132 is coupled to compress
sequencer I by line 136 and 138. Buffer controller 13~ rants a butter
cycle to compress sequencer fix on line 136, and compress sequencer 64
indicates on line 138 that a byte of data or a SOB is available to be
20 written eon buffer 58 and the buffer address has been updated. In this
manse , a bloclc of data is co~preFIsed and written to buffer 58.
Once a complete block of compressed data is avai~eb e Roy buffer C8,
the block is transmitted to a set of latches 150 over a line 152. the
avail2bi1ity of a complete block of compressed data is indicated by a
Reunion
3l23~ I
finished block line 139, which is set responsive to a complete block of
data having beer. transferred from date storage device 34 to first resister
38. Finished block line 139 is coupled to compress sequencer 64 which
nitrates transmission of the compressed block of data. Compress sequencer
64 is also coupled to storage medium 34 ho a line 141 to initiate transfer
of data from storage medium 34 to first fog suer 38.
Ye second data Tedium indicated at 154 receives the compressed data
from latches 150 at the rate required b-- second data medium 154. A device
sequencer 158 is coupled to second data tedium 154 by lines 160 and 162.
10. Line 160 provides requests for bytes of data from second data medium 154.
Line 162 provides an indication to second data medium 154 that a compressed
block of data has been provided to second data medium 154.
- Device sequencer 158 is also coupled to buffer control 132 by line 164
which provides requests for a buffer 58 cycle from second data medium 154.
Device sequencer 158 increments a device address resister 170 over a live
172 such that the device address register 170 contains the address of the
byte to be written to latches 150 from buffer 58. The device address
indicated by device address register 170 is provided by a line 174 to
selector 124 for provision to buffer I
The device address is also provided to A previous address comparator
176 and a compress address comparator 1/8 by line 174. The previous
; address coQrarator 176 compares the device address to the previous address
indicating the address of the last byte of the latest bloc of data
completely written to buffer 58. The previous address is provided to the
'~0984-Q L ?
.1~3~ pa!
--I " I
previous address comparator 176 by the previous address register 118 on a
line 1~0. Ike previous address is not inserted into previous address
register 118 until a complete block is WriteNow to buffer 58.
Inn a comparison is indicated on a line 182 winch is coupled to
device sequencer 158, device sequencer 158 stops the transmission of
compressed data from buffer 58 to second data medium 154 because a complete
block of data has been transmitted. The previous address is not changed
before a block is completely transmitted to second data medium !54. A
desired inter bloc zap is then established on second data medium 154 Chile
the previous address is changed and the next block of compressed data is
transmitted without interruption of the operation of second data medium
154. Thus, second data medium 154 can operate efficiently in a streaming
or a start stop mode.
Compress address comparator 178 is coupled to line 126 to receive the
address contained in compress address register 110 which indicates the
present address of the buffer 58 that data is being written into. An
address in device address register 170 equal to an address in compress
address register 110 indicates that data is available to be written into
buffer US, but that previously compressed data from that address has not
vet been transmitted to second data medium 154. Compress address
comparator 178 is coupled by a line 182 to buffer control I co Prevent
the Front of a huller cycle to compress sequencer fix and thus ensure that
data is rot retaken to buffer 58 until data having the tame address is
transmitted to second data medium 154.
R0984-0l7
I 1 ~60
In a further proofer embodiment, previous address resister lo is
loaded iota.. 2 buffer 58 address at which compressed data is stored rich is
a desired number of bytes from the byte currently being written to second
data medium 154. The number of bytes is predetermined as a function of
S the relative data transfer rates of storage device 34 and second data
medium 154 together with the predicted transformation characteristics OX
the data in storage device 34. This permits data to be arranged in other
than compressed blokes of a size defined by the compressibility of the
data. Since the address n previous address register 118 is changer as
data it being written to second data tedium 15~, second data medium 15~
arranges the data to best suit its characteristics. Inter bloc zaps are
inserted by second data medium 154 where and if desired.
In yet a further preferred embodiment, the second data tedium 154
comprises an interface to a communication system such as a packet sicken
system. In this embodiment, busier 53 provides compressed packets to suckered
data medium 154 as a function of the transmission bandwidth of the second
data medium. Buffer 53 serves as a buffer eon both the compression
characteristics of tile packet and access irregularities to the second data
medium, thus ensuring the availability of a compressed packet for
transmission.
rougher controller 13~ prioritizes access to buffer 58. ~ufrer 38
access is requested by device sequencer 15~ on line 164, compress sequencer
Go on line 13 end a refresh controlle~(r.ot shown) when buffer 58 is a
dyllamic memory. Highest prlorltv is given to device sequencer 158
followed by compress sequencer 64~ Lowest priority is given to refresh.
~09~4-017
- :~.;23.1l~
-14-
if all three request access simultaneously, buffer controller 132 grants
p irrupt as described above. In doing this, it places priority an trays-
milting data to second data medium 154 to keep second data medium 154
operating in a continuous manner. Thus, data is saved in a minimal remount
of time with the use of a minimum amount of second data medium 154, whether
it be magnetic tape or transmission bandwidth.
jowl the invention has been shown and described with reference to
preferred embodiments thereof, it will be understood by those skilled in
the art that various charges in form and details may be made therein
without departing from the spirit and scope of the invention.
~84-017
