CA2131627A1 - High-performance non-volatile ram protected write cache accelerator system - Google Patents

Abstract

The present invention implements a data storage system that can be coupled to a host computer system (32) for the transfer of data between the host (32) and a plurality of data storage (34, 40 and 42) devices. The data storage devices (34, 40 and 42) are coupled to a plurality of data transfer channels (86) with each data storage channel being coupled to at least a respective one of the data storage devices (34, 40 and 42). Each data transfer channel (86) includes a data buffer (85) and an autonomously operating controller (38) for transferring data between the channels (86) data buffer (85) and data storage device (34, 40 and 42). A
non-volatile random access storage memory (42) is provided to store cached pages of data. An interface (100) couples the data storage system to the host and through which data is transferred. A
reconfigurable data path (80) permits selective data transfer couplings between the data transfer channels, the non-volatile memory (42), and the interface (100). A controller (76) directs the configuration of the data path (80) and controls a direct memory access controller (114) for burst transferring data between the interface (100) and the channel data buffers (85), between interface and the non-volatile memory (42) and between the non-volatile memory (42) and the channel data buffers (85).

Description

W093~ 61 P~T~U~93tOl91l .

HIGH-PERFORMANC~
NON-VO~ATILE RAM PROTECTED
WRITE CACHR ACCELE~ATOR SYSTEM

~-~s-R fer nce to Related ApE~Lications . The p.esent application is related to the following Applications, all assigned to the Assignee of the , present Application:
1. PARA~LE~ I/O NETWORK FILE SERVER ARCHITECTUREI
0 in~ented by~ Row, et :al., AUSP7209, Application Serial Number 07/~04,g5~, filed September:8, 1989;

2. : MULTIP~E ~ FACILITY OPERATING SYSTEM
ARC~ITECTURE, invented by Hitz, et al./ AUSP7210, lication Seri~al Number:07/4~4,~85, ~iled September 8, 15 ~1989~

3. ENH~NCED VMEBUS PROTOCOL UTI~IZING SYNCHRONOUS
XaNDS~AKING ~ :BLOCK MODE DATA TRANSFER, in~ented by Daryl D. Starr, ~USP721I, Application Serial Number :;07/~0~,Ç3~6, f~iled~September 8, 1939;
: 2~0 ~4.~ : HIGH~SPEED, FLEXIBL. S~URCE/DESTINATION DATA
~: ~ BURST DIRECT:MæMORY ACCESS CO~TROL~RR, in~en~ed by Daryl D. Starr, AUSP7212, Application Serial Number 07/474,534, filed February 2, 1990; and 5. ~ BUS ~OCKING FIFO MULTI-PROCESSOR CO~MUNICATION

:;25: ~SYST~M, in~ented~by Pitts, et al., AUSP7213, Application Serial Number ~7/474,350, filed February 2, 1990.

W093/18461 PCT/US93/OlglI
213162~

BackqLround of the_I~vention l. Field of the Invention:
The present invention is generally related to caching disk co~trollers and, in particular, to disk controller systems pro~iding for high performance, integrity protected~ write data caches established as ~: integral components of a disk storage processor.

: 2. Description o__the Re~ated Art:
There have~been substantial advancement~ in all phases of co~puter technology and design. However, :certain aspects of:;modern complex operatin~ systems have great1y:increased~the re~uired perfoxmance of the data storage: sub-sys~tem. Specifically, UNIX and similar mul~i-tasking operating systems typically perform disk da:a ~ransfer operations divided into sub-operations as necessary to ~support the multi-tasking acti~ity of the operati~g sy~tem. ::~ Consequently, the actual data tranæfer operations performed by the disk sub-system are poorly optimized;and therefore directly degrade o~erall 20~ system perfo ~ nce.~ ~
A commonly~utilized component of the UNIX operating :system is a communications layer know as Netw~r~ File System:(NFS).~ ~The NFS layer provides a network control protocol that~permits convenient logical access to the di~k storage sub-sy tems of remote network connected computer systems.~ The design o~ the NFS protocol is es~entia11y state1eRs in that each atomic NFS operation proceeds without~:~presumption of the~ pre-existing data ; transfer state of the disk storage sub-system. Thus, many NFS directed data transfer operations are repeated .~ , : :

W~93/18461 2 1 3 1 6 2 7 P~T~VSg3JOl91I

and even poorly ordered in their submi~sion to a disk storage sub-system.
A typical solution to improving disk storage sub-s~stem performance has been to imple~ent a read data : 5 cache either in a specialized R~M cache memory or within a buffer pool formed ~ithin the main memory array of the computer ~ystem. The la~ter approach is particularly ad~antageous since, as in the UNIX operating system, ~irtually referenced memory objects can be stored in the read data cache. This, in turn, allows a greater pote~tial optimi~ation of the process of satisfying read data requests by reference to the virtual cache and :~ withou~ requiring a further time consuming virtual to phy~ical translations or e~en data tra~sfer operations through the disk ~storage sub-~ystem.
A ~separate, specialized cache RAM approach typically operates the cache in a physical block number :mode. An ad~antage presented by this approach i~ that : physical read data transfers can be ordered to maximize ;;20~the~data ~ransfer bandwidth of the actual sub-system ; disk~ drive~ However, the ~irtual referencability is lost.: ~
Write :data caching has also been utilized to enhance perfonmance. However, a substantial` data 25~ :i~tegri~y problem:arises in th~t data stQred by the disk s~orage ub-sy tem will not be accurate until all data write~ have completed to disk. A n ~ er of approaches : ha~e been adopted t~ mlnimize or prevent data corruptiQ~
in the instance where all write data~is not properly 3~0 upda~ed to disk. The simplest a~d most common approach :is to implement a write cache in main memory subject to control by the UNIX file system (UFS) and implement a flushing algorithm whereby all data in the cache is WO93J18461 P~T/US93/0l91~
213162~

written out to disk on a fairly frequent ~ime-controlled basis. Where even this limited exposure is considered too great a risk, a mechanism known as synchronous writes has been developed and widely applied. This mechanism, required for compliance with the NFS
protocols, requires that all data writes complete ~:~ entir@ly through to "~afe storage", typically a physical disk before being acknowledgPd as completed to the NFS
~er~er originating the data ^transfer. This of course has the disadvantage sf imposing a substantial time ; penalty on every NFS write.
Finally, a write data cache in main memory can be made substantially immune to the loss of data by ; isolating the buffer in non-volatile memory. This 15 : approach also has a~ number of difficul~ies associated with it, including management of the restoration of the ; state of the operating system when the system is re-booted~and ensuring that writes from the wri~e cache can : correctly complete to the disk storage sub-system reliably. An advantage of this write cache design is that all read and write data is logically managed by the :UNI~ file sy~tem directly. Therefore, all overlapping read: and write::;reguests can be cent~ally managed and resolved.: A corresponding disadvantage, of course, is ~hat the UFS :must be modified to understand and implement a write cache buffer.
~ Accordingly, there is a need for a comprehensive : ~ solution to improving the through put and control of cache memories in connection with a disk stox~ge subsystem in the context of complëx operating systems.

~:~, ..

WO 93/1~1 21316 2 7 PCT~US93/01911 .

Summarv of the Invention A general purpose of the present invention is to provide a high performance storage processor system incorporating non-vola.tile cache RAM for supporting an optimized write cache opera~ion with respect to multiple data storage de~ices.
This is achieved in the present in~ention by implementing a data storage system that can he coupled to a host computer system for the transfer of d~ta between the host and a: plurality of data storage devices. The data storage devices are co~pled to a ; : plurality of data transfer channels with each data storage channel be coupled to at least a respective one ; :o~ the~data storage devices. Each data ~ran~fer channel ,~ ~
lS includes a data:buffer and an au~onomously operating :: controller for tra~sferring between the channels data : buffer and data 6torage device. A non-volatile ra~dom access:storage~memory is~pro~ided to store cached pages of~data. An interface:couples the data torage system 2:0 :to the host and: through which d~ta is transferred. A
M~ rec~nfigurable data path permîts selective data transfer coupl~ings between the data transfers cha~el~, the non-: : volatile memory/; ~and the interface. A controller directs:the configuration of the data path and controls 2S~ a~direct memory:acce66 controller for burst transferring data betwee~ the interface and the channel data buffers, between the interface and the non-volatile memory and between ths~ non-volatile memory and the channel data buffers.
: 30 ~ By storing cache pages of:write data in the non-: ~olatile memory, an immediate ~acknowledgement of the ~::: data write can~be issued to the host computer ~ystem while allowing the controller to process the ::

:

W~93/1~4~1 PCT/US93/019ll 2~.3~6~ ' relationship between the cached pages of data in thenon-volatile memory for optimizing subsequent data transfer operations necessary to move the cached pages of data from the non-volatile memory to the channel data buffers.
Accordingly, an advantage of the present invention is that it directly support~ the management of multiple data storage channels in a consistent, highly integrated manner. Support for both simple and complex control arrays of data storage devices are transparen~ly managed.
: Another advantage of the present invention is that : : it manages and properly resolves all comhinations of o~erlapping data tran~fer reguests, including exact and ; 15 i~exact o~erIaps, read after write, and write after :
write ordering of data transfers.
A further adYantage of the present invention is that is queues data transfer request~ ~hrough a prioritization and arbitration scheme whereby write transfer operation ~from cache to disk are increasing : op~imized u~der~increasing data transfer ~ctivity.
Yet another advantage of the ~resent invention is that read re~uests ~re treated as cache bypas~es thereby :maximizing the availability of the cache for cached write requests opti~izations. Purther, the capability or selectiv~ly caching write reque~t~ ba~ed on file .~ ~ sy~tem identification is provided.
Still another advan~tage of the present invention is ~: that it-reguires :~ minimum change to a conventional multi-tasking operating system kernel in order to permit :~ complete implementation of the present i~vention. The ~ only change necessary is to allow the kernel file system :~ control Foftware to identify whether specific write WO93/1&461 2 1 3 1 6 2 7 PCT/US93/01911 requests are to be cached or not. Consequently, the present invention is isolated and correspondingly insensitive to the particular implementation, mechanics and operation of the sys~em software in and abo~e the storage processor device driver level. No change to the de~ice dri~er is re~uired.
A ~till further advantage of the present invention is that pxovides for a robus~ mechani~m for identifying the presence of ~alid càched pages of data on a restart ~0 of the operatlng system and for reco~structing the proper order of write operations for data found to exist ; in the non-~olatile cache memory. As part of the initial evaluatio~ of valid cache data present in the : non-~olatile memory, a control mechanism i5 maintained ~o both identify a~d verify the drive storage device configuration.
:: Yet still: ano~her ad~antage of the present invention is ;that it readily manages ~he presence of mirrored:drive storage configurations in a consistent ~and ~ BUb tantlally transparent operational manner : relative~to all:aspec~s of the pre~ent in~ention.

: Brief~De cription of the Drawinqs The3e and:other i~tended ad~antages and features of the~present in~ention will bec~me appreciated and better understood whe~ considered in connection with the accompanying drawings, wherein like referenced numerals throughout the fi~ures designate like parts, and wherein:
igure 1 is a block:diagram of a conventional approach for providing a non-volatile write cache in a multi-taski~g, multi-user operating system en~iro~ment;

.

W~93/18461 PCT/US93/01~11 2~3162~ ~ ~

Figure 2 is a block diagram of a computer system employing a high integrity write cache accelerator sub-system in accordance with a preferred embodiment of the present inventio~;
Figure 3a is a detailed block diagram of a preferred embodiment of the storage processor sub- ystem ~ ~ constructed in accordance with the present invention;
: Figure 3b is a simplified block diagram of a data channel interface unit in accordance with the preferred ~:10 embodiment of the present invention;
Figure 3c is a simplified block diagram of the preferred daughtercard implementation of the removable non-~olatile memory array with attached battery backup capabilit;~ as constructed in accordance with the present in~ention; and Figure 4 i:s a schematic representation of the : . software controlling d~ta structures and related hardware as implemented i~ a preferred embodime~t of the present invention.~

20: ~ Detailed Descri~tion of the I~vention A prior art ~approach to pro~iding for a non-~ola~ile write data cache in a compu~er system supporting a:multi-tasking, multi-user operating system is shown in Figure l~ The prior art syste~ lO includes a host proce or 14 coupled, in ~ typical fashion, to a VME backplane bu~ further æupporting a main memory array 16 and a disk sub-system 18, including a disk controller tha~ directly supports conventional SCSI or SCSI-2 data :transaction operations. A non-~olatile random access ~30 memory (RAM) array 20 is also supported by the bus 12.
:~ :The non-~olatile RAM array 20, at a minimum, maintains :write data ultimately destined to the disk system 18.

WO g3/184~1 Pcr/vss3/~lsll "-` 21311~27 A battery 2~ is provided to maintain power to the non-vola~ile RAM array 20 in the absence of ordinary system power. The battery 22 allows the cache data present in the Ilon-~rolatile R~q array 20 to be maintained ~chrough 5 a power cycli~g of the system 10, thereby allowing data maintained in the non-volatile RAM array 20 to be updated to the disk sub-system 18 as part of an error : recovery proces.0 An ad~antage of the system 10 is the data write operatio~s can be directed through the non-volatile RAM
array 20 resulting in a substantial increase in the apparent disk sub~system 18 performance. That is, where further operation for an: executing; process must wait until a pending write request has completed ~o ~he disk ~sub-system 18, completion of the write ~o the non-volatile ~M array:20 allows the write acknowledgment ~o be re~urned much sooner. Thusj apparent performa~ce is :
in~reased in:pr~portion to the ratio of di~k sub-system write:access spe~ed to the write access speed of the non-20; volatile RAM array 20.
O~:closer examination of the~:system 10, a number of ` di~advanta~e ~become apparent, parkicularly as the data ;:transfer Ioading of~the system 10 increa~es. Since all : di~k related ~data transfer operation3 must process 2~ thrQugh the host processor 14, a data write ~o the non-;`~ : volatile ~AM array 20 must eventually be followed by a ~, return transfer to the host proces~or 14 followed by the actual data write to the disk sub-~y~tem 18. However, : the data bandwidth of the bus 12 is finite. Thus, the : : ~ 30 design approach of system 10 will i~cur a sub tantially : increased bus loading factor, estimated to be on the :~ order o~ 50~, particularly increasing with data request ~ activity. In addition, since data transfer~ from thè
: :

W~93/18461 PCT/US93/01911 2~3l62~

non-volatile RAM 20 to the disk sub-system 18 are likely to be of low priority relati~e to other data : transactions, the non-~olatile RAM array 20 will typically tend to rapidly fill with data, resulting in a significant slow-down circumstance where the host 14 requires to write data to the non-volatile RA~ array ~0, but first an adequate amount of cached data must be flushed back to the host 14 from the non-volatile RAM
array in order to allow the initial write operation.
Finally, a limitation of the ~ys~em lO involve the need to manage the data in the non-volatile RAM array 20 dynamically through the operation of the system lO and as part of an error recovery routine following a system restart. In the former case, a ~o~trol structure :15~ ~ecessary to track the data prese~t in the non-volatile RAM array 20 would desirably be stored by the host processor 14 in the:host's own local memory in order to : speed the necas~a ~ management operations. However, such ~structures~would be lost o~ a system shut down.
20~ Therefore, a management control structures must be :maintained as an i~tegral component of the cached data within the non-volatile RAM array 20. Unfortunately, e~ery ac~ess of~such control ~tructures would result in urther traffic acosS the bus l2 and a correspo~ding 25; and likely significant reduction in the o~erall : performance o~ the ~y tem lO.
A high performance wrlte cache system 30, : constructed in accordance with the present in~ention, is ~, shown in Figure 2. There, a host processor 32, main memory 34, file processor 36 and:storage processor 38 are interconnected by a high-speed bus l2. The bus 12 and high-speed data transfer protocols, as well as the preferred architectural design and implementation of the : ::

W093/1g461 2 1 3 1 6 2 7 PCT/US93/01911 host processor 32, main memory 34, file processor 36 and storage processor 38 are disclosed and discussed in detail in the abo~e-identified co-pending related applications; which applications are hereby expxe~sly incorporated by reference herein.
The storage processor 38 includes a non-volatile RAM array 42 with battery backup 44 for selectively storing wri~e data inte~ded for any one of an array of di~ks 40, numbering from as few as one di~k drive to as many as 60 drives on lO parallel SCSI data channels.
~: The architecture of the system 30, fully consistent and with the present invention permits up to at least three fully functiona1 and independently operating s~orage processors 38, with respective attached non-volatile RAM
15~ arrays 42 a~d~ backup batteries 44, to be connected to the bu~ l2. Transfer of data between the bus 12 and ramote NFS clients and servers is provided by way of ::network :proces~or 46 directly supporting one or more Ethernet t~pe local area networks ( LAN9 ) 4 8 .
20~ In ~the preferred circumstance, NFS type data transfer ~equests::are processed through the ~etwork proces~or 46 and passed to the file processor 36. In urn, the file~ pxocessQr 36 identifies a storage location for:the::NFS request referenced data and pas~es this~;control: information to the storage proce~30r 38.
: Co~seque~tly, the~ storage processor 38 is operating : ~es~entially at a physical le~el; the control informatlon provided specifies data using disk and, in the ca e of SCSI drives, logical sector identifications.
~: 30; Consequently, all write cache co~trol operations and recovery procedures will, in accordance with the present : inven~ion, occur :~substantially outside of the ~:: conventional boundaries of an operating system executed ::

~3~6~ 12-by the computer system 30. That is, ~he write caching control, error recovery control and write data : optimiza~ion operations occur at a le~el below that of :~ the operating sys~em device driver level having responsibility for interface control over the storage :~ processor 38.
: The only fundamental kernel level change required ; for the full utilization of the present i~vention is implemented in the: UNIX file system control code as 10~ executed by the file ~processor 36. Specifically, the present invention contemplates utilization of a system : administration level file such as /etc/f~tab to identify file systems that are expressly to be treated as qubject to writ:e caching. : The default circumstance i9 to not ca:che write data operations. By identifying those file :systems :that are to be subject to write data caching, the~:file: processor~:36, in the processing of a data transfer ~re~ue t~ ir.~olving write data, preferably identifies whether: the data write may be cached by the 20~ type~0f:data~write:~request message sent to the storage :;proc~essor:38.~ T~is capability is of some substantial ;significance in~ that the / (root) and, typically, the /usr~file yst~m~::in a~SunOS implementation of the UNIX
operating: syætem should be~left uncached. While it is :25~the :~ultimate ~i~tention~ of the pre ent invent;ion to ensure that all~write data operations ultimately ~ucceed with a proper transfer of data to the disk 40, the transition prior~to the non-volat~ile RAM error reco~e~y and restoration:~remaining cached write data to the disk 40, requires that~at least the root file system must be assured of ~correct operation as necessary for in~itialiæation:to at least the maintenance mode le~el of the operating system.
:

: ~

WO93/18461 PCT/USg3/01g11 The only other alterations to the operating system executed by the system 30 is the addition of a system admini~tration u~Qer le~el program to administer the state of ~he non-volatile R~M array 42, to collect statistics, and obtain the current status of the non-~:~ ~olatile RAM array 42. This user le~el program need only be capable of passing a message ultimately to the low level emhedded control routines directly executed by the storage proce~sor 38. These system le~el commands include:
1. Init - this option i~ normally run automatically during rc.boot. It initializes the no~-Yolatile RAM array and puts it into a off state. In examining the contents of the non-volatile RAM array 42, if the storage processor 38 finds a~y unwritten cache pages, the storage processor 38 attempts a write data flush operation to transfer the data to the disks 40. If the : : writes are successful, the non-~olatile RAM array ~: 20 :is placed in the off state ~nd is otherwise ready :~ for use. Otherwise, the non-volatile RAM array is : put a dirty state indicating that the non-vo~atile RAM array 42 contains cache pages that cannot be written to ~he di~ks 40. If, in processing the : 25 control structure~, asso~iated wi~h the non-volatile RAM array 42 and maintained in non-:, volatile R~M, fails, the non-volatile RAM array is : put in a bad- chec~sum state. A ~rite to disk can ~: fail either to disk errors or a change in the disk ~ 30 40 configuration. The sy~tem administrator has the : : options to a) throw away the data and continue; b) correct the underlying cause of the dat~ error and retry the data write; c) reconfigure the system and ~ .
:

WO93/18461 PCT/US93~01911 ,. ~

2 ~3 ~ 6~ -14-reboot; and d) save the data and continue. If the storage processor is unable to place the non-volatile R~M array 42 in a off state, an error message is passed back to the host processor 32 and ultimately to the system administrator reflecting the error condition.
2. On ~page count] - puts the non-~olatile RAM array 42 in an on state. This command is , ~
~; ~ recognized only if the state of the ~on-volatile RAM array 42~ is currently o~f. The optional page_count ~alue allows the system administrator to specify a number of cache pages less than the maximum available number of cache page~ to be us~d for wxite ~caching. In a lMByte x 32 data space, ;15~ ~ l28 8KByte ~ache pages can be allocated. The default is to-use all available memory within the non-volakile R~M array 42 for cache pages~ Using a lesser number of cache pages may be of u e in e~aluati~g~performance of the write cache.
20~ 3. Off - ~put the non-~olatile RAM array 42 b~ck~ to~the~ off state. This command is only recognized if the non-volatile RAM array 42 is currentIy~in~the on state. In changing state, the torage processor 38 attempts to flush all 25 ~ u~written~cache ~page data stored by the no~-w latila RAM~array 42 to~ the disks 40. If the fl~sh fails, the non-volatile RAM array 42 is put in the ~ rty~state.

4.~;Purge - when the non-volaile RAM array 3 ~ 42 i~ dirty, this command causes all unwritten data present in the cache pages of ~he non-Yolatile RAM
array 42 to be discarded and forces the state of the non-volatile RAM array 42 to off .

W093/18461 PCT/US93/Olg11 ,~ 2I31627

5. Flush - when the non-vola~ile RAM array 42 is in either the on or dirty states, this command directs the storage processor 38 to initiate write operations to transfer all unwritten data in the non-volatile RAM array 42 to the disks 40. The ~tate of the ~on-~olatile RAM array is left either on, if the state was initially on and no wri~e failures were encountered in the write data tran~fer operations, or dirty, where the initial state was dirty or a write failure occurred, and off, where the initial state was dirty and no write ~ailures occ:-~red.
: . 6. State - this command .rects the storage proces~or 38 tc~ report back the current state o~
the non-~olatile RAM array 42.
7. Errors - this command directs the storage processor 38 to re~urn the li~t of currently un-writable cache pages as then mai~tai~ed in the no~-~olatile RAM:array 42.
2~0 ~ P multlple stage message passing protocol is utilized to imple~ent these co ~ nds. In general, a 128 byte me~sage is composed by a proce~sor, for example the host processor 32, and transferred to a reser~ed area of t:h~e main m~mory 34. A single word (32 bits wide) 25:- message descriptor is the~ pas~ed by the host processor : dire~tly to, for~example, a c~mmand ~IF0 o~ the storage processor 3B. This message descriptor identifies the location of the mes~age in main memory 34 a~d basic information as to the action to take with respect to ~he message. Typically, the storage processor 38 will undertake to transfer at least an image copy of the messa~e from main memory to` local memory onboard the storage processor 38. The storage processor 38 then , ,....

~3~6~ -16--e~aluates the message and, as appropriate, performs thecorrespon(.1. ing function or functions. The~, the storage proce~sor 38 will substantially overwrite the original message il:L main memory with a reply message gi~ing, for example, the completion status of the reguested operation and:reporting the state of the non-volatile RAM array 42. The storage processor 38 will then send a reply me!ssage descriptor to the command FIFO present in the ori~inating proce~ or, usually the file processor , 36.~ Upon ~3xamining~the reply message in m~in memory 34 : identified by: the reply message descrip~or, the originatiI^Ig processor can obtain the information pro~ided b~the message.
A det:ailed block~diagram of a storage processor SO, 15~ constructe~ in~accordance:with the preferred embodiment o~the prese~t in~ention, is shown in Figure 3a. A
mi~roprocessor 52:~ixects the ovexall operation of the torage~rocessor 50. The microproce~sor 52 is coupled :m~to:a~local processor~bus 54 wi~h attached ROM 56, static 20~ ;RAM~ 58, and: non-~volatile RAM 60 u~its. The ~ROM 56 pre erably:;co~tain an executable program ~ufficient to allow~;the microprocessor~ 52 to:boot and load an image opy~of the host~ operati~g system to the host processor 32~and,: ~ubsequently, to receive~ia the host proce3~0r ;25~ 3;2~an ~executable~ image of a~ co~trol program to be independently~executed ~b~ the microproce~sor 52 to establish cooperative operation among the proce~sors 32, 36,~ 38, and 46. ~This;:executable image is loaded In and executed from the~static RAM array 58. The st~tic RAM
30~:: array 58 also: is utilized to~3tore working program , ~ ~
~ariables and control~structures including, for purposes of the present invention, an:~active queue of data t}ansfer request~s. The non-volatile RAM uni~ ÇO is : :

WO93/18461 P~TJU~93/01911 ~ 2131627 utilized to maintain two significant control structures:
a non-volatile control structure, nvcntl, as defined in Table lo Table 1 Non-volatile control structure:
struct nvcntl {
int magic;
int cache page_cnt;
10 : char disk_serial_notMAX_DRIVES][12];
: ~ int check_sum;
}

: and an array of nvbuf co~trol ~tructures, where each is defined ln accorGance with Table 2:

Table 2 n-volatile data buffer control structure:
struct nvbuf {
~ ~1nt:~ equence_no;
int : sector;
hort sector_coun~;
u char di~k_number ~Max_No_Drives]tl2]
; :: . u:char :state;
::~int check_ um;

:::
:, Only a single nvc~tl control structure is maintained.
However, the number ~of nvbuf control structures is defined, i~ the preferred embodim nt of the presen~
invention, by the ~u~ber of cache pages that can be - : simultaneously allocated for use by the storage ::
: ~ processor 50.
:

: ~ : :

WO~3/1i~61 PCT/US93/01911 ~3~6~ -18-Also connected to the local bus 54 are a command FIFO 64, utilized for receiving message descriptors, a command register 70, utilized for sending message descriptors, and a data buffer 82 used for sending and receiving messages. The command FIF0 64 and command regi~ter 70 are both coupled to an internal data bus 66 that connects through a VM~ interface unit 6~ to the bus 12. The inter~al data bu~ 66 also connects through a : bi-directional register 74, intenmediate data bus 94, ~econd bi-directional register 78, a second~ry data bus ~: 80 and finally to the data buffer 82. Also connected to the secondary data bus 80 is an array of, in the preferred embodiment of the present invention, ten data channel interf:ace u~its, 860 through ~69. AS shown in :Figure 3b, each data channel interface unit 86 includes a data buffer 85 provided between the secondary data bus and a SCSI interface controller 87 capable of .
autonomous operation at:least for the transfer of data between a SCSI peripheral and~the data buffer 85.
~ Finally, a~ battery status indication signal is :: proYided essentially from the battery 88 to the local :bus:54~to allow status monitoring by the microprocessor r ~
In ~ support of the microprocessor 52, an : 25~ autonomo~ly:operatlng DM~ controller 72 i~ provided to ` control: the DM~ burs:t transfer of data between the VME
: interface unit 62, the data buffers 85 of the data channel interface units: 86, the data buffer 82 and, finally, a non-volatile cache memory 76. This non-30~ volatile cache: memory 76 is coupled to the intermediate data bus 94 for the bi-directional transfer of write cache data.

~:
~ .

W093/l84~1 PCr/~S93/OlglI
`` 2131627 .

The DM~ controller 72 generates a variety control and address signals on the control bus 84 to ~elect, initiate, a~d direct the operation of burst da~a transfers. In accordance with the preferred embodiment of the present invention, each burst transfer consists of 128 bytes. Cache data writes from the VME interface unit 62 are~ pa~sed by the register 74 to ~he non-volatile cache memory 76 for storage at an address provided by the DM~ controller 72 via the address line~ ~ 10 portion of the ~ontrol bus 84. Similarly, the DMA
controller 72 control DM~ burst tran~fers of write data from the non-volatile cache memory 76 through the bi-directional register 78 to the buffer 85 of one or more of the data channel interface units 86. Thus, an 5~ optimized, readily reconfigurablP data path is establi~hed to clQsely integrate the non-volatile memory array 76 with the balance of the storage proces~or 50.
However, in~accordance with the teachings of the pre3ent in~e~tion, a storage pro~essor 50 itself ~ y be 20~ subje~t failure. Thus, the non-~olatile memsry unit 60 and non-~olatile~ ache memory 7~ are preferably implemented on~a~ daughtercard detachable from the remainder~ of the~torage processor 50. Indeed, the torage proce~sQr 50 i5 quite capable of continued ~operation, albeit~without write data cachi~g capability, in the ab~ence of the non-volatile RAM unit 60 and non-:
~olatile cache memory 76.
As ~hown in Figure 3c, a daughtercard unit 98carries the non-;volatile RAM unit 60 and non-volatile cache memory 76 in their entireties. Address lines 54~, 84' and data lines 54", 94 extend off ~he daughtercard 98 to allow interconnection to the b~lance of the storage processor 50. Also included on the daughtercard WO93~18461 PCT/US93/01911 2~3~62~

98 i~ a sense switch unit 96 that performs the power switching function from standard system power V~ to ~he battery backup power source 90 V~AT. The sense switch 96 also perfo~ms the function of sensing the system power status and a low battery condition and providing corresponding signals on the control lines 92.
The value of the present invention is be~t illustrated when considering operation in support of NFS
write operations. In a conventional computer sy~tem without wxite caching, an NFS write operation ~ypically ~: re~uixes an average of 30 milliseconds to complete, since the write must physically complete to the disk.
Due to the stateless design of ~FS, data writes are requ'red to be synchronous with the disk sub-~ystem.
15~ That is, the write muRt complete $ully to disk prior to acknowledgment of the write data transfer; the process ~:~ initiating the NFS write request remalns blocked until the acknowledgeme~t is received.
A comparable NFS~write operation utilizing the non-:volatile cache memory 76, in accordance with the presentinYentio~l permits an ~FS write to complete in les~ than about one millisecond. The s~chronous NFS write ::
protocol is not violated since the acknowledgmen~ of the :write is not re~urned until the data i8 safely ~tored in ~: : :25~ the no~-~olatile cache memory. Naturally the integrity of such data mNst be insured until ultimately written to disk. ~ -On a loaded system the disparity between cached and: :; un-cached NFS writes increases. A typical NFS client will employ a number of basic I/O daemons (BIODs) in order to manage:multiple concurrent NFS operations. A
~:~ typical delay of an NFS write operation over an Ethernet network is on the order of about 42 milliseconds: 7 WO93/18461 2 1 3 1 ~ 2 7 PCT/VS~3/01911 , millisec4nds for Ethexnet transport, 3.5 milliseconds for network protocol processing, l.5 milliseconds for file sys~em proce~sing and 30 milliseconds for disk accessing and data return.
A si~gle NFS write operation results in one to four, typically two to three, synchronous writes being re~uired for inode and data accesses and updates. These accesses are typically sequential and therefore time accumulati~e. A period even where a large number of NFS
write data blocks àre transferred in fast succession over the network, and even with the typical case of four BIODs to allow parallel execution of NFS data write operations, the initial f~ur resulting NFS writes will ~till take on the order of ~bout l40 mil~iseconds~ si~ce ~he client has to wait until the last write is completed ~ and acknowledged.
:~ With the write cache of the pre~ent invention, fast ` uccessive da~a write operations c~n be completed to the non-~olatile cache memory with a total comparable , ~ ~
elapsed time of about 40 milliseconds or less. The : present invention further takes advantage of the -opportunity presented by maintaining write data in the non-~olatile cache memory to substantially optimize ~he ub equently:required data transfer operation~ to the physical di~k system. Particularly in the ca~e of the UNI~ operating ~y~tem, the:UFS will tend to write the inode and related indirect blocks repeatedly while writing file data ~equentially. The present invention ~:optimizes the data present in the non-volatile cache ~30 memory by discarding effectively ~verwritten blocks and : ::by coalescing sequential data blocks for transfer in a ~:single continuous write operation. The overwriting and co~lescing of blocks wlthin the non-volatile cache W093/1846l PCT/US93/0191l , .~ .~
~,~ 3~62~ ~

memory 76 directly eliminates and significantly optimizes the number and nature of physical disk transfer operations necessary for transferring the write cached data to disk.
In accordance with the present invention, the length of time that write data remains in the non-vol~tile cache memory is proportional to the data transfer loading o~ the storàge processor 50. That is, cache writes generally remain in the non-volatile cache : 10~ memory 76 for so long as read and uncached write data transfer operations are continuously being performed.
Howe~er, the degree of optimization obtainable through ~: ~ maintenance of delayed write data in the non-volatile cache memory 76 increases with the amount of time that ~the write data is dela~ed. Thus, optimization of the wrlte data operations of the torage proce ~or 50 ~ends to ~increas:e, at~ leaæt in partial compe~satio~, with :increasing or continuing read and uncached write data ransfer acti~ity.
20 ;~ A simplified block diagr~ of the ~ajor data structures and supporting hardware involved in data transfer operations through the storage proce~sor 50 are illustrated in Figure 4. The ~ystem 100 generally de8cribes he operation of the 8torage processor 50 in respo~se to mes~age~ received at 102 and, ultimately, the generation of DM~ control signal~ for the control of a block data transfer on the contxol lines 84. This proce~s i~ managed by the microprocessor 52 through the ~:~ us-e of an active request queue 104 formed as a control structure preferabIy within the SRAM u~it 58 of the storage processor 50, a ~vcntl control ~tructure 121 : : formed within the non-volatile RAM unit 60, a nvbuf array control structure 122 also formed within the non-~ .

W~93/1~61 PCr/USg3/01911 ~: r ~ 2 1 3 1 6 2 7 volatile R~M unit 60, and a delayed write queue 123, DMAwait queue 126 ~nd buffer wait queue 130, all of which are formed as control structures in the SRAM unit 58.
Finally, an array of cache page, each preferably consisting of an 8192 byte data page, are fonmed at fi~ed con ecutive locations within the non-volatile : cache memory 76.
By their construction in the S~AM unit 58 and non-volatile RAM unit 60, the active queue 104, delayed ~0 write ~ueue 123, DMA wait queue 126, buffer wait queue 130, non-volatile control structure 121 and non-volatile :~buffer array 122 are directly accessible by the microprocessor 52 as represented ~y the data control access paths 13~. Thus, the microproces~or 52 can :directly~ implement the transfer~ of da~a and control i~formatio~ among these structures. In addition, the microproce~sor has data and control access ~o a DM~
chan~el buffer ll0 utilized for communicating with a DMA
: processor 114. Together, the D ~ channel ~uffer 110 and DMA processor:ll4 form the DM~ controller 72 as shown in : Fisure 3a. ~The high-~peed DM~ controller and its operation are disclo~ed and di cussed in detail in the co-pe~ding related application, HIGH SPEED, ~LEXIBLE
SOURCE/D~STINATION ~DATA BURST DIRECT MEMORY ACCESS
~, :CONTROL~ER, which application i9 expre5sly i~corporated by reference herein.
.~ The D~ channel buffer 110 is, preferably, a dual ported memory ~utilized to store a~ array of channel :operation control structures. Bach of these chaDnel operation control structures are identically formatted ~:to define for the DM~ processor 114 a respective data transfer operation. In the preferred embodime~t of the present invention, there are twelve significant DMA

W093~18461 , PCT/VS93~01911 ?,~3~62~

channels. Ten of these channels correspond to the ten SCSI data channel interface units. Another channel is used for co~trolling the transfer of cache write' data , from the VME interface unit 62 to the non-volatile cache S memory 76. The twelfth DMA chan~el controls the transfer of a message~between the VME interface unit 62 and the data buffer~82.
The DM~ proces~or ~114 operates as an autonomous state machine ;perpetually scanning the DM~ channel 10~ ~bu~fer 110~ and~initiating ~NA ~channel operations for each of the~DMA channel~control structures found ready.
, Since the DMA processor 114 can direct a high-speed transfer a 128~byte~block~of~data~between data source~
~`s~ and~des~tinations, a~speed signif`icantly faster than the ;15~5~SI~interface~c;ontroller 87~can transfer data between its~associated~buffer 85 an ~SCSI periphexal, the DMA
procés~or 114 ca~effectively interlea~e multiple data tran~f~er~;operations~as a con'ee~uence of it~ yclic re-luation of~the~DM~channel control buffer 110. Once 20~ the~ DM~ process~or~-~114 has ~completed the~ entire data tra~;f~er~specif~ied by~a channel contr~l structure~ for a ",~ co~responding~D ~ cha ~ 1,~ the~ ~ transfer~ d interrupt, ~is~ provided~; to the~ mi~roprocessor 52 ~ia channèl~lnterrupt~lines~;116. ~ The~microprocessor 52, ;25~ e;~fective1y ;~as~;part of the interrupt servlce routine then~reevalu~tes;~the~co~tents~of~the~request queues~and c~ontrol,5tructures~ 104, 123, 126, 130, 121, 122 to determining whether another data transfer r~quest is pendin~ for the~ freed D ~ transfer channel. If a 30~ candidate request ~is identified,~the micropr~cessor 52 processes from ~the~request entry the necessary control n~ormation to~complete the DMA control structure within the DMA channel buffer 110. Once the new DMA control :

W093/1~61 2131627 PCT/US93/~1911 structure has been constructed, a flag is set in the DMA
control structure to signal the DM~ processor 114 on its next cyclic pass to cvnsider this new DM~ con~rol ~truc~ure for i~itiation of a corresponding channel DM~
transfer.
; The nvcntl control structure 121 co~tai~s a single instance of the nvcntl structure ~hown i~ Table 1. The variable magic ~ is set to a predetermined ~alue, preferably FF0055AA hex, to indicate that the non-~olatile RAM unit 60 and non-~olatile cache 76 have been initialized and valid write data may be stored. Any other value is considered to indicate tha~ the non-volatile RAM unit 60 and cache memory 76 are not initialized. The~:variable cache_page_cnt ~pecifies a 15 : size~o~ the non-volatile cache memory 76 ~y an integer :; CQU~ of fixed 8KBy~e cache pages that ~re available for : use. The array variable disk_serlal_no is u~ilized to store~ disk drive~ erial numbers for all disk dri~e~
attached to the~:storage processor 50. In the preferred : 20~ embodiment of the~present i~vention, each di k dri~e is : elec:tronically ~ erialized with a uni ~ e nu~her : : ` identifying the~par~icular di k drive unit. Finally, the check_sum variable holds a conventio~ally calculated check~um ~alue~.:;for the nv~ntl control sructure.
25~ ~Reralculation of~the check um ~alue allows the processor ~: ; 52 to check the ~data integrity of the nvc~l contxol struc~ure 1210, The ~vbuf cQntrol ~uffèr array 122 contains an array ~of nvbuf ~ ~structures having a one to one corre~pondence with the cache pages present in the non-~olatile cache memory 76. The data maintained in each nvbuf data ~tructure is sufficient to reconstruct a corre~ponding~delayed write queue entry upon evaluation .
.

WO g3~1~461 Pcr/uss3/olsll ~,~3~6~

by the microprocessor 52 following a restart of the operating system. Specifically, the ~equence_no variable maintai~s the count value ordering of cache data write requests. The order of cache page alloca~ion, and therefore of the corresponding requests, is neces~arily important in re toring the order of any repeated or o~erlapping data writes that may be pending in the reconstructed delayed write ~ueue 123. ~n the rare occurrence that the microproce~sor S2 wraps the 10 : se~uence number ~rom 2*32-1 to 0, the microprocessor 52 : will adjust the sequence numbers present in all of the ~: ~alid n~buf control structures in order to insure at least a monotonic a~cending order of sequence numbers at all times. The variabIes disl- number~ sector and 15~: sector count directly identify the disk and contiguous :sequence of logical di~k sectors that are to ultimately receive the cache write data. In ef$ect, the disk number identifies the DMA channel and data channel interface unit 86 that will be utiliæed for the eventual 2~0~ cache~data transfer to the disks 40. The state var}able a~ flag ~value ~es~entially ide~tifying whether the corresponding cache page contai~ ~alid data or is free for~reuse. Finally, the check sum variable i~ used ~o hold ~ a ~onventionally calculated checksum value 25: c~orresponding ~to the data within a specific, corresponding:in~tance of a nvbu~ control ~txucture.
~:, This checksum value is u ed to protect the data : integrity of it~ nv~uf control structure, particularly where there is a system failure while a specific nYbuf 3~ control structure is being modified~ In this case, only the write data associated with the n~buf control : : structure whose modification failed to complete, as : lndicated by a failed check~um compare, is deemed W0~3/18461 21 3 I 6 2 7 PCT/US~3/01911 corrupted. Thus, the loss of write data integrity is minimized to a specific cache page. Naturally, should a comparison of the stored and recalculated check sum ~alues for the n~cntl control structure 121 fail, then ~he integrity of the e~tire write cache is challenged.
In the preferred embodiment of the present in~ention, this challenge is resolved by simply purging and completely r initializing the NVRAM unit 60 and non-: : volatile cache memory 76.
~: ~ 10 The storage processor 50 m~intains the non-~olatile cache memory 76 in one of five states: un -ini tialized, off, dirty, o~, and bad-checksum. The default state of he non-~olatile cache memory 76 at power up i~ un-ini tialized. Consequently, there is no caching of data : 15 writes until a subsequent series of messages are eceived by the microprocessor 52 (initialize NVRAM and enable ~VRAM) directing the microproces~or 52 to place the ~on-volatile:cache ~emory 76 into an on state.
The cache controI messages that can be received by the microproce sor 52 for the management of the non-: ~olatile cache memory 76 parallel the commands is~uable at the system ~administrator user interface for c~ntrol o~ the ~on-volatile~ cache memory 76. The management messages in~.lude~Initialize NVRAM, Enable NVRAM, Disable : 25 ~NVRAM, Purge Data~a~d Flush Data. The Initialize NVRAM
message is normally i~sued durin~ the automatic sy~tem configuration operation following from the execution of rc.boot. 0~ receipt of ~he initialiæe ~VRAM message via the buffer 82, the~microproc~ssor 52 examines the value 3 0 of the magic variable in the nvcntl control structure : 121. If the ~alue is not set to the predetermined magic value, the nvcntl and array of nvbuf control structures 121, 122 are re-initialized or initialized for the first WO 93/184fil PCI`/US93~01911 . .

~6~l -2~-time, a log message may be issued through to the system console, and ~he state of the ~on-volatile cache memory : 76 is ~et to off. If the magic value is proper, the nvcntl control checksum is verified. If the verification fail~, all of the control struc~ures 121, 122 and any ~alid cache pages are considered corrupt.
The state of the non-volatile cache memory -76 is therefore set to bad-checksum. A purge of the control structures is required to change the state of the non-:~10 ~olatile cache memo~y 76 to off. If the checksum, however, is verified as correct, each of the nvbuf control structures are scanned to determine if there is :~ : an~ unwritten cache data present in the cache pages. If an nvbuf control structure is marked ~alid, but fails in a corresponding checksum ~erification, the integri~y of the non-~olatile memory is considered to have failed and the non-volatile cache memory 76 is placed in a bad-: ~ checksum state. However, for each ~alid a~d checksum verified nvbuf ~con~rol structure, the microprocessor 52 2~0 ~:will~then~erify the disk serial number i~ the n~cntl control structure 121 with the actua~ disk drive serial numbers as read from the disks. Where proper mat~hes are~found and~alid~ cache pages are present, the microprocessor~S2 t~hen initiates a write of the cache :25 page~data to a corresponding di k drive. If the disk erial :number ~verificatio~ or ~any cache page write ~:~ : attemptjfails,~ the non-volatile cache memory 76 is et to a dirty state. If the disk serial number verification failsl but no vaIid cache pages are ~present, the serial numbers ~in the nvcntl control : ~tructure are~updated (auto-reconfiguration). Where a cache page w~ite attempt failure;is correctable by the ~ ~system administrator, typically through the mapping of :~

W093/184~ 2 1 3 1 6 2 7 PCT/US93/0}~11 ;

a bad sector, a subsequent write operation, initiated in response to a system administrator user level flush command, and in ~urn Flush Data message, may then complete correctly. Once all valid cache pages successfully written to disk, the no~-volatile cache memory 76 is placed in an off state. Finally, the state of the non-volatile cache memory is returned in response to ~he initialize NVRAM message.
The Enable NVRAM message directs the microprocessor :
~;: 10 52 to enable the non-volatile cache memory write cache function and to set the state to on. This message is : only accepted when the current state of non-volatile :~ : cache memory 76 is of~.
:~ : The Disable NVRAM message dire-ts the microprocessor 52 to disable the non-volatile cache memory write cache ~unction. First all unwritten cache pages are ~lushed to the disks 40. Depending on whether the flu~h cache~write was entirely uccessful, the state of the non-~olatile cache memory is set ~o o~f 2~0~(successful) or~dirty (a flush write failed). Again, the~completion; status of the mes~age is returned in the message~reply to the di~able NVRAM message.
The Purge Data message direct~ the microproce~sor to purge all unwritte~ cache pages from the non-volatile cache memory 7~:and reinitialize all of the non-~olatile : ~ control 3tructuxes 12}, 122 and cache pages. The state o$ non-~olatile cache memory 75 is then set to off.
Finally, the~ Flush Data message forces a write o~
~ ~ , cache data from the non-volatile:cache ~emory 76 to the disk 40. This command will be accepted whenever the state of the non-volatile cache memory is dirty or on.
Consequently, the flush data mes~age is typica~ly used to flush out unwritten cache data after a disk media W~93/18461 PCT~US93/01911 ~3~6~ - 30-.

error, such as a bad sector, has been remapped. On a successful flush cache write, the state is changed to off. Otherwise the state remains set to dirty.
Three additional messages may be sen~ to the s~orage processor 50 to direct the actual transfer of : ~ :; data. The~e messages include a Write Data mes~age, a NV
Write Data message and a Read Data message. The Write Data message requests an uncached write of data to be transferred from~the;VME interface unit 62 directly to the disks 40.:: This~ message, like all messages, is actually transferred to~the buffer 82 for processing by : the microprocesso~ 52. In turn, a corresponding entry is made in the active:queue 104; this sequence of steps is logically ~hown by the line lO8. In the preferred lS~ embodiment of~the~present invention, the active queue 104~operates as~an~elev~or queue sorted by first by DM~
channel and:then ~ector. That is, the ins~rtio~ of the uncached:write ~reguests into the active gueue 104 is pèrform~d to;~maintain an order first based on 20~ ;disk ~umber,~:~pe ~ ted as neces~sary~ by the number of paràllel ~SCSI ~channels used for;;disk dri~es, a~d then ;;based on~ sector.~ ~Thus~, the active queue 104 effectively operates~as~multiple~ ele~ator queues, rather than a~ a single~or~even~multiple~first-in,:first-out queues, with~
`25~ ea~h~ o~ ~the~ elevator ~ueues~:~gueuing data trans~er requèsts for~a:~corresponding DM~channel.
Each entry~ in the active queue 104 contains sufficisnt info ~ tion:~to~construct:~a correspo~ding D~A
channel control structure that can then be transferred, 30:~ generally ~as ~indicated~ by the~ lin~ 112, ~y the microproce sor~52~to the DMA:channel buffer llO.
On~ receip~ of a DMA transfer done ~interrupt corresponding:to an uncached write operation present in , :

,: : :

W093/1 ~ ~ 2 I 31 6 2 7 PCT/US93/0191l ~he active queue 10~, the microprocessor 52 initiates the return of an acknowledgement reply message, deletes the uncached write entry from the active ~ueue 104 and, finally, promotes the next sequential active queue request for the freed DM~ channel to the DMA channel buffer 110.
:~ The Read Data mes~age is treated in a manner substantially similar to that of the uncached Write Data mes~age. The Read Da~a me3sage results in an entry being made in the appropriate elevator queue of the active queue 104, generally as indicated by the line 106. Eventually, the read data is obtained through the : operatio~ of the DM~ processor 114 and simultaneously stored via t;le VME interface unit 62 i~ main memory 34.
A~reply to the Read Data message is then issued by the : microproce~sor 52. The entry in the active gueue corresponding to the read data me~sage is then deleted.
:: Finally, the NV Write Data message i8 treated by the microprocessor 52 as a dire~tive to perfonm a data 0 : write operation to ~he ~on-volatile cache memory 76. An entry corresponding to the NV write message i~ made into the delayed write queue 123, generally as indicated by the line 120~ ~Depending on whether cache pages i~ the :: non-~volatile cache memory 76 are immediately available, 25:: an entry is made into the DM~ waiting queue 126 ~cache : pa~es available~,: as generally as indicated by the cache . write line 124, or into the buffer waiting queue 130 ; (all cache pages ~presently ~in use), generally as ind:icated by the line 128.
:~ ~ 30 The entry in the delayed write queue 123 is ubstantially eguivalent to an uncached write entry.
The entry in the DMA waiting ~ueue 126 for a cache write :~ contains information sufficient to construct a DM~
:, ~3~6~ - 32-channel control structure for the DMA channel buffer 110. The microprocessor 52 operates the DMA waiting ~ueue 126 as a first-in, first-out queue in selecting an entry for promotion to the DMA channel buffer llO. On completion of the DM~ transfer from the VME interface unit 62 to the no~-~olatile cache memory 76l the entry in the DMA waiting queue 126 is delated, the ; : corresponding entries in the delayed write ~ueue 123 and ~YbUf control structure 122 are updated to reflect : lO the identity and ~alidity of the cache pages holding the : write data. An acknowledgement reply message can then be issued.
In general, whe~ever the active queue 104 is empty, the~ microprocessor 52 examines the delayed write queue 15~:: l23 to ~elect an entry for promotion to the active queue 0:4. In general the entry defining the largest single data transfer :that~ca~ be made from the non-volatile cache~ memory ~76~:to the disks 40 is selected for promotion to the~active queue 104. Once this data is 20~ t~ransferred to:the~disks successfully, the corresponding caahe~pages:are marked as free by the microprocessor 52.
The~:bu~fer waiting:~queue 130 i8 then examined and the o1:dest ~ueue ~entry, :if any,: i~s promoted to the DM~
waiting queue;:126.: The correspondi~g entry in the 25~buf:fer waiting~ ueue 130 is deleted, while the ~ewly made entry in thé:DMA waiting gueue 126, is procesYed as though directly provided to the DMA walting ~ueue 126 as described a~ove.
^; Finally, a number of messages can be provided to 30~ the microprocessor 52 to obtain information regarding the state, condition, and performance of the non-volatile cache memory 76. A first message, Read Dirty ist or State message, obtains a reply message from the ~ :

WO93/1~61 PCT/US93/01911 ,..~.~

micxoprocessor 52 giving the current state of the non-volatile cache memory 76 or a list of dirty cache pages, including ~he sector, sector count, disk number and ~ta~e of each dirty cache page. Selection between a state or dirty page list reply is established by a flag byte provided in the Read Dirty ~ist or State message as ~ provided to the microprocessor 52.
:~ ~ NVRAM Status message, preferably issued by a host proces~or 32 executed status monitor daemon (statd) on lO ~ a periodic basis, directs the microprocessor 52 to p~epare a~reply ~essage specifying the number of cache pages used, number of messages waiting for cache, and num~er of messages waiting for DM~ hardware. Also returned i8 the average waiting ~ime for write data l5~ managed through the delayed write queue 123 as well as ; the total number of non-volatile cache memory writes, :coale3ced writes, and exact and inexac~ overlapping : writes handled through the non-volatile cache memory c~.
A~number~ of ~write cache protection~ as well as ~performance optimizations are performed by the mi~roprocessor ~52~in the management of all read and write~:data messages ~and data :transfers through the torage~processor~:~50. Since the ordering of successive read~a~d~ ~ ite~operatio~s mus~ be maintained, a sequence 25;;~number i as8igned~by:the microprocessor 52 to all cache write messages~in~the ordPr received and the orderiny o~
:read a~d uncached write messages is checked and managed on receipt.
In the case of a NV Write Data me~sage, the microprocessor;52 enters the me5sage in the delayed :
write queue 123 as soon as lt is received and :independent of the timing of the actual writ:e data ransfer to cache pages. This allows the microprocessor WO93~18461 PCT/US93/01911 213~6~

52 to immediately scan for and identify subsequent read : and write messages with overlapping sector addresses.
~ acknowledgment reply messàge is issued by the microprocessor 52 as soon as the data defined by an NV
Wri~e Data message h~ been transferred to cache pages - in the non-volatile cache memory 76. Thus, it is highly likely that another o~erlapping or contiguous write data block will be received, transferred to the non-volatile cache memory 76 and ac~nowledged before the overlapped or prior contiguous: da~a has been written out to the disks 40 Thus, there is an opportunity to optimize the : subsequent disk write operation by coalescing these two or more o~erlapping or continuous data blocks. In the instance of a Read Data me~sace, it is necessary for the :: 15 microproces~or 52 to scan the delayed write queue 123 immediately on receipt to determine if any overlapping NV Write Data message, either aw iting data or having data fully present in:the non-volatile cache memory 76, : ; are present. In accordance with the present in~ention, an NV Write Data message overlapped by a subsequent Read Data message is:immed:iately promoted to he active queue 104. The Read Data me~sage is also inserted in the .
active queue lO4, though behi~d the promoted NV Write ::: Data message s~ as to execute s~b~e~uently. If the : 25 ~equence order:of such a write and read seguence i9 ~ot so respectedi the read operation may properly complete, though r~turning improper data.
NV Write Data messages, in the preferred embodiment ~of the present invention, are limited to single 8K byte :~ 30 data transfers: that are aligned on 8K page boundaries the default size and character of NFS data write requests. NV Write Data messages that do ~ot conform to these criteria are treated as uncached Write Data WO93~1846~ 2 13 1 6 2 7 PCT/US93/~19ll .

messages in the preferred embodiment of the present invention. While these restrictions could be alleviated through the use of a dynamic storage allocation algorithm for controlling the placement of cache pages within the non-volatlle cache memory 76, the need fox such additional complexities is not genarally to be found neces~ary where NFS server use is the predominate operation.
For NV Write Data message~ that con$orm to the above criteria, a cache page is allocated in the non-, ~ola~ile cache memory 76. If a cache page cannot bealloca~ed, the message is effectively inserted into the buffer walti~g queue 130. Every time that the microprocessor 52 frees a cache page, the buffer w~iting queue 130 is scanned. When a cache page bec~mes a~ailable, either upon initial rereipt of an NV Write Data meB~age or after a cache paye has been freed, the ; NV Write Data me~sage is i~3erted ~o the DM~ waiting queu~ 126. If the DM~ waiting gueue 126 is otherwise empty, the DMA channel for controlling NV cache memory wrl~es is immedia~ely aYailable~ The DMA control , structur is th~n built in the DM~ channel buffer :~ ~ control structure for non-~olatile cache memory wri~es.
he DM~ wai~ing qu2ue iR reevaluated by the : :25 microprocessor ~2 each time the DM~ processor 114 i~ues ~; ~ a DMA transfer done interrupt corresponding to the non-j ~ volatile cache memory DM~ channel.
: When all data for an NV Write Data mes~age has been transferred to cache pages, the corresponding instances : 30 of the nvbuf control structures is updated for the corresponding cache pages. An acknowledgement reply message is then issued ~y the microprocessor 52. This insures that the write data is properly and fully stored W093~18461 PCT/US93/01911 3~62~ ' within the non-volatile cache memory and thereby protected i~ the event of a system shutdowrl prior to a further transfer to the disks 40.
In a~cordance with the present invention, a flag is associated with each of the entries in the delayed write queue 123 to signify whether the data corresponding to each of the messages has been fully fetched to cache pages. This flag must be updated prior to the corresponding write acknowledgement reply messa~e bei~g ::~ 10 i~ued by the microprocessor ~2 if synchronous wxites are to be correctIy handled. That is, the flag is updated in connection with completion of the data transfer request~pending at the top of queue in the DMA
waiting queue 126. Completion is signified by issuance 15~ of the DM~ tra~fer done interrupt corresponding to the no~-volatile cache memory DM~ channel. The entry in the ~:DM~ waiti~g queue l26 corresponding to the NV Write Data :me~ age i~ deleted f~llowing update of the nvbuf control ; structure 122 and prior to the is~uance of the cache 2~0 ~write a~cknowledge reply message by the microprocessor 52.:: Spe~îfically~, the acknowledgemen~ reply message, like all reply~més~sages where a non-~rivial reply is to be~ sent, is~ first formed in the buffer 82. The :micr~processor:52:then constructs a DM~ ~hannel control ::25 structure in~the DMA channel ~uffer llO for the DM~
buffer 82 to VME~interface unit 62 DM~ transfer channel.
For trivial replies, a null reply message descriptor is ~; written to the originating proceRsor' s command FIF0 via :`:the command register 70. In both cases, the : : 30 microprocessor 52 is then free to return to processing ; the entxies in the delayed write queue 123.
: : Continuous data block writes are coalesced 80 as to e serviced by a single disk request spa~ning multiple :

WO93/18461 ~ 31 6 2 7 PCT/US93/01911 cache pages. Repeated data blocks are identified and the earlier sequence numbered instances are deleted.
That is, the delayed write queue 123 is searched to ~ determine fir3t whether there are overlapping write : 5 requests. When a~ overlapping write is found, ~he overl~p is determined to be either exact or inexact.
The exact overlap mea~s that two write data reque3ts :~ : exactly reference (address) the same sectors of the same ;: ~ disk. In such case, the first write request is simply : lQ discarded and the corresponding cache pages are freed.
In order to insure that the subsequent data write can ul~imately be made ~o disk, the earlier data write cache : page is not:fre~d un~il the ~ubsequent data write block ha~ fetch completed and is present in another cache 15~ ~ page, and then only if transfer of the prior data block to a ~ata chaDnel interface unit 86 has not yet ~tarted.
An inexact overlap means that two wri~es overlap ` o~ly by a port~ion of their:addressed sectors. In the pre~erred emb~dimen~ of the present in~ention, the first 2~0;~ write is immediate1y promoted ~rom the delayed write ueue 123 to the: actiYe queue lO4. This immediate move i~sures that the :first write will reslllt in a data trans~er to di~k:~efore the s~ ~equent o~erlapping write is~tran~ferred to~di k. In the event that the system is 2S:~ r~started prior to the writing of at least the first inexact write to disk, there will be t~o cache pages identified in $he nvbuf control structuras 122. Both the active ~ueue~and delayed write queue lO4/ 723 will be lost as ~a consequence of the sys~em restart~
Howe~er, the sequence number variable present in each of the ~vbuf control ~tructure can be utilized by the microprocessor 52 in the reconstruction of the del~yed ~f `` ~ :
~ write queue 123 and the identification of the earlier of : ~:

::

WO93/1~61 PCT/US93J01911 2 13 ~ 62~ -38-the two o~erlapping cache pages. An entry corresponding to the earlier of the two overlapping cached pa~es can be reconstructed directly in the active queue 104-while a message entry for the subsequent cache page is entered in the delay write queue l23.
When two message entries are present in the active queue 104 with s~me di~k and overlapping sector addresses, they are erved in a first-in, first-out order. ~onsequently, the two messages will be served in ; lO their proper order and thereby preserve disk data integrity.
Whenever the microprocessor 52 detects that the active queue 10~4 i9~ empty, the delayed wri~e queue 123 is searched to f~ind the NV Write Data message that 15~defi~es a write data transfer to the largest number of sectors. This algorithm results in gi~ing ~he highest priority to coalesced writes and to freeing the largest ; number of cache pages as 800n as possible. This al~orithm also~recosnize~ that, particularly in the UNIX
file~ ystem, the writing of data blocks is typi~ally sequential while the writing of inodes and indirect blocks are typically repetitive~. The ~equential, coalesced writes, representing large blocks of onsecutive sectors will be written out to disk at 25~ high~r priority.~ ~Conversely, the smaller ~ector write blocks will have a s~b~ta~tially higher proportion of inode and indirect block writes and, therefore, be s~b~ect to a higher rate of discard becallse of ; repetitive writes~. Further, as is also typical with the UNIX file sy tem,~ata writes typically cluæter, thereby ~; resulting in a greater likelihood of data coaléscing , where the active queue 104 remains not empty for greater ~ ~ durations of ti~e. Thus, the delayed write control : ~: : :

W~g3/18461 21 3f PCTSUS93/01911 algorithms of the present invention tend to remain optimized under nearly all loading conditions.
Another circumstance that must be considered is where a NV Wri~e Data message has been promoted to the active queue 104 and is ready for processing by the DM~
processor 114 prior to completion of the data fetch into the cache pages. In this instance, ~he microprocessor 52 will hold off enabling that the corresponding SCSI
data channel interface unit 86 u~til the data fetch has .
completed. That is, the SCSI data transfer will be ini~iated by the microprocessor 52 in response ~o receipt of the DMA transfer done interrupt for corresponding to this SCSI channel.
Another circumstance that must be handled involves lS the utilization of mirrored write data disk drives. In accordance with the preferred embodiment of the present invention, a mirrored write generates, through the opexation of the UFS and device driver, two parallel di~k write me~sages. Two separate cache pages are 2~0 therefore allocated and separately fetched. The further processing necessary to manage the presence of ~ultiple non-~olatile write~messages in the delayed write ~ueue 123 is independe~t of the fact that two di~k writes are directed to ~mirrored drives. Thus, no special ~ ~ 25 instruc~ions are~ needed to manage such cache pages, sin~e each cache page is written to a respecti~e disk.
A fi~al circumstance in~olving NV Write Data me~sages occurs when there is a disk or media failure resulting in a NV Write Data message being held in the delayed write queue ~23, or even active queue 104, and the corresponding cache pages persisting in ~on-volatile cache memory 76. Since the cache page data write to disk failed, the cache pages are not freed and the data :

WO93/18~61 PCT/US93/01911 2~3~62~

is preserved. However, the corresponding n~buf control structures 122 are marked as dirty and subsequent cache function is disabled by the microprocessor- 52.
Preferably, a error log message is is~ued by the microprocessor 52 throu~h to ultimately the system console to i~form the system administrator. This allows manual intervention to attempt a fix of the disk hardware or media failure. If the failure ca~ be fixed, a flu3h me sage can be directed to the microprocessor 52 by th~ sy~tem administrator resulting potentially in a proper write of the cache page data to disk and a resetting of the status from dirty to off. The system admini~trator can ~hen is~ue an NVRAM enable signal to cause a resumption of NV write messages.
A further complication is where an NV write and o~erlapping read have been properly m~ved to the active queue 104, but~ the write data transfer fails. The corresponding entry in~the nv~uf control structures 122 is,~as before~marked dirty. In accordance with the 20~ preferred embodiment of the present in~ention, the o~erlapping read~request has a media error reply message return~by the microprocessor 52 and the dirty NV Write Data message i~ mo~ed~ack to the delay write queue 123.
:: :: mus:~ subsequent requests to the same disk ~nd DM~
25;~ channel ~ay be~processed through the active queue 104 while the~dirty NV Write Data request is maintained in ~; ~ the delayed write queue 123 $or comparison against s ~ se~uent reads and writes. A subsequent write request overlapping a dirty NV Write Data request already present in the delayed write queue 123 is not processed through the non-volatile cache memory 76 as the caching functionality was disabled on the o~currence of a dirty write. Rather, he request is allowed to process W093/1~61 21 31 6 2 7 PCT/US93~01911 through the active queue 104 in an attempt to write to the disk. Regardless of whether this subsequent request is successful, the dirty write request in the delayed write queue 123 is immediately removed as obsolete. If there is a failure, a m~dia error reply message is returned by the microprocessor 52 and the subsequent request is de~etedO Where there is an overlapping read following a dirty NV write, a media error reply message is immediately returned by the microproce ~or 52. Thus, ~: 10 the delayed wri~e queue 123 continues to be used in mana~ing a~d maintaining data integrity in the presence of a media or hardware error. That is, subsequent reads :and write~ overlapping a dirty NV write block will be identified by the microprocessor 52 through a search of the delayed write queue 123 with appropriate action being taken to maintain the cached data unless and until invalidated by sub~equent o~erlapping write or a purging of the non-volatile cache memory 76 under the direction ; of~ th2 system administrator.
:20 Thu~, a comprehensive high performance write cache ac~eleration system has been described. This system pro~ides the capability of providing write caching for an array of high performance data channels, resulting in : a substantially increased poten~ial c~mplexity yet :
25~ maintaining a~ e ficient design and optimum performance under widely-varying data transfer loads.
Na~urally, m~ny modificatio~ and variations of the present invention are possible in light of the abo~e teaching it is therefore to be understood that, wîthin the scope of the appended claims, as issued, the in~ention may be practiced otherwi~e than is : specifically described above.

Claims (3)

Claims

1. A data storage system coupled to a host computer system for the transfer of data, said data storage system comprising:
a) a plurality of data storage devices;
b) a plurality of data transfer channels, each said data storage channel being coupled to a respective one of said data storage devices, each data transfer channel including a data buffer and means for autonomously controlling the transfer of data between said data buffer and said respective one of said data storage devices;
c) memory means for non-volatile random access storage of data;
d) interface mean for transferring data;
e) data means for providing selective data transfer coupling between said data transfer channels, said memory means, and said interface means; and f) control means, coupled to said data means, for directing the transfer of data between said data transfer channels, said memory means, and said interface means, said control means including direct memory access control means, responsive to said control means, for burst transferring data between said interface means and said data buffers, between said interface means and said memory means and between said memory means and said data buffers.

2. The data storage system of Claim 1 wherein said memory means stores cache pages of data destined for storage by respective ones of said data storage devices, said data storage system further comprising control memory means for non-volatile random access storage of control data defining the status and destination of each cache page of data.

3. The data storage system of Claim 2 wherein the status of a cache page of data includes free and in-use, wherein said control means selectively directs the transfer of data from said interface to a predetermined first cache page, and wherein said control means modifies the control data defining the status of a predetermined second cache page.