WO1996029818A1

WO1996029818A1 - Progressive transmission of images

Info

Publication number: WO1996029818A1
Application number: PCT/GB1996/000623
Authority: WO
Inventors: Anil Bharath; Richard Kitney
Original assignee: Imperial College Of Science, Technology & Medicine
Priority date: 1995-03-17
Filing date: 1996-03-15
Publication date: 1996-09-26
Also published as: GB9505469D0; AU5012296A

Abstract

A method for the progressive transmission of images from a server to a workstation comprises encoding the image in multi-dimensional format, using the orthogonal wavelet transform, and storing the encoded image as a digital file on the server. A user requesting the image at a workstation sees, first of all, a small thumbnail image. If further detail is required, a request is made to the server which sends a 'detail' image consisting only of information needed to upgrade the low-resolution image to a higher-resolution image. This approach saves transmitting across the network information that has already been sent in connection with the original low-resolution image. The system may be of particular use in the hospital environment, for the transmission of medical images.

Description

PROGRESSIVE TRANSMISSION OF IMAGES

The invention relates to progressive transmission of images, and in particular although net exclusively to the progressive transmission of medical images across a computer network from a server to a workstation.

There is often a need, within the hospital environment, for physicians and surgeons to have rapid access to a variety of medical images such as, for example, x-ray images or CAT scan images. Typically, the images may be stored in digital form on a central server so that they may be called up as required by the user on a local workstation or display client.

The problem with such an arrangement is that the transmission of images, particularly detailed medical images, across a network puts a great strain on the available bandwidth. In a realistic hospital environment, the time taken for transmission of one detailed image is likely to be high because of the competition for bandwidth arising from other users of the system who may be attempting to view different images at the same time on other workstations. A further difficulty is that the user may wish to see only one of a series of perhaps rather similar images, and it is of course extremely inefficient to have to transmit each of those images, one by one, in full detail before the user is able to choose the image he or she actually requires.

One method of dealing with this problem is to present the user with a set of small " thumbnail" images, each of which is a low-resolution version of one of the images of the set the user is interested in. The user selects the required image using the displayed thumbnail images, the required image then being sent in higher- resolution form from the server to the workstation. There are two ways in which the image information may be updated from a low-resolution thumbnail image to a high-resolution display of the same image. The first way is to retransmit the full content of the high-resolution image. This represents a wastage of bandwidth, since some of the image information is already present at the viewing console. The second way in which the display may be updated is to transmit only the difference between the current image resolution and the desired high-resolution image from the server to the workstation. In the case of a wide area distributed viewing system, the retransmission of full resolution images represents a potential problem: for an individual image the retransmission of the full resolution image may not be significant; for the case in which there are many active viewing consoles, and a steady traffic of viewing and selection operations, the retransmission of the entire image may represent a significant bandwidth requirement over the transmission of the detail only.

It is an object of the present invention at least to alleviate the problems of the prior art.

It is a further object to provide an efficient progressive transmission system having an improved (smaller) bandwidth requirement.

It is a further object to provide a system in which progressive transmission may be applied to any selected portion of an image that the user wishes to see at higher-resolution.

According to a first aspect of the present invention there is provided a method of progressively transmitting images across a computer network from a server to a workstation, the method comprising.

(a) storing an image on the server as a digital file, the file comprising a low-resolution portion representative of a low-resolution version of the image, and a detail portion representative of detail in the image not contained within the low-resolution version;

(b) transmitting first information from the low- resolution portion to the workstation, and displaying the low-resolution version of the image at the workstation;

(c) selecting from the displayed image a user- defined area to be viewed at higher resolution;

(d) transmitting from the server second information from that part of the detail portion of the file which relates to the user-defined area;

(e) combining at the workstation the second information and that part of the first information which relates to the user-defined area to create a higher-resolution version of the user-defined area; and

(f) displaying at the workstation the higher- resolution version of the user-defined area.

The method of the present invention is particularly although not exclusively of use in the hospital environment, for the transmission of medical images such as echocardiographic images. There is often a requirements in a hospital environment to display a selection of mosaic of images, which may typically be image slices of a 3D data set, or frames of an image sequence. The present invention allows the individual images making up the mosaic to comprise small low-resolution "thumbnail" images. A user who wishes to see more detail for one particular image, simply selects the image (for example by clicking on it using the mouse). Further detail is then sent from the server, this further detail being combined with the information already held at the workstation to produce a slightly higher resolution image. Yet further information may be requested from the server if the user needs to see an even higher resolution version of the image. In each case, the only information that needs to be transmitted across the network is the detail information, that is the difference between the current low-resolution version of the image already held at the workstation, and the required higher-resolution version.

In some embodiments, the information may be sent automatically from the server. In other embodiments, a specific request may be sent by the workstation to the server each time further information is required. This request may conveniently be user-generated.

The preferred encoding method is to use the orthogonal wavelet transform. This is a transform which is discrete in both the time domain and in the scale-space (wavelet) domain. The small extent of the functions in the spatial domain allows for rapid convolution. The orthogonal nature of the functions used in the preferred embodiment guarantees that the minimum amount of information needs to be sent across the network.

The user may select a desired area of the image which he or she wishes to see in more detail. A desired region could for example be picked out on the screen with a mouse, preferably by dragging a box over the area of interest. If only an area of the image is selected by the user, the server sends information only on that selected area across the network.

The invention extends to a computer network and/or system for operating the method of the present invention.

The computer network may comprise a local area network or a wide area network, connected by wires or by a wire-less link. The term "network" includes any means of remote communication between computers, and accordingly encompasses (without limitation) communication via the telephone system and/or satellites. Such means of remote connection may be useful where for example the server is in a different hospital, and perhaps even in a different country, from the workstation.

The invention may be carried into practice in a number of ways and one specific embodiment will now be described, by way of example, with reference to the accompanying drawings, in which:

Figure 1 is a block diagram showing, schematically, the operation of the discrete wavelet transform;

Figure 2 (a) shows the discrete wavelet transform decomposition process;

Figure 2 (b) shows the decimation process at successive scales;

Figure 3 shows the Daubechies 4, 12 and 20 scaling functions, evaluated at different respective scales;

Figure 4 illustrates the decimation process in graphical form; and

Figure 5 shows how the multi-resolution image may be stored.

The preferred embodiment of the present invention is a method and/or system for the progressive transmission of medical images from a remote file server to a local workstation. Each image is stored on the server in a multi-resolution format comprising a low-resolution representation which may be called up as a small "thumbnail" image en the workstation, and a plurality of "detail" representations. If the user wishes to see the image at lowest resolution, the thumbnail image alone is sent. If, having viewed the thumbnail image, the user wishes to receive a higher-resolution representation, the first "detail" representation is passed along the network to the workstation. At the workstation, the "detail" representation is added to the low-resolution image to provide the required higher-resolution image If yet further resolution is required, the user can call upon the next "detail" representation, stored on the server, which in a like manner is added to the information which has previously been passed to the workstation to provide an image at the next highest resolution. The process may be continued until the user has the image resolution he or she requires, or has an image of the maximum resolution which is available from the server.

The first stage in the process is, of course, to take an original image and to encode it in digital form in a digital data file which forms the multi-resolution representation that is actually stored on the server. In order to split the original image up into a low-resolution image plus a plurality of "detail" representations or images, the preferred embodiment uses the orthogonal wavelet transform.

The orthogonal wavelet transform is a discrete transform, similar to the Foulier transform but instead of using sin functions wavelet mother functions are used instead. Each wavelet mother function has a limited extent in wavelet space, which contrasts with the s in functions used for Foulier transforms which are of course infinite in extent.

The wavelet transform has the following general form:

The discrete wavelet transform, which is used here, is the same, except that the values of a and b are restricted to discrete values only. In the preferred embodiment, orthogonal wavelets are chosen, for example the Daubechies wavelets shown in Figure 3.

For any given signal, one can compute a series of weights for discrete values of a and b which, when multiplied by the basis function and scaled leads back to the representation of f. This can be done, in principle, without losses (other than those arising from rounding errors). The fact that the wavelets are orthogonal guarantees the lowest possible error if certain terms are thrown away, for example if the user wishes to see an image of only intermediate resolution.

The advantage of the present scheme is that the full set of coefficients or weights W(a,b) can be arranged into a multi-resolution representation of the original signal.

As an illustration, consider the one-dimensional example shown in Figure 4. If one takes f(n) to be a discrete signal to be approximated, one can obtain a first approximation by calculating f' where f'(n) is calculated by determining the mean of adjacent samples within f(n). For example, f'(0) is half the sum of f(0 ) and f(1). Likewise, f'(1) is half the sum of f(2) and f(3). F' (n) may therefore be considered as an approximation g(n) to f(n). One may also define a "detail" representation h(n) as the difference between f(n) and g(n).

The process may be repeated, decimating the sample each time, thereby creating a plurality of individual "detail" representations, each indicative of the difference between the current approximation and the last previous approximation.

To store the image in multi-resolution format, the final approximated signal is kept, along with all of the detail signals.

The process may be represented schematically at each stage by the block diagram shown in Figure 1. At each stage, an approximation signal g(n) and a detail signal h(n) is determined from the input signal. The approximation signal is then fed back into the output and a further approximation and a further detail signal obtained. At an appropriate stopping point, the final very low-resolution g(n) output is retained, along with ail of the detail outputs h(n).

The structure shown in Figure 1 can be used for multi-resolution orthogonal biorthogonal, or quadrature mirror filters (QMF) where each of the above cases is specified by the chosen filter coefficients.

The filters are simply denoted as low pass, g(n) and high pass, h(n), to present a general form, but in fact these filters are related to scaling function, ø(x), and wavelet function ψ(x) [2] . The decomposition process is illustrated in Figure 2.

Each block has both a low pass filter and a nigh pass filter. The output of each filter is decimated by a factor of two. The resolution of the output of the low pass filter is also changed because of losing nigh frequency detail.

The output of the filter chain, A₂ ^-mS, signal represents a discrete approximation of signal at the resolution 2^-m. The other output is called the detail signal, D₂ ^-mS, at the resolution 2^-m. If the original signal has N samples, then the total number of samples in the detail and approximation signals is also N. Thus the discrete wavelet representation has the same total number of samples as the original signal. The smoothness of A2^jS depends on the shape of the scaling function. In general we would wish to have a smooth scaling function, but this function should satisfy some conditions such as orthogonality, support width etc. One of the important things is the spatial support (width) of the scaling function which divides the wavelet transform into compact into compact and non-compact wavelets.

In compactly supported wavelets the scaling function and the wavelet function have mere regularity at the expense of a large number of non-zero coefficients of h(n), which results in a large support for ø ( x ) . This support width increases linearly with their regularity in some wavelets such as the so-called Daubechies wavelets (Daubechies, 1988). As shown in Figure 3, increasing the number of coefficients of h(n), leads to an increase in the smoothness of the scaling function. Note that the three traces of Figure 3 have been evaluated at resolution in inverse proportion to their support, so support. Because of the compact support and orthogonality the Daubechies wavelets provide a suitable basis set. The Mallat basis set is constructed from splines in continuous space, then discretized, and hence is not compact.

A multi-resolution decomposition is a method of representing signals at different scales of magnification. The concept of multi-resolution decomposition was initially developed by Mallat [2] to address the problem of characterising image scenes in a scale independent manner. The approach of Mallat is essentially a pyramidal decomposition, along the lines of Burt and Adelson, but uses values of h(n) and g(n) such that the decomposition is onto an orthogonal basis. The basic idea is that of successive approximation, together with adding detail signal from one approximation to the next. Assume that we have a ladder of spaces such that

With the property that if

then

If W. is the orthogonal complement of V. in V\._: so we have

Thus, W. contains the detail signal necessary to go from V₁ to V₁₊₁. If we have an approximation of a signal at a resolution corresponding to V₀, then a better approximation to the signal is obtained by adding the detail signal corresponding W₂. This detail signal is the projection of the signal in W₂.

By interpolating the output

of the decomposition depleted in Figure (2), one obtains an approximation

of the signal S(n) at a resolution 2^-1. The interpolation is defined by the following equation:

where

denotes insertion of a zero between each sample of the discrete signal

This interpolation is a convolution between the scaling function ø and

This particular interpolation represents an example in going from a representation of the signal at one scale to an approximation of the signal at the scale immediately above it. If we wish to go from a representation of the signal at, say, a scale of 2^-j to an approximation of the signal at some scale 2^-j+1 , where l>1, it is necessary to dilate the scaling function appropriately.

The appropriate dilation may be performed as follows: if the scaling function of Equation (1 ) is at the resolution 1(i.e. 2^-1), then it belongs to V₀ . Since

one may write

This equation may be generalised to relate the scaling function at a scale 2 to the scaling function at a resolution 2 ^j+1 for any j. Therefore,

For the sake of simplicity, the above examples have assumed a one -dimensional signal rather than a two- dimensional image, which of course is what we are really interested in. In dealing with images rather than signals, it is necessary to use the corresponding two- dimensional discrete wavelet transform (DWT). The two-dimensional transform may again be represented by a repeated function block where, instead of a pair of outputs, we have four outputs corresponding to the low-frequency image component and the high-frequency vertical, horizontal and diagonal components. The latter three image components correspond to the detail signals.

For details of the two-dimensional decomposition, refer to Mallat [1]. It is possible to generalise Equaticn (3) to the two-dimensional case:

(4) due to the separability of the two-dimensional decomposition.

Figure 5 shows, in schematic form, how the information may be stored in a data file on the server. The lowest resolution image that is expected to be required (A) is stored, along with the sequences of detail information D₁, D₂ and D₃. If, for example, the lowest resolution image A is an 8 × 8 image, one will automatically obtain from the algorithm three sets of detail signals each of which also has size 3 × 3. The next level up is 16 × 16, and so on. To produce an image of 16 × 16 resolution , the three detail signals D₁ are added into the low-resolution image A. Similarly, to create an image of 32 × 32 resolution, the detail images D₂ are added onto the previously-created 16 × 16 image.

It will of course be understood that it is not necessary for the data file to be stored exactly as it is shown in Figure 5. All that is required is that the low- resolution image can be selected from the file and sent across the network to the workstation followed, if requested by the user, by each of the detail images in order.

In looking at medical images, it is often useful to be able to view a particular region of interest in greater detail. The infrastructure of orthogonal, multiresolution image decomposition provides an efficient way of doing this, in terms of bandwidth requirements. Consider a region of interest, such as a heart valve on a cardiac image. By selecting an approximate region of interest on a low resolution, interpolated image, the present infrastructure allows one to view that region at full resolution without needing to increase the resolution outside of the specified region.

The user first selects an area of interest (for example using a mouse), and that area is then redrawn on the screen in greater detail. For example, if the user wishes to view in more detail only the top left-hand corner of the thumbnail image A, he or she simply selects that area and a signal is sent to the server instructing it to send further detail appropriate to that area only. With the file structure shown in Figure 5, the server would send only the detail information contained within the top left-hand corners of each of the three D₂ images. If the user then wishes to view that enlarged image in yet more detail, the server would send only the top left- hand corners of the D₂ images.

In order to refine the detail over a particular user-selected area of the image, we need to determine the range of influence of all samples (pixels) within the area we are interested in, over all scales of the decomposition process. This range of influence is clearly a function of the width of the filters used, that is the impulse response of the filters used in the decomposition. Since we are typically using filters of very small length, this range of influence is fairly localised throughout all space. However, proper reconstruction can be carried out only if we can correctly identify all of the samples, at each scale, which influence the desired image within the selected area .

A general implementation involves tracing all of the influenced children of a pixel throughout the various components of the image in the transform space.

Although tedious, this is quite within the capabilities of the skilled man. If orthogonal filters are used, as they are in the preferred embodiment, then the range of influence simply becomes the direct children (4) of each pixel.

This allows great savings in bandwidth. Consider the requirement to view a region of microcalcification in a digital x-ray mammogram at a resolution of 256×256 pixels. Mammograms typically have full pixel dimensions of 4096×4096 pixels. In such a situation, immediate enhancement of the region requires ail the coefficients to go in resolution from, say a 512×512 icon to a full resolution region is less than 1% of the total coefficients necessary to rebuild the entire image at 4096x4096 resolution.

The results of our study show that the reconstruction of an image from its full wavelet decomposition represents by selecting an appropriate set of scales represents a valid method of building a multiresolution framework for progressive transmission of echocardiographic images. We have shown that not only does this framework support the progressive updating of "thumbnail" images, but also that smooth interpolation of these small images by an appropriately dilated scaling function lead to low resolution images at magnifications (scales) comparable to the original image. Such blurred images may be used, for example, to select with precision a particular region of interest, which one wishes to view in greater detail (i.e. at full resolution).

The clear advantage of adopting this approach to the clinical viewing of images in a networked client/server environment, such as a teleconferencing or remote diagnosis application, is that communications bandwidth is used optimally. Where a large number of clients are involved, and commercial networks are employed, the savings can be significant. Finally, even on local area networks using ethernet technology, the cut in overheads can represent a significant saving in bandwidth.

References

[1] Daubechies, I., "Orthonormal basis of compactly supported wavelets", Communications on Pure and Appl ied Ma thematics , vol. 41, pp 909-996, 1988.

[2] Mallat, S., "A Theory for multiresoiution signal decomposition: The wavelet representation", IEEE Transactions on Pattern Analysis and Machine In tell igence, vol. 11, no. 7, pp 674-692, 1988.

[3] Burt, P. and Adelson, E., "The Laplacian Pyramid as a compact image code", IEEE Transactions on Communica tions , vol. 31, no. 4, April 1983, pp 532-540.

Claims

CLAIMS :

1. A method of progressively transmitting an image across a computer network from a server to a workstation, the method comprising:

a, storing an image on the server as a digital file, the file comprising a low-resolution portion representative of a low-resolution version of the image, and a detail portion representative of detail in the image not contained within the low-resolution version;

(e) combining at the workstation the second information and that part of the first information which relates to the user-defined area to create a higher-resolution version of the user-defined area; andm

( f ) displaying at the workstation the higher- resolution version of the user-defined area.

2. A method as claimed in Claim 1 in which the detail portion of the file comprises a plurality of detail sub- portions, successive sub-portions containing information on successively finer detail within the image.

3. A method as claimed in Claim 2 in which third information from a first sub-portion is transmitted from the server, the third information being combined at the workstation with the first information to create a first intermediate resolution version of the image.

4. A method as claimed in Claim 3 in which fourth information from a second sup-portion is transmitted from the server, the fourth information being combined at the workstation with the first intermediate resolution version of the image to produce a second intermediate resolution version, of higher resolution than the first intermediate resolution version.

5. A method as claimed in Claim 4 in which fifth information from a third sub-portion is transmitted from the server, the fifth information being combined at the workstation with the second intermediate resolution version of the image to produce a third intermediate resolution version, of higher resolution than the second intermediate resolution version.

6. A method as claimed in any one of the preceding claims in which the digital file is encoded using a transform having a basis function of finite extent.

7 A method as claimed in any one of Claims 1 to 5 in which the digital file is encoded using the wavelet transform.

8 A method as claimed in any one of Claims 1 to 5 in which the file is encoded using the orthogonal wavelet transform.

9 A method as claimed in Claim 3 in which the orthogonal wavelet transform uses a Daubechies' wavelet.

10. A method as claimed in any one of the preceding claims in which the combining at the workstation includes interpolation

11. A method as claimed in any one of the preceding claims including storing a plurality of images on the server, and displaying on the workstation a corresponding plurality of low-resolution versions of the images for a user to select.

12. A method as claimed in any one of the preceding claims in which information from the detail portion is transmitted from the server on receipt of a request from the workstation.

13. A method as claimed in Claim 12 in which the request from the workstation is user-selectable.