Choosing
a storage solution
When designing a digital
edit suite (taking EDIT2's ///FAST silver. as an example, the following
consideratons concerning storage are important -
Drive manufacturer
Not all hard drives are
suitable for NLE video streaming applications. Choose a drive which is
already proven on the market, and for which the manufacturer offers at
least 3 years warranty. We have used both IBM and Seagate drives (our Power
vaults are fitted with Seagate drives). 10.000 rpm is necessary for video
streaming applications. Choose as large a disk cache as possible - 4MB
seems to be a current (2001) standard. Because of their mechanical construction,
you are more likely to experience a drive failure than a failure to any
other system component. It's not a question of "if" but rather "when" -
therefore a quality manufacturer, supported by a quality integrator such
as Dell, may mean major economic advantages when a drive eventually fails.
There are other drive manufacturers, though we have, as yet, no experience
of their products for such applications. Our previous experience with IBM
on warranty replacement is excellent. We have not yet experienced failures
on Seagate drives.
Data capacity
Itis important to have enough
storage, for the type of productions and style of editing to be undertaken.
Longer programmes, or the need to log and digitise large amounts of raw
material, dictate greater storage requirements. Most NLE systems allow
for editing on the basis of large amounts of data in off-line (low-resolution)
quality, and subsequent re-digitisation of the actual material used to
a higher resolution for play-out. If a system must be able to handle large
amounts of material at the higher release resolution, this will require
more storage capacity. Finally, there is the issue of flexibility. Analogue
videotape suites can handle many projects in which editing sessions overlap
- swap the raw and master tapes, and a new project can take over. In a
non-linear digital suite, a project loads the system until it is completed
and its data is erased - therefore additional storage must be allowed if
the facility must be able to handle overlapping projects. Similarly, many
editors prefer to have an amount of stock material - stock shots, graphicvs,
sound effects, permanently available in the suite. Space must be allocated
for this.
Number of drives
Whilst modern media drives
with speeds up to and over 10.000 rpm are quick to deliver data, all drives
are interrupted - albeit for a short time - when a data request necessitates
moving the read/write head. To deliver satisfactory data transfer rates
for video applications, it is necessary to spread the data over more than
one drive (rule of thumb is at least three drives), so that data may be
read from one drive, whilst the other drives can reposition their heads.
The ideal number of drives will also depend on the chosen RAID level and
the desired data capacity.
Number of stripe sets
/ RAID arrays
As drive manufacturers continue
to release larger and cheaper drives, it becomes important to consider
the upper limits of a single RAID or stripe set's data capacity. Initially
one would suppose that a large RAID is better than two small ones, but
this view does not take account of the risk of data loss. Despite RAID
strategies, there is always the risk of losing data due to drive failure
or lost configuration. The more data on the RAID, the more data which can
be lost! Therefore the size of a single RAID should never be greater, than
the amount of data which the user is prepared to redigitise or recover.
With a 500GB Power vault containing low-resolution (off-line quality) video/audio
data, this could mean up to 200 reels of Betacam SP footage! In EDIT2 we
have set the limit at 500GB, further expansion will then be based on additional
RAIDs rather than larger RAIDs.
RAID level
RAID means a Redundant Array
of Inexpensive Disks - a method by which several disks can be combined
so as to appear to the workstation as a single disk.
The fastest and most effective
use of a group of disks occurs when they are striped as single drive (just
a bundle of disks) - and data is spread equally accross all drives. Depending
on the controller, between 8 and 15 drives may be spriped together in this
manner - which is (maybe erroneously) referred to as RAID0. The shortcoming
of RAID0 is that if one disk fails, all data on the entire stripe set is
lost irretrievably.
RAID0 has been the preferred
striping method for many NLE manufacturers, many of whom have believed
that video applications cannot handle RAID levels with a higher level of
redundancy.
There are many other RAID
levels, most of which are either too slow in operation, or demand to much
disk space, for video applications -
-
RAID0 - Up to 8 (sometimes 15)
drives striped as one, no redundancy, capacity equal to all eight drives
together.
-
RAID1 - 2 drives, one a mirror
of the other, 100% redundancy, capacity equal to a single drive.
-
RAID2 - Not used widely
-
RAID3 - 5-15 drives, data spread
over all drives minus one, the last drive contains parity data from which
data on any single lost drive may be reconstructed. high redundancy, undue
overworking of parity drive. Capacity equal to number of drives together
minus approx. one drive. (Not recommended!)
-
RAID4 - not used widely
-
RAID5 - 5-15 drives, data and
parity information spread over all drives, data from any single drive failure
can be restored. Capacity equal to all drives together minus approx. one
drive.
RAID arrays can also be combined,
for example 2 separate RAID0's where one set mirros the other in RAID1
- this is called RAID10. Likewise two RAID3's or RAID5's can be mirrored,
or several mirrorsets can be arrayed in RAID3 or RAID5. None of these strategies
are presently sutable for realtime video applications.
Given the current choices,
RAID5
is the best method for protecting data for video editing applications.
RAID controller
Unintelligent RAID controllers,
including some integrated (on-board) controllers, can usually only handle
separate SCSI devices up to 15 per SCSI channel, or a simple RAID0 stripe
set. RAID levels with redundancy demand intelligent RAID controllers.
We have previously used the
Dell PERC2/DC RAID controller, but have upgraded to a U160 device. EDIT2
uses the Dell PERC3/DC (Dual channel) RAID controller. This controller
has on-board 128MB cache RAM, which enables higher sustained data transfer
rates because the cache can hold data whilst the drives are busy writing.
At present 128MB is the maximum cache size for the PERC controllers. We
have tried 64MB but this gave poorer results. The PERC range are manufactured
by American Megatrends Inc. (AMI) Similar cache technology is also available
from other RAID controller manufacturers such as Adaptec.
A battery backup for the
cache RAM is important to reduce the possibility of data loss in the event
of power outages during write operations.
With an intelligent controller,
there is no perceptable difference in performance between RAID0 and RAID5,
likewise there is no difference between for example a single 8 drive array
or two 4 drives arrays on separate controller channels.
In RAID5 write operations
are marginally slower, as both data and parity information must be written
to the drives. In read operations, RAID0 and RAID5 have the same speed.
Write operations are generally free of problems, as the system only handles
one video stream and up to four audio streams at once - under half the
capability of the system under playback.
Generally it is important to
understand the real-time nature of video streaming applications. Whereas
most users - using office applications - cannot perceive the difference
in speed between say a Pentium II 500MHz platform and a PIII 2x1GHZ platform,
as delays in terms of microseconds are unnoticeable, video applications
demand data on-time. There are no performance grey areas - transfer rate
is either fast enough, or too slow.
In EDIT2, with ///FAST silver.
MPEG2 and the Dell Power vault in 8x73GB RAID5 configuration, the storage
capacity achieved for each Power vault may be calculated -
The following data from ///FAST
Multimedia is used to calculate minutes of programme material per GB of
effective storage -
NDQ
level |
Data rate
M/bit |
Comparable to |
Min
per GB |
| 5 |
5 |
Off-line |
26,50 |
| 10 |
10 |
VHS |
13,00 |
| 15 |
15 |
S-VHS |
8,50 |
| 25 |
25 |
DV, DVCPro (25) |
5,00 |
| 33 |
33 |
Betacam SP |
4,00 |
| 50 |
50 |
Digital Betacam, DVCPro50 |
2,50 |
The capacity of each power
vault is calculated to include a redendancy for parity data and a redundancy
for that capacity, beyond which performance may suffer - on RAID0 using
an non-intelligent controller, performance redundancy can be as hish as
20% - Using the PERC2 and PERC3 controllers, we have not registered any
noticeable reduction in performance when working on an almost fully loaded
raid. However, as a matter of caution, our capacity calculations include
a performance redundancy of 5%
| Capacity / drive (GB) |
72 |
| Number of drives |
8 |
| RAID redundancy % |
10 |
| Performance redundancy
% |
5 |
| Total redundancy % |
15 |
|
|
| Nominal capacity of RAID |
489,60 |
To calculate storage capacity
in relation to reels of videotape, we apply the following rule of thumb
(this is accurate for our acquisition format - Betacam SP and our typical
shooting ratios - other systems and production environments may need to
work with other figures). This gives an estimate of the average amount
of material per production reel, which we would digitise -
| Source reel |
Betacam
SP BSP30 |
| Nominal running time
(minutes) |
34.00 |
| Useable footage % |
90 |
| Useable r/t (minutes) |
30.60 |
On the basis of the above
figures, a single 8x72GB Power vault has the following storage capacity
-
| NDQ |
Minutes |
Hours |
Reels |
| 5 |
12,974.40 |
216.24 |
424 |
| 10 |
6,364.80 |
106.08 |
208 |
| 15 |
4,161.60 |
69.36 |
136 |
| 25 |
4,448.00 |
40.80 |
80 |
| 33 |
1,958.40 |
32.64 |
64 |
| 50 |
1,224.00 |
20.40 |
40 |
The total capacity of EDIT2
is roughly twice the above figures. |
