Abstract base class to combine multiple files in a single one. More...

#include <MultiFileBase.h>

Public Member Functions
	MultiFileBase (const String &name, Int blockSize, Bool useODirect)
	Open or create a MultiFileBase with the given name. More...

virtual	~MultiFileBase ()
	The destructor flushes and closes the file. More...

Int	fileId (const String &name, Bool throwExcp=True) const
	Return the file id of a file in the MultiFileBase object. More...

Int	addFile (const String &name)
	Add a file to the MultiFileBase object. More...

void	deleteFile (Int fileId)
	Delete a file. More...

Int64	read (Int fileId, void *buffer, Int64 size, Int64 offset)
	Read a block at the given offset. More...

Int64	write (Int fileId, const void *buffer, Int64 size, Int64 offset)
	Write a block at the given offset. More...

void	flush ()
	Flush the file by writing all dirty data and all header info. More...

void	resync ()
	Resync with another process by clearing the buffers and rereading the header. More...

virtual void	reopenRW ()=0
	Reopen the underlying file for read/write access. More...

virtual void	fsync ()=0
	Fsync the file (i.e., force the data to be physically written). More...

String	fileName () const
	Get the file name of the MultiFileBase. More...

Bool	isWritable () const
	Is the file writable? More...

Bool	useODirect () const
	Will O_DIRECT be used? More...

Int64	blockSize () const
	Get the block size used. More...

uInt	nfile () const
	Get the nr of virtual files. More...

Int64	size () const
	Get the total nr of data blocks used. More...

const vector< MultiFileInfo > &	info () const
	Get the info object (for test purposes mainly). More...

const vector< Int64 > &	freeBlocks () const
	Get the free blocks (for test purposes mainly). More...

Protected Member Functions
void	setNewFile ()
	Set the flags and blockSize for a new MultiFile/HDF5. More...

Protected Attributes
String	itsName

Int64	itsBlockSize

Int64	itsNrBlock

Int64	itsHdrCounter

vector< MultiFileInfo >	itsInfo

std::shared_ptr< MultiFileBuffer >	itsBuffer

Bool	itsUseODirect

Bool	itsWritable

Bool	itsChanged

vector< Int64 >	itsFreeBlocks

Private Member Functions
void	writeDirty (MultiFileInfo &info)

virtual void	doAddFile (MultiFileInfo &)=0
	Do the class-specific actions on adding a file. More...

virtual void	doDeleteFile (MultiFileInfo &)=0
	Do the class-specific actions on deleting a file. More...

virtual void	flushFile ()=0
	Flush the file itself. More...

virtual void	close ()=0
	Flush and close the file. More...

virtual void	writeHeader ()=0
	Write the header info. More...

virtual void	readHeader (Bool always=True)=0
	Read the header info. More...

virtual void	extend (MultiFileInfo &info, Int64 lastblk)=0
	Extend the virtual file to fit lastblk. More...

virtual void	writeBlock (MultiFileInfo &info, Int64 blknr, const void *buffer)=0
	Write a data block. More...

virtual void	readBlock (MultiFileInfo &info, Int64 blknr, void *buffer)=0
	Read a data block. More...

Detailed Description

Abstract base class to combine multiple files in a single one.

Intended use:

Public interface

Review Status

Test programs:: tMultiFile

Synopsis

This class is a container file holding multiple virtual files. It is primarily meant as a container file for the storage manager files of a table to reduce the number of files used (especially for Lustre) and to reduce the number of open files (especially when concatenating tables).
A secondary goal is offering the ability to use an IO buffer size that matches the file system well (large buffer size for e.g. ZFS).

The SetupNewTable constructor has a StorageOption argument to define if a MultiFile has to be used and if so, the buffer size to use. It is also possible to specify that through aipsrc variables.

A virtual file is spread over multiple (fixed size) data blocks in the MultiFile. A data block is never shared by multiple files. For each virtual file MultiFile keeps a MultiFileInfo object telling the file size and the blocks numbers used for the file. When flushing the MultiFile, this meta info is written into a header block and, if needed, continuation blocks. On open and resync, it is read back.

A virtual file is represented by an MFFileIO object, which is derived from ByteIO and as such part of the casacore IO framework. It makes it possible for applications to access a virtual file in the same way as a regular file.

It is possible to delete a virtual file. Its blocks will be added to the free block list (which is also stored in the meta info).

Example

In principle it is possible to use the MultiFile functions directly. However, in general it is much easier to use an MFFileIO object per virtual file as shown below.

// Create a new MultiFile using a block size of 1 MB.
MultiFile mfile("file.mf', ByteIO::New, 1048576);
// Create a virtual file in it.
MFFileIO mf1(mfile, "mf1", ByteIO::New);
// Use it (for example) as the sink of AipsIO.
AipsIO stream (&mf1);
// Write values.
stream << (Int)10;
stream << True;
// Seek to beginning of file and read data in.
stream.setpos (0);
Int vali;
Bool valb;
stream >> vali >> valb;

To Do

write headers at alternating file positions (for robustness)
possibly write headers entirely at the end if larger than blocksize

Definition at line 164 of file MultiFileBase.h.

Constructor & Destructor Documentation

◆ MultiFileBase()

casacore::MultiFileBase::MultiFileBase	(	const String &	name,
		Int	blockSize,
		Bool	useODirect
	)

Open or create a MultiFileBase with the given name.

Upon creation the block size can be given. If 0, it uses the block size of the file system the file is on. If useODIrect=True, it means that O_DIRECT is used. If the OS does not support it, the flag will always be False. If True, the data buffers will have a proper alignment and size (as needed by O_DIRECT).

◆ ~MultiFileBase()

virtual casacore::MultiFileBase::~MultiFileBase ( )

virtual

The destructor flushes and closes the file.

Member Function Documentation

◆ addFile()

Int casacore::MultiFileBase::addFile ( const String & name )

Add a file to the MultiFileBase object.

It returns the file id. Only the base name of the given file name is used. In this way the MultiFileBase container file can be moved.

◆ blockSize()

Int64 casacore::MultiFileBase::blockSize ( ) const

inline

Get the block size used.

Definition at line 226 of file MultiFileBase.h.

References itsBlockSize.

◆ close()

virtual void casacore::MultiFileBase::close ( )

privatepure virtual

Flush and close the file.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ deleteFile()

void casacore::MultiFileBase::deleteFile ( Int fileId )

Delete a file.

It adds its blocks to the free block list.

◆ doAddFile()

virtual void casacore::MultiFileBase::doAddFile ( MultiFileInfo & )

privatepure virtual

Do the class-specific actions on adding a file.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ doDeleteFile()

virtual void casacore::MultiFileBase::doDeleteFile ( MultiFileInfo & )

privatepure virtual

Do the class-specific actions on deleting a file.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ extend()

virtual void casacore::MultiFileBase::extend	(	MultiFileInfo &	info,
		Int64	lastblk
	)

privatepure virtual

Extend the virtual file to fit lastblk.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ fileId()

Int casacore::MultiFileBase::fileId	(	const String &	name,
		Bool	throwExcp = `True`
	)		const

Return the file id of a file in the MultiFileBase object.

If the name is unknown, an exception is thrown if throwExcp is set. Otherwise it returns -1.

◆ fileName()

String casacore::MultiFileBase::fileName ( ) const

inline

Get the file name of the MultiFileBase.

Definition at line 214 of file MultiFileBase.h.

References itsName.

◆ flush()

void casacore::MultiFileBase::flush ( )

Flush the file by writing all dirty data and all header info.

◆ flushFile()

virtual void casacore::MultiFileBase::flushFile ( )

privatepure virtual

Flush the file itself.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ freeBlocks()

const vector<Int64>& casacore::MultiFileBase::freeBlocks ( ) const

inline

Get the free blocks (for test purposes mainly).

Definition at line 241 of file MultiFileBase.h.

References itsFreeBlocks.

◆ fsync()

virtual void casacore::MultiFileBase::fsync ( )

pure virtual

Fsync the file (i.e., force the data to be physically written).

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ info()

const vector<MultiFileInfo>& casacore::MultiFileBase::info ( ) const

inline

Get the info object (for test purposes mainly).

Definition at line 237 of file MultiFileBase.h.

References itsInfo.

Referenced by writeDirty().

◆ isWritable()

Bool casacore::MultiFileBase::isWritable ( ) const

inline

Is the file writable?

Definition at line 218 of file MultiFileBase.h.

References itsWritable.

◆ nfile()

uInt casacore::MultiFileBase::nfile ( ) const

Get the nr of virtual files.

◆ read()

Int64 casacore::MultiFileBase::read	(	Int	fileId,
		void *	buffer,
		Int64	size,
		Int64	offset
	)

Read a block at the given offset.

It returns the actual size read.

◆ readBlock()

virtual void casacore::MultiFileBase::readBlock	(	MultiFileInfo &	info,
		Int64	blknr,
		void *	buffer
	)

privatepure virtual

Read a data block.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ readHeader()

virtual void casacore::MultiFileBase::readHeader ( Bool always = True )

privatepure virtual

Read the header info.

If always==False, the info is only read if the header counter has changed.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ reopenRW()

virtual void casacore::MultiFileBase::reopenRW ( )

pure virtual

Reopen the underlying file for read/write access.

Nothing will be done if the file is writable already. Otherwise it will be reopened and an exception will be thrown if it is not possible to reopen it for read/write access.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

◆ resync()

void casacore::MultiFileBase::resync ( )

Resync with another process by clearing the buffers and rereading the header.

The header is only read if its counter has changed.

◆ setNewFile()

void casacore::MultiFileBase::setNewFile ( )

protected

Set the flags and blockSize for a new MultiFile/HDF5.

◆ size()

Int64 casacore::MultiFileBase::size ( ) const

inline

Get the total nr of data blocks used.

Definition at line 233 of file MultiFileBase.h.

References itsNrBlock.

◆ useODirect()

Bool casacore::MultiFileBase::useODirect ( ) const

inline

Will O_DIRECT be used?

Definition at line 222 of file MultiFileBase.h.

References itsUseODirect.

◆ write()

Int64 casacore::MultiFileBase::write	(	Int	fileId,
		const void *	buffer,
		Int64	size,
		Int64	offset
	)

Write a block at the given offset.

It returns the actual size written.

◆ writeBlock()

virtual void casacore::MultiFileBase::writeBlock	(	MultiFileInfo &	info,
		Int64	blknr,
		const void *	buffer
	)

privatepure virtual

Write a data block.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

Referenced by writeDirty().

◆ writeDirty()

void casacore::MultiFileBase::writeDirty ( MultiFileInfo & info )

inlineprivate

Definition at line 245 of file MultiFileBase.h.

References casacore::False, info(), and writeBlock().

◆ writeHeader()

virtual void casacore::MultiFileBase::writeHeader ( )

privatepure virtual

Write the header info.

Implemented in casacore::MultiHDF5, and casacore::MultiFile.

Member Data Documentation

◆ itsBlockSize

Int64 casacore::MultiFileBase::itsBlockSize

protected

Definition at line 279 of file MultiFileBase.h.

Referenced by blockSize().

◆ itsBuffer

std::shared_ptr<MultiFileBuffer> casacore::MultiFileBase::itsBuffer

protected

Definition at line 283 of file MultiFileBase.h.

◆ itsChanged

Bool casacore::MultiFileBase::itsChanged

protected

Definition at line 286 of file MultiFileBase.h.

◆ itsFreeBlocks

vector<Int64> casacore::MultiFileBase::itsFreeBlocks

protected

Definition at line 287 of file MultiFileBase.h.

Referenced by freeBlocks().

◆ itsHdrCounter

Int64 casacore::MultiFileBase::itsHdrCounter

protected

Definition at line 281 of file MultiFileBase.h.

◆ itsInfo

vector<MultiFileInfo> casacore::MultiFileBase::itsInfo

protected

Definition at line 282 of file MultiFileBase.h.

Referenced by info().

◆ itsName

String casacore::MultiFileBase::itsName

protected

Definition at line 278 of file MultiFileBase.h.

Referenced by fileName().

◆ itsNrBlock

Int64 casacore::MultiFileBase::itsNrBlock

protected

Definition at line 280 of file MultiFileBase.h.

Referenced by size().

◆ itsUseODirect

Bool casacore::MultiFileBase::itsUseODirect

protected

Definition at line 284 of file MultiFileBase.h.

Referenced by useODirect().

◆ itsWritable

Bool casacore::MultiFileBase::itsWritable

protected

Definition at line 285 of file MultiFileBase.h.

Referenced by isWritable().

The documentation for this class was generated from the following file:

casa/IO/MultiFileBase.h

Public Member Functions

Protected Member Functions

Protected Attributes

Private Member Functions

Detailed Description

Intended use:

Review Status

Synopsis

Example

To Do

Constructor & Destructor Documentation

◆ MultiFileBase()

◆ ~MultiFileBase()

Member Function Documentation

◆ addFile()

◆ blockSize()

◆ close()

◆ deleteFile()

◆ doAddFile()

◆ doDeleteFile()

◆ extend()

◆ fileId()

◆ fileName()

◆ flush()

◆ flushFile()

◆ freeBlocks()

◆ fsync()

◆ info()

◆ isWritable()

◆ nfile()

◆ read()

◆ readBlock()

◆ readHeader()

◆ reopenRW()

◆ resync()

◆ setNewFile()

◆ size()

◆ useODirect()

◆ write()

◆ writeBlock()

◆ writeDirty()

◆ writeHeader()

Member Data Documentation

◆ itsBlockSize

◆ itsBuffer

◆ itsChanged

◆ itsFreeBlocks

◆ itsHdrCounter

◆ itsInfo

◆ itsName

◆ itsNrBlock

◆ itsUseODirect

◆ itsWritable