Hdf5DataWriter Class Reference

#include <Hdf5DataWriter.hpp>

Collaboration diagram for Hdf5DataWriter:

Public Member Functions
	Hdf5DataWriter (DistributedVectorFactory &rVectorFactory, const std::string &rDirectory, const std::string &rBaseName, bool cleanDirectory=true, bool extendData=false, std::string datasetName="Data")
virtual	~Hdf5DataWriter ()
void	DefineFixedDimension (long dimensionSize)
void	DefineFixedDimension (const std::vector< unsigned > &rNodesToOuput, long vecSize)
void	DefineUnlimitedDimension (const std::string &rVariableName, const std::string &rVariableUnits, unsigned estimatedLength=1)
void	AdvanceAlongUnlimitedDimension ()
int	DefineVariable (const std::string &rVariableName, const std::string &rVariableUnits)
virtual void	EndDefineMode ()
void	PossiblyExtend ()
void	PutVector (int variableID, Vec petscVector)
void	PutStripedVector (std::vector< int > variableIDs, Vec petscVector)
void	PutUnlimitedVariable (double value)
void	Close ()
int	GetVariableByName (const std::string &rVariableName)
bool	ApplyPermutation (const std::vector< unsigned > &rPermutation, bool unsafeExtendingMode=false)
void	DefineFixedDimensionUsingMatrix (const std::vector< unsigned > &rNodesToOuput, long vecSize)
void	SetFixedChunkSize (const unsigned &rTimestepsPerChunk, const unsigned &rNodesPerChunk, const unsigned &rVariablesPerChunk)
Private Member Functions
void	CheckVariableName (const std::string &rName)
void	CheckUnitsName (const std::string &rName)
void	ComputeIncompleteOffset ()
void	OpenFile ()
hsize_t	CalculateNumberOfChunks ()
void	CalculateChunkDims (unsigned targetSize, unsigned pChunkSizeInBytes, bool pAllOneChunk)
void	SetChunkSize ()
Private Attributes
DistributedVectorFactory &	mrVectorFactory
bool	mCleanDirectory
bool	mUseExistingFile
bool	mIsInDefineMode
bool	mIsFixedDimensionSet
unsigned	mEstimatedUnlimitedLength
unsigned	mFileFixedDimensionSize
unsigned	mDataFixedDimensionSize
unsigned	mLo
unsigned	mHi
unsigned	mNumberOwned
unsigned	mOffset
bool	mNeedExtend
bool	mUseMatrixForIncompleteData
std::vector< DataWriterVariable >	mVariables
long unsigned	mCurrentTimeStep
Mat	mSinglePermutation
Mat	mDoublePermutation
Mat	mSingleIncompleteOutputMatrix
Mat	mDoubleIncompleteOutputMatrix
bool	mUseOptimalChunkSizeAlgorithm
hsize_t	mChunkSize [DATASET_DIMS]
hsize_t	mNumberOfChunks
hsize_t	mFixedChunkSize [DATASET_DIMS]

Detailed Description

A concrete HDF5 data writer class.

Definition at line 55 of file Hdf5DataWriter.hpp.

Constructor & Destructor Documentation

Hdf5DataWriter::Hdf5DataWriter	(	DistributedVectorFactory &	rVectorFactory,
		const std::string &	rDirectory,
		const std::string &	rBaseName,
		bool	cleanDirectory = `true`,
		bool	extendData = `false`,
		std::string	datasetName = `"Data"`
	)

Constructor.

Parameters:

	rVectorFactory	the factory to use in creating PETSc Vec and DistributedVector objects.
	rDirectory	the directory in which to write the data to file, relative to chaste test output.
	rBaseName	the name of the file in which to write the data
	cleanDirectory	whether to clean the directory (defaults to true)
	extendData	whether to try opening an existing file and appending to it.
	datasetName	The name of the HDF5 dataset to write, defaults to "Data".

The extendData parameter allows us to add to an existing dataset. It only really makes sense if the existing file has an unlimited dimension which we can extend. It also only makes sense if cleanDirectory is false, otherwise there won't be a file there to read...

Todo:: 1300 We can't set mDataFixedDimensionSize, because the information isn't in the input file. This means that checking the size of input vectors in PutVector and PutStripedVector is impossible.

Definition at line 52 of file Hdf5DataWriter.cpp.

Hdf5DataWriter::~Hdf5DataWriter ( ) [virtual]

Destructor.

Definition at line 256 of file Hdf5DataWriter.cpp.

References Close(), PetscTools::Destroy(), mDoubleIncompleteOutputMatrix, mDoublePermutation, mSingleIncompleteOutputMatrix, and mSinglePermutation.

Member Function Documentation

void Hdf5DataWriter::AdvanceAlongUnlimitedDimension ( )

Advance along the unlimited dimension. Normally this will be called when all variables in a row have been input.

Definition at line 1054 of file Hdf5DataWriter.cpp.

References EXCEPTION, mCurrentTimeStep, AbstractHdf5Access::mDatasetDims, mEstimatedUnlimitedLength, AbstractHdf5Access::mIsUnlimitedDimensionSet, and mNeedExtend.

Referenced by Hdf5DataWriter(), AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::Solve(), CardiacElectroMechanicsProblem< DIM, ELEC_PROB_DIM >::Solve(), and AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::Solve().

bool Hdf5DataWriter::ApplyPermutation	(	const std::vector< unsigned > &	rPermutation,
		bool	unsafeExtendingMode = `false`
	)

Apply a permutation to all occurences of PutVector Should be called when in define mode

Parameters:

	rPermutation	a forward/?reverse permutation
	unsafeExtendingMode	is true when we are extending a file which requires a permutation to be applied to it. In particular we are extending a cardiac simulation with "original node ordering"

Returns:: success value. A value "false" indictates that the permutation was empty or was the identity and was not applied

Definition at line 1109 of file Hdf5DataWriter.cpp.

References EXCEPTION, mDataFixedDimensionSize, mDoublePermutation, mFileFixedDimensionSize, mHi, mIsInDefineMode, mLo, mSinglePermutation, and PetscTools::SetupMat().

Referenced by AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::InitialiseWriter().

void Hdf5DataWriter::CalculateChunkDims	(	unsigned	targetSize,
		unsigned *	pChunkSizeInBytes,
		bool *	pAllOneChunk
	)			`[private]`

Calculate (and save in the member variables) chunk dimensions based on a target size. Chunks are kept as "square" as possible while wasting as little space around the edges of the dataset as possible.

Parameters:

`[in]`	targetSize	The target number of entries in each dimension in the chunk.
`[out]`	pChunkSizeInBytes	The size of the resulting chunk in bytes.
`[out]`	pAllOneChunk	Whether the dataset is spanned by a single chunk.

Definition at line 1220 of file Hdf5DataWriter.cpp.

References AbstractHdf5Access::DATASET_DIMS, mChunkSize, and AbstractHdf5Access::mDatasetDims.

Referenced by SetChunkSize().

hsize_t Hdf5DataWriter::CalculateNumberOfChunks ( ) [private]

Little method to calculate the number of chunks resulting from given chunk dimensions.

Returns:: The number of chunks resulting from chunk dimensions (stored in member variable).

Definition at line 1209 of file Hdf5DataWriter.cpp.

References AbstractHdf5Access::DATASET_DIMS, mChunkSize, and AbstractHdf5Access::mDatasetDims.

Referenced by SetChunkSize().

void Hdf5DataWriter::CheckUnitsName ( const std::string & rName ) [private]

Check name of unit is allowed, i.e. contains only alphanumeric & _, and isn't blank.

Parameters:

rName

unit name

Definition at line 543 of file Hdf5DataWriter.cpp.

References EXCEPTION.

Referenced by CheckVariableName(), and DefineVariable().

void Hdf5DataWriter::CheckVariableName ( const std::string & rName ) [private]

Check name of variable is allowed, i.e. contains only alphanumeric & _, and isn't blank.

Parameters:

rName

variable name

Definition at line 534 of file Hdf5DataWriter.cpp.

References CheckUnitsName(), and EXCEPTION.

Referenced by DefineVariable().

void Hdf5DataWriter::Close ( )

Close any open files.

Definition at line 1016 of file Hdf5DataWriter.cpp.

References AbstractHdf5Access::mFileId, mIsInDefineMode, AbstractHdf5Access::mIsUnlimitedDimensionSet, AbstractHdf5Access::mUnlimitedDatasetId, and AbstractHdf5Access::mVariablesDatasetId.

Referenced by CardiacElectroMechanicsProblem< DIM, ELEC_PROB_DIM >::Solve(), and ~Hdf5DataWriter().

void Hdf5DataWriter::ComputeIncompleteOffset ( ) [private]

Compute mOffset and mNumberOwned from mIncompleteNodeIndices when we have incomplete data.

Definition at line 482 of file Hdf5DataWriter.cpp.

References mHi, AbstractHdf5Access::mIncompleteNodeIndices, mLo, mNumberOwned, and mOffset.

Referenced by DefineFixedDimension(), DefineFixedDimensionUsingMatrix(), and Hdf5DataWriter().

void Hdf5DataWriter::DefineFixedDimension	(	const std::vector< unsigned > &	rNodesToOuput,
		long	vecSize
	)

Define the fixed dimension, assuming incomplete data output (subset of the nodes).

Parameters:

	rNodesToOuput	Node indexes to be output (precondition: to be monotonic increasing)
	vecSize

Definition at line 398 of file Hdf5DataWriter.cpp.

References ComputeIncompleteOffset(), DefineFixedDimension(), EXCEPTION, mFileFixedDimensionSize, AbstractHdf5Access::mIncompleteNodeIndices, and AbstractHdf5Access::mIsDataComplete.

void Hdf5DataWriter::DefineFixedDimension ( long dimensionSize )

Define the fixed dimension, assuming complete data output (all the nodes).

Parameters:

dimensionSize

The size of the dimension

Definition at line 373 of file Hdf5DataWriter.cpp.

References EXCEPTION, DistributedVectorFactory::GetHigh(), DistributedVectorFactory::GetLocalOwnership(), DistributedVectorFactory::GetLow(), mDataFixedDimensionSize, mFileFixedDimensionSize, mHi, mIsFixedDimensionSet, mIsInDefineMode, mLo, mNumberOwned, mOffset, and mrVectorFactory.

Referenced by DefineFixedDimension(), DefineFixedDimensionUsingMatrix(), ExtendedBidomainProblem< DIM >::DefineWriterColumns(), AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::DefineWriterColumns(), and AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::InitialiseHdf5Writer().

void Hdf5DataWriter::DefineFixedDimensionUsingMatrix	(	const std::vector< unsigned > &	rNodesToOuput,
		long	vecSize
	)

Define the fixed dimension, assuming incomplete data output (subset of the nodes) and using a matrix to convert from full to incomplete output (rather than picking required data values out one at a time).

Parameters:

	rNodesToOuput	Node indexes to be output (precondition: to be monotonic increasing)
	vecSize

Definition at line 423 of file Hdf5DataWriter.cpp.

References ComputeIncompleteOffset(), DefineFixedDimension(), EXCEPTION, mDataFixedDimensionSize, mDoubleIncompleteOutputMatrix, mFileFixedDimensionSize, mHi, AbstractHdf5Access::mIncompleteNodeIndices, AbstractHdf5Access::mIsDataComplete, mLo, mNumberOwned, mOffset, mSingleIncompleteOutputMatrix, mUseMatrixForIncompleteData, and PetscTools::SetupMat().

void Hdf5DataWriter::DefineUnlimitedDimension	(	const std::string &	rVariableName,
		const std::string &	rVariableUnits,
		unsigned	estimatedLength = `1`
	)

Define a variable.

Parameters:

	rVariableName	The name of the dimension
	rVariableUnits	The physical units of the dimension
	estimatedLength	An estimate of the unlimited dimension length for performance reasons

Definition at line 1034 of file Hdf5DataWriter.cpp.

References EXCEPTION, mEstimatedUnlimitedLength, mIsInDefineMode, AbstractHdf5Access::mIsUnlimitedDimensionSet, AbstractHdf5Access::mUnlimitedDimensionName, and AbstractHdf5Access::mUnlimitedDimensionUnit.

Referenced by ExtendedBidomainProblem< DIM >::DefineWriterColumns(), AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::DefineWriterColumns(), and AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::InitialiseHdf5Writer().

int Hdf5DataWriter::DefineVariable	(	const std::string &	rVariableName,
		const std::string &	rVariableUnits
	)

Define a variable.

Parameters:

	rVariableName	The name of the dimension
	rVariableUnits	The physical units of the dimension

Returns:: The identifier of the variable

Definition at line 499 of file Hdf5DataWriter.cpp.

References CheckUnitsName(), CheckVariableName(), EXCEPTION, mIsInDefineMode, DataWriterVariable::mVariableName, mVariables, and DataWriterVariable::mVariableUnits.

Referenced by AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::DefineExtraVariablesWriterColumns(), ExtendedBidomainProblem< DIM >::DefineWriterColumns(), BidomainProblem< DIM >::DefineWriterColumns(), AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::DefineWriterColumns(), and AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::InitialiseHdf5Writer().

void Hdf5DataWriter::EndDefineMode ( ) [virtual]

End the define mode of the DataWriter.

Definition at line 555 of file Hdf5DataWriter.cpp.

Referenced by AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::InitialiseHdf5Writer(), and AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::InitialiseWriter().

int Hdf5DataWriter::GetVariableByName ( const std::string & rVariableName )

Get the id of the given variable, the variable must already exist or an exception will be thrown.

Parameters:

rVariableName

variable name to look up

Returns:: HDF5 id for the given variable.

Definition at line 1089 of file Hdf5DataWriter.cpp.

References EXCEPTION, and mVariables.

Referenced by AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::DefineExtraVariablesWriterColumns(), ExtendedBidomainProblem< DIM >::DefineWriterColumns(), BidomainProblem< DIM >::DefineWriterColumns(), and AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::DefineWriterColumns().

void Hdf5DataWriter::OpenFile ( ) [private]

Opens an existing file or creates a new file, depending on mUseExistingFile.

This method sets mFileId.

Definition at line 278 of file Hdf5DataWriter.cpp.

References RelativeTo::Absolute, EXCEPTION, FileFinder::Exists(), FileFinder::GetAbsolutePath(), AbstractHdf5Access::mBaseName, mCleanDirectory, AbstractHdf5Access::mDatasetDims, AbstractHdf5Access::mDirectory, mEstimatedUnlimitedLength, mFileFixedDimensionSize, AbstractHdf5Access::mFileId, mNumberOfChunks, mUseExistingFile, mVariables, AbstractHdf5Access::mVariablesDatasetId, and SetChunkSize().

Referenced by EndDefineMode(), and Hdf5DataWriter().

void Hdf5DataWriter::PossiblyExtend ( )

Extend the dataset to the correct to the correct dimensions if needed.

Definition at line 1074 of file Hdf5DataWriter.cpp.

References AbstractHdf5Access::mDatasetDims, mNeedExtend, AbstractHdf5Access::mUnlimitedDatasetId, and AbstractHdf5Access::mVariablesDatasetId.

Referenced by PutStripedVector(), PutUnlimitedVariable(), and PutVector().

void Hdf5DataWriter::PutStripedVector	(	std::vector< int >	variableIDs,
		Vec	petscVector
	)

Write data for two variables from a PETSc vector to the dataset.

Parameters:

	variableIDs	vector of IDs of variables of the columns in the HDF5 file
	petscVector	the data

Definition at line 846 of file Hdf5DataWriter.cpp.

References PetscTools::CreateVec(), AbstractHdf5Access::DATASET_DIMS, PetscTools::Destroy(), EXCEPTION, mCurrentTimeStep, mDataFixedDimensionSize, mDoubleIncompleteOutputMatrix, mDoublePermutation, mFileFixedDimensionSize, AbstractHdf5Access::mIncompleteNodeIndices, AbstractHdf5Access::mIsDataComplete, mIsInDefineMode, mLo, mNumberOwned, mOffset, mUseMatrixForIncompleteData, AbstractHdf5Access::mVariablesDatasetId, and PossiblyExtend().

Referenced by AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::WriteOneStep(), ExtendedBidomainProblem< DIM >::WriteOneStep(), and BidomainProblem< DIM >::WriteOneStep().

void Hdf5DataWriter::PutUnlimitedVariable ( double value )

Write a single value for the unlimited variable (e.g. time) to the dataset.

Parameters:

value

the data

Definition at line 980 of file Hdf5DataWriter.cpp.

References PetscTools::AmMaster(), EXCEPTION, mCurrentTimeStep, mIsInDefineMode, AbstractHdf5Access::mIsUnlimitedDimensionSet, AbstractHdf5Access::mUnlimitedDatasetId, and PossiblyExtend().

Referenced by AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::WriteOneStep(), MonodomainProblem< ELEMENT_DIM, SPACE_DIM >::WriteOneStep(), ExtendedBidomainProblem< DIM >::WriteOneStep(), and BidomainProblem< DIM >::WriteOneStep().

void Hdf5DataWriter::PutVector	(	int	variableID,
		Vec	petscVector
	)

Write data for a given variable from a PETSc vector to the dataset.

Parameters:

	variableID	the variable id of the column in the HDF5 file
	petscVector	the data

Definition at line 741 of file Hdf5DataWriter.cpp.

References PetscTools::CreateVec(), AbstractHdf5Access::DATASET_DIMS, PetscTools::Destroy(), EXCEPTION, mCurrentTimeStep, mDataFixedDimensionSize, mFileFixedDimensionSize, AbstractHdf5Access::mIncompleteNodeIndices, AbstractHdf5Access::mIsDataComplete, mIsInDefineMode, mLo, mNumberOwned, mOffset, mSingleIncompleteOutputMatrix, mSinglePermutation, mUseMatrixForIncompleteData, AbstractHdf5Access::mVariablesDatasetId, and PossiblyExtend().

Referenced by AbstractCardiacProblem< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::WriteExtraVariablesOneStep(), AbstractDynamicLinearPdeSolver< ELEMENT_DIM, SPACE_DIM, PROBLEM_DIM >::WriteOneStep(), and MonodomainProblem< ELEMENT_DIM, SPACE_DIM >::WriteOneStep().

void Hdf5DataWriter::SetChunkSize ( ) [private]

This method sets the chunk size by building up in each dimension until a threshold is reached, unless user-specified values have been set using SetFixedChunkSize. By default, chunks of 128 K are used, which seems to be a good compromise. For large problems performance will usually improve by increasing this value (to e.g. 1 M).

Definition at line 1244 of file Hdf5DataWriter.cpp.

References CalculateChunkDims(), CalculateNumberOfChunks(), AbstractHdf5Access::DATASET_DIMS, mChunkSize, mFixedChunkSize, mNumberOfChunks, mUseOptimalChunkSizeAlgorithm, and NEVER_REACHED.

Referenced by EndDefineMode(), and OpenFile().

void Hdf5DataWriter::SetFixedChunkSize	(	const unsigned &	rTimestepsPerChunk,
		const unsigned &	rNodesPerChunk,
		const unsigned &	rVariablesPerChunk
	)

Use a particular chunk size, ignoring the algorithm that figures out a sensible value.

This method may be useful in very specific circumstances, e.g. if you want chunks to align perfectly with stripes on a Lustre file system, or if you have a parallel problem where every process can be assigned the same size and shape hyperslab of the HDF5 file (thus getting rid of contention between processes when reading/writing).

USE WITH CAUTION - as it can degrade performance.

Parameters:

	rTimestepsPerChunk	The number of unlimited dimension steps (usually time steps) per chunk.
	rNodesPerChunk	The number of objects (usually nodes) per chunk in the second dimension.
	rVariablesPerChunk	The number of objects (usually output variables) per chunk in the third dimension.

Definition at line 1197 of file Hdf5DataWriter.cpp.

References mFixedChunkSize, mIsInDefineMode, and mUseOptimalChunkSizeAlgorithm.