org.nd4j.linalg.dataset

Class MultiDataSet

java.lang.Object

org.nd4j.linalg.dataset.MultiDataSet

All Implemented Interfaces:

Serializable, MultiDataSet

public class MultiDataSet
extends Object
implements MultiDataSet

Implementation of MultiDataSet

Author:

Alex Black

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
MultiDataSet() Create a new (empty) MultiDataSet object (all fields are null)
MultiDataSet(INDArray[] features, INDArray[] labels) MultiDataSet constructor with no mask arrays
MultiDataSet(INDArray[] features, INDArray[] labels, INDArray[] featuresMaskArrays, INDArray[] labelsMaskArrays)
MultiDataSet(INDArray features, INDArray labels) MultiDataSet constructor with single features/labels input, no mask arrays
MultiDataSet(INDArray features, INDArray labels, INDArray featuresMask, INDArray labelsMask) MultiDataSet constructor with single features/labels input, single mask arrays

Method Summary

Modifier and Type	Method and Description
List<MultiDataSet>	asList() SplitV the MultiDataSet into a list of individual examples.
MultiDataSet	copy() Clone the dataset
void	detach() This method migrates this DataSet into current Workspace (if any)
boolean	equals(Object o)
List<Serializable>	getExampleMetaData() Get the example metadata, or null if no metadata has been set
<T extends Serializable> List<T>	getExampleMetaData(Class<T> metaDataType) Get the example metadata, or null if no metadata has been set Note: this method results in an unchecked cast - care should be taken when using this!
INDArray[]	getFeatures() Get all of the input features, as an array of INDArrays
INDArray	getFeatures(int index) Get a single feature/input array
INDArray	getFeaturesMaskArray(int index) Get the specified feature mask array.
INDArray[]	getFeaturesMaskArrays() Get the feature mask arrays.
INDArray[]	getLabels() Get all of the labels, as an array of INDArrays
INDArray	getLabels(int index) Get a single label/output array
INDArray	getLabelsMaskArray(int index) Get the specified label mask array.
INDArray[]	getLabelsMaskArrays() Get the labels mask arrays.
long	getMemoryFootprint() This method returns memory used by this DataSet
int	hashCode()
boolean	hasMaskArrays() Whether there are any mask arrays (features or labels) present for this MultiDataSet
boolean	isEmpty()
void	load(File from) Load the contents of this MultiDataSet from the specified file
void	load(InputStream from) Load the contents of this MultiDataSet from the specified stream.
static MultiDataSet	merge(Collection<? extends MultiDataSet> toMerge) Merge a collection of MultiDataSet objects into a single MultiDataSet.
void	migrate() This method migrates this MultiDataSet into current Workspace (if any)
int	numFeatureArrays() Number of arrays of features/input data in the MultiDataSet
int	numLabelsArrays() Number of arrays of label/output data in the MultiDataSet
void	save(File to) Save this MultiDataSet to the specified file
void	save(OutputStream to) Save this MultiDataSet to the specified stream.
void	setExampleMetaData(List<? extends Serializable> exampleMetaData) Set the metadata for this MultiDataSet By convention: the metadata can be any serializable object, one per example in the MultiDataSet
void	setFeatures(INDArray[] features) Set all of the features arrays for the MultiDataSet
void	setFeatures(int idx, INDArray features) Set a single features array (by index) for the MultiDataSet
void	setFeaturesMaskArray(int idx, INDArray maskArray) Set a single feature mask array by index
void	setFeaturesMaskArrays(INDArray[] maskArrays) Set the feature mask arrays
void	setLabels(INDArray[] labels) Set all of the labels arrays for the MultiDataSet
void	setLabels(int idx, INDArray labels) Set a single labels array (by index) for the MultiDataSet
void	setLabelsMaskArray(INDArray[] labelsMaskArrays) Set the labels mask arrays
void	setLabelsMaskArray(int idx, INDArray labelsMaskArray) Set a single labels mask array by index
void	shuffle() Shuffle the order of the examples in the MultiDataSet.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

MultiDataSet

public MultiDataSet()

Create a new (empty) MultiDataSet object (all fields are null)

MultiDataSet

public MultiDataSet(INDArray features,
                    INDArray labels)

MultiDataSet constructor with single features/labels input, no mask arrays

MultiDataSet

public MultiDataSet(INDArray features,
                    INDArray labels,
                    INDArray featuresMask,
                    INDArray labelsMask)

MultiDataSet constructor with single features/labels input, single mask arrays

MultiDataSet

public MultiDataSet(INDArray[] features,
                    INDArray[] labels)

MultiDataSet constructor with no mask arrays

MultiDataSet

public MultiDataSet(INDArray[] features,
                    INDArray[] labels,
                    INDArray[] featuresMaskArrays,
                    INDArray[] labelsMaskArrays)

Parameters:

features - The features (inputs) to the algorithm/neural network

labels - The labels (outputs) to the algorithm/neural network

featuresMaskArrays - The mask arrays for the features. May be null. Typically used with variable-length time series models, etc

labelsMaskArrays - The mask arrays for the labels. May be null. Typically used with variable-length time series models, etc

Method Detail

getExampleMetaData

public List<Serializable> getExampleMetaData()

Description copied from interface: MultiDataSet

Get the example metadata, or null if no metadata has been set

Specified by:

getExampleMetaData in interface MultiDataSet

Returns:

List of metadata instances

See Also:

#getExampleMetaData(Class)} for convenience method for types

getExampleMetaData

public <T extends Serializable> List<T> getExampleMetaData(Class<T> metaDataType)

Description copied from interface: MultiDataSet

Get the example metadata, or null if no metadata has been set
Note: this method results in an unchecked cast - care should be taken when using this!

Specified by:

getExampleMetaData in interface MultiDataSet

Type Parameters:

T - Type of metadata

Parameters:

metaDataType - Class of the metadata (used for opType information)

Returns:

List of metadata objects

setExampleMetaData

public void setExampleMetaData(List<? extends Serializable> exampleMetaData)

Description copied from interface: MultiDataSet

Set the metadata for this MultiDataSet
By convention: the metadata can be any serializable object, one per example in the MultiDataSet

Specified by:

setExampleMetaData in interface MultiDataSet

Parameters:

exampleMetaData - Example metadata to set

numFeatureArrays

public int numFeatureArrays()

Description copied from interface: MultiDataSet

Number of arrays of features/input data in the MultiDataSet

Specified by:

numFeatureArrays in interface MultiDataSet

numLabelsArrays

public int numLabelsArrays()

Description copied from interface: MultiDataSet

Number of arrays of label/output data in the MultiDataSet

Specified by:

numLabelsArrays in interface MultiDataSet

getFeatures

public INDArray[] getFeatures()

Description copied from interface: MultiDataSet

Get all of the input features, as an array of INDArrays

Specified by:

getFeatures in interface MultiDataSet

getFeatures

public INDArray getFeatures(int index)

Description copied from interface: MultiDataSet

Get a single feature/input array

Specified by:

getFeatures in interface MultiDataSet

setFeatures

public void setFeatures(INDArray[] features)

Description copied from interface: MultiDataSet

Set all of the features arrays for the MultiDataSet

Specified by:

setFeatures in interface MultiDataSet

setFeatures

public void setFeatures(int idx,
                        INDArray features)

Description copied from interface: MultiDataSet

Set a single features array (by index) for the MultiDataSet

Specified by:

setFeatures in interface MultiDataSet

getLabels

public INDArray[] getLabels()

Description copied from interface: MultiDataSet

Get all of the labels, as an array of INDArrays

Specified by:

getLabels in interface MultiDataSet

getLabels

public INDArray getLabels(int index)

Description copied from interface: MultiDataSet

Get a single label/output array

Specified by:

getLabels in interface MultiDataSet

setLabels

public void setLabels(INDArray[] labels)

Description copied from interface: MultiDataSet

Set all of the labels arrays for the MultiDataSet

Specified by:

setLabels in interface MultiDataSet

setLabels

public void setLabels(int idx,
                      INDArray labels)

Description copied from interface: MultiDataSet

Set a single labels array (by index) for the MultiDataSet

Specified by:

setLabels in interface MultiDataSet

hasMaskArrays

public boolean hasMaskArrays()

Description copied from interface: MultiDataSet

Whether there are any mask arrays (features or labels) present for this MultiDataSet

Specified by:

hasMaskArrays in interface MultiDataSet

getFeaturesMaskArrays

public INDArray[] getFeaturesMaskArrays()

Description copied from interface: MultiDataSet

Get the feature mask arrays. May return null if no feature mask arrays are present; otherwise, any entry in the array may be null if no mask array is present for that particular feature

Specified by:

getFeaturesMaskArrays in interface MultiDataSet

getFeaturesMaskArray

public INDArray getFeaturesMaskArray(int index)

Description copied from interface: MultiDataSet

Get the specified feature mask array. Returns null if no feature mask is present for that particular feature/input

Specified by:

getFeaturesMaskArray in interface MultiDataSet

setFeaturesMaskArrays

public void setFeaturesMaskArrays(INDArray[] maskArrays)

Description copied from interface: MultiDataSet

Set the feature mask arrays

Specified by:

setFeaturesMaskArrays in interface MultiDataSet

setFeaturesMaskArray

public void setFeaturesMaskArray(int idx,
                                 INDArray maskArray)

Description copied from interface: MultiDataSet

Set a single feature mask array by index

Specified by:

setFeaturesMaskArray in interface MultiDataSet

getLabelsMaskArrays

public INDArray[] getLabelsMaskArrays()

Description copied from interface: MultiDataSet

Get the labels mask arrays. May return null if no labels mask arrays are present; otherwise, any entry in the array may be null if no mask array is present for that particular label

Specified by:

getLabelsMaskArrays in interface MultiDataSet

getLabelsMaskArray

public INDArray getLabelsMaskArray(int index)

Description copied from interface: MultiDataSet

Get the specified label mask array. Returns null if no label mask is present for that particular feature/input

Specified by:

getLabelsMaskArray in interface MultiDataSet

setLabelsMaskArray

public void setLabelsMaskArray(INDArray[] labelsMaskArrays)

Description copied from interface: MultiDataSet

Set the labels mask arrays

Specified by:

setLabelsMaskArray in interface MultiDataSet

setLabelsMaskArray

public void setLabelsMaskArray(int idx,
                               INDArray labelsMaskArray)

Description copied from interface: MultiDataSet

Set a single labels mask array by index

Specified by:

setLabelsMaskArray in interface MultiDataSet

save

public void save(OutputStream to)
          throws IOException

Description copied from interface: MultiDataSet

Save this MultiDataSet to the specified stream. Stream will be closed after saving.

Specified by:

save in interface MultiDataSet

Throws:

IOException

save

public void save(File to)
          throws IOException

Description copied from interface: MultiDataSet

Save this MultiDataSet to the specified file

Specified by:

save in interface MultiDataSet

Throws:

IOException

load

public void load(InputStream from)
          throws IOException

Description copied from interface: MultiDataSet

Load the contents of this MultiDataSet from the specified stream. Stream will be closed after loading.

Specified by:

load in interface MultiDataSet

Throws:

IOException

load

public void load(File from)
          throws IOException

Description copied from interface: MultiDataSet

Load the contents of this MultiDataSet from the specified file

Specified by:

load in interface MultiDataSet

Throws:

IOException

asList

public List<MultiDataSet> asList()

Description copied from interface: MultiDataSet

SplitV the MultiDataSet into a list of individual examples.

Specified by:

asList in interface MultiDataSet

Returns:

List of MultiDataSets, each with 1 example

copy

public MultiDataSet copy()

Clone the dataset

Specified by:

copy in interface MultiDataSet

Returns:

a clone of the dataset

merge

public static MultiDataSet merge(Collection<? extends MultiDataSet> toMerge)

Merge a collection of MultiDataSet objects into a single MultiDataSet. Merging is done by concatenating along dimension 0 (example number in batch) Merging operation may introduce mask arrays (when necessary) for time series data that has different lengths; if mask arrays already exist, these will be merged also.

Parameters:

toMerge - Collection of MultiDataSet objects to merge

Returns:

a single MultiDataSet object, containing the arrays of

toString

public String toString()

Overrides:

toString in class Object

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

getMemoryFootprint

public long getMemoryFootprint()

This method returns memory used by this DataSet

Specified by:

getMemoryFootprint in interface MultiDataSet

Returns:

migrate

public void migrate()

Description copied from interface: MultiDataSet

This method migrates this MultiDataSet into current Workspace (if any)

Specified by:

migrate in interface MultiDataSet

detach

public void detach()

This method migrates this DataSet into current Workspace (if any)

Specified by:

detach in interface MultiDataSet

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface MultiDataSet

Returns:

True if the MultiDataSet is empty (all features/labels arrays are empty)

shuffle

public void shuffle()

Description copied from interface: MultiDataSet

Shuffle the order of the examples in the MultiDataSet. Note that this generally won't make any difference in practice unless the MultiDataSet is later split.

Specified by:

shuffle in interface MultiDataSet