Class ArrowReader
java.lang.Object
org.apache.arrow.vector.ipc.ArrowReader
- All Implemented Interfaces:
AutoCloseable, DictionaryProvider
- Direct Known Subclasses:
ArrowFileReader, ArrowScannerReader, ArrowStreamReader, NativeScanner.NativeReader, OrcStripeReader
Abstract class to read Schema and ArrowRecordBatches.
-
Nested Class Summary
Nested classes/interfaces inherited from interface DictionaryProvider
DictionaryProvider.MapDictionaryProvider -
Field Summary
Fields -
Constructor Summary
ConstructorsModifierConstructorDescriptionprotectedArrowReader(BufferAllocator allocator) protectedArrowReader(BufferAllocator allocator, CompressionCodec.Factory compressionFactory) -
Method Summary
Modifier and TypeMethodDescriptionabstract longReturn the number of bytes read from the ReadChannel.voidclose()Close resources, including vector schema root and dictionary vectors, and the underlying read source.voidclose(boolean closeReadSource) Close resources, including vector schema root and dictionary vectors.protected abstract voidClose the underlying read source.protected voidInitialize if not done previously.Get all dictionary IDs.Returns any dictionaries that were loaded along with ArrowRecordBatches.Returns the vector schema root.protected voidReads the schema and initializes the vectors.booleanReturns whether every loaded batch is checked withValueVectorUtility.validateFull(ValueVector).protected voidloadDictionary(ArrowDictionaryBatch dictionaryBatch) Load an ArrowDictionaryBatch to the readers dictionary vectors.abstract booleanLoad the next ArrowRecordBatch to the vector schema root if available.protected voidloadRecordBatch(ArrowRecordBatch batch) Load an ArrowRecordBatch to the readers VectorSchemaRoot.lookup(long id) Lookup a dictionary that has been loaded using the dictionary id.protected voidEnsure the reader has been initialized and reset the VectorSchemaRoot row count to 0.protected abstract SchemaRead the Schema from the source, will be invoked at the beginning the initialization.voidsetValidateBatches(boolean validateBatches) Sets whether every record batch and dictionary batch is checked withValueVectorUtility.validateFull(ValueVector)once it is loaded.
-
Field Details
-
allocator
-
dictionaries
-
-
Constructor Details
-
ArrowReader
-
ArrowReader
-
-
Method Details
-
setValidateBatches
public void setValidateBatches(boolean validateBatches) Sets whether every record batch and dictionary batch is checked withValueVectorUtility.validateFull(ValueVector)once it is loaded. This is off by default, because the check reads every value of the batch.Turn it on when the data comes from an untrusted source. The IPC format does not guarantee that offsets, views or run ends stay within the buffers they point into, and reading such a batch can read past the end of a buffer, without bounds checks when
arrow.enable_unsafe_memory_accessis set. With this option, a batch that fails the check is cleared and the load throwsValidateUtil.ValidateException.- Parameters:
validateBatches- whether to check each loaded batch
-
isValidateBatches
public boolean isValidateBatches()Returns whether every loaded batch is checked withValueVectorUtility.validateFull(ValueVector).- Returns:
- true if loaded batches are checked
- See Also:
-
getVectorSchemaRoot
Returns the vector schema root. This will be loaded with new values on every call to loadNextBatch.- Returns:
- the vector schema root
- Throws:
IOException- if reading of schema fails
-
getDictionaryVectors
Returns any dictionaries that were loaded along with ArrowRecordBatches.- Returns:
- Map of dictionaries to dictionary id, empty if no dictionaries loaded
- Throws:
IOException- if reading of schema fails
-
lookup
Lookup a dictionary that has been loaded using the dictionary id.- Specified by:
lookupin interfaceDictionaryProvider- Parameters:
id- Unique identifier for a dictionary- Returns:
- the requested dictionary or null if not found
-
getDictionaryIds
Description copied from interface:DictionaryProviderGet all dictionary IDs.- Specified by:
getDictionaryIdsin interfaceDictionaryProvider
-
loadNextBatch
Load the next ArrowRecordBatch to the vector schema root if available.- Returns:
- true if a batch was read, false on EOS
- Throws:
IOException- on error
-
bytesRead
public abstract long bytesRead()Return the number of bytes read from the ReadChannel.- Returns:
- number of bytes read
-
close
Close resources, including vector schema root and dictionary vectors, and the underlying read source.- Specified by:
closein interfaceAutoCloseable- Throws:
IOException- on error
-
close
Close resources, including vector schema root and dictionary vectors. If the flag closeReadChannel is true then close the underlying read source, otherwise leave it open.- Parameters:
closeReadSource- Flag to control if closing the underlying read source- Throws:
IOException- on error
-
closeReadSource
Close the underlying read source.- Throws:
IOException- on error
-
readSchema
Read the Schema from the source, will be invoked at the beginning the initialization.- Returns:
- the read Schema
- Throws:
IOException- on error
-
ensureInitialized
Initialize if not done previously.- Throws:
IOException- on error
-
initialize
Reads the schema and initializes the vectors.- Throws:
IOException
-
prepareLoadNextBatch
Ensure the reader has been initialized and reset the VectorSchemaRoot row count to 0.- Throws:
IOException- on error
-
loadRecordBatch
Load an ArrowRecordBatch to the readers VectorSchemaRoot.- Parameters:
batch- the record batch to load
-
loadDictionary
Load an ArrowDictionaryBatch to the readers dictionary vectors.- Parameters:
dictionaryBatch- dictionary batch to load
-