public class LargeListVector extends BaseValueVector implements RepeatedValueVector, FieldVector, PromotableVector
WARNING: Currently Arrow in Java doesn't support 64-bit vectors. This class follows the expected behaviour of a LargeList but doesn't actually support allocating a 64-bit vector. It has little use until 64-bit vectors are supported and should be used with caution. todo review checkedCastToInt usage in this class. Once int64 indexed vectors are supported these checks aren't needed.
Modifier and Type | Field and Description |
---|---|
protected CallBack |
callBack |
static String |
DATA_VECTOR_NAME |
static FieldVector |
DEFAULT_DATA_VECTOR |
protected String |
defaultDataVectorName |
static byte |
OFFSET_WIDTH |
protected long |
offsetAllocationSizeInBytes |
protected ArrowBuf |
offsetBuffer |
protected UnionLargeListReader |
reader |
protected ArrowBuf |
validityBuffer |
protected int |
valueCount |
protected FieldVector |
vector |
allocator, fieldReader, INITIAL_VALUE_ALLOCATION, MAX_ALLOCATION_SIZE, MAX_ALLOCATION_SIZE_PROPERTY
DEFAULT_REPEAT_PER_RECORD
Constructor and Description |
---|
LargeListVector(String name,
BufferAllocator allocator,
FieldType fieldType,
CallBack callBack)
Constructs a new instance.
|
Modifier and Type | Method and Description |
---|---|
<OUT,IN> OUT |
accept(VectorVisitor<OUT,IN> visitor,
IN value)
Accept a generic
VectorVisitor and return the result. |
<T extends ValueVector> |
addOrGetVector(FieldType fieldType)
Initialize the data vector (and execute callback) if it hasn't already been done,
returns the data vector.
|
void |
allocateNew()
Same as
allocateNewSafe() . |
boolean |
allocateNewSafe()
Allocate memory for the vector.
|
protected void |
allocateOffsetBuffer(long size) |
void |
clear()
Release any owned ArrowBuf and reset the ValueVector to the initial state.
|
void |
copyFrom(int inIndex,
int outIndex,
ValueVector from)
Copy a cell value from a particular index in source vector to a particular
position in this vector.
|
void |
copyFromSafe(int inIndex,
int outIndex,
ValueVector from)
Same as
copyFrom(int, int, ValueVector) except that
it handles the case when the capacity of the vector needs to be expanded
before copy. |
static LargeListVector |
empty(String name,
BufferAllocator allocator) |
void |
endValue(int index,
long size)
End the current value.
|
ArrowBuf[] |
getBuffers(boolean clear)
Return the underlying buffers associated with this vector.
|
int |
getBufferSize()
Get the size (number of bytes) of underlying buffers used by this
vector.
|
int |
getBufferSizeFor(int valueCount)
Returns the number of bytes that is used by this vector if it holds the given number
of values.
|
List<FieldVector> |
getChildrenFromFields()
The returned list is the same size as the list passed to initializeChildrenFromFields.
|
ArrowBuf |
getDataBuffer()
Gets the underlying buffer associated with data vector.
|
long |
getDataBufferAddress()
Gets the starting address of the underlying buffer associated with data vector.
|
FieldVector |
getDataVector()
Get the inner data vector for this list vector.
|
double |
getDensity()
Get the density of this ListVector.
|
long |
getElementEndIndex(int index) |
long |
getElementStartIndex(int index) |
Field |
getField()
Get information about how this field is materialized.
|
List<ArrowBuf> |
getFieldBuffers()
Get the buffers belonging to this vector.
|
List<BufferBacked> |
getFieldInnerVectors()
Deprecated.
This API will be removed as the current implementations no longer support inner vectors.
|
int |
getLastSet() |
Types.MinorType |
getMinorType() |
String |
getName()
Gets the name of the vector.
|
int |
getNullCount()
Get the number of elements that are null in the vector.
|
List<?> |
getObject(int index)
Get the element in the list vector at a particular index.
|
ArrowBuf |
getOffsetBuffer()
Gets the underlying buffer associated with offset vector.
|
long |
getOffsetBufferAddress()
Gets the starting address of the underlying buffer associated with offset vector.
|
protected int |
getOffsetBufferValueCapacity() |
UInt4Vector |
getOffsetVector()
Deprecated.
This API will be removed, as the current implementations no longer hold inner offset vectors.
|
UnionLargeListReader |
getReader()
Default implementation to create a reader for the vector.
|
protected FieldReader |
getReaderImpl()
Each vector has a different reader that implements the FieldReader interface.
|
TransferPair |
getTransferPair(String ref,
BufferAllocator allocator) |
TransferPair |
getTransferPair(String ref,
BufferAllocator allocator,
CallBack callBack) |
ArrowBuf |
getValidityBuffer()
Gets the underlying buffer associated with validity vector.
|
long |
getValidityBufferAddress()
Gets the starting address of the underlying buffer associated with validity vector.
|
int |
getValueCapacity()
Get the current value capacity for the vector.
|
int |
getValueCount()
Gets the number of values.
|
UnionLargeListWriter |
getWriter() |
int |
hashCode(int index)
Returns hashCode of element in index with the default hasher.
|
int |
hashCode(int index,
ArrowBufHasher hasher)
Returns hashCode of element in index with the given hasher.
|
void |
initializeChildrenFromFields(List<Field> children)
Initializes the child vectors
to be later loaded with loadBuffers.
|
protected void |
invalidateReader() |
boolean |
isEmpty(int index)
Check if element at given index is empty list.
|
boolean |
isNull(int index)
Check if element at given index is null.
|
int |
isSet(int index)
Same as
isNull(int) . |
void |
loadFieldBuffers(ArrowFieldNode fieldNode,
List<ArrowBuf> ownBuffers)
Load the buffers of this vector with provided source buffers.
|
TransferPair |
makeTransferPair(ValueVector target)
Makes a new transfer pair used to transfer underlying buffers.
|
UnionVector |
promoteToUnion() |
void |
reAlloc()
Resize the vector to increase the capacity.
|
protected void |
reallocOffsetBuffer() |
protected void |
replaceDataVector(FieldVector v) |
void |
reset()
Reset the ValueVector to the initial state without releasing any owned ArrowBuf.
|
void |
setInitialCapacity(int numRecords)
Set the initial record capacity.
|
void |
setInitialCapacity(int numRecords,
double density)
Specialized version of setInitialCapacity() for ListVector.
|
void |
setInitialTotalCapacity(int numRecords,
int totalNumberOfElements)
Specialized version of setInitialTotalCapacity() for ListVector.
|
void |
setLastSet(int value) |
void |
setNotNull(int index)
Sets the list at index to be not-null.
|
void |
setNull(int index)
Sets list at index to be null.
|
void |
setValueCount(int valueCount)
Sets the value count for the vector.
|
long |
startNewValue(long index)
Start a new value in the list vector.
|
checkBufRefs, close, getAllocator, getTransferPair, getValidityBufferSizeFromCount, iterator, releaseBuffer, toString, transferBuffer
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
close, getAllocator, getTransferPair, validate, validateFull
forEach, iterator, spliterator
public static final FieldVector DEFAULT_DATA_VECTOR
public static final String DATA_VECTOR_NAME
public static final byte OFFSET_WIDTH
protected ArrowBuf offsetBuffer
protected FieldVector vector
protected final CallBack callBack
protected int valueCount
protected long offsetAllocationSizeInBytes
protected String defaultDataVectorName
protected ArrowBuf validityBuffer
protected UnionLargeListReader reader
public LargeListVector(String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack)
name
- The name of the instance.allocator
- The allocator to use for allocating/reallocating buffers.fieldType
- The type of this list.callBack
- A schema change callback.public static LargeListVector empty(String name, BufferAllocator allocator)
public void initializeChildrenFromFields(List<Field> children)
FieldVector
initializeChildrenFromFields
in interface FieldVector
children
- the schemapublic void setInitialCapacity(int numRecords)
ValueVector
setInitialCapacity
in interface ValueVector
numRecords
- the initial record capacity.public void setInitialCapacity(int numRecords, double density)
setInitialCapacity
in interface DensityAwareVector
numRecords
- value countdensity
- density of ListVector. Density is the average size of
list per position in the List vector. For example, a
density value of 10 implies each position in the list
vector has a list of 10 values.
A density value of 0.1 implies out of 10 positions in
the list vector, 1 position has a list of size 1 and
remaining positions are null (no lists) or empty lists.
This helps in tightly controlling the memory we provision
for inner data vector.public void setInitialTotalCapacity(int numRecords, int totalNumberOfElements)
numRecords
- value counttotalNumberOfElements
- the total number of elements to to allow
for in this vector across all records.public double getDensity()
public List<FieldVector> getChildrenFromFields()
FieldVector
getChildrenFromFields
in interface FieldVector
public void loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers)
loadFieldBuffers
in interface FieldVector
fieldNode
- the fieldNode indicating the value countownBuffers
- the buffers for this Field (own buffers only, children not included)public List<ArrowBuf> getFieldBuffers()
getFieldBuffers
in interface FieldVector
@Deprecated public List<BufferBacked> getFieldInnerVectors()
getFieldInnerVectors
in interface FieldVector
public void allocateNew() throws OutOfMemoryException
allocateNewSafe()
.allocateNew
in interface ValueVector
OutOfMemoryException
- Thrown if no memory can be allocated.public boolean allocateNewSafe()
allocateNewSafe
in interface ValueVector
protected void allocateOffsetBuffer(long size)
public void reAlloc()
reAlloc
in interface ValueVector
protected void reallocOffsetBuffer()
public void copyFromSafe(int inIndex, int outIndex, ValueVector from)
copyFrom(int, int, ValueVector)
except that
it handles the case when the capacity of the vector needs to be expanded
before copy.copyFromSafe
in interface ValueVector
copyFromSafe
in class BaseValueVector
inIndex
- position to copy from in source vectoroutIndex
- position to copy to in this vectorfrom
- source vectorpublic void copyFrom(int inIndex, int outIndex, ValueVector from)
copyFrom
in interface ValueVector
copyFrom
in class BaseValueVector
inIndex
- position to copy from in source vectoroutIndex
- position to copy to in this vectorfrom
- source vector@Deprecated public UInt4Vector getOffsetVector()
getOffsetVector
in interface RepeatedValueVector
public FieldVector getDataVector()
getDataVector
in interface RepeatedValueVector
public TransferPair getTransferPair(String ref, BufferAllocator allocator)
getTransferPair
in interface ValueVector
public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack)
getTransferPair
in interface ValueVector
public TransferPair makeTransferPair(ValueVector target)
ValueVector
makeTransferPair
in interface ValueVector
target
- the target for the transfertransfer pair
that is used to transfer underlying
buffers into the target vector.public long getValidityBufferAddress()
FieldVector
getValidityBufferAddress
in interface FieldVector
public long getDataBufferAddress()
FieldVector
getDataBufferAddress
in interface FieldVector
public long getOffsetBufferAddress()
FieldVector
getOffsetBufferAddress
in interface FieldVector
public ArrowBuf getValidityBuffer()
ValueVector
getValidityBuffer
in interface ValueVector
public ArrowBuf getDataBuffer()
ValueVector
getDataBuffer
in interface ValueVector
public ArrowBuf getOffsetBuffer()
ValueVector
getOffsetBuffer
in interface ValueVector
public int getValueCount()
ValueVector
getValueCount
in interface ValueVector
public int hashCode(int index)
ValueVector
hashCode
in interface ValueVector
public int hashCode(int index, ArrowBufHasher hasher)
ValueVector
hashCode
in interface ValueVector
public <OUT,IN> OUT accept(VectorVisitor<OUT,IN> visitor, IN value)
ValueVector
VectorVisitor
and return the result.accept
in interface ValueVector
OUT
- the output result type.IN
- the input data together with visitor.public UnionLargeListWriter getWriter()
protected void replaceDataVector(FieldVector v)
public UnionVector promoteToUnion()
promoteToUnion
in interface PromotableVector
protected FieldReader getReaderImpl()
BaseValueVector
getReaderImpl
in class BaseValueVector
public UnionLargeListReader getReader()
BaseValueVector
BaseValueVector.getReaderImpl()
to initialize the reader appropriately.getReader
in interface ValueVector
getReader
in class BaseValueVector
public <T extends ValueVector> AddOrGetResult<T> addOrGetVector(FieldType fieldType)
addOrGetVector
in interface PromotableVector
public int getBufferSize()
getBufferSize
in interface ValueVector
public int getBufferSizeFor(int valueCount)
ValueVector
getBufferSizeFor
in interface ValueVector
valueCount
- the number of values to assume this vector containspublic Field getField()
ValueVector
getField
in interface ValueVector
public Types.MinorType getMinorType()
getMinorType
in interface ValueVector
public String getName()
ValueVector
getName
in interface ValueVector
getName
in class BaseValueVector
public void clear()
ValueVector
clear
in interface ValueVector
clear
in class BaseValueVector
public void reset()
ValueVector
reset
in interface ValueVector
public ArrowBuf[] getBuffers(boolean clear)
getBuffers
in interface ValueVector
clear
- Whether to clear vector before returning; the buffers will still be refcounted
but the returned array will be the only reference to thembuffers
that is used by this
vector instance.protected void invalidateReader()
public List<?> getObject(int index)
getObject
in interface ValueVector
index
- position of the elementpublic boolean isNull(int index)
isNull
in interface ValueVector
index
- position of elementpublic boolean isEmpty(int index)
index
- position of elementpublic int isSet(int index)
isNull(int)
.index
- position of elementpublic int getNullCount()
getNullCount
in interface ValueVector
public int getValueCapacity()
getValueCapacity
in interface ValueVector
protected int getOffsetBufferValueCapacity()
public void setNotNull(int index)
public void setNull(int index)
setNull
in interface FieldVector
index
- position in vectorpublic long startNewValue(long index)
index
- index of the value to startpublic void endValue(int index, long size)
index
- index of the value to endsize
- number of elements in the list that was writtenpublic void setValueCount(int valueCount)
Important note: The underlying vector does not support 64-bit allocations yet. This may throw if attempting to hold larger than what a 32-bit vector can store.
setValueCount
in interface ValueVector
valueCount
- value countpublic void setLastSet(int value)
public int getLastSet()
public long getElementStartIndex(int index)
public long getElementEndIndex(int index)
Copyright © 2023 The Apache Software Foundation. All rights reserved.