Package org.apache.arrow.memory

Class ArrowBuf

java.lang.Object
org.apache.arrow.memory.ArrowBuf
All Implemented Interfaces:
AutoCloseable
public final class ArrowBuf extends Object implements AutoCloseable
ArrowBuf serves as a facade over underlying memory by providing several access APIs to read/write data into a chunk of direct memory. All the accounting, ownership and reference management is done by ReferenceManager and ArrowBuf can work with a custom user provided implementation of ReferenceManager

Two important instance variables of an ArrowBuf: (1) address - starting virtual address in the underlying memory chunk that this ArrowBuf has access to (2) length - length (in bytes) in the underlying memory chunk that this ArrowBuf has access to

The management (allocation, deallocation, reference counting etc) for the memory chunk is not done by ArrowBuf. Default implementation of ReferenceManager, allocation is in BaseAllocator, BufferLedger and AllocationManager

  • Constructor Summary

    Constructors
    Constructor
    Description
    ArrowBuf(ReferenceManager referenceManager, @Nullable BufferManager bufferManager, long capacity, long memoryAddress)
    Constructs a new ArrowBuf.

  • Method Summary

    Modifier and Type
    Method
    Description
    long
    capacity()
     
    ArrowBuf
    capacity(long newCapacity)
    Adjusts the capacity of this buffer.
    void
    checkBytes(long start, long end)
    Allows a function to determine whether not reading a particular string of bytes is valid.
    ArrowBuf
    clear()
     
    void
    close()
     
    boolean
    equals(@Nullable Object obj)
     
    long
    getActualMemoryConsumed()
    Return that is Accounted for by this buffer (and its potentially shared siblings within the context of the associated allocator).
    byte
    getByte(long index)
    Get byte value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    getBytes(long index, byte[] dst)
    Copy data from this ArrowBuf at a given index in into destination byte array.
    void
    getBytes(long index, byte[] dst, int dstIndex, int length)
    Copy data from this ArrowBuf at a given index into destination byte array.
    void
    getBytes(long index, OutputStream out, int length)
    Copy a certain length of bytes from this ArrowBuf at a given index into the given OutputStream.
    void
    getBytes(long index, ByteBuffer dst)
    Copy data from this ArrowBuf at a given index into the destination ByteBuffer.
    void
    getBytes(long index, ArrowBuf dst, long dstIndex, int length)
    Copy a given length of data from this ArrowBuf starting at a given index into a dst ArrowBuf at dstIndex.
    char
    getChar(long index)
    Get char value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    double
    getDouble(long index)
    Get double value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    float
    getFloat(long index)
    Get float value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    long
    getId()
    Get the integer id assigned to this ArrowBuf for debugging purposes.
    int
    getInt(long index)
    Get int value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    long
    getLong(long index)
    Get long value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    long
    getPossibleMemoryConsumed()
    Returns the possible memory consumed by this ArrowBuf in the worse case scenario.
    ReferenceManager
    getReferenceManager()
    Get reference manager for this ArrowBuf.
    short
    getShort(long index)
    Get short value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
    int
    hashCode()
     
    long
    memoryAddress()
     
    ByteBuffer
    nioBuffer()
    Make a nio byte buffer from this arrowbuf.
    ByteBuffer
    nioBuffer(long index, int length)
    Make a nio byte buffer from this ArrowBuf.
    ByteOrder
    order()
    Returns the byte order of elements in this buffer.
    void
    print(StringBuilder sb, int indent)
    Print detailed information of this buffer into sb.
    void
    print(StringBuilder sb, int indent, org.apache.arrow.memory.BaseAllocator.Verbosity verbosity)
    Print information of this buffer into sb at the given indentation and verbosity level.
    long
    readableBytes()
    Returns the number of bytes still available to read in this buffer.
    byte
    readByte()
    Read the byte at readerIndex.
    void
    readBytes(byte[] dst)
    Read dst.length bytes at readerIndex into dst byte array
    long
    readerIndex()
    Get the index at which the next byte will be read from.
    ArrowBuf
    readerIndex(long readerIndex)
    Set the reader index for this ArrowBuf.
    ArrowBuf
    reallocIfNeeded(long size)
    Returns this if size is less than capacity(), otherwise delegates to BufferManager.replace(ArrowBuf, long) to get a new buffer.
    int
    refCnt()
     
    void
    setByte(long index, byte value)
    Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setByte(long index, int value)
    Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setBytes(long index, byte[] src)
    Copy data from a given byte array into this ArrowBuf starting at a given index.
    void
    setBytes(long index, byte[] src, int srcIndex, long length)
    Copy data from a given byte array starting at the given source index into this ArrowBuf at a given index.
    int
    setBytes(long index, InputStream in, int length)
    Copy a certain length of bytes from given InputStream into this ArrowBuf at the provided index.
    void
    setBytes(long index, ByteBuffer src)
    Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer.
    void
    setBytes(long index, ByteBuffer src, int srcIndex, int length)
    Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer starting at a given srcIndex for a certain length.
    void
    setBytes(long index, ArrowBuf src)
    Copy readableBytes() number of bytes from src ArrowBuf starting from its readerIndex into this ArrowBuf starting at the given index.
    void
    setBytes(long index, ArrowBuf src, long srcIndex, long length)
    Copy data from src ArrowBuf starting at index srcIndex into this ArrowBuf at given index.
    void
    setChar(long index, int value)
    Set char value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setDouble(long index, double value)
    Set double value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setFloat(long index, float value)
    Set float value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setInt(long index, int value)
    Set int value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setLong(long index, long value)
    Set long value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    ArrowBuf
    setOne(int index, int length)
    Deprecated.
    use setOne(long, long) instead.
    ArrowBuf
    setOne(long index, long length)
    Sets all bits to one in the specified range.
    void
    setShort(long index, int value)
    Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    void
    setShort(long index, short value)
    Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
    ArrowBuf
    setZero(long index, long length)
    Zero-out the bytes in this ArrowBuf starting at the given index for the given length.
    ArrowBuf
    slice()
    Returns a slice of only the readable bytes in the buffer.
    ArrowBuf
    slice(long index, long length)
    Returns a slice (view) starting at index with the given length.
    String
    toHexString(long start, int length)
    Return the buffer's byte contents in the form of a hex dump.
    String
    toString()
     
    long
    writableBytes()
    Returns the number of bytes still available to write into this buffer before capacity is reached.
    void
    writeByte(byte value)
    Set the provided byte value at the writerIndex.
    void
    writeByte(int value)
    Set the lower order byte for the provided value at the writerIndex.
    void
    writeBytes(byte[] src)
    Write the bytes from given byte array into this ArrowBuf starting at writerIndex.
    void
    writeBytes(byte[] src, int srcIndex, int length)
    Write the bytes from given byte array starting at srcIndex into this ArrowBuf starting at writerIndex.
    void
    writeDouble(double value)
    Set the provided double value at the writerIndex.
    void
    writeFloat(float value)
    Set the provided float value at the writerIndex.
    void
    writeInt(int value)
    Set the provided int value at the writerIndex.
    void
    writeLong(long value)
    Set the provided long value at the writerIndex.
    long
    writerIndex()
    Get the index at which next byte will be written to.
    ArrowBuf
    writerIndex(long writerIndex)
    Set the writer index for this ArrowBuf.
    void
    writeShort(int value)
    Set the provided int value as short at the writerIndex.

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • ArrowBuf

      public ArrowBuf(ReferenceManager referenceManager, @Nullable BufferManager bufferManager, long capacity, long memoryAddress)
      Constructs a new ArrowBuf.
      Parameters:
      referenceManager - The memory manager to track memory usage and reference count of this buffer
      capacity - The capacity in bytes of this buffer

  • Method Details

    • refCnt

      public int refCnt()

    • checkBytes

      public void checkBytes(long start, long end)
      Allows a function to determine whether not reading a particular string of bytes is valid.

      Will throw an exception if the memory is not readable for some reason. Only doesn't something in the case that AssertionUtil.BOUNDS_CHECKING_ENABLED is true.

      Parameters:
      start - The starting position of the bytes to be read.
      end - The exclusive endpoint of the bytes to be read.

    • getReferenceManager

      public ReferenceManager getReferenceManager()
      Get reference manager for this ArrowBuf.
      Returns:
      user provided implementation of ReferenceManager

    • capacity

      public long capacity()

    • capacity

      public ArrowBuf capacity(long newCapacity)
      Adjusts the capacity of this buffer. Size increases are NOT supported.
      Parameters:
      newCapacity - Must be in in the range [0, length).

    • order

      public ByteOrder order()
      Returns the byte order of elements in this buffer.

    • readableBytes

      public long readableBytes()
      Returns the number of bytes still available to read in this buffer.

    • writableBytes

      public long writableBytes()
      Returns the number of bytes still available to write into this buffer before capacity is reached.

    • slice

      public ArrowBuf slice()
      Returns a slice of only the readable bytes in the buffer.

    • slice

      public ArrowBuf slice(long index, long length)
      Returns a slice (view) starting at index with the given length.

    • nioBuffer

      public ByteBuffer nioBuffer()
      Make a nio byte buffer from this arrowbuf.

    • nioBuffer

      public ByteBuffer nioBuffer(long index, int length)
      Make a nio byte buffer from this ArrowBuf.

    • memoryAddress

      public long memoryAddress()

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • equals

      public boolean equals(@Nullable Object obj)
      Overrides:
      equals in class Object

    • getLong

      public long getLong(long index)
      Get long value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      8 byte long value

    • setLong

      public void setLong(long index, long value)
      Set long value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • getFloat

      public float getFloat(long index)
      Get float value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      4 byte float value

    • setFloat

      public void setFloat(long index, float value)
      Set float value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • getDouble

      public double getDouble(long index)
      Get double value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      8 byte double value

    • setDouble

      public void setDouble(long index, double value)
      Set double value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • getChar

      public char getChar(long index)
      Get char value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      2 byte char value

    • setChar

      public void setChar(long index, int value)
      Set char value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • getInt

      public int getInt(long index)
      Get int value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      4 byte int value

    • setInt

      public void setInt(long index, int value)
      Set int value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • getShort

      public short getShort(long index)
      Get short value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      2 byte short value

    • setShort

      public void setShort(long index, int value)
      Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • setShort

      public void setShort(long index, short value)
      Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • setByte

      public void setByte(long index, int value)
      Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • setByte

      public void setByte(long index, byte value)
      Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be written
      value - value to write

    • getByte

      public byte getByte(long index)
      Get byte value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      Parameters:
      index - index (0 based relative to this ArrowBuf) where the value will be read from
      Returns:
      byte value

    • readByte

      public byte readByte()
      Read the byte at readerIndex.
      Returns:
      byte value

    • readBytes

      public void readBytes(byte[] dst)
      Read dst.length bytes at readerIndex into dst byte array
      Parameters:
      dst - byte array where the data will be written

    • writeByte

      public void writeByte(byte value)
      Set the provided byte value at the writerIndex.
      Parameters:
      value - value to set

    • writeByte

      public void writeByte(int value)
      Set the lower order byte for the provided value at the writerIndex.
      Parameters:
      value - value to be set

    • writeBytes

      public void writeBytes(byte[] src)
      Write the bytes from given byte array into this ArrowBuf starting at writerIndex.
      Parameters:
      src - src byte array

    • writeBytes

      public void writeBytes(byte[] src, int srcIndex, int length)
      Write the bytes from given byte array starting at srcIndex into this ArrowBuf starting at writerIndex.
      Parameters:
      src - src byte array
      srcIndex - index in the byte array where the copy will being from
      length - length of data to copy

    • writeShort

      public void writeShort(int value)
      Set the provided int value as short at the writerIndex.
      Parameters:
      value - value to set

    • writeInt

      public void writeInt(int value)
      Set the provided int value at the writerIndex.
      Parameters:
      value - value to set

    • writeLong

      public void writeLong(long value)
      Set the provided long value at the writerIndex.
      Parameters:
      value - value to set

    • writeFloat

      public void writeFloat(float value)
      Set the provided float value at the writerIndex.
      Parameters:
      value - value to set

    • writeDouble

      public void writeDouble(double value)
      Set the provided double value at the writerIndex.
      Parameters:
      value - value to set

    • getBytes

      public void getBytes(long index, byte[] dst)
      Copy data from this ArrowBuf at a given index in into destination byte array.
      Parameters:
      index - starting index (0 based relative to the portion of memory) this ArrowBuf has access to
      dst - byte array to copy the data into

    • getBytes

      public void getBytes(long index, byte[] dst, int dstIndex, int length)
      Copy data from this ArrowBuf at a given index into destination byte array.
      Parameters:
      index - index (0 based relative to the portion of memory this ArrowBuf has access to)
      dst - byte array to copy the data into
      dstIndex - starting index in dst byte array to copy into
      length - length of data to copy from this ArrowBuf

    • setBytes

      public void setBytes(long index, byte[] src)
      Copy data from a given byte array into this ArrowBuf starting at a given index.
      Parameters:
      index - starting index (0 based relative to the portion of memory) this ArrowBuf has access to
      src - byte array to copy the data from

    • setBytes

      public void setBytes(long index, byte[] src, int srcIndex, long length)
      Copy data from a given byte array starting at the given source index into this ArrowBuf at a given index.
      Parameters:
      index - index (0 based relative to the portion of memory this ArrowBuf has access to)
      src - src byte array to copy the data from
      srcIndex - index in the byte array where the copy will start from
      length - length of data to copy from byte array

    • getBytes

      public void getBytes(long index, ByteBuffer dst)
      Copy data from this ArrowBuf at a given index into the destination ByteBuffer.
      Parameters:
      index - index (0 based relative to the portion of memory this ArrowBuf has access to)
      dst - dst ByteBuffer where the data will be copied into

    • setBytes

      public void setBytes(long index, ByteBuffer src)
      Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      src - src ByteBuffer where the data will be copied from

    • setBytes

      public void setBytes(long index, ByteBuffer src, int srcIndex, int length)
      Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer starting at a given srcIndex for a certain length.
      Parameters:
      index - index (0 based relative to the portion of memory this ArrowBuf has access to)
      src - src ByteBuffer where the data will be copied from
      srcIndex - starting index in the src ByteBuffer where the data copy will start from
      length - length of data to copy from src ByteBuffer

    • getBytes

      public void getBytes(long index, ArrowBuf dst, long dstIndex, int length)
      Copy a given length of data from this ArrowBuf starting at a given index into a dst ArrowBuf at dstIndex.
      Parameters:
      index - index (0 based relative to the portion of memory this ArrowBuf has access to)
      dst - dst ArrowBuf where the data will be copied into
      dstIndex - index (0 based relative to the portion of memory dst ArrowBuf has access to)
      length - length of data to copy

    • setBytes

      public void setBytes(long index, ArrowBuf src, long srcIndex, long length)
      Copy data from src ArrowBuf starting at index srcIndex into this ArrowBuf at given index.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      src - src ArrowBuf where the data will be copied from
      srcIndex - starting index in the src ArrowBuf where the copy will begin from
      length - length of data to copy from src ArrowBuf

    • setBytes

      public void setBytes(long index, ArrowBuf src)
      Copy readableBytes() number of bytes from src ArrowBuf starting from its readerIndex into this ArrowBuf starting at the given index.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      src - src ArrowBuf where the data will be copied from

    • setBytes

      public int setBytes(long index, InputStream in, int length) throws IOException
      Copy a certain length of bytes from given InputStream into this ArrowBuf at the provided index.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      in - src stream to copy from
      length - length of data to copy
      Returns:
      number of bytes copied from stream into ArrowBuf
      Throws:
      IOException - on failing to read from stream

    • getBytes

      public void getBytes(long index, OutputStream out, int length) throws IOException
      Copy a certain length of bytes from this ArrowBuf at a given index into the given OutputStream.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      out - dst stream to copy data into
      length - length of data to copy
      Throws:
      IOException - on failing to write to stream

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable

    • getPossibleMemoryConsumed

      public long getPossibleMemoryConsumed()
      Returns the possible memory consumed by this ArrowBuf in the worse case scenario. (not shared, connected to larger underlying buffer of allocated memory)
      Returns:
      Size in bytes.

    • getActualMemoryConsumed

      public long getActualMemoryConsumed()
      Return that is Accounted for by this buffer (and its potentially shared siblings within the context of the associated allocator).
      Returns:
      Size in bytes.

    • toHexString

      public String toHexString(long start, int length)
      Return the buffer's byte contents in the form of a hex dump.
      Parameters:
      start - the starting byte index
      length - how many bytes to log
      Returns:
      A hex dump in a String.

    • getId

      public long getId()
      Get the integer id assigned to this ArrowBuf for debugging purposes.
      Returns:
      integer id

    • print

      public void print(StringBuilder sb, int indent, org.apache.arrow.memory.BaseAllocator.Verbosity verbosity)
      Print information of this buffer into sb at the given indentation and verbosity level.

      It will include history if BaseAllocator.DEBUG is true and the verbosity.includeHistoricalLog are true.

    • print

      public void print(StringBuilder sb, int indent)
      Print detailed information of this buffer into sb.

      Most information will only be present if BaseAllocator.DEBUG is true.

    • readerIndex

      public long readerIndex()
      Get the index at which the next byte will be read from.
      Returns:
      reader index

    • writerIndex

      public long writerIndex()
      Get the index at which next byte will be written to.
      Returns:
      writer index

    • readerIndex

      public ArrowBuf readerIndex(long readerIndex)
      Set the reader index for this ArrowBuf.
      Parameters:
      readerIndex - new reader index
      Returns:
      this ArrowBuf

    • writerIndex

      public ArrowBuf writerIndex(long writerIndex)
      Set the writer index for this ArrowBuf.
      Parameters:
      writerIndex - new writer index
      Returns:
      this ArrowBuf

    • setZero

      public ArrowBuf setZero(long index, long length)
      Zero-out the bytes in this ArrowBuf starting at the given index for the given length.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      length - length of bytes to zero-out
      Returns:
      this ArrowBuf

    • setOne

      @Deprecated public ArrowBuf setOne(int index, int length)
      Deprecated.
      use setOne(long, long) instead.
      Sets all bits to one in the specified range.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      length - length of bytes to set.
      Returns:
      this ArrowBuf

    • setOne

      public ArrowBuf setOne(long index, long length)
      Sets all bits to one in the specified range.
      Parameters:
      index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
      length - length of bytes to set.
      Returns:
      this ArrowBuf

    • reallocIfNeeded

      public ArrowBuf reallocIfNeeded(long size)
      Returns this if size is less than capacity(), otherwise delegates to BufferManager.replace(ArrowBuf, long) to get a new buffer.

    • clear

      public ArrowBuf clear()