public class MessageSerializer extends Object
For schema messages, the serialization is simply the FB serialized Schema.
For RecordBatch messages the serialization is: 1. 4 byte little endian batch metadata header 2. FB serialized RowBatch 3. Padding to align to 8 byte boundary. 4. serialized RowBatch buffers.
Modifier and Type | Field and Description |
---|---|
static int |
IPC_CONTINUATION_TOKEN |
Constructor and Description |
---|
MessageSerializer() |
Modifier and Type | Method and Description |
---|---|
static int |
bytesToInt(byte[] bytes)
Convert an array of 4 bytes in little-endian to an native-endian i32 value.
|
static ArrowDictionaryBatch |
deserializeDictionaryBatch(Message message,
ArrowBuf bodyBuffer)
Deserializes an ArrowDictionaryBatch from a dictionary batch Message and data in an ArrowBuf.
|
static ArrowDictionaryBatch |
deserializeDictionaryBatch(MessageMetadataResult message,
ArrowBuf bodyBuffer)
Deserializes an ArrowDictionaryBatch from a dictionary batch Message and data in an ArrowBuf.
|
static ArrowDictionaryBatch |
deserializeDictionaryBatch(ReadChannel in,
ArrowBlock block,
BufferAllocator alloc)
Deserializes a DictionaryBatch knowing the size of the entire message up front.
|
static ArrowDictionaryBatch |
deserializeDictionaryBatch(ReadChannel in,
BufferAllocator allocator)
Deserializes an ArrowDictionaryBatch read from the input channel.
|
static ArrowMessage |
deserializeMessageBatch(MessageChannelReader reader)
Deserialize a message that is either an ArrowDictionaryBatch or ArrowRecordBatch.
|
static ArrowMessage |
deserializeMessageBatch(ReadChannel in,
BufferAllocator alloc)
Deserialize a message that is either an ArrowDictionaryBatch or ArrowRecordBatch.
|
static ArrowRecordBatch |
deserializeRecordBatch(Message recordBatchMessage,
ArrowBuf bodyBuffer)
Deserializes an ArrowRecordBatch from a record batch message and data in an ArrowBuf.
|
static ArrowRecordBatch |
deserializeRecordBatch(MessageMetadataResult serializedMessage,
ArrowBuf underlying)
Reads a record batch based on the metadata in serializedMessage and the underlying data buffer.
|
static ArrowRecordBatch |
deserializeRecordBatch(ReadChannel in,
ArrowBlock block,
BufferAllocator alloc)
Deserializes an ArrowRecordBatch knowing the size of the entire message up front.
|
static ArrowRecordBatch |
deserializeRecordBatch(ReadChannel in,
BufferAllocator allocator)
Deserializes an ArrowRecordBatch read from the input channel.
|
static ArrowRecordBatch |
deserializeRecordBatch(RecordBatch recordBatchFB,
ArrowBuf body)
Deserializes an ArrowRecordBatch given the Flatbuffer metadata and in-memory body.
|
static Schema |
deserializeSchema(Message schemaMessage)
Deserializes an Arrow Schema object from a schema message.
|
static Schema |
deserializeSchema(MessageMetadataResult message)
Deserializes an Arrow Schema object from a
MessageMetadataResult . |
static Schema |
deserializeSchema(ReadChannel in)
Deserializes an Arrow Schema read from the input channel.
|
static void |
intToBytes(int value,
byte[] bytes)
Convert an integer to a little endian 4 byte array.
|
static void |
longToBytes(long value,
byte[] bytes)
Convert a long to a little-endian 8 byte array.
|
static MessageMetadataResult |
readMessage(ReadChannel in)
Read a Message from the input channel and return a MessageMetadataResult that contains the
Message metadata, buffer containing the serialized Message metadata as read, and length of the
Message in bytes.
|
static ArrowBuf |
readMessageBody(ReadChannel in,
long bodyLength,
BufferAllocator allocator)
Read a Message body from the in channel into an ArrowBuf.
|
static ArrowBlock |
serialize(WriteChannel out,
ArrowDictionaryBatch batch) |
static ArrowBlock |
serialize(WriteChannel out,
ArrowDictionaryBatch batch,
IpcOption option)
Serializes a dictionary ArrowRecordBatch.
|
static ArrowBlock |
serialize(WriteChannel out,
ArrowRecordBatch batch)
Serializes an ArrowRecordBatch.
|
static ArrowBlock |
serialize(WriteChannel out,
ArrowRecordBatch batch,
IpcOption option)
Serializes an ArrowRecordBatch.
|
static long |
serialize(WriteChannel out,
Schema schema)
Serialize a schema object.
|
static long |
serialize(WriteChannel out,
Schema schema,
IpcOption option)
Serialize a schema object.
|
static ByteBuffer |
serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder,
byte headerType,
int headerOffset,
long bodyLength)
Deprecated.
|
static ByteBuffer |
serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder,
byte headerType,
int headerOffset,
long bodyLength,
IpcOption writeOption)
Serializes a message header.
|
static ByteBuffer |
serializeMetadata(ArrowMessage message)
Deprecated.
|
static ByteBuffer |
serializeMetadata(ArrowMessage message,
IpcOption writeOption)
Returns the serialized form of
RecordBatch wrapped in a Message . |
static ByteBuffer |
serializeMetadata(Schema schema)
Deprecated.
|
static ByteBuffer |
serializeMetadata(Schema schema,
IpcOption writeOption)
Returns the serialized flatbuffer bytes of the schema wrapped in a message table.
|
static long |
writeBatchBuffers(WriteChannel out,
ArrowRecordBatch batch)
Write the Arrow buffers of the record batch to the output channel.
|
static int |
writeMessageBuffer(WriteChannel out,
int messageLength,
ByteBuffer messageBuffer) |
static int |
writeMessageBuffer(WriteChannel out,
int messageLength,
ByteBuffer messageBuffer,
IpcOption option)
Write the serialized Message metadata, prefixed by the length, to the output Channel.
|
public static final int IPC_CONTINUATION_TOKEN
public static int bytesToInt(byte[] bytes)
bytes
- byte array with minimum length of 4 in little-endianpublic static void intToBytes(int value, byte[] bytes)
value
- integer value inputbytes
- existing byte array with minimum length of 4 to contain the conversion outputpublic static void longToBytes(long value, byte[] bytes)
value
- long value inputbytes
- existing byte array with minimum length of 8 to contain the conversion outputpublic static int writeMessageBuffer(WriteChannel out, int messageLength, ByteBuffer messageBuffer) throws IOException
IOException
public static int writeMessageBuffer(WriteChannel out, int messageLength, ByteBuffer messageBuffer, IpcOption option) throws IOException
out
- Output ChannelmessageLength
- Number of bytes in the message buffer, written as little Endian prefixmessageBuffer
- Message metadata buffer to be written, this does not include any
message body data which should be subsequently written to the Channeloption
- IPC write optionsIOException
- on errorpublic static long serialize(WriteChannel out, Schema schema) throws IOException
IOException
public static long serialize(WriteChannel out, Schema schema, IpcOption option) throws IOException
out
- where to write the schemaschema
- the object to serialize to outIOException
- if something went wrong@Deprecated public static ByteBuffer serializeMetadata(Schema schema)
public static ByteBuffer serializeMetadata(Schema schema, IpcOption writeOption)
public static Schema deserializeSchema(Message schemaMessage)
schemaMessage
- a Message of type MessageHeader.Schemapublic static Schema deserializeSchema(ReadChannel in) throws IOException
in
- the channel to deserialize fromIOException
- if something went wrongpublic static Schema deserializeSchema(MessageMetadataResult message)
MessageMetadataResult
. Format is from serialize().message
- a Message of type MessageHeader.Schemapublic static ArrowBlock serialize(WriteChannel out, ArrowRecordBatch batch) throws IOException
IOException
public static ArrowBlock serialize(WriteChannel out, ArrowRecordBatch batch, IpcOption option) throws IOException
out
- where to write the batchbatch
- the object to serialize to outIOException
- if something went wrongpublic static long writeBatchBuffers(WriteChannel out, ArrowRecordBatch batch) throws IOException
out
- the output channel to write the buffers tobatch
- an ArrowRecordBatch containing buffers to be writtenIOException
- on error@Deprecated public static ByteBuffer serializeMetadata(ArrowMessage message)
RecordBatch
wrapped in a Message
.public static ByteBuffer serializeMetadata(ArrowMessage message, IpcOption writeOption)
RecordBatch
wrapped in a Message
.public static ArrowRecordBatch deserializeRecordBatch(Message recordBatchMessage, ArrowBuf bodyBuffer) throws IOException
recordBatchMessage
- a Message of type MessageHeader.RecordBatchbodyBuffer
- Arrow buffer containing the RecordBatch dataIOException
- if something went wrongpublic static ArrowRecordBatch deserializeRecordBatch(ReadChannel in, BufferAllocator allocator) throws IOException
in
- Channel to read a RecordBatch message and data fromallocator
- BufferAllocator to allocate an Arrow buffer to read message body dataIOException
- on errorpublic static ArrowRecordBatch deserializeRecordBatch(ReadChannel in, ArrowBlock block, BufferAllocator alloc) throws IOException
in
- the channel to deserialize fromblock
- the object to deserialize toalloc
- to allocate buffersIOException
- if something went wrongpublic static ArrowRecordBatch deserializeRecordBatch(RecordBatch recordBatchFB, ArrowBuf body) throws IOException
recordBatchFB
- Deserialized FlatBuffer record batchbody
- Read body of the record batchIOException
- on errorpublic static ArrowRecordBatch deserializeRecordBatch(MessageMetadataResult serializedMessage, ArrowBuf underlying) throws IOException
IOException
public static ArrowBlock serialize(WriteChannel out, ArrowDictionaryBatch batch) throws IOException
IOException
public static ArrowBlock serialize(WriteChannel out, ArrowDictionaryBatch batch, IpcOption option) throws IOException
out
- where to serializebatch
- the batch to serializeoption
- options for IPCIOException
- if something went wrongpublic static ArrowDictionaryBatch deserializeDictionaryBatch(Message message, ArrowBuf bodyBuffer) throws IOException
message
- a message of type MessageHeader.DictionaryBatchbodyBuffer
- Arrow buffer containing the DictionaryBatch data
of type MessageHeader.DictionaryBatchIOException
- if something went wrongpublic static ArrowDictionaryBatch deserializeDictionaryBatch(MessageMetadataResult message, ArrowBuf bodyBuffer) throws IOException
message
- a message of type MessageHeader.DictionaryBatchbodyBuffer
- Arrow buffer containing the DictionaryBatch data
of type MessageHeader.DictionaryBatchIOException
- if something went wrongpublic static ArrowDictionaryBatch deserializeDictionaryBatch(ReadChannel in, BufferAllocator allocator) throws IOException
in
- Channel to read a DictionaryBatch message and data fromallocator
- BufferAllocator to allocate an Arrow buffer to read message body dataIOException
- on errorpublic static ArrowDictionaryBatch deserializeDictionaryBatch(ReadChannel in, ArrowBlock block, BufferAllocator alloc) throws IOException
in
- where to read fromblock
- block metadata for deserializingalloc
- to allocate new buffersIOException
- if something went wrongpublic static ArrowMessage deserializeMessageBatch(MessageChannelReader reader) throws IOException
reader
- MessageChannelReader to read a sequence of messages from a ReadChannelIOException
- if the message is not an ArrowDictionaryBatch or ArrowRecordBatchpublic static ArrowMessage deserializeMessageBatch(ReadChannel in, BufferAllocator alloc) throws IOException
in
- ReadChannel to read messages fromalloc
- Allocator for message dataIOException
- if the message is not an ArrowDictionaryBatch or ArrowRecordBatch@Deprecated public static ByteBuffer serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder, byte headerType, int headerOffset, long bodyLength)
public static ByteBuffer serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder, byte headerType, int headerOffset, long bodyLength, IpcOption writeOption)
builder
- to write the flatbuf toheaderType
- headerType fieldheaderOffset
- header offset fieldbodyLength
- body length fieldwriteOption
- IPC write optionspublic static MessageMetadataResult readMessage(ReadChannel in) throws IOException
in
- ReadChannel to read messages fromIOException
- on errorpublic static ArrowBuf readMessageBody(ReadChannel in, long bodyLength, BufferAllocator allocator) throws IOException
in
- ReadChannel to read message body frombodyLength
- Length in bytes of the message body to readallocator
- Allocate the ArrowBuf to contain message body dataIOException
- on errorCopyright © 2024 The Apache Software Foundation. All rights reserved.