Arrow Flight RPC

Arrow Flight is an RPC framework for efficient transfer of Flight data over the network.

See also

Flight protocol documentation

Documentation of the Flight protocol, including how to use Flight conceptually.

Flight API documentation

Python API documentation listing all of the various client and server classes.

Python Cookbook

Recipes for using Arrow Flight in Python.

Writing a Flight Service

Servers are subclasses of FlightServerBase. To implement individual RPCs, override the RPC methods on this class.

import pyarrow.flight as flight

class MyFlightServer(flight.FlightServerBase):
    def list_flights(self, context, criteria):
        info = flight.FlightInfo(...)
        yield info

Each RPC method always takes a ServerCallContext for common parameters. To indicate failure, raise an exception; Flight-specific errors can be indicated by raising one of the subclasses of FlightError.

To start a server, create a Location to specify where to listen, and create an instance of the server. (A string will be converted into a location.) This will start the server, but won’t block the rest of the program. Call FlightServerBase.serve() to block until the server stops.

# Listen to all interfaces on a free port
server = MyFlightServer("grpc://0.0.0.0:0")

print("Server listening on port", server.port)
server.serve()

Using the Flight Client

To connect to a Flight service, call pyarrow.flight.connect() with a location.

Cancellation and Timeouts

When making a call, clients can optionally provide FlightCallOptions. This allows clients to set a timeout on calls or provide custom HTTP headers, among other features. Also, some objects returned by client RPC calls expose a cancel method which allows terminating a call early.

On the server side, timeouts are transparent. For cancellation, the server needs to manually poll ServerCallContext.is_cancelled() to check if the client has cancelled the call, and if so, break out of any processing the server is currently doing.

Enabling TLS

TLS can be enabled when setting up a server by providing a certificate and key pair to FlightServerBase.

On the client side, use Location.for_grpc_tls() to construct the Location to listen on.

Enabling Authentication

Warning

Authentication is insecure without enabling TLS.

Handshake-based authentication can be enabled by implementing ServerAuthHandler. Authentication consists of two parts: on initial client connection, the server and client authentication implementations can perform any negotiation needed; then, on each RPC thereafter, the client provides a token. The server authentication handler validates the token and provides the identity of the client. This identity can be obtained from the ServerCallContext.

Custom Middleware

Servers and clients support custom middleware (or interceptors) that are called on every request and can modify the request in a limited fashion. These can be implemented by subclassing ServerMiddleware and ClientMiddleware, then providing them when creating the client or server.

Middleware are fairly limited, but they can add headers to a request/response. On the server, they can inspect incoming headers and fail the request; hence, they can be used to implement custom authentication methods.

Flight best practices