Package com.google.cloud.bigquery.storage.v1

Class JsonStreamWriter

java.lang.Object

com.google.cloud.bigquery.storage.v1.JsonStreamWriter

All Implemented Interfaces:

AutoCloseable

public class JsonStreamWriter
extends Object
implements AutoCloseable

A StreamWriter that can write JSON data (JSONObjects) to BigQuery tables. The JsonStreamWriter is built on top of a StreamWriter, and it simply converts all JSON data to protobuf messages then calls StreamWriter's append() method to write to BigQuery tables. It maintains all StreamWriter functions, but also provides an additional feature: schema update support, where if the BigQuery table schema is updated, users will be able to ingest data on the new schema after some time (in order of minutes).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	JsonStreamWriter.Builder

Method Summary

Modifier and Type	Method	Description
com.google.api.core.ApiFuture<AppendRowsResponse>	append(org.json.JSONArray jsonArr)	Writes a JSONArray that contains JSONObjects to the BigQuery table by first converting the JSON data to protobuf messages, then using StreamWriter's append() to write the data at current end of stream.
com.google.api.core.ApiFuture<AppendRowsResponse>	append(org.json.JSONArray jsonArr, long offset)	Writes a JSONArray that contains JSONObjects to the BigQuery table by first converting the JSON data to protobuf messages, then using StreamWriter's append() to write the data at the specified offset.
void	close()
com.google.protobuf.Descriptors.Descriptor	getDescriptor()	Gets current descriptor
long	getInflightWaitSeconds()	Returns the wait of a request in Client side before sending to the Server.
String	getLocation()	Gets the location of the destination
Map<String,AppendRowsRequest.MissingValueInterpretation>	getMissingValueInterpretationMap()
String	getStreamName()
String	getWriterId()
boolean	isClosed()
boolean	isUserClosed()
static JsonStreamWriter.Builder	newBuilder(String streamOrTableName, BigQueryWriteClient client)	newBuilder that constructs a JsonStreamWriter builder with TableSchema being initialized by StreamWriter by default.
static JsonStreamWriter.Builder	newBuilder(String streamOrTableName, TableSchema tableSchema)	newBuilder that constructs a JsonStreamWriter builder with BigQuery client being initialized by StreamWriter by default.
static JsonStreamWriter.Builder	newBuilder(String streamOrTableName, TableSchema tableSchema, BigQueryWriteClient client)	newBuilder that constructs a JsonStreamWriter builder.
void	setMissingValueInterpretationMap(Map<String,AppendRowsRequest.MissingValueInterpretation> missingValueInterpretationMap)	Sets the missing value interpretation map for the JsonStreamWriter.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

append

public com.google.api.core.ApiFuture<AppendRowsResponse> append(org.json.JSONArray jsonArr)
                                                         throws IOException,
                                                                com.google.protobuf.Descriptors.DescriptorValidationException

Writes a JSONArray that contains JSONObjects to the BigQuery table by first converting the JSON data to protobuf messages, then using StreamWriter's append() to write the data at current end of stream. If there is a schema update, the current StreamWriter is closed. A new StreamWriter is created with the updated TableSchema.

Parameters:

jsonArr - The JSON array that contains JSONObjects to be written

Returns:

ApiFuture<AppendRowsResponse> returns an AppendRowsResponse message wrapped in an ApiFuture

Throws:

IOException

com.google.protobuf.Descriptors.DescriptorValidationException

append

public com.google.api.core.ApiFuture<AppendRowsResponse> append(org.json.JSONArray jsonArr,
                                                                long offset)
                                                         throws IOException,
                                                                com.google.protobuf.Descriptors.DescriptorValidationException

Writes a JSONArray that contains JSONObjects to the BigQuery table by first converting the JSON data to protobuf messages, then using StreamWriter's append() to write the data at the specified offset. If there is a schema update, the current StreamWriter is closed. A new StreamWriter is created with the updated TableSchema.

Parameters:

jsonArr - The JSON array that contains JSONObjects to be written

offset - Offset for deduplication

Returns:

ApiFuture<AppendRowsResponse> returns an AppendRowsResponse message wrapped in an ApiFuture

Throws:

IOException

com.google.protobuf.Descriptors.DescriptorValidationException

getStreamName

public String getStreamName()

getWriterId

public String getWriterId()

Returns:

A unique Id for this writer.

getDescriptor

public com.google.protobuf.Descriptors.Descriptor getDescriptor()

Gets current descriptor

Returns:

Descriptor

getLocation

public String getLocation()

Gets the location of the destination

Returns:

Descriptor

getInflightWaitSeconds

public long getInflightWaitSeconds()

Returns the wait of a request in Client side before sending to the Server. Request could wait in Client because it reached the client side inflight request limit (adjustable when constructing the Writer). The value is the wait time for the last sent request. A constant high wait value indicates a need for more throughput, you can create a new Stream for to increase the throughput in exclusive stream case, or create a new Writer in the default stream case.

setMissingValueInterpretationMap

public void setMissingValueInterpretationMap(Map<String,AppendRowsRequest.MissingValueInterpretation> missingValueInterpretationMap)

Sets the missing value interpretation map for the JsonStreamWriter. The input missingValueInterpretationMap is used for all append requests unless otherwise changed.

Parameters:

missingValueInterpretationMap - the missing value interpretation map used by the JsonStreamWriter.

getMissingValueInterpretationMap

public Map<String,AppendRowsRequest.MissingValueInterpretation> getMissingValueInterpretationMap()

Returns:

the missing value interpretation map used for the writer.

newBuilder

public static JsonStreamWriter.Builder newBuilder(String streamOrTableName,
                                                  TableSchema tableSchema)

newBuilder that constructs a JsonStreamWriter builder with BigQuery client being initialized by StreamWriter by default.

The table schema passed in will be updated automatically when there is a schema update event. When used for Writer creation, it should be the latest schema. So when you are trying to reuse a stream, you should use Builder newBuilder( String streamOrTableName, BigQueryWriteClient client) instead, so the created Writer will be based on a fresh schema.

Parameters:

streamOrTableName - name of the stream that must follow "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+" or table name "projects/[^/]+/datasets/[^/]+/tables/[^/]+"

tableSchema - The schema of the table when the stream was created, which is passed back through WriteStream

Returns:

Builder

newBuilder

public static JsonStreamWriter.Builder newBuilder(String streamOrTableName,
                                                  TableSchema tableSchema,
                                                  BigQueryWriteClient client)

newBuilder that constructs a JsonStreamWriter builder.

The table schema passed in will be updated automatically when there is a schema update event. When used for Writer creation, it should be the latest schema. So when you are trying to reuse a stream, you should use Builder newBuilder( String streamOrTableName, BigQueryWriteClient client) instead, so the created Writer will be based on a fresh schema.

Parameters:

streamOrTableName - name of the stream that must follow "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+"

tableSchema - The schema of the table when the stream was created, which is passed back through WriteStream

client -

Returns:

Builder

newBuilder

public static JsonStreamWriter.Builder newBuilder(String streamOrTableName,
                                                  BigQueryWriteClient client)

newBuilder that constructs a JsonStreamWriter builder with TableSchema being initialized by StreamWriter by default.

Parameters:

streamOrTableName - name of the stream that must follow "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+"

client - BigQueryWriteClient

Returns:

Builder

close

public void close()

Specified by:

close in interface AutoCloseable

isClosed

public boolean isClosed()

Returns:

if a Json writer can no longer be used for writing. It is due to either the JsonStreamWriter is explicitly closed or the underlying connection is broken when connection pool is not used. Client should recreate JsonStreamWriter in this case.

isUserClosed

public boolean isUserClosed()

Returns:

if user explicitly closed the writer.