Package com.google.cloud.dataproc.v1

Class PySparkJob

  • All Implemented Interfaces:
    PySparkJobOrBuilder, com.google.protobuf.Message, com.google.protobuf.MessageLite, com.google.protobuf.MessageLiteOrBuilder, com.google.protobuf.MessageOrBuilder, Serializable
    public final class PySparkJob
extends com.google.protobuf.GeneratedMessageV3
implements PySparkJobOrBuilder
     A Dataproc job for running
 [Apache
 PySpark](https://spark.apache.org/docs/0.9.0/python-programming-guide.html)
 applications on YARN.
    Protobuf type google.cloud.dataproc.v1.PySparkJob
    See Also:
    Serialized Form

    • Field Detail

      • MAIN_PYTHON_FILE_URI_FIELD_NUMBER

        public static final int MAIN_PYTHON_FILE_URI_FIELD_NUMBER
        See Also:
        Constant Field Values

      • PYTHON_FILE_URIS_FIELD_NUMBER

        public static final int PYTHON_FILE_URIS_FIELD_NUMBER
        See Also:
        Constant Field Values

      • JAR_FILE_URIS_FIELD_NUMBER

        public static final int JAR_FILE_URIS_FIELD_NUMBER
        See Also:
        Constant Field Values

      • ARCHIVE_URIS_FIELD_NUMBER

        public static final int ARCHIVE_URIS_FIELD_NUMBER
        See Also:
        Constant Field Values

      • PROPERTIES_FIELD_NUMBER

        public static final int PROPERTIES_FIELD_NUMBER
        See Also:
        Constant Field Values

      • LOGGING_CONFIG_FIELD_NUMBER

        public static final int LOGGING_CONFIG_FIELD_NUMBER
        See Also:
        Constant Field Values

    • Method Detail

      • newInstance

        protected Object newInstance​(com.google.protobuf.GeneratedMessageV3.UnusedPrivateParameter unused)
        Overrides:
        newInstance in class com.google.protobuf.GeneratedMessageV3

      • getDescriptor

        public static final com.google.protobuf.Descriptors.Descriptor getDescriptor()

      • internalGetMapField

        protected com.google.protobuf.MapField internalGetMapField​(int number)
        Overrides:
        internalGetMapField in class com.google.protobuf.GeneratedMessageV3

      • internalGetFieldAccessorTable

        protected com.google.protobuf.GeneratedMessageV3.FieldAccessorTable internalGetFieldAccessorTable()
        Specified by:
        internalGetFieldAccessorTable in class com.google.protobuf.GeneratedMessageV3

      • getMainPythonFileUri

        public String getMainPythonFileUri()
         Required. The HCFS URI of the main Python file to use as the driver. Must
 be a .py file.
        string main_python_file_uri = 1 [(.google.api.field_behavior) = REQUIRED];
        Specified by:
        getMainPythonFileUri in interface PySparkJobOrBuilder
        Returns:
        The mainPythonFileUri.

      • getMainPythonFileUriBytes

        public com.google.protobuf.ByteString getMainPythonFileUriBytes()
         Required. The HCFS URI of the main Python file to use as the driver. Must
 be a .py file.
        string main_python_file_uri = 1 [(.google.api.field_behavior) = REQUIRED];
        Specified by:
        getMainPythonFileUriBytes in interface PySparkJobOrBuilder
        Returns:
        The bytes for mainPythonFileUri.

      • getArgsList

        public com.google.protobuf.ProtocolStringList getArgsList()
         Optional. The arguments to pass to the driver.  Do not include arguments,
 such as `--conf`, that can be set as job properties, since a collision may
 occur that causes an incorrect job submission.
        repeated string args = 2 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArgsList in interface PySparkJobOrBuilder
        Returns:
        A list containing the args.

      • getArgsCount

        public int getArgsCount()
         Optional. The arguments to pass to the driver.  Do not include arguments,
 such as `--conf`, that can be set as job properties, since a collision may
 occur that causes an incorrect job submission.
        repeated string args = 2 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArgsCount in interface PySparkJobOrBuilder
        Returns:
        The count of args.

      • getArgs

        public String getArgs​(int index)
         Optional. The arguments to pass to the driver.  Do not include arguments,
 such as `--conf`, that can be set as job properties, since a collision may
 occur that causes an incorrect job submission.
        repeated string args = 2 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArgs in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the element to return.
        Returns:
        The args at the given index.

      • getArgsBytes

        public com.google.protobuf.ByteString getArgsBytes​(int index)
         Optional. The arguments to pass to the driver.  Do not include arguments,
 such as `--conf`, that can be set as job properties, since a collision may
 occur that causes an incorrect job submission.
        repeated string args = 2 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArgsBytes in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the value to return.
        Returns:
        The bytes of the args at the given index.

      • getPythonFileUrisList

        public com.google.protobuf.ProtocolStringList getPythonFileUrisList()
         Optional. HCFS file URIs of Python files to pass to the PySpark
 framework. Supported file types: .py, .egg, and .zip.
        repeated string python_file_uris = 3 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPythonFileUrisList in interface PySparkJobOrBuilder
        Returns:
        A list containing the pythonFileUris.

      • getPythonFileUrisCount

        public int getPythonFileUrisCount()
         Optional. HCFS file URIs of Python files to pass to the PySpark
 framework. Supported file types: .py, .egg, and .zip.
        repeated string python_file_uris = 3 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPythonFileUrisCount in interface PySparkJobOrBuilder
        Returns:
        The count of pythonFileUris.

      • getPythonFileUris

        public String getPythonFileUris​(int index)
         Optional. HCFS file URIs of Python files to pass to the PySpark
 framework. Supported file types: .py, .egg, and .zip.
        repeated string python_file_uris = 3 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPythonFileUris in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the element to return.
        Returns:
        The pythonFileUris at the given index.

      • getPythonFileUrisBytes

        public com.google.protobuf.ByteString getPythonFileUrisBytes​(int index)
         Optional. HCFS file URIs of Python files to pass to the PySpark
 framework. Supported file types: .py, .egg, and .zip.
        repeated string python_file_uris = 3 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPythonFileUrisBytes in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the value to return.
        Returns:
        The bytes of the pythonFileUris at the given index.

      • getJarFileUrisList

        public com.google.protobuf.ProtocolStringList getJarFileUrisList()
         Optional. HCFS URIs of jar files to add to the CLASSPATHs of the
 Python driver and tasks.
        repeated string jar_file_uris = 4 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getJarFileUrisList in interface PySparkJobOrBuilder
        Returns:
        A list containing the jarFileUris.

      • getJarFileUrisCount

        public int getJarFileUrisCount()
         Optional. HCFS URIs of jar files to add to the CLASSPATHs of the
 Python driver and tasks.
        repeated string jar_file_uris = 4 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getJarFileUrisCount in interface PySparkJobOrBuilder
        Returns:
        The count of jarFileUris.

      • getJarFileUris

        public String getJarFileUris​(int index)
         Optional. HCFS URIs of jar files to add to the CLASSPATHs of the
 Python driver and tasks.
        repeated string jar_file_uris = 4 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getJarFileUris in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the element to return.
        Returns:
        The jarFileUris at the given index.

      • getJarFileUrisBytes

        public com.google.protobuf.ByteString getJarFileUrisBytes​(int index)
         Optional. HCFS URIs of jar files to add to the CLASSPATHs of the
 Python driver and tasks.
        repeated string jar_file_uris = 4 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getJarFileUrisBytes in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the value to return.
        Returns:
        The bytes of the jarFileUris at the given index.

      • getFileUrisList

        public com.google.protobuf.ProtocolStringList getFileUrisList()
         Optional. HCFS URIs of files to be placed in the working directory of
 each executor. Useful for naively parallel tasks.
        repeated string file_uris = 5 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getFileUrisList in interface PySparkJobOrBuilder
        Returns:
        A list containing the fileUris.

      • getFileUrisCount

        public int getFileUrisCount()
         Optional. HCFS URIs of files to be placed in the working directory of
 each executor. Useful for naively parallel tasks.
        repeated string file_uris = 5 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getFileUrisCount in interface PySparkJobOrBuilder
        Returns:
        The count of fileUris.

      • getFileUris

        public String getFileUris​(int index)
         Optional. HCFS URIs of files to be placed in the working directory of
 each executor. Useful for naively parallel tasks.
        repeated string file_uris = 5 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getFileUris in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the element to return.
        Returns:
        The fileUris at the given index.

      • getFileUrisBytes

        public com.google.protobuf.ByteString getFileUrisBytes​(int index)
         Optional. HCFS URIs of files to be placed in the working directory of
 each executor. Useful for naively parallel tasks.
        repeated string file_uris = 5 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getFileUrisBytes in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the value to return.
        Returns:
        The bytes of the fileUris at the given index.

      • getArchiveUrisList

        public com.google.protobuf.ProtocolStringList getArchiveUrisList()
         Optional. HCFS URIs of archives to be extracted into the working directory
 of each executor. Supported file types:
 .jar, .tar, .tar.gz, .tgz, and .zip.
        repeated string archive_uris = 6 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArchiveUrisList in interface PySparkJobOrBuilder
        Returns:
        A list containing the archiveUris.

      • getArchiveUrisCount

        public int getArchiveUrisCount()
         Optional. HCFS URIs of archives to be extracted into the working directory
 of each executor. Supported file types:
 .jar, .tar, .tar.gz, .tgz, and .zip.
        repeated string archive_uris = 6 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArchiveUrisCount in interface PySparkJobOrBuilder
        Returns:
        The count of archiveUris.

      • getArchiveUris

        public String getArchiveUris​(int index)
         Optional. HCFS URIs of archives to be extracted into the working directory
 of each executor. Supported file types:
 .jar, .tar, .tar.gz, .tgz, and .zip.
        repeated string archive_uris = 6 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArchiveUris in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the element to return.
        Returns:
        The archiveUris at the given index.

      • getArchiveUrisBytes

        public com.google.protobuf.ByteString getArchiveUrisBytes​(int index)
         Optional. HCFS URIs of archives to be extracted into the working directory
 of each executor. Supported file types:
 .jar, .tar, .tar.gz, .tgz, and .zip.
        repeated string archive_uris = 6 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getArchiveUrisBytes in interface PySparkJobOrBuilder
        Parameters:
        index - The index of the value to return.
        Returns:
        The bytes of the archiveUris at the given index.

      • getPropertiesCount

        public int getPropertiesCount()
        Description copied from interface: PySparkJobOrBuilder
         Optional. A mapping of property names to values, used to configure PySpark.
 Properties that conflict with values set by the Dataproc API may be
 overwritten. Can include properties set in
 /etc/spark/conf/spark-defaults.conf and classes in user code.
        map<string, string> properties = 7 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPropertiesCount in interface PySparkJobOrBuilder

      • containsProperties

        public boolean containsProperties​(String key)
         Optional. A mapping of property names to values, used to configure PySpark.
 Properties that conflict with values set by the Dataproc API may be
 overwritten. Can include properties set in
 /etc/spark/conf/spark-defaults.conf and classes in user code.
        map<string, string> properties = 7 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        containsProperties in interface PySparkJobOrBuilder

      • getPropertiesMap

        public Map<String,​String> getPropertiesMap()
         Optional. A mapping of property names to values, used to configure PySpark.
 Properties that conflict with values set by the Dataproc API may be
 overwritten. Can include properties set in
 /etc/spark/conf/spark-defaults.conf and classes in user code.
        map<string, string> properties = 7 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPropertiesMap in interface PySparkJobOrBuilder

      • getPropertiesOrDefault

        public String getPropertiesOrDefault​(String key,
                                     String defaultValue)
         Optional. A mapping of property names to values, used to configure PySpark.
 Properties that conflict with values set by the Dataproc API may be
 overwritten. Can include properties set in
 /etc/spark/conf/spark-defaults.conf and classes in user code.
        map<string, string> properties = 7 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPropertiesOrDefault in interface PySparkJobOrBuilder

      • getPropertiesOrThrow

        public String getPropertiesOrThrow​(String key)
         Optional. A mapping of property names to values, used to configure PySpark.
 Properties that conflict with values set by the Dataproc API may be
 overwritten. Can include properties set in
 /etc/spark/conf/spark-defaults.conf and classes in user code.
        map<string, string> properties = 7 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getPropertiesOrThrow in interface PySparkJobOrBuilder

      • hasLoggingConfig

        public boolean hasLoggingConfig()
         Optional. The runtime log config for job execution.
        .google.cloud.dataproc.v1.LoggingConfig logging_config = 8 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        hasLoggingConfig in interface PySparkJobOrBuilder
        Returns:
        Whether the loggingConfig field is set.

      • getLoggingConfig

        public LoggingConfig getLoggingConfig()
         Optional. The runtime log config for job execution.
        .google.cloud.dataproc.v1.LoggingConfig logging_config = 8 [(.google.api.field_behavior) = OPTIONAL];
        Specified by:
        getLoggingConfig in interface PySparkJobOrBuilder
        Returns:
        The loggingConfig.

      • isInitialized

        public final boolean isInitialized()
        Specified by:
        isInitialized in interface com.google.protobuf.MessageLiteOrBuilder
        Overrides:
        isInitialized in class com.google.protobuf.GeneratedMessageV3

      • writeTo

        public void writeTo​(com.google.protobuf.CodedOutputStream output)
             throws IOException
        Specified by:
        writeTo in interface com.google.protobuf.MessageLite
        Overrides:
        writeTo in class com.google.protobuf.GeneratedMessageV3
        Throws:
        IOException

      • getSerializedSize

        public int getSerializedSize()
        Specified by:
        getSerializedSize in interface com.google.protobuf.MessageLite
        Overrides:
        getSerializedSize in class com.google.protobuf.GeneratedMessageV3

      • equals

        public boolean equals​(Object obj)
        Specified by:
        equals in interface com.google.protobuf.Message
        Overrides:
        equals in class com.google.protobuf.AbstractMessage

      • hashCode

        public int hashCode()
        Specified by:
        hashCode in interface com.google.protobuf.Message
        Overrides:
        hashCode in class com.google.protobuf.AbstractMessage

      • parseFrom

        public static PySparkJob parseFrom​(ByteBuffer data)
                            throws com.google.protobuf.InvalidProtocolBufferException
        Throws:
        com.google.protobuf.InvalidProtocolBufferException

      • parseFrom

        public static PySparkJob parseFrom​(ByteBuffer data,
                                   com.google.protobuf.ExtensionRegistryLite extensionRegistry)
                            throws com.google.protobuf.InvalidProtocolBufferException
        Throws:
        com.google.protobuf.InvalidProtocolBufferException

      • parseFrom

        public static PySparkJob parseFrom​(com.google.protobuf.ByteString data)
                            throws com.google.protobuf.InvalidProtocolBufferException
        Throws:
        com.google.protobuf.InvalidProtocolBufferException

      • parseFrom

        public static PySparkJob parseFrom​(com.google.protobuf.ByteString data,
                                   com.google.protobuf.ExtensionRegistryLite extensionRegistry)
                            throws com.google.protobuf.InvalidProtocolBufferException
        Throws:
        com.google.protobuf.InvalidProtocolBufferException

      • parseFrom

        public static PySparkJob parseFrom​(byte[] data)
                            throws com.google.protobuf.InvalidProtocolBufferException
        Throws:
        com.google.protobuf.InvalidProtocolBufferException

      • parseFrom

        public static PySparkJob parseFrom​(byte[] data,
                                   com.google.protobuf.ExtensionRegistryLite extensionRegistry)
                            throws com.google.protobuf.InvalidProtocolBufferException
        Throws:
        com.google.protobuf.InvalidProtocolBufferException

      • parseFrom

        public static PySparkJob parseFrom​(com.google.protobuf.CodedInputStream input,
                                   com.google.protobuf.ExtensionRegistryLite extensionRegistry)
                            throws IOException
        Throws:
        IOException

      • newBuilderForType

        public PySparkJob.Builder newBuilderForType()
        Specified by:
        newBuilderForType in interface com.google.protobuf.Message
        Specified by:
        newBuilderForType in interface com.google.protobuf.MessageLite

      • toBuilder

        public PySparkJob.Builder toBuilder()
        Specified by:
        toBuilder in interface com.google.protobuf.Message
        Specified by:
        toBuilder in interface com.google.protobuf.MessageLite

      • newBuilderForType

        protected PySparkJob.Builder newBuilderForType​(com.google.protobuf.GeneratedMessageV3.BuilderParent parent)
        Specified by:
        newBuilderForType in class com.google.protobuf.GeneratedMessageV3

      • getDefaultInstance

        public static PySparkJob getDefaultInstance()

      • parser

        public static com.google.protobuf.Parser<PySparkJob> parser()

      • getParserForType

        public com.google.protobuf.Parser<PySparkJob> getParserForType()
        Specified by:
        getParserForType in interface com.google.protobuf.Message
        Specified by:
        getParserForType in interface com.google.protobuf.MessageLite
        Overrides:
        getParserForType in class com.google.protobuf.GeneratedMessageV3

      • getDefaultInstanceForType

        public PySparkJob getDefaultInstanceForType()
        Specified by:
        getDefaultInstanceForType in interface com.google.protobuf.MessageLiteOrBuilder
        Specified by:
        getDefaultInstanceForType in interface com.google.protobuf.MessageOrBuilder