Type Alias DataTaskDatasetState

Represents the execution state of a single dataset within a task run, including full load, CDC, and streaming progress.

type DataTaskDatasetState = {
    cdcStatus?: {
        ddlCount?: number;
        deleteCount?: number;
        incomingChangesCount?: number;
        insertCount?: number;
        lastProcessed?: string;
        message?: string;
        state?:
            | "QUEUED"
            | "PROCESSING"
            | "ACCUMULATING_CHANGES"
            | "COMPLETED"
            | "ERROR";
        totalProcessedCount?: number;
        unoptimizedRecordsCount?: number;
        updateCount?: number;
    };
    dataReadiness?: "READY"
    | "NOT_READY"
    | "ERROR";
    datasetId?: string;
    fullLoad?: {
        cachedChangesCount?: number;
        duration?: string;
        endTime?: string;
        failedRecordsCount?: number;
        fileStats?: FileStatistics;
        message?: string;
        startTime?: string;
        state?: "QUEUED" | "LOADING" | "COMPLETED" | "ERROR";
        totalProcessedCount?: number;
    };
    lastBatchOfChanges?: {
        duration?: string;
        endTime?: string;
        fileStats?: FileStatistics;
        message?: string;
        operationStats?: OperationStatistics;
        startTime?: string;
        state?: "QUEUED"
        | "PROCESSING"
        | "COMPLETED"
        | "ERROR";
        throughputInRecordsPerSecond?: number;
        totalProcessedCount?: number;
    };
    name?: string;
    sourceName?: string;
    streaming?: {
        lastProcessed?: string;
        message?: string;
        parseIssueCount?: number;
        recordsFilteredCount?: number;
        recordsWrittenCount?: number;
        state?: "QUEUED"
        | "RUNNING"
        | "ERROR";
        totalProcessedCount?: number;
        unoptimizedRecordsCount?: number;
    };
}
Index
cdcStatus? dataReadiness? datasetId? fullLoad? lastBatchOfChanges? name? sourceName? streaming?
cdcStatus?: {
    ddlCount?: number;
    deleteCount?: number;
    incomingChangesCount?: number;
    insertCount?: number;
    lastProcessed?: string;
    message?: string;
    state?:
        | "QUEUED"
        | "PROCESSING"
        | "ACCUMULATING_CHANGES"
        | "COMPLETED"
        | "ERROR";
    totalProcessedCount?: number;
    unoptimizedRecordsCount?: number;
    updateCount?: number;
}

Change Data Capture state for the dataset, tracking incremental changes applied and any errors.

Type Declaration

  • OptionalddlCount?: number

    Number of DDL statements executed during the last run

  • OptionaldeleteCount?: number

    delete portion of totalProcessedCount. Only available for some task types

  • OptionalincomingChangesCount?: number

    Number of incoming changes for the dataset. Only relevant for 'Iceberg Storage' and 'Streaming Transform' tasks.

  • OptionalinsertCount?: number

    Insert portion of totalProcessedCount. Only available for some task types

  • OptionallastProcessed?: string
  • Optionalmessage?: string
  • Optionalstate?: "QUEUED" | "PROCESSING" | "ACCUMULATING_CHANGES" | "COMPLETED" | "ERROR"
  • OptionaltotalProcessedCount?: number

    Total number of changes/DMLs applied to the dataset

  • OptionalunoptimizedRecordsCount?: number

    Number of records that are queryable via the view, but not yet merged into optimized Iceberg partitions. Only relevant for 'Iceberg Storage' and 'Streaming Transform' tasks.

  • OptionalupdateCount?: number

    update portion of totalProcessedCount. Only available for some task types

dataReadiness?: "READY" | "NOT_READY" | "ERROR"

Is the data ready for use?

datasetId?: string

Id of the dataset

fullLoad?: {
    cachedChangesCount?: number;
    duration?: string;
    endTime?: string;
    failedRecordsCount?: number;
    fileStats?: FileStatistics;
    message?: string;
    startTime?: string;
    state?: "QUEUED" | "LOADING" | "COMPLETED" | "ERROR";
    totalProcessedCount?: number;
}

Type Declaration

  • OptionalcachedChangesCount?: number

    Number of changes captured and cached during full load (CDC landing/replication tasks only)

  • Optionalduration?: string

    Duration in HH:MM:SS format (hours:minutes:seconds)

  • OptionalendTime?: string
  • OptionalfailedRecordsCount?: number

    Number of records that failed to load (currently only for knowledge marts)

  • OptionalfileStats?: FileStatistics

    Statistics for file-based tasks.

  • Optionalmessage?: string
  • OptionalstartTime?: string
  • Optionalstate?: "QUEUED" | "LOADING" | "COMPLETED" | "ERROR"
  • OptionaltotalProcessedCount?: number

    Number of records (or docs in knowledge marts) were loaded.

lastBatchOfChanges?: {
    duration?: string;
    endTime?: string;
    fileStats?: FileStatistics;
    message?: string;
    operationStats?: OperationStatistics;
    startTime?: string;
    state?: "QUEUED" | "PROCESSING" | "COMPLETED" | "ERROR";
    throughputInRecordsPerSecond?: number;
    totalProcessedCount?: number;
}

Type Declaration

  • Optionalduration?: string

    Duration in HH:MM:SS format (hours:minutes:seconds)

  • OptionalendTime?: string
  • OptionalfileStats?: FileStatistics

    Statistics for file-based tasks.

  • Optionalmessage?: string
  • OptionaloperationStats?: OperationStatistics

    Breakdown of operations for record-oriented tasks.

  • OptionalstartTime?: string
  • Optionalstate?: "QUEUED" | "PROCESSING" | "COMPLETED" | "ERROR"
  • OptionalthroughputInRecordsPerSecond?: number

    Throughput in records per second

  • OptionaltotalProcessedCount?: number
name?: string

Name of the dataset

sourceName?: string

Original name of the dataset, relevant only for data movement tasks

streaming?: {
    lastProcessed?: string;
    message?: string;
    parseIssueCount?: number;
    recordsFilteredCount?: number;
    recordsWrittenCount?: number;
    state?: "QUEUED" | "RUNNING" | "ERROR";
    totalProcessedCount?: number;
    unoptimizedRecordsCount?: number;
}

Real-time streaming state for the dataset, tracking record processing and transformation statistics.

Type Declaration

  • OptionallastProcessed?: string

    Timestamp of the latest source record inserted into the target dataset.

  • Optionalmessage?: string
  • OptionalparseIssueCount?: number

    Number of records that had parsing issues

  • OptionalrecordsFilteredCount?: number

    Total number of records filtered out and not written to the dataset

  • OptionalrecordsWrittenCount?: number

    Total number of records written to the dataset

  • Optionalstate?: "QUEUED" | "RUNNING" | "ERROR"
  • OptionaltotalProcessedCount?: number

    Total number of processed changes for the dataset

  • OptionalunoptimizedRecordsCount?: number

    Queryable records pending merge into optimized Iceberg partitions.

Settings
Member Visibility

On This Page
Properties