Skip to content

[Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode #26520

@johnjcasey

Description

@johnjcasey

What happened?

BQIO was lazily creating streams, which means that all our appends to a given stream would be appended at index 0.
This meant that for any second append to a stream, we would get an exception from BQ, and be forced to retry our work. This results in bad performance by itself, but combined with #26521 resulted in data loss for beam versions 2.44 - 2.47 inclusive.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions