BigQuery Storage Write API: JsonStreamWriter cannot transform BigDecimal with 9+ decimals to BigNumeric #2013
Description
Environment details
- OS type and version: OSX 11.6.3
- Java version: OpenSDK 11
- version(s): v2.28.3 and later
Steps to reproduce
- Define a TableSchema with a field of type
BIGNUMERIC - Serialize a JSON message to Proto containing more than 9 decimals, eg. the number
24.6789012345
Code example
This unit test could be added to JsonToProtoMessageTest to reproduce it:
@Test
public void testBigDecimalToBigNumericConversion() {
TableSchema ts =
TableSchema.newBuilder()
.addFields(
0,
TableFieldSchema.newBuilder()
.setName("bignumeric")
.setType(TableFieldSchema.Type.BIGNUMERIC)
.setMode(TableFieldSchema.Mode.REPEATED)
.build())
.build();
TestBignumeric expectedProto =
TestBignumeric.newBuilder()
.addBignumeric(
BigDecimalByteStringEncoder.encodeToBigNumericByteString(
new BigDecimal("24.6789012345")))
.build();
JSONObject json = new JSONObject();
json.put("bignumeric", Collections.singletonList(new BigDecimal("24.6789012345")));
DynamicMessage protoMsg =
JsonToProtoMessage.convertJsonToProtoMessage(TestBignumeric.getDescriptor(), ts, json);
assertEquals(expectedProto, protoMsg);
}Stack trace
[ERROR] com.google.cloud.bigquery.storage.v1.JsonToProtoMessageTest.testBigDecimalToBigNumericConversion Time elapsed: 0.007 s <<< ERROR!
com.google.cloud.bigquery.storage.v1.Exceptions$FieldParseError: ByteString scale cannot exceed 9: 24.6789012345
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessageImpl(JsonToProtoMessage.java:247)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessage(JsonToProtoMessage.java:139)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessageTest.testBigDecimalToBigNumericConversion(JsonToProtoMessageTest.java:1437)
...
Any additional information below
On the following lines, encodeToBigNumericByteString should be called instead of encodeToNumericByteString:
- https://github.com/googleapis/java-bigquerystorage/blob/v2.28.3/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java#L358
- https://github.com/googleapis/java-bigquerystorage/blob/v2.28.3/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java#L606
If you agree on this proposed approach, we can create PR with the following fix:
main...tradecloud:java-bigquerystorage:TC-8710-serialize-bigdecimal