Skip to content

Bugfix/issue 34 col name conflict#35

Merged
yokofly merged 8 commits intomainfrom
bugfix/issue-34-col-name-conflict
Oct 25, 2024
Merged

Bugfix/issue 34 col name conflict#35
yokofly merged 8 commits intomainfrom
bugfix/issue-34-col-name-conflict

Conversation

@yokofly
Copy link
Collaborator

@yokofly yokofly commented Oct 24, 2024

PR intends to:

  1. conf: set a larger batch limit when writing to target proton
  2. enhance the csv import (handle some edge case like create stream v(Hot int, hOT int) )
more internal see https://github.com/slingdata-io/issues/417
fix #34 

The old code intends to normalize every column to lower or upper case. 
This makes it difficult if we create a table with two columns that are almost the same:

create stream v(id int, Id float)

also because writing to csv the header is lowercase, we need to convert it to keep the old name.


the side effect: 
when importing a csv from other systems instead sling prob has an issue.
and I was thinking of the current PR hard code in many places,  need some time to prepare before sending it upstream.
  1. database_proton.go: refine the insert logic, to make [nan] in float can be treated as NULL.

  2. change proton template yaml: skip ALIAS columns. create stream v(id int, l int ALIAS id) this table only one column id

@yokofly yokofly merged commit aa5ebc5 into main Oct 25, 2024
@yokofly yokofly linked an issue Oct 25, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

column name issue when import data to target table

2 participants