Skip to content

fix: update timestamp layout and parsing method in Proton configuration#47

Merged
yokofly merged 1 commit intomainfrom
bugfix/issue-46-tz-conversion
Jul 10, 2025
Merged

fix: update timestamp layout and parsing method in Proton configuration#47
yokofly merged 1 commit intomainfrom
bugfix/issue-46-tz-conversion

Conversation

@yokofly
Copy link
Collaborator

@yokofly yokofly commented Jul 10, 2025

Fix #46

Issue Description

The issue occurs when incremental mode insert is used, and sling tries to directly compare UTC (CSV) with UTC+8 (Proton). The comparison is performed using to_time, which is not smart enough to detect the timezone properly.

Reproduction Steps

1. Create 2 streams in Proton

CREATE STREAM src
             (
               `id` int64,
               `_tp_time` datetime64(3, 'Asia/Shanghai') DEFAULT now64(3, 'Asia/Shanghai') CODEC(DoubleDelta, LZ4)
             );

CREATE STREAM tgt
             (
               `id` int64,
               `_tp_time` datetime64(3, 'Asia/Shanghai') DEFAULT now64(3, 'Asia/Shanghai') CODEC(DoubleDelta, LZ4)
             );

2. Insert an older record

INSERT INTO src(id, _tp_time) VALUES(1, '2025-07-18 22:00:00');

3. Run Proton → CSV → Proton transfer

./sling run --src-conn LOCAL --src-stream "src" --tgt-conn FILE --tgt-object "old.csv" --mode incremental --update-key _tp_time -d

./sling run --src-conn FILE --src-stream "old.csv" --tgt-conn LOCAL --tgt-object "tgt" --mode incremental --update-key _tp_time -d

4. Insert a new record (gap between newer and older records is less than 8 hours)

INSERT INTO src(id, _tp_time) VALUES(2, '2025-07-19 05:00:00');

5. Run Proton → CSV → Proton transfer again

./sling run --src-conn LOCAL --src-stream "src" --tgt-conn FILE --tgt-object "new.csv" --mode incremental --update-key _tp_time -d

./sling run --src-conn FILE --src-stream "new.csv" --tgt-conn LOCAL --tgt-object "tgt" --mode incremental --update-key _tp_time -d

6. Verify the final count

SELECT * FROM table(tgt);

Expected vs Actual Results

  • Expected: 2 rows returned
  • Actual (old sling): Only 1 row returned

@yokofly yokofly merged commit 9fc2a55 into main Jul 10, 2025
@yokofly yokofly deleted the bugfix/issue-46-tz-conversion branch July 10, 2025 06:52
@yokofly
Copy link
Collaborator Author

yokofly commented Jul 24, 2025

updated new test case:

  1. stream creation
CREATE STREAM src
             (
               `id` int64,
               `_tp_time` datetime64(3, 'Asia/Shanghai') DEFAULT now64(3, 'Asia/Shanghai') CODEC(DoubleDelta, LZ4)
             );

CREATE STREAM tgt
             (
               `id` int64,
               `_tp_time` datetime64(3, 'Asia/Shanghai') DEFAULT now64(3, 'Asia/Shanghai') CODEC(DoubleDelta, LZ4)
             );
INSERT INTO src(id, _tp_time) VALUES(1, '2025-07-18 22:00:00');
  1. stream => stream copy with sling ( check target table inserted one row)
    ./sling run --src-conn LOCAL --src-stream "src" --tgt-conn LOCAL --tgt-object "tgt" --mode incremental --update-key _tp_time -d

  2. insert another row

INSERT INTO src(id, _tp_time) VALUES(2, '2025-07-19 05:00:00');
  1. stream => stream copy with sling again ( Old version: nothing happen in new tgt table, new version: inserted new row)
    ./sling run --src-conn LOCAL --src-stream "src" --tgt-conn LOCAL --tgt-object "tgt" --mode incremental --update-key _tp_time -d

@kuiliu88
Copy link

Case verified on old/new version tools and works as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Timezone comparison bug during data restore with existing data

2 participants