import multiple tables at same time - 1 #2191
Conversation
| if err != nil { | ||
| utils.ErrExit("preparing for file import: %s", err) | ||
| } |
There was a problem hiding this comment.
Do we need to do this PrepareForFileImport here? we are already doing it in NewFileBatchProducer
| lastBatchNumber: lastBatchNumber, | ||
| lastOffset: lastOffset, | ||
| fileFullySplit: fileFullySplit, | ||
| completed: completed, |
There was a problem hiding this comment.
nit: completed: len(pendingBatches) == 0 && fileFullySplit
| return nil, err | ||
| } | ||
| if p.lineFromPreviousBatch != "" { | ||
| err = batchWriter.WriteRecord(p.lineFromPreviousBatch) |
There was a problem hiding this comment.
Add comment for explaining about this lineFromPreviousBatch
| } | ||
|
|
||
| // 3 batches should be produced | ||
| // while calculating for the first batch, the header is also considered |
There was a problem hiding this comment.
Oh right, while preparing the first batch - we add the bytes of the header to the batch's total bytes but for the further batches, we don't as we already have the header we don't include it in the batch's bytes.
I think worth testing if in some cases where the number of columns is huge can this header's bytes can also contribute to the batches' bytes and should be included.
Can you please add a TODO while we are adding a header to bthe atch file to fix this if required?
| assert.NotNil(t, batch1) | ||
| assert.Equal(t, int64(2), batch1.RecordCount) | ||
|
|
||
| // simulate a crash and recover |
There was a problem hiding this comment.
Nice test for recovery situation!
Describe the changes in this pull request
FileBatchProducerDescribe if there are any user-facing changes
How was this pull request tested?
Wrote unit tests.
To run integration tests:
Does your PR have changes that can cause upgrade issues?