fix: Switch to sequential processing in batch_process to resolve thread-safety issues

[fg-nava]

fix: Switch to sequential processing in batch_process to resolve thread-safety issues

Ticket

https://navalabs.atlassian.net/browse/DST-688

Changes

• Modified batch_process.py to use sequential processing instead of parallel processing
• Added explanatory comments about the thread-safety concerns with LiteLLM

Context for reviewers

This PR addresses the Docker deployment crashes (CPU >1400%) issue when processing CSVs with multiple rows. Investigation revealed that the high CPU usage was caused by thread-safety issues in the underlying LiteLLM client libraries when running in parallel.

The fix is simple but effective: we've removed the parallel processing implementation and switched to sequential processing. This change resolves the CPU spike issues we were seeing in Docker deployments.

Follow-up Items (to be tracked in separate tickets):

1. UI feedback issue: "File processed, results attached" message appears in logs but not in UI
2. Assessment needed: Determine if parallel processing is necessary for performance and if so, investigate thread-safe alternatives
3. Test coverage: Add integration tests for batch runs of _process_question (currently only have mock calls to chat engine)

Testing

Tested locally by:

• Runnin processing with multi-row CSVs
• Confirmed CPU usage remains stable
• Verified successful processing of all rows in sequential order

The change eliminates the >1400% CPU spikes previously observed in Docker deployments while maintaining functionality.

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
2845	2543	89%	80%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
app/src/batch_process.py	100%	🟢
TOTAL	100%	🟢

updated for commit: 025092d by action🐍

[fg-nava]

This can be closed without merging as it is upstream of #170

[yoomlam]

@fg-nava You should have merged this PR so that the title "fix: Switch to sequential processing in batch_process" is included as a separate commit message in main. Then merge #170, which will have a separate commit message that says "fix: Ensure UI feedback message for batch processing completion"