Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backend/frontend] add configuration to have better chances to get RSS Feed contents (#8736) #9244

Merged
merged 10 commits into from
Jan 24, 2025

Conversation

aHenryJard
Copy link
Member

@aHenryJard aHenryJard commented Dec 5, 2024

Proposed changes

Adding several configuration to get better chance to avoid HTTP 403 from public RSS Feed:

  • ingestion_manager.rss_feed.min_interval_minutes : wait for at least N minutes before next HTTP call for RSS Feed
  • ingestion_manager.rss_feed.user_agent: allow to overide user agent used in header for RSS Feed requests
  • ingestion_manager.csv_feed.min_interval_minutes : wait for at least N minutes before next HTTP call for CSV Feed

Adding some info and warn level log:

  • Info to know if an HTTP Call has been skipped because of min_interval_minutes or queue states
  • warn in case of 403 coming from a cloudflare challenge to know that it's not a credential issue.

Doc PR OpenCTI-Platform/docs#231

image

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case (coverage and e2e)
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

@github-actions github-actions bot added the filigran team use to identify PR from the Filigran team label Dec 5, 2024
@aHenryJard aHenryJard changed the title Investigate on RSS public Feed 403 issue. Investigate on RSS public Feed 403 issue (#8736) Dec 5, 2024
@aHenryJard aHenryJard changed the title Investigate on RSS public Feed 403 issue (#8736) [backend] add configuration to have better chances to get RSS Feed contents (#8736) Dec 11, 2024
@aHenryJard aHenryJard force-pushed the issue/8736-investigation branch from 2c316f3 to 6b967af Compare December 11, 2024 09:26
@aHenryJard aHenryJard changed the title [backend] add configuration to have better chances to get RSS Feed contents (#8736) [backend/frontend] add configuration to have better chances to get RSS Feed contents (#8736) Dec 11, 2024
@aHenryJard aHenryJard marked this pull request as ready for review December 11, 2024 09:28
Copy link

codecov bot commented Dec 11, 2024

Codecov Report

Attention: Patch coverage is 21.62162% with 29 lines in your changes missing coverage. Please review.

Project coverage is 65.23%. Comparing base (f681660) to head (75e4440).
Report is 10 commits behind head on master.

Files with missing lines Patch % Lines
...rm/opencti-graphql/src/manager/ingestionManager.ts 17.14% 29 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9244      +/-   ##
==========================================
- Coverage   65.26%   65.23%   -0.03%     
==========================================
  Files         630      630              
  Lines       60244    60273      +29     
  Branches     6760     6765       +5     
==========================================
+ Hits        39319    39320       +1     
- Misses      20925    20953      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@aHenryJard aHenryJard force-pushed the issue/8736-investigation branch from 432ffbf to 4a7b302 Compare January 8, 2025 09:42
@aHenryJard aHenryJard marked this pull request as draft January 8, 2025 09:43
@richard-julien richard-julien force-pushed the master branch 5 times, most recently from b414944 to c7f4cb7 Compare January 13, 2025 09:27
@aHenryJard aHenryJard force-pushed the issue/8736-investigation branch from 4a7b302 to 75e4440 Compare January 23, 2025 06:16
@aHenryJard aHenryJard marked this pull request as ready for review January 23, 2025 06:25
@aHenryJard aHenryJard merged commit bcf47c5 into master Jan 24, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[RSS Feed] Error 403 on accessible public feeds
4 participants