Support for MFA like authentik, Duo

[rostwal95]

Most of the private sites are behind MFA like authentik, duo. Automating the entire MFA flow could be difficult and a breach of security policies as well, if we can initiate an interactive session with the client, where he performs the MFA and upon completion we use the same session to start scraping provided robots.txt allows it, it will be highly useful and will serve many use cases.

[raznem]

Yeah sounds like a cool feature, but at the same time quite complex, or maybe there are some shortcuts?

[rostwal95]

It's not that complex if we let the user interact with MFA, if we handle that complexity in the backend, it could very well be the security breach and automatic MFA flows are a tedious task, if the site is behind MFA, backend can return the url back to the frontend with 200 OK, we should be able to open up a new session through playwright where the entire authentication flow could take place and we should be able to store the cookies as part of this process, once the authentication is done by the user we can retrieve the cookies and send that session in the backend, through the specified cookie we can validate if the session is valid by sending the token to auth endpoint and upon successful validation, start scraping the pages, we should also have options like allow domains and depth which restricts the scraping to the links with the same domain and the depth till when we want to scrape.

[rostwal95]

The reason this could be a really cool feature and stand out from other tools is most of the pages of an org be it web pages or other documents like pdf, txt, xslx are mostly behind MFA. If these orgs wants to create applications like RAG, our solution becomes much more relevant.

[raznem]

Ok, I see separate features that can be added:

1. Crawling the whole website by providing a link to it.
2. Ability to log in before extraction starts. The first version could provide a link in the terminal to open and log in to give cookies to the scraper. Initially, it can work without crawling the whole website, but with a list of pages from which to extract data.

[rostwal95]

Yeah, thanks for picking it up 🙂