IOC extraction process slow

[regulartim]

The celery tasks that extract data from elastic and write it to the GreedyBear DB have become much slower over time. This seems to correlate with a growing GreedyBear DB. On an empty DB, all extraction processes combined took about 20 seconds. Now, with about 500k IOCs in the DB, they take over 2 minutes.
One of the performance killers seems to be the _check_first_time_run method in attacks.py:

GreedyBear/greedybear/cronjobs/attacks.py

Lines 98 to 115 in 602fcf9

	def _check_first_time_run(self, honeypot_flag, general=False):
	all_ioc = IOC.objects.all()
	if not all_ioc:
	# plus, we extract the sensors addresses so we can whitelist them
	ExtractSensors().execute()
	self.first_time_run = True
	else:
	# if this is not the overall first time, it could that honeypot first time
	# FEEDS for a general honeypot it needs to be checked if it's in the list
	if not general:
	honeypot_ioc = IOC.objects.filter(**{f"{honeypot_flag}": True})
	else:
	honeypot_ioc = IOC.objects.filter(**{"general_honeypot__name__iexact": honeypot_flag})
	if not honeypot_ioc:
	# first time we execute this project.
	# So we increment the time range to get the data from the last 3 days
	self.first_time_run = True

In line 100 and 112 QuerySet evaluation is forced by testing it in a boolean context. To my understanding this means that all IOCs are fetched from the DB, which is very expensive. This should rather be done by using QuerySet.exists(). I will further investigate this and open a PR.

[drosetti]

I'll assign you this issue, thanks!