added s3 storage for attack event files #285

vorband · 2018-03-09T10:23:02Z

added log_s3.py logging module to store attack event files on external s3 storage
added s3storage config section in dist config
added botocore to requirements.txt
tested with pithos s3 (http) and AWS s3 (https).
Note: For pithos use signature_version "s3" , for AWS S3 uses "s3v4"

- added log_s3.py logging module to store attack event files on external s3 storage - added s3storage config section in dist config - added botocore to requirements.txt - tested with pithos s3 (http) and AWS s3 (https). - Note: For pithos use signature_version "s3" , for AWS S3 uses "s3v4"

glaslos · 2018-03-11T15:36:29Z

glastopf/glastopf.cfg.dist

+aws_access_key_id = YOUR_aws_access_key_id
+aws_secret_access_key = YOUR_aws_access_key_id
+bucket = glastopf
+region = europe


Maybe use eu-west-1?

This was meant as a dummy and needs to be set according to the configuration anyway. However, I set it to eu-west-1. Changed in e906972

glaslos · 2018-03-11T15:42:25Z

glastopf/modules/reporting/auxiliary/log_s3.py

+
+    def insert(self, attack_event):
+        if self._initial_connection_happend:
+            if attack_event.file_name is not None:


Have you thought about deduplication? I think you will see some of the file very often. Maybe add a flag to the event indicating this is a known file?

implemented in 0508e99
checks both locally and on the remote s3 for duplicate files to avoid upload. Other logging modules (except for log_sql) will need to be adapted in order to log this status flag.

- add flag for known file (known_file) to AttackEvent: set if file already locally exists - extended log_sql to new flag - added deduplication handling in log_s3 - local check (flag) and remote check (avoid upload)

Allow interpolation of environment variables from glastopf.cfg. This is useful in a docker environment in case you want settings to be controlled via docker-compose.yml. Interpolation is invoked in glastopf.cfg, for example ... [hpfeed] … chan_events = %(CHANEVENTS)s …

- changed filename to use sha256 hash - stix test to use sha256 - added flag for known rci for php cgi rce

vorband · 2018-03-23T08:56:49Z

changed s3 storage filename to use sha256 hash, added AttackEvent dict entry for file_sha256

…reliant log processing removed sha256 as default hash for files to maintain interoperability with md5 reliant log processing - removed previous changes in test_stix, php_cgi_rce, in order to still use md5 - added file_sha256 to AttackEvent event_dict dict - log_s3.py now stores on s3 and uses sha256 as filename

glaslos · 2018-04-11T15:20:11Z

glastopf/modules/reporting/auxiliary/log_s3.py

+        config = os.path.join(work_dir, config)
+        BaseLogger.__init__(self, config)
+        self.files_dir = os.path.join(data_dir, 'files/')
+        self.enabled = False


self.enabled = self.config.getboolean("s3storage", "enabled") and then use self.enabled.

I simply followed the way it was implemented in log_hpfeeds.py. However, I will change it according to your comments.

glaslos · 2018-04-11T15:21:04Z

glastopf/modules/reporting/auxiliary/log_s3.py

+        self.files_dir = os.path.join(data_dir, 'files/')
+        self.enabled = False
+        self._initial_connection_happend = False
+        self.options = {'enabled': self.enabled}


Are you using this?

nope. removed

glaslos · 2018-04-11T15:21:52Z

glastopf/modules/reporting/auxiliary/log_s3.py

+
+    def _start_connection(self, endpoint, accesskey, secretkey, version, region, bucket):
+        self.s3session = botocore.session.get_session()
+        self.s3session.set_credentials(accesskey, secretkey)


self.accesskey

glaslos · 2018-04-11T15:33:49Z

glastopf/modules/reporting/auxiliary/log_s3.py

+    def _start_connection(self, endpoint, accesskey, secretkey, version, region, bucket):
+        self.s3session = botocore.session.get_session()
+        self.s3session.set_credentials(accesskey, secretkey)
+        self.s3client = self.s3session.create_client(


are you sure this actually does a connection?

actually, it does not. I will perform a connection test instead.

glaslos · 2018-04-11T15:34:22Z

glastopf/modules/reporting/auxiliary/log_s3.py

+            self.options = {'enabled': self.enabled}
+            self.s3client = None
+            self.s3session = None
+            gevent.spawn(self._start_connection, self.endpoint, self.accesskey, self.secretkey, self.version, self.region, self.bucket)


maybe make this blocking so we ensure we have a connection

I added a test if bucket is available and if it fails during initial start, it will no longer try.
Will update the module now.

glaslos · 2018-04-11T15:34:54Z

glastopf/modules/reporting/auxiliary/log_s3.py

+        if self._initial_connection_happend:
+            if attack_event.file_sha256 is not None:
+                if attack_event.known_file:
+                    logger.debug('sha256:{0} / md5:{1} is a known file, it will not be uploaded.'.format(attack_event.file_sha256, attack_event.file_name))


return here and drop the else

glaslos · 2018-04-11T15:35:36Z

glastopf/modules/reporting/auxiliary/log_s3.py

+                            searchFile = self.s3client.list_objects_v2(Bucket=self.bucket, Prefix=attack_event.file_sha256)
+                            if (len(searchFile.get('Contents', []))) == 1 and str(searchFile.get('Contents', [])[0]['Key']) == attack_event.file_sha256:
+                                logger.debug('Not storing file (sha256:{0}) to s3 bucket "{1}" on {2} as it already exists in the bucket.'.format(attack_event.file_sha256, self.bucket, self.endpoint))
+                            else:


return in the if and drop the else

glaslos · 2018-04-11T15:36:17Z

glastopf/modules/reporting/auxiliary/log_s3.py

+        self._initial_connection_happend = True
+
+    def insert(self, attack_event):
+        if self._initial_connection_happend:


if the initial connection is blocking you can drop this if/else

vorband · 2018-05-14T11:43:39Z

changes implemented. hope it's ok now.

glaslos · 2018-05-14T13:32:21Z

Awesome, looks good!

Jc2k · 2018-06-04T16:42:13Z

Great to see this! Note to anyone finding this PR in future; Pithos after 8th March 2018 does support s3v4.

vorband and others added 2 commits March 9, 2018 11:06

Added s3 storage for attack event files

6e3bb15

glaslos requested changes Mar 11, 2018

View reviewed changes

vorband added 6 commits March 14, 2018 16:06

added deduplication handling for downloaded files

0508e99

- add flag for known file (known_file) to AttackEvent: set if file already locally exists - extended log_sql to new flag - added deduplication handling in log_s3 - local check (flag) and remote check (avoid upload)

changed region

e906972

changed logging level for debug output

b123a4e

filename sha256 hash

fe1c8c3

- changed filename to use sha256 hash - stix test to use sha256 - added flag for known rci for php cgi rce

Merge remote-tracking branch 'mushorg/master'

df5c5e5

glaslos requested changes Apr 11, 2018

View reviewed changes

implemented changes requested by glaslos for s3 logger

ea0bbf8

glaslos approved these changes May 14, 2018

View reviewed changes

glaslos merged commit 67c2a3f into mushorg:master May 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added s3 storage for attack event files #285

added s3 storage for attack event files #285

vorband commented Mar 9, 2018

glaslos Mar 11, 2018

vorband Mar 14, 2018

glaslos Mar 11, 2018

vorband Mar 14, 2018

vorband commented Mar 23, 2018 •

edited

Loading

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

glaslos Apr 11, 2018

vorband May 14, 2018

vorband commented May 14, 2018

glaslos commented May 14, 2018

Jc2k commented Jun 4, 2018

added s3 storage for attack event files #285

added s3 storage for attack event files #285

Conversation

vorband commented Mar 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vorband commented Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vorband commented May 14, 2018

glaslos commented May 14, 2018

Jc2k commented Jun 4, 2018

vorband commented Mar 23, 2018 •

edited

Loading