DENG: Handle OpenSSL.SSL.WantReadError

[jsu]

Summary

1. Taboola API started to show OpenSSL.SSL.WantReadError. We add requests.exceptions.ConnectionError to retry.

Deployment Instructions

1. merge
2. add new release
3. change requirements.txt in OpenMail repo.

[jsu]

The original error

Airflow

Message

[2020-07-30 15:37:04,354] {docker_operator.py:242} INFO - ERROR:ingestor:Account qoolmedia-yahoo-sc failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 313, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1840, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1646, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 752, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 682, in _update_chunk_length
    line = self._fp.fp.readline()
  File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into
    raise timeout("The read operation timed out")
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 560, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 781, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 430, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='backstage.taboola.com', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/OpenMail/etl/ingestor/ingest.py", line 72, in run
    worker.run(manager)
  File "/OpenMail/etl/ingestor/framework/worker.py", line 369, in run
    handle_from_download(manager)
  File "/OpenMail/etl/ingestor/framework/worker.py", line 391, in handle_from_download
    manager.task_row, f.name, parent_account_id=manager.parent_account_id
  File "/OpenMail/etl/ingestor/adapter/taboola.py", line 165, in download_fn
    common.gzip_json_write(tmp_file, results)
  File "/OpenMail/etl/ingestor/adapter/common.py", line 930, in gzip_json_write
    jsonlines.Writer(f).write_all(rows)
  File "/usr/local/lib/python3.6/site-packages/jsonlines/jsonlines.py", line 296, in write_all
    for obj in iterable:
  File "/OpenMail/etl/ingestor/adapter/taboola.py", line 132, in get_campaign_items
    client, account_id, campaign["id"]
  File "/src/pytaboola/pytaboola/services/base.py", line 41, in list
    return self.execute('GET', self.build_uri())['results']
  File "/src/pytaboola/pytaboola/services/base.py", line 28, in execute
    query_params=query_params, **payload)
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 311, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 391, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 338, in iter
    return fut.result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 394, in call
    result = fn(*args, **kwargs)
  File "/src/pytaboola/pytaboola/client.py", line 116, in execute
    timeout=self.timeout)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 686, in send
    r.content
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 828, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 757, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='backstage.taboola.com', port=443): Read timed out.

We catch ConnectionError and retry.