Error when connect from Elastalert2 in Docker to Opensearch in Docker

[buzzlightyear2k]

I faced this error in elastalert2 container:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 242, in main
    create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
  File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 24, in create_index_mappings
    esversion = get_version_from_cluster_info(es_client)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/util.py", line 562, in get_version_from_cluster_info
    esinfo = client.info()['version']
             ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/__init__.py", line 286, in info
    return self.transport.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 390, in perform_request
    raise e
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/http_requests.py", line 177, in perform_request
    raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))

The config file of elastalert2:

rules_folder: /opt/elastalert/rules

run_every:
  seconds: 10

buffer_time:
  minutes: 15

es_host: opensearch-node1
es_port: 9200

writeback_index: elastalert_status

alert_time_limit:
  days: 2

And this is the docker compose file i used to run opensearch and elastalert2:

---
version: '3'
services:
  opensearch-node1:
    image: 10.60.129.132:8890/opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true  # along with the memlock settings below, disables swapping
      - OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m  # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and higher
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536  # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600  # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: 10.60.129.132:8890/opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: 10.60.129.132:8890/opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - '5601'
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net
  elastalert:
    image: 10.60.129.132:8890/jertel/elastalert2:latest
    container_name: elastalert
    volumes:
      - ./elastalert.yaml:/opt/elastalert/config.yaml
      - ./rules:/opt/elastalert/rules
    links:
      - opensearch-node1
    networks:
      - opensearch-net    
  praeco:
    image: 10.60.129.132:8890/johnsusek/praeco:latest
    container_name: praeco
    ports:
      - 8080:8080
    links:
      - elastalert
      - opensearch-node1
    networks:
      - opensearch-net    

volumes:
  opensearch-data1:
  opensearch-data2:
networks:
  opensearch-net:  

[jertel]

Exec into the ElastAlert2 container, (or any other container on that same opensearch-net network) and try to curl -v http://opensearch-node1:9200 to see what happens. Also, check the logs on the opensearch-node1 container to see if it is outputing an error during the connection attempt. If neither of those yield clues, run tcpdump against the docker interface to watch the packet traffic between the ElastAlert2 container and the opensearch-node1 container.

[buzzlightyear2k]

Exec into the ElastAlert2 container, (or any other container on that same opensearch-net network) and try to curl -v http://opensearch-node1:9200 to see what happens. Also, check the logs on the opensearch-node1 container to see if it is outputing an error during the connection attempt. If neither of those yield clues, run tcpdump against the docker interface to watch the packet traffic between the ElastAlert2 container and the opensearch-node1 container.

@jertel When i curl to opensearch-node1 it return like this:

root@64b00ee46d7a:/tmp/nginx/praeco# curl -v http://opensearch-node1:9200
* Rebuilt URL to: http://opensearch-node1:9200/
*   Trying 172.27.0.2...
* TCP_NODELAY set
* Connected to opensearch-node1 (172.27.0.2) port 9200 (#0)
> GET / HTTP/1.1
> Host: opensearch-node1:9200
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host opensearch-node1 left intact
curl: (52) Empty reply from server

And this is log error on opensearch-node1 container:

io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682d6e6f6465313a393230300d0a557365722d4167656e743a206375726c2f372e35322e310d0a4163636570743a202a2f2a0d0a0d0a
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[netty-codec-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) [netty-common-4.1.110.Final.jar:4.1.110.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.110.Final.jar:4.1.110.Final]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682d6e6f6465313a393230300d0a557365722d4167656e743a206375726c2f372e35322e310d0a4163636570743a202a2f2a0d0a0d0a

[jertel]

From the docs: https://elastalert2.readthedocs.io/en/latest/configuration.html#global-configuration

Add use_ssl: true to the ElastAlert 2 config file

[buzzlightyear2k]

@jertel