Skip to content

Commit

Permalink
#53: Up to date and maintenance of all Ansible Collections - Non-HA a…
Browse files Browse the repository at this point in the history
…nd HA (#54)
  • Loading branch information
GeorgSchulz authored Oct 21, 2024
1 parent aee41cc commit 76f7e8b
Show file tree
Hide file tree
Showing 86 changed files with 2,761 additions and 368 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

- communitylab#26: Error handling in Conda Package Manager
- communitylab#52: Consider usage of Jupyter Enterprise Gateway instead of not maintained Jupyter on Hadoop

## [v1.5.0] - 2024-10-21

### Added

- communitylab#53: Up to date and maintenance of all Ansible Collections - Non-HA and HA
- communitylab#50: Simplifying architecture for IDE Non-HA setup
- communitylab#48: Downgrading OpenJDK and updating documentation images

Expand Down
46 changes: 26 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@ If you are german speaking you may be interested in my related academic work: [T
## 1. Prerequisites
### required
- Ubuntu (was tested on Ubuntu 24.04 LTS)
- Ansible (was tested on Ansible version 2.17.3)
- Python (was tested on Python version 3.12.5)
- Ansible (was tested on Ansible version 2.17.4)
- Python (was tested on Python version 3.12.6)

### optional
- Molecule (was tested on Molecule version 24.7.0)
- Molecule (was tested on Molecule version 24.9.0)
- Docker (was tested on Docker version 27.1.2, required for Ansible Molecule)
- Terraform (was tested on Terraform version v1.9.3)
- Go (was tested on Go version go1.18.1)
Expand All @@ -80,24 +80,30 @@ georg@notebook:~/git/CommunityLab$ bash requirements.sh

### 2.2 Ansible Molecule is available for following Ansible Collections
```console
georg@notebook:~/git/CommunityLab$ find . -name extensions
./collections/ansible_collections/jupyter/hub/extensions
./collections/ansible_collections/authentication/kerberos/extensions
./collections/ansible_collections/hadoop/hdfs/extensions
./collections/ansible_collections/hadoop/yarn/extensions
./collections/ansible_collections/bigdata/spark/extensions
./collections/ansible_collections/bigdata/zookeeper/extensions
./collections/ansible_collections/rdbms/postgres/extensions
./collections/ansible_collections/authorization/ldap/extensions
georg@notebook:~/git/CommunityLab$ find . -name molecule
./collections/ansible_collections/jupyter/hub/extensions/molecule
./collections/ansible_collections/authentication/kerberos/extensions/molecule
./collections/ansible_collections/hadoop/hdfs/extensions/molecule
./collections/ansible_collections/hadoop/yarn/extensions/molecule
./collections/ansible_collections/bigdata/spark/extensions/molecule
./collections/ansible_collections/bigdata/zookeeper/extensions/molecule
./collections/ansible_collections/rdbms/postgres/extensions/molecule
./collections/ansible_collections/authorization/ldap/extensions/molecule
```

### 2.3 The IDE can be installed using Ansible Molecule like this (Docker is required)
### 2.3 Each Ansible Collection has two Ansible Molecule scenarios - default and ha_setup
##### (The Ansible Collection bigdata.spark is an exception since the only purpose is the installation of common Apache Spark libraries)

- default (Simple installation process of the Ansible Collection without High Availability)
- ha_setup (More complex installation process of the Ansible Collection with High Availability)

### 2.4 The IDE can be installed using Ansible Molecule like this (Docker is required) - Non-HA IDE
```console
georg@notebook:~/git/CommunityLab$ cd collections/ansible_collections/jupyter/hub/extensions/
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ molecule converge
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ molecule converge -s default
```

### 2.4 Check container created by Ansible Molecule
### 2.5 Check container created by Ansible Molecule
```console
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Expand All @@ -109,25 +115,25 @@ a33331864a0e geerlingguy/docker-ubuntu2404-ansible "/usr/lib/systemd/sy…"
16591b433003 geerlingguy/docker-ubuntu2404-ansible "/usr/lib/systemd/sy…" About an hour ago Up About an hour instance-1
```

### 2.5 Get IP address of Docker container running JupyterHub
### 2.6 Get IP address of Docker container running JupyterHub
```console
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' instance-1
172.23.27.3
```

### 2.6 Start Firefox browser
### 2.7 Start Firefox browser
```console
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ firefox
```

### 2.7 Login to JupyterHub here using credentials of variable [ldap_users](./collections/ansible_collections/jupyter/hub/extensions/molecule/default/molecule.yml):
### 2.8 Login to JupyterHub here using credentials of variable [ldap_users](./collections/ansible_collections/jupyter/hub/extensions/molecule/default/molecule.yml):
https://172.23.27.3:8443

![IDE Docker](https://github.com/GeorgSchulz/CommunityLab/blob/master/images/ide_in_docker.bmp?raw=True)

### 2.6 Delete all Docker container using Ansible Molecule
### 2.9 Delete all Docker container using Ansible Molecule
```console
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ molecule destroy
georg@notebook:~/git/CommunityLab/collections/ansible_collections/jupyter/hub/extensions$ molecule destroy -s default
```

## 3. Use Hetzner Cloud
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
- name: Setup Kerberos server
hosts: kerberos
gather_facts: true
roles:
- role: ide.environment.user_and_groups
- role: tls.certs.issue
- role: tls.certs.distribute
- role: authorization.ldap.setup
- role: authentication.kerberos.setup
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
- name: Create
hosts: localhost
gather_facts: false
vars:
molecule_inventory:
all:
hosts: {}
tasks:
- name: Create docker network
community.docker.docker_network:
name: example.com
ipam_config:
- subnet: 172.23.27.0/24
gateway: 172.23.27.2
iprange: 172.23.27.0/26

- name: Create container
become: true
community.docker.docker_container:
name: "{{ item.name }}"
image: "{{ item.image }}"
state: started
command: "{{ item.command }}"
log_driver: json-file
publish_all_ports: true
cgroupns_mode: "{{ item.cgroupns_mode }}"
privileged: "{{ item.privileged }}"
volumes: "{{ item.volumes }}"
networks:
- name: example.com
ipv4_address: "{{ item.ipv4_address }}"
hostname: "{{ item.name }}.example.com"
tls_hostname: "{{ item.name }}.example.com"
capabilities:
- NET_ADMIN
register: result
loop: "{{ molecule_yml.platforms }}"

- name: Fail if container is not running
when: >
item.container.State.ExitCode != 0 or
not item.container.State.Running
ansible.builtin.include_tasks:
file: tasks/create-fail.yml
loop: "{{ result.results }}"
loop_control:
label: "{{ item.container.Name }}"

- name: Add container to molecule_inventory
vars:
inventory_partial_yaml: |
all:
vars:
ansible_connection: community.docker.docker
children:
kerberos1:
hosts:
instance-1:
kerberos2:
hosts:
instance-2:
kerberos:
children:
kerberos1:
kerberos2:
ldap1:
hosts:
instance-1:
ldap2:
hosts:
instance-2:
ldap:
children:
ldap1:
ldap2:
ansible.builtin.set_fact:
molecule_inventory: >
{{ molecule_inventory | combine(inventory_partial_yaml | from_yaml, recursive=true) }}
- name: Dump molecule_inventory
ansible.builtin.copy:
content: |
{{ molecule_inventory | to_yaml }}
dest: "{{ molecule_ephemeral_directory }}/inventory/molecule_inventory.yml"
mode: "0600"

- name: Force inventory refresh
ansible.builtin.meta: refresh_inventory

# we want to avoid errors like "Failed to create temporary directory"
- name: Validate that inventory was refreshed
hosts: all
gather_facts: false
tasks:
- name: Check uname
ansible.builtin.raw: uname -a
register: result
changed_when: false

- name: Display uname info
ansible.builtin.debug:
msg: "{{ result.stdout }}"
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
- name: Destroy molecule containers
hosts: all
gather_facts: false
tasks:
- name: Stop and remove container
delegate_to: localhost
become: true
community.docker.docker_container:
name: "{{ inventory_hostname }}"
state: absent
auto_remove: true

- name: Remove dynamic molecule inventory
hosts: localhost
gather_facts: false
tasks:
- name: Remove dynamic inventory file
become: true
ansible.builtin.file:
path: "{{ molecule_ephemeral_directory }}/inventory/molecule_inventory.yml"
state: absent
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
dependency:
name: galaxy
options:
requirements-file: requirements.yml
platforms:
- name: instance-1
image: docker.io/geerlingguy/docker-ubuntu2404-ansible
command: /usr/lib/systemd/systemd
pre_build_image: true
cgroupns_mode: host
ipv4_address: 172.23.27.3
privileged: true
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:rw
- /var/lib/containerd
- name: instance-2
image: docker.io/geerlingguy/docker-ubuntu2404-ansible
command: /usr/lib/systemd/systemd
pre_build_image: true
cgroupns_mode: host
ipv4_address: 172.23.27.4
privileged: true
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:rw
- /var/lib/containerd
provisioner:
name: ansible
log: false
options:
vvv: false
playbooks:
converge: converge.yml
inventory:
group_vars:
all:
molecule_deployment: true
ldap_external: false
ldap_user: openldap
ldap_uid: "5001"
ldap_group: openldap
ldap_gid: "4001"
service_user: "{{ ldap_user }}"
service_name: Open LDAP
service_uid: "{{ ldap_uid }}"
service_group: "{{ ldap_group }}"
service_gid: "{{ ldap_gid }}"
tls_user: "{{ ldap_user }}"
tls_group: "{{ ldap_group }}"
self_signed_certificates: true
certs_source:
- /tmp/selfsigned_certs/{{ ansible_fqdn }}/cert.pem
- /tmp/selfsigned_certs/{{ ansible_fqdn }}/chain.pem
- /tmp/selfsigned_certs/{{ ansible_fqdn }}/key.pem
certs_dest:
- cert.pem
- chain.pem
- key.pem
certs_mode:
- "0660"
- "0660"
- "0400"
domain: example.com
keystore_file: /etc/ssl/private/{{ ansible_fqdn }}.jks
keystore_password: changeit
truststore_file: /etc/ssl/certs/truststore.jks
truststore_password: changeit
ldap_server_address: "{% for host in groups.ldap | shuffle %}ldaps://{{ host }}.example.com:636{% if not loop.last %},{% endif %}{% endfor %}"
ldap_organization: dc=example,dc=com
ldap_user_search_base: ou=people,{{ ldap_organization }}
ldap_group_search_base: ou=groups,{{ ldap_organization }}
ldap_bind_user: cn=admin,{{ ldap_organization }}
ldap_password: changeit
ldap_bind_dn_template: uid={username},{{ ldap_user_search_base }}
ide_users_group: ide_users
ide_users_gid: "5001"
ldap_users:
- name: teppler
uid_number: 6000
gid_number: "{{ ide_users_gid }}"
password: albstadt
- name: anolle
uid_number: 6001
gid_number: "{{ ide_users_gid }}"
password: sigmaringen
- name: gschulz
uid_number: 6002
gid_number: "{{ ide_users_gid }}"
password: datascience
ldap_replication_user: cn=replicator,{{ ldap_organization }}
ldap_replication_password: changeit
ldap_kdc_service_password: changeit
ldap_kadmin_service_password: changeit
realm: COMMUNITY.LAB
realm_password: changeit
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
---
collections:
- name: community.docker
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
- name: Retrieve container log
ansible.builtin.command:
cmd: >-
{% raw %}
docker logs
{% endraw %}
{{ item.stdout_lines[0] }}
changed_when: false
register: logfile_cmd

- name: Display container log
ansible.builtin.fail:
msg: "{{ logfile_cmd.stderr }}"
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,10 @@
- krb5-admin-server

- name: Fetch stash and service.keyfile to ansible server
ansible.posix.synchronize:
ansible.builtin.fetch:
src: /etc/krb5kdc/{{ item }}
dest: /tmp/
mode: pull
flat: true
loop:
- .k5.{{ realm }}
- service.keyfile
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@
- krb5-kdc
- krb5-admin-server

- name: Pause 1 minute for krb5-kdc service to start
ansible.builtin.pause:
minutes: 1

- name: Start Kerberos services
ansible.builtin.systemd:
name: "{{ item }}"
Expand Down
Loading

0 comments on commit 76f7e8b

Please sign in to comment.