# Automated Connectware Deployment Using GitLab CI/CD & Ansible In this guide, we will set up a CI/CD pipeline for Connectware using Ansible and GitLab. Our Connectware infrastructure will be composed of: * 10 Siemens S7 connection services * 10 Modbus connection services * 1 data aggregation and transformation service * 1 SAP Cloud connection service GitLab CI/CD is a tool for software development that uses the concepts of Continuous Integration (CI) and Continuous Deployment/Delivery (CD), both fundamental parts of modern DevOps. Continuous Integration is the practice where developers frequently collaborate and merge code changes into a central repository where automated builds, tasks, and tests run. The build process helps ensure that the application is not broken whenever there are new changes in the commits. Continuous Deployment is an extension of CI that deploys the latest changes to a test or production environment. CI/CD helps deliver applications faster, with higher quality and fewer integration problems. It also removes development pressure on decisions for small changes and promotes faster iterations. ## Prerequisites To follow this guide, you will need the following: * A running instance of Cybus Connectware. * A valid [Connectware license](https://docs.cybus.io/2-1-0/documentation/installation-and-upgrades/licensing). * A Git environment (cloud-hosted or on-premises). In this guide, we will use GitLab. * [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/) installed on your system. * Access to the [Admin UI](https://docs.cybus.io/2-1-0/getting-started/admin-ui) with sufficient [user permissions](https://docs.cybus.io/2-1-0/documentation/user-management). * Basic knowledge of MQTT, Docker, and Ansible. If you want to refresh your knowledge, we recommend reading [ansible.com - Getting started with Ansible](https://docs.ansible.com/ansible/latest/getting_started/index.html). ## 1. Project Structure To deploy Connectware using Ansible and the CI/CD pipeline, we will set up a GitLab project with the following structure: ``` ├── README.md ├── .gitlab-ci.yml ├── ansible │ ├── ansible.cfg │ ├── hosts.yaml │ └── playbook.yaml └── services ├── data-aggregation.yml ├── modbus.yml ├── s7.yml └── sap.yml ``` Under `services`, we place the service commissioning files. Under `ansible`, we place our playbook and host configuration (more on this later). Finally, `.gitlab-ci.yml` is our pipeline configuration file. ## 2. Pipeline Configuration To provide GitLab with all the information required to run the CI/CD pipeline, we need to write a configuration file `.gitlab-ci.yml`. Here we describe: * The structure and order of jobs that we want to be executed (e.g., tasks, tests) * The decisions the pipeline should make when specific conditions are encountered {% hint style="info" %} You need to ensure that you have runners available. If you're using gitlab.com, you can skip this step as gitlab.com provides shared runners for you. {% endhint %} For self-hosted runners, see [Registering runners | GitLab](https://docs.gitlab.com/runner/register/). {% code lineNumbers="true" %} ```yaml variables: CONNECTWARE_VERSION: '1.1.3' stages: - deploy deploy-connectware: rules: - if: '$CI_PIPELINE_SOURCE == "web"' # Trigger from Web UI when: always # Always run - when: never # Else never run image: name: registry.cybus.io/cybus/cybus-connectware-ansible:latest entrypoint: [''] stage: deploy script: - 'chmod o-w ansible' - 'chmod 600 $SSH_KEY' - 'cd ansible && ansible-playbook playbook.yaml' ``` {% endcode %} The file uses the YAML text format and follows the GitLab keyword reference. Inside `.gitlab-ci.yml`, we define variables, stages (i.e., tasks), and the pipeline's trigger. * **Variables:** Connectware version to install * **Stages:** `deploy` is the only stage in our pipeline. In this stage, we run the `deploy-connectware` job * **Jobs:** `deploy-connectware` consists of rules, an image, and a script * **Rules:** When the pipeline is triggered * **Script:** Makes Ansible executable and executes the `ansible/playbook.yml` (which deploys Connectware and its services, see "Using Ansible for Connectware orchestration") * **Image:** Latest Ansible image from Cybus' registry * **Pipeline trigger:** In this case, through the GitLab Web UI (Run Pipeline button). This can also be configured to be triggered by a new commit or merge event
## 3. Ansible Playbook and Host Configuration To deploy Connectware and its services, we need to write a playbook with the same structure as seen in [Orchestrating Connectware with Ansible](https://docs.cybus.io/2-1-0/guides/ansible/orchestrating-connectware-with-ansible). One key difference might be the host. For on-premises, self-hosted Git environments, the hostname should be the IP address where you want Connectware to run. For example, `localhost`. However, if you are using a cloud solution such as gitlab.com, the host needs to point to a defined resource, such as a cloud environment. In our example, we will use an AWS EC2 instance to deploy Connectware. You can choose between two deployment approaches depending on your CI/CD pipeline requirements: * **Sequential Deployment**: Services deploy one after another. Simpler to implement and debug. * **Asynchronous Deployment**: Services deploy in parallel. Significantly faster for large-scale rollouts. ### Sequential Deployment For straightforward deployments where services are installed one at a time, use the standard `cybus.connectware.service` module. * Create an `ansible/playbook.yml` with the following content: {% code lineNumbers="true" %} ```yaml - name: Connectware Infrastructure Playbook hosts: AWS vars: connectwareVersion: "{{ lookup('env','CONNECTWARE_VERSION') }}" adminPassword: "{{ lookup('env','CONNECTWARE_ADMIN_PASSWORD') }}" license: "{{ lookup('env','CYBUS_CONNECTWARE_LICENSE') }}" tasks: - name: 'Deploy Connectware' become: yes cybus.connectware.instance: version: '{{ connectwareVersion }}' license: '{{ license }}' admin_password: '{{ adminPassword }}' - name: Install Services (S7) loop: - host: dev01 - host: dev02 - host: dev03 - host: dev04 - host: dev05 - host: dev06 - host: dev07 - host: dev08 - host: dev09 - host: dev10 loop_control: loop_var: s7 cybus.connectware.service: admin_password: '{{ adminPassword }}' id: '{{ s7.host }}_S7' commissioning_file: ../services/s7.yml parameters: ipAddress: '{{ s7.host }}' - name: Install Services (Modbus) loop: - host: dev11 - host: dev12 - host: dev13 - host: dev14 - host: dev15 - host: dev16 - host: dev17 - host: dev18 - host: dev19 - host: dev20 loop_control: loop_var: modbus cybus.connectware.service: admin_password: '{{ adminPassword }}' id: '{{ modbus.host }}_Modbus' commissioning_file: ../services/modbus.yml parameters: ipAddress: '{{ modbus.host }}' - name: Install SAP Connector Service cybus.connectware.service: admin_password: '{{ adminPassword }}' id: 'demo_sapconnector' commissioning_file: ../services/sap.yml - name: Install Data Mapping Service cybus.connectware.service: admin_password: '{{ adminPassword }}' id: 'demo_datamapping' commissioning_file: ../services/data-aggregation.yml ``` {% endcode %} **How It Works:** * The playbook uses loops to deploy multiple similar services (S7 and Modbus connections). * Each service deployment waits for completion before starting the next. * The host points to `AWS`, defined in our `hosts.yaml` file (see below). ### Asynchronous Deployment For CI/CD pipelines where speed is critical, use the `cybus.connectware.service_async` module to deploy multiple services in parallel. This approach is especially beneficial when deploying many services (in this example, 22 services). * Create an optimized `ansible/playbook.yml` with the following content: {% code lineNumbers="true" %} ```yaml - name: Connectware Infrastructure Playbook (Optimized) hosts: AWS vars: connectwareVersion: "{{ lookup('env','CONNECTWARE_VERSION') }}" adminPassword: "{{ lookup('env','CONNECTWARE_ADMIN_PASSWORD') }}" license: "{{ lookup('env','CYBUS_CONNECTWARE_LICENSE') }}" tasks: - name: 'Deploy Connectware' become: yes cybus.connectware.instance: version: '{{ connectwareVersion }}' license: '{{ license }}' admin_password: '{{ adminPassword }}' # Deploy S7 services in parallel - name: Deploy S7 Services loop: - host: dev01 - host: dev02 - host: dev03 - host: dev04 - host: dev05 - host: dev06 - host: dev07 - host: dev08 - host: dev09 - host: dev10 loop_control: loop_var: s7 cybus.connectware.service_async: admin_password: '{{ adminPassword }}' id: '{{ s7.host }}_S7' commissioning_file_b64: "{{ lookup('file', '../services/s7.yml') | b64encode }}" parameters: ipAddress: '{{ s7.host }}' async: 600 poll: 0 register: s7_jobs # Deploy Modbus services in parallel - name: Deploy Modbus Services loop: - host: dev11 - host: dev12 - host: dev13 - host: dev14 - host: dev15 - host: dev16 - host: dev17 - host: dev18 - host: dev19 - host: dev20 loop_control: loop_var: modbus cybus.connectware.service_async: admin_password: '{{ adminPassword }}' id: '{{ modbus.host }}_Modbus' commissioning_file_b64: "{{ lookup('file', '../services/modbus.yml') | b64encode }}" parameters: ipAddress: '{{ modbus.host }}' async: 600 poll: 0 register: modbus_jobs # Deploy remaining services in parallel - name: Deploy SAP and Data Mapping Services loop: - { id: 'demo_sapconnector', file: '../services/sap.yml' } - { id: 'demo_datamapping', file: '../services/data-aggregation.yml' } cybus.connectware.service_async: admin_password: '{{ adminPassword }}' id: '{{ item.id }}' commissioning_file_b64: "{{ lookup('file', item.file) | b64encode }}" async: 600 poll: 0 register: other_jobs # Wait for all services to complete - name: Wait for S7 services async_status: jid: '{{ item.ansible_job_id }}' loop: '{{ s7_jobs.results }}' register: s7_results until: s7_results.finished retries: 60 delay: 10 - name: Wait for Modbus services async_status: jid: '{{ item.ansible_job_id }}' loop: '{{ modbus_jobs.results }}' register: modbus_results until: modbus_results.finished retries: 60 delay: 10 - name: Wait for other services async_status: jid: '{{ item.ansible_job_id }}' loop: '{{ other_jobs.results }}' register: other_results until: other_results.finished retries: 60 delay: 10 ``` {% endcode %} **Key Differences:** * Uses `cybus.connectware.service_async` instead of `cybus.connectware.service`. * Requires `commissioning_file_b64` (base64-encoded file content) instead of `commissioning_file`. * Includes `async: 600` and `poll: 0` to start tasks without waiting. * Uses `async_status` tasks to wait for all deployments to complete. **Benefits:** * **Faster pipeline execution**: All services within a group deploy simultaneously. * **Better resource utilization**: Parallel deployments make better use of Connectware's capabilities. * **Time savings**: Can reduce deployment time from several minutes to under a minute. {% hint style="info" %} For CI/CD pipelines with many services, the asynchronous approach is recommended to minimize pipeline runtime and improve developer productivity. {% endhint %} ### Host Configuration In both playbook approaches, we are using environment variables such as: {% code lineNumbers="true" %} ```bash "{{ lookup('env','CYBUS_CONNECTWARE_LICENSE') }}" ``` {% endcode %} * To set up the values of these variables, go to GitLab and select **Settings > CI/CD > Variables**. It is best practice to place sensitive information as environment variables and NOT in the codebase. * Create another file called `hosts.yaml` with the following content: {% code lineNumbers="true" %} ```yaml --- all: hosts: AWS: ansible_connection: ssh ansible_host: demo-devops.cybus.io ansible_port: 22 ansible_user: ubuntu ansible_ssh_private_key_file: "{{ lookup('env','SSH_KEY') }}" ``` {% endcode %} **Result:** This will help Ansible deploy the Connectware infrastructure to our cloud endpoint (`demo-devops.cybus.io`). ## 4. Running the Pipeline Make sure your project has: * Ansible playbook and host configuration under `/ansible` * Connectware service commissioning files under `/services` * CI/CD configuration file `.gitlab-ci.yml` * Necessary environment variables * Valid deployment endpoint (host) * Available runners (either cloud-based or self-hosted) Then on the left panel, select **CI/CD** > **Pipelines** > **Run Pipeline**.
**Result:** Your pipeline should be running soon. After a few minutes, the build will finish and GitLab will update the status to **Passed**. You should be able to open your host endpoint and log in to Connectware. In case the pipeline fails, the status will be updated to **Failed**. * To see more information, click the status button. This will display the logs from the execution.