ONE-DL cloud Deployment

Using this section user can install ONES-DL backend AWS

Provisioning an EC2 Instance

AWS EC2 Instance Sizing for Event Ingestion

For efficient event ingestion, it's crucial to select the right EC2 instance type based on your event volume:

• 200K events/sec: Start with a minimum of an m5.4xlarge instance.

• Up to 400K events/sec: Scale up to an m5.8xlarge instance.

• Up to 600K events/sec: Opt for an m5.12xlarge instance.

• For higher volumes: Continue scaling in similar increments.

Document the `instance_id`, `vpc_id`, `region`, and `security_group_id` of the provisioned instance.

Install Docker

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
apt-cache policy docker-ce
sudo apt install docker-ce

Install Docker Compose

mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.3.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose
docker compose version

Install ONE-DL

• Copy the ONE-DL package to AWS Instance

• Untar the ONE-DL package

  • ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ tar -xvf one-dl.tar.gz
    docker-compose.yml
    one-dl-multitenant-installer.sh
    .env

• Modify the .env file with given input

  • ZOOKEEPER_SERVER_VALUE=    //Public DNS
    KAFKA_SERVER_VALUE=        //Public DNS
    SCHEMA_REGISTRY_SERVER_VALUE=    
    INSTANCE_ID=               //Instance ID
    VPC_ID=                    //VPC ID
    REGION=                    //REGION ID
    SG_ID=                     //SG ID

• Run ./one-dl-multitenant-installer.sh

aviz@aviz % ssh -i ones-dl-qa.pem ubuntu@34.215.5.89
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-1018-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

  System information as of Thu Mar  7 06:17:12 UTC 2024
  System load:  0.01513671875      Processes:                224
  Usage of /:   1.0% of 247.92GB   Users logged in:          1
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218
~
".env" 7L, 323B                                                                                                                                     1,1           All
  Memory usage: 0%                 IPv4 address for docker0: 172.17.0.1
  Swap usage:   0%                 IPv4 address for ens5:    172.31.28.5


Expanded Security Maintenance for Applications is not enabled.
39 updates can be applied immediately.
25 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable
7 additional security updates can be applied with ESM Apps.
Learn more about enabling ESM Apps service at https://ubuntu.com/esm


Last login: Thu Mar  7 06:10:37 2024 from 18.237.140.163
ubuntu@ip-172-31-28-5:~$
ubuntu@ip-172-31-28-5:~$ pwd
/home/ubuntu
ubuntu@ip-172-31-28-5:~$ cd ONES-DL-CLOUD/
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ pwd
/home/ubuntu/ONES-DL-CLOUD
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ls
one-dl.tar.gz
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ tar -xvf one-dl.tar.gz
docker-compose.yml
one-dl-multitenant-installer.sh
.env
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ vi .env
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218

ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cat .env
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218

ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cp .env .env-backup
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cat .env-backup
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218

ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ls
docker-compose.yml  one-dl-multitenant-installer.sh  one-dl.tar.gz

ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ./one-dl-multitenant-installer.sh
SHELL=/bin/bash
REGION=us-west-2
KAFKA_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com
PWD=/home/ubuntu/ONES-DL-CLOUD
LOGNAME=ubuntu
XDG_SESSION_TYPE=tty
SCHEMA_REGISTRY_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com
MOTD_SHOWN=pam
HOME=/home/ubuntu
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
VPC_ID=vpc-09a64a3439e9c1022
LC_TERMINAL=iTerm2
SSH_CONNECTION=43.224.130.156 61180 172.31.28.5 22
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=ubuntu
SG_ID=sg-04c3752b777d8f77e
LC_TERMINAL_VERSION=3.4.16
SHLVL=2
XDG_SESSION_ID=5
LC_CTYPE=C.UTF-8
XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT=43.224.130.156 61180 22
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
ZOOKEEPER_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
INSTANCE_ID=i-01300eb6edb18273b
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
SSH_TTY=/dev/pts/1
OLDPWD=/home/ubuntu
_=/usr/bin/env
us-west-2 i-01300eb6edb18273b vpc-09a64a3439e9c1022 sg-04c3752b777d8f77e
Error response from daemon: No such container: zookeeper
Error response from daemon: No such container: kafka
Error response from daemon: No such container: schema-registry
Error response from daemon: No such container: kafka-connect
Error response from daemon: No such container: zookeeper
Error response from daemon: No such container: kafka
Error response from daemon: No such container: schema-registry
Error response from daemon: No such container: kafka-connect
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-07f887d736a7c773c",
            "GroupId": "sg-04c3752b777d8f77e",
            "GroupOwnerId": "259789323334",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 29092,
            "ToPort": 29092,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0d446cf43dbf8c55a",
            "GroupId": "sg-04c3752b777d8f77e",
            "GroupOwnerId": "259789323334",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 9092,
            "ToPort": 9092,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-08d752b5cbfea8b1c",
            "GroupId": "sg-04c3752b777d8f77e",
            "GroupOwnerId": "259789323334",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 2181,
            "ToPort": 2181,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-058962812f2650b4e",
            "GroupId": "sg-04c3752b777d8f77e",
            "GroupOwnerId": "259789323334",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 8083,
            "ToPort": 8083,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-03a801549ac17a74d",
            "GroupId": "sg-04c3752b777d8f77e",
            "GroupOwnerId": "259789323334",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 8081,
            "ToPort": 8081,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-01ae4abe9d99c0086",
            "GroupId": "sg-04c3752b777d8f77e",
            "GroupOwnerId": "259789323334",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 8181,
            "ToPort": 8181,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}
[+] Running 10/10
 ✔ zookeeper 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                         5.1s
   ✔ 23828d760c7b Pull complete                                                                                                                                  0.6s
   ✔ e2670537dceb Pull complete                                                                                                                                  0.5s
   ✔ 3c059ccfa418 Pull complete                                                                                                                                  0.6s
   ✔ a23d33d59f2a Pull complete                                                                                                                                  0.7s
   ✔ 842a648f5439 Pull complete                                                                                                                                  0.9s
   ✔ 278244c5561f Pull complete                                                                                                                                  0.9s
   ✔ 0ab51e87b8dc Pull complete                                                                                                                                  1.0s
   ✔ 26ff518a75d1 Pull complete                                                                                                                                  1.4s
   ✔ 050c7bc3a5c4 Pull complete                                                                                                                                  1.1s
[+] Running 1/1
 ✔ Container zookeeper  Started                                                                                                                                  4.2s
[+] Running 12/12
 ✔ kafka 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                          9.4s
   ✔ fe36fc382320 Pull complete                                                                                                                                  0.7s
   ✔ 4250354b4fb7 Pull complete                                                                                                                                  2.1s
   ✔ c4c5f447179d Pull complete                                                                                                                                  0.3s
   ✔ 17fe3a92262f Pull complete                                                                                                                                  0.6s
   ✔ 5420596c14ab Pull complete                                                                                                                                  1.1s
   ✔ 0e55377ebe37 Pull complete                                                                                                                                  0.9s
   ✔ da7039bb2113 Pull complete                                                                                                                                  1.1s
   ✔ d389b3791c2e Pull complete                                                                                                                                  1.4s
   ✔ c24709eccb2a Pull complete                                                                                                                                  1.4s
   ✔ ad79ffd84e00 Pull complete                                                                                                                                  2.4s
   ✔ 218229bf30d9 Pull complete                                                                                                                                  1.7s
[+] Running 1/1
 ✔ Container kafka  Started                                                                                                                                      5.7s
[+] Running 12/12
 ✔ schema-registry 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                               17.9s
   ✔ fe36fc382320 Already exists                                                                                                                                 0.0s
   ✔ 4250354b4fb7 Already exists                                                                                                                                 0.0s
   ✔ c4c5f447179d Already exists                                                                                                                                 0.0s
   ✔ 17fe3a92262f Already exists                                                                                                                                 0.0s
   ✔ 5420596c14ab Already exists                                                                                                                                 0.0s
   ✔ 0e55377ebe37 Already exists                                                                                                                                 0.0s
   ✔ da7039bb2113 Already exists                                                                                                                                 0.0s
   ✔ d389b3791c2e Already exists                                                                                                                                 0.0s
   ✔ c24709eccb2a Already exists                                                                                                                                 0.0s
   ✔ e69ded016b1a Pull complete                                                                                                                                  8.9s
   ✔ 83e8ec805f43 Pull complete                                                                                                                                  0.3s
[+] Running 1/1
 ✔ Container schema-registry  Started                                                                                                                           18.7s
[+] Running 21/21
 ✔ kafka-connect 20 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                        18.1s
   ✔ a49367d57626 Pull complete                                                                                                                                  0.7s
   ✔ eeec2f37f4a0 Pull complete                                                                                                                                  2.8s
   ✔ ae805807db54 Pull complete                                                                                                                                  1.4s
   ✔ c553584e96cc Pull complete                                                                                                                                  1.1s
   ✔ 28769e081b94 Pull complete                                                                                                                                  2.4s
   ✔ 151cf144f79d Pull complete                                                                                                                                  1.9s
   ✔ 04292bb11042 Pull complete                                                                                                                                  2.3s
   ✔ c2e0c70921ea Pull complete                                                                                                                                  2.7s
   ✔ e8d7f71a6ca4 Pull complete                                                                                                                                  2.9s
   ✔ eb27eb7b8f05 Pull complete                                                                                                                                  4.7s
   ✔ 9bb8b7d5f6a0 Pull complete                                                                                                                                  3.3s
   ✔ 302eaef771e0 Pull complete                                                                                                                                 11.7s
   ✔ 1d0d5b44b6f1 Pull complete                                                                                                                                  3.8s
   ✔ 22df7873ee50 Pull complete                                                                                                                                  4.4s
   ✔ 4f4fb700ef54 Pull complete                                                                                                                                  4.7s
   ✔ 972adc085ab6 Pull complete                                                                                                                                  7.0s
   ✔ c4f994822767 Pull complete                                                                                                                                  6.1s
   ✔ 70547799762d Pull complete                                                                                                                                  7.0s
   ✔ 3f0c034e271c Pull complete                                                                                                                                  7.5s
   ✔ b6195a937e7b Pull complete                                                                                                                                  7.7s
[+] Running 1/1
 ✔ Container kafka-connect  Started                                                                                                                              4.6s
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ sudo su
root@ip-172-31-28-5:/home/ubuntu/ONES-DL-CLOUD# docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED              STATUS                             PORTS     NAMES
ab8aa46cc746   avizdock/kafka-connect:latest     "/etc/confluent/dock…"   28 seconds ago       Up 23 seconds (health: starting)             kafka-connect
346522f4baf5   confluentinc/cp-schema-registry   "/etc/confluent/dock…"   About a minute ago   Up About a minute                            schema-registry
23c50c44bc5b   confluentinc/cp-kafka:latest      "/etc/confluent/dock…"   2 minutes ago        Up 2 minutes                                 kafka
633eba10034d   zookeeper                         "/docker-entrypoint.…"   3 minutes ago        Up 3 minutes                                 zookeeper
root@ip-172-31-28-5:/home/ubuntu/ONES-DL-CLOUD#