ONE-DL cloud Deployment
Using this section user can install ONES-DL backend AWS
Provisioning an EC2 Instance
AWS EC2 Instance Sizing for Event Ingestion
For efficient event ingestion, it's crucial to select the right EC2 instance type based on your event volume:
200K events/sec: Start with a minimum of an
m5.4xlargeinstance.Up to 400K events/sec: Scale up to an
m5.8xlargeinstance.Up to 600K events/sec: Opt for an
m5.12xlargeinstance.For higher volumes: Continue scaling in similar increments.
Document the `instance_id`, `vpc_id`, `region`, and `security_group_id` of the provisioned instance.Install Docker
sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
apt-cache policy docker-ce
sudo apt install docker-ce
Install Docker Compose
mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.3.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose
docker compose version
Install ONE-DL
Copy the ONE-DL package to AWS Instance
Untar the ONE-DL package
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ tar -xvf one-dl.tar.gz docker-compose.yml one-dl-multitenant-installer.sh .env
Modify the .env file with given input
ZOOKEEPER_SERVER_VALUE= //Public DNS KAFKA_SERVER_VALUE= //Public DNS SCHEMA_REGISTRY_SERVER_VALUE= INSTANCE_ID= //Instance ID VPC_ID= //VPC ID REGION= //REGION ID SG_ID= //SG ID
Run
./one-dl-multitenant-installer.sh
aviz@aviz % ssh -i ones-dl-qa.pem ubuntu@34.215.5.89
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-1018-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Thu Mar 7 06:17:12 UTC 2024
System load: 0.01513671875 Processes: 224
Usage of /: 1.0% of 247.92GB Users logged in: 1
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218
~
".env" 7L, 323B 1,1 All
Memory usage: 0% IPv4 address for docker0: 172.17.0.1
Swap usage: 0% IPv4 address for ens5: 172.31.28.5
Expanded Security Maintenance for Applications is not enabled.
39 updates can be applied immediately.
25 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable
7 additional security updates can be applied with ESM Apps.
Learn more about enabling ESM Apps service at https://ubuntu.com/esm
Last login: Thu Mar 7 06:10:37 2024 from 18.237.140.163
ubuntu@ip-172-31-28-5:~$
ubuntu@ip-172-31-28-5:~$ pwd
/home/ubuntu
ubuntu@ip-172-31-28-5:~$ cd ONES-DL-CLOUD/
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ pwd
/home/ubuntu/ONES-DL-CLOUD
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ls
one-dl.tar.gz
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ tar -xvf one-dl.tar.gz
docker-compose.yml
one-dl-multitenant-installer.sh
.env
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ vi .env
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cat .env
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cp .env .env-backup
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cat .env-backup
ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com
INSTANCE_ID=i-06e36f82c35173cc9
VPC_ID=vpc-09a64a3439e9c1022
REGION=us-west-2
SG_ID=sg-0057cdb5b97e1d218
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ls
docker-compose.yml one-dl-multitenant-installer.sh one-dl.tar.gz
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ./one-dl-multitenant-installer.sh
SHELL=/bin/bash
REGION=us-west-2
KAFKA_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com
PWD=/home/ubuntu/ONES-DL-CLOUD
LOGNAME=ubuntu
XDG_SESSION_TYPE=tty
SCHEMA_REGISTRY_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com
MOTD_SHOWN=pam
HOME=/home/ubuntu
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
VPC_ID=vpc-09a64a3439e9c1022
LC_TERMINAL=iTerm2
SSH_CONNECTION=43.224.130.156 61180 172.31.28.5 22
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=ubuntu
SG_ID=sg-04c3752b777d8f77e
LC_TERMINAL_VERSION=3.4.16
SHLVL=2
XDG_SESSION_ID=5
LC_CTYPE=C.UTF-8
XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT=43.224.130.156 61180 22
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
ZOOKEEPER_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
INSTANCE_ID=i-01300eb6edb18273b
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
SSH_TTY=/dev/pts/1
OLDPWD=/home/ubuntu
_=/usr/bin/env
us-west-2 i-01300eb6edb18273b vpc-09a64a3439e9c1022 sg-04c3752b777d8f77e
Error response from daemon: No such container: zookeeper
Error response from daemon: No such container: kafka
Error response from daemon: No such container: schema-registry
Error response from daemon: No such container: kafka-connect
Error response from daemon: No such container: zookeeper
Error response from daemon: No such container: kafka
Error response from daemon: No such container: schema-registry
Error response from daemon: No such container: kafka-connect
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-07f887d736a7c773c",
"GroupId": "sg-04c3752b777d8f77e",
"GroupOwnerId": "259789323334",
"IsEgress": false,
"IpProtocol": "tcp",
"FromPort": 29092,
"ToPort": 29092,
"CidrIpv4": "0.0.0.0/0"
}
]
}
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-0d446cf43dbf8c55a",
"GroupId": "sg-04c3752b777d8f77e",
"GroupOwnerId": "259789323334",
"IsEgress": false,
"IpProtocol": "tcp",
"FromPort": 9092,
"ToPort": 9092,
"CidrIpv4": "0.0.0.0/0"
}
]
}
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-08d752b5cbfea8b1c",
"GroupId": "sg-04c3752b777d8f77e",
"GroupOwnerId": "259789323334",
"IsEgress": false,
"IpProtocol": "tcp",
"FromPort": 2181,
"ToPort": 2181,
"CidrIpv4": "0.0.0.0/0"
}
]
}
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-058962812f2650b4e",
"GroupId": "sg-04c3752b777d8f77e",
"GroupOwnerId": "259789323334",
"IsEgress": false,
"IpProtocol": "tcp",
"FromPort": 8083,
"ToPort": 8083,
"CidrIpv4": "0.0.0.0/0"
}
]
}
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-03a801549ac17a74d",
"GroupId": "sg-04c3752b777d8f77e",
"GroupOwnerId": "259789323334",
"IsEgress": false,
"IpProtocol": "tcp",
"FromPort": 8081,
"ToPort": 8081,
"CidrIpv4": "0.0.0.0/0"
}
]
}
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-01ae4abe9d99c0086",
"GroupId": "sg-04c3752b777d8f77e",
"GroupOwnerId": "259789323334",
"IsEgress": false,
"IpProtocol": "tcp",
"FromPort": 8181,
"ToPort": 8181,
"CidrIpv4": "0.0.0.0/0"
}
]
}
[+] Running 10/10
✔ zookeeper 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 5.1s
✔ 23828d760c7b Pull complete 0.6s
✔ e2670537dceb Pull complete 0.5s
✔ 3c059ccfa418 Pull complete 0.6s
✔ a23d33d59f2a Pull complete 0.7s
✔ 842a648f5439 Pull complete 0.9s
✔ 278244c5561f Pull complete 0.9s
✔ 0ab51e87b8dc Pull complete 1.0s
✔ 26ff518a75d1 Pull complete 1.4s
✔ 050c7bc3a5c4 Pull complete 1.1s
[+] Running 1/1
✔ Container zookeeper Started 4.2s
[+] Running 12/12
✔ kafka 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 9.4s
✔ fe36fc382320 Pull complete 0.7s
✔ 4250354b4fb7 Pull complete 2.1s
✔ c4c5f447179d Pull complete 0.3s
✔ 17fe3a92262f Pull complete 0.6s
✔ 5420596c14ab Pull complete 1.1s
✔ 0e55377ebe37 Pull complete 0.9s
✔ da7039bb2113 Pull complete 1.1s
✔ d389b3791c2e Pull complete 1.4s
✔ c24709eccb2a Pull complete 1.4s
✔ ad79ffd84e00 Pull complete 2.4s
✔ 218229bf30d9 Pull complete 1.7s
[+] Running 1/1
✔ Container kafka Started 5.7s
[+] Running 12/12
✔ schema-registry 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 17.9s
✔ fe36fc382320 Already exists 0.0s
✔ 4250354b4fb7 Already exists 0.0s
✔ c4c5f447179d Already exists 0.0s
✔ 17fe3a92262f Already exists 0.0s
✔ 5420596c14ab Already exists 0.0s
✔ 0e55377ebe37 Already exists 0.0s
✔ da7039bb2113 Already exists 0.0s
✔ d389b3791c2e Already exists 0.0s
✔ c24709eccb2a Already exists 0.0s
✔ e69ded016b1a Pull complete 8.9s
✔ 83e8ec805f43 Pull complete 0.3s
[+] Running 1/1
✔ Container schema-registry Started 18.7s
[+] Running 21/21
✔ kafka-connect 20 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 18.1s
✔ a49367d57626 Pull complete 0.7s
✔ eeec2f37f4a0 Pull complete 2.8s
✔ ae805807db54 Pull complete 1.4s
✔ c553584e96cc Pull complete 1.1s
✔ 28769e081b94 Pull complete 2.4s
✔ 151cf144f79d Pull complete 1.9s
✔ 04292bb11042 Pull complete 2.3s
✔ c2e0c70921ea Pull complete 2.7s
✔ e8d7f71a6ca4 Pull complete 2.9s
✔ eb27eb7b8f05 Pull complete 4.7s
✔ 9bb8b7d5f6a0 Pull complete 3.3s
✔ 302eaef771e0 Pull complete 11.7s
✔ 1d0d5b44b6f1 Pull complete 3.8s
✔ 22df7873ee50 Pull complete 4.4s
✔ 4f4fb700ef54 Pull complete 4.7s
✔ 972adc085ab6 Pull complete 7.0s
✔ c4f994822767 Pull complete 6.1s
✔ 70547799762d Pull complete 7.0s
✔ 3f0c034e271c Pull complete 7.5s
✔ b6195a937e7b Pull complete 7.7s
[+] Running 1/1
✔ Container kafka-connect Started 4.6s
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$
ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ sudo su
root@ip-172-31-28-5:/home/ubuntu/ONES-DL-CLOUD# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ab8aa46cc746 avizdock/kafka-connect:latest "/etc/confluent/dock…" 28 seconds ago Up 23 seconds (health: starting) kafka-connect
346522f4baf5 confluentinc/cp-schema-registry "/etc/confluent/dock…" About a minute ago Up About a minute schema-registry
23c50c44bc5b confluentinc/cp-kafka:latest "/etc/confluent/dock…" 2 minutes ago Up 2 minutes kafka
633eba10034d zookeeper "/docker-entrypoint.…" 3 minutes ago Up 3 minutes zookeeper
root@ip-172-31-28-5:/home/ubuntu/ONES-DL-CLOUD#Last updated