# ONE-DL cloud Deployment ## Provisioning an EC2 Instance ### AWS EC2 Instance Sizing for Event Ingestion For efficient event ingestion, it's crucial to select the right EC2 instance type based on your event volume: * **200K events/sec**: Start with a minimum of an `m5.4xlarge` instance. * **Up to 400K events/sec**: Scale up to an `m5.8xlarge` instance. * **Up to 600K events/sec**: Opt for an `m5.12xlarge` instance. * **For higher volumes**: Continue scaling in similar increments. {% hint style="info" %} {% code overflow="wrap" %} ```markdown Document the `instance_id`, `vpc_id`, `region`, and `security_group_id` of the provisioned instance. ``` {% endcode %} {% endhint %} ## Install Docker ```log sudo apt update sudo apt install apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt update apt-cache policy docker-ce sudo apt install docker-ce ``` ## Install Docker Compose ```log mkdir -p ~/.docker/cli-plugins/ curl -SL https://github.com/docker/compose/releases/download/v2.3.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose chmod +x ~/.docker/cli-plugins/docker-compose docker compose version ``` ## Install ONE-DL * Copy the ONE-DL package to AWS Instance * Untar the ONE-DL package * ```log ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ tar -xvf one-dl.tar.gz docker-compose.yml one-dl-multitenant-installer.sh .env ``` * Modify the .env file with given input * ``` ZOOKEEPER_SERVER_VALUE= //Public DNS KAFKA_SERVER_VALUE= //Public DNS SCHEMA_REGISTRY_SERVER_VALUE= INSTANCE_ID= //Instance ID VPC_ID= //VPC ID REGION= //REGION ID SG_ID= //SG ID ``` * Run `./one-dl-multitenant-installer.sh` ```log aviz@aviz % ssh -i ones-dl-qa.pem ubuntu@34.215.5.89 Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-1018-aws x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro System information as of Thu Mar 7 06:17:12 UTC 2024 System load: 0.01513671875 Processes: 224 Usage of /: 1.0% of 247.92GB Users logged in: 1 ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com INSTANCE_ID=i-06e36f82c35173cc9 VPC_ID=vpc-09a64a3439e9c1022 REGION=us-west-2 SG_ID=sg-0057cdb5b97e1d218 ~ ".env" 7L, 323B 1,1 All Memory usage: 0% IPv4 address for docker0: 172.17.0.1 Swap usage: 0% IPv4 address for ens5: 172.31.28.5 Expanded Security Maintenance for Applications is not enabled. 39 updates can be applied immediately. 25 of these updates are standard security updates. To see these additional updates run: apt list --upgradable 7 additional security updates can be applied with ESM Apps. Learn more about enabling ESM Apps service at https://ubuntu.com/esm Last login: Thu Mar 7 06:10:37 2024 from 18.237.140.163 ubuntu@ip-172-31-28-5:~$ ubuntu@ip-172-31-28-5:~$ pwd /home/ubuntu ubuntu@ip-172-31-28-5:~$ cd ONES-DL-CLOUD/ ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ pwd /home/ubuntu/ONES-DL-CLOUD ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ls one-dl.tar.gz ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ tar -xvf one-dl.tar.gz docker-compose.yml one-dl-multitenant-installer.sh .env ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ vi .env ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com INSTANCE_ID=i-06e36f82c35173cc9 VPC_ID=vpc-09a64a3439e9c1022 REGION=us-west-2 SG_ID=sg-0057cdb5b97e1d218 ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cat .env ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com INSTANCE_ID=i-06e36f82c35173cc9 VPC_ID=vpc-09a64a3439e9c1022 REGION=us-west-2 SG_ID=sg-0057cdb5b97e1d218 ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cp .env .env-backup ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ cat .env-backup ZOOKEEPER_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com KAFKA_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com SCHEMA_REGISTRY_SERVER_VALUE=ec2-35-93-19-159.us-west-2.compute.amazonaws.com INSTANCE_ID=i-06e36f82c35173cc9 VPC_ID=vpc-09a64a3439e9c1022 REGION=us-west-2 SG_ID=sg-0057cdb5b97e1d218 ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ls docker-compose.yml one-dl-multitenant-installer.sh one-dl.tar.gz ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ./one-dl-multitenant-installer.sh SHELL=/bin/bash REGION=us-west-2 KAFKA_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com PWD=/home/ubuntu/ONES-DL-CLOUD LOGNAME=ubuntu XDG_SESSION_TYPE=tty SCHEMA_REGISTRY_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com MOTD_SHOWN=pam HOME=/home/ubuntu LANG=C.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: VPC_ID=vpc-09a64a3439e9c1022 LC_TERMINAL=iTerm2 SSH_CONNECTION=43.224.130.156 61180 172.31.28.5 22 LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe %s USER=ubuntu SG_ID=sg-04c3752b777d8f77e LC_TERMINAL_VERSION=3.4.16 SHLVL=2 XDG_SESSION_ID=5 LC_CTYPE=C.UTF-8 XDG_RUNTIME_DIR=/run/user/1000 SSH_CLIENT=43.224.130.156 61180 22 XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop ZOOKEEPER_SERVER_VALUE=ec2-34-215-5-89.us-west-2.compute.amazonaws.com PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin INSTANCE_ID=i-01300eb6edb18273b DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus SSH_TTY=/dev/pts/1 OLDPWD=/home/ubuntu _=/usr/bin/env us-west-2 i-01300eb6edb18273b vpc-09a64a3439e9c1022 sg-04c3752b777d8f77e Error response from daemon: No such container: zookeeper Error response from daemon: No such container: kafka Error response from daemon: No such container: schema-registry Error response from daemon: No such container: kafka-connect Error response from daemon: No such container: zookeeper Error response from daemon: No such container: kafka Error response from daemon: No such container: schema-registry Error response from daemon: No such container: kafka-connect { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-07f887d736a7c773c", "GroupId": "sg-04c3752b777d8f77e", "GroupOwnerId": "259789323334", "IsEgress": false, "IpProtocol": "tcp", "FromPort": 29092, "ToPort": 29092, "CidrIpv4": "0.0.0.0/0" } ] } { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-0d446cf43dbf8c55a", "GroupId": "sg-04c3752b777d8f77e", "GroupOwnerId": "259789323334", "IsEgress": false, "IpProtocol": "tcp", "FromPort": 9092, "ToPort": 9092, "CidrIpv4": "0.0.0.0/0" } ] } { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-08d752b5cbfea8b1c", "GroupId": "sg-04c3752b777d8f77e", "GroupOwnerId": "259789323334", "IsEgress": false, "IpProtocol": "tcp", "FromPort": 2181, "ToPort": 2181, "CidrIpv4": "0.0.0.0/0" } ] } { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-058962812f2650b4e", "GroupId": "sg-04c3752b777d8f77e", "GroupOwnerId": "259789323334", "IsEgress": false, "IpProtocol": "tcp", "FromPort": 8083, "ToPort": 8083, "CidrIpv4": "0.0.0.0/0" } ] } { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-03a801549ac17a74d", "GroupId": "sg-04c3752b777d8f77e", "GroupOwnerId": "259789323334", "IsEgress": false, "IpProtocol": "tcp", "FromPort": 8081, "ToPort": 8081, "CidrIpv4": "0.0.0.0/0" } ] } { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-01ae4abe9d99c0086", "GroupId": "sg-04c3752b777d8f77e", "GroupOwnerId": "259789323334", "IsEgress": false, "IpProtocol": "tcp", "FromPort": 8181, "ToPort": 8181, "CidrIpv4": "0.0.0.0/0" } ] } [+] Running 10/10 ✔ zookeeper 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 5.1s ✔ 23828d760c7b Pull complete 0.6s ✔ e2670537dceb Pull complete 0.5s ✔ 3c059ccfa418 Pull complete 0.6s ✔ a23d33d59f2a Pull complete 0.7s ✔ 842a648f5439 Pull complete 0.9s ✔ 278244c5561f Pull complete 0.9s ✔ 0ab51e87b8dc Pull complete 1.0s ✔ 26ff518a75d1 Pull complete 1.4s ✔ 050c7bc3a5c4 Pull complete 1.1s [+] Running 1/1 ✔ Container zookeeper Started 4.2s [+] Running 12/12 ✔ kafka 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 9.4s ✔ fe36fc382320 Pull complete 0.7s ✔ 4250354b4fb7 Pull complete 2.1s ✔ c4c5f447179d Pull complete 0.3s ✔ 17fe3a92262f Pull complete 0.6s ✔ 5420596c14ab Pull complete 1.1s ✔ 0e55377ebe37 Pull complete 0.9s ✔ da7039bb2113 Pull complete 1.1s ✔ d389b3791c2e Pull complete 1.4s ✔ c24709eccb2a Pull complete 1.4s ✔ ad79ffd84e00 Pull complete 2.4s ✔ 218229bf30d9 Pull complete 1.7s [+] Running 1/1 ✔ Container kafka Started 5.7s [+] Running 12/12 ✔ schema-registry 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 17.9s ✔ fe36fc382320 Already exists 0.0s ✔ 4250354b4fb7 Already exists 0.0s ✔ c4c5f447179d Already exists 0.0s ✔ 17fe3a92262f Already exists 0.0s ✔ 5420596c14ab Already exists 0.0s ✔ 0e55377ebe37 Already exists 0.0s ✔ da7039bb2113 Already exists 0.0s ✔ d389b3791c2e Already exists 0.0s ✔ c24709eccb2a Already exists 0.0s ✔ e69ded016b1a Pull complete 8.9s ✔ 83e8ec805f43 Pull complete 0.3s [+] Running 1/1 ✔ Container schema-registry Started 18.7s [+] Running 21/21 ✔ kafka-connect 20 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 18.1s ✔ a49367d57626 Pull complete 0.7s ✔ eeec2f37f4a0 Pull complete 2.8s ✔ ae805807db54 Pull complete 1.4s ✔ c553584e96cc Pull complete 1.1s ✔ 28769e081b94 Pull complete 2.4s ✔ 151cf144f79d Pull complete 1.9s ✔ 04292bb11042 Pull complete 2.3s ✔ c2e0c70921ea Pull complete 2.7s ✔ e8d7f71a6ca4 Pull complete 2.9s ✔ eb27eb7b8f05 Pull complete 4.7s ✔ 9bb8b7d5f6a0 Pull complete 3.3s ✔ 302eaef771e0 Pull complete 11.7s ✔ 1d0d5b44b6f1 Pull complete 3.8s ✔ 22df7873ee50 Pull complete 4.4s ✔ 4f4fb700ef54 Pull complete 4.7s ✔ 972adc085ab6 Pull complete 7.0s ✔ c4f994822767 Pull complete 6.1s ✔ 70547799762d Pull complete 7.0s ✔ 3f0c034e271c Pull complete 7.5s ✔ b6195a937e7b Pull complete 7.7s [+] Running 1/1 ✔ Container kafka-connect Started 4.6s ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ ubuntu@ip-172-31-28-5:~/ONES-DL-CLOUD$ sudo su root@ip-172-31-28-5:/home/ubuntu/ONES-DL-CLOUD# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ab8aa46cc746 avizdock/kafka-connect:latest "/etc/confluent/dock…" 28 seconds ago Up 23 seconds (health: starting) kafka-connect 346522f4baf5 confluentinc/cp-schema-registry "/etc/confluent/dock…" About a minute ago Up About a minute schema-registry 23c50c44bc5b confluentinc/cp-kafka:latest "/etc/confluent/dock…" 2 minutes ago Up 2 minutes kafka 633eba10034d zookeeper "/docker-entrypoint.…" 3 minutes ago Up 3 minutes zookeeper root@ip-172-31-28-5:/home/ubuntu/ONES-DL-CLOUD# ```