Integration with AWS S3

AWS S3 (simple storage service) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Integrating AWS S3 with an AI platform enables advanced analytics, real-time insights, and predictive capabilities on machine data. AWS S3 excels in collecting, storing, and analyzing large volumes of machine-generated data from various sources, while AI platforms can leverage this data to perform deeper analysis, predictive modeling, and automation

This section covers:

• User Cases

• Onboard AWS S3 platform

• Deployment

Use Cases

• Application Flow monitoring using time-series data

• Fetch network logs, configurations, or event data stored in S3 for troubleshooting and performance analysis

• Real-time insights using application flow data

• Anomaly detection in security events

• Predictive maintenance using time-series data

• Automated response to network or system failures

Onboard AWS S3 Platform

Setup AWS Credentials and S3 Bucket

Before running the collector, ensure your AWS environment is configured correctly. You need to create an S3 bucket, generate an AWS IAM user, and attach the necessary policies to allow access to the bucket.

1. Create an S3 Bucket

   1. Go to the AWS Management Console and navigate to Amazon S3

   2. Click Create Bucket and follow the steps to create a new S3 bucket. Take note of the bucket name

      • Bucket name: <your-s3-bucket-name>

      • Region: Choose your preferred AWS region

      
2. Create an AWS IAM User

   • Navigate to the IAM dashboard in AWS

   • Click Users and then click Create User

   • Give the user a name (e.g., s3-collector-user)

   • TODO Select Programmatic access to allow API and CLI access for applications running outside the AWS ecosystem

   • Click Next to proceed to the permissions step

   • Access key creation with steps after creating the IAM user

   

   

   
3. Setup User Permissions

   • In the Set Permissions section, click Attach existing policies directly.

   • Search for and select the AmazonS3FullAccess policy.

   • Click Next to proceed to the next step.

4. Attach Inline Policies to the User

   • After creating the user, click on the user's name to view its details.

   • Navigate to the Permissions tab.

   • Click Add an inline policy to add a custom policy to provide specific access to your S3 bucket.

   • In the policy editor, configure the following policy below,

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::nflow-data-bucket",
                "arn:aws:s3:::nflow-data-bucket/*"
            ]
        }
    ]
}

Deployment

The AWS S3 Collector is part of the NCP package, that can be invoked from CLI to create integration

• Navigate to ingestion -> s3 inside the NCP directory

• Run the Installer ./install.sh

• Provide the installer with the information below,

  • AWS S3 Bucket Name

  • AWS Access Key and Secret

  • AWS Region

Uninstall AWS S3 Integration

To Uninstall AWS S3 integration run the below script from the NCP package,

• Navigate to ingestion -> s3 inside the NCP directory

• Run the Installer ./uninstall.sh