AWS S3

AWS S3 (simple storage service) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Integrating AWS S3 with an AI platform enables advanced analytics, real-time insights, and predictive capabilities on machine data. AWS S3 excels in collecting, storing, and analyzing large volumes of machine-generated data from various sources, while AI platforms can leverage this data to perform deeper analysis, predictive modeling, and automation

Overview

This section guides you through configuring and running the integration between AWS S3 and NCP

By following these steps, you'll enable NCP to:

• Connect to your S3 buckets to retrieve syslogs & flowlogs

• Index and parse data for contextual analysis and querying

• Allow users to ask natural language questions and access S3-stored insights via NCP’s chat interface

• Improve operational visibility by leveraging historical and real-time data stored in S3

This section covers:

• Onboard AWS S3 platform

• Configure AWS S3 as a Data Source in NCP

Onboard AWS S3 Platform

Setup AWS Credentials and S3 Bucket

Before running the collector, ensure your AWS environment is configured correctly. You need to create an S3 bucket, generate an AWS IAM user, and attach the necessary policies to allow access to the bucket.

1. Create an S3 Bucket

   1. Go to the AWS Management Console and navigate to Amazon S3

   2. Click Create Bucket and follow the steps to create a new S3 bucket. Take note of the bucket name

      • Bucket name: <your-s3-bucket-name>

      • Region: Choose your preferred AWS region

      
2. Create an AWS IAM User

   • Navigate to the IAM dashboard in AWS

   • Click Users and then click Create User

   • Give the user a name (e.g., s3-collector-user)

   • TODO Select Programmatic access to allow API and CLI access for applications running outside the AWS ecosystem

   • Click Next to proceed to the permissions step

   • Access key creation with steps after creating the IAM user

   

   

   
3. Setup User Permissions

   • In the Set Permissions section, click Attach existing policies directly.

   • Search for and select the AmazonS3FullAccess policy.

   • Click Next to proceed to the next step.

4. Attach Inline Policies to the User

   • After creating the user, click on the user's name to view its details.

   • Navigate to the Permissions tab.

   • Click Add an inline policy to add a custom policy to provide specific access to your S3 bucket.

   • In the policy editor, configure the following policy below,

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::nflow-data-bucket",
                "arn:aws:s3:::nflow-data-bucket/*"
            ]
        }
    ]
}

Configure AWS S3 as a Data Source in NCP

To integrate your AWS S3 bucket with NCP:

1. Log in to the NCP GUI

2. Navigate to Settings → Data Sources → AWS S3

   
3. Click Add Configuration

4. Provide the following details:

   • Name: A friendly name for this S3 configuration

   • S3 Bucket: The name of your target S3 bucket

   • AWS Region: The region the bucket is hosted in (e.g., us-east-1)

   • Authentication Method: Use your Access Key/Secret

   
5. Click Save to complete the setup