OpenSearchIngestion / Client / create_pipeline

create_pipeline#

OpenSearchIngestion.Client.create_pipeline(**kwargs)#

Creates an OpenSearch Ingestion pipeline. For more information, see Creating Amazon OpenSearch Ingestion pipelines.

Request Syntax

response = client.create_pipeline(
    PipelineName='string',
    MinUnits=123,
    MaxUnits=123,
    PipelineConfigurationBody='string',
    LogPublishingOptions={
        'IsLoggingEnabled': True|False,
        'CloudWatchLogDestination': {
            'LogGroup': 'string'
        }
    },
    VpcOptions={
        'SubnetIds': [
            'string',
        ],
        'SecurityGroupIds': [
            'string',
        ],
        'VpcAttachmentOptions': {
            'AttachToVpc': True|False,
            'CidrBlock': 'string'
        },
        'VpcEndpointManagement': 'CUSTOMER'|'SERVICE'
    },
    BufferOptions={
        'PersistentBufferEnabled': True|False
    },
    EncryptionAtRestOptions={
        'KmsKeyArn': 'string'
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)

Parameters:

PipelineName (string) –
[REQUIRED]

The name of the OpenSearch Ingestion pipeline to create. Pipeline names are unique across the pipelines owned by an account within an Amazon Web Services Region.
MinUnits (integer) –
[REQUIRED]

The minimum pipeline capacity, in Ingestion Compute Units (ICUs).
MaxUnits (integer) –
[REQUIRED]

The maximum pipeline capacity, in Ingestion Compute Units (ICUs).
PipelineConfigurationBody (string) –
[REQUIRED]

The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n.
LogPublishingOptions (dict) –
Key-value pairs to configure log publishing.
- IsLoggingEnabled (boolean) –
  
  Whether logs should be published.
- CloudWatchLogDestination (dict) –
  
  The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.
  - LogGroup (string) – [REQUIRED]
    
    The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/vendedlogs/OpenSearchService/pipelines.
VpcOptions (dict) –
Container for the values required to configure VPC access for the pipeline. If you don’t specify these values, OpenSearch Ingestion creates the pipeline with a public endpoint.
- SubnetIds (list) – [REQUIRED]
  
  A list of subnet IDs associated with the VPC endpoint.
  - (string) –
- SecurityGroupIds (list) –
  
  A list of security groups associated with the VPC endpoint.
  - (string) –
- VpcAttachmentOptions (dict) –
  
  Options for attaching a VPC to a pipeline.
  - AttachToVpc (boolean) – [REQUIRED]
    
    Whether a VPC is attached to the pipeline.
  - CidrBlock (string) –
    
    The CIDR block to be reserved for OpenSearch Ingestion to create elastic network interfaces (ENIs).
- VpcEndpointManagement (string) –
  
  Defines whether you or Amazon OpenSearch Ingestion service create and manage the VPC endpoint configured for the pipeline.
BufferOptions (dict) –
Key-value pairs to configure persistent buffering for the pipeline.
- PersistentBufferEnabled (boolean) – [REQUIRED]
  
  Whether persistent buffering should be enabled.
EncryptionAtRestOptions (dict) –
Key-value pairs to configure encryption for data that is written to a persistent buffer.
- KmsKeyArn (string) – [REQUIRED]
  
  The ARN of the KMS key used to encrypt buffer data. By default, data is encrypted using an Amazon Web Services owned key.
Tags (list) –
List of tags to add to the pipeline upon creation.
- (dict) –
  
  A tag (key-value pair) for an OpenSearch Ingestion pipeline.
  - Key (string) – [REQUIRED]
    
    The tag key. Tag keys must be unique for the pipeline to which they are attached.
  - Value (string) – [REQUIRED]
    
    The value assigned to the corresponding tag key. Tag values can be null and don’t have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

Return type:

dict

Returns:

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ],
                    'VpcAttachmentOptions': {
                        'AttachToVpc': True|False,
                        'CidrBlock': 'string'
                    },
                    'VpcEndpointManagement': 'CUSTOMER'|'SERVICE'
                }
            },
        ],
        'BufferOptions': {
            'PersistentBufferEnabled': True|False
        },
        'EncryptionAtRestOptions': {
            'KmsKeyArn': 'string'
        },
        'VpcEndpointService': 'string',
        'ServiceVpcEndpoints': [
            {
                'ServiceName': 'OPENSEARCH_SERVERLESS',
                'VpcEndpointId': 'string'
            },
        ],
        'Destinations': [
            {
                'ServiceName': 'string',
                'Endpoint': 'string'
            },
        ],
        'Tags': [
            {
                'Key': 'string',
                'Value': 'string'
            },
        ]
    }
}

Response Structure

(dict) –
- Pipeline (dict) –
  
  Container for information about the created pipeline.
  - PipelineName (string) –
    
    The name of the pipeline.
  - PipelineArn (string) –
    
    The Amazon Resource Name (ARN) of the pipeline.
  - MinUnits (integer) –
    
    The minimum pipeline capacity, in Ingestion Compute Units (ICUs).
  - MaxUnits (integer) –
    
    The maximum pipeline capacity, in Ingestion Compute Units (ICUs).
  - Status (string) –
    
    The current status of the pipeline.
  - StatusReason (dict) –
    
    The reason for the current status of the pipeline.
    - Description (string) –
      
      A description of why a pipeline has a certain status.
  - PipelineConfigurationBody (string) –
    
    The Data Prepper pipeline configuration in YAML format.
  - CreatedAt (datetime) –
    
    The date and time when the pipeline was created.
  - LastUpdatedAt (datetime) –
    
    The date and time when the pipeline was last updated.
  - IngestEndpointUrls (list) –
    
    The ingestion endpoints for the pipeline, which you can send data to.
    - (string) –
  - LogPublishingOptions (dict) –
    
    Key-value pairs that represent log publishing settings.
    - IsLoggingEnabled (boolean) –
      
      Whether logs should be published.
    - CloudWatchLogDestination (dict) –
      
      The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.
      - LogGroup (string) –
        
        The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/vendedlogs/OpenSearchService/pipelines.
  - VpcEndpoints (list) –
    
    The VPC interface endpoints that have access to the pipeline.
    - (dict) –
      
      An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.
      - VpcEndpointId (string) –
        
        The unique identifier of the endpoint.
      - VpcId (string) –
        
        The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.
      - VpcOptions (dict) –
        
        Information about the VPC, including associated subnets and security groups.
        
        SubnetIds (list) –
        
        A list of subnet IDs associated with the VPC endpoint.
        
        (string) –
        
        SecurityGroupIds (list) –
        
        A list of security groups associated with the VPC endpoint.
        
        (string) –
        
        VpcAttachmentOptions (dict) –
        
        Options for attaching a VPC to a pipeline.
        
        AttachToVpc (boolean) –
        
        Whether a VPC is attached to the pipeline.
        
        CidrBlock (string) –
        
        The CIDR block to be reserved for OpenSearch Ingestion to create elastic network interfaces (ENIs).
        
        VpcEndpointManagement (string) –
        
        Defines whether you or Amazon OpenSearch Ingestion service create and manage the VPC endpoint configured for the pipeline.
  - BufferOptions (dict) –
    
    Options that specify the configuration of a persistent buffer. To configure how OpenSearch Ingestion encrypts this data, set the EncryptionAtRestOptions. For more information, see Persistent buffering.
    - PersistentBufferEnabled (boolean) –
      
      Whether persistent buffering should be enabled.
  - EncryptionAtRestOptions (dict) –
    
    Options to control how OpenSearch encrypts buffer data.
    - KmsKeyArn (string) –
      
      The ARN of the KMS key used to encrypt buffer data. By default, data is encrypted using an Amazon Web Services owned key.
  - VpcEndpointService (string) –
    
    The VPC endpoint service name for the pipeline.
  - ServiceVpcEndpoints (list) –
    
    A list of VPC endpoints that OpenSearch Ingestion has created to other Amazon Web Services services.
    - (dict) –
      
      A container for information about VPC endpoints that were created to other services
      - ServiceName (string) –
        
        The name of the service for which a VPC endpoint was created.
      - VpcEndpointId (string) –
        
        The unique identifier of the VPC endpoint that was created.
  - Destinations (list) –
    
    Destinations to which the pipeline writes data.
    - (dict) –
      
      An object representing the destination of a pipeline.
      - ServiceName (string) –
        
        The name of the service receiving data from the pipeline.
      - Endpoint (string) –
        
        The endpoint receiving data from the pipeline.
  - Tags (list) –
    
    A list of tags associated with the given pipeline.
    - (dict) –
      
      A tag (key-value pair) for an OpenSearch Ingestion pipeline.
      - Key (string) –
        
        The tag key. Tag keys must be unique for the pipeline to which they are attached.
      - Value (string) –
        
        The value assigned to the corresponding tag key. Tag values can be null and don’t have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

Exceptions

OpenSearchIngestion.Client.exceptions.DisabledOperationException
OpenSearchIngestion.Client.exceptions.LimitExceededException
OpenSearchIngestion.Client.exceptions.ValidationException
OpenSearchIngestion.Client.exceptions.InternalException
OpenSearchIngestion.Client.exceptions.AccessDeniedException
OpenSearchIngestion.Client.exceptions.ResourceAlreadyExistsException
OpenSearchIngestion.Client.exceptions.ResourceNotFoundException