.png)
OpenSearch Integration Guide
Connect OpenSearch with enterprise applications through HTTP/JSON REST APIs, bulk operations, search queries, and selected alert notifications.
OpenSearch integration options at a glance
OpenSearch primarily integrates through HTTP/JSON REST APIs for index and document operations, search, aggregations, cluster administration, SQL or PPL queries, and plugin features. Its Bulk API supports grouped indexing, updates, and deletes, while scroll, point-in-time, search_after, multi-search, and multi-get support larger retrieval and batch-processing scenarios. Alerting and Notifications can provide selected webhook-style notifications, but not a universal document change stream. Authentication may use Basic Authentication, TLS, JWT, OpenID Connect, SAML, or AWS SigV4 depending on deployment. Martini can consume these APIs, transform JSON, expose controlled façades, schedule synchronization, and orchestrate retries.
| Integration point | Supported by OpenSearch? | Common use cases | How Martini supports it |
|---|---|---|---|
| REST APIs | Yes | Manage indexes, mappings, aliases, documents, searches, aggregations, cluster health, plugins, SQL, and PPL through HTTP/JSON. | Martini can consume OpenSearch REST endpoints, construct JSON requests, validate responses, transform results, and expose reusable APIs or workflows. |
| Bulk / async / batch APIs | Yes | Index, update, or delete many documents through the Bulk API; use multi-search, multi-get, scroll, point-in-time, search_after, and asynchronous search for batch scenarios. | Martini can batch source data, generate newline-delimited JSON, submit bounded requests, and process each bulk response item independently. |
| Webhooks / outbound callbacks | Limited | Alerting and Notifications can send selected monitor or alert notifications to configured destinations, depending on plugins and deployment configuration. | Martini can expose an authenticated API endpoint to receive selected notifications and route, enrich, or transform them. This is not a universal document-change webhook. |
| Database / analytics access | Limited | SQL and PPL APIs support search and analytics queries, but OpenSearch is not a general-purpose relational database. | Martini can call SQL or PPL REST endpoints and map result sets into reports, APIs, or downstream workflows. |
| Authentication | Yes | Deployments may use Basic Authentication, TLS, JWT, OpenID Connect, SAML, role-based access control, or AWS SigV4 depending on configuration. | Martini can keep credentials and environment-specific settings in protected configuration, call authenticated endpoints, and apply response and authorization checks. |
| File / attachment APIs | No | OpenSearch is document and search oriented rather than a general file or attachment repository. External files must normally be processed before indexing. | Martini can read files from supported external systems, transform their contents into JSON documents, and submit them to OpenSearch through REST or Bulk APIs. |
| Messaging | Limited | OpenSearch may participate in event-streaming or ingestion architectures through external systems and ingestion components, rather than a universal native messaging interface. | Martini can orchestrate messaging workflows around OpenSearch and call its REST APIs after consuming or receiving events. |
How OpenSearch exposes data and business events
OpenSearch REST APIs
OpenSearch exposes HTTP/JSON REST APIs for indexes, documents, searches, aggregations, cluster administration, ingest pipelines, plugins, SQL, and PPL. REST is the most portable and important integration mechanism for Martini.
Martini implementation pattern
Martini implementation pattern: A workflow or API receives a request, builds the appropriate OpenSearch JSON payload, calls the configured REST endpoint with environment-specific authentication, validates the response, and maps the result or error into the calling system's contract.
Implementation sequence
Bulk and batch APIs
The Bulk API supports grouped indexing, updates, and deletes using newline-delimited JSON. Multi-get, multi-search, scroll, point-in-time, search_after, and asynchronous search support additional batch and large-result patterns.
Martini implementation pattern
Martini implementation pattern: A scheduled or event-driven workflow groups source changes into bounded batches, generates bulk operations, submits them, and inspects each item in the response. Transient failures are retried selectively while permanent mapping or validation failures are isolated.
Implementation sequence
Alert notifications
OpenSearch Alerting and Notifications can send selected monitor or alert notifications to configured destinations when the required plugins and channels are available. These notifications are not a universal document-level change stream.
Martini implementation pattern
Martini implementation pattern: Martini exposes an authenticated API endpoint for selected OpenSearch notifications, validates the notification shape, enriches it with search or operational context when needed, and routes it to incident or collaboration applications.
Implementation sequence
SQL and PPL APIs
OpenSearch provides SQL and PPL query interfaces through REST APIs for analytics and reporting. Availability and feature coverage depend on the target version and deployment.
Martini implementation pattern
Martini implementation pattern: A workflow or API accepts an approved query or report request, applies allowlists and limits, invokes the SQL or PPL endpoint, and transforms the returned result into a report, response, or downstream message.
Implementation sequence
Common OpenSearch integration patterns
Pattern 1: Index application logs and events
When to use this pattern
Use this pattern when applications, agents, or messaging systems produce structured logs or events that must be searchable in OpenSearch. It supports normalization, enrichment, bounded bulk ingestion, and operational handling of rejected documents.
Integration direction
Example Mapping
| OpenSearch Field | Canonical Field | Target Field |
|---|---|---|
| timestamp | eventTimestamp | @timestamp |
| serviceName | sourceService | service.name |
| severity | eventSeverity | log.level |
| correlationId | traceCorrelationId | correlation.id |
Martini implementation pattern
Martini receives an API, webhook, or messaging payload, validates JSON, adds integration metadata and index-routing values, assigns a deterministic ID where appropriate, and submits a document or bulk request. It separates item-level mapping failures from transient transport or cluster failures and routes them accordingly.
Martini capabilities used
- workflows
- API consumption
- data mapping
- business rules
- error handling
Pattern 2: Synchronize source data into an OpenSearch index
When to use this pattern
Use this pattern when a business application or data service must be represented in OpenSearch for search or operational reporting. Incremental processing should use a source timestamp, version, cursor, or upstream event rather than assuming OpenSearch provides universal change capture.
Integration direction
Example Mapping
| OpenSearch Field | Canonical Field | Target Field |
|---|---|---|
| sourceId | businessObjectId | _id |
| updatedAt | lastChangedAt | updated_at |
| displayName | searchableName | name |
| status | lifecycleStatus | status |
Martini implementation pattern
A scheduled Martini workflow retrieves changed source objects, persists the cursor or last successful position, maps them to the index model, and uses deterministic IDs for idempotent upserts. Bulk responses are processed per item, with retryable failures retried and permanent mapping errors sent to an exception path.
Martini capabilities used
- scheduled workflows
- API consumption
- data mapping
- state and checkpoint handling
- error handling
Pattern 3: Expose a controlled OpenSearch search façade
When to use this pattern
Use this pattern when applications need search access without receiving direct OpenSearch credentials or unrestricted Query DSL access. The façade can stabilize the consumer contract and enforce index, field, pagination, and result-size policies.
Integration direction
Example Mapping
| OpenSearch Field | Canonical Field | Target Field |
|---|---|---|
| customerQuery | searchCriteria | query.bool.must |
| pageToken | searchCursor | search_after |
| pageSize | resultLimit | size |
| searchHit._source | resultItem | items[] |
Martini implementation pattern
Martini exposes a REST API that authenticates callers, validates filters and limits, translates approved parameters into OpenSearch Query DSL, invokes the Search API, and maps hits into a stable response. It rejects unsafe queries, enforces pagination, and records correlation information for troubleshooting.
Martini capabilities used
- API exposure
- API consumption
- data mapping
- business rules
- authentication and authorization
Pattern 4: Route OpenSearch alert notifications
When to use this pattern
Use this pattern for selected monitor or alert conditions when OpenSearch Alerting and Notifications are configured. It is appropriate for incident routing and enrichment, but not for complete document-level change propagation.
Integration direction
Example Mapping
| OpenSearch Field | Canonical Field | Target Field |
|---|---|---|
| monitor.name | alertSource | caller_id or source |
| trigger.name | alertRule | short_description |
| alert.severity | priority | priority |
| alert.id | deduplicationKey | correlation_id |
Martini implementation pattern
OpenSearch sends a selected notification to a Martini API endpoint. Martini authenticates and validates it, optionally retrieves additional OpenSearch context, applies severity and ownership rules, and calls the target incident API. Correlation identifiers prevent duplicate incidents during retries.
Martini capabilities used
- API exposure
- API consumption
- data mapping
- business rules
- error handling
Applications commonly integrated with OpenSearch
OpenSearch commonly participates in logging, observability, event-streaming, analytics, and incident-management architectures. Martini can coordinate these relationships through REST APIs, notification endpoints, scheduled workflows, and messaging integrations without requiring a dedicated OpenSearch connector.
| Application | Scenario | Direction | Martini Pattern |
|---|---|---|---|
| Fluent Bit | Collect and transform container, host, and application logs before indexing them for search and observability. | Fluent Bit → Martini → OpenSearch | Martini receives or consumes normalized log events, validates and enriches fields such as timestamp, service, severity, and correlation ID, then submits individual or bulk document requests to OpenSearch. Item-level failures are routed for retry or dead-letter handling. |
| Logstash | Parse, enrich, and route logs or event data into OpenSearch indexes. | Logstash → Martini → OpenSearch | A Martini workflow accepts transformed Logstash output, applies index-routing and document-ID rules, and invokes the OpenSearch Bulk API. The workflow records rejected items separately from transport failures. |
| Apache Kafka | Buffer and distribute high-volume events before they are indexed into OpenSearch. | Apache Kafka → Martini → OpenSearch | Martini consumes or receives event batches, maps event payloads into newline-delimited OpenSearch bulk operations, submits bounded batches, and tracks successful offsets or checkpoints after item-level response validation. |
| Kubernetes | Make container logs, metrics, and operational events available for search and troubleshooting. | Kubernetes → Martini → OpenSearch | Martini can receive operational events from agents or intermediary services, normalize Kubernetes metadata, and index documents into versioned indexes. Alert notifications can be routed to downstream operations tools. |
| Prometheus | Combine metrics-oriented monitoring with OpenSearch logs, traces, or alert context. | Prometheus → Martini → OpenSearch | Martini can coordinate selected monitoring events or alert context, enrich them with OpenSearch search results, and publish a normalized operational payload to the required target. Exact direction depends on the monitoring design. |
| Grafana | Visualize OpenSearch data alongside metrics and other observability sources. | OpenSearch → Martini → Grafana | Martini can expose a controlled search façade or prepare normalized data for Grafana-oriented queries, while OpenSearch remains the search and aggregation source. Access is constrained through API authentication and query validation. |
| Amazon CloudWatch | Consolidate AWS logs or operational data into OpenSearch for search and analysis. | Amazon CloudWatch → Martini → OpenSearch | Martini can receive or retrieve selected CloudWatch-related payloads through the surrounding AWS integration design, transform them into OpenSearch documents, and submit bounded bulk requests. AWS permissions and network access must be validated. |
| ServiceNow | Create or update incidents from selected OpenSearch alerts and operational conditions. | OpenSearch → Martini → ServiceNow | OpenSearch Alerting or Notifications sends a selected alert to a Martini API, which validates and enriches the notification, applies incident-routing rules, and calls ServiceNow APIs with idempotent correlation data. |
How to build a OpenSearch integration in Martini
Objective
Establish the OpenSearch endpoint and deployment-specific authentication model before building workflow logic.
Instructions in Martini
- Configure the OpenSearch base URL and TLS settings in environment-specific configuration.
- Select Basic Authentication, JWT, OpenID Connect, SAML-backed access, or AWS SigV4 according to the deployment.
- Store credentials, certificates, and signing material in protected secrets rather than workflow payloads.
- Apply least-privilege roles for the indexes and operations required.
Objective
Select the execution model that matches the data flow and avoids treating OpenSearch alerts as universal change events.
Instructions in Martini
- Use an API or webhook endpoint for inbound application or alert notifications.
- Use a scheduler for polling or source-system incremental synchronization.
- Use a messaging trigger when events arrive through an external event system.
- Define a cursor, timestamp, version, or source event contract for incremental processing.
Objective
Obtain source data or OpenSearch results with bounded requests and a recoverable position.
Instructions in Martini
- Call the required REST, Search, SQL, or PPL endpoint.
- Use search_after, point-in-time, scroll, or bounded pagination for large result sets.
- Persist the last successful cursor or source position where multiple executions are required.
- Avoid unbounded searches and oversized bulk payloads.
Objective
Coordinate calls, branching, enrichment, and response handling in a maintainable Martini workflow.
Instructions in Martini
- Separate transport failures from authentication, authorization, mapping, and item-level failures.
- Branch on OpenSearch response status and operation type.
- Add correlation identifiers and retain the request context needed for support.
- Use reusable services or workflow sections for common OpenSearch request behavior.
Objective
Convert source payloads into OpenSearch documents, queries, bulk operations, or downstream response contracts.
Instructions in Martini
- Map source fields to the index mapping and target document structure.
- Generate newline-delimited JSON for Bulk API operations when processing batches.
- Normalize timestamps, identifiers, severity, and nested fields consistently.
- Validate required fields and reject incompatible type changes before indexing.
Objective
Enforce index routing, access, deduplication, and query policies before writing or returning data.
Instructions in Martini
- Use deterministic document IDs for idempotent indexing and updates.
- Restrict façade searches to approved indexes, fields, filters, and result sizes.
- Route alert notifications according to severity, ownership, and correlation rules.
- Use versioned indexes and aliases when mapping changes require controlled rollout.
Common OpenSearch data objects used in integrations
| Object | Typical Use | Common target systems | Martini handling |
|---|---|---|---|
| Indexes | Logical collections used to store and search related JSON documents. | Logging platforms, observability tools, data services, and reporting applications | Martini can create, address, alias, and manage indexes through Index APIs, including versioned-index and rollover-style workflows. |
| Documents | JSON objects representing logs, events, operational data, or application search content. | OpenSearch indexes, APIs, dashboards, and downstream alerting workflows | Martini validates, maps, enriches, and indexes, updates, retrieves, or deletes documents, using deterministic IDs for idempotency. |
| Mappings | Field definitions and data types that control indexing and querying behavior. | OpenSearch index templates, versioned indexes, and reindexing workflows | Martini can apply controlled mapping configurations and route mapping failures for review rather than silently retrying permanent errors. |
| Aliases | Stable alternate names for one or more indexes, commonly used during reindexing or rollover. | Search façades, applications, and index lifecycle workflows | Martini can switch aliases as part of a controlled deployment or reindexing workflow after validating the replacement index. |
| Search queries and aggregations | JSON request structures for filtering, sorting, retrieving, and summarizing documents. | Search APIs, reporting services, Grafana, and application façades | Martini can validate query inputs, apply business restrictions, call Search APIs, and transform results into stable application responses. |
| Alerts and monitors | Configured conditions and notifications for operational or analytical events when relevant plugins are enabled. | Martini APIs, ServiceNow, Slack, Microsoft Teams, email, and incident workflows | Martini can receive selected notifications, enrich them, apply routing rules, and deliver them to downstream systems while preserving correlation data. |
Authentication and security considerations
Deployment-specific authentication
OpenSearch authentication depends on the deployment and enabled security configuration. Common patterns include Basic Authentication over TLS, JWT, OpenID Connect, SAML, and role-based permissions. Amazon OpenSearch Service may require AWS SigV4-signed requests.
Least privilege and secrets
Use roles limited to the required cluster actions, index patterns, document permissions, and fields. Store credentials, certificates, tokens, and signing material in protected Martini environment configuration or secrets rather than workflow payloads.
Network protection
Use HTTPS and validate endpoint, VPC, proxy, security-group, and endpoint-policy requirements. A façade can be used when consumers should not receive direct OpenSearch credentials or unrestricted query access.
Operational considerations for OpenSearch integrations
Throughput and rate limits
OpenSearch has no single universal rate limit across all deployments. Capacity can be constrained by managed-service quotas, proxies, thread pools, indexing pressure, JVM resources, shard design, and ingestion services. Keep bulk requests bounded and monitor partial failures.
Pagination and checkpoints
Use search_after, point-in-time searches, scroll, or other documented patterns for large result sets. Persist cursors or last successful positions when a synchronization spans multiple workflow executions.
Idempotency and bulk results
Use deterministic document IDs and inspect every item in a bulk response. An HTTP success response does not prove that every operation succeeded. Retry transient failures and route permanent mapping or validation failures for review.
Mappings and compatibility
Mapping changes can reject documents when field types change. Use explicit mappings, templates, versioned indexes, aliases, controlled reindexing, and deployment testing. Confirm API and plugin behavior for the exact OpenSearch version and distribution.
Why use Martini instead of scripts or point-to-point integrations?
Orchestrate more than HTTP calls
Martini combines OpenSearch API consumption with triggers, schedules, mappings, transformations, business rules, API exposure, messaging, and external application calls in maintainable workflows.
Handle operational complexity
Instead of duplicating retry, checkpoint, pagination, validation, and error-routing logic across scripts, Martini can centralize these behaviors and distinguish transport failures from item-level indexing failures.
Protect and stabilize interfaces
Martini can expose a controlled search façade, enforce query and result limits, keep credentials outside workflow payloads, and present stable application-specific contracts even when OpenSearch mappings or plugins evolve.
Frequently asked questions
OpenSearch is primarily integrated through HTTP/JSON REST APIs for document, index, search, aggregation, cluster, SQL, PPL, and plugin operations. Bulk APIs support high-volume ingestion, while selected Alerting and Notifications configurations can send webhook-style notifications. External applications, event systems, and files can be transformed into OpenSearch documents before indexing.
Yes. Martini can consume OpenSearch REST APIs, submit document and Bulk API requests, run search or SQL/PPL queries, expose a controlled search API façade, and receive selected OpenSearch alert notifications where the deployment supports them.
No. A dedicated OpenSearch connector is not required. Martini can integrate using OpenSearch's documented REST APIs, Bulk API, authentication methods, and selected notification endpoints. No Martini-native OpenSearch connector is documented in the supplied materials.
Lonti does not charge an additional per-connector or per-vendor fee to integrate OpenSearch with Martini. Integrations are subject to the provisioned capacity of the Martini environment. Separate costs may apply from OpenSearch, Amazon Web Services, infrastructure providers, or other third-party systems based on subscription, usage, and deployment model.
Use the REST APIs as the primary method, with the Bulk API for grouped writes and scroll, point-in-time, or search_after for large retrievals. SQL and PPL are useful for analytics. Selected Alerting and Notifications callbacks can support operational routing, but they should not be treated as universal document change capture.
OpenSearch Alerting and Notifications can send selected monitor or alert notifications to configured destinations, depending on installed plugins and configuration. This is limited support and does not provide a webhook for every index or document mutation. Document-level synchronization generally requires source events, an event stream, ingestion processing, or polling.
Martini can run scheduled or event-driven workflows, retrieve source changes using timestamps or cursors, map fields into OpenSearch document structures, and use deterministic IDs for idempotent upserts. It can generate bulk requests, validate mappings and data types, and persist checkpoints for recoverable incremental processing.
Martini can distinguish authentication, authorization, connection, timeout, throttling, cluster, mapping, malformed JSON, and bulk-item failures. Bulk responses should be inspected item by item; transient failures can be retried selectively, while permanent failures can be routed to an exception or dead-letter workflow. Deterministic document IDs reduce duplicates.
Related Martini documentation
Workflows
Operations
Integrate OpenSearch with Martini
Use Martini to build secure OpenSearch integrations that combine REST APIs, bulk ingestion, search façades, scheduled synchronization, alert routing, and maintainable workflow orchestration.