.png)
Amazon S3 Integration Guide
Connect Amazon S3 object storage with enterprise applications through signed REST APIs, event notifications, file workflows, and controlled data exchange.
Amazon S3 integration options at a glance
Amazon S3 provides a documented HTTP REST API for buckets, objects, metadata, versions, multipart uploads, policies, and configuration. It supports file-oriented uploads and downloads for JSON, XML, CSV, binary, and compressed content, with multipart operations for large objects. S3 can emit selected object and bucket events through Amazon SQS, Amazon SNS, AWS Lambda, or Amazon EventBridge rather than unrestricted webhooks. Authentication commonly uses AWS Signature Version 4 with IAM users, roles, or temporary credentials; presigned URLs support time-limited delegated access. Martini can orchestrate these APIs and event flows, transform files, apply validation and business rules, and expose reusable APIs for downstream systems.
| Integration point | Supported by Amazon S3? | Common use cases | How Martini supports it |
|---|---|---|---|
| REST APIs | Yes | Create and list buckets, list and retrieve objects, upload, copy, delete, manage metadata, versions, multipart uploads, policies, lifecycle rules, and encryption configuration. | Martini can consume the S3 HTTP API from workflows, construct signed requests where supported, map XML responses and headers, and expose APIs that orchestrate S3 operations. |
| File / attachment APIs | Yes | Upload, download, copy, delete, tag, and manage file-like objects containing JSON, XML, CSV, binary, image, document, or compressed content. | Martini can receive or generate files, validate content, transform formats, and coordinate S3 object operations as part of an end-to-end workflow. |
| Webhooks / outbound callbacks | Limited | S3 supports notifications for selected object and bucket events through Amazon SQS, Amazon SNS, AWS Lambda, and Amazon EventBridge; it does not provide unrestricted callbacks for every operation. | Martini can consume or receive event information through a supported AWS intermediary, then retrieve the object and execute idempotent processing logic. |
| Bulk / async / batch APIs | Yes | S3 Batch Operations applies copy, tagging, metadata, restore, Lambda invocation, retention, or legal-hold actions across objects listed in a manifest. | Martini can generate or update manifests, invoke or coordinate batch operations through APIs, poll status, and route completion results. |
| Authentication | Yes | Authenticated requests normally use AWS Signature Version 4 with IAM users, roles, or temporary credentials. Presigned URLs provide time-limited delegated access. | Martini can keep credentials and signing configuration in protected environment settings and orchestrate presigned upload or download flows without distributing long-lived credentials. |
| SOAP APIs | Legacy | Amazon S3 has a legacy SOAP interface, but it is not recommended for new integrations. | Martini can consume SOAP services, but new S3 integrations should use the documented REST API instead. |
| Database / analytics access | Legacy | S3 is object storage rather than a general-purpose database. S3 Select is legacy and unavailable to new customers; Athena and Glue are preferred for related analytics use cases. | Martini can coordinate S3 with database or analytics services through their supported APIs, but should not treat S3 itself as a relational database. |
| SDKs and command-line tools | Yes | AWS SDKs and the AWS CLI can handle request signing, retries, multipart transfers, pagination, and streaming. | Where the deployment model permits, Martini can use an approved JVM-compatible AWS library or call the HTTP API directly with Signature Version 4 handling. |
How Amazon S3 exposes data and business events
Amazon S3 REST APIs
Amazon S3 exposes an HTTP REST API for bucket and object operations, including listing, reading, writing, copying, deleting, metadata, versions, multipart uploads, policies, lifecycle rules, and encryption configuration. Many control-plane responses use XML, while object content can be text or binary.
Martini implementation pattern
Martini implementation pattern: a workflow authenticates with AWS Signature Version 4 or uses an approved AWS library, calls the required S3 operation, parses response headers or XML, transforms object content, and writes results to the target system. Protected environment configuration holds credentials, regions, bucket names, and other deployment-specific values.
Implementation sequence
S3 Event Notifications
S3 supports notifications for selected events such as object creation, deletion, restore, replication, and lifecycle activity. Destinations include Amazon SQS, Amazon SNS, AWS Lambda, and Amazon EventBridge; S3 does not provide unrestricted outbound webhooks for every API action.
Martini implementation pattern
Martini implementation pattern: receive the event through a supported AWS intermediary or API exposure, extract the bucket, object key, version, and event identifier, then retrieve the current object and process it idempotently. The workflow treats notifications as potentially duplicated, delayed, or unordered.
Implementation sequence
S3 File and Multipart Operations
S3 supports single-request uploads and downloads, object copying, metadata and tagging, byte-range access, and multipart uploads for large objects. An attachment is represented as an object with a key, content, metadata, and optional tags rather than through a separate attachment resource.
Martini implementation pattern
Martini implementation pattern: expose or consume an API for file exchange, validate content type and naming, stream or partition large payloads where appropriate, and orchestrate completion or abort operations. The workflow can generate a presigned URL when an external party needs controlled temporary access.
Implementation sequence
S3 Batch Operations
Amazon S3 Batch Operations performs asynchronous actions across object populations described by a manifest. Supported actions include copying objects, replacing tags or metadata, restoring objects, invoking Lambda, and applying retention or legal holds.
Martini implementation pattern
Martini implementation pattern: generate or update a manifest from a source system or S3 listing, invoke the relevant AWS operation, and poll or consume job status. Martini can apply business rules before submission and write batch results to an audit store or downstream application.
Implementation sequence
Common Amazon S3 integration patterns
Pattern 1: Process documents from S3 events
When to use this pattern
Use this pattern when applications place documents in S3 and downstream processing should begin shortly after an object is created. Because S3 notifications cover selected events and may be duplicated or unordered, the workflow should verify the object and use durable idempotency controls.
Integration direction
Example Mapping
| Amazon S3 Field | Canonical Field | Target Field |
|---|---|---|
| bucket | sourceBucket | repositoryBucket |
| object key | documentPath | fileName |
| content type | mimeType | documentType |
| version ID | sourceVersion | externalVersion |
Martini implementation pattern
Martini receives an SQS-delivered S3 event, checks the bucket/key/version processing key, retrieves the object, validates content and metadata, transforms the document or extracted payload, and calls the document API. Transient AWS or target errors are retried without creating duplicate downstream documents.
Martini capabilities used
- workflows
- API consumption
- event processing
- data mapping
- business rules
- error handling
Pattern 2: Synchronize S3 files to an enterprise application
When to use this pattern
Use this pattern when a business application receives periodic files from an S3 prefix and no suitable event route exists. A scheduled workflow lists objects using continuation tokens and uses metadata, keys, timestamps, versions, or a processing ledger to identify work.
Integration direction
Example Mapping
| Amazon S3 Field | Canonical Field | Target Field |
|---|---|---|
| object key | sourceFileName | importFileName |
| last modified | sourceModifiedAt | sourceTimestamp |
| eTag or checksum | contentFingerprint | externalFileHash |
| object body | sourcePayload | importPayload |
Martini implementation pattern
A Martini scheduler lists a bounded prefix, persists continuation state, downloads eligible files, parses JSON, XML, CSV, or other supported content, maps fields to the application model, and records completion in a durable ledger. Validation failures are quarantined and transient failures use bounded retries.
Martini capabilities used
- scheduled workflows
- API consumption
- pagination handling
- file processing
- data mapping
- validation
- retry handling
Pattern 3: Export application data to S3
When to use this pattern
Use this pattern when an application or database must publish controlled files for analytics, archival, exchange, or downstream AWS processing. The workflow can create stable object keys and apply content, retention, and encryption rules before upload.
Integration direction
Example Mapping
| Amazon S3 Field | Canonical Field | Target Field |
|---|---|---|
| application customerId | customerIdentifier | customers[].id |
| application updatedAt | changedAt | exportMetadata.updatedAt |
| application status | businessStatus | customers[].status |
| generated file | exportPayload | object body |
Martini implementation pattern
Martini retrieves source data, applies selection and business rules, transforms it to CSV, JSON, or XML, and uploads it to a controlled bucket and prefix using a signed request or approved AWS library. The workflow records the object key and result, and can expose completion information through a Martini API.
Martini capabilities used
- workflows
- API consumption
- data mapping
- JSON handling
- XML handling
- file processing
- secure configuration
Pattern 4: Controlled partner file exchange with presigned URLs
When to use this pattern
Use this pattern when an external partner needs to upload or download a specific object without receiving long-lived AWS credentials. Presigned URLs provide time-limited access, while Martini manages validation, status, and downstream routing.
Integration direction
Example Mapping
| Amazon S3 Field | Canonical Field | Target Field |
|---|---|---|
| object key | exchangeFileId | partnerFileReference |
| presigned URL expiry | accessExpiresAt | downloadExpiry |
| content type | mimeType | expectedContentType |
| object metadata | exchangeMetadata | partnerStatus |
Martini implementation pattern
Martini creates or obtains a presigned URL, returns controlled exchange metadata to the partner, and later receives an event or runs a scheduled check. It validates the completed object, routes acceptable content to internal systems, and prevents replay through an exchange identifier and processing ledger.
Martini capabilities used
- API exposure
- workflows
- secure configuration
- file validation
- business rules
- idempotency
- error handling
Applications commonly integrated with Amazon S3
Amazon S3 commonly participates in AWS event, processing, analytics, content-delivery, and external data-exchange architectures. Martini can coordinate S3 with named AWS services and enterprise applications using REST APIs, event services, scheduled workflows, mappings, and durable processing controls.
| Application | Scenario | Direction | Martini Pattern |
|---|---|---|---|
| AWS Lambda | Process newly created objects, validate files, generate derivatives, or enrich object metadata. | Amazon S3 → AWS Lambda | Use an S3 event notification routed to AWS Lambda, or let Martini orchestrate the S3 object retrieval and downstream processing after receiving an event through a supported intermediary. |
| Amazon EventBridge | Route selected S3 events to rules, workflows, API destinations, or other AWS services. | Amazon S3 → Amazon EventBridge → Martini | Route supported S3 events through EventBridge, expose or consume a Martini API as appropriate, then retrieve the current object, validate it, and invoke downstream workflows. |
| Amazon SQS | Buffer object-created notifications and decouple file arrival from processing workloads. | Amazon S3 → Amazon SQS → Martini | Consume or receive SQS-delivered S3 notifications, use the bucket and object key to retrieve the object, and persist an idempotency key before processing. |
| Amazon SNS | Fan out selected S3 notifications to multiple subscribers or processing paths. | Amazon S3 → Amazon SNS → Martini | Process SNS-delivered event information in a Martini workflow, apply routing rules, and send each object to the appropriate validation or transformation path. |
| Amazon Athena | Query structured data stored in S3 using SQL for reporting, validation, or analytical workflows. | Amazon S3 → Amazon Athena → Martini | Use S3 as the file source and coordinate Athena-oriented processing or result handling through Martini, while keeping analytical querying in the AWS service designed for that purpose. |
| AWS Glue | Discover schemas, catalog S3 objects, and run data preparation or ETL processes over stored files. | Amazon S3 → AWS Glue → Martini | Use Martini to coordinate file arrival, validation, metadata routing, and downstream calls around Glue processing, with status and audit information stored durably. |
| Amazon CloudFront | Distribute content stored in S3 through a content delivery network and controlled origin access. | Amazon S3 → Amazon CloudFront | Use Martini to publish or manage eligible S3 objects and associated metadata, while CloudFront handles content distribution and delivery controls. |
| Snowflake | Load files from S3 into tables or unload data from Snowflake to S3 for exchange and analytics. | Snowflake → Martini → Amazon S3 | Schedule or trigger Martini workflows to coordinate file creation, naming, validation, status tracking, and downstream Snowflake ingestion or export processing. |
How to build a Amazon S3 integration in Martini
Objective
Establish the S3 access model and environment configuration before building the workflow.
Instructions in Martini
- Choose the required bucket, region, prefixes, and S3 operations.
- Use IAM roles or temporary credentials where possible.
- Configure AWS Signature Version 4 or an approved AWS library for authenticated requests.
- Store credentials, signing parameters, bucket names, and KMS-related settings as protected environment configuration.
- Confirm network access to the regional S3 endpoint.
Objective
Select an event-driven, scheduled, or API-led entry point based on how files arrive and how quickly they must be processed.
Instructions in Martini
- Use an S3 notification through SQS, SNS, Lambda, or EventBridge for selected object events.
- Use a scheduler when the integration must discover objects by listing a prefix.
- Expose a Martini API when an application or partner initiates the exchange.
- Define the event, schedule, or request scope and expected delivery behavior.
Objective
Retrieve the relevant object or event context while accounting for pagination, versions, large files, and delayed notifications.
Instructions in Martini
- Extract and validate the bucket, object key, version ID, and event identifier.
- Handle continuation tokens when listing objects or versions.
- Use range requests, streaming, or multipart operations for large content.
- Verify that the referenced object is readable before downstream processing.
- Preserve the source key, version, metadata, and processing timestamp.
Objective
Coordinate S3 operations, transformations, downstream calls, and durable status tracking in a maintainable Martini workflow.
Instructions in Martini
- Separate event intake, object retrieval, transformation, target delivery, and completion recording into clear workflow stages.
- Apply routing rules based on bucket, prefix, object metadata, tags, content type, or business date.
- Use reusable services or APIs for common validation and status operations.
- Keep the workflow safe for duplicate and out-of-order notifications.
Objective
Convert object content and metadata into the canonical model expected by downstream systems.
Instructions in Martini
- Parse JSON, XML, CSV, binary, or compressed content according to the agreed file contract.
- Map S3 keys, metadata, tags, version IDs, and content fields to canonical fields.
- Normalize dates, encodings, delimiters, identifiers, and content types.
- Validate required fields and route invalid files to a controlled exception path.
Objective
Enforce processing eligibility, retention, naming, routing, and duplicate-prevention rules before writing results.
Instructions in Martini
- Use a durable processing key based on bucket, key, version, event ID, or checksum.
- Reject unsupported content types, expired exchange requests, and invalid naming conventions.
- Distinguish missing objects, delete markers, inaccessible objects, and validation failures.
- Apply tenant, document type, retention, and downstream routing rules.
Common Amazon S3 data objects used in integrations
| Object | Typical Use | Common target systems | Martini handling |
|---|---|---|---|
| Bucket | Top-level container for objects, notification configuration, policies, lifecycle rules, encryption settings, and access controls. | AWS services, file-processing applications, data platforms, enterprise APIs | Martini stores bucket and region configuration securely, invokes bucket APIs when required, and applies environment-specific routing and authorization rules. |
| Object | A stored file or payload with content, metadata, content type, tags, and an object key. | AWS Lambda, Amazon Athena, AWS Glue, Snowflake, ERP and CRM APIs | Martini retrieves or creates objects, validates content, transforms JSON, XML, CSV, or binary data, and records processing status. |
| Object key | The complete identifier used to address an object within a bucket; prefixes provide folder-like organization. | File-processing workflows, data lake conventions, downstream import jobs | Martini parses keys for tenant, source, date, document type, or business identifiers and uses validated naming rules for routing. |
| Object version | Historical version of an object when bucket versioning is enabled, allowing integrations to distinguish revisions. | Audit stores, archival workflows, downstream document systems | Martini uses version IDs where available to establish durable processing and avoid applying an event to the wrong object revision. |
| Multipart upload | Upload of a large object in separate parts that are completed or aborted as a unit. | Large-file exchange, data platforms, document-processing systems | Martini orchestrates initiation, part transfer, completion, retry, and cleanup while avoiding unnecessary in-memory buffering. |
| Delete marker | Versioned-bucket marker created when an object is deleted without permanently removing all historical versions. | Retention workflows, audit systems, synchronization processes | Martini distinguishes delete markers from missing or inaccessible objects and applies explicit deletion and retention rules. |
Authentication and security considerations
Authentication and authorization
Amazon S3 authenticated requests normally use AWS Signature Version 4 with IAM users, IAM roles, or temporary security credentials. Martini should keep credentials, signing configuration, regions, bucket names, prefixes, and KMS-related settings in protected environment configuration.
- Prefer IAM roles or temporary credentials over long-lived access keys where the deployment model permits.
- Restrict permissions to required buckets, prefixes, actions, and encryption keys.
- Use presigned URLs for controlled, time-limited delegated uploads or downloads.
- Protect presigned URLs as credentials while they remain valid.
- Do not log AWS credentials, authorization headers, object contents, or presigned URLs.
- Confirm bucket policies, Block Public Access, access points, VPC endpoints, TLS, and network controls for the deployment.
Operational considerations for Amazon S3 integrations
Reliability and processing controls
S3 listings, version listings, multipart uploads, and batch results may require pagination. S3 notifications can be duplicated, delayed, or unordered, so workflows should use durable idempotency keys based on the bucket, object key, version ID, event ID, or checksum.
- Handle continuation tokens and persist progress for large collections.
- Use exponential backoff for throttling, 503 Slow Down responses, connection failures, and transient service errors.
- Use streaming, range requests, or multipart transfers for large objects.
- Validate content type, encoding, compression, schema version, key conventions, and required metadata before processing.
- Distinguish missing objects, delete markers, inaccessible objects, and validation failures.
- Test versioned buckets, lifecycle behavior, KMS permissions, duplicate events, retries, and abandoned multipart uploads.
- Monitor workflow outcomes, object identifiers, processing latency, retries, and quarantined files without exposing sensitive content.
Why use Martini instead of scripts or point-to-point integrations?
Orchestration instead of isolated scripts
Martini provides a maintainable workflow layer around Amazon S3 APIs and event services. Rather than embedding request signing, file parsing, routing, retries, and target-specific mappings in separate scripts, teams can centralize the integration contract and operational behavior.
- Coordinate REST calls, event intake, scheduled discovery, file processing, and downstream APIs in one workflow.
- Reuse mappings, validation logic, authentication configuration, and status handling across integrations.
- Apply business rules for prefixes, metadata, tenants, schemas, retention, and duplicate prevention.
- Expose a controlled Martini API so applications do not need to implement S3 signing or orchestration themselves.
- Separate environment configuration from integration logic for safer deployment and maintenance.
- Use workflow error handling, retries, logging, and monitoring to support operational ownership at enterprise scale.
Frequently asked questions
Amazon S3 can be integrated through its HTTP REST API for bucket and object operations, file uploads and downloads, multipart transfers, metadata, versions, and batch processing. Selected S3 events can be routed through Amazon SQS, SNS, Lambda, or EventBridge. Authentication commonly uses AWS Signature Version 4 with IAM credentials, roles, or temporary credentials, while presigned URLs support controlled delegated access.
Yes. Martini can integrate with Amazon S3 by consuming the S3 REST API, orchestrating signed object and bucket operations, processing S3 notifications delivered through supported AWS services, and handling presigned URL exchanges. A native Martini Amazon S3 connector is not confirmed in the supplied documentation.
No. A dedicated Amazon S3 connector is not required. Martini can use Amazon S3’s documented REST API, AWS Signature Version 4, presigned URLs, object operations, and event notifications delivered through confirmed AWS messaging or event services.
Lonti does not charge an additional per-connector or per-vendor fee to integrate Amazon S3. The integration is subject to the provisioned capacity of the Martini environment. Separate costs may apply from AWS, cloud infrastructure, storage, data transfer, API usage, or other third-party services.
New implementations should generally use the S3 REST API, file and object operations, IAM-based access with Signature Version 4, and selected event notifications through SQS, SNS, Lambda, or EventBridge. SOAP is a legacy option and is not recommended for new S3 integrations. S3 should not be treated as a relational database; Athena or Glue may be more appropriate for analytics use cases.
S3 supports notifications for selected object and bucket events, including creation, deletion, restoration, replication, and lifecycle-related events. Destinations include Amazon SQS, SNS, Lambda, and EventBridge. These are not unrestricted webhooks for every S3 operation, and notification delivery may be duplicated, delayed, or unordered.
Martini can use S3 notifications for selected event-driven flows or scheduled workflows that list objects under a prefix. Reliable synchronization should handle continuation tokens, object versions, metadata, duplicate events, delayed delivery, stable naming conventions, and a durable processing ledger. Large files should use streaming, range requests, or multipart operations where appropriate.
Martini can map S3 object content and metadata into canonical and downstream models, validate JSON, XML, CSV, or other supported content, apply routing rules, and call target APIs. Workflows can use bounded retries and backoff for transient AWS errors, quarantine validation failures, and use idempotency keys to prevent duplicate downstream results. Martini can also expose an API façade that hides S3 request signing and orchestration from consuming applications.
Related Martini documentation
Data
Connect Amazon S3 to your enterprise workflows
Use Martini to orchestrate Amazon S3 APIs, event notifications, file processing, and downstream systems with secure configuration, reusable mappings, and operational controls.