.png)
Google Cloud Storage Integration Guide
Connect enterprise applications to Google Cloud Storage through REST APIs, secure object transfers, scheduled workflows, and selected event-notification paths.
Google Cloud Storage integration options at a glance
Google Cloud Storage provides a JSON REST API for managing projects, buckets, objects, metadata, uploads, downloads, copies, rewrites, compositions, lifecycle settings, and access policies. It also supports resumable transfers, batch requests, signed URLs, and selected object-change notifications through Pub/Sub and related Google Cloud event services. Authentication can use OAuth 2.0, service accounts, application default credentials, workload identity approaches, HMAC keys, or signed URLs. Martini can consume these APIs, orchestrate scheduled or event-driven workflows, transform file contents, validate metadata, and exchange objects with enterprise applications while applying retries and generation-aware idempotency.
| Integration point | Supported by Google Cloud Storage? | Common use cases | How Martini supports it |
|---|---|---|---|
| REST APIs | Yes | The Google Cloud Storage JSON API supports bucket and object administration, uploads, downloads, metadata, IAM, lifecycle rules, copying, rewriting, composing, and resumable uploads. | Martini can consume REST endpoints, manage authentication, map request and response data, and orchestrate multi-step object workflows. |
| Webhooks / outbound callbacks | Limited | Cloud Storage publishes selected object-change notifications through Pub/Sub and related Google Cloud event services rather than providing a general direct webhook for every operation. | Martini can receive or participate in the configured event delivery path through an API or messaging integration, then retrieve the current object. |
| Bulk / async / batch APIs | Limited | JSON API batch requests, resumable uploads, compose, rewrite, and Storage Transfer Service support high-volume or long-running transfers, but batch operations are not generally transactional. | Martini can coordinate batches, track per-object results, retry failed operations, and distinguish partial completion from full success. |
| File / attachment APIs | Yes | Object uploads, downloads, range requests, metadata retrieval, copies, rewrites, compositions, and signed URLs support enterprise file exchange. | Martini can download, validate, transform, and upload files while preserving content types, checksums, metadata, and object identifiers. |
| Authentication | Yes | Google Cloud Storage supports OAuth 2.0, service accounts, application default credentials, workload identity approaches, HMAC keys for the XML API, IAM permissions, and signed URLs. | Martini can use secure environment configuration and secrets for tokens or credentials and apply least-privilege access patterns. |
| XML API | Limited | Google Cloud Storage provides an XML API with an S3-compatible request style for selected use cases, while the JSON API is generally preferred for new integrations. | Martini can consume standards-based HTTP endpoints when the XML API is required, with authentication and request handling configured for the selected use case. |
| Database / analytics access | No | Cloud Storage is object storage and does not provide a general-purpose SQL interface for bucket contents; BigQuery and other services expose separate APIs. | Martini can orchestrate Cloud Storage with a separate analytics service, but should use that service's API for query execution. |
How Google Cloud Storage exposes data and business events
Google Cloud Storage REST APIs
The Google Cloud Storage JSON API is the primary integration mechanism for bucket and object administration and data operations. It supports listing, upload, download, metadata, copy, rewrite, compose, lifecycle, IAM, and resumable-transfer operations.
Martini implementation pattern
Martini implementation pattern: Martini workflows consume the JSON API using Google Cloud authentication, apply request and response mappings, follow pagination tokens, and store object names, generations, and operation outcomes for reliable processing.
Implementation sequence
Object-change notifications
Cloud Storage supports selected object-change notifications through Pub/Sub and related Google Cloud event services. These notifications are not a universal direct webhook for every bucket operation and may be retried or delivered out of order.
Martini implementation pattern
Martini implementation pattern: A configured Google Cloud event path reaches a Martini API or supported messaging boundary. Martini validates the event, uses the bucket, object name, generation, and event identifier to prevent duplicate processing, then retrieves current object state.
Implementation sequence
Resumable and batch transfers
Cloud Storage supports resumable uploads, JSON API batch requests, object composition, rewrite operations, and large-scale transfer services. These mechanisms support throughput and long-running transfers but do not provide a general transactional bulk interface.
Martini implementation pattern
Martini implementation pattern: Martini divides large or multi-object work into independently trackable operations, preserves transfer state, validates checksums where appropriate, and records per-object results so partial failures can be retried safely.
Implementation sequence
File and object exchange
File handling is a primary Cloud Storage capability covering simple, multipart, and resumable uploads, downloads, range requests, copies, rewrites, compositions, metadata, and signed URLs.
Martini implementation pattern
Martini implementation pattern: A workflow receives or retrieves a file, validates its content and metadata, transforms it when required, and writes it to Cloud Storage or another enterprise application. Signed URLs can provide narrowly scoped, time-limited access.
Implementation sequence
Common Google Cloud Storage integration patterns
Pattern 1: Ingest documents from a Cloud Storage bucket
When to use this pattern
Use this pattern for invoices, product catalogs, customer imports, reports, or other files delivered to an inbound bucket. It combines scheduled discovery with validation, transformation, downstream delivery, and archive handling.
Integration direction
Example Mapping
| Google Cloud Storage Field | Canonical Field | Target Field |
|---|---|---|
| name | sourceFileName | documentName |
| contentType | mimeType | fileType |
| generation | sourceGeneration | sourceVersion |
| size | fileSize | contentLength |
Martini implementation pattern
A scheduled Martini workflow lists objects page by page, filters already processed generations, downloads and validates each file, maps its contents to the target application, and archives or marks the source after a successful write. Invalid files are routed to an error outcome, while transient API failures are retried.
Martini capabilities used
- scheduled workflows
- API consumption
- file handling
- data mapping
- validation
- business rules
- error handling
Pattern 2: Process object events asynchronously
When to use this pattern
Use this pattern when selected object creation, deletion, archival, or metadata events should initiate processing without repeatedly scanning a bucket. Event delivery must be treated as retryable and potentially out of order.
Integration direction
Example Mapping
| Google Cloud Storage Field | Canonical Field | Target Field |
|---|---|---|
| bucket | sourceBucket | storageLocation |
| name | objectName | documentPath |
| generation | objectGeneration | sourceVersion |
| eventType | changeType | processingAction |
Martini implementation pattern
The configured Google Cloud notification path reaches Martini, which validates the event and checks a stable idempotency key before retrieving the referenced generation. The workflow transforms and routes the object, records processing state, and safely ignores or retries duplicate deliveries.
Martini capabilities used
- API exposure
- event-driven workflows
- API consumption
- idempotency rules
- data transformation
- error handling
Pattern 3: Export application data to Cloud Storage
When to use this pattern
Use this pattern for scheduled exports, audit files, partner exchanges, or downstream analytics ingestion. The workflow can generate JSON, CSV, XML, or another required format before uploading it to a controlled bucket path.
Integration direction
Example Mapping
| Google Cloud Storage Field | Canonical Field | Target Field |
|---|---|---|
| applicationId | sourceIdentifier | metadata.sourceId |
| createdAt | exportTimestamp | metadata.exportedAt |
| customerNumber | customerReference | objectName |
| fileChecksum | contentChecksum | crc32cOrMd5 |
Martini implementation pattern
Martini retrieves source data, applies business filters and transformations, generates the required file, and uploads it using the JSON API. The workflow records the bucket, object name, generation, and checksum and retries incomplete transfers without creating duplicate export objects.
Martini capabilities used
- scheduled workflows
- API consumption
- JSON and file transformation
- mapping
- business rules
- retry handling
Pattern 4: Controlled exchange with signed URLs
When to use this pattern
Use this pattern when an application or external party needs temporary access to a specific object without receiving broad bucket permissions. It is suitable for controlled uploads, downloads, and partner file exchange.
Integration direction
Example Mapping
| Google Cloud Storage Field | Canonical Field | Target Field |
|---|---|---|
| objectName | requestedObject | signedUrlResource |
| expiration | accessExpiry | urlExpiry |
| contentType | expectedMimeType | uploadConstraint |
| generation | resultingVersion | workflowState |
Martini implementation pattern
Martini exposes an API that returns or coordinates a narrowly scoped, time-limited signed URL. A follow-up workflow validates the resulting object, checks naming and content constraints, and routes it for processing while recording the generation and access outcome.
Martini capabilities used
- API exposure
- secure configuration
- workflow orchestration
- file validation
- business rules
- monitoring
Applications commonly integrated with Google Cloud Storage
Google Cloud Storage commonly participates in data, machine learning, analytics, and event-processing architectures. Martini can orchestrate these exchanges through the relevant application APIs and Cloud Storage REST operations.
| Application | Scenario | Direction | Martini Pattern |
|---|---|---|---|
| BigQuery | Load files from Cloud Storage into analytical tables or export query results to storage. | Google Cloud Storage → Martini → BigQuery | Use a scheduled or event-driven workflow to identify a completed object, validate its schema and generation, then invoke the relevant BigQuery API and record the load result. |
| Dataflow | Process batch or streaming data and write transformed datasets back to Cloud Storage. | Google Cloud Storage → Martini → Dataflow | Use Martini to coordinate object arrival, invoke or notify the Dataflow process through its available API, and route completion or failure outcomes to downstream workflows. |
| Dataproc | Provide Spark or Hadoop workloads with files stored in Cloud Storage. | Google Cloud Storage → Martini → Dataproc | Orchestrate input-file validation, Dataproc job submission, status checks, and output-object handling in a reusable Martini workflow. |
| Vertex AI | Exchange training datasets, model artifacts, and batch prediction files. | Google Cloud Storage → Martini → Vertex AI | Validate object metadata and file format before invoking Vertex AI operations, then store or route generated artifacts with generation-aware tracking. |
| Cloud Functions | Start serverless processing when selected Cloud Storage object events occur. | Google Cloud Storage → Martini → Cloud Functions | Use Cloud Storage notifications or an event service for initiation, then have Martini enrich the event, retrieve the referenced object, and coordinate subsequent application calls. |
| Pub/Sub | Deliver selected Cloud Storage object-change notifications for asynchronous processing. | Google Cloud Storage → Pub/Sub → Martini | Receive the event through the configured Google Cloud delivery path, use the bucket, object name, generation, and event identifier for idempotency, and retrieve current object state before processing. |
| Snowflake | Exchange analytical files between Cloud Storage and Snowflake stages. | Google Cloud Storage → Martini → Snowflake | Use Martini to validate and format export files, upload or retrieve objects, and coordinate Snowflake stage or load operations through the applicable Snowflake API. |
| Databricks | Support lakehouse ingestion and export using objects stored in Cloud Storage. | Google Cloud Storage → Martini → Databricks | Coordinate object availability, schema validation, Databricks ingestion or job execution, and archival or retry handling in a workflow. |
How to build a Google Cloud Storage integration in Martini
Objective
Configure Google Cloud Storage access using an environment-specific least-privilege credential and keep tokens, service account configuration, or signing material outside workflow definitions.
Instructions in Martini
- Select OAuth 2.0, service account, workload identity, HMAC, or signed URL access as appropriate
- Store credentials and endpoint configuration in Martini secure configuration
- Separate read, write, and administrative permissions where practical
Objective
Select a scheduled, API-driven, or event-driven entry point based on whether the workflow scans objects, receives a request, or responds to selected Cloud Storage notifications.
Instructions in Martini
- Use a scheduler for polling and batch ingestion
- Expose a Martini API for controlled uploads, downloads, or event delivery
- Use the configured Pub/Sub or Google Cloud event path for selected object changes
Objective
Locate and retrieve the required bucket or object while handling pagination, resumable transfers, ranges, and generation identifiers.
Instructions in Martini
- Follow page tokens until listing is complete
- Retrieve the referenced generation when event processing requires version accuracy
- Use resumable transfers for large files and verify completion
Objective
Convert object metadata and file contents into the canonical model required by the target application, with explicit validation for format and integrity.
Instructions in Martini
- Validate content type, size, naming, schema, encoding, and checksum
- Map JSON, CSV, XML, or binary content to the target structure
- Preserve relevant metadata and object identifiers
Objective
Apply routing, archive, overwrite, retention, and duplicate-processing rules before writing to downstream systems or changing the source object.
Instructions in Martini
- Use bucket, object name, generation, and event identifiers for idempotency
- Use generation or metageneration checks for concurrency-sensitive writes
- Route invalid or disallowed objects to a controlled error path
Objective
Upload, copy, compose, archive, or deliver the transformed result and record the resulting object path, generation, and business outcome.
Instructions in Martini
- Write through the Cloud Storage REST API or the target application API
- Record per-object success and failure status
- Avoid assuming that batch operations are transactionally atomic
Common Google Cloud Storage data objects used in integrations
| Object | Typical Use | Common target systems | Martini handling |
|---|---|---|---|
| Projects | Resource containers that own or provide access to buckets and related Google Cloud services. | BigQuery, Dataflow, Dataproc, Vertex AI, enterprise administration systems | Martini can use project identifiers in endpoint configuration, routing rules, credentials, and environment-specific workflows. |
| Buckets | Globally named containers with location, storage class, lifecycle, encryption, IAM, retention, and versioning configuration. | Data platforms, file exchange applications, analytics services | Martini can list, inspect, create, or manage buckets through REST workflows when the service account has the required permissions. |
| Objects | Files stored in buckets, including content, names, sizes, content types, checksums, timestamps, and metadata. | BigQuery, Dataflow, Dataproc, Vertex AI, Snowflake, Databricks | Martini can list, download, validate, transform, upload, copy, rewrite, compose, archive, or delete objects. |
| Object generations | Immutable versions created when object versioning is enabled and used for concurrency and idempotency control. | Workflow state stores, audit systems, downstream applications | Martini can retain generation identifiers in workflow state and use conditional reads or writes to avoid processing stale versions. |
| Object metadata | System and user-defined properties such as content type, checksums, cache settings, timestamps, and retention-related fields. | Document systems, data pipelines, compliance and archive systems | Martini can map, validate, preserve, and update metadata during object transfers and transformations. |
| Notifications | Configurations that publish selected object creation, deletion, archival, or metadata-change events to supported destinations. | Pub/Sub, Cloud Functions, event-processing workflows | Martini can consume the delivered event path, retrieve current object state, and apply duplicate and ordering protection. |
Authentication and security considerations
Authentication and least-privilege access
Google Cloud Storage supports OAuth 2.0, service accounts, application default credentials, workload identity approaches, HMAC keys for selected XML API use cases, IAM permissions, and signed URLs. Choose the narrowest access model that meets the workflow requirement.
- Store credentials, tokens, and signing configuration in Martini secure environment configuration.
- Separate read, write, and administrative permissions where practical.
- Use signed URLs with short expiration periods and narrowly scoped object access.
- Review bucket IAM, retention, encryption, versioning, and lifecycle settings as part of the integration design.
Operational considerations for Google Cloud Storage integrations
Design for reliable object processing
Cloud Storage listings are paginated, event deliveries can be duplicated or reordered, and object generations affect concurrency and version handling. Workflows should retrieve current state when necessary rather than relying only on an event payload.
- Follow page tokens until all objects have been processed.
- Use generation and metageneration preconditions for concurrency-sensitive writes.
- Use resumable transfers, timeouts, and checksum validation for large files.
- Record per-object results because batch operations can partially fail.
- Account for lifecycle rules, retention policies, legal holds, regional placement, and transfer costs.
- Validate file schemas, encodings, content types, and metadata before downstream processing.
Why use Martini instead of scripts or point-to-point integrations?
Reusable orchestration instead of isolated scripts
Martini provides a maintainable workflow layer around Google Cloud Storage APIs and event paths. It separates authentication, retrieval, transformation, business rules, downstream writes, and operational handling into reusable integration assets.
- Coordinate scheduled, API-led, and event-driven processing in one platform.
- Map JSON, CSV, XML, metadata, and file content to different target systems.
- Apply generation-aware idempotency, validation, retries, and partial-failure handling consistently.
- Expose controlled APIs for uploads, downloads, signed URL flows, or partner exchanges.
- Centralize environment configuration, monitoring, troubleshooting, and deployment practices.
Frequently asked questions
Google Cloud Storage can be integrated through its JSON REST API for bucket and object operations, file uploads and downloads, metadata management, copying, rewriting, composition, resumable transfers, signed URLs, and selected object-change notifications delivered through Pub/Sub or related Google Cloud event services.
Yes. Martini can consume the Google Cloud Storage REST API, use Google-supported authentication methods, orchestrate file and object workflows, expose APIs for controlled exchange, and participate in event-driven designs when the configured Google Cloud delivery path reaches Martini.
No. A dedicated Google Cloud Storage connector is not required. Martini can integrate using Google Cloud Storage's native REST APIs, authentication methods, file operations, signed URLs, and selected notification paths.
Lonti does not charge an additional per-connector or per-vendor fee to integrate Google Cloud Storage with Martini. Integrations are subject to the provisioned capacity of the Martini environment; separate costs may apply from Google Cloud, infrastructure providers, or other third-party services.
The JSON REST API is the preferred method for new integrations. Use resumable uploads for large files, signed URLs for controlled temporary access, and batch or rewrite operations for suitable high-volume or long-running transfers. The XML API is available for selected compatibility scenarios.
Cloud Storage supports selected object-change notifications through Pub/Sub and related Google Cloud event services, including events such as creation, deletion, archival, and metadata changes depending on the mechanism. This is not a universal direct webhook facility, and deliveries may be duplicated or out of order.
Synchronization can be scheduled by listing objects and following pagination, event-driven through selected notifications, or initiated by an API request. Reliable workflows track object names, generations, checksums, and processing state so that updates, overwrites, retries, and versioned objects are handled safely.
Martini can map object metadata and file contents into canonical and target models, validate schemas and checksums, apply business rules, and route failures for retry or review. Workflows should use bucket, object name, generation, and event identifiers as idempotency keys and record per-object outcomes.
Related Martini documentation
Workflows
Data
Connect Google Cloud Storage with Martini
Use Martini to build secure, reliable Google Cloud Storage integrations for object exchange, event-driven processing, data exports, and enterprise workflow orchestration.