View a markdown version of this page

TwelveLabs Marengo Embed 3.0 - Amazon Bedrock
Migrate from TwelveLabs Marengo Embed 2.7 to TwelveLabs Marengo Embed 3.0TwelveLabs Marengo Embed 3.0 request parametersTwelveLabs Marengo Embed 3.0 responseTwelveLabs Marengo Embed 3.0 code examples

TwelveLabs Marengo Embed 3.0

The TwelveLabs Marengo Embed 3.0 model generates enhanced embeddings from video, text, audio, image, or multi-input (text with multiple images) inputs. This latest version offers improved performance and accuracy for similarity search, clustering, and other machine learning tasks.

  • Provider — TwelveLabs

  • Model ID — twelvelabs.marengo-embed-3-0-v1:0

Marengo Embed 3.0 delivers several key enhancements:

  • Extended video processing capacity – Process up to 4 hours of video and audio content. Files can be up to 6 GB, which is double the capacity of previous versions. This makes it ideal for analyzing full sporting events, extended training videos, and complete film productions.

  • Enhanced sports analysis – The model delivers significant improvements. It provides better understanding of gameplay dynamics, player movements, and event detection.

  • Global multilingual support – Expanded language capabilities from 12 to 36 languages. This enables global organizations to build unified search and retrieval systems that work seamlessly across diverse regions and markets.

  • Multimodal search precision – Combine images and descriptive text in a single embedding request. This merges visual similarity with semantic understanding to deliver more accurate and contextually relevant search results.

  • Reduced embedding dimension – Reduced from 1024 to 512, which can help reduce storage costs.

The TwelveLabs Marengo Embed 3.0 model supports the Amazon Bedrock Runtime operations in the following table.

API operation Supported model types Input modalities Output modalities

InvokeModel

US East (N. Virginia) – Base models and Inference profiles

Europe (Ireland) – Inference profiles

Asia Pacific (Seoul) - Base models

Text

Image

Multi-input (text with multiple images)

Note: Text and image interleaved is also supported.

Embedding
StartAsyncInvoke

US East (N. Virginia) – Base models

Europe (Ireland) – Base models

Asia Pacific (Seoul) - Base models

Video

Audio

Image

Text

Multi-input (text with multiple images)

Note: Text and image interleaved is also supported.

Embedding
Note

Use InvokeModel to generate embeddings for search query. Use StartAsyncInvoke to generate embeddings for assets at a large scale.

The following quotas apply to the input:

Input modality Maximum
Text 500 tokens
Image 5 MB per image
Video (S3) 6 GB, 4 hour length
Audio (S3) 6 GB, 4 hour length
Note

If you define audio or video inline by using base64-encoding, make sure that the request body payload doesn't exceed the Amazon Bedrock 25 MB model invocation quota.

Migrate from TwelveLabs Marengo Embed 2.7 to TwelveLabs Marengo Embed 3.0

Important

TwelveLabs Marengo Embed 2.7 will be deprecated. Embeddings that you created with TwelveLabs Marengo Embed 2.7 are NOT compatible with TwelveLabs Marengo Embed 3.0. You must regenerate all embeddings with TwelveLabs Marengo Embed 3.0.

To migrate from TwelveLabs Marengo Embed 2.7 to TwelveLabs Marengo Embed 3.0, complete the following steps.

  1. Update your code to use the new nested input structure and parameter changes described in the following section.

  2. Update the model ID to twelvelabs.marengo-embed-3-0-v1:0.

  3. Test your integration with the new version.

The following are the key changes between TwelveLabs Marengo Embed 2.7 and TwelveLabs Marengo Embed 3.0.

  • Nested input structure – Input parameters are now organized inside objects based on the inputType value. For example, if inputType is text, nest inputText within a text object.

  • Combined text and image inputs – A new text_image input type allows you to combine text and image in a single request. This input type is not supported in TwelveLabs Marengo Embed 2.7.

  • Parameter changes – The embeddingOption parameter values changed:

    • TwelveLabs Marengo Embed 2.7: visual-text, visual-image, audio

    • TwelveLabs Marengo Embed 3.0: visual, audio, transcription

The following examples show the structural changes for text input.

TwelveLabs Marengo Embed 2.7 (flat structure)

{
    "inputType": "text",
    "inputText": "your-text",
    "embeddingOption": "visual-text"
}

TwelveLabs Marengo Embed 3.0 (nested structure)

{
    "inputType": "text",
    "text": {
        "inputText": "your-text"
    },
    "embeddingOption": "visual"
}

The following examples show the structural changes for media inputs (image, video, or audio).

TwelveLabs Marengo Embed 2.7 (flat structure)

{
    "inputType": "image|video|audio",
    "mediaSource": {
        "s3Location": {
            "bucketName": "your-bucket",
            "objectKey": "your-key"
        }
    },
    "embeddingOption": "visual-image|audio"
}

TwelveLabs Marengo Embed 3.0 (nested structure)

{
    "inputType": "image|video|audio",
    "image|video|audio": {
        "mediaSource": {
            "s3Location": {
                "bucketName": "your-bucket",
                "objectKey": "your-key"
            }
        }
    },
    "embeddingOption": "visual|audio|transcription"
}

TwelveLabs Marengo Embed 3.0 request parameters

When you make a request, the field in which the model-specific input is specified depends on the API operation:

The format of the model input depends on the input modality:

Text
{
    "inputType": "text",
    "text": {
        "inputText": "string"
    }
}
Image
{
  "inputType": "image",
  "image": {
    "mediaSource": {
      "base64String": "base64-encoded string", // base64String OR s3Location, exactly one
      "s3Location": {
        "uri": "s3://amzn-s3-demo-bucket/folder/dog.jpg",
        "bucketOwner": "123456789012"
      }
    }
  }
}
Text & image
{
  "inputType": "text_image",
  "text_image": {
    "inputText": "man walking a dog",
    "mediaSource": {
      "base64String": "base64-encoded string", // base64String OR s3Location, exactly one
      "s3Location": {
        "uri": "s3://amzn-s3-demo-bucket/folder/dog.jpg",
        "bucketOwner": "123456789012"
      }
    }
  }
}
Audio
{
  "inputType": "audio",
  "audio": {
    "mediaSource": {
      "base64String": "base64-encoded string", // base64String OR s3Location, exactly one
      "s3Location": {
        "uri": "s3://amzn-s3-demo-bucket/audio/a.wav",
        "bucketOwner": "123456789012"
      }
    },
    "startSec": 0,
    "endSec": 6,
    "segmentation": {
      "method": "fixed", 
      "fixed": {
        "durationSec": 6
      }
    },
    "embeddingOption": [
      "audio",
      "transcription"
    ], // optional, default=both
    "embeddingType": [
      "separate_embedding"
    ], // optional, default=["separate_embedding"]
    "embeddingScope": [
      "clip",
      "asset"
    ] // optional, one or both
  }
}
Video
{
  "inputType": "video",
  "video": {
    "mediaSource": {
      "base64String": "base64-encoded string", // base64String OR s3Location, exactly one
      "s3Location": {
        "uri": "s3://amzn-s3-demo-bucket/video/clip.mp4",
        "bucketOwner": "123456789012"
      }
    },
    "startSec": 0,
    "endSec": 6,
    "segmentation": {
      "method": "dynamic", // dynamic OR fixed, exactly one
      "dynamic": {
        "minDurationSec": 4
      }
      "method": "fixed",
      "fixed": {
        "durationSec": 6
      }
    },
    "embeddingOption": [
      "visual",
      "audio", 
      "transcription"
    ], // optional, default=all
    "embeddingType": [
      "separate_embedding"
    ], // optional, default=["separate_embedding"]
    "embeddingScope": [
      "clip",
      "asset"
    ] // optional, one or both
  }
}
Multi-input
{
  "inputType": "multi_input",
  "multi_input": {
    "inputText": "<@img1> walking a dog with <@img2>", // optional
    "mediaSources": [
      {
        "name": "img1", // required if inputText uses <@name> placeholders
        "mediaType": "image",
        "base64String": "base64-encoded string", // base64String OR s3Location, exactly one
        "s3Location": {
          "uri": "s3://amzn-s3-demo-bucket/images/person.jpg",
          "bucketOwner": "123456789012"
        }
      },
      {
        "name": "img2",
        "mediaType": "image",
        "base64String": "base64-encoded string", // base64String OR s3Location, exactly one
        "s3Location": {
          "uri": "s3://amzn-s3-demo-bucket/images/dog.jpg",
          "bucketOwner": "123456789012"
        }
      }
    ]
  }
}

Expand the following sections for details about the input parameters:

Modality for the embedding.

  • Type: String

  • Required: Yes

  • Valid values: text | image | text_image | audio | video | multi_input

Text to be embedded.

  • Type: String

  • Required: Yes (for compatible input types)

  • Compatible input types: Text

Contains information about the media source.

  • Type: Object

  • Required: Yes (if compatible type)

  • Compatible input types: Image, Video, Audio

The format of the mediaSource object in the request body depends on whether the media is defined as a Base64-encoded string or as an S3 location.

  • Base64-encoded string

    {
    "mediaSource": {
        "base64String": "base64-encoded string"
    }
}

    • base64String – The Base64-encoded string for the media.

  • S3 location – Specify the S3 URI and the bucket owner.

    {
    "s3Location": {
        "uri": "string",
        "bucketOwner": "string"
    }
}

    • uri – The S3 URI containing the media.

    • bucketOwner – The AWS account ID of the S3 bucket owner.

Specifies which types of embeddings to retrieve.

  • Type: List

  • Required: No

  • Valid values for list members:

    • visual – Visual embeddings from the video.

    • audio – Embeddings of the audio in the video.

    • transcription – Embeddings of the transcribed text.

  • Default value:

    • Video: ["visual", "audio", "transcription"]

    • Audio: ["audio", "transcription"]

  • Compatible input types: Video, Audio

Specifies the scope of the embeddings to retrieve.

  • Type: List

  • Required: No

  • Valid values for list members:

    • clip – Returns embeddings for each clip.

    • asset – Returns embeddings for the entire asset.

  • Compatible input types: Video, Audio

The time point in seconds of the clip where processing should begin.

  • Type: Double

  • Required: No

  • Minimum value: 0

  • Default value: 0

  • Compatible input types: Video, Audio

The time point in seconds where processing should end.

  • Type: Double