AI Features

Renderly ships three AI building blocks that feed the render pipeline:

• Auto-captions - transcribe any audio/video URL into captions with real word-level timings (word-by-word highlighting works out of the box).
• AI voiceover - turn text into narration audio, stored in your media library.
• AI images - turn a text prompt into an image, stored in your media library.

All are available two ways: as standalone endpoints under /api/v1/ai/*, and - the recommended way - declaratively inside POST /renders via special replacement values, so one API call produces a narrated, word-synced-captioned, illustrated video.

Pricing

AI operations use the same credit balance as rendering, at flat, predictable rates:

Credits are deducted when the operation runs and refunded automatically if it fails. Every operation is logged as an AI job you can inspect via GET /ai/jobs/{jobId}.

An identical request is free. Renderly stores every voiceover, transcription and image you make. When you send the same request again, you get the stored result back for 0 credits, with cached: true in the response (cached: false when the work ran and was charged). This also applies inside a render, so retrying a failed render does not pay again for the narration, captions or images it already made. What counts as identical:

• Voiceover: the same text (leading and trailing spaces ignored), voice, speed and provider. An omitted voice or speed counts as the default. A voiceover whose stored audio you deleted from your media library is made again and charged.
• Auto-captions: the same sourceUrl (the exact string), startTime, endTime and language. fps does not matter: the caption overlay is rebuilt for the fps you send. Stored transcriptions do not expire, so when you replace the file behind a URL, change the URL too (for example ?v=2) to get a new transcription.
• Images: see AI images below.

Failed operations are never stored, and the cache is per user.

Free captioning has two conditions:

• No explicit clip window. Passing startTime / endTime inside transcribe forces a real transcription pass over that window, even when the source is narration you generated - the free path only applies when the whole generated clip is captioned.
• Speechify only. The openai voiceover provider (the fallback for languages Speechify cannot speak) returns no word timings, so there is nothing to reuse - captioning its narration is billed as a normal transcription at 0.25 credits per started minute.

Auto-captions

curl -X POST https://renderly.video/api/v1/ai/captions \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "sourceUrl": "https://your-bucket.s3.amazonaws.com/video.mp4" }'

The response contains two representations of the same result:

• captions - an array of caption "pages" (a few words each), every word carrying startMs / endMs / confidence.
• captionOverlay - a ready-to-paste caption overlay you can drop straight into a composition's overlays array.

Optional fields: language (ISO-639-1 hint, auto-detected when omitted) and fps (defaults to 30; used for the overlay's frame timings).

Captioning a clip of a longer file

If your composition plays only a cut of a longer source (a podcast highlight, a webinar excerpt), pass startTime / endTime (seconds into the source file):

{
  "sourceUrl": "https://your-bucket.s3.amazonaws.com/full-podcast.mp4",
  "startTime": 76,
  "endTime": 85
}

Only that window is transcribed, word timings come back clip-relative (0 = clip start, ready to drop onto a caption overlay aligned with the clip), and you're billed for the window only - 9 seconds costs 0.25 credits even if the file is an hour long.

Limits: files up to 5 GB, audio up to 3 hours. The source URL must be publicly reachable (uploaded Renderly assets qualify). Most files complete within the request; very long media returns status: "PROCESSING" - poll GET /ai/jobs/{jobId} until it completes. Retrying the same request while a job is in flight resumes that job instead of charging again, and retrying it after the job completed returns the stored captions for free.

AI voiceover

curl -X POST https://renderly.video/api/v1/ai/voiceover \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "text": "Welcome to 42 Oak Street.", "voice": "henry" }'

Returns the mp3's public url, its measured durationMs, a mediaAssetId, creditsUsed, cached (true when an identical earlier voiceover was returned for free), and words (an array of { word, startMs, endMs } timings for every spoken word) - the audio is saved to your media library, so it's immediately usable as a sound overlay src or a replacement value. Twelve curated voices: claudette, emily, geffenv1, henry (default), hugh_32, jacob, jordan, julie, monica, nick, phil, ruby; any Speechify catalogue voice id also works, including non-English ones across 36 locales. Optional speed (0.5-4). Text up to 4,096 characters per request. Sending the same request again returns the stored voiceover for 0 credits. When an identical request is still running, the call waits up to 30 seconds for its result, then returns 409 (retry: the retry returns that voiceover without charging again).

Because the words timings come back with the audio, captioning this narration is free - pass its overlay name as transcribe.source in a render and no transcription charge applies (see below).

AI images

curl -X POST https://renderly.video/api/v1/ai/images \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "prompt": "A sunlit kitchen with a marble island", "tier": "quality", "size": "1K", "aspectRatio": "16:9" }'

Fields: prompt (1-2,000 characters, required), tier (quality by default, or fast), size (1K by default, or 2K), aspectRatio (1:1 by default, or 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 4:5, 5:4, 21:9), style (up to 500 characters) and seed (an integer from 0 to 2147483647, 0 by default).

The response is { "success": true, "data": { "jobId", "url", "mediaAssetId", "creditsUsed", "cached", "width", "height" } }. The image is saved to your media library, so it is immediately usable as an image overlay src or a replacement value. Errors: 400 for a bad request or a refused prompt, 402 for too few credits, 409 when an identical image is still being made by an earlier request (retry - the retry returns that image without charging again), 502 when the image provider fails (credits are refunded), 503 when AI is not configured.

Pricing and the free cache. A quality image costs 0.5 credits and a fast image costs 0.2, at 1K. A 2K image costs double (1.0 and 0.4). The same prompt, style, size, aspect ratio, seed and tier returns the stored image free, with cached: true. The seed is a cache key: the same prompt, style, size, ratio and seed returns the stored image for free (default seed 0). Send a different seed to make a new version.

Pick the size for the frame. Use size: "2K" for a full-screen slide in a 1080p video - a 1K 16:9 image is about 1344x768 and is enlarged to fill the frame.

Images inside a render

Give an isDynamic image overlay a prompt instead of a URL:

{
  "projectId": "your_project_id",
  "imageStyle": "flat vector illustration, warm palette",
  "imageSeed": 42,
  "replacements": {
    "slide1": { "image": { "prompt": "A sunlit kitchen with a marble island", "size": "2K" } },
    "slide2": { "image": { "prompt": "A garden with a wooden deck at dusk", "tier": "fast" } },
    "slide3": { "image": { "prompt": "A mascot waving at the front door", "style": "3D Pixar render" } }
  }
}

• The value is { "image": { "prompt", "tier"?, "size"?, "aspectRatio"?, "style"? } }.
• The aspect ratio follows the overlay box: it defaults to the nearest supported ratio of the box. Set aspectRatio to override it.
• imageStyle (up to 500 characters) and imageSeed (an integer from 0 to 2147483647) sit at the top level of the request and apply to every image in the render, so imageStyle keeps the images in one style. The seed is a cache key: the same prompt, style, size, ratio and seed returns the stored image for free (default seed 0). Send a different seed to make a new version.
• style (up to 500 characters) sets the style for one image. It replaces imageStyle for that image and is never joined with it, so the other images keep imageStyle. A blank style is ignored. In the example above, slide3 uses "3D Pixar render" and the other two use the flat vector style. Two slots with the same prompt but different styles are two images, and each one is charged.
• At most 20 images per render.
• Before anything is charged, the render checks that your balance covers the uncached voiceovers and the uncached images, and that the paying account covers the render. If not, it returns 402 and charges nothing.
• A preview skips images and lists them under aiSkipped.images. Send confirmAiCost: true to generate them in the preview and be billed.

One call: narration + synced captions + render

The most powerful form: declare AI values directly in a render's replacements. Given a project with an isDynamic sound overlay named narration and an isDynamic caption overlay named subtitles:

curl -X POST https://renderly.video/api/v1/renders \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "your_project_id",
    "replacements": {
      "headline": "42 Oak Street",
      "narration": {
        "tts": { "text": "This three bedroom home features a renovated kitchen and a large garden.", "voice": "henry" }
      },
      "subtitles": {
        "transcribe": { "source": "narration" }
      }
    }
  }'

What happens server-side, in order:

1. tts - the narration audio is generated, uploaded to your library, and its URL swapped into the narration sound overlay. The overlay (and the composition, if needed) is stretched to the audio's real length, and render credits are computed on the extended duration.

2. transcribe - the subtitles caption overlay is filled with word-synced captions transcribed from the referenced overlay's audio. source names any overlay with media (here the narration that was just generated - guaranteeing captions match the voiceover); use { "url": "https://..." } instead to transcribe an external file, and add language to skip auto-detection. When source points at narration you generated with tts in the same request, the word timings come back with the audio and this step is free - no transcription charge.

   Trimmed clips are handled automatically: when source is a video overlay that plays a cut of a longer file (videoStartTime + durationInFrames), only that portion is transcribed and billed, and caption timings are aligned to the clip as it plays - including speed changes and the caption overlay's own start frame. Pass startTime / endTime (seconds into the source file) inside transcribe to override the window explicitly.

3. The render proceeds as normal - captions burn in with the styling you gave the caption overlay in the editor.

This replaces a multi-step pipeline (script → TTS service → transcription service → compose → render) with a single request. Feed it rows from a spreadsheet or product feed and every record gets its own narrated, captioned video.

Caps: at most 3 tts, 3 transcribe and 20 image values per render. AI credits are charged in addition to render credits and itemized as separate AI jobs.

Slideshow example: images + narration + captions

Given an isDynamic image overlay for each slide (slide1, slide2, slide3), a sound overlay named narration and a caption overlay named subtitles:

{
  "projectId": "your_project_id",
  "imageStyle": "soft watercolor, muted colors",
  "replacements": {
    "slide1": { "image": { "prompt": "A quiet harbor at sunrise", "size": "2K" } },
    "slide2": { "image": { "prompt": "Fishing boats leaving the harbor", "size": "2K" } },
    "slide3": { "image": { "prompt": "A market stall full of fresh fish", "size": "2K" } },
    "narration": { "tts": { "text": "Every morning the harbor wakes before the town does.", "voice": "henry" } },
    "subtitles": { "transcribe": { "source": "narration" } }
  }
}

The three images, the narration and the captions are made in one request. Image and voiceover credits are charged in addition to render credits.

In the editor

The same captioning engine powers the editor: select a video overlay → AI tab → Generate Captions. You'll see the credit cost before it runs, and the generated caption overlay lands at the playhead, ready to restyle.

Setting up a template for AI replacements

1. In the editor, add a sound overlay for the narration slot and a caption overlay, styled how you want.
2. Name them (e.g. narration, subtitles) and flag both Dynamic - see Templates & Personalization.
3. They now show up in GET /projects/{id}/variables and accept the tts / transcribe values above.
4. An isDynamic image overlay needs nothing extra to accept a prompt. It accepts { "image": { "prompt": ... } } as well as a URL.

The narration slot needs no placeholder audio. A sound overlay with an empty src ("") is valid - it plays nothing in previews and renders safely if no replacement is provided, then the tts replacement fills it at render time. Alternatively, set a real sample narration as the default src so template previews sound finished - it gets replaced per render like any other variable. Don't point it at a fake/nonexistent URL: a dead audio link will fail any render that doesn't override it.