AI Features

Renderly ships two AI building blocks that feed the render pipeline:

• Auto-captions — transcribe any audio/video URL into captions with real word-level timings (word-by-word highlighting works out of the box).
• AI voiceover — turn text into narration audio, stored in your media library.

Both are available two ways: as standalone endpoints under /api/v1/ai/*, and — the recommended way — declaratively inside POST /renders via special replacement values, so one API call produces a narrated, word-synced-captioned video.

Pricing

AI operations use the same credit balance as rendering, at flat, predictable rates:

Credits are deducted when the operation runs and refunded automatically if it fails. Every operation is logged as an AI job you can inspect via GET /ai/jobs/{jobId}.

Auto-captions

curl -X POST https://renderly.video/api/v1/ai/captions \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "sourceUrl": "https://your-bucket.s3.amazonaws.com/video.mp4" }'

The response contains two representations of the same result:

• captions — an array of caption "pages" (a few words each), every word carrying startMs / endMs / confidence.
• captionOverlay — a ready-to-paste caption overlay you can drop straight into a composition's overlays array.

Optional fields: language (ISO-639-1 hint, auto-detected when omitted) and fps (defaults to 30; used for the overlay's frame timings).

Captioning a clip of a longer file

If your composition plays only a cut of a longer source (a podcast highlight, a webinar excerpt), pass startTime / endTime (seconds into the source file):

{
  "sourceUrl": "https://your-bucket.s3.amazonaws.com/full-podcast.mp4",
  "startTime": 76,
  "endTime": 85
}

Only that window is transcribed, word timings come back clip-relative (0 = clip start, ready to drop onto a caption overlay aligned with the clip), and you're billed for the window only — 9 seconds costs 0.25 credits even if the file is an hour long.

Limits: files up to 5 GB, audio up to 3 hours. The source URL must be publicly reachable (uploaded Renderly assets qualify). Most files complete within the request; very long media returns status: "PROCESSING" — poll GET /ai/jobs/{jobId} until it completes. Retrying the same sourceUrl while a job is in flight resumes that job instead of charging again.

AI voiceover

curl -X POST https://renderly.video/api/v1/ai/voiceover \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "text": "Welcome to 42 Oak Street.", "voice": "nova" }'

Returns the mp3's public url, its measured durationMs, and a mediaAssetId — the audio is saved to your media library, so it's immediately usable as a sound overlay src or a replacement value. Voices: alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer; optional speed (0.25–4). Text up to 4,096 characters per request.

One call: narration + synced captions + render

The most powerful form: declare AI values directly in a render's replacements. Given a project with an isDynamic sound overlay named narration and an isDynamic caption overlay named subtitles:

curl -X POST https://renderly.video/api/v1/renders \
  -H "Authorization: Bearer rnd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "your_project_id",
    "replacements": {
      "headline": "42 Oak Street",
      "narration": {
        "tts": { "text": "This three bedroom home features a renovated kitchen and a large garden.", "voice": "nova" }
      },
      "subtitles": {
        "transcribe": { "source": "narration" }
      }
    }
  }'

What happens server-side, in order:

1. tts — the narration audio is generated, uploaded to your library, and its URL swapped into the narration sound overlay. The overlay (and the composition, if needed) is stretched to the audio's real length, and render credits are computed on the extended duration.

2. transcribe — the subtitles caption overlay is filled with word-synced captions transcribed from the referenced overlay's audio. source names any overlay with media (here the narration that was just generated — guaranteeing captions match the voiceover); use { "url": "https://..." } instead to transcribe an external file, and add language to skip auto-detection.

   Trimmed clips are handled automatically: when source is a video overlay that plays a cut of a longer file (videoStartTime + durationInFrames), only that portion is transcribed and billed, and caption timings are aligned to the clip as it plays — including speed changes and the caption overlay's own start frame. Pass startTime / endTime (seconds into the source file) inside transcribe to override the window explicitly.

3. The render proceeds as normal — captions burn in with the styling you gave the caption overlay in the editor.

This replaces a multi-step pipeline (script → TTS service → transcription service → compose → render) with a single request. Feed it rows from a spreadsheet or product feed and every record gets its own narrated, captioned video.

Caps: at most 3 tts and 3 transcribe values per render. AI credits are charged in addition to render credits and itemized as separate AI jobs.

In the editor

The same captioning engine powers the editor: select a video overlay → AI tab → Generate Captions. You'll see the credit cost before it runs, and the generated caption overlay lands at the playhead, ready to restyle.

Setting up a template for AI replacements

1. In the editor, add a sound overlay for the narration slot and a caption overlay, styled how you want.
2. Name them (e.g. narration, subtitles) and flag both Dynamic — see Templates & Personalization.
3. They now show up in GET /projects/{id}/variables and accept the tts / transcribe values above.

The narration slot needs no placeholder audio. A sound overlay with an empty src ("") is valid — it plays nothing in previews and renders safely if no replacement is provided, then the tts replacement fills it at render time. Alternatively, set a real sample narration as the default src so template previews sound finished — it gets replaced per render like any other variable. Don't point it at a fake/nonexistent URL: a dead audio link will fail any render that doesn't override it.