Tornevall Networks

← Back to docs

Whisper Transcriptions

Whisper Transcriptions

Tools provides a queue-based Whisper transcription service for media URLs and uploaded audio/video files.

This guide focuses on the public contract: what the feature does, how users interact with it, which endpoints exist, and what clients should expect in requests and responses.

What the feature does

Whisper in Tools can:

  • queue media for transcription
  • show live queue/job progress
  • show completed transcript segments while a job is still transcribing
  • store the finished transcript
  • optionally generate transcript analysis
  • optionally generate transcript translations
  • optionally attach estimated speaker labels when available
  • create a public transcript share page
  • re-transcribe a completed retained-media job with another supported model while keeping earlier completed passes available

Access models

Signed-in web/JWT API

The ordinary Whisper UI and authenticated API use:

  • signed-in web session auth, or
  • JWT bearer auth from POST /api/account/login

User permission requirements:

  • whisper.use for ordinary queue access
  • whisper.manage for full-queue/admin actions such as run-now and all-user visibility
  • provider_openai when transcript analysis/translations should run for a non-admin user

Token-authenticated transcribe API

Tools now also exposes a separate server-to-server transcription API for token-based integrations.

Auth requirements:

  • an active personal token with the access scope whisper.api (the built-in generator still creates a provider_whisper_api token row for convenience)
  • recommended transport: Authorization: Bearer YOUR_API_TOKEN
  • token owner must have whisper.api
  • token owner must also have normal Whisper access (whisper.use)
  • admin users bypass ordinary permission checks

Legacy X-Api-Key or apikey transport may still exist for backwards compatibility, but new integrations should use the Authorization header.

Queue behavior

Whisper jobs are processed asynchronously.

Job statuses:

  • queued
  • downloading
  • transcribing
  • finalizing
  • completed
  • failed

Jobs can also expose a queue origin:

  • queue_channel="web"
  • queue_channel="api"

Signed-in queue/detail views and authenticated /api/whisper/jobs* payloads can therefore show whether a job came from the regular user queue or the token-authenticated API queue.

Admin-owned jobs are prioritized ahead of non-admin jobs when queued work is claimed.

Web UI

/whisper

The signed-in queue UI lets users:

  • submit a media URL
  • upload one or more media files in the same file selection
  • choose model and language hints
  • set an optional title/label and free-text note
  • select analysis/translation language preferences
  • follow live queue progress
  • open job detail pages

For local uploads, multiple files can be selected at once. Tools uploads them one at a time and creates a separate Whisper job for each file with the same selected model, language, analysis, translation, and diarization settings.

During transfer, the page shows both overall batch transfer progress and separate progress/status for each file. After a file transfer completes, its row shows that Tools is waiting for the queue job to be created. If one file fails, the remaining files continue and successful jobs stay in the queue. The effective upload limit applies per file.

The authenticated API remains a single-file contract: media_file means one file per request. Multi-file behavior in /whisper is client-side batching on top of the existing enqueue API.

For URL jobs, Tools normally honors the configured yt-dlp proxy. If that proxy itself cannot be resolved or reached, the runner retries the same download directly so a temporary proxy outage does not immediately fail the job. Installations that require every media request to use the proxy can enable strict proxy mode; in that mode no direct retry is made.

Facebook extraction can return an anonymous "cannot parse data" result even when a saved signed-in session is available. Tools now treats that response as a reason to continue through the configured cookie sources instead of stopping after the anonymous attempts.

/whisper/jobs/{jobId}

The signed-in detail page shows:

  • current status and progress
  • source title/description
  • a runtime card that opens on the timestamped workflow log and can switch to the live transcript
  • transcript
  • transcript analysis
  • transcript translations
  • speaker-aware transcript when available
  • public share status
  • the current/primary transcription pass plus earlier completed passes when the job has been re-transcribed

Completed jobs can create a public transcript share page.

Completion email

When a Whisper job reaches a final completed or failed state and the owner has an email address, the owner report links back to the public Tools job page and never uses a local-only job address.

The report also contains a speaker-diarization summary. It states whether diarization was requested or disabled, shows its current or final status, and includes the provider, detected speaker count, labelled segment count, and safe warning or error text when those values are available.

Re-transcription passes

When the source media for a completed job is still retained, the transcript owner can choose Re-transcribe and select another model that is actually supported by the current Whisper runtime. The selected model must differ from the current pass. The existing completed pass is archived before the new pass enters the normal queue, so trying a heavier or otherwise different model does not destroy the previous transcript.

The job detail page keeps the ordinary job as the current/primary pass for backwards compatibility and shows earlier completed passes as read-only history. Each archived pass records the model and language used, its transcript, timestamped transcript segments, and the transcript-dependent analysis, translations, diarization and speaker-editor state that belonged to that pass. The new primary pass starts without those old transcript-dependent results; they can be generated again from the new transcript instead of being silently carried across models.

Re-transcription is owner-only even when another administrator can view the job. It is unavailable while the current pass is not completed, while translation or speaker-diarization work is still pending, or after the retained source media has been released. The UI shows the reason instead of offering a broken action. URL jobs reuse the retained local copy selected for the new pass; Tools does not silently download the external URL again. With the current remote-worker contract, retained-media URL re-transcriptions therefore stay on the local runner so a worker cannot accidentally re-fetch the original URL.

Model size is a capacity, compute and accuracy/robustness tradeoff rather than a simple vocabulary-size choice. Larger models generally require more resources and can perform better on difficult, multilingual or noisy speech. turbo is an optimized model derived from large-v3 and is useful when a stronger speed/accuracy balance is preferred.

Transcript-based job titles

For uploaded media, Tools tracks whether the current job title is still only the original upload filename. When the transcript completes, a user with OpenAI access can have that filename fallback replaced automatically with a short title based on the transcript content. A title explicitly supplied or edited by the user is never replaced, and a title-generation failure does not change an otherwise successful Whisper job into a failed job.

The completed transcript view also offers a dedicated title-suggestion action beside the transcript. It reuses the existing transcript metadata AI helper but fills only the editable title field. The suggestion is a preview: it is not persisted until the transcript owner reviews it and explicitly saves the metadata form. The existing title-and-description auto-fill action remains available separately.

Automatic and manually requested title-generation operations are written to the dedicated Whisper audit log as operational events. Transcript text and generated title content are not stored in that audit context.

While Whisper is transcribing, the live transcript view is updated through the same job-detail polling as the workflow log. Text appears one completed Whisper segment at a time, not word by word. The card keeps a bounded scroll area so a long transcript does not keep expanding the page layout. Incoming segments keep the view at the newest text only while the scrollbar is already at the bottom. Scrolling upward pauses that follow behavior until the scrollbar returns to the bottom.

The Live progress card now shows a prominent, color-coded runner state. It distinguishes a normally running job from a delayed heartbeat, a runner that has likely stopped, a stale job that is likely dead, a queued job waiting for a runner, and a finished job. The exact last heartbeat and last output times remain visible under the state.

Speaker diarization has a separate progress indicator in the speaker section. The transcription progress remains at 100% once the transcript itself is complete, while a queued or running diarization rerun is shown independently. The detail page follows the structured pyannote progress events already stored in the timestamped workflow log, so a measurable provider stage can show its current stage name, completed/total work, stage percentage, observed elapsed time, and age of the latest provider update. That percentage is explicitly stage-local and is never presented as the percentage for the whole diarization run. When the current provider stage has no trustworthy completed/total measurement, the diarization bar remains deliberately indeterminate instead of inventing a value. Queue, worker-claim, and provider-start markers are treated as boundaries for a new rerun, so retained provider events from the previous run are discarded immediately. If the transcript owner opens the page while a rerun is already pending or running, the rerun control remains available in a disabled state and confirms the current backend capability before becoming actionable again after a terminal transition. A terminal completed diarization still shows the whole-run indicator as 100%. If older stored jobs are missing timestamped JSON segments, Tools can also recover timings from generated subtitle artifacts before diarization without replacing the saved transcript, analysis, or translations.

If an active job has no runner heartbeat for 90 seconds, the page marks it as likely stopped. This warning is separate from the longer automatic stale-recovery threshold, so the UI can warn early without changing when a job is automatically reclaimed.

Workflow-log timestamps and all job-detail timestamps are shown explicitly in Europe/Stockholm, including Swedish standard time and daylight-saving time. The estimated finish uses the same time zone and includes an approximate remaining time. It prefers completed transcript segment timing when available, falls back to the current transcribing progress before that, and is recalculated by the existing polling as processing speed changes.

Each new run also mirrors the evolving text into a dedicated TXT snapshot after every completed segment. The job page shows the snapshot's storage and physical paths in the storage path trace. Long path values wrap inside their own rows so they do not overlap adjacent details.

Public transcript share page

Completed transcripts can be exposed through a tokenized public page under:

  • /shared/whisper/transcript/{token}

The share page is intended for reading/transcript sharing, not queue administration.

For token-authenticated API submissions, Tools can now create that share automatically when the transcript completes successfully, and the callback payload includes the direct share URL.

Authenticated Whisper API (/api/whisper/*)

These endpoints use signed-in web/JWT auth, not the dedicated Whisper API token.

GET /api/whisper/status

Returns queue counters and capability flags.

Typical response shape:

{
  "ok": true,
  "summary": {
    "queued": 3,
    "processing": 1,
    "completed": 21,
    "failed": 2
  },
  "can_manage_all": false,
  "config": {
    "enabled": true,
    "default_model": "large",
    "upload_max_mb": 64,
    "upload_limit": {
      "configured_mb": 200,
      "php_upload_max_mb": 64,
      "php_post_max_mb": 128,
      "effective_max_mb": 64,
      "effective_max_label": "64 MB",
      "limited_by_php": true
    },
    "ytdlp_configured": true
  }
}

upload_max_mb now reflects the practical/effective limit for uploaded Whisper media on the current host, and additive config.upload_limit can explain when PHP upload/body limits are lower than Whisper's own configured cap.

GET /api/whisper/jobs?limit=100

Returns visible Whisper jobs for the authenticated user.

POST /api/whisper/jobs

Queues a new Whisper job.

Supported request styles:

  • JSON/form body with source_url
  • multipart/form-data with media_file

Important rule:

  • send either source_url or media_file, not both
  • media_file is one file per request; the web UI's multi-file mode sends separate requests and creates separate jobs
  • if the uploaded file is too large for the current host, uploaded only partially, or is blocked by temporary-storage/PHP upload errors, the endpoint now returns a clearer 422 validation error under media_file instead of only the generic "failed to upload" wording

Example JSON body:

{
  "source_url": "https://example.com/audio.mp3",
  "source_label": "Interview with customer",
  "source_note": "Recorded support follow-up call.",
  "model": "large",
  "language": "sv",
  "analysis_language": "sv",
  "translation_target_languages": ["en"]
}

GET /api/whisper/jobs/{jobId}

Returns one visible Whisper job. The job returned here remains the current/primary transcription pass even after re-transcription, so existing clients that expect one transcript continue to use the same contract.

Additive job fields now include:

  • queue_channel
  • queue_channel_label
  • source_type
  • source_label
  • source_note
  • source_mime
  • source_size_bytes
  • source_duration_seconds
  • source_duration_human
  • stage_label
  • stage_detail
  • runtime_log[]
  • liveness
  • analysis
  • translations[]
  • diarization
  • share
  • callback (primarily relevant for API-queue jobs)

The additive liveness.state value can be inactive, active, quiet, unresponsive, stale, or suspect. unresponsive means the heartbeat has been absent long enough that the runner likely stopped, while stale means the existing automatic stale-recovery threshold has also been reached.

GET /api/whisper/jobs/{jobId}/revisions

Returns re-transcription state for one visible job: the explicit current/primary pass, supported model choices, whether the owner can start another pass, a human-readable unavailable reason when not, and earlier completed revision history. Historical revisions can include the transcript plus the analysis, translations, diarization and speaker-editor state that belonged to that archived pass.

Reading follows the ordinary Whisper job-visibility boundary. Starting a new pass remains owner-only.

POST /api/whisper/jobs/{jobId}/retranscribe

Queues a new transcription pass from the retained source media.

Example body:

{
  "model": "large"
}

Guardrails:

  • caller must own the job and have normal Whisper access
  • current pass must be completed
  • selected model must be supported and different from the current model
  • retained media must still be available
  • pending translation or speaker-diarization work must finish first
  • URL-source re-transcription reuses retained local media instead of silently downloading the original URL again

The current completed pass is archived transactionally before the job is queued again. Its transcript-dependent analysis, translations, diarization and speaker state remain attached to that history item, while the new primary pass starts clean. A new pass is also a new terminal-notification cycle for the owner.

POST /api/whisper/jobs/{jobId}/analyze

Runs transcript analysis for a completed transcript.

Guardrails:

  • transcript must already exist
  • non-admin users must have OpenAI access

POST /api/whisper/jobs/{jobId}/cancel

Requests cooperative cancellation for an actively processing job.

POST /api/whisper/jobs/{jobId}/restart

Queues a failed/queued job for retry.

DELETE /api/whisper/jobs/{jobId}

Deletes a non-processing job.

POST /api/whisper/run-now

Admin/manager helper endpoint.

Request body can include:

{
  "limit": 1,
  "reset_failed": true
}

Token-authenticated transcribe API (/api/whisper/transcribe/*)

This is the dedicated server-to-server callback API.

GET /api/whisper/transcribe/status

Returns queue counters for the token-authenticated API queue channel.

GET /api/whisper/transcribe/jobs?limit=100

Returns visible jobs from the API queue channel.

GET /api/whisper/transcribe/jobs/{jobId}

Returns one visible API-queue job.

POST /api/whisper/transcribe

Queues a new token-authenticated Whisper job.

Required field:

  • callback_url

Supported submission styles:

  • URL jobs using source_url
  • multipart file jobs using media_file

Upload validation guidance:

  • when the uploaded file is larger than the current practical host limit, the API can now return 422 with errors.media_file[] explaining the effective Whisper upload limit
  • the same media_file validation path is also used for partial uploads, missing temp-folder failures, write failures, and other PHP upload transport errors before the job is queued

The token API accepts the same additive metadata as the ordinary queue endpoint, including:

  • source_label
  • source_note
  • model
  • language
  • analysis_language
  • translation_target_languages[]
  • disable_diarization

Example JSON body:

{
  "source_url": "https://example.com/audio.mp3",
  "callback_url": "https://api.example.test/whisper/callback",
  "source_label": "Customer interview",
  "source_note": "Transcribe and send the final result back to our integration.",
  "model": "large",
  "language": "en",
  "analysis_language": "en",
  "translation_target_languages": ["sv"]
}

Example success response:

{
  "ok": true,
  "message": "Whisper API job queued. A callback will be sent when the job reaches a terminal state.",
  "job": {
    "id": 123,
    "queue_channel": "api",
    "queue_channel_label": "API queue",
    "status": "queued",
    "callback": {
      "url": "https://api.example.test/whisper/callback",
      "status": "pending",
      "http_status": null,
      "last_attempt_at": null,
      "delivered_at": null,
      "error": null
    },
    "share": null
  }
}

Callback contract

When a token-authenticated Whisper API job reaches terminal completed or failed, Tools sends one JSON POST to the submitted callback_url.

Callback envelope:

{
  "ok": true,
  "event": "whisper.job.completed",
  "job": {
    "job_id": 123,
    "status": "completed",
    "status_label": "Completed",
    "queue_channel": "api",
    "queue_channel_label": "API queue",
    "source": "Customer interview",
    "model": "large",
    "language": "en",
    "job_url": "https://tools.example.test/whisper/jobs/123",
    "share_url": "https://tools.example.test/shared/whisper/transcript/example-token-redacted",
    "transcript_text": "...",
    "analysis_text": "...",
    "translations": [],
    "share": {
      "url": "https://tools.example.test/shared/whisper/transcript/example-token-redacted"
    }
  }
}

Failure callbacks use event="whisper.job.failed" and can include failure_error instead of transcript/share data.

Client guidance:

  • treat callbacks as asynchronous terminal-state notifications
  • store them idempotently by job.job_id
  • do not assume a share URL exists on failed jobs
  • do not assume transcript analysis/translations are always present for every account

Error handling

Typical error classes:

  • 401 unauthenticated / token rejected
  • 403 missing permission
  • 404 job not found or not visible to the caller
  • 422 validation or business-rule failure
  • 429 throttled
  • 5xx temporary backend/provider failure

Rate limiting

Whisper API routes use a general throttle:120,1 policy.

Clients should still implement normal backoff for repeated polling or transient failures.

Safe client recommendations

  • Prefer Authorization: Bearer YOUR_API_TOKEN
  • Treat callback_url as required for token-authenticated submissions
  • Expect jobs to finish asynchronously
  • Surface queue_channel and queue_channel_label in operator/debug UIs
  • Treat transcript_text as the primary result and speaker_aware_transcript as additive helper output
  • Treat share.url as public access and handle it carefully

Enabling diarization after transcription

A completed transcript can queue speaker diarization afterwards even when diarization was disabled during the original transcription. Tools reuses the retained local audio/video file. When timestamped transcript segments are already stored, the rerun goes directly to diarization. If those older segment records are missing, the worker regenerates only the timestamped segments from the retained media, stores them on the existing job, and then runs diarization without replacing the saved transcript text, analysis, or translations.

Post-transcription diarization therefore requires the source media to still exist in Whisper storage, but saved transcript segments are no longer required up front. If the media has already been purged, the request is rejected clearly instead of creating a broken queue item. If segment recovery fails later, the completed transcript remains available and only the diarization attempt is marked failed. Enabling diarization afterwards also persists the preference on the job, so later runs behave as ordinary diarization reruns. During a pending or running rerun, the transcript owner's rerun control remains disabled; when the run reaches completed, failed, unavailable, or skipped, the UI rechecks the current backend capability before enabling that control. This allows a page opened mid-run to recover the action without a manual reload while keeping backend authorization and current job state authoritative.