Sign inSign up

cenk1cenk2/hermes-bridge-runs

By cenk1cenk2

•Updated 2 days ago

Brokers the Hermes runs API and holds jobs across Hermes restarts.

Image
0

10K+

cenk1cenk2/hermes-bridge-runs repository overview

⁠hermes-bridge-runs

A broker in front of a Hermes⁠ API server that speaks its own /v1/runs API. Every create becomes a job with an id of its own, held in Redis while Hermes is busy or down and resubmitted when a run is interrupted, so a caller that polls the job keeps getting an answer through Hermes restarts and broker rollouts. Talks directly to any deployment that exposes /v1/runs, and authenticates nobody: the gateway in front of it does.

⁠Image

docker run --rm -p 8080:8080 cenk1cenk2/hermes-bridge-runs:latest
TagSource
latestThe latest release.
v<version>The release @kilic.dev/hermes-bridge-runs@<version> created by semantic-release.

⁠Environment

Durations are milliseconds unless noted.

VariableDefaultDescription
PORT8080Port the HTTP server binds to.
LOG_LEVELinfoLowest level logged: error, warn, info, debug or verbose.
LOG_FORMATjsonjson for one JSON object per line, text for readable local lines.
REDIS_URLrequiredRedis or Valkey holding every queue and all shared state.
QUEUE_PREFIXhermes-bridge-runsPrefix of every Redis key and queue.
QUEUE_RETENTION_COMPLETED3600Seconds a completed job is kept.
QUEUE_RETENTION_FAILED86400Seconds a failed job is kept.
QUEUE_TRANSACTION_RETRIES10Attempts of a Redis transaction that keeps conflicting.
RUNS_RETENTION86400000Lifetime of a job, its idempotency key and its event stream after their last change.
RUNS_BODY_LIMIT4194304Largest create body in bytes.
RUNS_PROXY_ENABLEDfalseAnswer GET /v1/runs/{id} for an id no job has with the run of that id on Hermes, for runs created on Hermes before the broker took over.
RUNS_EVENT_MAX_LENGTH1000Events kept per job, trimmed from the oldest.
RUNS_EVENT_BLOCK5000Longest wait of an event stream for its next event before it checks whether its client is still there.
HERMES_URLrequiredBase URL of the Hermes API server, e.g. https://hermes.example.com/v1.
HERMES_API_KEYrequiredKey sent to the Hermes API server.
HERMES_INSTRUCTIONSnoneInstructions for a create whose body carries none.
HERMES_EVENTS_IDLE_TIMEOUT120000Silence on a run event stream before it is dropped and the run is polled.
DRIVER_SEEN_TTL86400000Lifetime of a seen idempotency key.
DRIVER_QUEUE_LOCK_TTL30000Lifetime of a thread lock.
DRIVER_QUEUE_LOCK_RENEW_INTERVAL10000Renewal interval of a held thread lock.
DRIVER_QUEUE_SWEEP_INTERVAL1000Interval of the sweep for threads with due work.
DRIVER_BACKOFF_INITIAL5000First retry delay while a run create gets 429, a 5xx or no connection.
DRIVER_BACKOFF_MAX60000Largest retry delay.
DRIVER_POLICY_BATCHfalseUnused: every job is one run of its own.
DRIVER_POLICY_STEERfalseUnused: a job is never steered.
DRIVER_POLICY_RESUBMIT_MAX0New runs for a job whose run ends interrupted, when the create has no X-Bridge-Resubmit.
DRIVER_POLICY_BACKOFF_WINDOW600000Time a job is held while Hermes is busy or unavailable before it fails, when the create has no X-Bridge-Hold.
DRIVER_POLICY_APPROVALdeny-stopdeny-stop stops a run after denying its approval request, deny-continue lets it go on.
DRIVER_FOLLOW_LEASE_TTL30000Lifetime of a run lease; another replica takes the run over once it expires.
DRIVER_FOLLOW_LEASE_RENEW_INTERVAL10000Renewal interval of a held run lease.
DRIVER_FOLLOW_SWEEP_INTERVAL5000Interval of the sweep for runs nobody follows.
DRIVER_FOLLOW_POLL_INTERVAL5000Interval between polls of a run without a stream.
DRIVER_FOLLOW_RETENTION86400000Time a finished run's record is kept.
DRIVER_HEARTBEAT_IDLE1200000Silence in a thread before it reports idle.
DRIVER_HEARTBEAT_SWEEP_INTERVAL60000Interval of the idle sweep.
BOARD_ENABLEDfalseDeliver thread state, source metadata and independent board/post reporting flags to the agent runs receiver; per-thread flags do not disable transport.
BOARD_URLrequired when enabledEndpoint the states are posted to, e.g. https://webhooks.kilic.dev/v1/hooks/hermes.
BOARD_TIMEOUT10000Timeout of one delivery POST or reconciliation status query.
BOARD_RETRIES3Retries of a post that fails or answers other than 2xx.
BOARD_REFRESH_INTERVAL60000Minimum interval between immediate working deliveries from real activity, shared across replicas; not the maintenance interval.
BOARD_RECONCILE_INTERVAL3600000Startup and hourly maintenance replay retained projections and repair known active runs; independent of event delivery and receiver rendering.
BOARD_TTL86400000Source lifetime of retained board projections and deduplication. Replays and status queries do not renew it.

⁠Slack

The shared core no longer polls reactions or manually resolves answer cards. The runs application posts no direct Slack answer cards. Runs board reports, independent reporting controls and the job lifecycle are unchanged. Historical Slack messages and unused Redis resolution records are left untouched. This application release includes the shared-core removal.

⁠Board

Reports include optional reporting: { board: boolean, post: boolean }: the receiver filters board visibility and agent announcement posts independently. Missing reporting or either missing flag means true. Both-disabled reports are still retained and delivered so the receiver can clear stale visibility; flags never suppress execution, results, errors or SSE. The resolved policy survives activity, terminal events, retries, outbox delivery and hourly reconciliation without renewing source expiry.

With BOARD_ENABLED, every thread posts its state to BOARD_URL as JSON {"key","platform","sessionId","state","title","link","links","error","runId","at","sourceActivityAt","reporting"}, next to the job status. running means WORKING while a turn is queued or executing; done means IDLE immediately after completion. Failures, stops and denied commands remain distinct as failed, stopped and approval_denied. Real tool, message, todo and thought activity refreshes running at most once per BOARD_REFRESH_INTERVAL across replicas; silence never changes an active turn to idle, and synthetic idle sweeps send no board updates. Working and completed entries expire from their last known source activity, not their last delivery; a later real turn reports working again. sessionId is optional and comes from the create's session_id, key is the thread key runs-<run_id>. Optional create context supplies title, link (from context.url) and links ([{"text","url"}]); without it, title falls back to the thread key and source links are absent. Attribution is stored with the job and driver input and retained on activity refreshes and retries. error carries the failure or denied command. at is an epoch-millisecond version that increases when a new projection is persisted; it orders updates, not receipt time. Optional sourceActivityAt is epoch milliseconds of the last known real accepted, queued, tool, message, thought, todo or terminal lifecycle activity. Even activity coalesced within BOARD_REFRESH_INTERVAL persists a new immutable projection with its source time. Silence, replay and unchanged status checks do not create activity or a fresh version. A repaired source change uses its known activity time, not query time. Settled and superseded occurrences reject late callbacks.

The complete latest projection and its pending delivery are stored atomically with a retained index, including terminal threads after driver cleanup. Driver lifecycle transitions await only Redis persistence; board delivery and platform/Slack HTTP remain detached. Immediate delivery keeps the BOARD_RETRIES budget. After releasing delivery ownership, the sender checks for a newer due pending version and drains it immediately, including reports persisted by another replica while delivery was finishing. The exhausted version itself waits for startup or hourly reconciliation rather than retrying in a tight loop. An acknowledgement clears only its exact version, so it cannot discard a newer report. Reconciliation also replays acknowledged latest projections to restore lost receiver history, byte-for-byte with the same at, source time, session, links, error and reporting policy. Redis token-owned locks coordinate replicas; at-least-once delivery can repeat an identical payload.

Receiver integration must deduplicate equal/older at without another announcement and expire entries at sourceActivityAt + HERMES_TTL, using only the remaining delay and ignoring expired replays. Receipt time is transport diagnostics, never source activity. Deploy that receiver support before enabling repeated replay; legacy JSON fields and the endpoint remain unchanged. Receiver rendering remains event-driven at 60 seconds, and daily message rotation belongs to the receiver, not this sender.

Active-run repair uses the established HERMES_URL and HERMES_API_KEY through GET /v1/runs/{id}; it never sends driver input, recovers a driver thread, or emits platform activities. The driver record and board version must still match the snapshot taken before the query. Explicit stop and denied-stop state override raw run status, while an interrupted run eligible for driver resubmission stays running. Hermes updated_at is seconds of a status write, not user activity: only a recognized activity/terminal event can supply a recovery source time, and interruption queries never use it because GET can synthesize it. Otherwise repair preserves a known board source time or the original follower start time; unknown or expired provenance is skipped. Terminal history with neither a retained projection nor a surviving driver source cannot be reconstructed, and no timestamps are invented for it. Existing refresh, retry and TTL setting names remain unchanged.

⁠Routes

Errors answer the Hermes envelope {"error":{"message","type","code"}}.

MethodPathDescription
GET/healthzLiveness: answers 200 ok.
GET/readyzReadiness: answers 200 ok, and 503 once a shutdown starts draining.
POST/v1/runsCreates a job from {"input","instructions"?,"session_id"?,"context"?,"reporting"?} and answers 202 {"run_id","status","replayed"}. With an Idempotency-Key, the same key, input and instructions answer the same run_id with replayed: true and Idempotency-Replayed: true on any replica, and different input or instructions answer 409 idempotency_key_conflict. 400 on a body that fails the schema, a malformed Idempotency-Key (invalid_idempotency_key) or a malformed X-Bridge-* header (invalid_header).
GET/v1/runs/{id}The job as a Hermes run: {"object":"hermes.run","run_id","status","session_id","created_at","updated_at","output","error","usage","attempts","held_since"}, times in seconds. 404 run_not_found for an unknown id, unless RUNS_PROXY_ENABLED hands it to Hermes.
GET/v1/runs/{id}/eventsServer-sent events of the job in the Hermes vocabulary, each with an id:; a Last-Event-ID header resumes after that event on any replica. Ends after run.completed, run.failed, run.cancelled or run.interrupted, and when the replica drains.
POST/v1/runs/{id}/stopStops the job and answers it with status: stopping; a finished job answers as it is.
anyany otherAnswers 404.

⁠Runs

The broker enqueues every create as a durable job in Redis instead of calling Hermes inline: a create answers immediately with the job id, the job survives Hermes restarts and rollouts, and RUNS_PROXY_ENABLED additionally exposes runs that were created on Hermes directly. RUNS_EVENT_BLOCK bounds how long an event stream waits for its next event before it verifies the client is still connected; raise it on high-latency links if streams close early.

Every job is a thread of its own, runs-<run_id>, that never batches or steers, so one job is one Hermes run at a time and a stop stops only that job. The job id is the idempotency key of its first Hermes create and <run_id>#<attempt> of each resubmit. A session_id in the body is passed to Hermes as the session of the run and included in board reports.

⁠Reporting visibility

A create may include top-level "reporting": { "board": false, "post": true } to hide its board entry while keeping agent announcement posts, or control either flag independently. Each flag is optional and must be a boolean; each omitted flag defaults to true, including {} or an absent object. Invalid reporting answers 400. Both flags default on for existing callers.

The first job to claim an Idempotency-Key owns its resolved reporting policy. Changed or omitted reporting on a replay does not change that policy, including queued replay on another replica; a legacy job without stored flags keeps both on. The execution fingerprint remains input plus instructions only. Reporting is stored on the job and its driver thread, never in the Hermes execution body, prompt or input metadata. Board state retains the complete report including reporting after driver cleanup, so failed delivery, restart and hourly replay preserve false values exactly. Receivers must support these independent flags before callers opt out; this sender does not filter either channel locally or change BOARD_ENABLED.

⁠Source context

A create may include display-only context without changing its prompt or instructions:

{
  "input": "Review the promotion.",
  "instructions": "Run the review skill.",
  "context": {
    "title": "Review staging promotion",
    "url": "https://kargo.example.com/project/staging",
    "links": [
      { "text": "Promotion", "url": "https://kargo.example.com/project/staging/promotion/1" },
      { "text": "Source", "url": "https://gitlab.example.com/group/project/-/merge_requests/12" }
    ]
  }
}

context.title is required when context is supplied: 1–500 characters, not whitespace-only. context.url is optional. context.links is optional with at most 10 entries; each requires nonblank text of 1–100 characters and a url. Every URL is at most 2048 characters, HTTP or HTTPS only, without username or password. Invalid context answers 400. Values are preserved, not extracted from or appended to the prompt, and context is never sent to Hermes.

The execution fingerprint remains input plus instructions only. A replay with changed or omitted context returns the original job and retains its original context, including a replay of a queued job on another replica. Adding context to a replay of a legacy job does not add attribution. Busy retries and interrupted resubmits keep the original driver input metadata. Existing creates without context keep working unchanged.

HeaderDefaultDescription
X-Bridge-HoldDRIVER_POLICY_BACKOFF_WINDOWMilliseconds the job is held while Hermes is busy or unavailable before it fails.
X-Bridge-ResubmitDRIVER_POLICY_RESUBMIT_MAXNew runs for the job when its run ends interrupted.
Jobstatus
Created, not yet taken by Hermes, or held while Hermes is busy or unavailable (held_since set)queued
Taken by Hermes, resubmitted after an interrupt, or stopping after a denied approvalrunning
Finishedcompleted with output as a string, or failed, cancelled or interrupted with error

attempts counts the Hermes runs of the job. A job, its idempotency key and its events are kept for RUNS_RETENTION after their last change.

⁠Logs

Every line carries a message that is a complete sentence fixed per event, its source class as context, and its ids, numbers and reasons as fields at the root of the JSON object ({"level":"log","message":"Created a job.","context":"RunsJobService","job":"job_...","key":"...","hold":2400000,"resubmit":3}); text prints the same fields inline. Durations are durationMs; bodies are never logged.

  • info: one line per HTTP request (method, path, status, duration, remote address; /healthz and /readyz only at debug), each created, replayed, held, accepted, stopped and finished job, each proxied run, run creation, busy backoff, follows and takeovers, finished runs with status and duration, Redis connections, the drain on shutdown and the runs, threads and event streams it hands over or ends.
  • debug: every appended job event, opened and closed event stream, run stream event, Hermes call (method, path, status, duration), thread lock and run lease acquire, renew and release, and Redis transaction retry.
  • warn and error: anything retried, rejected, dropped or failed, with its error.

⁠Development

The application lives in apps/runs of the workspace and runs on the shared core in packages/core; see the repository README.

pnpm install
pnpm lint
pnpm test
pnpm build
pnpm --filter @kilic.dev/hermes-bridge-runs start

⁠Platform boundary

The broker is platform-neutral. GitLab webhook classification, review skill selection, label filtering and MR lookups belong to the GitLab hook bridge, not the runs API. No GitLab credentials or review-specific endpoint are loaded here. Generic jobs, context, reporting, idempotency, polling, stop and SSE behavior are unchanged.

Removal of the former /v1/runs/reviews endpoint must be released and promoted only after the gateway routes every automatic GitLab review to the GitLab bridge. An old gateway still calling that endpoint would receive 404. Retain the old broker image until the gateway migration is verified; do not maintain a permanent GitLab compatibility path in runs.

Tag summary

Content type

Image

Digest

sha256:e8366ef33…

Size

72 MB

Last updated

2 days ago

docker pull cenk1cenk2/hermes-bridge-runs