Execution Timeline (15 min) →
Execution Outcomes
Task Types
Schedule Types
Recent Executions →
Recent Events →
Top Groups →
Agents →
Pipeline Stages →
Dependency Map →
▶ Live Output
Timeline (1 hour)
Recent Events
Execution History
Dependencies & Connections
Create Job
0 * * * * *
Example: To react when job "health-check" outputs an error pattern:
• On "health-check", add an Output Trigger with pattern "ERROR" and severity "error"
• On this job, set Event Kind to
output.matched, Severity to error, Source Job to health-check
Schedule Window (optional)
{{VAR_NAME}}.New Script
print(msg)— append to job outputhttp_get(url)— returns#{status, body}http_post(url, body)— returns#{status, body}shell_exec(cmd)— returns#{exit_code, stdout, stderr}env_var(name)— read environment variablesleep_ms(ms)— pause executionfail(msg)— mark execution as failedudp_send(addr, data)/tcp_send(addr, data)hex_encode(string)/hex_decode(hex)
FROM— base imageRUN— execute command during buildCOPY/ADD— add files to imageWORKDIR— set working directoryENV— set environment variableEXPOSE— document portCMD— default commandENTRYPOINT— container entry command
Global Variables
| Name | Value | Updated |
|---|
Connections
New Connection
AI Job Builder
Describe what you want in plain English. AI generates the full job configuration — you review and save.
AI is not configured. Add your API key in Settings to enable.
Supports Anthropic Claude and OpenAI GPT.
Generated Job
Custom Agents
Custom agents use a pull-based model — they poll the controller for work, execute it however they want, and post the result back. Build agents in any language with just an HTTP client. No server needed.
How It Works
Custom agents separate workflow definition from implementation:
- The UI defines the workflow — admins configure task types (name, description, form fields) on the agent card in the dashboard
- The agent code defines the implementation — the agent handles whatever task data it receives
This means you can reconfigure task types without restarting the agent, and non-developers can set up workflows through the dashboard.
Setup
- Start the agent — it registers with
agent_type: "custom"and begins polling - Configure task types in the UI — go to Agents page, click the custom agent card, add task types with field definitions, and save
- Create a job — select "Custom Agent" execution mode, pick your agent, fill in the dynamic form, and trigger
Protocol
All agent endpoints require no API key.
1. Register
POST /api/agents/register
{
"name": "my-python-agent",
"tags": ["python", "ml"],
"hostname": "ml-box",
"address": "0.0.0.0",
"port": 0,
"agent_type": "custom"
}
// Returns: {"agent_id": "uuid", "heartbeat_interval_secs": 10}
2. Discover Task Types (optional)
GET /api/agents/{agent_id}/task-types
// Returns: [{"name": "python", "fields": [...]}, ...]
Useful for logging on startup or validating that your agent handles the configured types.
3. Poll for Work
GET /api/agent-queue/{agent_id}/next
// Returns 204 if no work, or 200 with job JSON:
{
"queue_id": "uuid",
"execution_id": "uuid",
"job_id": "uuid",
"agent_id": "uuid",
"task": {
"type": "custom",
"agent_task_type": "python",
"data": {"script": "print('hello')", "args": "--verbose"}
},
"callback_url": "http://controller:8080/api/callbacks/execution-result"
}
Your agent switches on task.agent_task_type and reads field values from task.data.
4. Report Result
POST {callback_url}
{
"execution_id": "...", "job_id": "...", "agent_id": "...",
"status": "succeeded", "exit_code": 0,
"stdout": "output", "stderr": "",
"stdout_truncated": false, "stderr_truncated": false,
"started_at": "2026-03-25T10:00:00Z",
"finished_at": "2026-03-25T10:00:05Z"
}
Task Type Definitions
Configured per-agent in the UI. Each definition has a name, optional description, and a fields array:
{
"name": "python",
"description": "Run a Python script",
"fields": [
{"name": "script", "label": "Script", "field_type": "textarea", "required": true},
{"name": "args", "label": "Arguments", "field_type": "text", "required": false}
]
}
Supported field types: text, textarea, number, select, password
Select fields include an options array with value/label pairs.
Task types can also be managed via the API: GET/PUT /api/agents/{id}/task-types
Queue Behavior
- Jobs are queued in the controller database until the agent polls
- Unclaimed jobs are failed after 5 minutes
- Claimed but unreported jobs are failed after 10 minutes
- The UI shows a "queued" badge (amber) for pending custom agent jobs
- Polling also acts as a heartbeat — no separate call needed
Execution Modes
| Mode | Task Types | Targets |
|---|---|---|
| Standard | Shell, HTTP, SQL, FTP, Script | Controller / Specific Agent / Any / All |
| Custom Agent | Defined per agent in UI | Specific custom agent |
Python Example
A complete working example is at examples/custom_agent.py:
pip install requests
python3 examples/custom_agent.py
# Configure with:
KRONFORCE_URL=http://controller:8080 \
AGENT_NAME=my-agent \
AGENT_TAGS=python,ml \
python3 examples/custom_agent.py
gRPC Agent Example
A custom agent that calls gRPC services using grpcurl. Supports reflection and proto file discovery.
brew install grpcurl
KRONFORCE_AGENT_KEY=kf_your_key python3 examples/grpc_agent.py
Configure two task types on the agent card:
- grpc_call — call a method (address, service, method, JSON data, optional proto file, plaintext toggle, metadata headers)
- grpc_list — list services or methods via reflection
Scripting
Kronforce supports two script types:
- Rhai Scripts (
.rhai) — lightweight scripting language embedded in the Rust binary. Use thescripttask type. - Dockerfile Scripts (
.dockerfile) — Docker image definitions. Use thedocker_buildtask type to build images and optionally run containers.
Both types are managed in Toolbox → Scripts with syntax highlighting and a reference panel. Select the script type when creating a new script.
Managing Scripts
Use Toolbox → Scripts (with syntax-highlighted editor), the API, or drop .rhai files in the scripts directory.
# Create/update via API
curl -X PUT http://localhost:8080/api/scripts/health-check \
-H "Authorization: Bearer kf_your_key" \
-H "Content-Type: application/json" \
-d '{"code": "let resp = http_get(\"https://api.example.com/health\");\nif resp.status != 200 { fail(\"down\"); }\nprint(\"OK\");"}'
# Or drop a file
echo 'print("hello");' > ./scripts/hello.rhai
Available Functions
| Function | Returns | Description |
|---|---|---|
print(msg) | — | Appends to job output |
http_get(url) | #{status, body} | HTTP GET request |
http_post(url, body) | #{status, body} | HTTP POST request |
shell_exec(cmd) | #{exit_code, stdout, stderr} | Run a shell command |
env_var(name) | string | Read environment variable |
sleep_ms(ms) | — | Sleep for N milliseconds |
fail(msg) | — | Mark execution as failed |
udp_send(addr, data) | #{sent, error} | Send string via UDP |
tcp_send(addr, data) | #{response, error} | Send string via TCP |
udp_send_hex(addr, hex) | #{sent, error} | Send raw bytes via UDP |
tcp_send_hex(addr, hex) | #{response_hex, response, error} | Send raw bytes via TCP |
hex_encode(s) | string | String to hex |
hex_decode(hex) | string | Hex to string |
Examples
Health check with Slack notification
let resp = http_get("https://api.example.com/health");
if resp.status != 200 {
http_post("https://hooks.slack.com/services/T00/B00/xxx",
`{"text": "API is DOWN! Status: ${resp.status}"}`);
fail("Health check failed");
}
print("API is healthy");
Run command and check output
let result = shell_exec("df -h / | tail -1 | awk '{print $5}'");
print("Disk usage: " + result.stdout);
if parse_int(result.stdout.replace("%", "")) > 90 {
fail("Disk usage critical");
}
TCP protocol check
let result = tcp_send("192.168.1.10:6379", "PING\r\n");
if result.error != "" {
fail("Redis unreachable: " + result.error);
}
print("Redis: " + result.response);
Sandboxing
- 1,000,000 operations max (prevents infinite loops)
- 256KB string size max
- Timeout enforced by the job's
timeout_secssetting - No direct file system access (use
shell_exec)
Task Types
| Type | Execution | Config Fields |
|---|---|---|
shell | Runs sh -c (or sudo -n -u with run_as) | command |
http | In-process HTTP request via reqwest | method, url, headers, body, expect_status |
sql | Shells out to psql/mysql/sqlite3 | driver, connection_string, query |
ftp | Uses curl for FTP/FTPS/SFTP | protocol, host, port, username, password, direction, remote_path, local_path |
script | Rhai scripting engine | script_name |
custom | Dispatched to custom agent | agent_task_type, data (fields defined per agent in UI) |
file_push | Base64 decode + write to filesystem | filename, destination, content_base64, permissions, overwrite |
kafka | kafka-console-producer via shell | broker, topic, message, key, properties |
rabbitmq | amqp-publish via shell | url, exchange, routing_key, message, content_type |
mqtt | mosquitto_pub via shell | broker, port, topic, message, qos, username, password |
redis | redis-cli PUBLISH via shell | url, channel, message |
mcp | Calls a tool on an MCP server via JSON-RPC | server_url, tool, arguments |
The first 5 types are built-in and run on the controller or standard agents. The custom type is for custom agents. File Push deploys files to agents. Message queue types (Kafka, RabbitMQ, MQTT, Redis) shell out to CLI tools.
Shell
Runs a command via sh -c. Supports run_as for sudo execution.
{"type": "shell", "command": "echo hello"}
HTTP
Makes an in-process HTTP request. Response body is captured as stdout, status code as exit code.
{"type": "http", "method": "get", "url": "https://api.example.com/health", "expect_status": 200}
SQL
Executes a query via psql, mysql, or sqlite3 CLI tools. Query results are captured as output.
{"type": "sql", "driver": "postgres", "connection_string": "postgresql://user:pass@host/db", "query": "SELECT count(*) FROM orders"}
FTP/SFTP
File transfers via curl. Supports FTP, FTPS, and SFTP protocols.
{"type": "ftp", "protocol": "sftp", "host": "ftp.example.com", "username": "user", "password": "pass", "direction": "upload", "local_path": "/data/report.csv", "remote_path": "/uploads/report.csv"}
Custom
Dispatched to a custom agent. Task types and fields are defined per-agent in the UI. Cannot run locally.
{"type": "custom", "agent_task_type": "python", "data": {"script": "print('hello')", "args": "--verbose"}}
File Push
Upload a file in the job form and deploy it to an agent. Content is base64-encoded in the task JSON (5MB limit).
{"type": "file_push", "filename": "app.conf", "destination": "/opt/app/app.conf", "content_base64": "...", "permissions": "644", "overwrite": true}
Kafka
Publish a message to a Kafka topic via kafka-console-producer.
{"type": "kafka", "broker": "localhost:9092", "topic": "events", "message": "{\"event\":\"test\"}", "key": "user-123"}
RabbitMQ
Publish a message to a RabbitMQ exchange via amqp-publish.
{"type": "rabbitmq", "url": "amqp://guest:guest@localhost:5672", "exchange": "events", "routing_key": "user.created", "message": "{\"user\":\"alice\"}"}
MQTT
Publish a message to an MQTT topic via mosquitto_pub.
{"type": "mqtt", "broker": "localhost", "port": 1883, "topic": "sensors/temp", "message": "22.5", "qos": 1}
Redis
Publish a message to a Redis Pub/Sub channel via redis-cli.
{"type": "redis", "url": "redis://localhost:6379", "channel": "notifications", "message": "{\"type\":\"alert\"}"}
MCP (Model Context Protocol)
Call tools on MCP-compatible servers. Connects to MCP servers via HTTP.
{"type": "mcp", "server_url": "http://localhost:8000/mcp", "tool": "analyze_logs", "arguments": {"path": "/var/log/app.log"}}
Use "Discover Tools" in the job form to browse available tools from a server. Tool results are captured as stdout. The arguments field supports {{VAR_NAME}} variable substitution.
Quick Start with the Test Server
# Install the MCP Python SDK
pip install mcp
# Run the included example server (starts HTTP server)
python3 examples/mcp_test_server.py
Then create an MCP job with:
- Server URL:
http://localhost:8000/mcp - Tool:
greet - Arguments:
{"name": "World"}
Available test tools: greet, add, system_info, word_count, reverse.
API Reference
All /api/* endpoints (except health, agent registration, polling, and callbacks) require an API key via the Authorization: Bearer kf_... header.
Jobs
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/jobs | List all jobs (paginated, filterable by status, searchable) |
| POST | /api/jobs | Create a job |
| GET | /api/jobs/{id} | Get job details |
| PUT | /api/jobs/{id} | Update a job |
| DELETE | /api/jobs/{id} | Delete a job |
| POST | /api/jobs/{id}/trigger | Trigger a job now |
Executions
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/jobs/{id}/executions | List executions for a job |
| GET | /api/executions | List all executions (filterable, searchable) |
| GET | /api/executions/{id} | Get execution details with output |
| POST | /api/executions/{id}/cancel | Cancel a running execution |
Agents
| Method | Endpoint | Auth | Description |
|---|---|---|---|
| GET | /api/agents | Yes | List all agents |
| GET | /api/agents/{id} | Yes | Get agent details |
| DELETE | /api/agents/{id} | Yes | Deregister an agent |
| PUT | /api/agents/{id}/task-types | Yes | Update custom agent task type definitions |
| GET | /api/agents/{id}/task-types | No | Get task type definitions (for agent discovery) |
| POST | /api/agents/register | No | Register an agent |
| GET | /api/agent-queue/{id}/next | No | Poll for work (custom agents) |
| POST | /api/agents/{id}/heartbeat | No | Agent heartbeat |
| POST | /api/callbacks/execution-result | No | Report execution result |
Events
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/events | List events (paginated) |
Scripts
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/scripts | List all scripts |
| GET | /api/scripts/{name} | Get script code |
| PUT | /api/scripts/{name} | Create or update script |
| DELETE | /api/scripts/{name} | Delete a script |
API Keys
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/keys | List API keys |
| POST | /api/keys | Create a key (name, role, optional allowed_groups) |
| DELETE | /api/keys/{id} | Revoke a key |
Group scoping: Set allowed_groups when creating a key to restrict it to specific job groups. Admin keys always see everything. Keys without group restrictions see all groups.
Approval Workflows
Jobs with approval_required: true create a pending_approval execution when triggered instead of running immediately.
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/executions/{id}/approve | Approve a pending execution (requires write access) |
Job Versions
Every job create/update saves a full snapshot. Query the version history for audit trail or rollback reference.
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/jobs/{id}/versions | List version history (newest first) |
Secret Variables
Variables marked as secret have their values masked (••••••••) in API responses and the UI. Secret values are still substituted into task fields at runtime via {{VAR_NAME}}.
Job Templates
Save any job as a reusable template, then create new jobs from it.
- Save — click "Save as Template" on any job's detail page
- Create — click "Template" on the Monitor → Jobs tab to browse and select
- Share — export via
GET /api/templates, import viaPOST /api/templates
Templates capture task config, target, notifications, retry, SLA, and approval settings. Schedule and name are set fresh on each use.
Priority Scheduling
Jobs have a priority field (default 0). Higher priority jobs are scheduled first when multiple are due simultaneously. Set via the job create/update API or the Advanced tab in the job modal.
SLA Deadlines
Set an SLA deadline on a job to track whether it completes on time. Configure in the job modal Advanced tab:
- Deadline — time by which the job must finish (HH:MM, UTC)
- Warning — minutes before the deadline to fire a warning event
When a job with an SLA deadline is still running:
- At
deadline - warning_mins: firessla.warningevent (severity: warning) - At
deadline: firessla.breachevent (severity: error)
SLA events trigger notifications (Slack, email, PagerDuty, etc.) and can also trigger other jobs via event-driven scheduling.
OIDC/SSO (Optional)
Set KRONFORCE_OIDC_ISSUER and KRONFORCE_OIDC_CLIENT_ID to enable SSO. The login screen shows a "Sign in with SSO" button alongside API key login.
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/auth/oidc/config | Check if OIDC is enabled |
| GET | /api/auth/oidc/login | Redirect to IdP for login |
| GET | /api/auth/oidc/callback | IdP callback (creates session) |
| POST | /api/auth/logout | Clear SSO session |
Role mapping: Use KRONFORCE_OIDC_ROLE_CLAIM (default: groups) to specify which claim to read. Set KRONFORCE_OIDC_ADMIN_VALUES and KRONFORCE_OIDC_OPERATOR_VALUES to map claim values to roles. Unmatched users get the KRONFORCE_OIDC_DEFAULT_ROLE (default: viewer).
Notifications
Three notification channels, all configured in Settings:
- Email (SMTP) — send to email addresses via any SMTP server
- SMS (Webhook) — send via Twilio or any SMS webhook API
- Webhook (Slack / Teams / PagerDuty) — post to incoming webhook URLs with format-specific payloads
Per-job notification triggers: on failure, on success, on assertion failure. Per-job recipient override or fall back to global recipients.
System alerts: agent went offline notification.
Prometheus Metrics
Scrape GET /metrics (no auth required) for Prometheus-compatible metrics:
kronforce_executions_total— total executions by statuskronforce_jobs_total— total number of jobskronforce_agents_total— total registered agentskronforce_groups_total— total job groupskronforce_jobs_by_task_type— jobs by task typekronforce_db_ok,kronforce_db_size_bytes,kronforce_db_wal_size_bytes— database health
Other
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/health | Health check with DB status (no auth) |
| GET | /metrics | Prometheus metrics (no auth) |
| GET | /api/auth/me | Current key/session info |
| GET | /api/mcp/tools | Discover tools from an MCP server (query: server_url) |
Global Variables
Global variables are shared key-value pairs that can be referenced in any task field. Manage them in Toolbox → Variables or via the /api/variables API.
Using Variables in Tasks
Reference variables using {{VAR_NAME}} syntax in any task field — shell commands, HTTP URLs, SQL queries, message bodies, etc.
| Task Field | Example |
|---|---|
| Shell command | curl {{API_HOST}}/health |
| HTTP URL | {{BASE_URL}}/api/status |
| Kafka topic | {{ENV}}_events |
Variables are resolved controller-side before execution, so they work for both local and remote agent jobs. Shell variables like ${HOME} are unaffected.
Output Extraction Targets
Each extraction rule has a target that controls where the extracted value goes:
- Variable (default) — writes the extracted value to a global variable (set the "Write to var" field). Other jobs can reference it with
{{VAR_NAME}}. - Output — replaces the execution's saved stdout with just the extracted values. Useful for filtering noisy output down to the data you care about.
Patterns without capture groups use the full match. Use (\d+) to extract just a portion, or \d+% to extract the full match.
Updating Variables from Job Output
Set the target to Variable and enter a variable name in the "Write to var" field to automatically update a global variable with the extracted value after each run.
Example pipeline:
- Job A runs
deploy.shand outputsversion=2.4.1 - An extraction rule captures
versionwith target Variable and write_to_variableDEPLOY_VERSION - Job B uses
echo "Deployed {{DEPLOY_VERSION}}"in its command
Variable Name Rules
Names must match [A-Za-z0-9_]+ (letters, numbers, underscores only).
Variable Expiration
Variables can have an optional expiration (30, 90, 180, or 365 days). Set this when creating a variable in Toolbox → Variables. Expired variables show a red "expired" badge in the UI. Useful for rotating secrets and temporary configuration that should be reviewed periodically.
Unresolved Variables
If a {{VAR}} reference doesn't match any defined variable, the placeholder is left as-is and a warning is logged. The job still runs.
Connections
Connections are named credential profiles for external systems. Instead of embedding passwords in task definitions, reference a connection by name. Credentials are encrypted at rest using AES-256-GCM.
Supported Types
| Type | Config Fields | Used By |
|---|---|---|
| PostgreSQL | connection_string | SQL task |
| MySQL | connection_string | SQL task |
| SQLite | connection_string (path) | SQL task |
| FTP / SFTP | host, port, username, password, private_key | FTP task |
| HTTP | base_url, auth_type, token/username/password/header | HTTP task |
| Kafka | broker, SASL username/password | Kafka publish/consume |
| MQTT | broker, port, username, password, client_id | MQTT publish/subscribe |
| RabbitMQ | url | RabbitMQ publish/consume |
| Redis | url, password | Redis publish/read |
| MongoDB | connection_string | Future use |
| SSH | host, port, username, password, private_key | Future use |
| SMTP | host, port, username, password | Future use |
| S3 / MinIO | endpoint, bucket, region, access_key, secret_key | Future use |
Using a Connection in a Job
Add a "connection": "my-conn-name" field to any task that supports it. The connection's credentials are merged into the task at execution time. Inline fields take precedence.
{
"task": {
"type": "sql",
"driver": "postgres",
"query": "SELECT count(*) FROM orders",
"connection": "prod-db"
}
}
The connection_string is resolved from the "prod-db" connection at runtime. You only configure the password once.
HTTP Auth Injection
HTTP connections support 4 auth types:
- None — no auth headers injected
- Bearer — adds
Authorization: Bearer <token> - Basic — adds
Authorization: Basic <base64> - Custom Header — adds a named header (e.g.,
X-API-Key: value)
API
GET /api/connections # List all (masked)
GET /api/connections/{name} # Get one (masked)
POST /api/connections # Create
PUT /api/connections/{name} # Update
DELETE /api/connections/{name} # Delete
POST /api/connections/{name}/test # Test connectivity
Sensitive fields (password, token, secret_key, private_key) are masked as ******** in API responses. When updating, send ******** to preserve the existing value.
AI Assistant
When enabled, the Create Job modal shows a "Describe what this job should do" prompt. Type a natural language description and click Generate to have AI fill in all form fields — name, task type, schedule, timeout, notifications, and more.
Setup
Configure AI from Settings → AI Assistant (no restart needed), or via environment variables:
KRONFORCE_AI_API_KEY=sk-ant-... # Anthropic or OpenAI API key
KRONFORCE_AI_PROVIDER=anthropic # "anthropic" (default) or "openai"
KRONFORCE_AI_MODEL=claude-sonnet-4-5-20250514 # Optional model override
The AI page is always visible in the sidebar. When no key is configured, it shows setup instructions with disabled controls. No data is sent to the AI provider except the user's description text.
Examples
| You type | AI generates |
|---|---|
| "back up postgres every night at 3am" | Shell task with pg_dump, cron 0 0 3 * * *, timeout 1h, notify on failure |
| "check if our API is healthy every 5 minutes" | HTTP GET task, cron 0 */5 * * * *, expect 200, Monitoring group |
| "run ETL extract daily at 6am weekdays only" | Shell task, cron 0 0 6 * * 1-5, ETL group |
| "clean up temp files older than 7 days every Sunday" | Shell task with find /tmp -mtime +7 -delete, cron 0 0 0 * * 0 |
How It Works
The AI receives a system prompt describing all Kronforce task types, schedule formats, and configuration options. It returns a JSON job definition that is parsed and used to populate every field in the form. You review and edit before saving — the AI never creates jobs directly.
API
POST /api/ai/generate-job
{"prompt": "back up postgres every night at 3am"}
# Returns:
{"job": {"name": "postgres-backup", "task": {...}, "schedule": {...}, ...}}
Job Groups
Groups let you organize jobs into logical categories (e.g., "ETL", "Monitoring", "Deploys"). Every job belongs to a group — new jobs default to the Default group unless you choose another.
Managing Groups
- Pipelines tab — view all groups in the Stages view, create new groups, rename or delete existing ones
- Job modal — pick a group from the dropdown when creating or editing a job
- Bulk assign — on Monitor → Jobs, select jobs with checkboxes, click "Set Group"
Group Rules
- Names: 1–50 characters, letters, numbers, spaces, hyphens, underscores
- The Default group always exists and cannot be deleted
- Deleting a group moves its jobs to Default
- Renaming a group updates all jobs in that group
API
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/jobs/groups | List all group names |
| POST | /api/jobs/groups | Create an empty group ({"name": "ETL"}) |
| PUT | /api/jobs/bulk-group | Assign group to multiple jobs ({"job_ids": [...], "group": "ETL"}) |
| PUT | /api/jobs/rename-group | Rename a group ({"old_name": "ETL", "new_name": "Data Pipeline"}) |
Filter jobs by group: GET /api/jobs?group=ETL
Execution Retry
Jobs can automatically retry on failure or timeout. Configure retry settings in the Advanced tab of the job create/edit modal.
Settings
| Field | Default | Description |
|---|---|---|
| Max Retries | 0 (disabled) | Number of retry attempts after the initial run |
| Retry Delay | 0 seconds | Initial delay between retries |
| Backoff Multiplier | 1.0 (fixed) | Multiply delay by this factor each attempt (e.g., 2.0 = exponential backoff) |
How It Works
- Retries trigger only on Failed or Timed Out status
- No retry on success, cancellation, or assertion failure
- Delay formula:
delay_secs × backoff^(attempt - 1), capped at 1 hour - Each retry creates a new execution linked to the original (
retry_offield) - Execution detail shows "Attempt 2/3" for retry executions
Example
A job with max_retries: 3, retry_delay_secs: 5, retry_backoff: 2.0:
- Attempt 1: runs immediately (fails)
- Attempt 2: waits 5s, retries (fails)
- Attempt 3: waits 10s, retries (fails)
- Attempt 4: waits 20s, retries (succeeds — done)
API
Set retry config on job creation or update:
{"retry_max": 3, "retry_delay_secs": 5, "retry_backoff": 2.0}
Rate Limiting
All API endpoints are rate limited to prevent abuse. Exceeding the limit returns 429 Too Many Requests.
Tiers
| Tier | Default Limit | Scope | Endpoints |
|---|---|---|---|
| Public | 30/min | Per source IP | Health, dashboard |
| Authenticated | 120/min | Per API key | All authenticated API routes |
| Agent | 600/min | Per API key | Agent register, heartbeat, queue, callbacks |
Response Headers
| Header | Description |
|---|---|
X-RateLimit-Limit | Max requests per minute for this tier |
X-RateLimit-Remaining | Requests remaining in current window |
Retry-After | Seconds until window resets (on 429 only) |
Configuration
| Variable | Default |
|---|---|
KRONFORCE_RATE_LIMIT_ENABLED | true |
KRONFORCE_RATE_LIMIT_PUBLIC | 30 |
KRONFORCE_RATE_LIMIT_AUTHENTICATED | 120 |
KRONFORCE_RATE_LIMIT_AGENT | 600 |
Set any limit to 0 to disable that tier. Set KRONFORCE_RATE_LIMIT_ENABLED=false to disable all rate limiting.
Audit Log
The audit log records all sensitive operations with actor attribution. It is separate from events and has its own retention policy.
What's Recorded
| Operation | Resource |
|---|---|
key.created, key.revoked | API keys |
job.created, job.updated, job.deleted, job.triggered | Jobs |
script.saved, script.deleted | Scripts |
settings.updated | Settings |
variable.created, variable.updated, variable.deleted | Variables |
agent.deregistered | Agents |
group.renamed | Groups |
Each entry records: timestamp, actor (API key name + ID), operation, resource type, resource ID, and details.
API (Admin Only)
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/audit-log | List audit entries (paginated) |
Filters: ?operation=job.created, ?actor=deploy-bot, ?since=2026-03-01T00:00:00Z
Retention
Default: 90 days (separate from events/executions). Configure via Settings: audit_retention_days.
MCP Server
Kronforce exposes an MCP (Model Context Protocol) server that AI assistants and MCP clients can connect to. This lets tools like Claude, Cursor, or custom AI agents discover and manage your jobs through the standard MCP protocol.
Connecting
The MCP server is available at POST /mcp using the Streamable HTTP transport. Connect with your API key:
- URL:
http://your-controller:8080/mcp - Auth:
Authorization: Bearer kf_your_key - Headers:
Accept: application/json, text/event-stream
Available Tools
| Tool | Description | Min Role |
|---|---|---|
list_jobs | List jobs (filterable by group, status, search) | Viewer |
get_job | Get job details by name or ID | Viewer |
create_job | Create a new job | Operator |
trigger_job | Trigger a job to run now | Operator |
list_executions | List recent executions | Viewer |
get_execution | Get execution output by ID | Viewer |
list_agents | List registered agents | Viewer |
list_groups | List job groups | Viewer |
list_events | List system events | Viewer |
get_system_stats | System overview stats | Viewer |
Tools are filtered by the API key's role — viewers see read-only tools, operators can create and trigger jobs.
Configuration
The MCP server is enabled by default. Disable with KRONFORCE_MCP_ENABLED=false.
Cron Expressions
6-field cron with second-level precision: sec min hour dom month dow
| Expression | Description |
|---|---|
* * * * * * | Every second |
0 * * * * * | Every minute |
0 0 * * * * | Every hour |
0 0 9 * * * | Daily at 9:00 AM |
0 0 9 * * 1-5 | Weekdays at 9:00 AM |
0 */5 * * * * | Every 5 minutes |
*/30 * * * * * | Every 30 seconds |
Supports: *, ranges (1-5), lists (1,3,5), steps (*/5, 1-30/5).
Day-of-month / day-of-week: follows POSIX OR semantics. When both are set (not *), the job fires if either matches. Example: 0 0 0 15 * 5 fires on the 15th of any month or any Friday.
Schedule Types
| Type | JSON | Description |
|---|---|---|
| One-shot | {"type": "one_shot", "value": "2026-04-01T00:00:00Z"} | Fires once |
| Cron | {"type": "cron", "value": "0 * * * * *"} | Recurring schedule |
| On-demand | {"type": "on_demand"} | Triggered manually only |
| Event | {"type": "event", "value": {...}} | Fires on system events |
| Calendar | {"type": "calendar", "value": {"anchor": "last_day", ...}} | Business-day expressions (last day, nth weekday, skip weekends/holidays) |
| Interval | {"type": "interval", "value": {"interval_secs": 1800}} | Fixed delay from last execution completion |
Event Triggers
Jobs can fire reactively when system events occur. Set a job's schedule to "Event" and configure what to listen for.
Available Event Kinds
| Kind | When it fires |
|---|---|
job.created | A job is created |
job.updated | A job is edited |
job.deleted | A job is deleted |
job.triggered | A job is manually triggered |
execution.completed | A job execution finishes (success or failure) |
output.matched | An output trigger pattern matched stdout/stderr |
agent.registered | An agent registers |
agent.offline | An agent heartbeat times out |
agent.unpaired | An agent is removed |
group.completed | All jobs in a group have succeeded |
key.created | An API key is created |
key.revoked | An API key is revoked |
Trigger Configuration
| Field | Description |
|---|---|
| Event Kind | Which event to listen for. Use wildcards: output.*, job.*, or * for all. |
| Severity | Optional. Only trigger on events with this severity (error, warning, info, success). |
| Source Job | Optional. Only trigger when the event came from a job with this name. Matches as a substring. |
Example: React to Output Patterns
This is the most powerful pattern — chain jobs based on what another job outputs:
- Job A ("health-check"): In the Advanced section, add an Output Trigger with pattern
ERRORand severityerror - Job B ("restart-service"): Set schedule to Event, pick
output.matched, set severity toerror, set Source Job tohealth-check - When Job A runs and its output contains "ERROR", Job B fires automatically
Example: React to Job Failures
- Job B ("cleanup"): Set schedule to Event, pick
execution.completed, set severity toerror, set Source Job toetl-pipeline - Whenever "etl-pipeline" fails, "cleanup" fires automatically
Extending Kronforce
Kronforce doesn't use a plugin system. Instead, it provides composable primitives that cover most use cases. Here's how common Jenkins plugins map to Kronforce patterns:
Jenkins Plugin Equivalents
| Jenkins Plugin | Kronforce Equivalent | How |
|---|---|---|
| Git / SCM | Shell task | git clone, git pull in a shell command |
| Slack Notification | Built-in Slack | Settings → Notifications → Slack webhook URL |
| Email Extension | Built-in SMTP | Settings → Notifications → SMTP config |
| PagerDuty | Built-in PagerDuty | Settings → Notifications → routing key |
| Docker Pipeline | Shell + Dockerfile scripts | Shell task with docker build/run, or Dockerfile script type |
| HTTP Request | HTTP task | GET/POST/PUT/DELETE with headers, body, auth, assertions |
| Database plugins | SQL task + Connections | Named connection (Postgres/MySQL/SQLite) + SQL query |
| SSH / SSH Agent | SSH connection + Shell | Named SSH connection, or shell task with target agent |
| Credentials Binding | Secret variables + Connections | {{MY_SECRET}} substituted at runtime, encrypted at rest |
| Pipeline DSL | Dependencies + Events | Job B depends on Job A; event triggers react to completions |
| Parameterized Build | Job parameters | Define parameters in job config, pass at trigger time |
| Cron Trigger | Cron schedule | 6-field cron expression on any job |
| Webhook Trigger | Built-in webhooks | Enable webhook on a job, trigger via unique URL |
| Approval Gate | Built-in approval | Set approval_required: true on any job |
| Retry / Naginator | Built-in retry | retry_max + retry_delay_secs on any job |
| Build Timeout | Built-in timeout | timeout_secs on any job |
| Prometheus Metrics | Built-in /metrics | Scrape endpoint with job/execution counters |
| LDAP / SAML / OIDC | Built-in OIDC | Okta, Azure AD, Google via OIDC config |
| Custom Build Step | Custom agent | Write a Python/Go/Node agent for any task type |
| gRPC / Protobuf | Custom agent | gRPC agent example in docs |
| ML / GPU Workloads | Custom agent | Run agent on GPU machine, define train-model task type |
| Kubernetes | Shell task or custom agent | kubectl apply in shell, or K8s-native custom agent |
| Terraform / Ansible | Shell task | terraform apply, ansible-playbook as shell commands |
| S3 / Artifact Upload | S3 connection + Shell | Named S3/MinIO connection + aws s3 cp |
Custom Agents: The Extension Point
When built-in task types aren't enough, write a custom agent in any language. A custom agent is just an HTTP client that:
- Registers with the controller (
POST /api/agents/register) - Long-polls for work (
GET /api/agent-queue/{id}/next?wait=30) - Executes the task however it wants
- Posts the result back (
POST {callback_url})
Custom agents define their own task types with UI-configurable fields. For example, a Python ML agent might define a train-model task type with fields for dataset URL, model architecture, and hyperparameters. These appear as form fields in the Builder when creating a job targeting that agent.
See the Custom Agents section for the full protocol and examples.
Rhai Scripts: Inline Logic
For lightweight automation that doesn't need an external agent, use Rhai scripts. Rhai is a sandboxed scripting language with built-in HTTP, JSON, math, and string functions. Scripts run on the controller — no agent needed.
Use cases: health checks, API polling, data validation, conditional logic, multi-step HTTP workflows.
See the Scripting section for the API and examples.
Output Rules: Reactive Workflows
Jobs can extract values from their output and trigger downstream actions:
- Extraction — capture regex matches or JSON paths from stdout into variables
- Assertions — fail a job if output doesn't match expected patterns
- Triggers — fire other jobs based on output content
- Forward — POST output to an external URL (webhook relay)
Combined with event triggers and dependencies, these compose into arbitrarily complex workflows without plugins.
MCP Server: AI Tool Integration
Kronforce includes a built-in MCP server that exposes jobs, executions, and agents as tools for AI assistants (Claude, etc.). This lets AI agents trigger jobs, check statuses, and read output programmatically.
API: Build Anything
Every feature in Kronforce is accessible via the REST API. Build custom dashboards, CLI tools, Slack bots, or integrations with any system that speaks HTTP. The API supports pagination, filtering, and real-time streaming (SSE).
From Cron
Use the built-in crontab importer to convert your cron entries automatically:
crontab -l | kronforce-import-crontab kf_your_key
kronforce-import-crontab kf_your_key --group Monitoring < /etc/cron.d/jobs
kronforce-import-crontab kf_your_key --dry-run < mycrontab
The importer converts 5-field cron to 6-field (adds seconds), handles @daily/@hourly shortcuts, and generates job names from commands. Find it at scripts/kronforce-import-crontab.
Manual conversion: prepend 0 for the seconds field. */5 * * * * becomes 0 */5 * * * *.
From Jenkins
Import Jenkinsfiles and Jenkins config.xml with the Jenkins importer:
kronforce-import-jenkins kf_your_key Jenkinsfile --pipeline
kronforce-import-jenkins kf_your_key config.xml --group CI-CD
kronforce-import-jenkins kf_your_key ./jenkins-jobs/ --pipeline --dry-run
The --pipeline flag wires stages as a Kronforce dependency chain — trigger the first stage and the rest cascade automatically. The importer handles:
- Declarative pipeline stages with
sh/batsteps - Cron triggers (converts Jenkins
Hsyntax) - Agent labels → agent target tags
- Retry and timeout settings
- Freestyle job config.xml
- Bulk import from
jobs/*/config.xmldirectories
Find it at scripts/kronforce-import-jenkins.
From Airflow / Rundeck
No automated importer (too many format variants), but the mapping is straightforward:
| Concept | Kronforce Equivalent |
|---|---|
| DAG / Job Group | Group + dependency chain + pipeline schedule |
| BashOperator / Script Step | Shell task |
| HttpOperator / HTTP Step | HTTP task |
| XCom / Key Storage | Variables + output extraction ({{VAR}}) |
| Connections / Secrets | Secret variables |
| RBAC / ACL | API key roles + OIDC SSO |
See the full Migration Guide for detailed mapping tables and step-by-step instructions.