File Uploads (v0.2.0)

libscadable v0.2.0 adds streaming file upload from devices. Customer code calls one of four functions; the library handles auth, TLS, chunked transfer encoding, and parsing the server response. Compile-time opt-in — when the feature is off (default), it adds zero code, zero RAM, zero TLS work.

Opt in via Kconfig

idf.py menuconfig
  → Component config
    → SCADABLE Gateway
      → [*] Enable file upload feature

Or in sdkconfig.defaults:

CONFIG_SCD_UPLOAD_ENABLE=y

When off, the four scadable_upload_* symbols don't ship in your binary. When on, you can pull files of any size up to the per-request cap (50 MB by default).

Public API

#include "scadable.h"

#if defined(CONFIG_SCD_UPLOAD_ENABLE)

typedef struct scd_upload* scd_upload_handle_t;

esp_err_t scadable_upload_begin(const char *filename,
                                const char *content_type,
                                scd_upload_handle_t *out);
esp_err_t scadable_upload_chunk(scd_upload_handle_t h,
                                const void *bytes, size_t len);
esp_err_t scadable_upload_end  (scd_upload_handle_t h,
                                char *file_id_out, size_t file_id_max);
void      scadable_upload_abort(scd_upload_handle_t h);

#endif

Pattern: begin → chunk × N → end. On any error from chunk, call abort to free the handle. end finalizes the request and parses the server-assigned file_id out of the response.

Minimal customer code

#include "scadable.h"

void upload_crashdump(void) {
    scd_upload_handle_t up;
    if (scadable_upload_begin("crash-0001.bin",
                              "application/octet-stream",
                              &up) != ESP_OK) return;

    uint8_t buf[4096];
    size_t  n;
    while (read_crashdump_chunk(buf, sizeof(buf), &n)) {
        if (scadable_upload_chunk(up, buf, n) != ESP_OK) {
            scadable_upload_abort(up);
            return;
        }
    }

    char file_id[64];
    if (scadable_upload_end(up, file_id, sizeof(file_id)) == ESP_OK) {
        ESP_LOGI("crash", "uploaded as %s", file_id);
    }
}

One-shot for an in-memory buffer:

scd_upload_handle_t up;
scadable_upload_begin("hello.txt", "text/plain", &up);
scadable_upload_chunk(up, "hello world", 11);
scadable_upload_end(up, NULL, 0);

Auth — automatic, via the device cert

The library reads the device certificate's Common Name from NVS at boot and sends it as X-Device-CN: SC-<device_id> on every upload. The server resolves the CN through the provisioning record to find the device's namespace and org, and stamps all three on the stored metadata.

You don't pass org / namespace / device IDs from firmware — they're derived server-side. The whole identity story for uploads is: "the device cert says who you are, and the server figures out where the file belongs."

Devices that aren't provisioned: 403. Devices outside an org-owned namespace: 403.

Wire protocol (for testing without the library)

PUT https://uploads.scadable.com/v1/upload HTTP/1.1
X-Device-CN:       SC-a3f8b2c4e6d8
X-Filename:        crash-0001.bin
X-Content-Type:    application/octet-stream
Transfer-Encoding: chunked

<chunked body>

HTTP/1.1 200 OK
{
  "file_id":      "f_...",
  "org_id":       "org_...",
  "namespace_id": "ns_...",
  "device_cn":    "SC-...",
  "filename":     "crash-0001.bin",
  "content_type": "application/octet-stream",
  "uploaded_at":  1748678400,
  "size_bytes":   524288,
  "sha256":       "..."
}

Equivalent curl (handy for debugging):

curl -i -X PUT \
  -H "X-Device-CN: SC-<your_device_id>" \
  -H "X-Filename: crash-0001.bin" \
  -H "X-Content-Type: application/octet-stream" \
  --data-binary @./crash-0001.bin \
  https://uploads.scadable.com/v1/upload

Listing uploads from the dashboard

GET /api/orgs/{org_id}/uploads
  ?limit=<n>                  (default 50, max 200)
  &ns=<ns_id>                 (optional — narrow to one namespace)
  &device=<cn>                (optional — narrow to one device)
  &from=<unix>&to=<unix>      (optional — time range; default last 7 days)

Returns a newest-first array of upload metadata (no bytes — separate fetch endpoint arrives in v0.3.0).

GET /api/orgs/{org_id}/uploads/stats

Returns { count, bytes_used, bytes_quota, last_upload_at } for a dashboard tile.

Limits

Exceeding per-request → 413 Payload Too Large. Exceeding per-org → 507 Insufficient Storage.

Memory footprint

Adding CONFIG_SCD_UPLOAD_ENABLE=y:

You only pay the per-upload cost while an upload is in flight; closing the handle returns it.

Storage today, S3 tomorrow

v0.2.0 stores file bytes directly in Valkey:

scadable:upload:{file_id}              hash    metadata
scadable:upload:{file_id}:bytes        string  content
scadable:org:{org_id}:uploads          zset    score=uploaded_at, member=file_id
scadable:ns:{ns_id}:uploads            zset    same
scadable:device:{cn}:uploads           zset    same
scadable:org:{org_id}:upload_bytes_used integer (quota counter)

v0.3.0+ swaps the storage layer to S3 (likely customer-owned buckets). The device API doesn't change — only the backend implementation behind uploads.scadable.com/v1/upload does. Customers running v0.2.0 keep working without re-flashing.

What's NOT in v0.2.0

• Download/fetch from device. Upload is one-way (device → cloud).
• Resumable uploads. A drop mid-upload starts over.
• Concurrent uploads per device. Pattern is begin/chunk/end serially.
• Server-side decryption. Files are stored as-sent. v0.3.0 may add envelope encryption if there's demand for at-rest application-level protection (today the storage layer encrypts at rest at the infra level).

Related

• API reference — the rest of the v0.1.0 library surface.
• Quickstart — first device onboarding.
• Troubleshooting — upload-related error messages.