We've also launched a bug bounty program and Veria Labs is continuing to audit the proxy. More fixes will ship in upcoming versions.

The two high-severity issues (CVE-2026-35029 and GHSA-69x8-hrgq-fjj8) both require the attacker to already have a valid API key for the proxy. These are not exploitable by unauthenticated users.

The critical-severity issue (CVE-2026-35030) is an authentication bypass, but only affects deployments with enable_jwt_auth explicitly enabled, which is off by default. The default LiteLLM configuration is not affected, and no LiteLLM Cloud customers had this feature enabled.

Vulnerabilities​

CVE-2026-35030: Authentication bypass via OIDC cache collision (Critical)​

Found by Veria Labs.

When enable_jwt_auth is enabled, LiteLLM cached OIDC userinfo using token[:20] as the cache key. JWTs from the same signing algorithm share the same header prefix, so an attacker could forge a token that hits another user's cache entry and inherit their session. We fixed this by keying the cache on sha256(token) instead.

Most deployments are not affected. This requires enable_jwt_auth: true, which is off by default. If you can't upgrade, disable JWT auth as a workaround.

Full advisory: GHSA-jjhc-v7c2-5hh6

CVE-2026-35029: Privilege escalation via /config/update (High)​

Found by Lakera.

/config/update didn't check the caller's role. Any authenticated user could modify the proxy's runtime configuration, which could lead to arbitrary file read, admin account takeover, or remote code execution. We now require the proxy_admin role on this endpoint.

Full advisory: GHSA-53mr-6c8q-9789

Password hash exposure and pass-the-hash login (High)​

Weak hashing originally reported by GitHub user hamzayevmaqsud (#15484). The full chain was identified by Luca Vandenweghe and Maarten De Rammelaere of iO Digital.

Passwords were stored as unsalted SHA-256 hashes, and in some cases plaintext. Several API endpoints returned the hash to any authenticated user, and /v2/login accepted the raw hash as a credential without re-hashing it, so a stolen hash was as good as the password itself. We've moved to scrypt with random salts and stripped hashes from all API responses.

Full advisory: GHSA-69x8-hrgq-fjj8

Bug bounty program​

After the supply chain incident and these disclosures it was clear we needed more external eyes on the project. We've set up a bug bounty program so researchers have a way to report issues.

Bounties are currently paid for P0 (supply chain) and P1 (unauthenticated proxy access) vulnerabilities:

We plan on expanding the program further in the coming months. More info about the bug bounty program is available here.

What's next​

Veria Labs is continuing to work with us on a broader audit of the proxy. Security advisories sent through Github will be responded to within five business days. We'll publish advisories as issues are confirmed and fixed.

Agenda​

• Product updates and roadmap progress
• Reliability and security updates
• Open Q&A with the team

How to contribute​

Add your thoughts to this ticket to help us shape the agenda.

Register​

Register here: LiteLLM April Townhall Form

We will hold the townhall from 7:30 AM to 8:30 AM PST on Zoom.

For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.

Building on the roadmap from our security incident, CI/CD v2 introduces isolated environments, stronger security gates, and safer release separation for LiteLLM.

What changed​

• Security scans and unit tests run in isolated environments.
• Validation and release are separated into different repositories, making it harder for an attacker to reach release credentials.
• Trusted Publishing for PyPI releases - this means no long-lived credentials are used to publish releases.
• Immutable Docker release tags - this means no tampering of Docker release tags after they are published Learn more. Note: work for GHCR docker releases is planned as well.
• Docker image signing with Cosign - all release images are signed so users can independently verify they came from us.

Verify Docker image signatures​

Starting from v1.83.0-nightly, all LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Expected output:

The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

What's next​

Moving forward, we plan on:

• Adopting OpenSSF (this is a set of security criteria that projects should meet to demonstrate a strong security posture - Learn more)

  • We've added Scorecard and Allstar to our Github

• Adding SLSA Build Provenance to our CI/CD pipeline - this means we allow users to independently verify that a release came from us and prevent silent modifications of releases after they are published.

We hope that this will mean you can be confident that the releases you are using are safe and from us.

The principle​

The new CI/CD pipeline reflects the principles, outlined below, and is designed to be more secure and reliable:

• Limit what each package can access
• Reduce the number of sensitive environment variables
• Avoid compromised packages
• Prevent release tampering

How to help:​

Help us plan April's stability sprint - https://github.com/BerriAI/litellm/issues/24825

We are partnering with Vanta to recertify LiteLLM's compliance for SOC 2 Type 2 and ISO 27001.

As part of this process, we are also identifying independent auditors to validate and verify our compliance posture.

This is part of our commitment to being the most secure and transparent AI Gateway possible.

We wanted to use that time to walk through what we know, what we've done so far, and how we're improving LiteLLM's release and security processes going forward. This post is a written version of that update. Slides available here

What happened​

On March 24, 2026 at 10:39 UTC, LiteLLM v1.82.7 was pushed to PyPI. Version v1.82.8 was published soon after. Those packages were live for about 40 minutes before being quarantined by PyPI. By 16:00 UTC, the LiteLLM team had worked with PyPI to delete the affected packages.

At this point, our understanding is that this was a supply-chain incident affecting those two published versions.

How did this happen?​

Our understanding is that the issue came from the compromised Trivy security scanner dependency in our CI/CD pipeline.

There were three major contributing factors:

1. Shared CI/CD environment​

At the time, everything was running on CircleCI, and all steps shared a common environment. That increased blast radius: if one component was compromised, it could potentially access credentials or context intended for other parts of the pipeline.

2. Static credentials in environment variables​

Release credentials, including credentials for PyPI, GHCR, and Docker publishing, were available as static secrets in the environment. That meant a compromised step could access long-lived release credentials.

3. Unpinned Trivy dependency​

In our security scanning component, we had an unpinned Trivy dependency. Our present understanding is that a compromised Trivy package ran during the scan, had access to environment variables, and enabled attackers to obtain those credentials.

In summary: a compromised package in CI had access to secrets it should not have had, and those secrets were then used in the release path.

What we've already done​

In the last 3 days, we've taken the following steps:

1. Minimize Scope of Impact​

Prevented further key abuse​

We deleted or rotated all impacted or adjacent secret keys, including PyPI, GitHub, Docker, and related credentials. Out of an abundance of caution, we've also rotated LiteLLM maintainer accounts.

Prevent branch attacks​

We removed roughly 6,000 open branches and added an auto-deletion policy for branches merged into main. This reduces the surface area for branch-based abuse.

Pinned CI/CD dependencies​

We've pinned all Github Actions, and are working on pinning all CircleCI dependencies as well.

Paused releases​

We've paused new releases until we've confirmed codebase security and put stronger release controls in place.

2. Secured LiteLLM​

Forensic analysis​

We are working with Google's Mandiant cybersecurity team to confirm the source of the attack and verify the security of the codebase. We also confirmed that no malicious code was pushed to main.

Confirm Application Security​

In parallel, we are working with whitehat hackers at Veria Labs to verify application security and review improvements to our CI/CD process.

We have also confirmed that the last 20 LiteLLM releases contain no indicators of compromise, and that no unauthenticated attacks can be made against LiteLLM Proxy based on our current investigation. Check Security Blog for release verification.

Created a security working group​

We created a new security working group inside LiteLLM focused on:

• Building threat models
• Auditing the build process and dependencies

If you're interested in joining the security working group, please file an issue here.

3. Improved CI/CD​

We've already begun making structural changes to how releases are built and published. These align with our goals (covered in the next section) around isolated environments, ephemeral credentials, and release auditing.

Roadmap​

We plan on following 4 guiding principles for our new CI/CD pipeline:

1. Limit what each package can access
2. Reduce the number of sensitive environment variables
3. Avoid compromised packages
4. Prevent release tampering

Isolated environments​

We are breaking our CI/CD into 4 semantic concepts:

1. Unit tests
2. Integration tests
3. Security scans
4. Release publishing

And will be running each of these in isolated environments.

This will limit the damage that any single compromised component can cause.

Ephemeral credentials​

We plan to move to ephemeral credentials for PyPI (Trusted Publisher) and GHCR (Token-based authentication) releases. This will reduce the risk of credentials being leaked or compromised.

We have already begun doing this:

• PyPI Trusted Publisher on GitHub Actions PR
• GHCR Token-based authentication on GitHub Actions PR

Release auditing​

Our goal is to allow users to independently verify that a release came from us and prevent silent modifications of releases after they are published.

This will ensure, your releases are safe, even when:

• Stolen PyPI/GHCR credentials are used to publish malicious releases
• Tampered registry artifacts are published
• Tag mutations are made after the release is published

We believe that Cosign is a good fit for this, and have shipped it in PR #24683.

How to verify a Docker image with Cosign​

Starting from v1.83.0-nightly, all LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key that was introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Expected output:

The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Avoid Compromised Packages​

• Move to pinned, verified SHAs for packages and actions used in CI/CD, avoiding latest wherever possible.
• Add a cooldown period before upgrading to a new version of a package - allows more time to investigate and verify the new version.

We've added zizmor to help us catch issues such as unpinned dependencies and credential leakage. commit.

Frequently Asked Questions​

Q: Did you observe any lateral movement into your corporate environment during this incident?

A: No. Our investigation to date, conducted in coordination with external security experts, has found no evidence of lateral movement into our internal corporate systems. The incident was isolated to the CI/CD pipeline and the release path for specific versions (v1.82.7 and v1.82.8). As a proactive measure, we have rotated all potentially impacted or adjacent secrets—including PyPI, GitHub, and Docker credentials—and updated maintainer account security to ensure continued isolation.

Q: Do you expect delays in future product releases due to these new security measures?

A: We are committed to balancing security with speed. While we have temporarily paused releases to implement stronger controls, we are moving quickly to automate our new security protocols. We are currently implementing isolated CI/CD environments, ephemeral credentials (via Trusted Publishers), and release auditing with Cosign. These improvements are designed to be integrated into our automated pipeline, allowing us to maintain a fast release cadence while ensuring every package is verified and secure.

Q: Were older packages impacted?

Our current findings show no indicators of compromise in the last 20 versions of LiteLLM. This was manually verified by our team and independently reviewed by Veria Labs.

We have also published the verified versions for users to use. Check Security Blog for release verification.

Questions & Support​

If you believe your systems may be affected, contact us immediately:

• Security: security@berri.ai
• Support: support@berri.ai
• Slack: Reach out to the LiteLLM team directly here

Hiring​

We are currently hiring for:

• DevOps Engineer - to keep ci/cd secure and running smoothly
• Security Engineer - to keep the application secure

If you're interest in joining, please apply here

Status: Active investigation Last updated: March 27, 2026

Update (March 30): A new clean version of LiteLLM is now available (v1.83.0). This was released by our new CI/CD v2 pipeline which added isolated environments, stronger security gates, and safer release separation for LiteLLM.

Update (March 27): Review Townhall updates, including explanation of the incident, what we've done, and what comes next. Learn more

Update (March 27): Added Verified safe versions section with SHA-256 checksums for all audited PyPI and Docker releases.

Update (March 26): Added checkmarx[.]zone to Indicators of compromise

Update (March 25): Added community-contributed scripts for scanning GitHub Actions and GitLab CI pipelines for the compromised versions. See How to check if you are affected. s/o @Zach Fury for these scripts.

TLDR;​

• The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages were live on March 24, 2026 from 10:39 UTC for about 40 minutes before being quarantined by PyPI.
• We believe that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow.
• Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages.
• We have paused all new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe. Updated: We have now released a new safe version of LiteLLM (v1.83.0) by our new CI/CD v2 pipeline which added isolated environments, stronger security gates, and safer release separation for LiteLLM. We have also verified the codebase is safe and no malicious code was pushed to main.

Overview​

LiteLLM AI Gateway is investigating a suspected supply chain attack involving unauthorized PyPI package publishes. Current evidence suggests a maintainer's PyPI account may have been compromised and used to distribute malicious code.

At this time, we believe this incident may be linked to the broader Trivy security compromise, in which stolen credentials were reportedly used to gain unauthorized access to the LiteLLM publishing pipeline.

This investigation is ongoing. Details below may change as we confirm additional findings.

Confirmed affected versions​

The following LiteLLM versions published to PyPI were impacted:

• v1.82.7: contained a malicious payload in the LiteLLM AI Gateway proxy_server.py
• v1.82.8: contained litellm_init.pth and a malicious payload in the LiteLLM AI Gateway proxy_server.py

If you installed or ran either of these versions, review the recommendations below immediately.

Note: These versions have already been removed from PyPI.

What happened​

Initial evidence suggests the attacker bypassed official CI/CD workflows and uploaded malicious packages directly to PyPI.

These compromised versions appear to have included a credential stealer designed to:

• Harvest secrets by scanning for:
  • environment variables
  • SSH keys
  • cloud provider credentials (AWS, GCP, Azure)
  • Kubernetes tokens
  • database passwords
• Encrypt and exfiltrate data via a POST request to models.litellm.cloud, which is not an official BerriAI / LiteLLM domain

Who is affected​

You may be affected if any of the following are true:

• You installed or upgraded LiteLLM via pip on March 24, 2026, between 10:39 UTC and 16:00 UTC
• You ran pip install litellm without pinning a version and received v1.82.7 or v1.82.8
• You built a Docker image during this window that included pip install litellm without a pinned version
• A dependency in your project pulled in LiteLLM as a transitive, unpinned dependency (for example through AI agent frameworks, MCP servers, or LLM orchestration tools)

You are not affected if any of the following are true:

LiteLLM AI Gateway/Proxy users: Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages.

• You are using LiteLLM Cloud
• You are using the official LiteLLM AI Gateway Docker image: ghcr.io/berriai/litellm
• You are on v1.82.6 or earlier and did not upgrade during the affected window
• You installed LiteLLM from source via the GitHub repository, which was not compromised

How to check if you are affected​

pip show litellm

export GITHUB_TOKEN="your-github-pat"

python find_litellm_github.py

#!/usr/bin/env python3
"""
Scan all GitHub Actions jobs in a GitHub org that ran between
0800-1244 UTC today and identify any that installed litellm 1.82.7 or 1.82.8.

Adjust WINDOW_START / WINDOW_END to cover March 24, 2026 if running later.
"""

import io
import os
import re
import sys
import zipfile
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone

import requests

GITHUB_URL   = "https://api.github.com"
ORG          = "your-org"  # <-- set to your GitHub organization
TOKEN        = os.environ.get("GITHUB_TOKEN", "")

TODAY        = datetime.now(timezone.utc).date()
WINDOW_START = datetime(TODAY.year, TODAY.month, TODAY.day,  8,  0, 0, tzinfo=timezone.utc)
WINDOW_END   = datetime(TODAY.year, TODAY.month, TODAY.day, 12, 44, 0, tzinfo=timezone.utc)

TARGET_VERSIONS = {"1.82.7", "1.82.8"}
VERSION_PATTERN = re.compile(r"litellm[=\-](\d+\.\d+\.\d+)", re.IGNORECASE)

SESSION = requests.Session()
SESSION.headers.update({
    "Authorization": f"Bearer {TOKEN}",
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
})


def get_paginated(url, params=None):
    params = dict(params or {})
    params.setdefault("per_page", 100)
    page = 1
    while True:
        params["page"] = page
        resp = SESSION.get(url, params=params, timeout=30)
        if resp.status_code == 404:
            return
        resp.raise_for_status()
        data = resp.json()
        if isinstance(data, dict):
            items = next((v for v in data.values() if isinstance(v, list)), [])
        else:
            items = data
        if not items:
            break
        yield from items
        if len(items) < params["per_page"]:
            break
        page += 1


def parse_ts(ts_str):
    if not ts_str:
        return None
    return datetime.fromisoformat(ts_str.replace("Z", "+00:00"))


def get_repos():
    repos = []
    for r in get_paginated(f"{GITHUB_URL}/orgs/{ORG}/repos", {"type": "all"}):
        repos.append({"id": r["id"], "name": r["name"], "full_name": r["full_name"]})
    return repos


def get_runs_in_window(repo_full_name):
    created_filter = (
        f"{WINDOW_START.strftime('%Y-%m-%dT%H:%M:%SZ')}"
        f"..{WINDOW_END.strftime('%Y-%m-%dT%H:%M:%SZ')}"
    )
    url = f"{GITHUB_URL}/repos/{repo_full_name}/actions/runs"
    runs = []
    for run in get_paginated(url, {"created": created_filter, "per_page": 100}):
        ts = parse_ts(run.get("run_started_at") or run.get("created_at"))
        if ts and WINDOW_START <= ts <= WINDOW_END:
            runs.append(run)
    return runs


def get_jobs_for_run(repo_full_name, run_id):
    url = f"{GITHUB_URL}/repos/{repo_full_name}/actions/runs/{run_id}/jobs"
    jobs = []
    for job in get_paginated(url, {"filter": "all"}):
        ts = parse_ts(job.get("started_at"))
        if ts and WINDOW_START <= ts <= WINDOW_END:
            jobs.append(job)
    return jobs


def fetch_job_log(repo_full_name, job_id):
    url = f"{GITHUB_URL}/repos/{repo_full_name}/actions/jobs/{job_id}/logs"
    resp = SESSION.get(url, timeout=60, allow_redirects=True)
    if resp.status_code in (403, 404, 410):
        return ""
    resp.raise_for_status()

    content_type = resp.headers.get("Content-Type", "")
    if "zip" in content_type or resp.content[:2] == b"PK":
        try:
            with zipfile.ZipFile(io.BytesIO(resp.content)) as zf:
                parts = []
                for name in sorted(zf.namelist()):
                    with zf.open(name) as f:
                        parts.append(f.read().decode("utf-8", errors="replace"))
                return "\n".join(parts)
        except zipfile.BadZipFile:
            pass
    return resp.text


def check_job(repo_full_name, job):
    job_id   = job["id"]
    job_name = job["name"]
    run_id   = job["run_id"]
    started  = job.get("started_at", "")

    log_text = fetch_job_log(repo_full_name, job_id)
    if not log_text:
        return None

    found_versions = set()
    context_lines  = []
    for line in log_text.splitlines():
        m = VERSION_PATTERN.search(line)
        if m:
            ver = m.group(1)
            if ver in TARGET_VERSIONS:
                found_versions.add(ver)
                context_lines.append(line.strip())

    if not found_versions:
        return None

    return {
        "repo":       repo_full_name,
        "run_id":     run_id,
        "job_id":     job_id,
        "job_name":   job_name,
        "started_at": started,
        "versions":   sorted(found_versions),
        "context":    context_lines[:10],
        "job_url":    job.get("html_url", f"https://github.com/{repo_full_name}/actions/runs/{run_id}"),
    }


def main():
    if not TOKEN:
        print("ERROR: Set GITHUB_TOKEN environment variable.", file=sys.stderr)
        sys.exit(1)

    print(f"Time window : {WINDOW_START.isoformat()} -> {WINDOW_END.isoformat()}")
    print(f"Hunting for : litellm {', '.join(sorted(TARGET_VERSIONS))}")
    print()

    print(f"Fetching repositories for org '{ORG}'...")
    repos = get_repos()
    print(f"  Found {len(repos)} repositories")
    print()

    jobs_to_check = []

    print("Scanning workflow runs for time window...")
    for repo in repos:
        full_name = repo["full_name"]
        try:
            runs = get_runs_in_window(full_name)
        except requests.HTTPError as e:
            print(f"  WARN: {full_name} - {e}", file=sys.stderr)
            continue
        if not runs:
            continue
        print(f"  {full_name}: {len(runs)} run(s) in window")
        for run in runs:
            try:
                jobs = get_jobs_for_run(full_name, run["id"])
            except requests.HTTPError as e:
                print(f"    WARN: run {run['id']} - {e}", file=sys.stderr)
                continue
            for job in jobs:
                jobs_to_check.append((full_name, job))

    total = len(jobs_to_check)
    print(f"\nFetching logs for {total} job(s)...")
    print()

    hits = []
    with ThreadPoolExecutor(max_workers=8) as pool:
        futures = {
            pool.submit(check_job, full_name, job): (full_name, job["id"])
            for full_name, job in jobs_to_check
        }
        done = 0
        for future in as_completed(futures):
            done += 1
            full_name, jid = futures[future]
            try:
                result = future.result()
            except Exception as e:
                print(f"  ERROR {full_name} job {jid}: {e}", file=sys.stderr)
                continue
            if result:
                hits.append(result)
            print(
                f"  [{done}/{total}] {full_name} job {jid}" +
                (f"  *** HIT: litellm {result['versions']} ***" if result else ""),
                flush=True,
            )

    print()
    print("=" * 72)
    print(f"RESULTS: {len(hits)} job(s) installed litellm {' or '.join(sorted(TARGET_VERSIONS))}")
    print("=" * 72)

    if not hits:
        print("No matches found.")
        return

    for h in sorted(hits, key=lambda x: x["started_at"]):
        print()
        print(f"  Repo      : {h['repo']}")
        print(f"  Job       : {h['job_name']} (#{h['job_id']})")
        print(f"  Run ID    : {h['run_id']}")
        print(f"  Started   : {h['started_at']}")
        print(f"  Versions  : litellm {', '.join(h['versions'])}")
        print(f"  URL       : {h['job_url']}")
        print(f"  Log lines :")
        for line in h["context"]:
            print(f"    {line}")


if __name__ == "__main__":
    main()

export GITLAB_TOKEN="your-gitlab-pat"

python find_litellm_jobs.py

#!/usr/bin/env python3
"""
Scan all GitLab CI/CD jobs in a GitLab group that ran between
0800-1244 UTC today and identify any that installed litellm 1.82.7 or 1.82.8.

Adjust WINDOW_START / WINDOW_END to cover March 24, 2026 if running later.
"""

import os
import re
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone

import requests

GITLAB_URL = "https://gitlab.com"
GROUP_NAME = "YourGroup"  # <-- set to your GitLab group name
TOKEN = os.environ.get("GITLAB_TOKEN", "")

TODAY = datetime.now(timezone.utc).date()
WINDOW_START = datetime(TODAY.year, TODAY.month, TODAY.day, 8, 0, 0, tzinfo=timezone.utc)
WINDOW_END   = datetime(TODAY.year, TODAY.month, TODAY.day, 12, 44, 0, tzinfo=timezone.utc)

TARGET_VERSIONS = {"1.82.7", "1.82.8"}
VERSION_PATTERN = re.compile(r"litellm[=\-](\d+\.\d+\.\d+)", re.IGNORECASE)

HEADERS = {"PRIVATE-TOKEN": TOKEN}
SESSION = requests.Session()
SESSION.headers.update(HEADERS)


def get_paginated(url, params=None):
    params = dict(params or {})
    params.setdefault("per_page", 100)
    page = 1
    while True:
        params["page"] = page
        resp = SESSION.get(url, params=params, timeout=30)
        resp.raise_for_status()
        data = resp.json()
        if not data:
            break
        yield from data
        if len(data) < params["per_page"]:
            break
        page += 1


def get_group_id(group_name):
    resp = SESSION.get(f"{GITLAB_URL}/api/v4/groups/{group_name}", timeout=30)
    resp.raise_for_status()
    return resp.json()["id"]


def get_all_projects(group_id):
    projects = []
    for p in get_paginated(
        f"{GITLAB_URL}/api/v4/groups/{group_id}/projects",
        {"include_subgroups": "true", "archived": "false"},
    ):
        projects.append({"id": p["id"], "name": p["path_with_namespace"]})
    return projects


def parse_ts(ts_str):
    if not ts_str:
        return None
    ts_str = ts_str.replace("Z", "+00:00")
    return datetime.fromisoformat(ts_str)


def jobs_in_window(project_id):
    matching = []
    url = f"{GITLAB_URL}/api/v4/projects/{project_id}/jobs"
    params = {"per_page": 100, "scope[]": ["success", "failed", "canceled", "running"]}

    page = 1
    while True:
        params["page"] = page
        resp = SESSION.get(url, params=params, timeout=30)
        if resp.status_code == 403:
            return matching
        resp.raise_for_status()
        jobs = resp.json()
        if not jobs:
            break

        stop_early = False
        for job in jobs:
            ts = parse_ts(job.get("started_at") or job.get("created_at"))
            if ts is None:
                continue
            if ts > WINDOW_END:
                continue
            if ts < WINDOW_START:
                stop_early = True
                continue
            matching.append(job)

        if stop_early or len(jobs) < 100:
            break
        page += 1

    return matching


def fetch_trace(project_id, job_id):
    url = f"{GITLAB_URL}/api/v4/projects/{project_id}/jobs/{job_id}/trace"
    resp = SESSION.get(url, timeout=60)
    if resp.status_code in (403, 404):
        return ""
    resp.raise_for_status()
    return resp.text


def check_job(project_name, project_id, job):
    job_id   = job["id"]
    job_name = job["name"]
    ref      = job.get("ref", "")
    started  = job.get("started_at", job.get("created_at", ""))

    trace = fetch_trace(project_id, job_id)
    if not trace:
        return None

    found_versions = set()
    for match in VERSION_PATTERN.finditer(trace):
        ver = match.group(1)
        if ver in TARGET_VERSIONS:
            found_versions.add(ver)

    if not found_versions:
        return None

    context_lines = []
    for line in trace.splitlines():
        if VERSION_PATTERN.search(line):
            ver_match = VERSION_PATTERN.search(line)
            if ver_match and ver_match.group(1) in TARGET_VERSIONS:
                context_lines.append(line.strip())

    return {
        "project":    project_name,
        "project_id": project_id,
        "job_id":     job_id,
        "job_name":   job_name,
        "ref":        ref,
        "started_at": started,
        "versions":   sorted(found_versions),
        "context":    context_lines[:10],
        "job_url":    f"{GITLAB_URL}/{project_name}/-/jobs/{job_id}",
    }


def main():
    if not TOKEN:
        print("ERROR: Set GITLAB_TOKEN environment variable.", file=sys.stderr)
        sys.exit(1)

    print(f"Time window : {WINDOW_START.isoformat()} -> {WINDOW_END.isoformat()}")
    print(f"Hunting for : litellm {', '.join(sorted(TARGET_VERSIONS))}")
    print()

    print(f"Resolving group '{GROUP_NAME}'...")
    group_id = get_group_id(GROUP_NAME)

    print("Fetching projects...")
    projects = get_all_projects(group_id)
    print(f"  Found {len(projects)} projects")
    print()

    all_jobs_to_check = []

    print("Scanning job listings for time window...")
    for proj in projects:
        try:
            jobs = jobs_in_window(proj["id"])
        except requests.HTTPError as e:
            print(f"  WARN: {proj['name']} - {e}", file=sys.stderr)
            continue
        if jobs:
            print(f"  {proj['name']}: {len(jobs)} job(s) in window")
        for j in jobs:
            all_jobs_to_check.append((proj["name"], proj["id"], j))

    total = len(all_jobs_to_check)
    print(f"\nFetching traces for {total} job(s)...")
    print()

    hits = []
    with ThreadPoolExecutor(max_workers=10) as pool:
        futures = {
            pool.submit(check_job, pname, pid, job): (pname, job["id"])
            for pname, pid, job in all_jobs_to_check
        }
        done = 0
        for future in as_completed(futures):
            done += 1
            pname, jid = futures[future]
            try:
                result = future.result()
            except Exception as e:
                print(f"  ERROR checking {pname} job {jid}: {e}", file=sys.stderr)
                continue
            if result:
                hits.append(result)
            print(f"  [{done}/{total}] checked {pname} job {jid}" +
                  (f"  *** HIT: litellm {result['versions']} ***" if result else ""),
                  flush=True)

    print()
    print("=" * 72)
    print(f"RESULTS: {len(hits)} job(s) installed litellm {' or '.join(sorted(TARGET_VERSIONS))}")
    print("=" * 72)

    if not hits:
        print("No matches found.")
        return

    for h in sorted(hits, key=lambda x: x["started_at"]):
        print()
        print(f"  Project   : {h['project']}")
        print(f"  Job       : {h['job_name']} (#{h['job_id']})")
        print(f"  Branch/tag: {h['ref']}")
        print(f"  Started   : {h['started_at']}")
        print(f"  Versions  : litellm {', '.join(h['versions'])}")
        print(f"  URL       : {h['job_url']}")
        print(f"  Log lines :")
        for line in h["context"]:
            print(f"    {line}")


if __name__ == "__main__":
    main()

CI/CD scripts contributed by the community (original gist). Review before running.

Indicators of compromise (IoCs)​

Review affected systems for the following indicators:

• litellm_init.pth present in your site-packages
• Outbound traffic or requests to models.litellm[.]cloud This domain is not affiliated with LiteLLM
• Outbound traffic or requests to checkmarx[.]zone This domain is not affiliated with LiteLLM

Immediate actions for affected users​

If you installed or ran v1.82.7 or v1.82.8, take the following actions immediately.

1. Rotate all secrets​

Treat any credentials present on the affected systems as compromised, including:

• API keys
• Cloud access keys
• Database passwords
• SSH keys
• Kubernetes tokens
• Any secrets stored in environment variables or configuration files

2. Inspect your filesystem​

Check your site-packages directory for a file named litellm_init.pth:

find /usr/lib/python3.13/site-packages/ -name "litellm_init.pth"

If present:

• remove it immediately
• investigate the host for further compromise
• preserve relevant artifacts if your security team is performing forensics

3. Audit version history​

Review your:

• Local environments
• CI/CD pipelines
• Docker builds
• Deployment logs

Confirm whether v1.82.7 or v1.82.8 was installed anywhere.

Pin LiteLLM to a known safe version such as v1.82.6 or earlier, or to a later verified release once announced.

Response and remediation​

The LiteLLM AI Gateway team has already taken the following steps:

• Removed compromised packages from PyPI
• Rotated maintainer credentials and established new authorized maintainers
• Engaged Google's Mandiant security team to assist with forensic analysis of the build and publishing chain

Verify Docker image signatures​

Starting from v1.83.0-nightly, all LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Expected output:

The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Verified safe versions​

We have audited every LiteLLM release published between v1.78.0 and v1.82.6 across both PyPI and Docker. Each artifact was verified by:

1. Downloading the published artifact and computing its SHA-256 digest
2. Scanning for the known indicators of compromise (IOCs)
3. Comparing the artifact contents against the corresponding Git commit in the BerriAI/litellm repository

All versions listed below are confirmed clean.

Questions and support​

If you believe your systems may be affected, contact us immediately:

• Security: security@berri.ai
• Support: support@berri.ai
• Slack: Reach out to the LiteLLM team directly

For real-time updates, follow LiteLLM (YC W23) on X.

Summary​

When a custom guardrail returned the full LiteLLM request/data dictionary, the guardrail response logged by LiteLLM could include secret_fields.raw_headers, including plaintext Authorization headers containing API keys or other credentials.

This information could then propagate to logging and observability surfaces that consume guardrail metadata, including:

• Spend logs in the LiteLLM UI: visible to admins with access to spend-log data
• OpenTelemetry traces: visible to anyone with access to the relevant telemetry backend

LLM calls, proxy routing, and provider execution were not blocked by this bug. The impact was exposure of sensitive request headers in observability and logging paths.

Background​

LiteLLM keeps internal request data (including request headers) for use during the call. That data is not meant to be written to logs or telemetry.

When custom guardrails run, their outcomes are logged so they can appear in spend logs, OpenTelemetry traces, and other observability backends. If a guardrail returned the full request payload instead of a minimal result, that internal request data could be included in what was logged. Before the fix, the guardrail logging path did not strip that data before sending it to those systems.

Root Cause​

The root cause was incomplete sanitization in the guardrail logging path. When building the payload that gets sent to spend logs and traces, LiteLLM prepared guardrail responses for logging but did not strip internal request data (such as headers) from them. If a guardrail returned a response that included that data, it was passed through to the logging and observability systems unchanged.

Impact​

This issue required all of the following:

1. A custom guardrail returned the full LiteLLM request/data dictionary, or another response object containing secret_fields.
2. LiteLLM logged that guardrail response through the standard guardrail logging path.
3. An operator, admin, or telemetry consumer had access to the resulting logs or traces.

When those conditions were met, sensitive values could become visible through:

• Spend logs / UI responses: guardrail metadata could be included in spend-log payloads rendered in the admin UI.
• OpenTelemetry traces: guardrail_response could be written as a span attribute on guardrail spans.
• Other downstream observability backends: any integration consuming the same guardrail metadata could receive the leaked values.

This was a logging and telemetry exposure bug. It did not let callers bypass auth, access other tenants directly, or change model behavior, but it could expose plaintext credentials to people with access to those observability systems.

Guidance For Users​

• Upgrade to LiteLLM 1.82.3+.
• If you operated custom guardrails that return the full request/data dict, review whether spend logs or telemetry traces were retained during the affected period.
• Rotate any credentials that may have appeared in Authorization or other forwarded request headers in those systems.
• Apply least-privilege access controls to spend-log views and telemetry backends that may contain request-derived metadata.

Usage​

model_list:
  - model_name: gpt-5.4-mini
    litellm_params:
      model: openai/gpt-5.4-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.4-nano
    litellm_params:
      model: openai/gpt-5.4-nano
      api_key: os.environ/OPENAI_API_KEY

litellm --config /path/to/config.yaml

# GPT-5.4-mini
curl -X POST "http://localhost:4000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.4-mini",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

# GPT-5.4-nano
curl -X POST "http://localhost:4000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.4-nano",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}]
  }'

from litellm import completion

# GPT-5.4-mini
response = completion(
    model="openai/gpt-5.4-mini",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.choices[0].message.content)

# GPT-5.4-nano
response = completion(
    model="openai/gpt-5.4-nano",
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
)
print(response.choices[0].message.content)

Notes​

• Both models support function calling, vision, and tool-use — see the OpenAI provider docs for advanced usage.
• GPT-5.4-nano is the most cost-effective option for simple tasks; GPT-5.4-mini offers a balance of speed and capability.

What's New​

Four new endpoints for video character operations:

• Create character - Upload a video to create a reusable asset
• Get character - Retrieve character metadata
• Edit video - Modify generated videos
• Extend video - Continue clips with character consistency

Available from: LiteLLM v1.83.0+

Quick Example​

import litellm

# Create character from video
character = litellm.avideo_create_character(
    name="Luna",
    video=open("luna.mp4", "rb"),
    custom_llm_provider="openai",
    model="sora-2"
)
print(f"Character: {character.id}")

# Use in generation
video = litellm.avideo(
    model="sora-2",
    prompt="Luna dances through a magical forest.",
    characters=[{"id": character.id}],
    seconds="8"
)

# Get character info
fetched = litellm.avideo_get_character(
    character_id=character.id,
    custom_llm_provider="openai"
)

# Edit with character preserved
edited = litellm.avideo_edit(
    video_id=video.id,
    prompt="Add warm golden lighting"
)

# Extend sequence
extended = litellm.avideo_extension(
    video_id=video.id,
    prompt="Luna waves goodbye",
    seconds="5"
)

Via Proxy​

# Create character
curl -X POST "http://localhost:4000/v1/videos/characters" \
  -H "Authorization: Bearer sk-litellm-key" \
  -F "video=@luna.mp4" \
  -F "name=Luna"

# Get character
curl -X GET "http://localhost:4000/v1/videos/characters/char_abc123def456" \
  -H "Authorization: Bearer sk-litellm-key"

# Edit video
curl -X POST "http://localhost:4000/v1/videos/edits" \
  -H "Authorization: Bearer sk-litellm-key" \
  -H "Content-Type: application/json" \
  -d '{
    "video": {"id": "video_xyz789"},
    "prompt": "Add warm golden lighting and enhance colors"
  }'

# Extend video
curl -X POST "http://localhost:4000/v1/videos/extensions" \
  -H "Authorization: Bearer sk-litellm-key" \
  -H "Content-Type: application/json" \
  -d '{
    "video": {"id": "video_xyz789"},
    "prompt": "Luna waves goodbye and walks into the sunset",
    "seconds": "5"
  }'

Managed Character IDs​

LiteLLM automatically encodes provider and model metadata into character IDs:

What happens:

Upload character "Luna" with model "sora-2" on OpenAI
  ↓
LiteLLM creates: char_abc123def456 (contains provider + model_id)
  ↓
When you reference it later, LiteLLM decodes automatically
  ↓
Router knows exactly which deployment to use

Behind the scenes:

• Character ID format: character_<base64_encoded_metadata>
• Metadata includes: provider, model_id, original_character_id
• Transparent to you - just use the ID, LiteLLM handles routing

How it works​

Flow of generating ephemeral token

Proxy Setup​

model_list:
  - model_name: gpt-4o-realtime
    litellm_params:
      model: openai/gpt-4o-realtime-preview-2024-12-17
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      mode: realtime

Azure: use model: azure/gpt-4o-realtime-preview, api_key, api_base.

litellm --config /path/to/config.yaml

Try it live​

Client Usage​

1. Get token - POST /v1/realtime/client_secrets with LiteLLM API key and { model }.

2. WebRTC handshake - Create RTCPeerConnection, add mic track, create data channel oai-events, send SDP offer to POST /v1/realtime/calls with Authorization: Bearer <encrypted_token> and Content-Type: application/sdp.

3. Events - Use the data channel for session.update and other events.

// 1. Token
const r = await fetch("http://proxy:4000/v1/realtime/client_secrets", {
  method: "POST",
  headers: { "Authorization": "Bearer sk-litellm-key", "Content-Type": "application/json" },
  body: JSON.stringify({ model: "gpt-4o-realtime" }),
});
const { client_secret } = await r.json();
const token = client_secret.value;

// 2. WebRTC
const pc = new RTCPeerConnection();
const audio = document.createElement("audio");
audio.autoplay = true;
pc.ontrack = (e) => (audio.srcObject = e.streams[0]);
const ms = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(ms.getTracks()[0]);
const dc = pc.createDataChannel("oai-events");
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

const sdpRes = await fetch("http://proxy:4000/v1/realtime/calls", {
  method: "POST",
  headers: { "Authorization": `Bearer ${token}`, "Content-Type": "application/sdp" },
  body: offer.sdp,
});
await pc.setRemoteDescription({ type: "answer", sdp: await sdpRes.text() });

// 3. Events
dc.send(JSON.stringify({ type: "session.update", session: { instructions: "..." } }));

FAQ​

Q: What do I do if I get a 401 Token expired error?
A: Tokens are short-lived. Get a fresh token right before creating the WebRTC offer.

Q: Which key should I use for /v1/realtime/calls?
A: Use the encrypted token from client_secrets, not your raw API key.

Q: Should I pass the model parameter when making the call?
A: No, the encrypted token already encodes all routing information including model.

Q: How do I resolve Azure api-version errors?
A: Set the correct api_version in litellm_params (or via the AZURE_API_VERSION environment variable), along with the right api_base and deployment values.

Q: What if I get no audio?
A: Make sure you grant microphone permission, ensure pc.ontrack assigns the audio element with autoplay enabled, check your network/firewall for WebRTC traffic, and inspect the browser console for ICE or SDP errors.

Docker Image​

docker pull ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch

Usage​

model_list:
  - model_name: gpt-5.4
    litellm_params:
      model: openai/gpt-5.4
      api_key: os.environ/OPENAI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.4",
    "messages": [
      {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ]
  }'

from litellm import completion

response = completion(
    model="openai/gpt-5.4",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
)

print(response.choices[0].message.content)

Notes​

• Restart your container to get the cost tracking for this model.
• Use /responses for better model performance.
• GPT-5.4 supports reasoning, function calling, vision, and tool-use — see the OpenAI provider docs for advanced usage.

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.80.8-stable.1

pip install litellm==v1.80.8-stable.1

What's New​

Supports all four thinking levels:

• MINIMAL: Ultra-fast responses with minimal reasoning
• LOW: Simple instruction following
• MEDIUM: Balanced reasoning for complex tasks
• HIGH: Maximum reasoning depth (dynamic)

Quick Start​

from litellm import completion

response = completion(
    model="gemini/gemini-3.1-flash-lite-preview",
    messages=[{"role": "user", "content": "Extract key entities from this text: ..."}],
)

print(response.choices[0].message.content)

from litellm import completion

# Use MEDIUM thinking for complex reasoning tasks
response = completion(
    model="gemini/gemini-3.1-flash-lite-preview",
    messages=[{"role": "user", "content": "Analyze this dataset and identify patterns"}],
    reasoning_effort="medium",  # low, medium , high
)

print(response.choices[0].message.content)

model_list:
  - model_name: gemini-3.1-flash-lite
    litellm_params:
      model: gemini/gemini-3.1-flash-lite-preview
      api_key: os.environ/GEMINI_API_KEY
  
  # Or use Vertex AI
  - model_name: vertex-gemini-3.1-flash-lite
    litellm_params:
      model: vertex_ai/gemini-3.1-flash-lite-preview
      vertex_project: your-project-id
      vertex_location: us-central1

litellm --config /path/to/config.yaml