Forty minutes. That is how long two malicious LiteLLM releases sat on PyPI back in March. 24 March, 10:39 UTC, versions 1.82.7 and 1.82.8 go live. Around 11:19 they are gone, and the project told anyone who installed before 16:00 UTC that day to assume the worst.
As I write this, five months later almost to the day, credentials stolen in those forty minutes still work.
The malware was competent but ordinary. The gap is the story.
The chain
Start at the entry point, because it should make you uncomfortable. Nobody phished a LiteLLM maintainer. The attacker came in through Trivy, a vulnerability scanner, compromised earlier in March in a separate supply chain attack. A PyPI API token got exposed through the compromised Trivy dependency, and a publishing token is a skeleton key. You do not open a pull request. You do not pass review. You do not touch the project's CI at all. You push straight to the index and the index says thank you.
So the release process everyone trusts, the pipeline with the gates and the checks, was bypassed by design. The attacker held the one credential that sits above all of it.
Then there is the payload delivery, which deserves more attention than it got. Version 1.82.8 shipped a file called litellm_init.pth. Python has an ancient feature where .pth files in site-packages execute at interpreter startup. Not at import. Startup. Once that file is on disk, every Python process on the machine runs the payload. Your linter runs it. Your database migration runs it. You never import litellm once and it makes no difference.
On a CI runner that means the harvest happens at job start, when the environment is at its richest. Environment variables, SSH keys, cloud credentials, Kubernetes tokens, database passwords. And because this is 2026, the model keys too, OPENAI_API_KEY and ANTHROPIC_API_KEY sitting right there in env.
24 Mar 10:39 UTC . . . 1.82.7 / 1.82.8 live on PyPI
24 Mar ~11:19 UTC . . . pulled. exposure window: ~40 min
13 Aug . . . . . . . . Hudson Rock publishes the fallout
today . . . . . . . . . tested credentials still valid
What forty minutes buys
Hudson Rock put numbers on it in August. A 153GB archive. 433,909 files. 118,829 CI runner dumps, tied to 2,488 corporate domains. CloudSEK maps the exposure to more than 2,500 organisations, and the names in the pile are not small: NVIDIA, Cisco, Deloitte, Volkswagen, FedEx, Siemens, X Corp.
Read those numbers against the window. Nobody triaged anything in forty minutes. No human saw an alert and made a decision inside that window. Automated builds pulled the poisoned versions, executed the payload at interpreter startup and shipped their own secrets out, at machine speed, on the attacker's behalf. The attack was over before the defenders' day started.
The five months
Here is the part that should actually scare you. PyPI pulled the packages, LiteLLM published an advisory, and for most affected teams that was the end of the incident. Quarantine got treated as closure. The artifact is gone, therefore the problem is gone.
Except the incident does not end when the package dies. It ends when the credentials the package saw are dead. Those are different events, and the distance between them turned a forty minute compromise into a five month one.
One researcher checked an organisation that claimed it had rotated everything. Almost every credential he tested still worked.
That tracks with everything we know about rotation. Finding secrets is a solved problem. Rotating them is not, because rotation means knowing what breaks when the value changes, and that knowledge is usually in one engineer's head, and that engineer may have left. So the dashboard says remediated while the attacker's copy keeps working.
Rotate on exposure. Not on confirmation. If the artifact ran where your secrets live, the secrets are gone. Behave accordingly.
Your scanner is a dependency
Sit with the entry point again. The ingress for one of the worst CI/CD compromises of the year was a security tool. Trivy sits inside build pipelines on purpose, with publishing rights and tokens in reach, and we extend it a trust we would never extend to a random utility library. There is no technical basis for that trust. A scanner is code you pull from the internet, running with more privilege than the code it inspects.
Threat model your security tooling like any other dependency, because the attackers already do.
What actually stops this
In our pipeline the control that turns this whole incident into a non event is boring: a minimum package age. Installs route through a proxy that refuses anything published less than 48 hours ago. The malicious versions lived for forty minutes. Under a 48 hour hold, nothing in the building could have pulled them, not a dev shell, not a runner, and nobody had to be fast or even awake. The window closed because the window was never open.
The hold costs something. Occasionally a fix you genuinely need shipped that morning and someone has an argument with the proxy. That argument is the price, and against 118,829 runner dumps it is nothing.
Beyond that, the homework this incident sets is short. Grep your build logs for litellm 1.82.7 and 1.82.8 around 24 March, and treat any hit as full compromise of everything that environment held. Go and look at what your CI runners actually expose in env at job start, because that inventory is precisely what got dumped here. Audit site-packages for .pth files you cannot explain. Then look hard at how many of your credentials are long lived, because a stolen token that expires in an hour is a very different artifact from one that still works in August.
Forty minutes of attacker effort. Five months of defender debt. The ratio is the lesson, and it only moves in one direction: mechanically, before the window opens, because nothing human moves inside it.
Sources:
The Hacker News: malicious LiteLLM releases tied to Trivy compromise (CloudSEK)
Help Net Security: Hudson Rock on the stolen credential leak
No comments:
Post a Comment