Mock It Before You Rock It

As agents increasingly interface with external systems to do useful work, the ability to sandbox workflows has become extremely critical not only for us humans to verify logic but for agents writing workflows to iterate faster and catch blatant errors before accidentally making a change on a real system.

So this, in essence is the system architecture. Before diving into the objectives and data flows, I'm just adding the github repos 1 and 2 you can use to replicate the setup (there's one secret that you'll have to manually add in order to use SOPS for committing encrypted secrets that are decryptable by argocd repo server and swap out certs but pish posh, I'm sure that's figure-outable). And I've shown a hubspot mock server, but this could be literally any external provider out there.

System Objectives

What I wanted is to basically have the ability for workloads (and in this case I use a Temporal worker to illustrate) to safely exercise external API integrations without contacting the actual third party systems.

1. Prevent unintended external side effects: This is the primary goal. Outbound API calls should be redirected to internal mocks so we don't accidentally create contacts, send emails or modify customer data.
2. Application code should be unchanged: We literally want to test the application code as it will behave when we deploy it in prod. So the application should make normal HTTPS calls to real domains and the onus of sandboxing things falls on our mechanism.
3. Scope sandboxing to intended workloads: The mechanism is activated only for certain namespaces (in our case the sandbox-ns-1) controlled by an explicit label. We can instruct humans and agents to test their code in these namespaces.
4. Control DNS: One of the decisions I thought was important, was to use a separate DNS deployment. This way we can control some basic rules that "hey refuse HTTP egresses" or "wait we don't mock that. stop it".
5. Observe every external API call: MITMProxy is going to intercept any api call that exits our workload. Ergo we can quickly observe from container logs if there's egresses that our workload attempts but aren't intended for it.
6. Contain sensitive CA material: We give our workloads only the public CA trust certificate and add the signing key only to the mitmproxy sidecar. Right now I just add the root CA pkey but ideally we'd have an intermediate key to protect against leaks (a good prod system will do this).
7. Make the setup reproducible: Lastly but most importantly, make this entire setup reproducible. Sure I'm using SOPS so there's like some manual configuration steps not controlled by gitops and also I use a local registry so there's that additional hurdle to jump through but more or less I think this ones also satisfied (waiting for feedback on this one). In actual production, we could use hashicorp vault with vault secrets operator and a remote container registry.

Interestingly, I thought the exact same setup can be used for testing malware also? I don't know much about cybersecurity techniques but maybe just block external calls instead of mocking them and log them?

The Setup

I deployed a TypeScript Temporal worker to the sandbox namespace through the sandbox application repository.

Its workflow has two activities:

1. Create a HubSpot contact
2. List HubSpot contacts

Both activities make normal HTTPS requests to api.hubapi.com. The worker has no proxy configuration and no test-specific base URL.

I also deployed a HubSpot WireMock server with fixture mappings for those two routes. Its responses are static for this demo, which is enough to prove that the worker can exercise an external integration without contacting the real provider.

So the flow is this:

1. The worker activity (doesn't matter which one) makes a normal HTTPS request to a route on "api.hubapi.com"
2. The injected iptables init container redirects outbound TCP/443 traffic to the Pod-local mitmproxy sidecar.
3. mitmproxy dynamically issues a leaf certificate for api.hubapi.com, signed by the sandbox root CA. The worker trusts it because the public sandbox CA certificate was mounted into the Pod during admission.
4. MITMProxy derives the internal wiremock service destination from the hostname (and we currently add a configmap to tell it what we support in terms of mock servers) returning "api-hubapi-com.mock-servers-ns-1.svc.cluster.local:8080".
5. Wiremock returns a hubspot shaped response.

To support more providers, all I need to do is to deploy its mock service and mappings and add its external hostname to the routing configmap.

Also in actual usage, we'd create these deployments through kubectl commands (humans) or k8s api (agents) not through a static application in argocd. But I'm just trying to demo here so its sufficient.

The Run

If you observe carefully, the contact I wanted to add was [email protected] but it ended up creating [email protected] (because that's what the mock server is configured to statically return). Similarly, the contact list has "Ravi Sanghavi" and "Neelam Shanghavi" and it will always invariably return this because of the configs in the mock server.

Caveats

• A sandboxed workload can bypass the sidecar by using some other protocol (other than HTTPS), direct IP or privileged networking. We can use network policies (cilium/istio) to control this.
• The sandbox CA private key exists as a k8s secret but ideally we need to rotate it periodically.
• Certificate-pinned clients will reject the dynamically issued certificate. It won't generally be a problem but just something to bear in mind in case some sdk bundles it along.

So yeah. That was my attempt to sandbox external calls for workloads.