Lua Libraries in Istio's EnvoyFilter

Last week at work, we hit a very interesting problem — is it possible to run an Envoy plugin inside of an Istio sidecar? The plugin I was specifically interested in was Moesif, a really nice API request observability tool we use.

Now, the sidecar proxies in Istio are Envoy, so it seems like a reasonable enough request.

The problem of customizing Envoy configuration directly in Istio is solved by Istio's EnvoyFilter resource. You can see examples here.

Now, the Moesif plugin wants to run some Lua on outgoing requests. The EnvoyFilter documentation has a clear way to do that; it's a somewhat typical use case.

However, there's a problem — the Lua it wants to run depends on the Moesif library. This use case is closer to embedded Lua than a full-fledged application with a package manager. Even if we were setting up a VM we could download the library in the user data script. We need a way to get the Moesif library into the sidecar.

Istio does have a way to edit the sidecar configuration (the Sidecar resource), but it's more focused on the traffic going in and out than on filesystem of the sidecar. After some searching, I did come across an annotation that was exactly what I needed: sidecar.istio.io/userVolume.

apiVersion: v1
kind: Pod
metadata:
  labels:
    # matches `workloadSelector` in EnvoyFilter
    # allows you to turn on and off the EnvoyFilter for some workload
    super-special-plugin: activated
  annotations:
    sidecar.istio.io/userVolume: "<user-volume-json>"
    sidecar.istio.io/userVolumeMount: "<user-volume-mount-json>"
  name: nginx
spec:
  containers:
    - image: nginx
      name: nginx

With the omitted values looking like this:

{
    "lua-libs": {
        "configMap": {
            "name": "lua-libs",
            "items": [
                {
                    "key": "moesif-core-base64.lua",
                    "path": "moesif/core/base64.lua"
                },
                {
                    "key": "moesif-core-lib_deflate.lua",
                    "path": "moesif/core/lib_deflate.lua"
                },
                {
                    "key": "moesif-plugins-log.lua",
                    "path": "moesif/plugins/log.lua"
                },
                {
                    "key": "moesif-core-helpers.lua",
                    "path": "moesif/core/helpers.lua"
                },
                {
                    "key": "moesif-core-zzlib.lua",
                    "path": "moesif/core/zzlib.lua"
                },
                {
                    "key": "moesif-core-json.lua",
                    "path": "moesif/core/json.lua"
                },
                {
                    "key": "moesif-core.lua",
                    "path": "moesif/core.lua"
                }
            ]
        }
    }
}

{
    "lua-libs": {
        "mountPath": "/usr/local/share/lua/5.1", // Must be on Lua package path!
        "readOnly": true
    }
}

In the userVolume JSON, we're referencing the library as a ConfigMap. That's the best way I could come up with going from a folder of Lua code to files in the sidecar.

folder of Lua code -> ConfigMap -> user volume in sidecar

Going from the folder of Lua code to a ConfigMap was a simpler problem. I did the following, in Pulumi:

1. Recursively search the folder and build an array of all Lua files in the folder
2. Build a ConfigMap where a file like moesif/core/helpers.lua would have a key of moesif-core-helpers.lua and a value of the minified contents of the file. (Note that / is not a valid character in keys of ConfigMaps).

Once I verified that the library was present in the sidecar then things worked pretty smoothly.

$ kubectl exec -it nginx --container=istio-proxy -- ls /usr/local/share/lua/5.1
moesif

Let's go over the pros and cons of this approach.

Pros:

• You can easily turn the EnvoyFilter on/off with labels
• It works (we've tried about 3 other outbound traffic monitoring setups with Moesif, just getting one working is not a given. These tools are usually meant to monitor inbound traffic, not outbound.)
• This approach doesn't have as many TLS issues as others, since it's the first thing that catches the traffic as it leaves the original container, and in our setup, we have Istio originate TLS later in the Envoy listener's filter chain. The closer to the edge of your network you put a tool like this, the more likely you are to have TLS issues. There's a tradeoff there.

Cons:

• You have to specify that sidecar.istio.io/userVolume annotation on every Deployment. If you have some good abstraction over Kubernetes resources then this is less of a problem. For us, it's a problem.
• If you have any retries implemented in Istio, because that comes later in the filter chain, it's not visible to Moesif. Moesif just sees one request with the status code of the final call, and an increased latency. So 500's that we want to catch can be covered up.
• It's not simple.

If you've had similar issues, please let me know what approach you took!