CipherStash Proxy operations & troubleshooting

Useful Settings

Optimise logging for Observability

It is common for customers to push CipherStash Proxy logs to logging platforms like Datadog or ELK.

To run CipherStash Proxy with structured json logs, all output to stdout and with color disable:

1cipherstash-proxy --log-format structured --log-output stdout --nocolor true

Testing, CI & migrations

The CipherStash Proxy maintains a connection pool with limits defined in your configuration.

If your testing setup drops and recreates the database, the connection pool will error with ConnectionReset - Connection reset by peer.

Postgres invalidates any open connections if the database state changes. Any open connections in the pool will be reset, and the proxy will need to reset the pool and reconnect.

Any tests that hit the proxy as the pool is reset may experience the ConnectionReset error.

The actual error may be different, as everything depends on the timing of the drop, the tests connecting to the proxy, and pool settings in both the connecting test application and the proxy itself.

In general: if your tests error, check for database drop and recreate.

Mitigations

Reload or restart

Often the easiest option is to reload config or restart the CipherStash Proxy with each test run, after the drop.

CipherStash Proxy will reload the pool configuration when it receives a SIGHUP signal.

Existing connections will be closed and new connections established.

1kill -s SIGHUP $(pgrep -f cipherstash-proxy)

Alternatively, the process can be killed and restarted.

1kill -s SIGINT $(pgrep -f cipherstash-proxy)

In most cases, the proxy startup time is fast.

Example actual startup time in development environment using min_pool_size of 100 (creates 100 connections on start):

1Start: 2024-06-24T08:17:48.038280Z
2Done:  2024-06-24T08:17:48.394084Z

Your mileage may vary.

test_on_checkout

The test_on_checkout setting ensures that a health check is made every time a connection is acquired from the pool. If the connection fails the check, a new connection is established, within the constraints of the pool configuration and capacity.

Enabling test_on_checkout ensures that connections invalidated by the test suite will be restablished.

Note: the ConnectionReset error is still logged, but a new connection will be acquired and client statements executed without error.

use_dedicated_connection

The use_dedicated_connection configuration option forces the CipherStash Proxy to use a new connection for every client, bypassing the pool altogether.

As the pool is bypassed, the number of open connections may exceed the configured pool capacity and test_on_checkout is generally a better option.

Errors

AllServersDown

A connection cannot be acquired from the pool.

1Could not get connection from pool: { pool_name: "..", username: "..", error: "AllServersDown" }

In most cases there will be an underlying error explaining why a connection cannot be acquired.

Common causes include:

• the Postgres server itself has run out of available connections (check the max_connection config in postgresql.conf)
• an issue connecting to the Postgres server (eg a transient network glitch)
• an issue with the database credentials (generally the CipherStash Proxy will not start if a connection cannot be acquired, but credentials may change)