Posts on 3manuek

Upgrading Postgres Major, and Django model with Logical Replication.

Sat, 15 Nov 2025 00:00:00 +0000

The story

A customer needed to upgrade their Aurora RDS Postgres from v13 to v16 due to v13’s EOL. This is a common requirement, and so far, a straightforward process. The complexity arose when they needed to upgrade their application model, which was taking a long time to run due that most of the tables required a full rewrite.

The Django migration was taking several hours to run, which made a blue/green upgrade and subsequent migration execution unfeasible due to the long downtime it would incur.

Online Backups with no archives in PGBackRest

Sun, 28 Sep 2025 00:00:00 +0000

Even tho is not documented in the official documentation, it is possible to avoid storing archives on PGBackRest setups. In certain scenarios, you may want to disregard storing archives for reducing costs, specially if you are using cloud storage for your backups.

Those situations could be:

Tests environments, whether you run intensive tests but you don’t need or care about PITR (Point in Time Recovery).
- Also, you want to test your backup configuration rather than just having logical dumps in test environments.
Highly updated Postgres databases, where you can recover changes by other means, such as scrapping or restoring from external sources.

⚠️

Not storing archives may lead you to data loss if you don’t have a proper strategy for handling the deltas.
You still need to store archives during backup execution if you want online backups. Otherwise, they won’t be recoverable.
If you don’t want to store archives at all, you need to run backup offline. That is, stopping your instance and executing pgbackrest backup.

Configuration

Configuration on your postgresql.conf:

Evaluating PGCat's Sharding by hash

Sun, 09 Mar 2025 00:00:00 +0000

ℹ️ PGCat’s sharding feature is currently experimental. The code for the laboratory can be found at lab_pgcat.

PGCat sharding mechanisms

As of today, PGCat supports 2 mechanisms of sharding through extended syntax:

By setting the shard explicitly: SET SHARD TO '<index>';, which allows you to do deterministic sharding, whether you choose your shard according to a rule, such a lookup, region, customer’s group, etc. This is great if you have one of those well delimited or an even distribution. But, still a nice approach and kind scalable. We won’t be focusing on this strategy in this post, as its implementation relies on custom requirements.
Setting sharding_function to use one of the available functions: pg_bigint_hash and sha1. The extended syntax SET SHARDING KEY TO '<value>'; will calculate the index. Not very clear from docs how sha1 function is used, so this post will focus on pg_bigint_hash case. Shard by hash is a bold strategy, particularly if you expect to have a large workload, and you need to have enough compute across all shards. This extended syntax can be done through comments, see pgcat Sharding documentation. In this laboratory, we’ll focus on the pg_bigint_hash function. It is not clear from PGCat’s documentation how sha1 should be implemented, by I’ll extend the laboratory to cover it – that is, if I overcome my skill issues :P .

At this point, you may be aware of the complexities of implementing sharding, and what limitations we expect from the hashing approach. Keep in mind that this PGCat feature is tied to the Postgres partition, based on the same HASH_PARTITION_SEED. See also pgcat hash seed.

[BabelfishPG] Enabling Statements Execution Time logging and Plans

Fri, 25 Oct 2024 00:00:00 +0000

Introduction

ℹ️ Laboratory for this post can be found at babelfishpg-lab.

⚠️ In this post, we cover 4.2.0 version. There is a slight difference in the logging settings and format from 4.1.1.

Even tho BabelfishPG is a Postgres flavor, the configuration is done through its extensions (babelfish_tds and babelfish_tsql). In this post, we focus on how to log query timings and enable showing query plans.

For Postgres vanilla, you can log slow queries by setting the log_min_duration_statements, but this won’t log TSQL statements. Queries execute through a different backend fork, so targeted statements executed through the TDS protocol, won’t get logged.

Cascading timeouts through Pool (PGBouncer)

Wed, 02 Oct 2024 00:00:00 +0000

Combining `query_timeout` and `statement_timeout`

In pgbouncer’s documentation it’s stated that the query_timeout should be set slightly higher that Postgres’ statement_timeout. Although this apply in most of the cases, it depends on the business requirements.

Generally, the statement_timeout should be set to the 99th percentile of your statements duration. However, there are cases were certain statements require larger timeouts, due to particularities like a large customer set, or larger fields, as in TOAST compression cases.

[BabelfishPG] Using tds_fdw to access BabelfishPG

Fri, 06 Sep 2024 00:00:00 +0000

It supports TDS, right?

Some things happen once in a lifetime, and the story around this is quite particular. A customer required something that at the beginning sound counter intuitive: migrate an existing Postgres database to BabelfishPG.

The thing was that the application was a critical business core, with a large amount of code that would require years to migrate fully for supporting another storage strategy. But the real reason was that their customer didn’t want to stick to any private licensing model, and required to use Open Source solutions.

[BabelfishPG] Pooling TDS connections in BabelfishPG with FreeTDS

Thu, 15 Aug 2024 00:00:00 +0000

Next post will cover performance tests using tdspool.

BabelfishPG connection architecture

Inherited from Postgres connection architecture, each connection trhough the TDS port will instance a Postgres backend. As in Postgres, BabelfishPG needs a middleware for funnel connections through the TDS port for avoiding running out of connections and processing capacity in the database server.

For Postgres, we have plenty of options, like PGBouncer, Odyssey, pgcat, name it. For T-SQL (read as MSSQL compatible language), there aren’t many open sourced solutions.

[Earthly +v0.8.10 feature]: Disabling Earthly internal labels

Mon, 01 Jul 2024 00:00:00 +0000

NOTE: This has not been yet updated in the Earthly documentation, but I’m pretty sure it will be soon (and I’ll update this post accordingly)

What does labels do to reproducible builds?

By default, Earthly adds dev.earthly.* labels in the built image. You can find these by doing a docker inspect <imageID> | jq -r '.[].Config.Labels'.

Here’s an example of the image configuration including the dev.earthly.* labels:

 "Config": {
 ...
 "Labels": {
 "dev.earthly.built-by": "homebrew",
 "dev.earthly.git-sha": "6b41f8409d7ffef0d25072c2c04250074b6e3c72",
 "dev.earthly.version": "v0.8.14",
 ...
 }
 }

You may probably infer what’s the potential issue with these values.

Implementing a functional multi-process function in Bash

Sat, 01 Jun 2024 00:00:00 +0000

If you’re in this post, is because you need to implement a consistent way to add parallelization to your scripts. This implementation is limited, as it does not contemplates grouping jobs for setting different process groups with different settings or priorities.

Although, probably most of the cases you just need to execute code that runs inside a block or just simply does something in parallel. The parallel command is a good option, but it requires to loose some code readability, particularly on code blocks that might inject a complex semantic.

Testcontainers' customization for Postgres tests

Sun, 26 May 2024 00:00:00 +0000

Testcontainers allows you to test your code with ephemeral containers right inside your tests. It provides different modules for simplifying the process, however, sometimes you many need to customize the container beyond the default parameters or it contents.

Source Code of the laboratory. All examples are functional, follow the instructions in the README file for setting up. The csv file is generated by the Docker Compose setup, you can regenerate the example by running docker compose up -d.

Building dynamically-linked binaries in distroless images

Tue, 14 May 2024 00:00:00 +0000

Case

I’ve been developing an application in Go that uses SQLite as a local cache of the backend. The story would end there if it wasn’t that I decided to go for a distroless image for opmitization purposes, and also, why not?

The thing here is that for enabling certain features of the database/sql driver, the artifact should be dynamically linked through CGO_ENABLED=1 and with the -tags "fts5" (for the FTS5 feature support, see this issue comment).

Using parallel with multiple arguments

Mon, 01 Apr 2024 00:00:00 +0000

A handy tool for parallelization

If you are reading this post, is because you have heard about parallel. It is a GNU developed tool for paralelizing commands across sets of arguments. I’ll be honest, I use to forget about its existence until I need to do quick and dirty things when dealing with tasks that require paralelization, and whenever time is a constraint.

It has plenty of options and arguments, although once you get up on its usage, it is very handy in many cases.

Ergodox Keyboard Layout for Colemak, QWERTY and Dvorak

Wed, 24 Jan 2024 00:00:00 +0000

ℹ️ This is my current MacOS Layout.

A few months ago, I bought the Ergodox OrthoLinear Split Keyboard. After a period of trying it, I’ve found that the QWERTY layout was considerably uncomfortable to type in the keyboard disposition. I wasn’t the only one, of course, so I started the journey for searching a suitable layout. During a period of time, I switched back and forth between three layouts (QWERTY, Dvorak, and Colemak), although I picked Colemak at the end.

[Useless Code] Swapping variable values with no additional libraries

Thu, 23 Feb 2023 00:00:00 +0000

Background

There was a contest in an internet thread about how to code a variable swap as low-level and minimal as possible. So, I ended up writing a low-level function in C calling instructions for a modern amd64 platform.

#include <stdio.h>
#include <stdlib.h>

void swapshit(int *arg1, int *arg2) {

 __asm__ __volatile__ ("movl %2, %%eax;"
 "movl %3, %%ebx;"
 "movl %%eax, %0;"
 "movl %%ebx, %1;"
 : "=g" (*arg2) , "=g" (*arg1)
 : "a" (*arg1), "b" (*arg2) );
}

int main(int argc, char *argv[])
{
 int arg1 = atoi(argv[1]);
 int arg2 = atoi(argv[2]);
 swapshit(&arg1,&arg2);
 printf("arg1 es %d , arg2 es %d\n",arg1,arg2);
 return 0;
}

The explanation of the above is quite simple. We use registers to allocate the values and call them in a different order using movl instruction. The main is just a wrapper that passes arguments.

Switching layout and key disposition experience

Mon, 12 Dec 2022 00:00:00 +0000

This post has no intention to discuss what layout might be better in terms of efficiency or effort. There are a considerable amount of great articles related to all the analisys across different layouts, and even tools for ingesting text and estimate the effort of typing the text for several layouts.

You’ll find this article more useful if you already decided to switch to Colemak, and make it your primary layout. Personally, I have found very comfortable to write in Dvorak, although there are some perks in Colemak regarding shortcut compatibility among other benefits that made me do the choice. You can read a very interesting comparision at this Xahlee post.

Posts on 3manuek

Upgrading Postgres Major, and Django model with Logical Replication.

The story

Online Backups with no archives in PGBackRest

Configuration

Evaluating PGCat's Sharding by hash

PGCat sharding mechanisms

[BabelfishPG] Enabling Statements Execution Time logging and Plans

Introduction

Cascading timeouts through Pool (PGBouncer)

Combining query_timeout and statement_timeout

[BabelfishPG] Using tds_fdw to access BabelfishPG

It supports TDS, right?

[BabelfishPG] Pooling TDS connections in BabelfishPG with FreeTDS

BabelfishPG connection architecture

[Earthly +v0.8.10 feature]: Disabling Earthly internal labels

What does labels do to reproducible builds?

Implementing a functional multi-process function in Bash

Testcontainers' customization for Postgres tests

Building dynamically-linked binaries in distroless images

Case

Using parallel with multiple arguments

A handy tool for parallelization

Ergodox Keyboard Layout for Colemak, QWERTY and Dvorak

[Useless Code] Swapping variable values with no additional libraries

Background

Switching layout and key disposition experience

Combining `query_timeout` and `statement_timeout`