🌐 AI搜索 & 代理 主页
Skip to content

Commit 40ce12e

Browse files
fabriziomellofabriziomello
committed
Add pg_upgrade test to CI
It's long we need to have binary upgrade tests in our CI. One example is timescale#6935 (and others) that can be prevented if we have such kind of test in our CI. To implement the `pg_upgrade` test we used the python Testing Framework for PostgreSQL (https://github.com/postgrespro/testgres). Unfortunately the testing framework don't have the ability to retain the pg_upgrade log files after a successful execution, then we created a PR to make it possible and we'll use this patched version until we get the code merged and released on upstream. postgrespro/testgres#125 Closes timescale#3868 timescale#4428
1 parent 00fc782 commit 40ce12e

File tree

4 files changed

+260
-0
lines changed

4 files changed

+260
-0
lines changed

‎.github/workflows/code_style.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,8 +107,24 @@ jobs:
107107
psutil pygithub pglast
108108
pip list
109109
pip list --user
110+
111+
# Using e375302a until 1.10.1 get released including the following PR
112+
# https://github.com/postgrespro/testgres/pull/125
113+
- name: Checkout testgres
114+
uses: actions/checkout@v4
115+
with:
116+
repository: 'postgrespro/testgres'
117+
path: 'testgres'
118+
ref: e375302a114cd4df3ceed54d6526f250c44c08e7
119+
120+
- name: Build and install testgres
121+
run: |
122+
cd testgres
123+
python setup.py install --user
124+
110125
- name: Checkout source
111126
uses: actions/checkout@v4
127+
112128
- name: Run prospector
113129
run: |
114130
find . -type f -name "*.py" -print -exec prospector {} + -exec black {} +

‎.github/workflows/pg_upgrade-test.yaml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: pg_upgrade test
2+
"on":
3+
push:
4+
branches:
5+
- main
6+
- prerelease_test
7+
pull_request:
8+
9+
jobs:
10+
pg_upgrade_test:
11+
name: pg_upgrade test from PG${{ matrix.pg_version_old }} to PG${{ matrix.pg_version_new }}
12+
runs-on: 'ubuntu-latest'
13+
strategy:
14+
matrix:
15+
include:
16+
- pg_version_old: 14 # 14 to 15
17+
pg_version_new: 15
18+
- pg_version_old: 14 # 14 to 16
19+
pg_version_new: 16
20+
- pg_version_old: 15 # 15 to 16
21+
pg_version_new: 16
22+
fail-fast: false
23+
env:
24+
OUTPUT_DIR: ${{ github.workspace }}/pg_upgrade_test
25+
26+
steps:
27+
- name: Install Linux Dependencies
28+
run: |
29+
sudo apt-get update
30+
sudo apt-get install pip postgresql-common libkrb5-dev
31+
32+
# Using e375302a until 1.10.1 get released including the following PR
33+
# https://github.com/postgrespro/testgres/pull/125
34+
- name: Checkout testgres
35+
uses: actions/checkout@v4
36+
with:
37+
repository: 'postgrespro/testgres'
38+
path: 'testgres'
39+
ref: e375302a114cd4df3ceed54d6526f250c44c08e7
40+
41+
- name: Build and install testgres
42+
run: |
43+
cd testgres
44+
python setup.py install --user
45+
46+
- name: Install PostgreSQL ${{ matrix.pg_version_old}} and ${{ matrix.pg_version_new }}
47+
run: |
48+
yes | sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
49+
echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
50+
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -
51+
sudo apt-get update
52+
sudo apt-get install -y \
53+
postgresql-${{ matrix.pg_version_old }} postgresql-server-dev-${{ matrix.pg_version_old }} \
54+
postgresql-${{ matrix.pg_version_new }} postgresql-server-dev-${{ matrix.pg_version_new }}
55+
sudo apt-get install -y --no-install-recommends \
56+
timescaledb-2-postgresql-${{ matrix.pg_version_old }} \
57+
timescaledb-2-postgresql-${{ matrix.pg_version_new }}
58+
59+
- name: Checkout TimescaleDB
60+
uses: actions/checkout@v4
61+
62+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_old}}
63+
env:
64+
BUILD_DIR: pg${{ matrix.pg_version_old }}
65+
run: |
66+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_old }}/bin:$PATH"
67+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
68+
make -j -C pg${{ matrix.pg_version_old }}
69+
sudo make -j -C pg${{ matrix.pg_version_old }} install
70+
71+
- name: Build and install TimescaleDB on PostgreSQL ${{ matrix.pg_version_new}}
72+
env:
73+
BUILD_DIR: pg${{ matrix.pg_version_new }}
74+
run: |
75+
PATH="/usr/lib/postgresql/${{ matrix.pg_version_new }}/bin:$PATH"
76+
./bootstrap -DCMAKE_BUILD_TYPE=Release -DWARNINGS_AS_ERRORS=OFF -DASSERTIONS=ON -DLINTER=OFF -DGENERATE_DOWNGRADE_SCRIPT=OFF -DREGRESS_CHECKS=OFF -DTAP_CHECKS=OFF
77+
make -j -C pg${{ matrix.pg_version_new }}
78+
sudo make -j -C pg${{ matrix.pg_version_new }} install
79+
80+
- name: Run pg_upgrade test
81+
env:
82+
PGVERSIONOLD: ${{ matrix.pg_version_old }}
83+
PGVERSIONNEW: ${{ matrix.pg_version_new }}
84+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
85+
run: |
86+
scripts/test_pg_upgrade.py
87+
diff -u \
88+
"${OUTPUT_DIR}/post.pg${PGVERSIONOLD}.log" \
89+
"${OUTPUT_DIR}/post.pg${PGVERSIONNEW}.log" | \
90+
tee "${DIFFFILE}"
91+
if [[ -s "${DIFFFILE}" ]]; then
92+
echo "pg_upgrade test for ${PGVERSIONOLD} -> ${PGVERSIONNEW} failed"
93+
exit 1
94+
fi
95+
96+
- name: Show pg_upgrade diffs
97+
if: always()
98+
env:
99+
DIFFFILE: ${{ env.OUTPUT_DIR }}/upgrade_check.diff
100+
DIROLD: pg${{ matrix.pg_version_old }}
101+
DIRNEW: pg${{ matrix.pg_version_new }}
102+
run: |
103+
cd ${OUTPUT_DIR}
104+
cat ${DIFFFILE}
105+
tar czf /tmp/pg_upgrade_artifacts.tgz \
106+
${DIFFFILE} \
107+
${OUTPUT_DIR}/*.log \
108+
${OUTPUT_DIR}/${DIROLD}/logs/* \
109+
${OUTPUT_DIR}/${DIRNEW}/logs/* \
110+
${OUTPUT_DIR}/${DIRNEW}/data/pg_upgrade_output.d/*
111+
112+
- name: Upload pg_upgrade logs
113+
if: always()
114+
uses: actions/upload-artifact@v4
115+
with:
116+
name: pg_upgrade logs from ${{ matrix.pg_version_old }} to ${{ matrix.pg_version_new }}
117+
path: /tmp/pg_upgrade_artifacts.tgz

‎scripts/test_pg_upgrade.py‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#!/usr/bin/env python3
2+
3+
import os
4+
import sys
5+
6+
from shutil import rmtree
7+
from testgres import get_new_node, PostgresNode
8+
9+
10+
# Accessor functions
11+
def set_default_conf(node: PostgresNode, unix_socket_dir):
12+
node.default_conf(fsync=True, allow_streaming=False, allow_logical=False)
13+
node.append_conf(unix_socket_directories=unix_socket_dir)
14+
node.append_conf(timezone="GMT")
15+
node.append_conf(client_min_messages="warning")
16+
node.append_conf(max_prepared_transactions="100")
17+
node.append_conf(max_worker_processes="0")
18+
node.append_conf(shared_preload_libraries="timescaledb")
19+
node.append_conf("timescaledb.telemetry_level=off")
20+
21+
22+
def write_bytes_to_file(filename, nbytes):
23+
with open(filename, "wb") as f:
24+
f.write(nbytes)
25+
f.close()
26+
27+
28+
def getenv_or_error(envvar):
29+
return os.getenv(envvar) or sys.exit(f"Environment variable {envvar} not defined")
30+
31+
32+
def getenv_or_default(envvar, default):
33+
return os.getenv(envvar) or default
34+
35+
36+
def initialize_node(working_dir, prefix, port, bin_dir, base_dir):
37+
node = get_new_node(prefix=prefix, port=port, bin_dir=bin_dir, base_dir=base_dir)
38+
node.init()
39+
set_default_conf(node, working_dir)
40+
return node
41+
42+
43+
# Globals
44+
pg_version_old = getenv_or_error("PGVERSIONOLD")
45+
pg_version_new = getenv_or_error("PGVERSIONNEW")
46+
47+
pg_node_old = f"pg{pg_version_old}"
48+
pg_node_new = f"pg{pg_version_new}"
49+
50+
pg_port_old = getenv_or_default("PGPORTOLD", "54321")
51+
pg_port_new = getenv_or_default("PGPORTNEW", "54322")
52+
53+
test_version = getenv_or_default("TEST_VERSION", "v8")
54+
55+
pg_bin_old = getenv_or_default("PGBINOLD", f"/usr/lib/postgresql/{pg_version_old}/bin")
56+
pg_bin_new = getenv_or_default("PGBINNEW", f"/usr/lib/postgresql/{pg_version_new}/bin")
57+
58+
working_dir = getenv_or_default(
59+
"OUTPUT_DIR",
60+
f"/tmp/pg_upgrade_output/{pg_version_old}_to_{pg_version_new}",
61+
)
62+
63+
pg_data_old = getenv_or_default("PGDATAOLD", f"{working_dir}/{pg_node_old}")
64+
pg_data_new = getenv_or_default("PGDATAOLD", f"{working_dir}/{pg_node_new}")
65+
66+
pg_database_test = getenv_or_default("PGDATABASE", "pg_upgrade_test")
67+
68+
if os.path.exists(working_dir):
69+
rmtree(working_dir)
70+
os.makedirs(working_dir)
71+
72+
# Real testing code
73+
print(f"Initializing nodes {pg_node_old} and {pg_node_new}")
74+
node_old = initialize_node(
75+
working_dir, pg_node_old, pg_port_old, pg_bin_old, pg_data_old
76+
)
77+
node_old.start()
78+
79+
node_new = initialize_node(
80+
working_dir, pg_node_new, pg_port_new, pg_bin_new, pg_data_new
81+
)
82+
83+
print(f"Creating {pg_database_test} database on node {pg_node_old}")
84+
node_old.safe_psql(filename="test/sql/updates/setup.roles.sql")
85+
node_old.safe_psql(query=f"CREATE DATABASE {pg_database_test};")
86+
node_old.safe_psql(dbname=pg_database_test, query="CREATE EXTENSION timescaledb;")
87+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/pre.testing.sql")
88+
node_old.safe_psql(
89+
dbname=pg_database_test,
90+
filename=f"test/sql/updates/setup.{test_version}.sql",
91+
)
92+
node_old.safe_psql(dbname=pg_database_test, query="CHECKPOINT")
93+
node_old.safe_psql(dbname=pg_database_test, filename="test/sql/updates/setup.check.sql")
94+
95+
# Run new psql over the old node to have the same psql output
96+
node_new.port = pg_port_old
97+
(code, old_out, old_err) = node_new.psql(
98+
dbname=pg_database_test, filename="test/sql/updates/post.pg_upgrade.sql"
99+
)
100+
node_new.port = pg_port_new
101+
102+
# Save output to log
103+
write_bytes_to_file(f"{working_dir}/post.{pg_node_old}.log", old_out)
104+
node_old.stop()
105+
106+
print(f"Upgrading node {pg_node_old} to {pg_node_new}")
107+
res = node_new.upgrade_from(old_node=node_old, options=["--retain"])
108+
node_new.start()
109+
110+
(code, new_out, new_err) = node_new.psql(
111+
dbname=pg_database_test,
112+
filename="test/sql/updates/post.pg_upgrade.sql",
113+
)
114+
115+
# Save output to log
116+
write_bytes_to_file(f"{working_dir}/post.pg{pg_version_new}.log", new_out)
117+
node_new.stop()

‎test/sql/updates/post.pg_upgrade.sql‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
-- This file and its contents are licensed under the Apache License 2.0.
2+
-- Please see the included NOTICE for copyright information and
3+
-- LICENSE-APACHE for a copy of the license.
4+
5+
\pset format aligned
6+
\pset tuples_only off
7+
8+
\ir post.catalog.sql
9+
\ir post.policies.sql
10+
\ir post.functions.sql

0 commit comments

Comments
 (0)