Skip to content

Commit 8c16ad3

Browse files
nathan-bossartnathan-bossart
committed
Allow using syncfs() in frontend utilities.
This commit allows specifying a --sync-method in several frontend utilities that must synchronize many files to disk (initdb, pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade). On Linux, users can specify "syncfs" to synchronize the relevant file systems instead of calling fsync() for every single file. In many cases, using syncfs() is much faster. As with recovery_init_sync_method, this new option comes with some caveats. The descriptions of these caveats have been moved to a new appendix section in the documentation. Co-authored-by: Justin Pryzby Reviewed-by: Michael Paquier, Thomas Munro, Robert Haas, Justin Pryzby Discussion: https://postgr.es/m/20210930004340.GM831%40telsasoft.com
1 parent cccc6cd commit 8c16ad3

File tree

21 files changed

+271
-11
lines changed

21 files changed

+271
-11
lines changed

‎doc/src/sgml/config.sgml

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10511,15 +10511,9 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir'
1051110511
On Linux, <literal>syncfs</literal> may be used instead, to ask the
1051210512
operating system to synchronize the whole file systems that contain the
1051310513
data directory, the WAL files and each tablespace (but not any other
10514-
file systems that may be reachable through symbolic links). This may
10515-
be a lot faster than the <literal>fsync</literal> setting, because it
10516-
doesn't need to open each file one by one. On the other hand, it may
10517-
be slower if a file system is shared by other applications that
10518-
modify a lot of files, since those files will also be written to disk.
10519-
Furthermore, on versions of Linux before 5.8, I/O errors encountered
10520-
while writing data to disk may not be reported to
10521-
<productname>PostgreSQL</productname>, and relevant error messages may
10522-
appear only in kernel logs.
10514+
file systems that may be reachable through symbolic links). See
10515+
<xref linkend="syncfs"/> for more information about using
10516+
<function>syncfs()</function>.
1052310517
</para>
1052410518
<para>
1052510519
This parameter can only be set in the

‎doc/src/sgml/filelist.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,7 @@
183183
<!ENTITY acronyms SYSTEM "acronyms.sgml">
184184
<!ENTITY glossary SYSTEM "glossary.sgml">
185185
<!ENTITY color SYSTEM "color.sgml">
186+
<!ENTITY syncfs SYSTEM "syncfs.sgml">
186187

187188
<!ENTITY features-supported SYSTEM "features-supported.sgml">
188189
<!ENTITY features-unsupported SYSTEM "features-unsupported.sgml">

‎doc/src/sgml/postgres.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,7 @@ break is not needed in a wider output rendering.
294294
&acronyms;
295295
&glossary;
296296
&color;
297+
&syncfs;
297298
&obsolete;
298299

299300
</part>

‎doc/src/sgml/ref/initdb.sgml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,28 @@ PostgreSQL documentation
365365
</listitem>
366366
</varlistentry>
367367

368+
<varlistentry id="app-initdb-option-sync-method">
369+
<term><option>--sync-method</option></term>
370+
<listitem>
371+
<para>
372+
When set to <literal>fsync</literal>, which is the default,
373+
<command>initdb</command> will recursively open and synchronize all
374+
files in the data directory. The search for files will follow symbolic
375+
links for the WAL directory and each configured tablespace.
376+
</para>
377+
<para>
378+
On Linux, <literal>syncfs</literal> may be used instead to ask the
379+
operating system to synchronize the whole file systems that contain the
380+
data directory, the WAL files, and each tablespace. See
381+
<xref linkend="syncfs"/> for more information about using
382+
<function>syncfs()</function>.
383+
</para>
384+
<para>
385+
This option has no effect when <option>--no-sync</option> is used.
386+
</para>
387+
</listitem>
388+
</varlistentry>
389+
368390
<varlistentry id="app-initdb-option-sync-only">
369391
<term><option>-S</option></term>
370392
<term><option>--sync-only</option></term>

‎doc/src/sgml/ref/pg_basebackup.sgml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -594,6 +594,31 @@ PostgreSQL documentation
594594
</listitem>
595595
</varlistentry>
596596

597+
<varlistentry>
598+
<term><option>--sync-method</option></term>
599+
<listitem>
600+
<para>
601+
When set to <literal>fsync</literal>, which is the default,
602+
<command>pg_basebackup</command> will recursively open and synchronize
603+
all files in the backup directory. When the plain format is used, the
604+
search for files will follow symbolic links for the WAL directory and
605+
each configured tablespace.
606+
</para>
607+
<para>
608+
On Linux, <literal>syncfs</literal> may be used instead to ask the
609+
operating system to synchronize the whole file system that contains the
610+
backup directory. When the plain format is used,
611+
<command>pg_basebackup</command> will also synchronize the file systems
612+
that contain the WAL files and each tablespace. See
613+
<xref linkend="syncfs"/> for more information about using
614+
<function>syncfs()</function>.
615+
</para>
616+
<para>
617+
This option has no effect when <option>--no-sync</option> is used.
618+
</para>
619+
</listitem>
620+
</varlistentry>
621+
597622
<varlistentry>
598623
<term><option>-v</option></term>
599624
<term><option>--verbose</option></term>

‎doc/src/sgml/ref/pg_checksums.sgml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,28 @@ PostgreSQL documentation
139139
</listitem>
140140
</varlistentry>
141141

142+
<varlistentry>
143+
<term><option>--sync-method</option></term>
144+
<listitem>
145+
<para>
146+
When set to <literal>fsync</literal>, which is the default,
147+
<command>pg_checksums</command> will recursively open and synchronize
148+
all files in the data directory. The search for files will follow
149+
symbolic links for the WAL directory and each configured tablespace.
150+
</para>
151+
<para>
152+
On Linux, <literal>syncfs</literal> may be used instead to ask the
153+
operating system to synchronize the whole file systems that contain the
154+
data directory, the WAL files, and each tablespace. See
155+
<xref linkend="syncfs"/> for more information about using
156+
<function>syncfs()</function>.
157+
</para>
158+
<para>
159+
This option has no effect when <option>--no-sync</option> is used.
160+
</para>
161+
</listitem>
162+
</varlistentry>
163+
142164
<varlistentry>
143165
<term><option>-v</option></term>
144166
<term><option>--verbose</option></term>

‎doc/src/sgml/ref/pg_dump.sgml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1179,6 +1179,27 @@ PostgreSQL documentation
11791179
</listitem>
11801180
</varlistentry>
11811181

1182+
<varlistentry>
1183+
<term><option>--sync-method</option></term>
1184+
<listitem>
1185+
<para>
1186+
When set to <literal>fsync</literal>, which is the default,
1187+
<command>pg_dump --format=directory</command> will recursively open and
1188+
synchronize all files in the archive directory.
1189+
</para>
1190+
<para>
1191+
On Linux, <literal>syncfs</literal> may be used instead to ask the
1192+
operating system to synchronize the whole file system that contains the
1193+
archive directory. See <xref linkend="syncfs"/> for more information
1194+
about using <function>syncfs()</function>.
1195+
</para>
1196+
<para>
1197+
This option has no effect when <option>--no-sync</option> is used or
1198+
<option>--format</option> is not set to <literal>directory</literal>.
1199+
</para>
1200+
</listitem>
1201+
</varlistentry>
1202+
11821203
<varlistentry>
11831204
<term><option>--table-and-children=<replaceable class="parameter">pattern</replaceable></option></term>
11841205
<listitem>

‎doc/src/sgml/ref/pg_rewind.sgml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,28 @@ PostgreSQL documentation
284284
</listitem>
285285
</varlistentry>
286286

287+
<varlistentry>
288+
<term><option>--sync-method</option></term>
289+
<listitem>
290+
<para>
291+
When set to <literal>fsync</literal>, which is the default,
292+
<command>pg_rewind</command> will recursively open and synchronize all
293+
files in the data directory. The search for files will follow symbolic
294+
links for the WAL directory and each configured tablespace.
295+
</para>
296+
<para>
297+
On Linux, <literal>syncfs</literal> may be used instead to ask the
298+
operating system to synchronize the whole file systems that contain the
299+
data directory, the WAL files, and each tablespace. See
300+
<xref linkend="syncfs"/> for more information about using
301+
<function>syncfs()</function>.
302+
</para>
303+
<para>
304+
This option has no effect when <option>--no-sync</option> is used.
305+
</para>
306+
</listitem>
307+
</varlistentry>
308+
287309
<varlistentry>
288310
<term><option>-V</option></term>
289311
<term><option>--version</option></term>

‎doc/src/sgml/ref/pgupgrade.sgml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,29 @@ PostgreSQL documentation
190190
variable <envar>PGSOCKETDIR</envar></para></listitem>
191191
</varlistentry>
192192

193+
<varlistentry>
194+
<term><option>--sync-method</option></term>
195+
<listitem>
196+
<para>
197+
When set to <literal>fsync</literal>, which is the default,
198+
<command>pg_upgrade</command> will recursively open and synchronize all
199+
files in the upgraded cluster's data directory. The search for files
200+
will follow symbolic links for the WAL directory and each configured
201+
tablespace.
202+
</para>
203+
<para>
204+
On Linux, <literal>syncfs</literal> may be used instead to ask the
205+
operating system to synchronize the whole file systems that contain the
206+
upgraded cluster's data directory, its WAL files, and each tablespace.
207+
See <xref linkend="syncfs"/> for more information about using
208+
<function>syncfs()</function>.
209+
</para>
210+
<para>
211+
This option has no effect when <option>--no-sync</option> is used.
212+
</para>
213+
</listitem>
214+
</varlistentry>
215+
193216
<varlistentry>
194217
<term><option>-U</option> <replaceable>username</replaceable></term>
195218
<term><option>--username=</option><replaceable>username</replaceable></term>

‎doc/src/sgml/syncfs.sgml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
<!-- doc/src/sgml/syncfs.sgml -->
2+
3+
<appendix id="syncfs">
4+
<title><function>syncfs()</function> Caveats</title>
5+
6+
<indexterm zone="syncfs">
7+
<primary>syncfs</primary>
8+
</indexterm>
9+
10+
<para>
11+
On Linux <function>syncfs()</function> may be specified for some
12+
configuration parameters (e.g.,
13+
<xref linkend="guc-recovery-init-sync-method"/>), server applications (e.g.,
14+
<application>pg_upgrade</application>), and client applications (e.g.,
15+
<application>pg_basebackup</application>) that involve synchronizing many
16+
files to disk. <function>syncfs()</function> is advantageous in many cases,
17+
but there are some trade-offs to keep in mind.
18+
</para>
19+
20+
<para>
21+
Since <function>syncfs()</function> instructs the operating system to
22+
synchronize a whole file system, it typically requires many fewer system
23+
calls than using <function>fsync()</function> to synchronize each file one by
24+
one. Therefore, using <function>syncfs()</function> may be a lot faster than
25+
using <function>fsync()</function>. However, it may be slower if a file
26+
system is shared by other applications that modify a lot of files, since
27+
those files will also be written to disk.
28+
</para>
29+
30+
<para>
31+
Furthermore, on versions of Linux before 5.8, I/O errors encountered while
32+
writing data to disk may not be reported to the calling program, and relevant
33+
error messages may appear only in kernel logs.
34+
</para>
35+
36+
</appendix>

0 commit comments

Comments
 (0)