FastTransfer Configuration

MigratorXpress delegates all data movement to FastTransfer (>= 0.15.0 required), which uses a streaming architecture. Rows flow directly from source to target without being buffered entirely in memory, so tables with billions of rows can be transferred on modest hardware.

FastTransfer Location

--fasttransfer_dir_path is optional. When omitted, MigratorXpress looks for FastTransfer in the engine/ subdirectory next to the MigratorXpress binary. Pass the flag to use a FastTransfer installed elsewhere. The FastTransfer version is checked at startup; an error is logged if it is older than 0.15.0.

FastTransfer is started directly from an argument list, without a shell, using the absolute path of the executable: passwords containing ", $, `, % or spaces reach it unchanged. Passwords and connection strings are masked in the logged command and in FastTransfer's logged output.

Connection Types

Database	As source	As target
PostgreSQL	pgsql (pgcopy when the target is PostgreSQL)	pgcopy
Oracle	oraodp	--
SQL Server	mssql (odbc for an odbc auth entry)	msbulk
Netezza	nzsql	--

For a SQL Server source and a PostgreSQL target, tables with date/time (datetime2, datetime, smalldatetime, datetimeoffset, time) or binary (binary, varbinary, image) columns are read through a query (--query) that converts these columns to text that PostgreSQL parses back exactly. Other tables are read with --sourceschema / --sourcetable.

Key Parameters

Parameter	Type	Default	Description
--fasttransfer_dir_path	Path	engine/ next to the binary	Directory containing the FastTransfer binary (optional)
--fasttransfer_p	Integer	1	Number of parallel streams within FastTransfer for large table partitioning
--ft_large_table_th	Integer	250,000,000	Tables with an estimated row count greater than or equal to this value are eligible for parallel transfer
--n_jobs	Integer	1	Number of tables to transfer simultaneously

Dual-Level Parallelism

MigratorXpress supports two levels of parallelism:

• --n_jobs controls how many tables are transferred concurrently.
• --fasttransfer_p controls how many parallel streams each large table uses internally.

For example, --n_jobs 4 --fasttransfer_p 2 transfers 4 tables simultaneously, and any table at or above the large table threshold is split into 2 parallel streams (except SQL Server sources, see below).

Parallel Transfer Methods

Each source database uses a different method to partition tables for parallel transfer:

Source Database	Method	Key Column Required	Description
PostgreSQL	Ctid	No	Splits by physical tuple ID
Oracle	Rowid	No	Splits by physical ROWID; requires SELECT_CATALOG_ROLE
Netezza	NZDataSlice	No	Splits by internal data slice distribution
SQL Server	None	N/A	No intra-table parallelism; --fasttransfer_p has no effect

Method Selection Logic

MigratorXpress selects the parallel transfer method automatically based on the source database type. No manual configuration is needed. If --fasttransfer_p is 1 or the table's estimated row count is below --ft_large_table_th, the table is transferred as a single stream regardless of the source type. Row counts are database estimates unless --compute_nbrows true is set.

Performance Tuning

PostgreSQL Sources

Ctid-based parallelism works well with minimal configuration. Increase --fasttransfer_p for tables with hundreds of millions of rows. Start with 2-4 parallel streams and scale up based on available CPU and I/O bandwidth.

Oracle Sources

Rowid-based parallelism requires SELECT_CATALOG_ROLE (see Oracle configuration). Oracle parallel transfers benefit from higher stream counts on large tables. Consider 4-8 streams for tables exceeding 100M rows.

Netezza Sources

NZDataSlice parallelism aligns with Netezza's internal data distribution. Set --fasttransfer_p to match or be a factor of the number of data slices on the appliance.

SQL Server Sources

SQL Server sources use non-partitioned transfers. Parallelism is achieved at the table level via --n_jobs. For SQL Server sources, increase --n_jobs rather than --fasttransfer_p.

SQL Server Target Options

When SQL Server is the target database, MigratorXpress supports additional options for columnstore indexes and foreign key handling. With a PostgreSQL target, --cci_threshold and --aci_threshold are ignored.

Columnstore Indexes

Parameter	Type	Default	Description
--cci_threshold	Integer	1,000,000	Create Clustered Columnstore Index for tables above this row count
--aci_threshold	Integer	100,000,000	Use archive compression (COLUMNSTORE_ARCHIVE) above this row count

Tables >= cci_threshold but < aci_threshold get a standard CCI. Tables >= aci_threshold get a CCI with DATA_COMPRESSION = COLUMNSTORE_ARCHIVE. Tables below cci_threshold remain as standard rowstore tables.

Foreign Key Mode

Parameter	Type	Default	Description
--fk_mode	String	disabled	FK enforcement for copy_fk (SQL Server target only): disabled (created with NOCHECK and disabled), untrusted (NOCHECK but enabled), trusted (full enforcement)

Row Count Options

Parameter	Type	Default	Description
--compute_nbrows	Boolean	false	true to use COUNT(*) for exact row counts instead of database estimates
--without_xid	Flag	false	Disable transaction ID / ROWID watermark usage in Netezza and Oracle (forced on by --drop_tables_if_exists)

Configuration Examples

Small Migration (< 10 tables, < 1M rows total)

./MigratorXpress --auth ./credentials.json \
  --source_db_auth_id oracle_prod \
  --source_schema_name HR \
  --target_db_auth_id mssql_target \
  --target_schema_name hr \
  --fasttransfer_dir_path ./FastTransfer_linux-x64_v0.15.0/ \
  --migration_db_auth_id ms_mig_log \
  --task_list translate create transfer diff

Default parallelism is sufficient. Single-stream transfer per table, one table at a time.

Medium Migration (10-100 tables, up to 100M rows)

./MigratorXpress --auth ./credentials.json \
  --source_db_auth_id oracle_prod \
  --source_schema_name SALES \
  --target_db_auth_id mssql_target \
  --target_schema_name sales \
  --fasttransfer_dir_path ./FastTransfer_linux-x64_v0.15.0/ \
  --migration_db_auth_id ms_mig_log \
  --n_jobs 4 \
  --fasttransfer_p 2 \
  --task_list translate create transfer diff

Four tables in parallel, large tables split into 2 streams each.

Large Migration (100+ tables, billions of rows)

./MigratorXpress --auth ./credentials.json \
  --source_db_auth_id oracle_prod \
  --source_schema_name WAREHOUSE \
  --target_db_auth_id mssql_target \
  --target_schema_name warehouse \
  --fasttransfer_dir_path ./FastTransfer_linux-x64_v0.15.0/ \
  --migration_db_auth_id ms_mig_log \
  --n_jobs 8 \
  --fasttransfer_p 4 \
  --ft_large_table_th 500000 \
  --task_list translate create transfer diff

Eight tables in parallel, large tables (>= 500,000 rows) split into 4 streams. Adjust thresholds based on hardware capacity.

Monitoring Performance

During and after a migration, you can query the migration tracking database to check progress and identify slow tables.

Check Overall Progress

Per-table jobs live in sourcetargetloads (one row per table and task). status is NULL (or NOT STARTED) before a job runs, then STARTED, DONE, or FAILED.

SELECT
    l.desc_step,
    COUNT(*) AS total_tables,
    SUM(CASE WHEN s.status = 'DONE' THEN 1 ELSE 0 END) AS done,
    SUM(CASE WHEN s.status = 'STARTED' THEN 1 ELSE 0 END) AS running,
    SUM(CASE WHEN s.status = 'FAILED' THEN 1 ELSE 0 END) AS failed
FROM sourcetargetloads AS s
JOIN loadsteps AS l ON l.id_step = s.id_step
WHERE s.run_id = 'YOUR_RUN_ID'
GROUP BY l.desc_step;

Identify Slow Tables

Each finished job writes an eventlog row with its elapsed time. Step 3 is data_transfer:

SELECT
    src_schemaname,
    src_tablename,
    eventstatus,
    elapsedtimeseconds
FROM eventlog
WHERE eventrunid = 'YOUR_RUN_ID'
    AND id_step = 3
    AND eventstatus IN ('DONE', 'FAILED')
ORDER BY elapsedtimeseconds DESC;

Troubleshooting

FastTransfer binary not found: If you rely on auto-detection, check that FastTransfer is in the engine/ directory next to the MigratorXpress binary. Otherwise verify that --fasttransfer_dir_path points to the directory containing the FastTransfer executable. Check file permissions and ensure the binary is executable (chmod +x FastTransfer on Linux).

Transfer stalls on large tables: Reduce --fasttransfer_p if the source or target database is under heavy load. Check network bandwidth between the MigratorXpress host and the databases.

Out of memory errors: FastTransfer's streaming architecture normally prevents memory issues. If they occur, reduce --n_jobs to lower the number of concurrent table transfers.

Slow transfer throughput:

• Increase --fasttransfer_p for large tables.
• Increase --n_jobs if the bottleneck is small-table overhead rather than individual table speed.
• Check that the network path between source and target is not saturated.
• Verify that the source database is not under heavy concurrent load.

See Also

• Migration Tasks -- Task execution order and combinations
• Database Connections -- Auth file configuration
• Quick Start Guide -- Getting started with MigratorXpress
• CLI Reference -- Full command-line reference