External Program Connector

info

Available from Version 6.0.3686

The External Program connector reads from and writes to a data source through a command-line program that you provide. The program can be written in any language, such as Python, PowerShell, Node.js, Go or C#. It only has to follow a small protocol: Data Sync starts it once per operation, passes its input as files and reads its output from files.

You keep all of Data Sync's features, including the schema map, compare, lookups, calculated columns and the Connection Library. The program only has to know how to talk to your data source.

info

This connector is different from the External Program step in the Run Tool and in Ouvvi. Those steps run a program as part of a workflow. This connector uses a program as the source or target of a sync.

tip

You don't have to write the program yourself. The protocol specification is written so that an AI assistant such as Claude can build a working program from it. See Build a Program with AI.

When to Use It

Option	Use it when
A built-in connector	Data Sync already connects to your system.
JSON API Connector	The data is behind a REST API that you can describe without code.
External Program	You need code to reach the data, for example an unusual API, a file format, a command-line tool or a vendor SDK, and you don't want to build a .NET connector.
Custom Connector	You want a .NET connector that is installed into Data Sync.

Connecting

To access the connector, open the connection window, expand the Simego folder and select External Program.

1. Set ProgramPath to your program, or to the interpreter of a script. If you use an interpreter, set ScriptPath to the script.
2. Open Parameters and enter a value for each one. Data Sync asks the program which parameters it needs and adds them for you.
3. Choose a table from the DataSource drop-down, which is filled from the program's list of datasources.
4. Click Connect.

Data Sync checks the program's protocol version and that every required parameter has a value, then loads the columns into the schema map.

Program and Script Paths

Program	ProgramPath	ScriptPath
Executable	C:\Tools\MyConnector.exe	Empty
Python	py, or the full path of python.exe	C:\Connectors\orders.py
Node.js	node	C:\Connectors\orders.js
PowerShell	pwsh	C:\Connectors\orders.ps1
.NET	dotnet	C:\Connectors\Orders.dll

A name without a folder, such as node, is found on the PATH.

Properties

Property	Description
ProgramPath	The program to run, or the interpreter of a script.
ScriptPath	The script for the interpreter, passed as its first argument. Leave this empty for a program that runs by itself.
Parameters	The settings the program needs, such as a folder or a connection string.
DataSource	The datasource to read and write. You can type a name if it isn't listed.
WorkingDirectory	The folder the program runs in. If empty, this is the folder containing the script, or else the program.
TimeoutSeconds	How long one command may run before it is stopped. The default is 300.
DateTimeHandling	How System.DateTime values are converted: Utc (the default), Local or Unspecified. See Dates and Times.
UpdateBatchSize	The number of changes passed to the program each time it runs to write. The default is 100.

Parameters and Secrets

A program declares its parameters, and marks passwords, tokens and keys as secret. Secret values are stored encrypted when you have installed an Encryption Key in DataSync. They are only decrypted to write a temporary configuration file, which only the current user can read and which is deleted afterwards. Parameter values are never passed on the command line, so they can't be seen in Task Manager or in logs.

Use absolute paths in parameters. A relative path is taken from the working directory.

Reading and Writing

The columns the program marks as unique are the key of the datasource. Data Sync reads the whole datasource into memory, and the whole read must finish within TimeoutSeconds.

When the program is the target, Data Sync sends the differences it finds in batches of UpdateBatchSize, always in the order add, update, delete. A program can be read-only, or support only some of these actions.

Each change that fails is reported on its own, with the program's error message, and the other changes in the batch still apply.

Data Sync shows progress while the program runs. If you press stop, or a Data Preview has enough rows, a program that supports stopping finishes the row in hand and exits. Any other program is ended straight away. Changes that weren't applied are picked up by the next sync.

caution

As with any target, rows that exist only in the target are deleted. Run a Compare and check the results before you run a Sync with a new program, and test it against a copy of the data first.

Dates and Times

DateTimeHandling controls how System.DateTime values are converted. Columns of type System.DateTimeOffset and date-only columns of type Simego.DataSync.Date aren't affected by this setting.

	Utc	Local	Unspecified
Program returns a time with a zone	Converted to UTC	Converted to the local time of the Data Sync machine	The time as written, ignoring the zone
Program returns a time without a zone	Taken to be UTC	Taken to be local time	The time as written
Data Sync writes	UTC, with Z	UTC, with Z	No zone and no conversion

Connection Library

Saving the connection to the Connection Library keeps the program, working directory, timeout, date handling and parameters in one place. Each project keeps its own DataSource and UpdateBatchSize. Saved connections appear under External Program Connections, with one entry for each datasource.

Running on a Schedule

When a project runs from the Run Tool or an Ouvvi agent, the program runs on that machine as that account. The program and any interpreter it needs must be installed there, and the account needs access to everything the program uses.

Troubleshooting

To find out what a program is doing, run it by hand from a terminal. The testing steps show how. Anything the program writes to the file passed in --logfile appears in the Data Sync Output log.

Problem	What to check
The program implements protocol version…	The program's info command must return "protocol": 1.
You must specify a value for the parameter(s)…	Enter a value for each required item in Parameters.
The 'read' command didn't finish within…	Increase TimeoutSeconds or make the program faster.
The 'read' command failed with exit code…	The rest of the message comes from the program.
The drop-down for DataSource is empty	Check that Parameters are set, then look in the Output log.
It works in a terminal but not in Data Sync	Check the working directory, the PATH, and which account Data Sync runs as.
The same rows are updated on every sync	The program returns values differently from how they were written. See How a Program Must Behave.
Text has the wrong characters	Every file must be UTF-8. Windows PowerShell 5.1 and Python on Windows don't use UTF-8 by default.
No progress is shown until the end	The program must write rows to the output file as it goes, and flush it regularly.