Program Protocol

This page is a guide to the protocol between the External Program Connector and your program. For every rule and edge case, see the full specification:

Download the External Program Connector Protocol, version 1

tip

The specification is also what you give to an AI assistant. See Build a Program with AI.

Commands

A program is a command-line tool with one subcommand per operation. prog stands for your program, or the interpreter and script.

prog info                                    --outfile info.json
prog datasources                [--config c] --outfile datasources.json
prog schema <datasource>        [--config c] --outfile schema.json
prog read   <datasource>        [--config c] [--schema-file schema.json] --outfile data.jsonl [--max-rows N]
prog write  add|update|delete <datasource> [--config c] [--schema-file schema.json] --infile changes.jsonl --outfile results.jsonl

common options: [--format jsonl] [--errfile error.json] [--logfile log.txt]

Programs that support stopping also receive --stop-file <path> on read and write.

Command	Purpose	Output
info	The protocol version, capabilities and parameters. It is called first and must work with no configuration.	JSON document
datasources	The tables or datasources the program exposes.	JSON document
schema	The columns of one datasource.	JSON document
read	The rows of one datasource.	JSON Lines, one row per line
write add, update or delete	Applies changes to one datasource.	JSON Lines, one result per change

How the Program Is Run

• Data Sync starts the program directly, once per operation, and passes all data through files with absolute paths. Files are created in a private temporary folder and deleted afterwards.
• Nothing is sent to stdin and stdout is ignored, so you can print anything for people running the program by hand.
• All files are UTF-8. Property names are case-sensitive.
• Parameters arrive in the JSON file named by --config, as string values. For testing by hand, a program should also accept DATASYNC_PARAM_<NAME> environment variables, with the name in upper case. If --config is passed, ignore the environment variables.

Describing the Program

info declares what the program can do and which parameters it needs:

{
  "protocol": 1,
  "name": "CSV Folder",
  "version": "1.0.0",
  "supports": {
    "datasources": true,
    "maxRows": true,
    "write": ["add", "update", "delete"],
    "stop": true
  },
  "parameters": [
    {"name": "DataDir", "description": "Folder containing the CSV files.", "required": true, "secret": false}
  ]
}

Leave write empty or out for a read-only program. Set datasources to false for a program with a single datasource, which Data Sync then calls default.

datasources lists the names that Data Sync passes back to the other commands:

{"datasources": [{"name": "Customers"}, {"name": "Products"}]}

schema describes the columns of one datasource:

{
  "datasource": "Products",
  "columns": [
    {"name": "ID", "data-type": "System.Int32", "unique": true, "allow-null": false, "read-only": true},
    {"name": "Name", "data-type": "System.String", "length": 100},
    {"name": "Price", "data-type": "System.Decimal"},
    {"name": "Updated", "data-type": "System.DateTime"}
  ]
}

Columns with "unique": true form the key. A datasource that can be written must have a key. Declare a key that the data store generates as both unique and read-only.

Reading

read writes one compact JSON object per line. When --schema-file is passed, return only the columns it lists. Write rows as you read them and flush regularly, so Data Sync can show progress.

{"ID":1,"Name":"Chai","Price":18.00,"Updated":"2024-03-01T09:30:00Z"}
{"ID":2,"Name":"Chang\nLarge bottle","Price":19,"Updated":null}

Writing

Each line of --infile is one change, with a key and the values to write:

{"key":null,"values":{"Name":"Ikura","Price":31.00}}
{"key":{"ID":7},"values":{"Price":30.00}}
{"key":{"ID":9}}

These examples are an add with a generated key, an update and a delete. An update contains only the columns that changed. A column present with null is set to null, and a column that is missing is left unchanged.

Each line of --outfile is the result of one change, written as soon as the change is committed. row is the zero-based line number of the change. An add returns the key of the new row.

{"row":0,"ok":true,"key":{"ID":78}}
{"row":1,"ok":true}
{"row":2,"ok":false,"error":"no row has the key ID=9"}

Apply each change on its own, so that one failure doesn't stop or undo the others.

Situation	Result
add with a key that already exists	ok:false. Never overwrite the existing row.
update of a row that doesn't exist	ok:false
delete of a row that doesn't exist	ok:true, so that a repeated run succeeds.
A read-only column in values	Ignore it.
A column the datasource doesn't have	ok:false, naming the column.
A value the data store rejects	ok:false, with the data store's message.

Stopping

If info declares "stop": true, the program receives --stop-file <path>. The file doesn't exist at the start. Check for it at least once a second. When it appears, finish the row in hand, close the output and exit with code 4. The program is killed if it hasn't exited after 30 seconds. A program that doesn't support stopping is killed straight away.

Data Types

data-type	JSON value
System.String	String
System.Int16, System.Int32, System.Int64, System.Byte	Number
System.Decimal, System.Double, System.Single	Number, or a string for values a double can't hold exactly
System.Boolean	true or false
System.DateTime	ISO 8601 string, converted by the DateTimeHandling setting
Simego.DataSync.Date	A date with no time, such as 2024-03-01
System.DateTimeOffset	ISO 8601 string with an offset
System.Guid	String
System.Byte[]	Base64 string
Newtonsoft.Json.Linq.JToken	Any JSON value, kept as JSON
Newtonsoft.Json.Linq.JArray	JSON array

null is always JSON null, never an empty string. Any other data-type makes the schema command fail, so use System.String for anything else.

Exit Codes

Code	Meaning
0	Success.
1	Error. Write {"error": "message"} to --errfile.
2	Usage error: an unknown command, a missing or unknown option, or an unsupported format. Also write to --errfile.
3	write only: some changes failed, and the results file says which.
4	Stopped because the --stop-file appeared. The output so far is valid.

How a Program Must Behave

Data Sync reads every row from both sides, matches them by key and compares them column by column. This means a program must follow these rules:

• Keys are unique and stable. Rows can't be matched if a key is duplicated or changes between runs.
• Each value is written the same way every time. A value that is returned differently shows as changed on every run.
• What is written reads back unchanged. If the data store rounds or trims a value, read should return the stored value and the schema should say so, for example with a length.
• read returns every row, or fails. Data Sync does its own filtering. If the program can't get all of the rows, it must exit with code 1. When the program is the source, a read that silently returns too few rows makes Data Sync delete the missing rows from the target.
• Changes can be repeated. A change with no result is tried again on the next run, so applying it twice must be safe.

Testing a Program

Everything Data Sync does can be done from a terminal. Parameters can be passed with environment variables:

export DATASYNC_PARAM_DATADIR=/data/csv

prog info --outfile info.json                     # exit 0; protocol 1
prog datasources --outfile ds.json                # exit 0; the datasources
prog schema Products --outfile schema.json        # exit 0; has a "unique" column if it can be written
prog read Products --outfile rows.jsonl           # exit 0; one JSON object per line
prog read Missing --outfile x --errfile err.json  # exit 1; err.json says what is wrong
prog read Products --outfile x --bogus 1          # exit 2

echo '{"key":null,"values":{"Name":"Test","Price":1.5}}' > add.jsonl
prog write add Products --infile add.jsonl --outfile res.jsonl

Check the exit code with echo $?, or $LASTEXITCODE in PowerShell. The specification has a full checklist in section 12.