Synchronise a SharePoint Folder with the File System
In this guide we show a method to use Simego Data Synchronisation Studio to copy files from a SharePoint Document Library to a Local Folder. Data Sync will automatically detect which files have changed in SharePoint and copy them to the local folder so that the Directory Structure matches SharePoint.
Begin Setup
Before you start you will need the following:
- Windows 10 or Windows Server 2016-2019
- Simego Data Synchronisation Studio
- SharePoint Document Library on SharePoint Online
- Local Folder
Connect Source to SharePoint Document Library
Data Sync can connect to SharePoint Online or an OnPremise SharePoint Document Library. The following details are for a connection to SharePoint Online.
For SharePoint Online you can connect either with Modern Authentication (Oauth2) or your Azure AD credentials if your organization still supports it.
In Data Sync click the Connect DataSource icon on the source to open the Datasource connection window. Then choose the SharePoint Online connector under the Microsoft SharePoint category.
Enter the URL to your SharePoint Site this the URL to the root of your site. Examples of the URL are listed below.
https://company.sharepoint.com/https://company.sharepoint.com/subsitehttps://company.sharepoint.com/subsite/subsite2
Choose the Authentication Method to connect to your SharePoint Site. Choosing the default Simego OAuth Application will use a pre-registered multi-tenant Azure AD Oauth2 Application to connect to your SharePoint Online instance.
You can learn more about supported authentication methods here. Click the Authorize Connection button, this will open your default browser and ask you to sign in to Azure AD and grant access to allow Data Sync to connect to your SharePoint instance.
Once the Authorization is complete go back to Data Sync and choose your SharePoint Document Library from the list and click Connect & Create Library Connection. This will create a connection in the Connection Library, you only need to do this once per site as you will be able to access all the lists and libraries in that site from the connection library window.
Once your connected to the SharePoint Document Library, Data Sync will load the library data schema into the source connection and create a default schema map.
Connect Target to Local Folder
The next step is to connect the target to the local folder where you want your SharePoint documents copied to. This is the folder which Data Sync will keep synchronised with the SharePoint Document Library.
This is a 1-Way sync where document changes in SharePoint are copied to the folder. The timestamp of the change in SharePoint is also applied to the local file when it is copied so that they remain the same.
Map Source to Target
The schema map defines the mapping of the source to the target. For this SharePoint to Folder synchronisation we simply want to map the filename to the target along with the Created and Modified dates.
Data Sync includes a default mapping rule for this type of project so that the mapping can be created automatically for you. When the mapping rules window is displayed select the SharePoint Document Library to Folder mapping rule and click Apply.
The schema map should then be shown like this mapping URL Path from SharePoint to FullFileName on the target.
The mapping rule also sets WebFriendlyPaths to true on the target connection so that the local file paths use a forward slash (/) rather than a backslash (\).
Compare and Synchronise
Next we use Data Sync to load the list of Documents stored in SharePoint and in the Folder and compare these to work out which documents need to be ADDED, UPDATED or DELETED so that the local folder matches SharePoint.
Delete operations are always disabled by default on new Data Sync projects to enable Delete you need to set the EnableDelete property in the target connection to true.
To start the synchronisation process click the Run Compare button on the main toolbar. This will start the process to load the file list from SharePoint and the local folder. At this point only the data is loaded and compared to create the changeset, no files would have been written at this point.
We can now review the changes that are to be made before running the copy operation to copy the files from SharePoint to the local folder.
Clicking on the 19 Record(s) Only in Source (A) ADD (19) to Target (B) row in the changeset shows us the file list to be ADDED to the folder. We can the review this and once we happy to continue click the Synchronise button to start the copy process.
Click the Start button to start the actual synchronisation.
This completes the synchronisation, if you then look in the target folder you will see that your files have been copied. Any folders in SharePoint have also been created in the local filesystem.
You can then go and add additional documents to SharePoint or edit existing documents. When you then compare and synchronise again you will see that only those documents which were changed in SharePoint are copied.
Automate and Schedule
Now that your project is configured you can save it and then schedule it to run via the Data Sync Run Tool, Windows Task Scheduler, or Ouvvi Automation Server.
You may want to automate your project so that it runs on a regular basis, to do this you can use either the Run Tool or Ouvvi.
Ouvvi gives more options for scheduling, with both time and event based triggers. So you could configure a trigger to start your project whenever a change is detected in your SQL table or SharePoint List. Note that this requires using a modified date time stamp in your SQL table.
Alternatively you can use the Run Tool to schedule the project to run using Windows Task Scheduler.
Run via the Run Tool
The Run Tool is an additional program that comes linked to Data Sync and enables you to build out your data integration jobs.
You can use this to group Data Sync projects that need to run in order and add additional step types such as adding an status report to email you when the project has run and if there were any failures. Each step is run in the order it appears in the list and you can apply conditional rules so that the next step can run if the previous one succeeded or had data changes for example. To find out more please see our Run Tool Documentation.
To open your project in the Run Tool, in your Data Sync project open the Tools menu and select Open in Run Tool.
You can then click onto the green Run button to check it runs as expected. If there are data changes you will be provided with a count of how many items were added updated or deleted
Run via the Command Line
Another option is to Run the Run Tool project or single Data Sync project from the command line. To do this pass the path to the project file like this:
Run Tool Project
%DATASYNCINSTALLPATH%\Simego.DataSync.Run.exe -execute "D:\DSProjects\myproject.dsrun"
Data Sync Project
%DATASYNCINSTALLPATH%\Simego.DataSync.Run.exe -execute "D:\DSProjects\myproject.dsprj"
Automate with Windows Task Scheduler
You can then schedule your Run Tool project to run when you need it to using Windows Task Scheduler. For the full details on how to do this see our Task Scheduler Documentation.
Automate with Ouvvi
An alternative option, with more scheduling capabilities is to use Ouvvi Automation Server. Ouvvi enables you to fully schedule and manage all of your Data Integration Projects, from Data Sync projects to SQL Statements to Powershell Scripts.
It provides full logging and documentation capabilities to fully manage your integration operations.
To find out more see our Ouvvi Documentation or send us an email.
Additional Configuration
Listview Threshold
SharePoint has a Listview Threshold limit of 5000 items in a view, while a Document Library can store upto 30 million documents. Data Sync can read more than the 5000 item limit provided there is no filter or sort on the view. Ensure that you set the sort column to the ID column and remove any filters for Data Sync to read more than the Listview Threshold limit.
Filter Documents
You can filter documents on the source through the use of a SharePoint View. However be aware of the Listview Threshold Limit of 5000 items, if you have more than 5000 items make sure that you add a index in SharePoint for the columns you are using in the filter.
Once you have defined the view in SharePoint you can select this view from the View list on the source SharePoint connection.
Connector Documentation
We have further information for the SharePoint connector in our documentation here.
If you want to synchronise a File System to a SharePoint Document Library check out our other guide here.