ETL Refresh

Refresh & other DataSelf ETL processes:

DataSelf ETL processes you can monitor with Window's Task Manager:

• dsrefresh.exe The main refresh program.
• dspackage.exe
• dsmngto.exe

NOTES:

• In Windows Task Manager select Show processes from all users.
• In Windows Task Manager click the Image Name column header to sort the processes alphabetically.

See also: How to Abort a Refresh.

Log Records for Scheduled ETL Refresh Jobs

Scheduled ETL Refresh Jobs are configured in refresh batches by DataSelf ETL's Refresh Wizard.
The refresh batches are configured by the Refresh Wizard to be run under the control of the job scheduler feature of SQL Server Agent. SQL Server Agent is a service of Microsoft SQL Server. By default, SQL Server Agent is up and running whenever SQL Server is running.

SQL Server Agent (Scheduler) Log

Microsoft SQL Server's SQL Server Agent is the job scheduler that is configured by the Refresh Wizard.

ETL Log File

If the DataSelf ETL refresh software was successfully started messages will be posted to the ETL Log File. The ETL log file is one of the first places to look for clues if a scheduled refresh fails to complete successfully.

For more about how to find and read the ETL log file see View ETL Log.
For more about configuring ETL log file options see ETL_Settings_and_Configuration#Configuring_ETL_Logging.

ETL Refresh Diagnosis & Repair

_{(Trouble Shooting)}

• ETL Refresh Diagnosis & Repair - Troubleshooting, diagnosis & repair of ETL issues

How to Abort a Refresh

To abort a ETL refresh batch that is still running:

1. Open the Windows Task Manager.
2. Click the Process tab.
3. Right-click on dsrefresh.exe.
4. Click End Process Tree

In most cases aborting dsrefresh.exe will stop the entire DataSelf refresh process.

Abort Refresh Troubleshooting:

1. :IF the ETL refuses to open or you experience other ETL related issues, then rebooting the server and/or killing the other ETL processes is a next step.
2. Reboot the server if possible.
3. If rebooting is not possible yet, kill the following apps if they are still running: dspackage.exe and dsmngto.exe .

Network Speed Transfer Stats

It might take a while to bring large data volumes across a network. The following are data transfer speed stats we usually experience in our clients' deployments.

• Connection via ODBC (ex.: Providex, Pervasive, FoxPro): 10MB/sec to 20MB/sec.
• Connection directly to MS SQL Server: 20MB/sec to 1GB/sec. The wide variance is because some clients have old servers with slow networks.
• Connection via OData: peak at 200KB/sec to 500KB/sec. The transfer is not continuous but done in short bursts at every 5 to 20 seconds. OData data transfer is turtle slow, and because of that, we set up incremental data refresh for tables with large data volumes.

If your data transfer is slower than the stats above, there are likely network issues to be addressed. Please contact us to discuss.

Stats were collected using Windows Task Manager (Performance tab, network section). These data transfer rates apply regardless of what tools we used to extract the data (such as ETL, Tableau, Power BI, Crystal, SSRS and Excel).

Keywords: refresh, import, update, extract, load, data warehouse, dataself