Datascope Plus integration

DataScope Plus (DSP) is a reference data source from Refinitiv. Files are published to an FTP server every 15 minutes; Refinery downloads these files and converts them to a kdb+ database. There are four necessary steps to set up DSP integration:

  1. Configuring a report in the DSP GUI

  2. Setting up passwordless login to the FTP server

  3. Configuring Refinery to use the correct user id for DSP

  4. Configuring the report names that are downloaded by Refinery

To be able to set up DSP integration, the pre-requisites are:

  1. Valid DSP Account

  2. SFTP access from server where KX Refinery is installed to prodcddis.thomsonreuters.com

Configuring a report in the DSP GUI

Log into the DataScope GUI by going to <https://hosted.datascope.reuters.com/DataScope/> and enter your user credentials:

Click on the DataScope Select dropdown, the DataScope Plus option will be available on the left-hand side under Standard Solutions selection:

Click on the DataScope Plus Select dropdown, the Create option will be available on the right-hand side under Subscriptions selection:

Click on New Report:

On the Create New Report screen, choose the desired subscriptions from the left-hand side, pick a Report Name, pick a Report Description and finally select Reference as the Report Type:

Click on the Fields tab on the left-hand side of the Create New Report screen, search for the field to add, select the appropriate field by highlighting and click the Right Arrow button to move the field to the right-hand side and list of chosen fields.

The report should be configured to use the below fields.

Note

The order of the fields matters.

After completing all of the above steps and adding the desired fields, the final step is to click Save to save the report.

Configuring passwordless SFTP login to DSP file server

Log into the server which has the Refinery installed.

Navigate into your home directory and create a .ssh folder, use the -p flag on the mkdir command to not return an error if .ssh already exists.

Execute the code below on the server:

**a)**  \[kxuser\@s626207nj2sl03 \~\]\$ **cd \~**

**b)**  \[kxuser\@s626207nj2sl03 \~\]\$ **mkdir -p .ssh**

**c)**  \[kxuser\@s626207nj2sl03 \~\]\$ **cd .ssh**

Run ssh-keygen to generate the keys and check that they have been created, no passphrase is generated and all defaults are accepted.

Execute the code below on the server

**d)**  \[kxuser\@s626207nj2sl03 .ssh\]\$ **ssh-keygen**

Result from ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/data01/home/kxuser/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /data01/home/kxuser/.ssh/id_rsa.

Your public key has been saved in /data01/home/kxuser/.ssh/id_rsa.pub.

The key fingerprint is:

5a:3a:4b:55:c5:fa:cc:ac:06:7e:84:94:45:14:4d:ca kxuser\@s626207nj2sl03.uswhwk6.savvis.net

The key's Randomart image is:

SFTP onto the prodcddis.thomsonreuters.com

Command format $ sftp r<userid>@prodcddis.thomsonreuters.com:

Check if there is an authorized_keys file in the .ssh folder

e) sftp> ls .ssh

f) .ssh/authorized_keys

If there is an authorized_keys file, retrieve it by executing the below command and exit the SFTP session

g) sftp> get .ssh/authorized_keys

h) Fetching /.ssh/authorized_keys to authorized_keys

i) /.ssh/authorized_keys 100% 1259 1.2KB/s 00:00

j) sftp> exit

If there is no authorized_keys file, just exit the session and then create one by using the touch command

k) sftp> exit

l) [kxuser@s626207nj2sl02 ~]$ touch authorized_keys

Append the public key that you generated earlier to the authorized_keys file

m) [kxuser@s626207nj2sl02 ~]$ cat .ssh/id_rsa.pub >>authorized_keys

SFTP back onto prodcddis.thomsonreuters.com and put the modified authorized_keys file into the .ssh folder:

Test passwordless login by connecting back to prodcddis.thomsonreuters.com server; you should not be prompted for your password:

Configuration in Delta Control

Log into Delta Control UI:

Search for *dspConnection in the Search Bar; when .daas.cfg.dspConnectionDetails:<DEFAULT> becomes visible, double left click on the item:

The attribute screen will appear, add the appropriate user id to the Username section. Click Ctrl+S to save the attributes.

Configuring the report names that are downloaded by Refinery

Currently this is configured using a batch file for the sftp command that downloads the DSP files. You configure the batch file location in the configuration parameter .daas.cfg.dspConnectionDetails mentioned in the previous section.

By default, its location is

<install-dir>/packages/DaaSPlatform_xxx/interfaces/dsp/batchjob.txt

By default the batch file pulls all REF reports

[kxuser@s626207nj2sl02 dsp]$ cat batchjob.txt

mget *REF*

exit

This makes use of mget to pull multiple files at a time; the regex can be updated to pull specific file names instead of all files, e.g.:

[kxuser@s626207nj2sl02 dsp]$ cat batchjob.txt

mget my-report-number-1.REF*

mget my-report-number-2.REF*

exit

If setup has been successful, the logs of the emea_tr_clientFileUpload_a will contain messages indicating successful download from DSP and successful processing.

Possible errors are:

  1. Incorrect SSH key setup preventing passwordless login

  2. Mismatch between the expected columns and those contain in the DSP report

  3. Failed unzip of download file -- if this occurs, remove files from <install-dir>/delta-data/fileStore/secmaster/dsp/<current-date>