Refinery request/report generation¶
Refinery has been integrated with a number of upstream data sources. This documentation describes those sources and how to interact with them.
TRTH framework¶
The TRTH Framework has been developed to facilitate the following scenarios:
-
API level requests.
-
Manual uploads into the platform from downloads via the DSS GUI
The Framework includes the following components:
-
DC Report Management Framework
-
Java API client
-
RFA Velocity C++ Feed
-
Daas_processingDaemon
-
Spawned feed monitor
-
Spawned tickerplant
-
Spawned persistence engine
-
Daas_dataMergeDaemon
-
Spawned merge slaves
-
Daas_hdbCopy Process
Guidelines¶
-
It is good practice to do a test from start to finish testing the integrity of the download and data downloaded. This would preferably be done once a new exchange map is being used in a request.
-
A TickHistoryRaw request must only contain RICs within a single asset class.
Request processing strategies
1. The symbol list (RIC list) for each submission should be:
-
the full list of RICS for the asset class
-
separate submissions for each asset class.
Current issues and resolutions details
-
TRTH has no facility to provide a Refresh message (complete snapshot image). Without this data the RFA Velocity Feed cannot process the raw TRTH data. The Refresh message for each RIC takes place over the weekend and so our request date ranges are altered to include the previous Saturday. This means that the TRTH granularity is defaulted to a WEEK, unless TR provides a more suitable solution via the API.
-
The requests are asset class specific, meaning the API client will flag an error if the input RIC list are not all the same asset class. This limitation is due to the RFA Velocity Feed only being able to process one asset class at a time.
-
DSS validation issue: The Historical Reference report template returns an empty string for the asset category in some instances. This has resulted in a workaround being added to the TRTH Framework where the user can remove the validation by setting applyValidation to false and specifying the specific asset class in the assetClass parameter.
-
There is a known issue where a timeout seems to occur when the trthClient is downloading a request. The file does not get downloaded in its entirety, resulting in an end of file error when it is processed by the feed. Logic to control this has been implemented, but if it does fail then the file must be manually downloaded from the DSS GUI.
How to add a new exchange map¶
-
Go to the TrthExchangeMaps directory
a. "cd" to
delta/delta-bin/software/TrthExchangeMaps_x_x_x/ExchangeMaps -
SCP the folder to the directory
a. Note it must be in the following naming convention:
exchangeMap-<DATE>-<Version>b. At minimum the folder must contain the following files:
i. enumtype.def ii. exchgmap.txt -
Restart/Update processes:
a. The
DS_launch_REPORT_Aworkflow will need to be stopped and restarted so that TrthClient can pick up the new exchange map.b. Update/Create soft-link and then stop and restart the RFAVelocityFeed so the new exchange map can be picked up
How to update the RDM field dictionary file¶
This file is additive and so the feed must always point to the latest copy. TR will release a new version with the exchange maps and the following details outline how to and where to replace the current copy.
-
Go to the RDMFieldDict directory
a. "cd" to
delta/delta-bin/software/TrthExchangeMaps_x_x_x/RDMFieldDict -
Delete the old RDMFieldDictionary file
a.
rm -rf RDMFieldDictionary -
SCP the new RDMFieldDictionary to the current location
a. The file must be named:
RDMFieldDictionary -
Stop and restart the RFAVelocityFeed so the new file can be picked up
Update/create soft-link for the RFA velocity feed¶
- Create a latest directory
mkdir -p latest
The -p applies to the case where the directory already exists, in which case use the command below to clear the directory:
rm -rf latest/*
-
Copy the latest exchange map files into the latest directory
a.
cp exchangeMap-2017.04.29-v8.0/* latest/ -
Stop and restart the RFAVelocityFeed so the new files can be picked up
How to update configuration files¶
Configuration files:
-
TRTH/DSS Credentials
-
TRTH Client Proxy Parameters
Note
If you do not need to specify any proxy parameters, then the configuration can be left blank. This is a likely scenario if the deployment box has open access to the internet.
- Go to the Report/Request Manager dashboard and the select the second tab: TRTH/DSS Config Editor to inspect the configuration files.
-
Update/Edit the configurations as necessary.
a. To enter a new value double click on the cell
b. The box will appear orange after you enter your value and click out of the box
c. The dashboard will notify the user that there are un-saved changes by highlighting them in orange.
d. To save the changes select save at the bottom of the screen.
Editing the RFA velocity feed parameters¶
The feed parameters for replaying TRTH RAW files can be updated if
required. The following configuration file must be edited on Delta
Control UI: .daas.cfg.trth.rfaVelocityFeed
The two parameters that can be edited freely are:
-
playback-delay: accepts millisecond value - defaults to 1 -
playback-rate: accepts millisecond value - defaults to -1 (meaning publish as fast as possible)
Manual uploads into the system¶
The upload file must contain the following naming convention:
<USER-ID>.<REQUEST-ID>_<REQUEST-NAME>_<DATE-TIME>_<EXCHANGE MAP FOLDER NAME>_<FILE-PART>_<ASSET-CLASS>.<DSS-DATE-TIME>.<DSS-SERVER-DETAILS>.csv.gz
Example:
9010723.12_TRTH.RAW.EQ_20170510T193000.842_exchangeMap-2017.04.29-v8.0_1-of-1_RFA-EQUITY.20170510.203039.2000000000402546.tm01n01.csv.gz.2017.05.10T233008.720
Note
No underscores can be included in the name other than the defaults.
Note
The request must start on a Saturday in order to obtain a refresh message for each RIC.
Note
The RICs must all be of the one asset class.
In order for the file to be uploaded into the system, it must be placed in the following directory:
TRTH_TICKHISTORYRAW=${DELTADATA_HOME}/DaaSData/trth/downloads/TickHistoryRaw
How to use the report manager for API-level requests¶
The Report/Request Manager is a self-contained component for creating and scheduling reports. Its functionality has been extended to allow the creation and scheduling of TRTH/DSS requests. The underlying structure of the request is created from the Template Editor. The report template must be created to allow requests to be submitted.
The default report templated shipped with Refinery is called: TRTH.DSS.ReportTemplate.
-
Target Engine:
cg:ds_gw_report -
Report Analytic:
.daas.trth.request -
Report Actions:
.daas.trth.requestAction
Note
Do not edit this default template!
Creating a new Report Instance
From the Instance Editor, click Create a new Report Instance.
A report instance must be created from the TRTH.DSS.ReportTemplate report template. This represents an individual TRTH/DSS request.
This will take the details from the Template and apply them to the current Instance.
Now select the newly created instance from the dropdown menu.
Note
The Details Tab will appear blank as the parameters must be entered in the Advanced Details tab.
In the Advanced Detailsb tab, the Action description can be changed from the default used in the Template Editor. To customize Report Parameters from Template defaults, uncheck the use default and enter a value.
Timeout is set to default of zero ("infinite"). Changing the value will set a millisecond timeout for the report.
Enter the Request Parameters
-
Target Engine:
cg: ds\_gw\_report -
Timeout: This can be any suitable number or left unspecified (e.g. 3600000 - 1 hour in milliseconds). The process timeout is in seconds. The report will be cancelled if the execution time exceeds the timeout value. Zero indicates no timeout.
-
Report Action:
.daas.trth.requestAction(This should not be changed!) -
Action Parameters: Uncheck the Use Default check box to enter a value
Note: .z.d is the current UTC time and .z.D is the system's local time. In a system running on UTC, they are equivalent. In a system that is set up for a different time zone (i.e. non-UTC), these will give different results.
| Parameter | Valid inputs | Comments |
|---|---|---|
| requestType | DSS: CorporateActionsStandard Composite TRTHv2: TickHistoryRaw TickHistoryMarketDepth |
|
| startDateTime & endDateTime | - .z.D-<N> e.g. .z.D-1, .z.D-2*- .z.D-1 is a valid input return yesterday’s data. - You can also specify the date time explicitly in the following format: 2016-11-28T16:34:02.034 |
- If input is not of the .z.D-<N> type, the explicitly entered date-time must be in .z.Z format (http://code.kx.com/wiki/Reference/dotzdotz) |
| ricType | - allRics - eqL2 - eq - fi - futL2 - fut - fx - idx - mn - lisOpt - ricChain - eqL2Legacy - futL2Legacy - <watchlist_name> |
- Only one input is accepted. - The RICs are pulled from the watch lists stored in the platform. - If “ricChain” is entered then the ricChains will be taken from either manualRicInput or csvUpload. - This input will not be overwritten if a value has been specified for manualRicInput or csvUpload. - <watchlist_name> is an underscore-separated list describing a particular feedhandler process. The list comprises of:- region- feedType- dataSource- datatype- assetClass- procNum- env - e.g. emea_rfavelocity_tr_marketData_eq_0_a |
| manualRicInput | Only input accepted is a symList (eg. VOD.L`IBM`XXX) |
- It can accept any number of inputs - This input will not be overwritten if a value has been specified for csvUpload. |
| csvUpload | - e.g. testUpload.csv |
- The user must specify the full file name including the .csv extension. - The file must be uploaded via the Client Upload directory by selecting the trth directory. - It must be a CSV file and it will read in the first column names sym. |
| applyValidation | - True- False |
- If true the trthClient will apply validation to the request and determine the asset class- This field defaults to false. - Field must be set to false for moneyMarket asset class |
| assetClass | - EQUITY- FUTURE- FOREX- FIXED- INDEX- MNYM- LISOPT |
- If applyValidation is false then the user can specify the asset class. - The inputs must be capitalized. |
| marketView | - L1- mbpL2- legacyL2 |
- Market View must be L1 for TickHistoryRaw. - Market View must be mbpL2 or legacyL2 for TickHistoryMarketDepth. - If not specified it Defaults to L1. |
Set report scheduling
This can be a one-off report or a frequently scheduled report:
One-off schedule:
Daily schedule:
Report permissions
Note
Do not change or set any values on the Permissions tab.
Save the Report Instance
The final step is to run the report. If scheduled, the report will run at the defined time. However, you can run reports directly from the Instances tab by clicking Run.
The status of the report can be found under the Run Info Tab. If successful, the log will show a green success or a red fail.
Use the Date, Report and/or RunID filters to search. Remember to click Submit when running a filter.
Logging
-
The logging is self-descriptive. It will log the processing of the request from start to finish. Errors will be denoted as "FAILED".
-
The TrthClient logs all its messages with the prefix TRTH-Java-Client.
-
The trth framework within kdb signals which process is currently handling the request.
Logging Explained:
Initializing report
Initializing - parameters and report ID generated
Got handle for ds_gw_report
Function defined : .daas.trth.request
Sending function to remote process
Result received from remote process
Running actions against report
Running action .daas.trth.requestAction
Report Action Update: requestType Validated
Report Action Update: dateTimes Validated - startDateTime:2017-06-13T00:00:00.000 endDateTime:2017-06-13T23:59:59.999
Report Action Update: RicList Validated
Report Action Update: Validation being applied to request: false
Report Action Update: Market View Validated:L1
Report Action Update: Success -> Sending request to TRTH Java Client
Action .daas.trth.requestAction completed successfully
Stats dictionary returned from action .daas.trth.requestAction
Report complete
At this stage in processing, the report manager process will send the request to the Java trthClient
TRTH-Java-Client: Request {x_x_x_x_1-of-1} accepted by TRTH Java client
TRTH-Java-Client: No Validation logic being applied to requestID: x_x_x_x_1-of-1
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Deleted schedule request files on API with the same names as new request
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Instrument list created
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Instrument list identifiers appended to instrument list
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Report template created
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Report request submitted
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Completed scheduling request
TRTH-Java-Client: Request {x_x_x_x_1-of-1_RFA-EQUITY} is being downloaded by the TRTH Java client
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Request report status received
TRTH-Java-Client: Downloaded 100% of file: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz.notes.txt
TRTH-Java-Client: Downloaded File: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz.notes.txt
TRTH-Java-Client: Downloaded 25% of file: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz
TRTH-Java-Client: Downloaded 50% of file: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz
TRTH-Java-Client: Downloaded 75% of file: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz
TRTH-Java-Client: Downloaded 100% of file: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz
TRTH-Java-Client: Downloaded File: 9010723.x_x_x_x_1-of-1_RFA-EQUITY.XXX.csv.gz
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Report files downloaded
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Deleted schedule request files on DSS API
TRTH-Java-Client: RequestID: {x_x_x_x_1-of-1_RFA-EQUITY} - Completed downloading request
At this stage in processing, the RAW file has been saved to trth/downloads/.. where the FW will then pick up the file
emea_tr_trth_fw_0_a.1: File part 1-of-1 has been picked up by processing daemon and is about to be processed by the Feed
At this stage in the processing, daemon has ran an instance of the feed and it is currently being processing the RAW file
Daas_processingDaemon_a.1: The Feed has finished processing in the File Watcher
End of processing
Use Cases
-
If a feed goes down for an hour on a Monday, TRTH can be used to back fill that data; then we can make the request for that day and the data will be upserted into the database. The data should match exactly as the feed used the original exchange time and not the time it was received in the feed.
-
If the data is corrupt or if the data was played back using an outdated exchange map, then these dates must be deleted from the database and then TRTH must be run.
-
A TickHistoryRaw request can be run nightly to get the previous days data.
Backfill
- A single request can be made to back fill all data for a system.
Example Backfill report instance:
Obtain T-1 data
-
By scheduling a job to run daily and setting the start and end times to .z.D-1, the request will retrieve the previous day's data.
-
Note. Due to the current refresh message constraint, the daily request will behave in the following pattern
- (This is due to instrument IMAGE record only being published by TR on SATURDAY, with all intermediate records being deltas):
-
i. Monday Request -- Retrieves Sat, Sun, Mon
ii. Tuesday Request -- Retrieves Sat, Sun, Mon, Tue
iii. Wednesday Request -- Retrieves Sat, Sun, Mon, Tue, Wed
iv. Thursday Request -- Retrieves Sat, Sun, Mon, Tue, Wed, Thu
v. Fri Request -- Retrieves Sat, Sun, Mon, Tue, Wed, Thu, Fri
Example report instance:
Note
If running multiple asset class within the platform, separate report instances for each asset class must be created and scheduled as a request can only be made per asset class.
Recovery
-
Recover a day's data by overwriting the data on disk.
-
The Report Instance name must contain the word "recovery" to save the data down and overwrite any previous data that existed for the report dates only.
-
The RICs that were subscribed to on that day should be contained in the "watch lists"; these RICs can be pulled into the TRTH request by specifying the corresponding asset type in the "ricType" parameter.
-
By running this request the only data that will be contained in the HDB date after the trthEOD has run will be the new data from the request.
-
Due to this logic the request can only be made for the entire day.
-
TRTH EOD Logic
The end-of-day logic performs two distinct functions based on the request:
-
If the Report Instance name contains the word "recovery" or the core HDB (where the data is being copied to) is empty, then it will use "set" to save the date down and overwrite any previous data that existed for the report dates only.
-
Any other condition will use "upsert" to upsert the data into the core HDB.
-
If is critical to note that if you save down a request with the same data, then you will have duplicate data on disk.
-
The reason this occurs is because data is not keyed on disk in a HDB table.
-
How to run TRTH EOD manually
Open the daas_mergeDaemon IDE within Delta Control and run the
following command:
.daas.merge.eodMoveToMain[];
Then proceed to check the data processing dashboard and daas_mergeDaemon logs.
Datascope select (DSS)¶
The data is loaded in via the client upload file watcher to the secmaster, and then distributed throughout the system to the valid processes.
The following report instances should not be altered in any way.
Corporate actions¶
The CorporateActionsStandard request is scheduled to download all of the reference for all Equity RICs contained within the Equity watchlist each night. It is deployed preconfigured as part of the Refinery platform:
- Its report instance name is: DSS.CorAx
Reference data¶
The Composite request is scheduled to download all of the reference for all RICs contained within the watchlists each night. It is deployed preconfigured as part of the Refinery platform:
- Its report instance name is: DSS.Composite
DSS EOD¶
The EOD for both Composite and CorporateActionsStandard requests are saved down intraday. The emea_tr_clientFileUpload filewatcher process reads in the file and saves it down to the associated HDB.
Start of day report¶
The Start of Day Report runs once a day at 8 a.m. and gives an overview
of the status of the system at that time. The results of the above
checks are collated and a CSV file is created using results where the
status is not OK. This CSV file is attached to an email sent to the
monitoring email address (assigned in configuration parameter
DS_EMAIL_ADDRESSES).
If there are no issues with the system, an email will still be sent with the subject "SOD Report: ALL OK".
The time at which the report will run is configurable through the
Report/Request Manger dashboard. In the instance Editor tab select
SODReport from the dropdown and then under the scheduling tab you can
edit the Start Date to the preferred time
RIC chain reduction¶
Background¶
Downloading of futures contract RICs at Refinitiv via DataScope and the DataScope API use the ‘RIC chain’ construction of ‘parent’ and ‘constituent’ RIC to create a list of RICs that relate to the futures contracts for the months ahead for the asset class in question.
This can produce enormous volumes of downloaded data for a client, so there is a constituent RIC reduction capability built into the TRTH client for futures levels 1 and 2. This may be invoked with the addition of an additional key value pair added in the TRTH configuration.
With agreement from the client, they may wish to reduce their volume of report downloads using this mechanism. It is also useful for testing to reduce volumes while downloading the full quota in the watchlist. This can be also be a useful tool for disk sizing.
The key for this pair is: numRICChainMonthsToProcess. Its value is an integer. E.g.
It is set in the DS_LAUNCH_COMMANDLINE_PARAMS:trthClient. N.B. The TRTH client must be restarted in order to use this parameter as it is a JVM argument and included at runtime when the client starts. Also note that if the key value pair is absent from the configuration, no RIC chain reduction will occur and the log will report that it has not been found.
How it works¶
Constituent RIC chains follow a format that identifies them to the month to which the contract relates, typically following the pattern below:
<ricPrefix><monthLetter><monthNumber>
Using this mechanism, the futures contract months ahead can be reduced by: - Retrieving the list of constituent RICs - Sorting into their correct month order - Reducing the number to download based on the parameter passed in
In order for this to work, the constituent RICs for a RIC chain must follow a pattern that is capable of being sorted. However, there are exceptions to this rule, meaning that the constituent RIC does not pertain to a month, but is some other indicator, a total for example. When these special RICs are in the constituent RIC chain, they cannot be used for RIC reduction. This is because the DataScope API does not guarantee to return the list in the correct order; once received they must be sorted and they can only be sorted if they all conform to the pattern.
RIC chain 0#CUU: is an example of a constituent RIC list that will sort as its constituent RICs follow the expected pattern:
CUUU1 CUUV1 CUUX1 CUUZ1 CUUF2 CUUG2 CUUH2 CUUJ2 CUUK2 CUUM2 CUUN2 CUUQ2 CUUU2 CUUV2 CUUX2 CUUZ2 CUUF3 CUUG3 CUUH3 CUUJ3
RIC chain 0#FCPO: is an example of a constituent RIC list that will not sort as its constituent RICs contain the FCPO-TOT RIC in the chain that does not follow the pattern. No special handling has been added for this; if any of the list of RICs does not conform to the pattern in any way, it is returned untouched by the sorting mechanism.
FCPO-TOT FCPOU1 FCPOV1 FCPOX1 FCPOZ1 FCPOF2 FCPOG2 FCPOH2 FCPOJ2 FCPOK2 FCPOM2 FCPON2 FCPOQ2 FCPOU2 FCPOX2 FCPOF3 FCPOH3 FCPOK3 FCPON3 FCPOU3
If the RIC chain is capable of being reduced, and the numRICChainMonthsToProcess is set, the chain will be reduced. Using the example of the 0#CUU:RIC the list returned will be:
CUUQ1 CUUU1 CUUV1 if the parameter is set to 3 and:
CUUQ1 CUUU1 CUUV1 CUUX1 CUUZ1, CUUF2 if the request is done within the same month and the month boundary has not been crossed and the parameter is set to 6 – see Month Boundary below.
Month boundary¶
It is inherent in the nature of a future that it has contracts in the months ahead. If the processing for any RIC chain crosses a month boundary, then the earliest month that is now available will be used, as past months drop off the bottom of the stack. In this example, CUUQ1 will drop off and the first RIC in the list will beCUUU1.
Using the example above for 0#CUU: the list returned will be:
CUUU1 CUUV1 CUUX1 CUUZ1 CUUF2 CUUG2
Location in code¶
The method that does this work is called:
getSortedConstituentRicsSubset
and is located within the com.fd.daas.trthClient.DssMessaging.java class of the TRTH client.
Logging¶
Logging is to be found in the
Example logging output is as follows (numRICChainMonthsToProcess is set to 3)
Successfully reduced:
INFO com.fd.daas.trthClient.DssMessaging - Current ricChain - 0#SM:|
INFO com.fd.daas.trthClient.DssMessaging - ricChain - 0#SM: has constituent RICs - [SMF2, SMF3, SMH2, SMH3, SMK2, SMK3, SMN1, SMN2, SMN24, SMN3, SMQ1, SMQ2, SMQ3, SMU1, SMU2, SMU3, SMV1, SMV2, SMV24, SMV3, SMZ1, SMZ2, SMZ24, SMZ3]
INFO com.fd.daas.trthClient.TrthUtils - RIC Chain ricPrefix: SM
INFO com.fd.daas.trthClient.TrthUtils - Only word characters found in RICs, further checking to confirm conform to
Non-standard naming found; original list returned:
INFO com.fd.daas.trthClient.DssMessaging - numRICChainMonthsToProcess is set and its value is : 3
INFO com.fd.daas.trthClient.DssMessaging - Current ricChain - 0#GF:|
INFO com.fd.daas.trthClient.DssMessaging - RIC Chain - 0#GF: has constituent RICs - [.GFTVO.FX, GFQ1, GFV1, GFZ1, XAUFIXAM=]
INFO com.fd.daas.trthClient.TrthUtils - RIC Chain ricPrefix: GF
INFO com.fd.daas.trthClient.TrthUtils - Non standard characters found in the following RICs : [.GFTVO.FX, XAUFIXAM=]
INFO com.fd.daas.trthClient.TrthUtils - RIC chain 0#GF: constituent RICs original list [.GFTVO.FX, GFQ1, GFV1, GFZ1, XAUFIXAM=]
INFO com.fd.daas.trthClient.DssMessaging - 5 original constituent RIC list : [.GFTVO.FX, GFQ1, GFV1, GFZ1, XAUFIXAM=]
INFO com.fd.daas.trthClient.DssMessaging - 5 processed constituent RIC list [.GFTVO.FX, GFQ1, GFV1, GFZ1, XAUFIXAM=]
INFO com.fd.daas.trthClient.DssMessaging - {2189_2021.07.27.FUT.L1.GF.manualrun_20210728T095301.966_exchangeMap-2019.05.13-v8.2.5.1} - Number of RICs returned from ricChains: [5]
Conclusion¶
Using this mechanism, the report downloads for the client it was produced for were reduced by approx. two thirds, speeding up delivery for the client when they had identified that they did not need so many months ahead for historical data for their analysis of Futures.
N.B. This will vary depending on the proportion of the RIC chains that conform to the pattern that will allow reduction to take place; if none conform to the pattern, then no reduction will take place.