User types and segregation of duties

The application user type behaves very differently and has completely different requirements to the data science user type.

Application user type

  • Mission critical
  • Time sensitive, losing money every second offline
  • Requires high performance
  • Requires high availability
  • Requires specific functionality
  • Applications are tested and certified

Data science user type

  • Not mission critical
  • Not time sensitive, can withstand small outages
  • Requires good performance
  • Requires good availability
  • Requires specific functionality
  • Requires highly flexible access to data and functionality.
  • Requires full power of the q language
  • Wants to upload new data on demand
  • Wants to save results for later reuse.
  • Wants to share results and code with other data scientists.
  • Wants to connect any data science tools they wish (r, matlab etc)

Applications and data science are separated in Refinery both logically and physically.

Segregation

App Prod (and DR) are for application users. Sandboxes are for Data Science!.

Sandboxes

Sandboxes can be either real-time or historical only (as in diagram) and run on separate hardware to App production.


Sandboxes

Data nodes are multi-tenant; user Sandboxes are single-tenant.


Each user has their own UNIX account. This provides:

  • Security and stability: no user can access another’s files or kill another’s process.
  • Flexibility: each user manages their own space and home directories.
  • Simple administration: standard UNIX user permissions, nothing complicated.
  • Separation of the underlying data nodes from the user space at OS level. Users cannot access or damage the underlying data nodes.