Persisting to S3

This example will generate data for the last 10 days on a new volume, and then start the Storage Manager and Data Access components with that existing volume for an initial import.

This example also sets up a very simple SP that generates more data every minute, providing an example that queries real time and historical data.

Create a new PVC and data generator for initial import

Save the following YAML to datagen.yml, then apply with kubectl apply -f datagen.yml.

This will create a new volume, and a one off job to save the generated data to the volume.

In order to use this script, you must have a kdb-license secret, and an existing imagePullSecret.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hdb-s3-claim
spec:
  storageClassName: rook-cephfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hdb-gen-script
data:
  gen.q: |
    // Generate data for the last 10 days, 10,000 rows per day.
    // Creates
    // /data/hdb
    //    data/
    //      sym
    //      2022.01.01
    //        trade
    //      2022.01.02
    //        trade
    //          ...
    system "cd /data/hdb";
    d:asc .z.d - til 10;
    {[d;n]sv[`;.Q.par[`:data/;d;`trade],`]set .Q.en[`:data;([]sym:`$'n?.Q.A;time:("p"$d)+til n;price:n?100f;size:n?50f)];}[;10000] each d
    exit 0;
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hdb-gen-sa
  labels:
    app.kubernetes.io/name: hdb-gen-sa
    app.kubernetes.io/instance: hdb-gen-sa
---
apiVersion: batch/v1
kind: Job
metadata:
  name: hdb-gen-job
spec:
  template:
    spec:
      imagePullSecrets:
        - name: kxi-nexus-pull-secret
      serviceAccountName: hdb-gen-sa
      securityContext:
        {}
      restartPolicy: Never
      containers:
        - name: generator
          tty: true
          stdin: true
          securityContext:
            {}
          image: dev-registry.internal-insights.kx.com/kx_helm_test:0.0.4
          imagePullPolicy: Always
          command:
            - ./opt/kx/startq.sh
            - -testScript
            - /tmp/gen.q
          env:
            - name: KDB_LICENSE_B64
              valueFrom:
                secretKeyRef:
                  name: kdb-license
                  key: license
          volumeMounts:
            - name: hdb-gen-script
              mountPath: /tmp/gen.q
              subPath: gen.q
            - name: hdb-volume
              mountPath: /data/hdb
          ports:
            - name: hdb-gen
              containerPort: 5060
              protocol: TCP
      volumes:
        - name: hdb-gen-script
          configMap:
            name: hdb-gen-script
        - name: hdb-volume
          persistentVolumeClaim:
            claimName: hdb-s3-claim

Generate sample historical data

We are going to create a configmap that houses our generator script, and use a Job to run it.

Deploy the Assembly CR

To deploy the assembly, download the s3-persistence.yaml from the release repository and deploy to your cluster with the command.

You will want to replace the sm.tiers store for the s3 tier to set your bucket URL.

kubectl apply -f s3-persistence.yaml

Deleting this demo

First uninstall the assembly:

kubectl delete asm s3-storage

Once that completes, delete the resources we created earlier:

kubectl delete job hdb-gen-job
kubectl delete configmap hdb-gen-script
kubectl delete pvc hdb-s3-claim
kubectl delete sa hdb-gen-sa