Configuring, Launching TIBCO ComputeDB Clusters

Before you configure the TIBCO ComputeDB cluster, check the system requirements. In case you have not yet provisioned TIBCO ComputeDB, you can follow the instructions here.
TIBCO recommends that you have at least 8 GB of memory and 4 cores available even for simple experimentation with TIBCO ComputeDB.

Launching Single Node Cluster with Default Configuration

If you want to launch the cluster either on Amazon EC2 or on a Kubernetes cluster, you can follow the instructions listed here (AWS) and here (Kubernetes)

If you are launching on a single node, for example, on your laptop or on a linux server you have access to, you can do so using this simple command:

./sbin/snappy-start-all.sh 

This launches a single locator, lead and a data server. You can go to the following URL on your browser to view the cluster dashboard:

http://(localhost or hostname or machineIP):5050

By default, the cluster uses the following ports:

Cluster Component	Port	Memory Used
Lead	5050 (http port used for dashboard.) 8090 (Port used to submit Spark streaming or batch jobs.) 10000 (Port for hive thrift server.)	4 GB
Locator	1527 (Port used by JDBC clients.) 10334 (Ports used for all cluster members to communicate with each other.)	1 GB
Server	1528 (Port used by ODBC or JDBC clients)	4 GB

All the artifacts created such as the server - logs, metrics, and the database files are all stored in a folder called work in the product home directory. Click the individual member URLs on the dashboard to view the logs.

Also see:

• Connecting with JDBC

• Connecting with ODBC

• Submitting a Spark Job

Configuring and Launching a Multi-node Cluster

Provision TIBCO ComputeDB and ensure that all the nodes are setup appropriately. If all the nodes are SSH enabled and can share folders using NFS or some shared file system, you can proceed to Capacity Planning. A shared file system is not a requirement but preferred.

Step 1: Capacity Planning

You must consider the capacity required for storage capacity (in-memory and disk) as well as the computational capacity that is required (memory and CPU). You can skip this step if the volume of data is low and feel you have abundant resources for the client workload.

Planning Storage Capacity

TIBCO ComputeDB is optimized for managing all the data in the memory. When data cannot fit in the memory, it automatically overflows the data to disk. To achieve the highest possible performance, we recommend you go through this exercise below. The capacity required is dependent on several variables such as input data format, the data types in use, use of indexes, number of redundant copies in the cluster, cardinality of the individual columns (compression ratio can vary a lot) and so on.

A general rule of thumb for compressed data (say Parquet) is to configure about 1.5X the compressed size on disk. Else, you can also use the following steps:

1. Define your external tables to access the data sources. This only creates catalog entries in TIBCO ComputeDB.
   For example, when loading data from a folder with data in CSV format, you can do the following using the Snappy Shell:

   create external table t1 using csv options(inferSchema 'true', header 'true', path '<some folder or file containing the CSV data>') ;
               // Spark Scala/Java/Python API example is omitted for brevity.
   

   More examples for loading data from external sources are available here

2. Decide if you want to manage this data in a Columnar or Row format.

3. Load a sample of the data:

   create table t1_inmem using column as (select * from t1 where rand() < 0.1) ;
   

   This loads about 10% of the data from the external data source. Note that the fraction value (0.1, which is 10%) should be reduced when working with large data sets or increased when the data set is small. You must repeat this process for all the tables that you want to manage in the memory.

4. Check the dashboard to view the actual memory consumed. Simply extrapolate to 100% (10 times for the example above). The amount of disk space is about two times the memory space requirement.

Estimating Computation Capacity

TIBCO recommends to configure off-heap memory when using the Enterprise edition that is TIBCO ComputeDB. This option is not available in the community edition of TIBCO ComputeDB where all data must be managed in the JVM heap memory. Even when off-heap is configured, you must also configure enough JVM heap memory for Spark temporary caching, computations, and buffering when the data is loaded.

TIBCO recommends to allocate at least 1 GB of memory per core to JVM heap for computations.

For example, when running on 8 core servers, configure JVM heap to be 8 GB. By default, 50% of the off-heap memory is available as computational memory. While, you may not need this much computational capacity when large off-heap is configured, it is still recommended for reserving enough capacity if working with large data sets.

More complex the analytical processing, especially large aggregations, greater the space requirement in off-heap. For example, if your per server off-heap storage need is 100 GB then, allocate an additional 30 GB of off-heap for computations. Even if your data set is small, you must still allocated at least a few Gigabytes of off-heap storage for computations.

Step 2: Configure Core Cluster Component Properties

Configuration files for locator, lead, and server should be created in the conf folder located in the TIBCO ComputeDB home directory with names locators, leads, and servers.

To do so, you can copy the existing template files servers.template, locators.template, leads.template, and rename them to servers, locators, leads. These files should contain the hostnames of the nodes (one per line) where you intend to start the member. You can modify the properties to configure individual members. For more details refer to Cluster Components Configuration section.

Configuring Core Lead properties

The following core properties must be set in the conf/leads file:

Properties	Description	Default Value
hostname (or IP)	The hostname on which a TIBCO ComputeDB locator is started
heap-size	Sets the maximum heap size for the Java VM, using TIBCO ComputeDB default resource manager settings. For example, -heap-size=8g It is recommended to allocate minimum 6-8 GB of heap size per lead node. If you use the -heap-size option, by default TIBCO ComputeDB sets the critical-heap-percentage to 95% of the heap size, and the eviction-heap-percentage to 85.5% of the critical-heap-percentage. TIBCO ComputeDB also sets resource management properties for eviction and garbage collection if the JVM supports them.
dir	Working directory of the member that contains the TIBCO ComputeDB Server status file and the default location for the log file, persistent files, data dictionary, and so forth.	/work
classpath	Location of user classes required by the TIBCO ComputeDB Server. This path is appended to the current classpath	Appended to the current classpath
spark.executor.cores	The number of cores to use on each server.
spark.jars

Example Configurations

In the following configuration, you set the heap size for the lead and specify the working directory location:

localhost   -dir=/opt/snappydata/data/lead -heap-size=6g

You can add a line for each of the Lead members that you want to launch. Typically only one. In production, you may launch two.

In the following configuration, you are specifying the number of cores to use on each server:

localhost -spark.executor.cores=16

Configuring Core Locator Properties

The following core properties must be set in the conf/locators file:

Properties	Description	Default Value
client-port	The port that the network controller listens for client connections in the range of 1 to 65535.	1527
dir	Working directory of the member that contains the TIBCO ComputeDB Server status file and the default location for the log file, persistent files, data dictionary, and so forth.	Current directory

Configuring Core Server Properties

The following core properties must be set in the conf/servers file:

Properties	Description	Default Value
memory-size	Specifies the total memory that can be used by the node for column storage and execution in off-heap.	The default value is either 0 or it gets auto-configured in specific scenarios
heap-size	Sets the maximum heap size for the Java VM, using TIBCO ComputeDB default resource manager settings. For example, -heap-size=8g It is recommended to allocate minimum 6-8 GB of heap size per lead node. If you use the -heap-size option, by default TIBCO ComputeDB sets the critical-heap-percentage to 95% of the heap size, and the eviction-heap-percentage to 85.5% of the critical-heap-percentage. TIBCO ComputeDB also sets resource management properties for eviction and garbage collection if the JVM supports them.
dir	Working directory of the member that contains the TIBCO ComputeDB Server status file and the default location for the log file, persistent files, data dictionary, and so forth.	Current directory

Configuration Examples

cat <product_home>/conf/servers
node1_hostname -dir=/nfs/opt/snappy-db1/server1 -heap-size=8g -memory-size=42g
node1_hostname -dir=/nfs/opt/snappy-db1/server2 -heap-size=8g -memory-size=42g

// You can launch more than one data server on a host 

node2_hostname -dir=/nfs/opt/snappy-db1/server3 -heap-size=8g -memory-size=42g
node2_hostname -dir=/nfs/opt/snappy-db1/server4 -heap-size=8g -memory-size=42g

Configuration Reference

See Configuration Reference section for more details.

TIBCO ComputeDB Properties

For the complete list of TIBCO ComputeDB properties, refer to List of Properties.