In the previous article, we reviewed several practical performance hints that let us exploit SSDs at their peak. Today you will see how to go further and improve random write workloads of a database by replacing regular SSDs with Intel Optane SSDs powered by cutting-edge 3D-XPoint technology.

What might be wrong with sustained random write workloads on good-old SSDs? Those who read the previous article will remember that regular SSDs have to perform garbage collection routines endlessly by erasing blocks with stale data. Since any kind of garbage collection inevitably leads to performance degradation, SSDs manufactures came up with the over-provisioning technique that reserves some amount of space for cleaning needs. However, this space is limited and can be exhausted pretty quick depending on a workload.

Thus, it is a prevalent situation to get the top I/O numbers on regular SSDs in the first hours of operation and then suddenly hit a significant performance drop while keeping the workload the same.  The picture below shows this exact tendency:

As a software guy, I was always curious to know how the things work at the hardware level and how to apply the knowledge for more advanced optimizations in applications. Take Java Memory Model for instance. The model grounds its memory consistency and visibility properties on keywords such as volatile or synchronize. But these are just the language keywords and you start looking around how JVM engineers could turn the model in life. At some point, you will breathe out revealing that the model utilizes a low-level instruction set for mutexes and memory barriers at the very bottom of the software pie running on physical machines. Nice, these are the instructions a CPU understands but the curiosity drives you further because it is still vague how all the memory consistency guarantees can be satisfied on multi-CPU machines with several CPU registers and caches. Well, the hardware guys took care of this by supporting the cache coherence protocol. And finally you, as a software guy, can develop highly-performant applications that halt CPUs and invalidate their caches only on purpose with all these volatile, synchronize and final keywords.

Apache Ignite veterans tapped into the knowledge above and, undoubtedly, could deliver one of the fastest in-memory database and computational platform. Presently, the same people are optimizing Ignite Native Persistence - Ignite's distributed and transactional persistence layer. Being a part of that community, let me share some tips about solid-state drives (SSDs) that you, as a software guy, can exploit in Ignite or other disk-based databases deployments.

SSD Level Garbage Collection

Garbage collection (GC) term is used not only by Java developers to describe the process of purging dead objects from Java heap residing in RAM. Hardware guys use the same term for the same purpose but in relation to SSDs.

In simple words, an SSD stores data in pages. Pages are grouped in blocks (usually 128/256 pages per block). The SSD driver can write data directly into an empty page but can clean the whole blocks only. Thus, to reclaim the space occupied by invalid data, all the valid data from one block has to be first copied into empty pages of another block. Once this happens, the driver will purge all the data from the first block giving more space for new data arriving from your applications.
This process happens in the background and called with a familiar term - garbage collection (GC).

So, if you suddenly observe a performance drop under a steady load like it's shown in Figure 1. below, do not be trapped blaming your application or Apache Ignite. The drop might be caused by SSD GC routines.



Let me give you several hints on how to decrease the impact of the SSD GC on the performance of applications.

Separate Disk Devices for WAL and Data/Index Files

Apache Ignite arranges data and indexes in special partition files on disk. This type of architecture does not require you to have all the data in RAM, if something is missing there Apache Ignite will find the data on disk in these files.


However, referring to Figure 2., every data (1) that is received by Apache Ignite cluster node will be stored in RAM and persisted (2) in a write-ahead log (WAL) first. This is done by performance reasons and once the update is in the WAL, your application will get the acknowledgment and be able to execute its logic. Then, in the background, the checkpointing process will update the partition files by copying dirty pages from RAM to disk (4). Specific WAL files will be archived over the time and can be safely removed because all the data will be already in the partition files.

So, what's the performance hint here? Consider using separate SSDs for the partition files and the WAL. Apache Ignite actively writes to both places, thus, by having separate physical disk devices for each you may double the overall write throughput. See how to tweak the configuration for that.

SSD Over-provisioning

As the Java heap, SSD requires free space to perform efficiently and to avoid significant performance drops due to the GC. All SSD manufactures reserve some amount of space for that purpose. This is called over-provisioning.

Here are you, as a software guy, should keep in mind that the performance of random writes on a 50% filled disk is much better than on a 90% filled disk because of the SSDs over-provisioning and GC. Consider buying SSDs with higher over-provisioning rate and make sure a manufacturer supports the tools to adjust it.

Anything else?

That's enough for the beginning. If you are a sort of the guy who wants to get most of the hardware by tweaking page size or swapping settings refer to this tuning page maintained by Apache Ignite community.

Bothersome Ad

Apache Ignite 1.7.0 has been rolled out recently and among the new changes you can find a killer one that was being awaited by many Apache Ignite users and customers for a long time - the Non-Collocated Distributed joins support for SQL queries. So this post will be fully dedicated to this feature, and I'll try to shed some light on how the non-collocated distributed joins work and how they are different from the traditional (affinity collocation based) joins available in Apache Ignite.

In the previous post we were talking about the situation when an Ignite cluster is deployed in Google Compute Engine network and we need to have nodes auto-discovery mechanism.

Well, Apache Ignite has Compute Engine specific solution to fill this gap but what's about many others cloud platforms? What if I want to use AWS, Rackspace, GoGrid or other cloud provider?

Do you need to switch back to annoying and fragile static IP configuration? Luckily, the answer - you don't!

Since 1.1.0-incubating release Ignite has a so called TcpDiscoveryCloudIpFinder that supports a variety of cloud providers out of the box. Actually, under the hood this IP finder is integrated with well-known Apache jclouds multi-cloud toolkit. Using jclouds the IP finder retrieves IP addresses of all virtual machines in a cloud, talks to them and forms a list of active Ignite nodes (the machines with running Ignite instances). Every node stores a copy of such an up-to-date list. All this lets the nodes to discover each other automatically.

Having this in mind, let's set up and use TcpDiscoveryCloudIpFinder. As an example I'll keep using Google Compute Engine platform but you can use any provider from this list.

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryCloudIpFinder ipFinder = new TcpDiscoveryCloudIpFinder();

// Configuration for Google Compute Engine.
ipFinder.setProvider("google-compute-engine");
ipFinder.setIdentity(yourServiceAccountEmail);
ipFinder.setCredential(pathToYourPemFile);
ipFinder.setZones(Arrays.asList("us-central1-a", "asia-east1-a"));

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

First three parameters that are passed to the instance of the IP finder in the code above are set according to jclouds requirements:

• Provider name is equal to a value stored in "Maven Artifact ID" column of the following table. Compute Engine's value is "google-compute-engine", that's why it's passed to ipFinder.setProvider();

• Both identity and credential are provider specific. To get a provider specific instructions click on a provider name in "Provider" column in the same table. Google Compute authentication method is an exception, it's authentication method is covered on this page.

This post I want to dedicate to one useful feature of Apache Ignite that I had a chance to contribute to the project and that became a part of it since release 1.1.0-incubating.

Actually, this will be my first post about this Apache project and for those who are unfamiliar with it I would recommend to overview a good quick start documentation that can be found here.

The feature I'm going to cover is related to nodes discovery process located in a cluster in case when the cluster is a set of virtual machines deployed on Google Compute Engine. Historically Apache Ignite offered us two ways for the discovery process - leveraging multicast protocol or pre-configuring nodes' IP addresses. Unfortunately, neither way works well for Google VMs (virtual machines) instances: multicast is not supported in Compute Engine network at all and static IP configuration is not flexible (machine IP address may change from time to time, machines can be removed or added to a cluster, etc.).

However, there is a solution to this problem called TcpDiscoveryGoogleStorageIpFinder.

This IP finder does exactly what its name says. It lets virtual nodes located in Compute Engine network discover each other automatically using Google Cloud Storage where Apache Ignite's kernal will keep track of all cluster's nodes.

What you need to do to activate this feature is set TcpDiscoveryGoogleStorageIpFinder as an IP finder for a VM node.

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryGoogleStorageIpFinder ipFinder = new TcpDiscoveryGoogleStorageIpFinder();

ipFinder.setServiceAccountId(yourServiceAccountId);
ipFinder.setServiceAccountP12FilePath(pathToYourP12Key);
ipFinder.setProjectName(yourGoogleCloudPlatformProjectName);

ipFinder.setBucketName("your_bucket_name");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

That's all on the code side. Easy enough, right?
Now let's go through the parameters that have to be set in order to make the IP finder workable.

Considering that you're setting up everything from scratch perform the following steps:

• First you need to sign in Google Developer Console;

• Create a new project, open its "Overview" section, find "Project Number" and pass it to ipFinder.setProjectName() method call;

• Open "APIs and auth"->"Credentials" section and create a service account by pressing on "Create new Client ID" button. Set account's email address to ipFinder.setServiceAccountId(). Generate and download new account's P12 key and pass a full path to its location using ipFinder.setServiceAccountP12FilePath();

• Set unique Google Storage bucket name to ipFinder.setBucketName(). Note, that the name must be unique across the whole Google Cloud Platform and not just in your project.

So these are all the steps you need to do if you want to run a cluster of Ignite nodes in Compute Engine network with nodes auto-discovery enabled.

In the next post I will share another one generic solution that will enable nodes auto-discovery not only for Compute Engine network but for many other cloud platforms. Stay turned!