As per our standards at LightningChart Ltd., we have always maintained publicly available performance benchmarks of our charting libraries since 2021. Recently, one of our competitors published an article titled “A Comprehensive Benchmark Suite to Compare Chart Performance”, and there are critical issues in their content that we cannot stay silent about.

We identified this performance comparison as an intent to highlight SciChart and downplay other libraries maliciously. Their benchmark, available at Chart Bench: A Comprehensive Benchmark Suite to Compare JavaScript Chart Library Performance, does not represent real data visualization use cases.

The blog mentioned in the video can be found at NPM’s Blog Archive.

Even when it is not to anyone’s surprise that a company’s own marketing materials are meant to support its own products, the content published by this company is attempting to mislead its own readers. This should not be accepted. We’ve summarized the key issues with their benchmarks into the following points:

Completely Unrepresentative Benchmarks of Real Use Cases

SciChart themselves have mentioned a clearly visible disclaimer on their GitHub page, which is not being disclosed in their blog posts.

What does this disclaimer actually mean for those considering SciChart?
In practice, here’s how SciChart’s benchmark applications look, which is completely unrepresentative of real use cases.

Forced Charts Re-rendering

Nearly all the tests are based on programmatically tweaking the value axis interval by a small amount every frame, forcing charts to re-render.

Misleading Scoring System

Their scoring system places exponential value on FPS (how many times per second the chart is re-rendered). However, they are also separately counting FPS and Frames, which are the same metric, and even raised both metrics to the power of 1.5.

Biased Result Maximization

Finally, they run the tests with a 240 FPS ultra-high refresh rate monitor to maximize the effects of the aforementioned tricks. Their test suites for other libraries also show clear malicious intent. More on that later.

For the average viewer, none of this is noticeable. Even for an insider reader, this is not noticeable unless one actually runs the tests on one’s own computer and makes an effort to evaluate them. We understand that comparing chart performance is difficult, and even with these details laid out above, it is still unclear why and how all of this matters. So, let’s take a look at another way of doing the comparison for reference.

In LightningChart JS performance tests, we separately run tests for measuring A) Static use case performance and B) real-time use case performance. The scoring is performed as follows:

Static Use Cases

score = (numChannels x dataPerCh) / loadTimeSeconds

Real-Time Use Cases

score = numChannels x newDataPerSecond x FPS

Why are these metrics important?

The term “Performance” is ambiguous, and SciChart has chosen to stay hidden behind this ambiguity to mislead its readers. When we talk about Static Use Cases, we refer to situations where the user has a dataset X and wants to display it as soon as possible. In this case, it is critical to understand that the only “performance” that matters is how fast the user can display that data!

However, SciCharts’ tests involve playing around with the value axis interval, measuring FPS, and then telling readers that this means that the chart did a “good job”. The only situation where FPS is meaningful is in real-time use cases. i.e., new data is displayed in the chart every frame. But there is no excuse to raise it to the power of 1.5 and further multiply by frame count (which is the exact same measurement!) and further raise to the power of 1.5.

The above-mentioned facts are the most important issues with SciChart performance comparison materials. They are very deliberately “comparing performance” in a non-sensical way with use cases that do not reflect reality. We identify this as a major concern as they communicate this misleading information across their channels as conclusive evidence. For example:

All in all, our initial look into their materials resulted in 20 pages of issues. Some examples of malicious intent in other library apps include:

• SciChart has selected a 30-month-old library version on the basis that later versions require the download of a free trial.
• In SciChart apps, light configurations are set, whereas for other libraries, all defaults are used, or even slower configurations are applied.
• For LightningChart’s benchmark, they have used the wrong features that look completely different from the respective SciChart tests in order to perform more slowly.
• In several LightningChart benchmarks, they do not utilize features for best performance (such as shared contexts, shared timestamps, max. sample counts, disabling animations, …), so they have either not spent the effort to look up our documentation or have deliberately left these out. Even though, in some cases, there are automatic warnings in the console.

We reiterate our commitment to delivering the best charting technology and performance in all our libraries. Our benchmarks are public and can be consulted on GitHub.