HPCC Systems 10.2.0.0 Platform Release Announcement

This release brings notable improvements in performance, operational efficiency, and user experience across the HPCC Systems Platform.

Highlights include enabling jemalloc by default, faster index operations, Roxie runtime optimizations, improved storage handling, and UI/feature enhancements in ECL Watch.

Performance Improvements

Memory Management and Allocation

• HPCC-35673 Enable jemalloc by default — jemalloc is now enabled by default for improved memory efficiency and reduced fragmentation.
• HPCC-35485 Add jemalloc allocator config option for LD_PRELOAD — Adds configurability for jemalloc via LD_PRELOAD, simplifying deployment and tuning.

For more information, see Expert settings in HPCC Helm chart

Roxie Runtime and I/O Optimizations

• HPCC-35022 CRoxieFileCache::lookupFile should allow concurrent lookups — Reduces the time Roxie takes to start up when the parallelQueryLoadThreads or numResolveFilenameThreads options are used.
• HPCC-34955 Support using io_uring for tcp reads in roxie — Adds the capability for Roxie to use TCP as the network transport layer through the transportProtocol option.
• HPCC-35550 Remove Roxie code to model differences in compression speed — Simplifies compression handling to reduce overhead and improve consistency.

Storage and Indexing

• HPCC-35242 Allow indexes to be loaded in parallel — Parallel index loading significantly reduces startup and recovery times.
• HPCC-34619 Disk Page Cache — When Roxie is reading data from remote storage, this adds an option to cache frequent reads on local SSD/NVMe, improving performance.
• HPCC-35626 Default save dali store in binary format — Uses binary format for Dali store by default for improved load/save performance.
• HPCC-35296 Improve AWS S3 implementation — Improvements to AWS SDK integration, thread safety, error handling, and performance optimizations including read-ahead buffering and multipart upload support for large files.

Data Ingestion and Heap Tuning

• HPCC-35359 Enable new quoted csv partitioner in 10.0.x by default — Significantly reduces the time to partition large CSV files that use quoted fields.
• HPCC-35285 Introduce a RHFlimitedcount heap option to reduce contention — Adds heap configuration to reduce contention under high-concurrency workloads.

Notable Changes and Features

UI and User Experience

• HPCC-35578 Promote ActivitiesCard to default — The ActivitiesCard UI component is now the default, improving ECL Watch dashboard usability.

Data Access

• HPCC-34832 Add ParquetReader method to get Record Structure from Parquet file — Adds a method to extract record structure from Parquet, simplifying schema inspection and tooling.

Documentation Updates

• HPCC-35580 Further updates to the index documentation — Enhances index documentation with clearer guidance and updated examples.

  See Index Representation in the HPCC Systems Platform

Platform and Deployment

• HPCC-35516 Support striped storage in bare-metal — Adds support for striped storage on bare-metal deployments.

Additional Notes

For more details, follow the linked pull requests in each section.