Release Notes for Storm 2.3.0
JIRA issues addressed in the 2.3.0 release of Storm. Documentation for this
release is available at the Apache Storm
project site.
New Feature
- [STORM-3271] - Launch storm workers in docker containers
- [STORM-3388] - Launch workers inside container using runc runtime
Improvement
- [STORM-3638] - Add fast and more reliable process liveness check
- [STORM-3641] - switch JCQueue metrics to new metrics API
- [STORM-3642] - update AutoTGT metric to new API
- [STORM-3644] - improve PacemakerClient error messaging
- [STORM-3647] - Add OFF-HEAP macro for worker childopts similar to LIMIT-MEM
- [STORM-3648] - Add meter to track worker heartbeat rate
- [STORM-3650] - Ackers and metricComponents are not distributed evenly
- [STORM-3653] - Document treatment of common nodes in favored/unfavored nodes in scheduling
- [STORM-3654] - JCQueue shouldn't include executor string in the metric name
- [STORM-3656] - Change handling of Hadoop TGT renewal exception
- [STORM-3657] - storm.messaging.netty.authentication should be true if it is true in either the topology conf or cluster conf
- [STORM-3659] - User page should show storm Version in the Owner topologies table
- [STORM-3660] - Remove messaging queue for updating Executor credentials
- [STORM-3661] - Netty server should handle incoming messages better
- [STORM-3662] - Continue to support Pacemaker for 2.x topologies
- [STORM-3667] - Exception message on submitting topology due to incorrect worker max heap size should be more straightforward
- [STORM-3670] - Separate configurations for daemon metric reporters and topology metrics reporters
- [STORM-3671] - Add TopologySummary method to NImbus for optimal UI queries
- [STORM-3673] - update BuiltinMetrics to use v2 Metrics API
- [STORM-3675] - Eliminate ps command and use /proc/ status in ServerUtils
- [STORM-3676] - Reduce debug spew to scheduler log
- [STORM-3682] - Upgrade netty client metrics to use V2 API
- [STORM-3685] - Detect Loops in Topology at Submit
- [STORM-3691] - Refactor Resource Aware Strategies (Base, Generic, Default)
- [STORM-3692] - Handle UID return from Files.getOwner() at process liveness check
- [STORM-3694] - all V2 metric reporters to report metric short names with dimensions
- [STORM-3697] - Add metric for capacity
- [STORM-3706] - Cluster.needsSchedulingRas always succeeds
- [STORM-3707] - Add meter to track update blob failures
- [STORM-3708] - ConstraintSolverConfig LOG messages should include topology id
- [STORM-3709] - Reject topology submission if missing spout
- [STORM-3714] - Add rate information for TaskMetrics
- [STORM-3715] - Add Caching to HDFS BlobStore
- [STORM-3719] - Add configuration for AsyncLocalizer updateBlobs frequency
- [STORM-3721] - Change child pom.xml reference to parent pom.xml
- [STORM-3724] - Use blobstore dir modtime to avoid update lookups by HDFSBlobstore
- [STORM-3737] - Share Worker Metric Registry For Guice AOP Based Metrics Integeration
- [STORM-3739] - Scheduling should sort numa zones by host groups
- [STORM-3744] - IntelliJ does not find shaded classes
- [STORM-3749] - improve logging on server error in StormServerHandler
- [STORM-3755] - While scheduling multiple ackers with executor use best effort basis
- [STORM-3758] - Zookeeper reconnect events sometime cause workers to restart
- [STORM-3774] - Migrate Cgroup metrics to V2
- [STORM-3778] - convert SpoutThrottlingMetrics to V2 API
- [STORM-3780] - switch ErrorReportingMetrics to V2 API
- [STORM-3786] - V2 metrics tick may overreport or not report at all
- [STORM-3790] - Add meter to track failures WorkerTokenAuthorizer getPassword
- [STORM-3791] - update metric documentation
- [STORM-3793] - Add metric to track backpressure status for a task
Bug
- [STORM-3553] - Upgrade JQuery to 3.5.1
- [STORM-3624] - Race condition on ArtifactoryConfigLoader.load
- [STORM-3649] - Logic error regarding storm.supervisor.medium.memory.grace.period.ms
- [STORM-3652] - Last error not displayed in Topology summary storm ui
- [STORM-3655] - Worker should commit suicide On Change in its assignment.
- [STORM-3658] - Problematic worker stays alive because of a deadlock and race condition caused by ShutdownHooks
- [STORM-3663] - Topology with Mockito 1.x fails to run unit tests
- [STORM-3666] - Validate component name in rebalance command and fix --executor option
- [STORM-3677] - Fix Worker Suicide Function if assignment is null
- [STORM-3679] - misuse of nodeId as hostname in LoadAwareShuffleGrouping
- [STORM-3684] - receive-queue V2 metrics shouldn't have "_system" as componentId if it is from a system task
- [STORM-3690] - UI has NullPointerException when the scheduler is not ResourceAwareScheduler
- [STORM-3695] - Add Timer rates to datapoints for V2 metrics tick
- [STORM-3696] - ClientSupervisorUtils.processLauncherAndWait ignores InterruptedException
- [STORM-3699] - fix flight.bash to support flight recording on openJDK8u262 or newer
- [STORM-3701] - Race Condition between cleanup thread and download tasks
- [STORM-3704] - Cosmetic: columns shifted in "Topology summary" table
- [STORM-3712] - Check-Updated-License-Files CI job has been broken
- [STORM-3720] - BlobStoreFile getModTime() never updates after first call
- [STORM-3723] - ServerUtils.isAnyPosixProcessPidDirAlive might return wrong result
- [STORM-3725] - DRPC spout will crash when any one of DRPC server is down
- [STORM-3727] - SUPERVISOR_SLOTS_PORTS could be list of Longs
- [STORM-3728] - Workers are not able to connect to Pacemaker if pacemaker.auth.method is KERBEROS
- [STORM-3729] - Assigning memory greater and equal than 2048m will make assgin memory for slot values to 1m
- [STORM-3731] - Remove unused nashorn import in storm-loadgen:OutputStream.java
- [STORM-3733] - AsyncLocalizer stuck looking for missing topology
- [STORM-3735] - Kyro serialization fails on some metric tuples when topology.fall.back.on.java.serialization is false
- [STORM-3740] - Asynchronous background blob download can cause orphaned blob references
- [STORM-3741] - Maven Surefire plugin corrupts output and creates an excessive log file
- [STORM-3748] - prevent concurrent modification when fetching v2 metrics
- [STORM-3763] - Backpressure message ignored by the receiver caused the topology to not progress
- [STORM-3765] - NPE in DRPCSimpleACLAuthorizer.readAclFromConfig when drpc.authorizer.acl has no values
- [STORM-3767] - NPE on getComponentPendingProfileActions
- [STORM-3769] - Failed adding references to blobs: FileNotFoundException
- [STORM-3775] - topology.blobstore.map can cause supervisor restarts
- [STORM-3785] - Rate metrics are wrongly divided by 1000000
Comment
- [STORM-3687] - Add a warning about possible issues on a mixed cluster if the StormCommon.systemTopology implementation is changed
Dependency upgrade
Documentation
Test
- [STORM-3743] - Add new topologies for TestLargeCluster
Sub-task
- [STORM-3674] - explicitly specify pip2 and pip3 in travis-install.sh
- [STORM-3681] - Enable basic Travis ARM CI job
- [STORM-3711] - Enable all the modules in ARM CI