Kylin will generate temporary files in HDFS during the cube building; Besides, when purge/drop/merge cubes, some parquet files may be left in HDFS and will no longer be queried; Although Kylin has started to do some automated garbage collection, it might not cover all cases; You can do an offline storage cleanup periodically.
Working Dir(ROOT)
In above directory tree, the directory which end with "managed by tool" means StorageCleanupJob will try to check and delete useless files under these directory.
For directory table_snapshot, dict/global_dict, parquet/{CUBE_NAME}, parquet/{CUBE_NAME}/{SEGMENT_NAME} , Kylin will mark files which is unreferenced and stale(by checking last modified time) as garbage.
For directory job_tmp, Kylin will only check last modified time.
Option | Data Type | Default Value | Comment |
---|---|---|---|
delete | Boolean | false | Boolean, whether or not to do real delete operation. Default value is false, means a dry run. |
cleanupTableSnapshot | Boolean | true | Boolean, whether or not to delete unreferenced snapshot files. Default value is true . |
cleanupGlobalDict | Boolean | true | Boolean, whether or not to delete unreferenced global dict files. Default value is true . |
cleanupJobTmp | Boolean | false | Boolean, whether or not to delete job tmp files. Default value is false . |
cleanupThreshold | Integer | 168 | Integer, used to specific delete unreferenced storage that have not been modified before how many hours (recent files are protected). Default value is 168 hours. |
[root@cdh-master apache-kylin-4.0.0-SNAPSHOT-bin]# bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob -help Retrieving hive dependency... Retrieving hadoop conf dir... Retrieving Spark dependency... ... Running org.apache.kylin.rest.job.StorageCleanupJob -help usage: org.apache.kylin.rest.job.StorageCleanupJob -cleanupGlobalDict <cleanupGlobalDict> Boolean, whether or not to delete unreferenced global dict files. Default value is true . -cleanupJobTmp <cleanupJobTmp> Boolean, whether or not to delete job tmp files. Default value is false . -cleanupTableSnapshot <cleanupTableSnapshot> Boolean, whether or not to delete unreferenced snapshot files. Default value is true . -cleanupThreshold <cleanupThreshold> Integer, used to specific delete unreferenced storage that have not been modified before how many hours (recent files are protected). Default value is 168 hours. -delete <delete> Boolean, whether or not to do real delete operation. Default value is false, means a dry run. |
bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob |
bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true |
bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true \ --cleanupJobTmp ture -cleanupTableSnapshot false \ -cleanupGlobalDict false --cleanupThreshold 24 |