Tests added in package org.apache.hadoop.fs , for the FileSystem API, in 2.0:
There are essentially 2 major types of tests in this package which are relevant to the HCFS working group.
1) Those that exercise the FileSystem and 2) Those that exercise the new AbstractFileSystem.
Of those test types, we have "HCFS" tests, which are generic, and other tests, which either Mock a FileSystem implementation (MOCK), are abstract in their implementation (ABSTRACT), test the LocalFileSystem (LOCAL), or test HDFS specifically (HDFS).
FileSystem Tests
These tests build off of the FileSystem-1.0 test library.
File System Implementation |
Class Name |
Description |
HCFS |
TestFileSystemCanonicalization |
Checks "canonicalization" of the file system URL |
MOCK |
TestFileSystemInitialization |
Check if FileSystem can be properly initialized if URLStreamHandlerFactory is set |
MOCK |
TestFileSystemTokens |
|
ABSTRACT |
TestFilterFileSystem |
|
ABSTRACT |
TestFilterFs |
|
LOCAL |
TestFsShellCopy |
|
HDFS |
TestHarFileSystemBasics |
This test class checks basic operations with HarFileSystem including various initialization cases, getters, and modification methods. |
LOCAL |
TestListFiles |
This class tests the FileStatus API. |
LOCAL |
TestLocalFileSystemPermission |
Test permissions on local fs impl |
LOCAL |
TestLocalFSFileContextCreateMkdir |
|
LOCAL |
TestLocalFSFileContextMainOperations |
|
LOCAL |
TestLocalFSFileContextSymlink |
Test symbolic links using FileContext and LocalFs. |
LOCAL |
TestLocalFsFCStatistics |
Tests the File Context Statistics for LocalFileSystem |
LOCAL |
TestLocal_S3FileContextURI |
|
S3 |
TestS3_LocalFileContextURI |
|
Generic |
TestTrash |
Tests trash intervals |
AbstractFileSystem Tests
These are new to Hadoop 2.X. Many of these tests define a protected parameter which for the FileContext which can be overriden by extending classes, making them well suited for HCFS testing in 2.0 AbstractFileSystem implementations.
HCFS |
FCStatisticsBaseTest |
Base Test to FileContext |
HCFS |
FileContextCreateMkdirBaseTest |
Base Test for mkdir operations |
HCFS |
FileContextMainOperationsBaseTest |
Base Test for FileContext operations (similar to TestFileSystem) |
HCFS |
FileContextPermissionBase |
Generic test for FileContext permissions |
HCFS |
FileContextSymlinkBaseTest |
Testing of symlinks creation and handling |
HCFS |
FileContextURIBase |
Testing multiple filesystem URIs together |
HCFS |
FileContextUtilBase |
Testing util() provided by FileContext |
HCFS |
TestAfsCheckPath |
Testing handling of paths with different schemes |
LOCAL |
TestFcLocalFsPermission |
? |
LOCAL |
TestFcLocalFsUtil |
|
LOCAL |
TestFileContextDeleteOnExit |
|
HCFS |
TestFileContextResolveAfs |
|
|
TestFilterFs |
|
Other Tests
FileContextTestHelper |
TestAvroFSInput |
TestCommandFormat |
TestDFVariations |
TestDelegationTokenRenewer |
TestFileStatus |
TestFsOptions |
TestFsShellCopy |
Sub packages, specific implementation tests
kfs |
loadGenerator |
permission |
s3 |
s3native |
shell |
slive |
viewfs |
Other Relevant (Not necessarily 2.X related) Test Frameworks (Swift, KFS, etc...)
Swift-File-System
Many of these tests are decoupled from swift semantics by inheriting from the base class "SwiftFileSystemBaseTest", and thus, they may be adoptable as HCFS tests readily. In addition to generic tests, the swift-file-system test suite also includes Swift specific functional tests (for example, reconciliation of swift object paths with files ). These are taken from Steve Loughran's github Fork of the swift-file-system repository.
Generic (extends SwiftFileSystemBaseTest.java) |
TestSwiftFileSystemExtendedContract |
Test semantics of file operations in corner cases (i.e. opening a file that doesnt exist) |
Generic (extends SwiftFileSystemBaseTest) |
TestSwiftFileSystemRename |
Testing that renames are consistent (i.e. after rename, file contents are in new path, etc..) . Also tests that invalid renames handled appropriately. |
SwiftFS |
TestSwiftFileSystemContract |
Implementation of Swift specific hooks for the generic hadoop FileSystemContractTest |
SwiftFS |
TestSwiftFileSystemPartitionedUploads |
Test Swift specific partitioning behavior on writes |
SwiftFS |
SwiftFileSystemBaseTest |
Generic Base test class which can be failed by subclasses. From HCFS perspective, this class decouples subclasses from Swift semantics, that they can mostly agnostic to FileSystem impl |
Generic (extends SwiftFileSystemBaseTest) |
TestSwiftFileSystemRead |
Tests behaviour when "overreading" a file |
Generic (extends SwiftFileSystemBaseTest) |
TestSwiftFileSystemDelete |
Tests delete semantics and recursive delete policy |
Generic (extends SwiftFileSystemBaseTest) |
TestSwiftFileSystemLsOperations |
Tests semantics of list operations and corner cases (i.e. ls empty directorys, ls on root directory, etc.. ) |
SwiftFS |
NativeSwiftFileSystemContractBaseTest |
Implementation of Swift specific hooks for Generic FileSYstemContractBaseTest - extended by TestSwiftFileSystemContract |
SwiftFS |
TestSwiftObjectPath |
Tests that files represented as object paths in SwiftFS are consistent and correct |
Generic (extends SwiftFileSystemBaseTest) |
TestSwiftFileSystemConcurrency |
Tests concurrent operations using Java threads. Generic tests but extends SwiftFSTest |
SwiftFS |
TestSwiftFileSystemBasicOps |
Hardcoded to SwiftFS in some places, but appears generalizable - tests basic fs operations (put, create, delete, ...) |
SwiftFS |
TestSwiftFileSystemDirectories |
Test file and directory semantics. Adaptable to a generic test. |
Generic (extends SwiftFileSystemBaseTest) |
SwiftFileSystemTestBlockLocation |
Tests that blocks are provided in FileSystem getFileBlockLocations call |
Other tests in the Hadoop-Swift-Integration repository:
InMemorySwiftObjectStore |
An admittedly "unrealistic" mock of Swift storage engine for testing. |
http |
package with unit tests for Rest clients connecting to Swift. |
snative |
package with extended version of SwiftFileSystem specific for testing large uploads. |
Conclusions
Hadoop 2.X adds several new tests and the FileContext related tests provide extension points for other HCFS tests.
This is an initial analysis. Comparison of missing tests would be much appreciated.
We have used the JDiff tool to diff between 1.X (1.1) and the 2.X branches (2.1-beta) and summarized classes by inspection in order to identify the ability to generalize these tests, but this may be prone to error. Comments and/or edits are welcome.
Meanwhile, other frameworks (i.e. swift) have implemented their own tests which in many cases can be abstracted into HCFS tests which are directly applicable to any FileSystem implementation.
More work will need to be done to normalize and integrate the corpus of existing tests into a single, robust, implementation independent testing suite that the community can share.