Tests added in package org.apache.hadoop.fs , for the FileSystem API, in 2.0:

There are essentially 2 major types of tests in this package which are relevant to the HCFS working group.

1) Those that exercise the FileSystem and 2) Those that exercise the new AbstractFileSystem.

Of those test types, we have "HCFS" tests, which are generic, and other tests, which either Mock a FileSystem implementation (MOCK), are abstract in their implementation (ABSTRACT), test the LocalFileSystem (LOCAL), or test HDFS specifically (HDFS).

FileSystem Tests

These tests build off of the FileSystem-1.0 test library.

File System Implementation

Class Name

Description

HCFS

TestFileSystemCanonicalization

Checks "canonicalization" of the file system URL

MOCK

TestFileSystemInitialization

Check if FileSystem can be properly initialized if URLStreamHandlerFactory is set

MOCK

TestFileSystemTokens

ABSTRACT

TestFilterFileSystem

ABSTRACT

TestFilterFs

LOCAL

TestFsShellCopy

HDFS

TestHarFileSystemBasics

This test class checks basic operations with HarFileSystem including various initialization cases, getters, and modification methods.

LOCAL

TestListFiles

This class tests the FileStatus API.

LOCAL

TestLocalFileSystemPermission

Test permissions on local fs impl

LOCAL

TestLocalFSFileContextCreateMkdir

LOCAL

TestLocalFSFileContextMainOperations

LOCAL

TestLocalFSFileContextSymlink

Test symbolic links using FileContext and LocalFs.

LOCAL

TestLocalFsFCStatistics

Tests the File Context Statistics for LocalFileSystem

LOCAL

TestLocal_S3FileContextURI

S3

TestS3_LocalFileContextURI

Generic

TestTrash

Tests trash intervals

AbstractFileSystem Tests

These are new to Hadoop 2.X. Many of these tests define a protected parameter which for the FileContext which can be overriden by extending classes, making them well suited for HCFS testing in 2.0 AbstractFileSystem implementations.

HCFS

FCStatisticsBaseTest

Base Test to FileContext

HCFS

FileContextCreateMkdirBaseTest

Base Test for mkdir operations

HCFS

FileContextMainOperationsBaseTest

Base Test for FileContext operations (similar to TestFileSystem)

HCFS

FileContextPermissionBase

Generic test for FileContext permissions

HCFS

FileContextSymlinkBaseTest

Testing of symlinks creation and handling

HCFS

FileContextURIBase

Testing multiple filesystem URIs together

HCFS

FileContextUtilBase

Testing util() provided by FileContext

HCFS

TestAfsCheckPath

Testing handling of paths with different schemes

LOCAL

TestFcLocalFsPermission

?

LOCAL

TestFcLocalFsUtil

LOCAL

TestFileContextDeleteOnExit

HCFS

TestFileContextResolveAfs

TestFilterFs

Other Tests

FileContextTestHelper

TestAvroFSInput

TestCommandFormat

TestDFVariations

TestDelegationTokenRenewer

TestFileStatus

TestFsOptions

TestFsShellCopy

Sub packages, specific implementation tests

kfs

loadGenerator

permission

s3

s3native

shell

slive

viewfs

Other Relevant (Not necessarily 2.X related) Test Frameworks (Swift, KFS, etc...)

Swift-File-System

Many of these tests are decoupled from swift semantics by inheriting from the base class "SwiftFileSystemBaseTest", and thus, they may be adoptable as HCFS tests readily. In addition to generic tests, the swift-file-system test suite also includes Swift specific functional tests (for example, reconciliation of swift object paths with files ). These are taken from Steve Loughran's github Fork of the swift-file-system repository.

Generic (extends SwiftFileSystemBaseTest.java)

TestSwiftFileSystemExtendedContract

Test semantics of file operations in corner cases (i.e. opening a file that doesnt exist)

Generic (extends SwiftFileSystemBaseTest)

TestSwiftFileSystemRename

Testing that renames are consistent (i.e. after rename, file contents are in new path, etc..) . Also tests that invalid renames handled appropriately.

SwiftFS

TestSwiftFileSystemContract

Implementation of Swift specific hooks for the generic hadoop FileSystemContractTest

SwiftFS

TestSwiftFileSystemPartitionedUploads

Test Swift specific partitioning behavior on writes

SwiftFS

SwiftFileSystemBaseTest

Generic Base test class which can be failed by subclasses. From HCFS perspective, this class decouples subclasses from Swift semantics, that they can mostly agnostic to FileSystem impl

Generic (extends SwiftFileSystemBaseTest)

TestSwiftFileSystemRead

Tests behaviour when "overreading" a file

Generic (extends SwiftFileSystemBaseTest)

TestSwiftFileSystemDelete

Tests delete semantics and recursive delete policy

Generic (extends SwiftFileSystemBaseTest)

TestSwiftFileSystemLsOperations

Tests semantics of list operations and corner cases (i.e. ls empty directorys, ls on root directory, etc.. )

SwiftFS

NativeSwiftFileSystemContractBaseTest

Implementation of Swift specific hooks for Generic FileSYstemContractBaseTest - extended by TestSwiftFileSystemContract

SwiftFS

TestSwiftObjectPath

Tests that files represented as object paths in SwiftFS are consistent and correct

Generic (extends SwiftFileSystemBaseTest)

TestSwiftFileSystemConcurrency

Tests concurrent operations using Java threads. Generic tests but extends SwiftFSTest

SwiftFS

TestSwiftFileSystemBasicOps

Hardcoded to SwiftFS in some places, but appears generalizable - tests basic fs operations (put, create, delete, ...)

SwiftFS

TestSwiftFileSystemDirectories

Test file and directory semantics. Adaptable to a generic test.

Generic (extends SwiftFileSystemBaseTest)

SwiftFileSystemTestBlockLocation

Tests that blocks are provided in FileSystem getFileBlockLocations call

Other tests in the Hadoop-Swift-Integration repository:

InMemorySwiftObjectStore

An admittedly "unrealistic" mock of Swift storage engine for testing.

http

package with unit tests for Rest clients connecting to Swift.

snative

package with extended version of SwiftFileSystem specific for testing large uploads.

Conclusions

Hadoop 2.X adds several new tests and the FileContext related tests provide extension points for other HCFS tests.

This is an initial analysis. Comparison of missing tests would be much appreciated.

We have used the JDiff tool to diff between 1.X (1.1) and the 2.X branches (2.1-beta) and summarized classes by inspection in order to identify the ability to generalize these tests, but this may be prone to error. Comments and/or edits are welcome.

Meanwhile, other frameworks (i.e. swift) have implemented their own tests which in many cases can be abstracted into HCFS tests which are directly applicable to any FileSystem implementation.

More work will need to be done to normalize and integrate the corpus of existing tests into a single, robust, implementation independent testing suite that the community can share.

HCFS/FileSystem-2.0-Tests (last edited 2013-07-01 02:35:51 by JayVyas)