Differences between revisions 8 and 9
Revision 8 as of 2013-06-28 19:12:50
Size: 4566
Editor: JayVyas
Comment:
Revision 9 as of 2013-06-28 19:20:06
Size: 4526
Editor: JayVyas
Comment:
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
||''' Impl Test ''' ||'''Test''' ||'''What it does''' ||'''Runtime Test (main)''' ||
||no FS ||TestDU.java ||Validates DU.java works properly || ||
||no FS ||TestGlobPattern.java ||Validates Glob Pattern Matching Support || ||
|| ||DFSCIOTest.java ||Why do we have this AND TestDFSIO ? ||x ||
||no, but references org.apache.hadoop.hdfs.MiniDFSCluster ||TestDisableCache.java ||Validates the ability to turn FS caching on and off || ||
||no, but references org.apache.hadoop.hdfs.MiniDFSCluster ||TestHarFileSystem.java ||Validates [[http://hadoop.apache.org/docs/stable/hadoop_archives.html|Hadoop Archives (HAR)]] Support || ||
|| ||DistributedFSCheck.java ||Validates FS Consistency ||x ||
|| ||FSMainOperationsBaseTest.java ||Validates FS Operations. Needs to be disambiguated from FileSystemContractBaseTest || ||
||no, but references org.apache.hadoop.hdfs.MiniDFSCluster ||TestFileSystem.java ||Validates Reads, Seeks and Writes via a M/R Job. Needs to be disambiguated from DFSCIOTest and TestDFSIO ||x ||
|| ||FileSystemContractBaseTest.java ||Validates FS Impl adherence to FS Contract || ||
|| ||FileSystemTestHelper.java ||Helper/Utility Class for FS Testing || ||
||no, but references and configured with RawLocalFileSystem.||TestFsShellReturnCode.java ||Validates that Hadoop FS chmod, chown, chgrp commands work correctly || ||
||no FS ||TestPath.java ||I think this is a Helper class for testing Paths. Seems to be a mix of local and (perhaps) FS Tests || ||
|| ||TestGetFileBlockLocations.java ||Validates the ability to determine Block/File Locations || ||
|| ||TestTrash.java ||Validates FS support of Trash Operations || ||
||no, but references org.apache.hadoop.hdfs.MiniDFSCluster ||TestCopyFiles.java ||Test Copy Operations but needs to be disambiguated from FileSystemContractBaseTest and other FS Test Classes || ||
|| ||TestTruncatedInputBug.java ||Test for HADOOP-1489. Needs to be changed from Explicit Local to FS || ||
|| ||TestDFSIO.java ||Validate Read and Write Throughput ||x ||
||no FS ||TestGlobPaths.java ||Tests Glob Paths. Needs to be disambiguated from other Glob Tests || ||
||no but depends on org.apache.hadoop.hdfs.MiniDFSCluster ||TestUrlStreamHandler.java ||High level validation of FS Input Streams || ||
||x ||AccumulatingReducer.java ||Reducer that accumulates values based on their type || ||
||x ||TestLocalDirAllocator.java ||Does not use FS. This is a LocalFileSystem test. || ||
||x ||TestFileSystemCaching.java ||Does not use FS. This is a LocalFileSystem test. || ||
||x ||TestLocalFileSystem.java ||Does not use FS. This is a LocalFileSystem test. || ||
||x ||TestFileUtil.java ||Does not use FS. This is a LocalFileSystem test || ||
||x ||TestLocalFileSystemPermission.java ||Does not use FS. This is a LocalFileSystem test || ||
||x ||IOMapperBase.java ||Not sure where this is used. Seems redundant || ||
||x ||TestChecksumFileSystem.java ||Does not use FS. This is a ChecksumFileSystem/InMemoryFileSystem test || ||
||x ||TestGlobExpander.java ||Helper class for Glob Operations. Not clear who uses it || ||
||x ||TestHardLink.java ||Does not use FS. We should remove. Validates Single and Multi File Hard linking || ||
||x ||TestFSMainOperationsLocalFileSystem.java ||Does Nothing. Recommend that we remove this out of the FS Library || ||
Line 5: Line 36:
|| ''' Impl Test ''' || '''Test''' || '''What it does''' || '''Runtime Test (main)''' ||
|| no FS || TestDU.java || Validates DU.java works properly || ||
|| no FS || TestGlobPattern.java || Validates Glob Pattern Matching Support || ||
|| || DFSCIOTest.java || Why do we have this AND TestDFSIO ? || x ||
|| no, but references org.apache.hadoop.hdfs.MiniDFSCluster || TestDisableCache.java || Validates the ability to turn FS caching on and off || ||
|| no, but references org.apache.hadoop.hdfs.MiniDFSCluster || TestHarFileSystem.java || Validates [[http://hadoop.apache.org/docs/stable/hadoop_archives.html | Hadoop Archives (HAR)]] Support || ||
|| || DistributedFSCheck.java || Validates FS Consistency|| x ||
|| || FSMainOperationsBaseTest.java || Validates FS Operations. Needs to be disambiguated from FileSystemContractBaseTest || ||
|| no, but references org.apache.hadoop.hdfs.MiniDFSCluster || TestFileSystem.java || Validates Reads, Seeks and Writes via a M/R Job. Needs to be disambiguated from DFSCIOTest and TestDFSIO || x ||
|| || FileSystemContractBaseTest.java || Validates FS Impl adherence to FS Contract || ||
|| || FileSystemTestHelper.java || Helper/Utility Class for FS Testing || ||
|| || TestFsShellReturnCode.java || Validates that Hadoop FS chmod, chown, chgrp commands work correctly || ||
|| no FS || TestPath.java || I think this is a Helper class for testing Paths. Seems to be a mix of local and (perhaps) FS Tests || ||
|| || TestGetFileBlockLocations.java || Validates the ability to determine Block/File Locations || ||
|| || TestTrash.java || Validates FS support of Trash Operations || ||
|| no, but references org.apache.hadoop.hdfs.MiniDFSCluster || TestCopyFiles.java || Test Copy Operations but needs to be disambiguated from FileSystemContractBaseTest and other FS Test Classes|| ||
|| || TestTruncatedInputBug.java || Test for HADOOP-1489. Needs to be changed from Explicit Local to FS || ||
|| || TestDFSIO.java || Validate Read and Write Throughput || x ||
|| no FS || TestGlobPaths.java || Tests Glob Paths. Needs to be disambiguated from other Glob Tests || ||
|| no but depends on org.apache.hadoop.hdfs.MiniDFSCluster || TestUrlStreamHandler.java || High level validation of FS Input Streams || ||
||x|| AccumulatingReducer.java || Reducer that accumulates values based on their type || ||
||x|| TestLocalDirAllocator.java || Does not use FS. This is a LocalFileSystem test. || ||
||x|| TestFileSystemCaching.java || Does not use FS. This is a LocalFileSystem test. || ||
||x|| TestLocalFileSystem.java || Does not use FS. This is a LocalFileSystem test. || ||
||x|| TestFileUtil.java || Does not use FS. This is a LocalFileSystem test || ||
||x|| TestLocalFileSystemPermission.java || Does not use FS. This is a LocalFileSystem test || ||
||x|| IOMapperBase.java || Not sure where this is used. Seems redundant || ||
||x|| TestChecksumFileSystem.java || Does not use FS. This is a ChecksumFileSystem/InMemoryFileSystem test || ||
||x|| TestGlobExpander.java || Helper class for Glob Operations. Not clear who uses it || ||
||x|| TestHardLink.java || Does not use FS. We should remove. Validates Single and Multi File Hard linking || ||
||x|| TestFSMainOperationsLocalFileSystem.java || Does Nothing. Recommend that we remove this out of the FS Library || ||

Line 38: Line 40:
||''' Impl Test ''' ||'''Test''' ||'''What it does''' ||
|| ||permission ||Validates the Hadoop FS Plugin adherence to Permissions ||
|| ||slive ||[[https://issues.apache.org/jira/browse/HDFS-708|Slive is a stress test for the Hadoop FS]] ||
|| ||loadGenerator ||A load testing tool for testing Hadoop FS behavior under different mixes of read, write and list requests ||
||x ||ftp ||Validates the Hadoop FS Plugin for an FTP Server ||
||x ||kfs ||Validates the Hadoop FS Plugin for the Kosmos FS. Now defunct due to the [[https://wiki.apache.org/hadoop/HCFS|Quantcast FS Plugin]] ||
||x ||s3 ||Validates the Hadoop FS Plugin for the Amazon S3 Object Store ||
||x ||s3native ||Pseudo S3 Implementation to allow testing of the Hadoop FS Plugin for S3 without connecting to Amazon S3 ||
Line 39: Line 49:
|| ''' Impl Test ''' || '''Test''' || '''What it does''' ||
|| || permission || Validates the Hadoop FS Plugin adherence to Permissions ||
|| || slive || [[https://issues.apache.org/jira/browse/HDFS-708 | Slive is a stress test for the Hadoop FS ]] ||
|| || loadGenerator || A load testing tool for testing Hadoop FS behavior under different mixes of read, write and list requests ||
|| x || ftp || Validates the Hadoop FS Plugin for an FTP Server ||
|| x || kfs || Validates the Hadoop FS Plugin for the Kosmos FS. Now defunct due to the [[https://wiki.apache.org/hadoop/HCFS | Quantcast FS Plugin]] ||
|| x || s3 || Validates the Hadoop FS Plugin for the Amazon S3 Object Store ||
|| x || s3native || Pseudo S3 Implementation to allow testing of the Hadoop FS Plugin for S3 without connecting to Amazon S3 ||

Initial Analysis of Hadoop FileSystem 1.0 Tests by the HFCS working group.

The term "Impl Test" used below is intended to assert that the test validates a single particular Hadoop FileSystem Implementation and should not belong with a Hadoop FS Test Library that focuses on any FileSystem Plugins ability to implement the Hadoop FileSystem contract.

Impl Test

Test

What it does

Runtime Test (main)

no FS

TestDU.java

Validates DU.java works properly

no FS

TestGlobPattern.java

Validates Glob Pattern Matching Support

DFSCIOTest.java

Why do we have this AND TestDFSIO ?

x

no, but references org.apache.hadoop.hdfs.MiniDFSCluster

TestDisableCache.java

Validates the ability to turn FS caching on and off

no, but references org.apache.hadoop.hdfs.MiniDFSCluster

TestHarFileSystem.java

Validates Hadoop Archives (HAR) Support

DistributedFSCheck.java

Validates FS Consistency

x

FSMainOperationsBaseTest.java

Validates FS Operations. Needs to be disambiguated from FileSystemContractBaseTest

no, but references org.apache.hadoop.hdfs.MiniDFSCluster

TestFileSystem.java

Validates Reads, Seeks and Writes via a M/R Job. Needs to be disambiguated from DFSCIOTest and TestDFSIO

x

FileSystemContractBaseTest.java

Validates FS Impl adherence to FS Contract

FileSystemTestHelper.java

Helper/Utility Class for FS Testing

no, but references and configured with RawLocalFileSystem.

TestFsShellReturnCode.java

Validates that Hadoop FS chmod, chown, chgrp commands work correctly

no FS

TestPath.java

I think this is a Helper class for testing Paths. Seems to be a mix of local and (perhaps) FS Tests

TestGetFileBlockLocations.java

Validates the ability to determine Block/File Locations

TestTrash.java

Validates FS support of Trash Operations

no, but references org.apache.hadoop.hdfs.MiniDFSCluster

TestCopyFiles.java

Test Copy Operations but needs to be disambiguated from FileSystemContractBaseTest and other FS Test Classes

TestTruncatedInputBug.java

Test for HADOOP-1489. Needs to be changed from Explicit Local to FS

TestDFSIO.java

Validate Read and Write Throughput

x

no FS

TestGlobPaths.java

Tests Glob Paths. Needs to be disambiguated from other Glob Tests

no but depends on org.apache.hadoop.hdfs.MiniDFSCluster

TestUrlStreamHandler.java

High level validation of FS Input Streams

x

AccumulatingReducer.java

Reducer that accumulates values based on their type

x

TestLocalDirAllocator.java

Does not use FS. This is a LocalFileSystem test.

x

TestFileSystemCaching.java

Does not use FS. This is a LocalFileSystem test.

x

TestLocalFileSystem.java

Does not use FS. This is a LocalFileSystem test.

x

TestFileUtil.java

Does not use FS. This is a LocalFileSystem test

x

TestLocalFileSystemPermission.java

Does not use FS. This is a LocalFileSystem test

x

IOMapperBase.java

Not sure where this is used. Seems redundant

x

TestChecksumFileSystem.java

Does not use FS. This is a ChecksumFileSystem/InMemoryFileSystem test

x

TestGlobExpander.java

Helper class for Glob Operations. Not clear who uses it

x

TestHardLink.java

Does not use FS. We should remove. Validates Single and Multi File Hard linking

x

TestFSMainOperationsLocalFileSystem.java

Does Nothing. Recommend that we remove this out of the FS Library

Hadoop FS Test sub-packages

Impl Test

Test

What it does

permission

Validates the Hadoop FS Plugin adherence to Permissions

slive

Slive is a stress test for the Hadoop FS

loadGenerator

A load testing tool for testing Hadoop FS behavior under different mixes of read, write and list requests

x

ftp

Validates the Hadoop FS Plugin for an FTP Server

x

kfs

Validates the Hadoop FS Plugin for the Kosmos FS. Now defunct due to the Quantcast FS Plugin

x

s3

Validates the Hadoop FS Plugin for the Amazon S3 Object Store

x

s3native

Pseudo S3 Implementation to allow testing of the Hadoop FS Plugin for S3 without connecting to Amazon S3

HCFS/FileSystem-1.0-Tests (last edited 2013-06-28 19:20:06 by JayVyas)