This document attempts to answer some of the more frequently asked questions regarding various aspects of CommonsNet. These questions are typically asked over and over again on the mailing lists, as a courtesy to the developers, we ask that you read this document before posting to the mailing lists.
Q: How do I debug FTP applications?
A: You can add a protocol command listener; for example:
ftp.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
Q: How can I set a connection timeout?
A: setDefaultTimeout does not set the connect timeout. It provides a default socket timeout. Only in J2SE 1.4 was the ability to specify a timeout on connect added to the Socket API. Since Commons Net 1.2.x has a J2SE 1.2 compatibility requirement, the ability to specify a connect timeout is not included. The way to workaround this is to implement your own SocketFactory and set it with SocketClient.setSocketFactory (FTPClient is a subclass of SocketClient). When you implement the SocketFactory, add a setConnectTimeout method or some such. Inside of the createSocket methods, use the J2SE 1.4 connect method with the timeout. We could actually provide socket factory that subclasses DefaultSocketFactory to do this without breaking backward compatibility, but that would have to be discussed further. The way to do it is to compile it only if J2SE >= 1.4 is being used.
SocketClient could check for availability of the J2SE 1.4 connect method and instantiate the J2SE >= 1.4 factory if available (using Class.forName and newInstance). The setDefaultTimeout method could then be changed to also set the connect timeout in the new factory if being used. If users want this functionality enough, the best chance of getting it implemented soon is to submit a patch.
Q: On which server operating systems is FTPClient able to bring back a parsed listing?
A: Currently parsers have been written for UNIX, Windows, OS/2, Netware, VMS and OS400. These can be autodetected by the code without being specified. Additionally, the code correctly handles cases where NT or OS400 servers have been configured to look like unix servers. There is also a means of specifying a user-written parser by class name. All of these parsers return FTPFile objects with as much information about the file as can be gathered from the listing.
If none of these is sufficient, FTPClient.listNames() may always be called. This retrieves just the file name without other information.
Q: The FTP parser I am connecting to uses older software and non-English names of months and/or non-standard (standard American, that is) date formats, and therefore parsing fails. Is there a workaround?
A: Beginning with commons-net 1.4.0, there are new mechanisms for configuring various aspects of the server file entry parsing expectations. Please see http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html
Q: Why are my files corrupt after transfer?
A: The most common cause for this is when the file is transfered as ASCII but the contents of the file are not ASCII and the file should be transferred as BINARY. RFC 959 says the default transfer mode should be ASCII. FTPClient conforms to the standard. You must explicitly call setFileType(FTP.BINARY_FILE_TYPE); to request binary transfer mode after logging in to the FTP server.
Q: How do I get passive mode correctly started?
A: Passive mode has to be established each time you connect to an FTP server. After you connect to a server with FTPClient.connect(), you can call
FTPClient.enterLocalPassiveMode() to enable passive mode. From then on, FTPClient will take care of issuing a PASV command each time it opens a data connection until you change the data connection mode by calling
enterLocalActiveMode() or you close the connection. If you call
enterLocalPassiveMode before connecting to a server, the data connection mode will be reset to the default active mode upon initial connection establishment. If you always want to use passive mode and don't want to call
enterLocalPassiveMode all the time, you can subclass FTPClient and override
connectAction() like so:
protected void _connectAction_() throws IOException {
super._connectAction_();
enterLocalPassiveMode();
}
Also see http://cephas.net/blog/2004/05/27/commons-net-ftp-and-active-vs-pasv-connections/
Q: Using the NNTPClient, retrieving a list of headers takes forever. Why is that?
Normal retrieval of headers using a program like Grabit takes about 10 seconds for a group containing about 3000 articles. This piece of code takes about 2 minutes. All the time, the network is hardly used. How can it be fixed?
try
{
Date startTime = new Date();
log.debug("Connecting to " + host + "...");
client.connect(host);
client.setTcpNoDelay(true);
NewsgroupInfo info = new NewsgroupInfo();
int articles = 0
if (client.selectNewsgroup(group, info))
{
articles = info.getArticleCount();
log.debug(group + " selected. Contains " + articles + " articles.");
}
Reader r = null;
BufferedReader rr = null;
for (int i = start; i <= end; i++)
{
r = client.retrieveArticleHeader(i);
if (r != null)
{
rr = new BufferedReader(r);
String header = "";
while (rr.ready())
header += rr.readLine() + "\n";
}
}
client.disconnect();
log.debug("Disconnected.");
Date endTime = new Date();
long update = (endTime.getTime() - startTime.getTime())/1000;
if (update < 60)
log.info("Update took " + update + " seconds.");
else
log.info("Update took " + update/60 + " minute(s).");
}
catch (IOException e)
{
log.error(e);
}
A: After some thinking and browsing of the Commons-Net sourcecode, I come to the conclusion that the reason why updating all headers is so slow, is the fact that each message header is retrieved separately with a new command send to the server each time.
Update: This is indeed an issue. There are two fixes which I hope to implement - one is the utilisation of an extended NNTP command like XOVER or XHDR to retrieve the info, the other is the most useful one - actually taking a look at the parsing code and seeing where we can optimise that. I am confident we can improve the efficiency by a great deal - there is a lot of String creation in the internal reply parsing routines. - RoryWinston
UPDATE: Rory's patch to include XOVER was applied in August or September 2003.
UPDATE: Has anyone ever thought that reading and writing characters might be less performant than reading and writing bytes? Maybe changing the API to use streams instead of readers/writers might give some speedup. At least it makes it possible to attach a MIME parser like http://www.mime4j.org/
Q: Are the Commons-Net classes thread-safe? For example, can you have multiple instances of FTPClient, each with its own thread running inside?
A: Multiple instances of FTPClient can be used to connect to multiple (or the same) FTP server and concurrently retrieve files. If you want to share a single FTPClient instance between multiple threads, you must serialize access to the object with critical sections.
Q: Does FTPClient support FTP connections through an FTP proxy server?
A: Paul Buchanan answers this question here: http://marc.theaimsgroup.com/?l=jakarta-commons-user&m=107877944806547&w=2
The summary is:
Since the Net project uses java.net.Socket under the covers, you have to set up the Socket settings. If the proxy you are using is a SOCKS4 or SOCKS5 proxy, you can set the following System Properties:
System.getProperties().put( "socksProxyPort", "1080");
System.getProperties().put( "socksProxyHost" ,"proxy.host.address");
From what I can tell, 1080 is the default port for SOCKS proxies. The following site documents many of the networking System Properties available:
http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
</pre>
In addition if you need to identify with the SOCKS server, use something like:
import java.net.Authenticator;
import java.net.PasswordAuthentication;
public class Auth extends Authenticator {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(user, pass
.toCharArray());
}
}
Q: What's the point of the SMTP and POP3 packages when JavaMail provides more functionality?
A: NetComponents, now Commons Net, was written before JavaMail. NetComponents had different goals than JavaMail. NetComponents was and Commons Net is a low level API. JavaMail is a high level API. The NetComponents POP3 API is something one would have used to implement a JavaMail POP3 provider if JavaMail didn't ship with one already. People who have to deal with large messages, such as those containing attachments, like the low-level API which allows you to deal directly with the data stream unlike JavaMail which used to, and may still, buffer messages entirely in RAM.
Q: The storeFile method of FTPClient class calls copyStream method of org.apache.commons.net.io.Utils. The method in Utils takes a CopyStreamListener listener as a parameter. Is there anyway I can have a listener when calling storeFile? This way i can get the update of the bytes transferred from another thread
A: If you need to use a CopyStreamListener, you can manually open the connection using the method retrieveFileStream instead of storeFile and copy the contents from one file to another using the method copyStream.
try {
InputStream stO =
new BufferedInputStream(
ftp.retrieveFileStream("foo.bar"),
ftp.getBufferSize());
OutputStream stD =
new FileOutputStream("bar.foo");
org.apache.commons.net.io.Util.copyStream(
stO,
stD,
ftp.getBufferSize(),
/* I'm using the UNKNOWN_STREAM_SIZE constant here, but you can use the size of file too */
org.apache.commons.net.io.CopyStreamEvent.UNKNOWN_STREAM_SIZE,
new org.apache.commons.net.io.CopyStreamAdapter() {
public void bytesTransferred(long totalBytesTransferred,
int bytesTransferred,
long streamSize) {
// Your progress Control code here
}
});
ftp.completePendingCommand();
} catch (Exception e) { ... }
This snippet is a simple version of retrieveFile method. If you consult the original version, you will see that all you need to do is change the copyStream method according to your needs.
Q:After I perform a file transfer to the server, printWorkingDirectory() returns null.
*A:*You need to call completePendingCommand() after transferring the file. See the following snippet for a guide:
FTPClient client = new FTPClient();
client.addProtocolCommandListener(new PrintCommandListener(new
PrintWriter(System.out)));
client.connect("127.0.0.1");
client.login("rory", "rory");
for (FTPFile file : client.listFiles()) {
System.out.printf("%s %s [%d bytes]\n" ,(file.isDirectory() ? "[D]" : " "),
file.getName(), file.getSize());
}
client.enterLocalPassiveMode();
client.setFileType(FTPClient.BINARY_FILE_TYPE);
client.retrieveFile("A.pdf", new FileOutputStream("c:\\temp\\A.pdf"));
// Upload a file
InputStream fis = new FileInputStream("c:\\temp\\B.pdf");
OutputStream os = client.storeFileStream("B.pdf");
byte buf[] = new byte[8192];
int bytesRead = fis.read(buf);
while (bytesRead != -1) {
os.write(buf, 0, bytesRead);
bytesRead = fis.read(buf);
}
fis.close();
os.close();
client.completePendingCommand();
client.changeWorkingDirectory("Dir1");
for (FTPFile file : client.listFiles()) {
System.out.printf("%s %s [%d bytes]\n" ,(file.isDirectory() ? "[D]" : " "),
file.getName(), file.getSize());
}
System.out.println(client.printWorkingDirectory());
client.changeToParentDirectory();
client.changeWorkingDirectory("/D2");
System.out.println(client.printWorkingDirectory());
client.logout();
client.disconnect();
Q:There doesn't seem to be any opensource TFTP servers available. Has anyone written one for Commons/Net?
*A:*Yes. Commons/Net actually has a TFTP server. Unfortunately, due to a disagreement among the committers over whether Commons/Net should expose a new useful tool, or whether it should stay true to the Client-Side tools only mantra, the TFTP Server is not included in the binary builds, and no javadoc is built or exposed for it.
Instead, the TFTP Server was put into the JUnit tests. In the 2.0 release, you can find the TFTP server here in the source package: /commons-net-2.0-src/src/test/java/org/apache/commons/net/tftp/TFTPServer.java
In SVN, it lives here: TFTPServer
If you download the source package, and build it, it should be included in the tests jar file.