1.文件上传下载的常用方法
文件上传下载是一种非常常见的功能,特别是在web服务网站。
常用的文件上传下载协议有以下几种:
- FTP(File Transfer Protocol):是一种用于在计算机间传输文件的标准网络协议。它使用客户端-服务器架构,通过对服务器的连接进行认证和授权,允许用户将文件上传到服务器或从服务器下载文件。
- HTTP(s):超文本传输协议(HTTP)是用于在互联网上发送和接收超文本的协议。HTTP允许通过HTTP请求和响应进行文件的上传和下载。通过HTTPS可以提供更安全的传输。
- SCP(Secure Copy):SCP是一种安全的文件传输协议,基于SSH(Secure Shell)协议。它可以在本地计算机和远程计算机之间进行文件传输,提供了加密和身份验证的功能。
- SFTP(SSH File Transfer Protocol):SFTP是一个基于SSH的安全文件传输协议。它使用SSH进行身份验证和加密,可以在本地计算机和远程计算机之间进行安全的文件传输。
- 除了以上几种应用层上的标准协议,我们甚至可以使用原生的socket构建文件服务器。使用Tcp搭建文件服务,采用的协议为私有协议。
2.服务端代码实现
本文演示如何使用netty搭建基于http协议的文件下载服务。(例子来自于netty官方example,这里做了简化,去掉ssl,去掉浏览器文件缓存)
2.1、构建ServerBootstrap
文件下载基于http协议实现,所以跟创建http服务很类似
public final class HttpStaticFileServer {private EventLoopGroup bossGroup = new NioEventLoopGroup(1);private EventLoopGroup workerGroup = new NioEventLoopGroup();private HttpFileConfig ftpConfig;public HttpStaticFileServer(HttpFileConfig ftpConfig) {this.ftpConfig = ftpConfig;}public void start() throws Exception {try {ServerBootstrap b = new ServerBootstrap();b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class).childHandler(new ChannelInitializer<SocketChannel>() {@Overridepublic void initChannel(SocketChannel ch) {ch.pipeline().addLast(new HttpServerCodec());ch.pipeline().addLast(new HttpObjectAggregator(65536));ch.pipeline().addLast(new HttpStaticFileServerHandler(ftpConfig));}});Channel ch = b.bind(ftpConfig.getPort()).sync().channel();System.err.println("http file server listens at " +"http://127.0.0.1:" + ftpConfig.getPort() + '/');ch.closeFuture().sync();} catch (Exception e) {shutdown();throw e;}}public void shutdown() {bossGroup.shutdownGracefully();workerGroup.shutdownGracefully();}}
总共添加3个handler,作用如下
HttpServerCodec:将HTTP请求消息从字节流解码为HttpRequest对象,将HTTP响应消息从HttpResponse对象编码为字节流。它还负责处理HTTP的编解码细节,如HTTP头部的解析和生成、HTTP消息的分块处理等。
HttpObjectAggregator:将多个HTTP请求或响应消息进行聚合,形成一个完整的HttpObject对象。在HTTP请求中,它可以将HTTP请求头部和请求体合并为一个FullHttpRequest对象;在HTTP响应中,它可以将HTTP响应头部和响应体合并为一个FullHttpResponse对象。HttpStaticFileServerHandler:自定义类,用于处理上一个节点解析的FullHttpRequest。
2.2、自定义类处理客户端的http请求
public class HttpStaticFileServerHandler extends SimpleChannelInboundHandler<FullHttpRequest> {private HttpFileConfig httpConfig;HttpStaticFileServerHandler(HttpFileConfig config) {this.httpConfig = config;}@Overridepublic void channelRead0(ChannelHandlerContext ctx, FullHttpRequest request) throws Exception {if (!request.decoderResult().isSuccess()) {sendError(ctx, BAD_REQUEST);return;}if (request.method() != GET) {sendError(ctx, METHOD_NOT_ALLOWED);return;}final String uri = request.uri();final String path = sanitizeUri(uri);if (path == null) {sendError(ctx, FORBIDDEN);return;}File file = new File(path);if (file.isHidden() || !file.exists()) {sendError(ctx, NOT_FOUND);return;}if (file.isDirectory()) {if (uri.endsWith("/")) {sendFileListing(ctx, file, uri);} else {sendRedirect(ctx, uri + '/');}return;}if (!file.isFile()) {sendError(ctx, FORBIDDEN);return;}RandomAccessFile raf;try {raf = new RandomAccessFile(file, "r");} catch (FileNotFoundException ignore) {sendError(ctx, NOT_FOUND);return;}long fileLength = raf.length();HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);HttpUtil.setContentLength(response, fileLength);setContentTypeHeader(response, file);response.headers().set(HttpHeaderNames.CONTENT_DISPOSITION, String.format("filename='%s'", URLEncoder.encode(file.getName(), "UTF-8")));if (HttpUtil.isKeepAlive(request)) {response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.KEEP_ALIVE);}// Write the initial line and the header.ctx.write(response);// Write the content.ChannelFuture sendFileFuture;ChannelFuture lastContentFuture;sendFileFuture =ctx.write(new DefaultFileRegion(raf.getChannel(), 0, fileLength), ctx.newProgressivePromise());lastContentFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);sendFileFuture.addListener(new ChannelProgressiveFutureListener() {@Overridepublic void operationProgressed(ChannelProgressiveFuture future, long progress, long total) {if (total < 0) {System.err.println(future.channel() + " Transfer progress: " + progress);} else {System.err.println(future.channel() + " Transfer progress: " + progress + " / " + total);}}@Overridepublic void operationComplete(ChannelProgressiveFuture future) {System.err.println(future.channel() + " Transfer complete.");}});if (!HttpUtil.isKeepAlive(request)) {lastContentFuture.addListener(ChannelFutureListener.CLOSE);}}@Overridepublic void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {cause.printStackTrace();if (ctx.channel().isActive()) {sendError(ctx, INTERNAL_SERVER_ERROR);}}private String sanitizeUri(String uri) {try {uri = URLDecoder.decode(uri, "UTF-8");} catch (UnsupportedEncodingException e) {throw new Error(e);}if (uri.isEmpty() || uri.charAt(0) != '/') {return null;}uri = uri.replace('/', File.separatorChar);// 安全验证if (!httpConfig.getSafeRule().test(uri)) {return null;}// Convert to absolute path.return httpConfig.getFileDirectory() + File.separator + uri;}private void sendFileListing(ChannelHandlerContext ctx, File dir, String dirPath) {FullHttpResponse response = new DefaultFullHttpResponse(HTTP_1_1, OK);response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/html; charset=UTF-8");StringBuilder buf = new StringBuilder().append("<!DOCTYPE html>\r\n").append("<html><head><meta charset='utf-8' /><title>").append("Listing of: ").append(dirPath).append("</title></head><body>\r\n").append("<h3>Listing of: ").append(dirPath).append("</h3>\r\n").append("<ul>").append("<li><a href=\"../\">..</a></li>\r\n");for (File f : dir.listFiles()) {if (f.isHidden() || !f.canRead()) {continue;}String name = f.getName();if (!httpConfig.getSafeRule().test(name)) {continue;}buf.append("<li><a href=\"").append(name).append("\">").append(name).append("</a></li>\r\n");}buf.append("</ul></body></html>\r\n");ByteBuf buffer = Unpooled.copiedBuffer(buf, CharsetUtil.UTF_8);response.content().writeBytes(buffer);buffer.release();ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);}private void sendRedirect(ChannelHandlerContext ctx, String newUri) {FullHttpResponse response = new DefaultFullHttpResponse(HTTP_1_1, FOUND);response.headers().set(HttpHeaderNames.LOCATION, newUri);// Close the connection as soon as the error message is sent.ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);}private void sendError(ChannelHandlerContext ctx, HttpResponseStatus status) {FullHttpResponse response = new DefaultFullHttpResponse(HTTP_1_1, status, Unpooled.copiedBuffer("Failure: " + status + "\r\n", CharsetUtil.UTF_8));response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/plain; charset=UTF-8");ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);}private static void setContentTypeHeader(HttpResponse response, File file) {MimetypesFileTypeMap mimeTypesMap = new MimetypesFileTypeMap();response.headers().set(HttpHeaderNames.CONTENT_TYPE, mimeTypesMap.getContentType(file.getPath()));}
}
对于客户端的请求,如果是目录则枚举目录文件,封装成html的<a>超链接标签;如果是文件,则提供下载。
其中下载用到了DefaultFileRegion工具。
2.3、DefaultFileRegion解析
DefaultFileRegion可以利用零拷贝(zero-copy)技术,将文件内容直接从文件系统读取,并将数据传输到网络中,而无需经过中间缓冲区的复制。这种方式可以大大提高文件传输的效率,并减少CPU和内存的消耗。非常适用于需要传输大文件或者高并发的文件传输场景。
使用DefaultFileRegion写入文件之后,需要再写入特殊的httpconent表示接受,单例内容LastHttpContent.EMPTY_LAST_CONTENT。
需要注意的是,DefaultFileRegion只适用于文件服务器直接推送给客户端情况下使用。如果文件服务器需要将文件加载到内存,则不适合使用。改用下面的HttpChunkedInput。
2.4、使用HttpChunkedInput传输大文件
使用HttpChunkedInput,可以将需要传输的数据分割成多个HttpContent块,并通过Netty的HTTP编解码器进行传输。这样,发送方可以在需要的时候生成和发送每个块,接收方可以在接收到每个块时进行处理。
HttpChunkedInput可以与Netty的HTTP编解码器无缝集成,使得在进行HTTP分块传输时非常方便。对于需要处理大量数据的应用场景,使用HttpChunkedInput可以提高性能和降低内存消耗,从而更好地处理大规模数据传输。
使用HttpChunkedInput,则不需要手动添加LastHttpContent.EMPTY_LAST_CONTENT。内部代码自行添加
public HttpContent readChunk(ByteBufAllocator allocator) throws Exception {if (this.input.isEndOfInput()) {if (this.sentLastChunk) {return null;} else {this.sentLastChunk = true;return this.lastHttpContent;}} else {ByteBuf buf = (ByteBuf)this.input.readChunk(allocator);return buf == null ? null : new DefaultHttpContent(buf);}}
ServerBootstrap添加ChunkedWriteHandler()配合处理大文件分块问题
ServerBootstrap b = new ServerBootstrap();b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class).childHandler(new ChannelInitializer<SocketChannel>() {@Overridepublic void initChannel(SocketChannel ch) {ch.pipeline().addLast(new HttpServerCodec());ch.pipeline().addLast(new HttpObjectAggregator(65536));ch.pipeline().addLast(new ChunkedWriteHandler());ch.pipeline().addLast(new HttpStaticFileServerHandler(ftpConfig));}});
2.5文件服务配置类
配置类复制端口设定,文件目录设定,文件安全检测策略设定等。
public class HttpFileConfig {/*** 需要提供对外服务的文件目录*/private String fileDirectory;/*** 对外服务端口,默认为8080*/private int port = 8080;/*** url安全规则判定,默认运行访问制定目录下所有文件*/private Predicate<String> safeRule = new Predicate<String>() {@Overridepublic boolean test(String s) {return true;}};//省略setter/getter}
2.6启动入口
public class Main {public static void main(String[] args) throws Exception {HttpFileConfig cfg = new HttpFileConfig();cfg.setFileDirectory("F:\\projects");HttpStaticFileServer fileServer = new HttpStaticFileServer(cfg);fileServer.start();}
}
3.客户端演示
3.1netty客户端
使用netty,也可以实现文件下载功能。
public class HttpDownloadClient {private static final String SERVER_HOST = "localhost";private static final int SERVER_PORT = 8080;private static EventLoopGroup group = new NioEventLoopGroup();public static void main(String[] args) throws Exception {try {Bootstrap b = new Bootstrap();b.group(group).channel(NioSocketChannel.class).option(ChannelOption.TCP_NODELAY, true).handler(new ChannelInitializer<SocketChannel>() {@Overridepublic void initChannel(SocketChannel ch) throws Exception {ch.pipeline().addLast(new HttpClientCodec());ch.pipeline().addLast(new ChunkedWriteHandler());ch.pipeline().addLast(new HttpObjectAggregator(65536));ch.pipeline().addLast(new HttpFileDownloadClientHandler());}});ChannelFuture future = b.connect(SERVER_HOST, SERVER_PORT).sync();// 构建HTTP GET请求DefaultHttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/pom.xml");// 设置请求头信息,指定下载文件的名称和保存路径requestSetting(request);// 发送HTTP GET请求future.channel().writeAndFlush(request);future.channel().closeFuture().sync();} finally {group.shutdownGracefully();}}private static void requestSetting(DefaultHttpRequest request) {request.headers().set(HttpHeaderNames.HOST, SERVER_HOST);request.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);request.headers().set(HttpHeaderNames.ACCEPT_ENCODING, HttpHeaderValues.GZIP);request.headers().set(HttpHeaderNames.ACCEPT_CHARSET, "ISO-8859-1,utf-8;q=0.7,*;q=0.7");request.headers().set(HttpHeaderNames.USER_AGENT, "Netty File Download Client");request.headers().set(HttpHeaderNames.ACCEPT_LANGUAGE, "en-us,en;q=0.5");request.headers().set(HttpHeaderNames.ACCEPT, "*/*");request.headers().set(HttpHeaderNames.RANGE, "bytes=0-");request.headers().set(HttpHeaderNames.CONTENT_TYPE, "application/octet-stream");request.headers().set(HttpHeaderNames.CONTENT_LENGTH, HttpHeaderValues.ZERO);}static class HttpFileDownloadClientHandler extends SimpleChannelInboundHandler<FullHttpResponse> {@Overridepublic void channelActive(ChannelHandlerContext ctx) throws Exception {}@Overridepublic void channelInactive(ChannelHandlerContext ctx) throws Exception {}private static final Pattern FILENAME_PATTERN = Pattern.compile("filename\\*?=\"?(?:UTF-8'')?([^\";]*)\"?;?");@Overrideprotected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse response) throws Exception {System.out.println(response);String contentType = response.headers().get(HttpHeaderNames.CONTENT_TYPE);if (contentType.contains("html")) {if (response.getDecoderResult().isSuccess() && response.content().isReadable()) {String html = response.content().toString(io.netty.util.CharsetUtil.UTF_8);System.out.println(html);}} else if (contentType.contains("application/octet-stream")) {Matcher matcher = FILENAME_PATTERN.matcher(response.headers().get(HttpHeaderNames.CONTENT_DISPOSITION));String fileName = "f@" + System.currentTimeMillis();if (matcher.find()) {fileName = matcher.group(1).replaceAll("'", "").replaceAll("\"", "");System.out.println("下载文件:" + fileName);}RandomAccessFile file = new RandomAccessFile("download/"+fileName, "rw");FileChannel fileChannel = file.getChannel();fileChannel.write(response.content().nioBuffer()); // 将数据从ByteBuffer写入到RandomAccessFile}}}
}
通过解析 FullHttpResponse的Header,如果是html文本则显示html内容;如果是字节流则保存到本地目录。
本地测试了以下,一个3.5G的电影,大概需要60秒。
进一步拓展
1.增加用户界面;
2.增加文件断点续传;
3.多线程下载;
3.2浏览器客户端
直接在浏览器显示,当然最方便。