杰霸霸博客

主题

Hadoop整合S3 ​

S3(Simple Storage Service)相比HDFS更加通用，是Hadoop存算分离架构中的一种存储解决方案。Hadoop整合S3后，Hadoop可以访问和使用S3存储。

1. S3依赖添加 ​

Hadoop要访问s3a://协议，而Hadoop自带S3依赖但没有在classpath里，需要手动添加。位置如下：

shell
$HADOOP_HOME/share/hadoop/tools/lib/hadoop-aws-3.3.6.jar
$HADOOP_HOME/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.367.jar

配置HADOOP_CLASSPATH环境变量：

shell
## 首先通过hadoop classpath得到现有classpath
HADOOP_CLASSPATH=$(hadoop classpath)
## 添加S3依赖
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*

如果配置yarn-site.xml文件，里面指明了hadoop classpath的话需要修改调整：

xml
<property>
    <name>yarn.application.classpath</name>
    <value>D:\hadoop-3.3.6\etc\hadoop;D:\hadoop-3.3.6\share\hadoop\common;D:\hadoop-3.3.6\share\hadoop\common\lib\*;D:\hadoop-3.3.6\share\hadoop\common\*;D:\hadoop-3.3.6\share\hadoop\hdfs;D:\hadoop-3.3.6\share\hadoop\hdfs\lib\*;D:\hadoop-3.3.6\share\hadoop\hdfs\*;D:\hadoop-3.3.6\share\hadoop\yarn;D:\hadoop-3.3.6\share\hadoop\yarn\lib\*;D:\hadoop-3.3.6\share\hadoop\yarn\*;D:\hadoop-3.3.6\share\hadoop\mapreduce\*;D:\hadoop-3.3.6\share\hadoop\tools\lib\*;D:\hadoop-3.3.6\share\hadoop\tools\*</value>
</property>

2. 配置访问S3 ​

在Hadoop的配置文件中添加S3的访问信息，修改core-site.xml文件：

xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.s3a.endpoint</name>
        <value>http://172.16.44.75:8082</value>
    </property>
    <property>
        <name>fs.s3a.access.key</name>
        <value>你的访问密钥</value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <value>你的密钥</value>
    </property>
    <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.connection.timeout</name>
        <value>60000</value>
    </property>
    <property>
        <name>fs.s3a.socket.timeout</name>
        <value>60000</value>
    </property>
    <property>
        <name>fs.s3a.attempts.maximum</name>
        <value>10</value>
    </property>
    <property>
        <name>fs.s3a.etag.checksum.enabled</name>
        <value>true</value>
    </property>
</configuration>

更改完毕并重启集群后，Hadoop即可访问S3。

3. 验证S3访问 ​

shell
PS D:\hadoop-3.3.6\sbin> hadoop fs -ls s3a://gbasehd/
2026-06-08 10:07:20,638 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2026-06-08 10:07:20,666 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2026-06-08 10:07:20,666 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
Found 4 items
drwxrwxrwx   - Administrator Administrator          0 2026-06-08 10:07 s3a://gbasehd/hbase
drwxrwxrwx   - Administrator Administrator          0 2026-06-08 10:07 s3a://gbasehd/paimon
drwxrwxrwx   - Administrator Administrator          0 2026-06-08 10:07 s3a://gbasehd/test
drwxrwxrwx   - Administrator Administrator          0 2026-06-08 10:07 s3a://gbasehd/user
2026-06-08 10:07:21,811 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2026-06-08 10:07:21,811 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2026-06-08 10:07:21,811 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.