spark on kubernetes运行测试

测试环境

● kubernetes 1.20.15
● default命名空间
● spark 3.1.2
● kubectl

运行架构

在这里插入图片描述

构建镜像

  1. 配置JAVA_HOME
  2. 下载spark二进制包spark-3.1.2-bin-hadoop3.2.tgz并解压
  3. 修改kubernetes/dockerfiles/spark/Dockerfile文件
ARG java_image_tag=11-jre-slimFROM openjdk:${java_image_tag}# 默认185,有root需求这里可以修改为0
ARG spark_uid=0RUN set -ex && \
sed -i 's/http:\/\/deb.\(.*\)/https:\/\/deb.\1/g' /etc/apt/sources.list && \
apt-get update && \
ln -s /lib /lib64 && \
apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps && \
mkdir -p /opt/spark && \
mkdir -p /opt/spark/examples && \
mkdir -p /opt/spark/work-dir && \
touch /opt/spark/RELEASE && \
rm /bin/sh && \
ln -sv /bin/bash /bin/sh && \
echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
rm -rf /var/cache/apt/*# 可以提前将依赖的第三方jar包拷贝到jars目录下
COPY jars /opt/spark/jars
COPY bin /opt/spark/bin
COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
COPY kubernetes/tests /opt/spark/tests
COPY data /opt/spark/dataENV SPARK_HOME /opt/sparkWORKDIR /opt/spark/work-dir
RUN chmod g+w /opt/spark/work-dir
RUN chmod a+x /opt/decom.shENTRYPOINT [ "/opt/entrypoint.sh" ]# Specify the User that the actual main process will run as
USER ${spark_uid}
  1. 构建镜像与推送
./bin/docker-image-tool.sh -r harbor.gistack.cn/bigdata/spark-on-k8s -t 3.1.2 build
./bin/docker-image-tool.sh -r harbor.gistack.cn/bigdata/spark-on-k8s -t 3.1.2 push# 构建其它语言镜像
./bin/docker-image-tool.sh -r harbor.gistack.cn/bigdata/spark-on-k8s \-t 3.1.2-py -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build./bin/docker-image-tool.sh -r harbor.gistack.cn/bigdata/spark-on-k8s \-t 3.1.2-R -p ./kubernetes/dockerfiles/spark/bindings/R/Dockerfile build

启动spark任务

  1. 为spark driver创建SA账户及授权
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
  1. 提交自带实例任务
./bin/spark-submit \--master k8s://https://192.168.190.123:6443 \--deploy-mode cluster \--name spark-pi \--class org.apache.spark.examples.SparkPi \--conf spark.executor.instances=5 \--conf spark.kubernetes.container.image=harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2 \--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar
  1. 客户端运行日志
[root@k8stest spark-3.1.2-bin-hadoop3.2]# ./bin/spark-submit \
>   --master k8s://https://192.168.190.123:6443 \
>   --deploy-mode cluster \
>   --name spark-pi \
>   --class org.apache.spark.examples.SparkPi \
>   --conf spark.executor.instances=5 \
>   --conf spark.kubernetes.container.image=harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2 \
>   --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>   local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar
22/06/27 18:00:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/06/27 18:00:00 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
22/06/27 18:00:01 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
22/06/27 18:00:03 INFO LoggingPodStatusWatcherImpl: State changed, new state:pod name: spark-pi-02432381a49a8c05-drivernamespace: defaultlabels: spark-app-selector -> spark-a119352471db48d0a4a1083c79b48e0f, spark-role -> driverpod uid: bcd6ba46-a58e-42ad-9185-4355fbf78cdbcreation time: 2022-06-27T10:00:02Zservice account name: sparkvolumes: spark-local-dir-1, spark-conf-volume-driver, spark-token-g8sl4node name: vm190116start time: 2022-06-27T10:00:02Zphase: Pendingcontainer status:container name: spark-kubernetes-drivercontainer image: harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2container state: waitingpending reason: ContainerCreating
22/06/27 18:00:03 INFO LoggingPodStatusWatcherImpl: State changed, new state:pod name: spark-pi-02432381a49a8c05-drivernamespace: defaultlabels: spark-app-selector -> spark-a119352471db48d0a4a1083c79b48e0f, spark-role -> driverpod uid: bcd6ba46-a58e-42ad-9185-4355fbf78cdbcreation time: 2022-06-27T10:00:02Zservice account name: sparkvolumes: spark-local-dir-1, spark-conf-volume-driver, spark-token-g8sl4node name: vm190116start time: 2022-06-27T10:00:02Zphase: Pendingcontainer status:container name: spark-kubernetes-drivercontainer image: harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2container state: waitingpending reason: ContainerCreating
22/06/27 18:00:03 INFO LoggingPodStatusWatcherImpl: Waiting for application spark-pi with submission ID default:spark-pi-02432381a49a8c05-driver to finish...
22/06/27 18:00:04 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Pending)
22/06/27 18:00:05 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Pending)
22/06/27 18:00:05 INFO LoggingPodStatusWatcherImpl: State changed, new state:pod name: spark-pi-02432381a49a8c05-drivernamespace: defaultlabels: spark-app-selector -> spark-a119352471db48d0a4a1083c79b48e0f, spark-role -> driverpod uid: bcd6ba46-a58e-42ad-9185-4355fbf78cdbcreation time: 2022-06-27T10:00:02Zservice account name: sparkvolumes: spark-local-dir-1, spark-conf-volume-driver, spark-token-g8sl4node name: vm190116start time: 2022-06-27T10:00:02Zphase: Runningcontainer status:container name: spark-kubernetes-drivercontainer image: harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2container state: runningcontainer started at: 2022-06-27T10:00:04Z
22/06/27 18:00:06 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:07 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:08 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:09 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:10 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:11 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:12 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:13 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:14 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:15 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:16 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:17 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:18 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:19 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:20 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:21 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:22 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:23 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:24 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:25 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:26 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:27 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Running)
22/06/27 18:00:27 INFO LoggingPodStatusWatcherImpl: State changed, new state:pod name: spark-pi-02432381a49a8c05-drivernamespace: defaultlabels: spark-app-selector -> spark-a119352471db48d0a4a1083c79b48e0f, spark-role -> driverpod uid: bcd6ba46-a58e-42ad-9185-4355fbf78cdbcreation time: 2022-06-27T10:00:02Zservice account name: sparkvolumes: spark-local-dir-1, spark-conf-volume-driver, spark-token-g8sl4node name: vm190116start time: 2022-06-27T10:00:02Zphase: Succeededcontainer status:container name: spark-kubernetes-drivercontainer image: harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2container state: terminatedcontainer started at: 2022-06-27T10:00:04Zcontainer finished at: 2022-06-27T10:00:26Zexit code: 0termination reason: Completed
22/06/27 18:00:27 INFO LoggingPodStatusWatcherImpl: Application status for spark-a119352471db48d0a4a1083c79b48e0f (phase: Succeeded)
22/06/27 18:00:27 INFO LoggingPodStatusWatcherImpl: Container final statuses:container name: spark-kubernetes-drivercontainer image: harbor.gistack.cn/bigdata/spark-on-k8s:3.1.2container state: terminatedcontainer started at: 2022-06-27T10:00:04Zcontainer finished at: 2022-06-27T10:00:26Zexit code: 0termination reason: Completed
22/06/27 18:00:27 INFO LoggingPodStatusWatcherImpl: Application spark-pi with submission ID default:spark-pi-02432381a49a8c05-driver finished
22/06/27 18:00:27 INFO ShutdownHookManager: Shutdown hook called
22/06/27 18:00:27 INFO ShutdownHookManager: Deleting directory /tmp/spark-db110761-ab56-46b8-a40a-d811d28ff296
[root@k8stest spark-3.1.2-bin-hadoop3.2]#

  1. kubernetes上运行情况：
    在这里插入图片描述
    运行结束：在这里插入图片描述

  2. spark driver Pod日志

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ grep SPARK_JAVA_OPT_
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z x ']'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
+ case "$1" in
+ shift 1
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.42.5.96 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
22/06/27 10:00:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/06/27 10:00:08 INFO SparkContext: Running Spark version 3.1.2
22/06/27 10:00:08 INFO ResourceUtils: ==============================================================
22/06/27 10:00:08 INFO ResourceUtils: No custom resources configured for spark.driver.
22/06/27 10:00:08 INFO ResourceUtils: ==============================================================
22/06/27 10:00:08 INFO SparkContext: Submitted application: Spark Pi
22/06/27 10:00:08 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/06/27 10:00:08 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
22/06/27 10:00:08 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/06/27 10:00:08 INFO SecurityManager: Changing view acls to: root
22/06/27 10:00:08 INFO SecurityManager: Changing modify acls to: root
22/06/27 10:00:08 INFO SecurityManager: Changing view acls groups to: 
22/06/27 10:00:08 INFO SecurityManager: Changing modify acls groups to: 
22/06/27 10:00:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/06/27 10:00:09 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
22/06/27 10:00:09 INFO SparkEnv: Registering MapOutputTracker
22/06/27 10:00:09 INFO SparkEnv: Registering BlockManagerMaster
22/06/27 10:00:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/06/27 10:00:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/06/27 10:00:09 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/06/27 10:00:09 INFO DiskBlockManager: Created local directory at /var/data/spark-da112425-86c7-49b2-a0ab-c6609af9ec9e/blockmgr-0d1c82f2-2e1f-48c7-8698-283c8812642f
22/06/27 10:00:09 INFO MemoryStore: MemoryStore started with capacity 413.9 MiB
22/06/27 10:00:09 INFO SparkEnv: Registering OutputCommitCoordinator
22/06/27 10:00:10 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/06/27 10:00:10 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-pi-02432381a49a8c05-driver-svc.default.svc:4040
22/06/27 10:00:10 INFO SparkContext: Added JAR local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar at file:/opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar with timestamp 1656324008416
22/06/27 10:00:10 WARN SparkContext: The jar local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar has been added already. Overwriting of added jars is not supported in the current version.
22/06/27 10:00:10 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
22/06/27 10:00:13 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5 running: 0.
22/06/27 10:00:13 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
22/06/27 10:00:13 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
22/06/27 10:00:13 INFO NettyBlockTransferService: Server created on spark-pi-02432381a49a8c05-driver-svc.default.svc:7079
22/06/27 10:00:13 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/06/27 10:00:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-pi-02432381a49a8c05-driver-svc.default.svc, 7079, None)
22/06/27 10:00:13 INFO BlockManagerMasterEndpoint: Registering block manager spark-pi-02432381a49a8c05-driver-svc.default.svc:7079 with 413.9 MiB RAM, BlockManagerId(driver, spark-pi-02432381a49a8c05-driver-svc.default.svc, 7079, None)
22/06/27 10:00:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-pi-02432381a49a8c05-driver-svc.default.svc, 7079, None)
22/06/27 10:00:13 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-pi-02432381a49a8c05-driver-svc.default.svc, 7079, None)
22/06/27 10:00:13 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
22/06/27 10:00:13 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
22/06/27 10:00:13 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
22/06/27 10:00:13 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
22/06/27 10:00:20 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.42.5.97:54996) with ID 3,  ResourceProfileId 0
22/06/27 10:00:20 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.42.5.99:60040) with ID 2,  ResourceProfileId 0
22/06/27 10:00:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.42.5.97:43211 with 413.9 MiB RAM, BlockManagerId(3, 10.42.5.97, 43211, None)
22/06/27 10:00:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.42.5.99:33117 with 413.9 MiB RAM, BlockManagerId(2, 10.42.5.99, 33117, None)
22/06/27 10:00:21 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.42.5.98:38884) with ID 1,  ResourceProfileId 0
22/06/27 10:00:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.42.5.98:40293 with 413.9 MiB RAM, BlockManagerId(1, 10.42.5.98, 40293, None)
22/06/27 10:00:24 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.42.6.35:48062) with ID 4,  ResourceProfileId 0
22/06/27 10:00:24 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
22/06/27 10:00:24 INFO BlockManagerMasterEndpoint: Registering block manager 10.42.6.35:35003 with 413.9 MiB RAM, BlockManagerId(4, 10.42.6.35, 35003, None)
22/06/27 10:00:24 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
22/06/27 10:00:24 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
22/06/27 10:00:24 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
22/06/27 10:00:24 INFO DAGScheduler: Parents of final stage: List()
22/06/27 10:00:24 INFO DAGScheduler: Missing parents: List()
22/06/27 10:00:24 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
22/06/27 10:00:25 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.1 KiB, free 413.9 MiB)
22/06/27 10:00:25 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1816.0 B, free 413.9 MiB)
22/06/27 10:00:25 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-pi-02432381a49a8c05-driver-svc.default.svc:7079 (size: 1816.0 B, free: 413.9 MiB)
22/06/27 10:00:25 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1388
22/06/27 10:00:25 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
22/06/27 10:00:25 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0
22/06/27 10:00:25 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.42.5.98, executor 1, partition 0, PROCESS_LOCAL, 4597 bytes) taskResourceAssignments Map()
22/06/27 10:00:25 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (10.42.5.97, executor 3, partition 1, PROCESS_LOCAL, 4597 bytes) taskResourceAssignments Map()
22/06/27 10:00:25 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.42.5.98:40293 (size: 1816.0 B, free: 413.9 MiB)
22/06/27 10:00:25 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.42.5.97:43211 (size: 1816.0 B, free: 413.9 MiB)
22/06/27 10:00:26 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 848 ms on 10.42.5.97 (executor 3) (1/2)
22/06/27 10:00:26 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 905 ms on 10.42.5.98 (executor 1) (2/2)
22/06/27 10:00:26 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
22/06/27 10:00:26 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 1.165 s
22/06/27 10:00:26 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/06/27 10:00:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/06/27 10:00:26 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.368000 s
Pi is roughly 3.1293956469782347
22/06/27 10:00:26 INFO SparkUI: Stopped Spark web UI at http://spark-pi-02432381a49a8c05-driver-svc.default.svc:4040
22/06/27 10:00:26 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
22/06/27 10:00:26 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
22/06/27 10:00:26 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
22/06/27 10:00:26 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/06/27 10:00:26 INFO MemoryStore: MemoryStore cleared
22/06/27 10:00:26 INFO BlockManager: BlockManager stopped
22/06/27 10:00:26 INFO BlockManagerMaster: BlockManagerMaster stopped
22/06/27 10:00:26 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/06/27 10:00:26 INFO SparkContext: Successfully stopped SparkContext
22/06/27 10:00:26 INFO ShutdownHookManager: Shutdown hook called
22/06/27 10:00:26 INFO ShutdownHookManager: Deleting directory /var/data/spark-da112425-86c7-49b2-a0ab-c6609af9ec9e/spark-562c9819-1859-46c6-9c08-4487303da431
22/06/27 10:00:26 INFO ShutdownHookManager: Deleting directory /tmp/spark-3e96bc2f-e078-4078-a24c-8f1fb9d75162

总结

  1. 通过Client提交spark任务到kubernetes apiserver；
  2. kubernetes上启动spark driver Pod作为spark master角色；
  3. spark driver通过sa账户创建spark exec pod来运行任务；
  4. 无论spark driver还是spark exec都是通过kubernets scheduler调度完成分配的；
  5. 运行任务完成后自动删除spark exec pod，spark driver状态变为completed。

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.rhkb.cn/news/458441.html

如若内容造成侵权/违法违规/事实不符，请联系长河编程网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

HBuilder X 中Vue.js基础使用2（三）

HBuilder X 中Vue.js基础使用2（三）

一、条件渲染 1、条件判断 v-if ： 表达式返回真值时才被渲染 v-else ：表达式返回为假时不被渲染 2、 分支条件判断 v-else-if ：使用v-if , v-else-if 和 v-else 来表示其他的条件分支 3、显示隐藏 v-show v-show true 把节点显示 …
阅读更多...
持续深化信创布局，途普科技与统信软件完成产品兼容性互认证

持续深化信创布局，途普科技与统信软件完成产品兼容性互认证

近日，由北京途普科技有限公司（以下简称“途普科技”）自主研发的TopGraph图数据库及知识图谱构建平台已成功完成统信服务器操作系统V20的兼容性互认证，标志着途普科技在国产自控技术上又迈出了坚实的一步。 在各项严格的测试环节中…
阅读更多...
技术成神之路：设计模式（二十一）外观模式

技术成神之路：设计模式（二十一）外观模式

相关文章：技术成神之路：二十三种设计模式(导航页) 介绍 外观模式（Facade Pattern）是一种结构型设计模式，它为子系统中的一组接口提供一个统一的接口。外观模式定义了一个高层接口，使得子系统更容易使用。 …
阅读更多...
XJ02、消费金融｜消费金融业务模式中的主要主体

XJ02、消费金融｜消费金融业务模式中的主要主体

根据所持有牌照类型的不同，消费金融服务供给方主要分为商业银行、汽车金融公司、消费金融公司和小贷公司，不同类型机构定位不同、提供消费金融服务与产品类型也各不相同。此外，互联网金融平台也成为中国消费金融业务最重要的参与方之一，虽其并非持牌金融机构，但借助其流量…
阅读更多...
D50【python 接口自动化学习】- python基础之类

D50【python 接口自动化学习】- python基础之类

day50 init方法 学习日期：20241027 学习目标：类 -- 64 init方法：如何为对象传递参数？ 学习笔记： 魔术方法 init方法 class Klass(object):# 定义初始化方法，类实例化时自动进行初始化def __init__(self…
阅读更多...
AGI 之 【Dify】 之 Dify 在 Windows 端本地部署调用 Ollama 本地下载的大模型，实现 API 形式进行聊天对话

AGI 之 【Dify】 之 Dify 在 Windows 端本地部署调用 Ollama 本地下载的大模型，实现 API 形式进行聊天对话

AGI 之 【Dify】 之 Dify 在 Windows 端本地部署调用 Ollama 本地下载的大模型，实现 API 形式进行聊天对话 目录 AGI 之 【Dify】 之 Dify 在 Windows 端本地部署调用 Ollama 本地下载的大模型，实现 API 形式进行聊天对话 一、简单介绍 二、创建一个聊…
阅读更多...
基于SSM+小程序的旅游社交登录管理系统（旅游4）

基于SSM+小程序的旅游社交登录管理系统（旅游4）

👉文末查看项目功能视频演示获取源码sql脚本视频导入教程视频 1、项目介绍 ​ 本旅游社交小程序功能有管理员和用户。管理员有个人中心，用户管理，每日签到管理，景点推荐管理，景点分类管理，防疫查询管理&a…
阅读更多...
洞察前沿趋势！2024深圳国际金融科技大赛——西丽湖金融科技大学生挑战赛技术公开课指南

洞察前沿趋势！2024深圳国际金融科技大赛——西丽湖金融科技大学生挑战赛技术公开课指南

在当前信息技术与“互联网”深度融合的背景下，金融行业的转型升级是热门话题，创新与发展成为金融科技主旋律。随着区块链技术、人工智能技术、5G通信技术、大数据技术等前沿科技的飞速发展，它们与金融领域的深度融合，正引领着新型…
阅读更多...
Golang 怎么高效处理ACM模式输入输出

Golang 怎么高效处理ACM模式输入输出

文章目录 问题bufio.NewReader高效的原理 再次提交 问题 最近在练习牛客上单调栈题目时，要求自己处理出入输出，也就是读题库要求的输入，计算最终结果，并打印输出 当我用fmt.Scan处理输入，用fmt.Println处理输出时&am…
阅读更多...
R语言笔记（五）：Apply函数

R语言笔记（五）：Apply函数

文章目录 一、Apply Family二、apply(): rows or columns of a matrix or data frame三、Applying a custom function四、Applying a custom function "on-the-fly"五、Applying a function that takes extra arguments六、Whats the return argument?七、Optimized…
阅读更多...
linux开机自启动三种方式

linux开机自启动三种方式

方式一、 1：rc.local 文件 1、执行命令：编辑 “/etc/rc.local” vi /ect/rc.local 2、然后在文件最后一行添加要执行程序的全路径。 例如，每次开机时要执行一个 hello.sh，这个脚本放在 / usr 下面，那就可以在 “/et…
阅读更多...
深入了解 Android 中的命名空间：`xmlns:tools` 和其他常见命名空间

深入了解 Android 中的命名空间：`xmlns:tools` 和其他常见命名空间

在 Android 开发中，xmlns （.xml的namespace）命名空间是一个非常重要的概念。通过引入不同的命名空间，可以使用不同的属性来设计布局、设置工具属性或者支持自定义视图等。除了 xmlns:tools 以外，还有很多常见的命名空间…
阅读更多...
动态IP是什么？

动态IP是什么？

随着互联网成为人们生活的重要组成部分，以信息传递为主导的时代种，网络连接质量对我们的工作效率、学习进度以及娱乐体验等方面都有很大影响。 动态IP，作为网络连接中的一种重要IP代理形式，越来越受到用户的欢迎。本文将深入解析…
阅读更多...
计算机网络-CSMA/CD协议笔记及“争用期”的理解

计算机网络-CSMA/CD协议笔记及“争用期”的理解

假设a和b是总线型网络上相距最远的两个节点。 从零这个时刻a节点会往信道上发送数据，那么a节点发送的第一个比特，需要经过τ这么长的时间，也就是经过一个单向的传播时延之后。它的这个信号才可以被最远的这个节点检测到。那如果b结点在τ这个…
阅读更多...
以bat脚本实现自动识别盘符名称

以bat脚本实现自动识别盘符名称

以bat脚本实现自动识别盘符名称 引言以bat脚本实现自动识别盘符名称运行结果 引言 请听题，如何自动识别电脑盘符的名称，比如，F盘的盘符名称为office，我应该如何自动识别呢？ 这里我是以bat脚本实现 以bat脚本实现自动…
阅读更多...
平均误差ME、均方误差MSE、均方根误差RMSE、平均均方根误差ARMSE辨析

平均误差ME、均方误差MSE、均方根误差RMSE、平均均方根误差ARMSE辨析

四个性能指标的定义和作用的解释 ME(k) - 平均误差(Mean Error) 公式: M E ( k ) = ( 1 / M ) ∗ Σ ( x k − x ^ k ) , m = 1 , . . . , M ME(k) = (1/M) * Σ(xk - x̂k), m = 1, ..., M ME(k)=(1/M)∗Σ(xk−
阅读更多...
VUE3实现古典音乐网站源码模板

VUE3实现古典音乐网站源码模板

文章目录 1.设计来源1.1 网站首页页面1.2 古典音乐页面1.3 著名人物页面1.4 古典乐器页面1.5 历史起源页面1.6 登录页面1.7 注册页面 2.效果和源码2.1 动态效果2.2 目录结构 源码下载万套模板，程序开发，在线开发，在线沟通 作者：xc…
阅读更多...
【Unity踩坑】UWP应用未通过Windows应用认证：API不支持

【Unity踩坑】UWP应用未通过Windows应用认证：API不支持

在将Unity项目导出为XAML类型的UWP项目后，通过Visual Studio打包成功，但在进行Windows应用认证时结果是Failed。 其中的错误是某些dll里用到了Windows SDK不支持的API。 本次问题中涉及到的具体dll有两个：gilzoide-sqlite-net.dll和D3D12Cor…
阅读更多...
排序

排序

插入排序（最有价值） 类似于摸牌 InsertSort：O(N^2)；最好：O(N) 最坏情况：逆序有序 最好情况：O(N)顺序有序 比冒泡排序实际应用更高效 以下是单趟排序，实现多趟需要再嵌套一个fo…
阅读更多...
IDEA初探：深入理解 Structure 功能

IDEA初探：深入理解 Structure 功能

一、Structure - 类视图 Structure 是 IDEA 中的一个视图工具，它提供了对当前文件中结构元素的快速访问。通过 Structure，我们可以方便地查看和导航到代码中的各个部分，从而提高代码编辑和浏览的效率。 1.1 基本概念 Structure 视图以树形结…
阅读更多...
最新文章
推荐文章

Copyright @ 2022~2023