spark 写入mysql 中文数据 显示?? 或者 乱码

目录

前言

Spark报错:

解决办法:

总结一下:

报错:

解决:


前言

用spark写入mysql中,查看中文数据 显示?? 或者 乱码

Spark报错:

Sat Nov 23 19:15:59 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:02,903{yy/MM/dd java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 more
2024-11-23 19:16:02,921{yy/MM/dd Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 76 in stage 12.0 failed 1 times, most recent failure: Lost task 76.0 in stage 12.0 (TID 242) (LAPTOP-P70BBG66 executor driver): java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 moreDriver stacktrace:at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2253)at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2202)at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2201)at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2201)at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1078)at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1078)at scala.Option.foreach(Option.scala:274)at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1078)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2440)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2382)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2371)at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2202)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2223)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2242)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2267)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$1(RDD.scala:1020)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:1018)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.saveTable(JdbcUtils.scala:854)at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:63)at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:301)at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:817)at com._02.gy.GZ.zhibiao.Test01$.main(Test01.scala:61)at com._02.gy.GZ.zhibiao.Test01.main(Test01.scala)
Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 more
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,282{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,330{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,386{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)Process finished with exit code 1

解决办法:

我试了网上的很多办法:在saprk里面更改编码,直接把这一列更改为UTF-8

.withColumn("ChangeRecordState", decode($"ChangeRecordState", "UTF-8"))

使用新的driver,然后更改maven里面的jdbc链接版本

properties.setProperty("driver", "com.mysql.cj.jdbc.Driver")

等等....都没有解决问题

所以我开始着手自己解决,问题的核心无非就是编码冲突问题,因为已经写入进去了,所以连接配置啥的都没问题,主要解决编码问题

spark中可以打印出中文字段所以spark基本没有什么问题,我还是用了老方法

mysqlURL的配置项上加了?useUnicode=true&characterEncoding=UTF-8

val mysqlUrl = "jdbc:mysql://bigdata1:3306/shtd_industry?useUnicode=true&characterEncoding=UTF-8"

主要解决的是mysql,所以我开始查询mysql的默认编码是什么,我找到了命令

SHOW TABLE STATUS LIKE 'machine_state_time';

找到Collation字段对应的就是默认的编码  --  latin1_swedish_ci

找到了默认编码不是UTF-8,所以这里问题很明显了,就是把 latin1_swedish_ci 变成 UTF-8

所以我又接着查询,把表转换编码的命令

alter table machine_state_time character set utf8;

结果还是不行,当我试着找出问题,才发现我只把表改成了UTF-8,没有把字段改成UTF-8

alter table machine_state_time convert to character set utf8;

这样mysql的问题也解决了,运行一下还是报错

找到问题,scala语言也需要设置编码方式,所以我又加了一条配置项

System.setProperty("file.encoding", "UTF-8")

这样就可以了,运行成功,然后查看一下mysql中的内容

总结一下:

报错:

主要原因是因为编码冲突,scala,spark,mysql-connect连接,mysql三个编码不一致

解决:

设置scala的编码:

System.setProperty("file.encoding", "UTF-8")

设置spark的编码:

saprk本身就是UTF-8 不用设置

设置mysqlURL的编码:

val mysqlUrl = "jdbc:mysql://bigdata1:3306/shtd_industry?useUnicode=true&characterEncoding=UTF-8"

设置mysql本身的编码:

用两条命令,分别是:

//修改目标表所有字段编码格式为utf8
alter table machine_state_time convert to character set utf8;
//修改目标表表编码格式为utf8
alter table machine_state_time character set utf8;

好的,今天的分享就到这里了,希望对你有帮助,有帮助可以点点赞,评论评论

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.rhkb.cn/news/479545.html

如若内容造成侵权/违法违规/事实不符,请联系长河编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

微信小程序条件渲染与列表渲染的全面教程

微信小程序条件渲染与列表渲染的全面教程 引言 在微信小程序的开发中,条件渲染和列表渲染是构建动态用户界面的重要技术。通过条件渲染,我们可以根据不同的状态展示不同的内容,而列表渲染则使得我们能够高效地展示一组数据。本文将详细讲解这两种渲染方式的用法,结合实例…

ctfshow

1,web153 大小写绕过失败 使用.user.ini 来构造后⻔ php.ini是php的⼀个全局配置⽂件,对整个web服务起作⽤;⽽.user.ini和.htaccess⼀样是⽬录的配置⽂件,.user.ini就是⽤户⾃定义的⼀个php.ini,我们可以利⽤这个⽂件来构造后⻔和…

【大数据学习 | Spark-SQL】Spark-SQL编程

上面的是SparkSQL的API操作。 1. 将RDD转化为DataFrame对象 DataFrame: DataFrame是一种以RDD为基础的分布式数据集,类似于传统数据库中的二维表格。带有schema元信息,即DataFrame所表示的二维表数据集的每一列都带有名称和类型。这样的数…

DINO-X:一种用于开放世界目标检测与理解的统一视觉模型

摘要 本文介绍了由IDEA Research开发的DINO-X,这是一个统一的以对象为中心的视觉模型,具有迄今为止最佳的开放世界对象检测性能。DINO-X采用了与Grounding DINO 1.5 [47]相同的基于Transformer的编码器-解码器架构,以追求面向开放世界对象理…

MySQL系列之远程管理(安全)

导览 前言Q:如何保障远程登录安全一、远程登录的主要方式1. 用户名/口令2. SSH3. SSL/TLS 二、使用TLS协议加密连接1. 服务端2. 客户端 结语精彩回放 前言 在我们的学习或工作过程中,作为开发、测试或运维人员,经常会通过各类客户端软件&…

扫振牙刷设计思路以及技术解析

市面上目前常见的就两种:扫振牙刷和超声波牙刷 为了防水,表面还涂上了一层防水漆 一开始的电池管理芯片,可以让充电更加均衡。 如TP4056 第一阶段以恒流充电;当电压达到预定值时转入第二阶段进行恒压充电,此时电流逐…

Hot100 - 除自身以外数组的乘积

Hot100 - 除自身以外数组的乘积 最佳思路: 此问题的关键在于通过两次遍历,分别计算从左侧和右侧开始的累积乘积,以此避免使用额外的除法操作。 时间复杂度: 该算法的时间复杂度为 O(n),因为我们只需要遍历数组两次。…

一个vue项目如何运行在docker

将 Vue.js 应用程序通过 Docker 发布是一个非常常见的做法,它可以帮助你轻松地部署应用到不同的环境中。下面是一个简单的指南,介绍如何为 Vue.js 项目创建 Dockerfile 并进行构建和运行。 第一步:安装 Docker 确保你的开发机器上已经安装了…

【公益接口】不定时新增接口,仅供学习

文章日期:2024.11.24 使用工具:Python 文章类型:公益接口 文章全程已做去敏处理!!! 【需要做的可联系我】 AES解密处理(直接解密即可)(crypto-js.js 标准算法&#xff…

使用phpStudy小皮面板模拟后端服务器,搭建H5网站运行生产环境

一.下载安装小皮 小皮面板官网下载网址:小皮面板(phpstudy) - 让天下没有难配的服务器环境! 安装说明(特别注意) 1. 安装路径不能包含“中文”或者“空格”,否则会报错(例如错误提示:Cant cha…

DolphinDB 登陆伦敦!携手中英人工智能协会共话 AI 未来

11 月 9 日,DolphinDB 联合中英人工智能协会(CBAIA)在全球人工智能中心、今年三位诺贝尔奖得主的诞生地——伦敦盖茨比计算神经科学中心举办 AI 技术交流会。来自人工智能、量化投资等领域的 150 多位全球专家齐聚一堂,共同探讨人…

爬虫与反爬-Ja3指纹风控(Just a moment...)处理方案及参数说明

概述:本文将针对 Ja3 指纹检测风控进行处理,举例了一个案例并使用两种不同的破解方案进行突破,同时深入了解指纹间不同字符所代表的含义 指纹检测背景: 1、每一个设备、软件都有独属于自己的设备信息、版本号、加密算法、椭圆算…

数据结构——排序算法第二幕(交换排序:冒泡排序、快速排序(三种版本) 归并排序:归并排序(分治))超详细!!!!

文章目录 前言一、交换排序1.1 冒泡排序1.2 快速排序1.2.1 hoare版本 快排1.2.2 挖坑法 快排1.2.3 lomuto前后指针 快排 二、归并排序总结 前言 继上篇学习了排序的前面两个部分:直接插入排序和选择排序 今天我们来学习排序中常用的交换排序以及非常稳定的归并排序 快排可是有多…

华为云云连接+squid进行正向代理上网冲浪

1 概述 ‌Squid‌是一个高性能的代理缓存服务器,主要用于缓冲Internet数据。它支持多种协议,包括FTP、gopher、HTTPS和HTTP。Squid通过一个单独的、非模块化的、I/O驱动的进程来处理所有的客户端请求,这使得它在处理请求时具有较高的效率‌。…

杰发科技AC7803——不同晶振频率时钟的配置

计算公式 PLL_POSDIV [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62] PLL_PREDIV_1 1 2 4 USE_XTAL 24M SYSCLK_FREQ 64M SYSCLK_DIVIDER 1 VCO USE_XTAL*…

攸信技术:运动文化激发企业活力,赋能体育行业新未来

在攸信技术,运动文化如同春日暖阳,温暖着每一位员工的心。这份文化,源自盈趣科技的深厚底蕴,橙色不仅传递着3POS文化中的激情与活力,更成为了攸信人共同的精神标识。公司的每一个角落,都洋溢着对运动的热爱…

【ubuntu24.04】GTX4700 配置安装cuda

筛选显卡驱动显卡驱动 NVIDIA-Linux-x86_64-550.135.run 而后重启:最新的是12.6 用于ubuntu24.04 ,但是我的4700的显卡驱动要求12.4 cuda

LightRAG - 更快更便宜的GraphRAG

检索增强生成(Retrieval-Augmented Generation, RAG)已经成为提升大型语言模型(LLMs)能力的重要方法之一,通过整合外部知识,显著改善了生成内容的质量和相关性。 RAG 的局限性 传统的 RAG 系统虽然表现优…

TCP/IP协议攻击与防范

一、TCP/IP协议攻击介绍 1.1 Internet的结构​ LAN:局域网 WAN:广域网 WLAN:无线局域网 私有IP地址与公有IP地址? 私有地址:A类:10.0.0.0~10.255.255.255 B类:172.16.0.0~172.31.255.255…

机器学习模型——线性回归

文章目录 前言1.基础概念2.代价函数3.单变量线性回归3.1加载数据3.2初始化超参数3.3梯度下降算法3.3.1初次梯度下降3.3.2 多次梯度下降3.3.3结果可视化 前言 随着互联网数据不断累积,硬件不断升级迭代,在这个信息爆炸的时代,机器学习已被应用…