Spark supports automatic detection of partitions
For example: the directory is as follows:
When df is read, different paths are passed in, and the schemas are different. As you can see from the figure below, spark can detect the partitions under the current path.
You can control this parameter to decide whether to enable automatic type speculation (the type is controlled, if it is set to false, it is the partition field string type)
spark.sql.sources.partitionColumnTypeInference.enabled
Test code:
The location of the source code for partition type speculation:
