Creating pyspark of SparkSession

1.from pyspark.sql import SparkSession

2.spark = SparkSession.builder.master("spark://master:7077") \
.appName('compute_customer_age') \
.config('spark.executor.memory','2g') \
.enableHiveSupport() \
.getOrCreate()

3. Create complete

4. can be used to build DataFrame | used to access the hive

4.1DataFrame

documentDF = spark.createDataFrame([
("Hi I heard about Spark".split(" "), ),
("I wish Java could use case classes".split(" "), ),
("Logistic regression models are neat".split(" "), )
], ["text"])

4.2 Access hive

sql = """
"""
df = spark.sql(sql)
df.show()