Compared with the Scala language, Python has its unique advantages and a wide range of applications, python call interface, so Spark also launched PySpark, which provides an interface using the Python language on the framework, python interface to the data request is received post scientists It provides a convenient framework.
As we all know, Spark framework mainly by Scala language, it also contains a small amount of Java code. Spark for user programming interface is Scala. However, Python in the field of scientific data has been occupying an important position. Engineers still have large amounts of data using a variety of data processing and scientific computing Python libraries, such as numpy, Panda, scikit-learn and so on. At the same time, Python's entry threshold was significantly lower than Scala.
For this reason, Spark has launched PySpark, it provides a set of Python interfaces on the Spark framework to facilitate data scientist. This paper analyzes the source code to achieve the level of implementation of the principle of PySpark, including the following:
PySpark multi-process architecture;
Python calls Java and Scala interfaces;
Python driver RDD, SQL interfaces;
performing inter-process communication end and serialization ;
Panda UDF;
summary.
1.PySpark multi-process architecture
PySpark use multi-process architecture, which Python and the JVM process are separate. Python and appear on the JVM process drivers and actuators. When submitting PySpark Python script by spark-submit, driver-side Python script will run directly from python and start the JVM body. However, RDD or Python call operation data frame will be called by the Java interface Py4j.
In terms of executor, just the opposite. The driver first actuator JVM startup process,python automatic writing papers and then start a child process of Python on the JVM to execute Python's UDF. Sockets for inter-process communication. The overall architecture is shown below: How 2.Python driver calls the Java interface
described above, after PySpark jobs submitted by spark-submit, the driver first run Python scripts submitted by users. However, most API Spark are provided Scala or Java, it is necessary to be able to invoke Java interfaces with Python. PySpark used here Py4j as open source library. When you create a Python objects SparkContext end, JVM has actually started, and created a Scala end of SparkContext object. Code is implemented in python / pyspark / context.py in:
If you have an Internet problem, you can ask me, thank you! If you want to learn artificial intelligence together, welcome message exchange.
The complete code
python API calls in several ways:
- urllib2
- httplib2
- pycurl
- requests
urllib2
import urllib2, urllib
github_url = 'https://api.github.com/user/repos'
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, github_url, 'user', '***')
auth = urllib2.HTTPBasicAuthHandler(password_manager) # create an authentication handler
opener = urllib2.build_opener(auth) # create an opener with the authentication handler
urllib2.install_opener(opener) # install the opener...
request = urllib2.Request(github_url, urllib.urlencode({'name':'Test repo', 'description': 'Some test repository'})) # Manual encoding required
handler = urllib2.urlopen(request)
print handler.read()
2. httplib2
import urllib, httplib2
github_url = '
h = httplib2.Http(".cache")
h.add_credentials("user", "******", "
data = urllib.urlencode({"name":"test"})
resp, content = h.request(github_url, "POST", data)
print content
3. pycurl
import pycurl, json
github_url = "
user_pwd = "user:*****"
data = json.dumps({"name": "test_repo", "description": "Some test repo"})
c = pycurl.Curl()
c.setopt(pycurl.URL, github_url)
c.setopt(pycurl.USERPWD, user_pwd)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, data)
c.perform()
4. requests
Requests Import, JSON
github_url = "
Data = json.dumps ({ 'name': 'Test', 'Description': 'the repo some Test'})
R & lt requests.post = (github_url, Data, the auth = ( 'User', '*****'))
Print r.json
more ways can call the API to perform actions, but requests the code this way most concise and clear, is recommended.