Life is too short I use Python
1. Write at the beginning
hello brothers
Previously, Python scripts were often written for data processing, data transfer, and model training. Running the script may take some time as the volume and complexity of the data increases. You can do some other work at the same time while waiting for data processing to complete.
To achieve this, a set of Python scripts were written to solve this problem. Use these scripts to send process updates, visualizations, and completion notifications to your phone. When you have these moments of freedom once in a while, you can enjoy rather than worry about your model's progress.
2. What is needed
The first question is, what do you need to know? It depends on what you're doing. For the author, there are three main processing tasks that may take up time:
- model training
- Data processing and/or transfer
- financial model
We need to analyze each case in detail.
3. Model training
Update every n epochs, must include key metrics. For example, loss and accuracy on training and validation sets. Then complete the notification, including:
-
Visualization of key metrics during training (again, loss and accuracy on training and validation sets)
-
Other less important but still useful information such as local model directory, training time, model architecture, etc.
-
Predicted output, for text generation, output the generated text (or a sample of it); for image generation
The output is a (hopefully) cool visualization
Take the example of training a neural network to reproduce a given art style. We need to focus on the images generated from the model, the loss and accuracy plots, the current training time, and the name of a model.
import notify
START= datetime.now() # this line would be placed before modeltraining begins
MODELNAME="SynthwaveGAN" # giving us ourmodel name
NOTIFY=100 # so we send an update notification every100 epochs
# for each epoch e,we would include the following code
if e % notify_epoch ==0and e !=0:
# here we createthe email body message
txt = (f"{
MODELNAME} update as of "
f"{
datetime.now().strftime( %H:%M:%S )}.")
# we build theMIME message object with notify.message
msg = notify.message(
subject= Synthwave GAN ,
text=txt,
img=[
f ../visuals/{
MODELNAME}/epoch_{
e}_loss.png ,
f ../visuals/{
MODELNAME}/epoch_{
e}_iter_{
i}.png
]
) # note that weattach two images here, the loss plot and
# ...a generated image output from our model
notify.send(msg) # we then send the message
Every 100 epochs, an email is sent with all of the above. Here is one of the emails:
4. Data processing and transmission
It's not very interesting, but in terms of time consumption, it ranks first.
Take the example of uploading bulk data to SQLServer using Python (for those without BULK INSERT). At the end of the upload script, there will be a simple message notifying that the upload is complete.
import os
import notify
from data importSql # seehttps://jamescalam.github.io/pysqlplus/lib/data/sql.html
dt =Sql( database123 , server001 ) # setup theconnection to SQL Server
for i, file inenumerate(os.listdir( ../data/new )):
dt.push_raw(f ../data/new/{
file} ) # push a file to SQL Server
# once the upload is complete, send a notification
# first we create the message
msg = notify.message(
subject= SQL Data Upload ,
text=f Data upload complete, {
i} filesuploaded. ,
)
# send the message
notify.send(msg)
If an error is thrown occasionally, you can also add a try-except statement to catch the error and add it to a list for inclusion in update and/or completion emails.
5. Financial Model
Everything running in financial modeling is actually very fast, so I can only provide an example here.
Take the Cash Flow Modeling Tool as an example. In reality, this process only takes 10-20 seconds, but now imagine you are the hottest quantitative analyst on Wall Street, processing several million loans. In this email, you may want to include an advanced profiling portfolio. It is possible to randomly select some loans and visualize the key figures over a given time period - given a small sample to cross-check the performance of the model.
end = datetime.datetime.now() # get the ending datetime
# get the total runtime in hours:minutes:seconds
hours,rem =divmod((end - start).seconds, 3600)
mins,secs =divmod(rem, 60)
runtime= {
:02d}:{
:02d}:{
:02d} .format(hours, mins,secs)
# now built our message
notify.msg(
subject="Cashflow Model Completion",
text=(f {
len(model.output)} loansprocessed.
f Total runtime:{
runtime} ),
img=[
../vis/loan01_amortisation.png ,
../vis/loan07_amortisation.png ,
../vis/loan01_profit_and_loss.png ,
../vis/loan07_profit_and_loss.png
]
)
notify.send(msg) # and send it
6. Code
All the functions above are excerpted from a script called notify.py. Outlook will be used in the sample code. Two Python libraries are needed here, email and smtplib:
email : Used to manage email messages. With this library it is possible to set up the email message itself, including subject, body and attachments.
smtplib : handles SMTP connections. Simple Mail Transfer Protocol (SMTP) is the protocol used by most email systems to allow mail to be sent over the Internet.
Seven, MIME
The message itself is constructed using the MIMEMultipart object from the email module. There are also three MIME subclasses that need to be used and attached to the MIMEMultipart object:
-
mimetext: This will contain the email "payload", the text in the body of the email.
-
mimeimage : This is used to include images in emails.
-
mimeapplication: The application object for MIME messages. That is, file attachments.
In addition to these subclasses, there are other parameters, such as the Subject value in MimeMultipart. All of this adds up to the structure below.
What happens when you put them all together:
import os
from email.mime.text importMIMEText
from email.mime.image importMIMEImage
from email.mime.application importMIMEApplication
from email.mime.multipart importMIMEMultipart
defmessage(subject="PythonNotification", text="", img=None, attachment=None):
# build messagecontents
msg =MIMEMultipart()
msg[ Subject ] = subject # add in thesubject
msg.attach(MIMEText(text)) # add text contents
# check if wehave anything given in the img parameter
if img isnotNone:
# if we do, wewant to iterate through the images, so let s check that
# what we haveis actually a list
iftype(img) isnot list:
img = [img] # if it isn t alist, make it one
# now iteratethrough our list
for one_img in img:
img_data =open(one_img, rb ).read() # read the imagebinary data
# attach theimage data to MIMEMultipart using MIMEImage, we add
# the givenfilename use os.basename
msg.attach(MIMEImage(img_data, name=os.path.basename(one_img)))
# we do the samefor attachments as we did for images
if attachment isnotNone:
iftype(attachment) isnot list:
attachment = [attachment] # if it isn t a list, make it one
for one_attachment in attachment:
withopen(one_attachment, rb ) as f:
# read in theattachment using MIMEApplication
file =MIMEApplication(
f.read(),
name=os.path.basename(one_attachment)
)
# here we editthe attached file metadata
file[ Content-Disposition ] =f attachment; filename="{os.path.basename(one_attachment)}"
msg.attach(file) # finally, addthe attachment to our message object
return msg
This script is fairly simple. At the top, there are imports (this is the MIME part that was covered before) and Python's os library.
Next define a function called message. This allows to call functions with different arguments and easily build an email message object. For example, an email with multiple images and attachments can be written like this:
email_msg= message(
text="Model processing complete,please see attached data.",
img=[ accuracy.png , loss.png ],
attachments=[ data_in.csv , data_out.csv ]
)
First, the MIMEMultipart object is initialized and assigned to msg; then, the email subject is set using the "subject" key. The attach method adds different MIME subclasses to the MIMEMultipart object. You can use the MIMEText subclass to add the email body.
For image img and attachment, you can pass nothing but a file path, or pass a set of file paths. We handle it by first checking if the argument is None, and if it is None, pass it; otherwise, check the given data type, not a list, and create one, which makes it possible to iterate over the items with the for loop below.
Next, use the MIMEImage and MIMEApplication subclasses to attach images and files, respectively. Use os.basename to get the filename from the given filepath, including the filename in the attachment name.
8. SMTP
Now that the email message object is constructed, the next step is to send it. This is where the smtplib module comes into play. The code is also very simple with one exception.
Different SMTP addresses are required due to dealing directly with different email providers and their respective servers. Type "outlook smtp" into google. Without clicking the page, you can get the server address smtp-mail.outlook.com and port number 587.
Both methods are required when initializing the SMTP object using smtplib.SMTP. SMTP - send function near the beginning:
import smtplib
import socket
defsend(server= smtp-mail.outlook.com , port= 587 , msg):
# contain followingin try-except in case of momentary network errors
try:
# initialiseconnection to email server, the default is Outlook
smtp = smtplib.SMTP(server, port)
# this is the Extended Hello command, essentially greeting our SMTP or ESMTP server
smtp.ehlo()
# this is the Start Transport Layer Security command, tells the server we will
# becommunicating with TLS encryption
smtp.starttls()
# read email andpassword from file
withopen( ../data/email.txt , r ) as fp:
email = fp.read()
withopen( ../data/password.txt , r ) as fp:
pwd = fp.read()
# login tooutlook server
smtp.login(email, pwd)
# sendnotification to self
smtp.sendmail(email, email, msg.as_string())
# disconnectfrom the server
smtp.quit()
except socket.gaierror:
print("Network connection error, email notsent.")
smtp.ehlo() and smtp.starttls() are both SMTP commands. ehlo (extended Hello) essentially says hello to the server. starttls informs the server that an encrypted Transport Level Security (TLS) connection will be used for communication.
After this, just read the email and password from the file, store in email and pwd respectively. Then, log in to the SMTP server using smtp.login. Login and send email using smtp.sendmail.
I always send myself notifications, but in the case of automated reports, I may want to send the email elsewhere. For this, I would change the destination_address: smtp.sendmail(email, destination_address, msg.as_string).
Finally, terminate the session and close the connection to smtp.quit.
Put all of this in a try-except statement. In the event of a momentary loss of network connectivity, it will not be possible to connect to the server. results in socket.gaierror. Use the try-except statement to prevent your program from crashing if the network connection fails.
The author uses it for ML model training update and data transfer completion. It doesn't matter if the mail isn't sent. This simple, passive handling of connection loss is appropriate.
Nine, put together
Now that the two parts of the code have been written, we can send the email:
# builda message object
msg = message(text="See attached!", img= important.png ,
attachment= data.csv )send(msg) #send the email (defaults to Outlook)
This is all email notification and/or automation of the whole process using Python. Thanks to the email and smptlib libraries, the setup process is very easy.
One more thing to note, the public email provider server address and TLS port.
For any time-consuming processing or training task, progress updates and completion notifications are often the real liberation. Python, make work better!
Brothers learn python, sometimes they don't know how to learn and where to start. After mastering some basic grammar or doing two cases, I don't know what to do next, and I don't know how to learn more advanced knowledge.
So for these big brothers, I have prepared a lot of materials, PDF e-books, and source code! It's all on the card below.
Alright guys, that's all for today's sharing. I recommend a set of Python tutorials for you, covering most of the common cases and actual combat. I hope it will be helpful to you!