QQ group 1070535031
follows the idea of the previous article, in this article we will implement the entire process.
Experimental needs
To follow this article to learn and experiment, you need the environment in the previous blog post and the extracted UK data. ( Learning and sharing-NILM load decomposition based on deep learning (2) electrical appliance data extraction )
The code idea of this article is in the previous one, this article focuses on the related code implementation. Take a look at the ideas ( learning sharing-NILM load decomposition based on deep learning (3) deep learning processing, basic ideas )
Experimental environment:
pycharm, python 3.6+, keras library, numpy library, matplotlib library
and UKData.
Define global parameters
First quote all required libraries, as follows:
import numpy as np
import os
import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop
from keras.models import load_model
Then define some global parameters:
# 全局参数定义
dimension = 5 #电器的个数
#电器id定义
fridge_id = 0
tv_id = 1
kettle_id = 2
microwave_id = 3
washerdryer_id = 4
#电器开启状态阈值
fridge_threshold = 20
tv_threshold = 20
kettle_threshold = 20
microwave_threshold = 10
washerdryer_threshold = 0
#消噪音阈值
air_threshold = 5
Introduce the above parameters one by one
dimension : is the number of categories of electrical appliances we train. Five are set up in the current preliminary experiment.
Electrical appliance id : It is the self-increasing id of the five electrical appliances we selected. (Here we choose: refrigerator, TV, kettle, microwave oven and washing dryer)
Electrical appliance turn-on state threshold : The function of this is to shield some interference, that is, when the value is lower than this value, the appliance is considered not to be turned on.
Noise elimination threshold : Noise elimination of total work. The value below the change is considered noise and ignored.
Read appliance data
fridge_data = np.load('UKData/Fridge freezer0.npy')
print(fridge_data.shape)
fridge_data = fridge_data[:1600000]
print(fridge_data.shape)
fridge_data = np.where(fridge_data > air_threshold,fridge_data,0)
television_data = np.load('UKData/Television0.npy')
print(television_data.shape)
television_data = television_data[:1600000]
print(television_data.shape)
television_data = np.where(television_data > air_threshold, television_data,0)
kettle_data = np.load('UKData/Kettle0.npy')
print(kettle_data.shape)
kettle_data = kettle_data[0:1600000]
print(kettle_data.mean)
kettle_data = np.where(kettle_data > air_threshold, kettle_data,0)
microwave_data = np.load('UKData/Microwave0.npy')
print(microwave_data.shape)
microwave_data = microwave_data[:1600000]
print(microwave_data.shape)
microwave_data = np.where(microwave_data > air_threshold, microwave_data,0)
washerdryer_data = np.load('UKData/Washer dryer0.npy')
print(washerdryer_data.shape)
washerdryer_data = washerdryer_data[:1600000]
print(washerdryer_data.shape)
washerdryer_data = np.where(washerdryer_data > air_threshold, washerdryer_data,0)
The above code directly selects the electrical equipment we used for training from the electrical appliances we read before, reads it out, and takes out the data (useful work) of the same length (1.600000). At the same time, the value below the threshold air_threshold is taken to be 0, the purpose is to remove the drying.
Make a label
def create_label(data, application_id, power_threshold, dimension):
labels = np.zeros((len(data), dimension))
for i, value in enumerate(data):
if value[0] > power_threshold:
labels[i,application_id] = 1
return labels
In the above method, the input is data, appliance id, opening state threshold, and number of appliances.
The output is a one-dimensional array of the length of the number of appliances. At a certain moment, when the corresponding appliance is turned on, the position of the corresponding id is 1, and when it is turned off, it is 0.
Data is the load at each moment, and label is the on-state of the electrical appliance at each moment.
Example: The label defaults to [0,0,0,0,0] at each moment. If the appliance with id 1 is turned on at a certain moment, the label at that moment is [0,1,0,0,0]
The following code is to generate five appliances, each appliance label
fridge_labels = create_label(fridge_data,fridge_id,fridge_threshold,dimension)
tv_labels = create_label(television_data,tv_id,tv_threshold,dimension)
kettle_labels = create_label(kettle_data,kettle_id,kettle_threshold,dimension)
microwave_labels = create_label(microwave_data,microwave_id,microwave_threshold,dimension)
washerdryer_labels = create_label(washerdryer_data,washerdryer_id,washerdryer_threshold,dimension)
Combining data and standardization
We integrate the data of five electrical appliances, and the useful work is directly added together to become the sum of useful work at every moment.
The label labels are also combined together, because the labels generated above correspond to one column for each appliance, so they can be added directly.
sum_label = fridge_labels + tv_labels + kettle_labels + microwave_labels + washerdryer_labels
sum_data_num = fridge_data + television_data + kettle_data + microwave_data + washerdryer_data
Then you can draw a picture to see, our label
#画图查看数据
epochs = range(1,len(sum_label[:,0])+1)
plt.plot(epochs,sum_label[:,0],label='fridge_labels')
plt.plot(epochs,sum_label[:,1],label='tv_labels')
plt.plot(epochs,sum_label[:,2],label='kettle_labels')
plt.plot(epochs,sum_label[:,3],label='microwave_labels')
plt.plot(epochs,sum_label[:,4],label='washerdryer_labels')
plt.title('Training and validation accuracy')
plt.legend()
plt.show()
It looks like this, each color is turned on
Considering the training efficiency and effect, we normalize the data as follows:
mean = sum_data_num[:1600000].mean(axis=0)
sum_data = sum_data_num - mean
std = sum_data[:1600000].std(axis=0)
sum_data /= std
Iterator
We only used 16w of data for one room, but the amount of data used is still relatively large, for the sake of more data later. We use iterators for data input to reduce memory usage. Each time the iterator takes data and inputs it into the training iteration.
def generator(data, label, lookback, delay, min_index, max_index, shuffle=False, batch_size=128, step=1):
if max_index is None:
max_index = len(data) - delay - 1
i = min_index + lookback
while 1:
if shuffle:
rows = np.random.randint(min_index + lookback, max_index, size=batch_size)
else:
if i + batch_size >= max_index:
i = min_index + lookback
rows = np.arange(i, min(i + batch_size, max_index))
i += len(rows)
samples = np.zeros((len(rows),
lookback // step,
data.shape[-1]))
targets = np.zeros((len(rows),dimension))
for j, row in enumerate(rows):
indices = range(rows[j] - lookback, rows[j], step)
samples[j] = data[indices]
targets[j] = label[rows[j] + delay]
yield samples, targets
This piece of logic may be a bit complicated, take your time.
Parameters:
data : data.
label : label.
lookback : The window size mentioned in the previous section, here is the window to look back.
delay : predict the result of the next few frames (here we default to 0, because it is the result of predicting the decomposition of the current frame, so you don’t need to look into the future)
min_index : start from which frame of the data to look at
max_index : start from which frame of the data to end
shuffle：Whether it is random
batch_size：The number of each batch in the iteration
step：Every time data is fetched, skip back a few frames
The meaning of this method is to take the data in the range of min_index to max_index in the total data, each batch is batch_size, and each time the data of the lookback window size is retrieved, one step is taken back and then taken down. once. Shuffle is whether to randomly start the position.
The delay can be ignored.
If you really don’t understand, you can chat with me privately in the group, or wait for me to start the live broadcast.
Split data iterator
First, define some hyper-parameters of the iterator:
window size, step backward, prediction of the current frame (delay=0), batch size
lookback = 20
step = 1
delay = 0
batch_size = 128
The specific size setting of each parameter will be explained in detail when I analyze the function and influence of each super parameter in my later article.
Then is to define each iterator:
train_gen = generator(sum_data,
sum_label,
lookback=lookback,
delay=delay,
min_index=0,
max_index=800000,
shuffle=True,
step=step,
batch_size=batch_size)
val_gen = generator(sum_data,
sum_label,
lookback=lookback,
delay=delay,
min_index=800001,
max_index=1200000,
step=step,
batch_size=batch_size)
test_gen = generator(sum_data,
sum_label,
lookback=lookback,
delay=delay,
min_index=1200001,
max_index=None,
step=step,
batch_size=batch_size)
A total of 160w frames of data, I used the first 80w as training, 80-120w as verification, and 120-160w as test.
The training set data uses random processing.
Then calculate the number of batches for each iteration
train_steps = 800000 // batch_size
val_steps = (1200000 - 800001 -lookback) // batch_size
test_steps = (len(sum_data) - 1200001 -lookback) // batch_size
Build a simple network to test
#双向循环网络
model = Sequential()
model.add(layers.Bidirectional(
layers.GRU(32,dropout=0.1), input_shape=(None,sum_data.shape[-1])))
model.add(layers.Dense(dimension,activation='sigmoid'))
model.compile(optimizer=RMSprop(), loss='binary_crossentropy',metrics=['acc'])
history = model.fit_generator(train_gen,
steps_per_epoch=train_steps,
epochs=2,
validation_data=val_gen,
validation_steps=val_steps)
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1,len(loss)+1)
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and Validation loss')
# plt.legend()
plt.show()
plt.clf() #清空图表
acc_values = history.history['acc']
val_acc_values = history.history['val_acc']
plt.plot(epochs,acc_values,'bo',label='Training acc') #bo是蓝色圆点
plt.plot(epochs,val_acc_values,'b',label='Validation acc') #b是蓝色实线
plt.title('Training and validation accuracy')
plt.legend()
plt.show()
What I use here is the keras method to do the test, two-way GRU, link one layer of fully connected output.
Then set the parameters, optimizer, and output, and iterate 2 times.
After training, I did not save the model, because the above network is just a test, not our established complete network structure.
You can run to see if there are any problems with the above process.
After running, two graphs will be drawn, one is loss and the other is acc.
The operation process is as follows:
Quietly wait for 2 iterations.
It can be seen that in two iterations, loss is decreasing and converging, and accuracy acc is increasing. Looking at the verification acc will reflect the robustness of the model in training.
Build a network for training
Below I provide a relatively high-quality network structure after a series of verifications, for reference.
However, I hope that students can study and practice various network structure combinations, including the adjustment of various super parameters.
For example, fully connected networks, convolutional networks, and recurrent networks. And various combinations of the above networks.
Only by constantly trying and accumulating experience can we design and train better models.
循环网络+一维卷积
model = Sequential()
model.add(layers.Conv1D(32, 5, activation='relu', input_shape=(None,sum_data.shape[-1])))
model.add(layers.MaxPooling1D(3))
model.add(layers.Conv1D(32, 5, activation='relu'))
model.add(layers.LSTM(32, dropout=0.1))
# model.add(layers.GlobalMaxPooling1D())
model.add(layers.Dense(dimension,activation='sigmoid'))
model.compile(optimizer=RMSprop(), loss='binary_crossentropy',metrics=['acc'])
history = model.fit_generator(train_gen,
steps_per_epoch=train_steps,
epochs=20,
validation_data=val_gen,
validation_steps=val_steps)
# model.save('model.h5')
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1,len(loss)+1)
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and Validation loss')
# plt.legend()
plt.show()
plt.clf() #清空图表
acc_values = history.history['acc']
val_acc_values = history.history['val_acc']
plt.plot(epochs,acc_values,'bo',label='Training acc') #bo是蓝色圆点
plt.plot(epochs,val_acc_values,'b',label='Validation acc') #b是蓝色实线
plt.title('Training and validation accuracy')
plt.legend()
plt.show()
My above model: One-dimensional convolution is used to refine data, LSTM (or two-way LSTM) learns the law of change, and the fully connected layer outputs the result.
Hyperparameters (such as the size of each layer, convolution kernel, window size, etc.) and the number of layers (two layers of lstm can be used, the last two layers can be fully connected, etc.) various combinations, you need to try, welcome everyone to the group To discuss the experimental results together.
Test set
Finally, run the test set to make predictions on the data not involved in training.
测试集测试
test_loss, test_acc = model.evaluate_generator(test_gen,steps=test_steps)
print('test acc : ', test_acc)
If you don’t understand, or have questions about ideas, you can leave a message to me.
Or QQ group 1070535031
For many parameters and usage in the code, there is no detailed description. I will write a separate article later to elaborate.
