I haven't seen a question related to this, yet it pops up despite several of my efforts to the contrary, so I was hoping that someone could help me understand what's going on.
I'm new to Python's Gekko package, and I'm trying to run a non-linear solver to the resolve what's going on. I'm getting a weird error message (FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/03/vyh22j3j45j2rqjmnygfw2l80000gn/T/tmpatrk8y36gk_model0/options.json') that seems programmatic/syntactical rather than mathematical. It's one function in a much larger function/data/input, so it's hard for me to provide enough to help you replicate it -- but --
The y (i.e. the result from self.get_days_energy(date = date)) is a vector of energy values, as is the exogenous inputs from self.get_exogenous_inputs_of_day(date). Z variables are latent state variables, and A and B are transition matrices for latent states and exogenous variables, respectively. The goal of this function is to update the three aforementioned variables.
Here'e the function:
def daily_weight_fit(self, date):
"""
Update function to the weights of the dynamic system, this will be called
once a day after the data has arrived.
"""
m = GEKKO(remote = False)
A = m.Array(
m.Var,
(self.state_weights.shape),
value = 1,
lb = -10,
ub = 10,
)
B = m.Array(
m.Var,
(self.input_weights.shape),
value = 1,
lb = -100,
ub = 100,
)
dates = pd.date_range(start=self.starting_date, end=date)
date_list = dates.tolist()
hours = list(range(24))
y = [[self.get_days_energy(date = date)] for date in dates]
flat_y = np.reshape(y, -1)
timesteps = len(flat_y)
u = [self.get_exogenous_inputs_of_day(date) for date in dates for hour in hours]
z = m.Array(m.Var, (timesteps, 4), lb = -1, ub = 1)
C = np.array([0, 0, 1, 0])
m.Obj(
m.sqrt(
m.sum([(flat_y[i] - z[i][3])**2 for i in range(len(flat_y))])
)
)
for i in range(timesteps - 2):
state_contribution = np.dot(A, z[i])
ex_contribution = np.dot(B, u[i])
for j in range(4):
m.Equation(z[i + 1][j] == state_contribution[j] + ex_contribution[j])
m.options.solver = 1
m.solve()
return A, B, z
Here's the error output:
----------------------------------------------------------------
APMonitor, Version 0.9.2
APMonitor Optimization Suite
----------------------------------------------------------------
--------- APM Model Size ------------
Each time step contains
Objects : 1
Constants : 0
Variables : 1849
Intermediates: 0
Connections : 361
Equations : 1793
Residuals : 1793
Number of state variables: 1849
Number of total equations: - 1793
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 56
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter Objective Convergence
0 4.12076E+08 8.12928E+01
1 5.38520E+02 1.00000E+00
2 5.38512E+02 3.75145E-12
3 5.38450E+02 2.00000E+00
4 5.38506E+02 1.80710E-01
NEGATIVE NDF: -1
5 1.10270E+07 3.34638E-01
6 4.89845E+06 1.93472E+00
7 4.73333E+05 3.99952E+00
8 3.76178E+05 2.00000E+00
9 1.69753E+05 1.31302E+00
Iter Objective Convergence
10 1.90770E+07 5.86504E-01
11 3.03623E+16 3.34638E-01
12 9.38385E+11 3.34304E-01
13 1.14353E+12 1.09011E-02
14 8.91928E+12 7.65181E-03
Error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x108c7718d
#1 0x108c7661b
#2 0x7fff661ccf59
#3 0x108a92b6b
apm_mac(33870,0x7fff9e9c1380) malloc: *** error for object 0x7fb371862e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Program received signal SIGABRT: Process abort signal.
Backtrace for this error:
#0 0x108c7718d
#1 0x108c7661b
#2 0x7fff661ccf59
Error: 'results.json' not found. Check above for additional error details
Traceback (most recent call last):
.... (my own irrelevant function trace).....
m.solve()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gekko/gekko.py", line 2145, in solve
self.load_JSON()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gekko/gk_post_solve.py", line 13, in load_JSON
f = open(os.path.join(self._path,'options.json'))
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/03/vyh22j3j45j2rqjmnygfw2l80000gn/T/tmp42metzl9gk_model0/options.json'
Any tips or advice would be much appreciated! John Hedengren, hi and I love your work!!
You have discovered a new bug with the APOPT solver. The iteration summary shows that the problem is diverging so some of the variables may be going to Infinity, beyond machine precision.
- A first option is to try a different solver such as BPOPT with
m.options.SOLVER=2or IPOPT withm.options.SOLVER=3. - A second option is to bound some of your decision variables or watch out for divide by zero in your equations or objective function.
Whenever the solver crashes, please consider sending the contents of your run directory in m.path so that developers can fix the bug.
Background Info:
The options.json or results.json files are written by the underlying C++/Fortran executable either locally with m=GEKKO(remote=False) or through the web-service m=GEKKO(remote=True). In all cases, the executable produces the results as json files either in the local run directory m.path or else on the remote server. When the executable crashes due to a bug, the json files are not produced and it leads to the error that you observed. You can also see the solver stops at iteration 14. With remote=False, the executable runs locally from the Gekko bin directory (apm for Linux, apm_mac for MacOS, or apm.exe for Windows) but produces the files in the run directory m.path.