Using Highs in cluster network optimizaion

[trevorb1]

Checklist

• I am using the current master branch
• I am running on an up-to-date pypsa-usa environment. Update via conda env update -f envs/environment.yaml

The Issue

When using highs in the cluster/simplify network scripts, an error is thrown by pyomo. If only highs is passed to the SolverFactory, the following error is thrown:

opt = po.SolverFactory("highs")
opt.solve(m)

Solver (asl) did not exit normally
  File "/home/trevor/master/pypsa-usa/workflow/scripts/cluster_network.py", line 247, in distribute_clusters
    results = opt.solve(m)
              ^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/scripts/cluster_network.py", line 317, in busmap_for_n_clusters
    n_clusters = distribute_clusters(
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/scripts/cluster_network.py", line 379, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/scripts/simplify_network.py", line 278, in <module>
    clustering = clustering_for_n_clusters(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
pyomo.common.errors.ApplicationError: Solver (asl) did not exit normally

If you pass the solver appsi_highs, as described by pyomo with the command pyomo help --solvers, the following error is raised:

opt = po.SolverFactory("appsi_highs")
if not opt.has_capability("quadratic_objective"):
...

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'LegacySolver' object has no attribute 'has_capability'

Steps To Reproduce

1. Change solver configuration to the following:

  solver:
    name: highs
    options: highs-default

2. Run the workflow

Expected Behavior

The clustering optimization should run via ipopt if highs is selected for the quadratic optimization

pypsa-usa/workflow/scripts/cluster_network.py

Lines 186 to 252 in a056649

	def distribute_clusters(n, n_clusters, focus_weights=None, solver_name="cbc"):
	"""
	Determine the number of clusters per country.
	"""
	L = (
	n.loads_t.p_set.mean()
	.groupby(n.loads.bus)
	.sum()
	.groupby([n.buses.country, n.buses.sub_network])
	.sum()
	.pipe(normed)
	)
	N = n.buses.groupby(["country", "sub_network"]).size()
	assert (
	n_clusters >= len(N) and n_clusters <= N.sum()
	), f"Number of clusters must be {len(N)} <= n_clusters <= {N.sum()} for this selection of countries."
	if focus_weights is not None:
	total_focus = sum(list(focus_weights.values()))
	assert (
	total_focus <= 1.0
	), "The sum of focus weights must be less than or equal to 1."
	for country, weight in focus_weights.items():
	L[country] = weight / len(L[country])
	remainder = [
	c not in focus_weights.keys() for c in L.index.get_level_values("country")
	]
	L[remainder] = L.loc[remainder].pipe(normed) * (1 - total_focus)
	logger.warning("Using custom focus weights for determining number of clusters.")
	assert np.isclose(
	L.sum(),
	1.0,
	rtol=1e-3,
	), f"Country weights L must sum up to 1.0 when distributing clusters. Is {L.sum()}."
	m = po.ConcreteModel()
	def n_bounds(model, *n_id):
	return (1, N[n_id])
	m.n = po.Var(list(L.index), bounds=n_bounds, domain=po.Integers)
	m.tot = po.Constraint(expr=(po.summation(m.n) == n_clusters))
	m.objective = po.Objective(
	expr=sum((m.n[i] - L.loc[i] * n_clusters) ** 2 for i in L.index),
	sense=po.minimize,
	)
	opt = po.SolverFactory(solver_name)
	if not opt.has_capability("quadratic_objective"):
	logger.warning(
	f"The configured solver `{solver_name}` does not support quadratic objectives. Falling back to `ipopt`.",
	)
	opt = po.SolverFactory("ipopt")
	results = opt.solve(m)
	assert (
	results["Solver"][0]["Status"] == "ok"
	), f"Solver returned non-optimally: {results}"
	return pd.Series(m.n.get_values(), index=L.index).round().astype(int)

Error Message

No response

Anything else?

The pypsa-eur team updated their clustering to use linopy; would be nice to follow their implementation and allow us to drop the pyomo dependency

https://github.com/PyPSA/pypsa-eur/blob/51f8c2935ae3f80b15ad0ab71708ad006ad49da9/scripts/cluster_network.py#L172-L230

[trevorb1]

Thanks for the patch, @ktehranchi! I agree with pushing this to master. You okay if I reopen this though, as the patch doesnt address the underlying problem of highs not being used for the cluster network?

[ktehranchi]

Agreed- i also had an issue with pyomo on my cluster runs this week... it wasn't reading my gurobi license correctly, so i might take this on and replace pyomo with linopy soon