Support matpower dict #928

yasirroni · 2024-10-08T15:00:50Z

This code support PowerModels to accept matlab_data in the form of Dict.

But, to be honest, later in the next PR, I want it to accept Dict{Any, Any} and InfractructureModels to accept AbstractMatrix.

Here is current usage:

    function convert_matrix_to_vector_of_reals(value::Any)
        if isa(value, AbstractMatrix)
            return [Vector(row) for row in eachrow(value)]
        else
            return value 
        end
    end

    function convert_dict_value_matrix_to_vector_of_reals(data::Dict)
        for (key, value) in data
            data[key] = convert_matrix_to_vector_of_reals(value)
        end
        return data
    end

    function convert_python_to_matlab_data(dict::Dict{Any, Any})
        new_dict = Dict{String, Any}()
        for (key, value) in dict
            new_dict[string(key)] = convert_matrix_to_vector_of_reals(value)
        end
        return new_dict
    end

    matlab_data = convert_python_to_matlab_data(matlab_data)
    data = PowerModels.parse_matpower(matlab_data, func_name=func_name, colnames=colnames, validate=true)
    result = solve_opf(data, ACPPowerModel, nlp_solver)

Where matlab_data id Dict{Any, Any}that containsMatrix{Float64}`.

ccoffrin · 2024-10-09T03:43:42Z

I don't understand the objective of this PR. Not inclined to accept at this time.

yasirroni · 2024-10-09T04:23:05Z

Hi @ccoffrin, the goal is to be able to pass a plain Dict{String, Any} from Python to PowerModels via Python code:

Main.matlab_data = mpc
Main.eval("""
    matlab_data = convert_python_to_matlab_data(matlab_data)
    data = PowerModels.parse_matpower(matlab_data, func_name=func_name, colnames=colnames, validate=true)
    result = solve_opf(data, ACPPowerModel, nlp_solver)
""")

yasirroni · 2024-10-10T02:49:08Z

Hi @ccoffrin, you also can look at the @testset to see a case where this PR is useful. User will be able to pass a Dict with a shape almost identical with plain text that follows .m file.

ccoffrin · 2024-10-19T23:32:56Z

At this point in time I do not want to add support for another data format convention. We currently support: Matpower txt files, PSSE v33 txt files, and JSON data in the PowerModels data structure format.

It appears that a small amount of code can be used to support your preferred format, keep that code in your related tooling.

yasirroni · 2024-10-20T03:23:21Z

Thanks @ccoffrin. But, let me clarify more as it seems that you still not understand my case. Can your parser read this matpower case69.m file?

function mpc = case69
%CASE69  Power flow data for 69 bus distribution system
%   Please see CASEFORMAT for details on the case file format.
%
%   Data from ...
%       M. E. Baran and F. F. Wu, "Optimal capacitor placement on radial
%       distribution systems," in IEEE Transactions on Power Delivery,
%       vol. 4, no. 1, pp. 725-734, Jan. 1989, doi: 10.1109/61.19265.
%       https://doi.org/10.1109/61.19265
%
%   Derived "from a portion of the PG&E distribution system".
%
%   Also in ...
%       D. Das, Optimal placement of capacitors in radial distribution
%       system using a Fuzzy-GA method, International Journal of Electrical
%       Power & Energy Systems, Volume 30, Issues 6–7, July–September 2008,
%       Pages 361-367
%       https://doi.org/10.1016/j.ijepes.2007.08.004
%
%   Modifications:
%     v2 - 2020-09-30 (RDZ)
%         - Cite original source (Baran & Wu)
%         - Specify branch parameters in Ohms, loads in kW.
%         - Added code for explicit conversion of loads from kW to MW and
%           branch parameters from Ohms to p.u.
%         - Set BASE_KV to 12.66 kV (instead of 12.7)
%         - Slack bus Vmin = Vmax = 1.0
%         - Gen Qmin, Qmax, Pmax magnitudes set to 10 (instead of 999)
%         - Branch flow limits disabled, i.e. set to 0 (instead of 999)
%         - Add gen cost.

%% MATPOWER Case Format : Version 2
mpc.version = '2';

%%-----  Power Flow Data  -----%%
%% system MVA base
mpc.baseMVA = 10;

%% bus data
%	bus_i	type	Pd	Qd	Gs	Bs	area	Vm	Va	baseKV	zone	Vmax	Vmin
mpc.bus = [ %% (Pd and Qd are specified in kW & kVAr here, converted to MW & MVAr below)
	1	3	0	0	0	0	1	1	0	12.66	1	1	1;
	2	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	3	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	4	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	5	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	6	1	2.6	2.2	0	0	1	1	0	12.66	1	1.1	0.9;
	7	1	40.4	30	0	0	1	1	0	12.66	1	1.1	0.9;
	8	1	75	54	0	0	1	1	0	12.66	1	1.1	0.9;
	9	1	30	22	0	0	1	1	0	12.66	1	1.1	0.9;
	10	1	28	19	0	0	1	1	0	12.66	1	1.1	0.9;
	11	1	145	104	0	0	1	1	0	12.66	1	1.1	0.9;
	12	1	145	104	0	0	1	1	0	12.66	1	1.1	0.9;
	13	1	8	5.5	0	0	1	1	0	12.66	1	1.1	0.9;
	14	1	8	5.5	0	0	1	1	0	12.66	1	1.1	0.9;
	15	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	16	1	45.5	30	0	0	1	1	0	12.66	1	1.1	0.9;
	17	1	60	35	0	0	1	1	0	12.66	1	1.1	0.9;
	18	1	60	35	0	0	1	1	0	12.66	1	1.1	0.9;
	19	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	20	1	1	0.6	0	0	1	1	0	12.66	1	1.1	0.9;
	21	1	114	81	0	0	1	1	0	12.66	1	1.1	0.9;
	22	1	5.3	3.5	0	0	1	1	0	12.66	1	1.1	0.9;
	23	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	24	1	28	20	0	0	1	1	0	12.66	1	1.1	0.9;
	25	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	26	1	14	10	0	0	1	1	0	12.66	1	1.1	0.9;
	27	1	14	10	0	0	1	1	0	12.66	1	1.1	0.9;
	28	1	26	18.6	0	0	1	1	0	12.66	1	1.1	0.9;
	29	1	26	18.6	0	0	1	1	0	12.66	1	1.1	0.9;
	30	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	31	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	32	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	33	1	14	10	0	0	1	1	0	12.66	1	1.1	0.9;
	34	1	19.5	14	0	0	1	1	0	12.66	1	1.1	0.9;
	35	1	6	4	0	0	1	1	0	12.66	1	1.1	0.9;
	36	1	26	18.6	0	0	1	1	0	12.66	1	1.1	0.9;
	37	1	26	18.6	0	0	1	1	0	12.66	1	1.1	0.9;
	38	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	39	1	24	17	0	0	1	1	0	12.66	1	1.1	0.9;
	40	1	24	17	0	0	1	1	0	12.66	1	1.1	0.9;
	41	1	1.2	1	0	0	1	1	0	12.66	1	1.1	0.9;
	42	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	43	1	6	4.3	0	0	1	1	0	12.66	1	1.1	0.9;
	44	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	45	1	39.2	26.3	0	0	1	1	0	12.66	1	1.1	0.9;
	46	1	39.2	26.3	0	0	1	1	0	12.66	1	1.1	0.9;
	47	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	48	1	79	56.4	0	0	1	1	0	12.66	1	1.1	0.9;
	49	1	384.7	274.5	0	0	1	1	0	12.66	1	1.1	0.9;
	50	1	384.7	274.5	0	0	1	1	0	12.66	1	1.1	0.9;
	51	1	40.5	28.3	0	0	1	1	0	12.66	1	1.1	0.9;
	52	1	3.6	2.7	0	0	1	1	0	12.66	1	1.1	0.9;
	53	1	4.3	3.5	0	0	1	1	0	12.66	1	1.1	0.9;
	54	1	26.4	19	0	0	1	1	0	12.66	1	1.1	0.9;
	55	1	24	17.2	0	0	1	1	0	12.66	1	1.1	0.9;
	56	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	57	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	58	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	59	1	100	72	0	0	1	1	0	12.66	1	1.1	0.9;
	60	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	61	1	1244	888	0	0	1	1	0	12.66	1	1.1	0.9;
	62	1	32	23	0	0	1	1	0	12.66	1	1.1	0.9;
	63	1	0	0	0	0	1	1	0	12.66	1	1.1	0.9;
	64	1	227	162	0	0	1	1	0	12.66	1	1.1	0.9;
	65	1	59	42	0	0	1	1	0	12.66	1	1.1	0.9;
	66	1	18	13	0	0	1	1	0	12.66	1	1.1	0.9;
	67	1	18	13	0	0	1	1	0	12.66	1	1.1	0.9;
	68	1	28	20	0	0	1	1	0	12.66	1	1.1	0.9;
	69	1	28	20	0	0	1	1	0	12.66	1	1.1	0.9;
];

%% generator data
%	bus	Pg	Qg	Qmax	Qmin	Vg	mBase	status	Pmax	Pmin	Pc1	Pc2	Qc1min	Qc1max	Qc2min	Qc2max	ramp_agc	ramp_10	ramp_30	ramp_q	apf
mpc.gen = [
	1	0	0	10	-10	1	100	1	10	0	0	0	0	0	0	0	0	0	0	0	0;
];

%% branch data
%	fbus	tbus	r	x	b	rateA	rateB	rateC	ratio	angle	status	angmin	angmax
mpc.branch = [  %% (r and x specified in ohms here, converted to p.u. below)
	1	2	0.0005	0.0012	0	0	0	0	0	0	1	-360	360;
	2	3	0.0005	0.0012	0	0	0	0	0	0	1	-360	360;
	3	4	0.0015	0.0036	0	0	0	0	0	0	1	-360	360;
	4	5	0.0251	0.0294	0	0	0	0	0	0	1	-360	360;
	5	6	0.366	0.1864	0	0	0	0	0	0	1	-360	360;
	6	7	0.381	0.1941	0	0	0	0	0	0	1	-360	360;
	7	8	0.0922	0.047	0	0	0	0	0	0	1	-360	360;
	8	9	0.0493	0.0251	0	0	0	0	0	0	1	-360	360;
	9	10	0.819	0.2707	0	0	0	0	0	0	1	-360	360;
	10	11	0.1872	0.0619	0	0	0	0	0	0	1	-360	360;
	11	12	0.7114	0.2351	0	0	0	0	0	0	1	-360	360;
	12	13	1.03	0.34	0	0	0	0	0	0	1	-360	360;
	13	14	1.044	0.34	0	0	0	0	0	0	1	-360	360;
	14	15	1.058	0.3496	0	0	0	0	0	0	1	-360	360;
	15	16	0.1966	0.065	0	0	0	0	0	0	1	-360	360;
	16	17	0.3744	0.1238	0	0	0	0	0	0	1	-360	360;
	17	18	0.0047	0.0016	0	0	0	0	0	0	1	-360	360;
	18	19	0.3276	0.1083	0	0	0	0	0	0	1	-360	360;
	19	20	0.2106	0.069	0	0	0	0	0	0	1	-360	360;
	20	21	0.3416	0.1129	0	0	0	0	0	0	1	-360	360;
	21	22	0.014	0.0046	0	0	0	0	0	0	1	-360	360;
	22	23	0.1591	0.0526	0	0	0	0	0	0	1	-360	360;
	23	24	0.3463	0.1145	0	0	0	0	0	0	1	-360	360;
	24	25	0.7488	0.2475	0	0	0	0	0	0	1	-360	360;
	25	26	0.3089	0.1021	0	0	0	0	0	0	1	-360	360;
	26	27	0.1732	0.0572	0	0	0	0	0	0	1	-360	360;
	3	28	0.0044	0.0108	0	0	0	0	0	0	1	-360	360;
	28	29	0.064	0.1565	0	0	0	0	0	0	1	-360	360;
	29	30	0.3978	0.1315	0	0	0	0	0	0	1	-360	360;
	30	31	0.0702	0.0232	0	0	0	0	0	0	1	-360	360;
	31	32	0.351	0.116	0	0	0	0	0	0	1	-360	360;
	32	33	0.839	0.2816	0	0	0	0	0	0	1	-360	360;
	33	34	1.708	0.5646	0	0	0	0	0	0	1	-360	360;
	34	35	1.474	0.4873	0	0	0	0	0	0	1	-360	360;
	3	36	0.0044	0.0108	0	0	0	0	0	0	1	-360	360;
	36	37	0.064	0.1565	0	0	0	0	0	0	1	-360	360;
	37	38	0.1053	0.123	0	0	0	0	0	0	1	-360	360;
	38	39	0.0304	0.0355	0	0	0	0	0	0	1	-360	360;
	39	40	0.0018	0.0021	0	0	0	0	0	0	1	-360	360;
	40	41	0.7283	0.8509	0	0	0	0	0	0	1	-360	360;
	41	42	0.31	0.3623	0	0	0	0	0	0	1	-360	360;
	42	43	0.041	0.0478	0	0	0	0	0	0	1	-360	360;
	43	44	0.0092	0.0116	0	0	0	0	0	0	1	-360	360;
	44	45	0.1089	0.1373	0	0	0	0	0	0	1	-360	360;
	45	46	0.0009	0.0012	0	0	0	0	0	0	1	-360	360;
	4	47	0.0034	0.0084	0	0	0	0	0	0	1	-360	360;
	47	48	0.0851	0.2083	0	0	0	0	0	0	1	-360	360;
	48	49	0.2898	0.7091	0	0	0	0	0	0	1	-360	360;
	49	50	0.0822	0.2011	0	0	0	0	0	0	1	-360	360;
	8	51	0.0928	0.0473	0	0	0	0	0	0	1	-360	360;
	51	52	0.3319	0.114	0	0	0	0	0	0	1	-360	360;
	9	53	0.174	0.0886	0	0	0	0	0	0	1	-360	360;
	53	54	0.203	0.1034	0	0	0	0	0	0	1	-360	360;
	54	55	0.2842	0.1447	0	0	0	0	0	0	1	-360	360;
	55	56	0.2813	0.1433	0	0	0	0	0	0	1	-360	360;
	56	57	1.59	0.5337	0	0	0	0	0	0	1	-360	360;
	57	58	0.7837	0.263	0	0	0	0	0	0	1	-360	360;
	58	59	0.3042	0.1006	0	0	0	0	0	0	1	-360	360;
	59	60	0.3861	0.1172	0	0	0	0	0	0	1	-360	360;
	60	61	0.5075	0.2585	0	0	0	0	0	0	1	-360	360;
	61	62	0.0974	0.0496	0	0	0	0	0	0	1	-360	360;
	62	63	0.145	0.0738	0	0	0	0	0	0	1	-360	360;
	63	64	0.7105	0.3619	0	0	0	0	0	0	1	-360	360;
	64	65	1.041	0.5302	0	0	0	0	0	0	1	-360	360;
	11	66	0.2012	0.0611	0	0	0	0	0	0	1	-360	360;
	66	67	0.0047	0.0014	0	0	0	0	0	0	1	-360	360;
	12	68	0.7394	0.2444	0	0	0	0	0	0	1	-360	360;
	68	69	0.0047	0.0016	0	0	0	0	0	0	1	-360	360;
];

%%-----  OPF Data  -----%%
%% generator cost data
%	1	startup	shutdown	n	x1	y1	...	xn	yn
%	2	startup	shutdown	n	c(n-1)	...	c0
mpc.gencost = [
	2	0	0	3	0	20	0;
];


%% convert branch impedances from Ohms to p.u.
[PQ, PV, REF, NONE, BUS_I, BUS_TYPE, PD, QD, GS, BS, BUS_AREA, VM, ...
    VA, BASE_KV, ZONE, VMAX, VMIN, LAM_P, LAM_Q, MU_VMAX, MU_VMIN] = idx_bus;
[F_BUS, T_BUS, BR_R, BR_X, BR_B, RATE_A, RATE_B, RATE_C, ...
    TAP, SHIFT, BR_STATUS, PF, QF, PT, QT, MU_SF, MU_ST, ...
    ANGMIN, ANGMAX, MU_ANGMIN, MU_ANGMAX] = idx_brch;
Vbase = mpc.bus(1, BASE_KV) * 1e3;      %% in Volts
Sbase = mpc.baseMVA * 1e6;              %% in VA
mpc.branch(:, [BR_R BR_X]) = mpc.branch(:, [BR_R BR_X]) / (Vbase^2 / Sbase);

%% convert loads from kW to MW
mpc.bus(:, [PD, QD]) = mpc.bus(:, [PD, QD]) / 1e3;

That case contains an octave code:

Vbase = mpc.bus(1, BASE_KV) * 1e3;      %% in Volts
Sbase = mpc.baseMVA * 1e6;              %% in VA
mpc.branch(:, [BR_R BR_X]) = mpc.branch(:, [BR_R BR_X]) / (Vbase^2 / Sbase);

%% convert loads from kW to MW
mpc.bus(:, [PD, QD]) = mpc.bus(:, [PD, QD]) / 1e3;

Available at https://github.com/MATPOWER/matpower/blob/master/data/case69.m

Cases like that require octave/matlab to properly run and read the case.

In Python, we can use something like:

import os

from matpower import path_matpower, start_instance

from matpowercaseframes import CaseFrames

m = start_instance()

CASE_NAME = "case69.m"

case_path = os.path.join(path_matpower, "data", CASE_NAME)
with open(case_path, "r") as f:
    print(f.read())

cf_lc = CaseFrames(CASE_NAME, load_case_engine=m)
cf.infer_numpy()
mpc = cf.to_dict()

And then, I hope PowerModels.jl can read that dict from python.

from julia import Main

Main.matlab_data = mpc
Main.eval("""
    matlab_data = convert_python_to_matlab_data(matlab_data)
    data = PowerModels.parse_matpower(matlab_data, func_name=func_name, colnames=colnames, validate=true)
    result = solve_opf(data, ACPPowerModel, nlp_solver)
""")

Maybe, there exist similar approach in julia that did not parse the file directly, but run octave to read it and get the data. In that case, the data will most likely be in the same format as the Dict that I support here.

Please reconsider.

ccoffrin · 2024-10-26T21:44:46Z

Support for things like case69.m is outside the scope of PowerModels. My suggested work around would be to: load case69.m in Matlab (or octave) the save the resulting case data in the format of the other static data matpower data files.

The workflow you propose is reasonable but the code does not need to be in PowerModels. It can live in some other open repo that is used for the research activity you have in mind.

yasirroni · 2024-10-27T02:21:32Z

@ccoffrin this is so sad that my PR as simple as moving the location of your matlab_data, func_name, colnames = _IM.parse_matlab_string(data_string, extended=true) outside function _parse_matpower_string(data_string::String) can't be merged.

And this PR is implementing a well-known best practice of programming concept, that is Single Responsibility Principle. "Each function should do one thing only, and do it well", https://softwareengineering.stackexchange.com/questions/166884/should-i-extract-specific-functionality-into-a-function-and-why.

And make it easier for someone like me that want to send the data from Python, without the need to write plain text of .m file.

In my benchmark, the task is to get data from Python. I will write other code to write plain text of .m file then. Thus, the benchmark for PowerModels.jl will became slower due to read and write thousands of lines of text.

Thanks for your time.

yasirroni added 4 commits October 7, 2024 19:44

support matlab_data to parse_matpower

8892253

fix type

169c205

change test data to use PowerModels case30.m

2f0a9da

test func_name and colnames optional

af8e610

yasirroni mentioned this pull request Oct 9, 2024

The length of result['solution']['gen'] did not match the inputted pm_data['gen] #929

Closed

ccoffrin closed this Oct 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support matpower dict #928

Support matpower dict #928

yasirroni commented Oct 8, 2024

ccoffrin commented Oct 9, 2024

yasirroni commented Oct 9, 2024 •

edited

Loading

yasirroni commented Oct 10, 2024 •

edited

Loading

ccoffrin commented Oct 19, 2024

yasirroni commented Oct 20, 2024

ccoffrin commented Oct 26, 2024

yasirroni commented Oct 27, 2024

Support matpower dict #928

Support matpower dict #928

Conversation

yasirroni commented Oct 8, 2024

ccoffrin commented Oct 9, 2024

yasirroni commented Oct 9, 2024 • edited Loading

yasirroni commented Oct 10, 2024 • edited Loading

ccoffrin commented Oct 19, 2024

yasirroni commented Oct 20, 2024

ccoffrin commented Oct 26, 2024

yasirroni commented Oct 27, 2024

yasirroni commented Oct 9, 2024 •

edited

Loading

yasirroni commented Oct 10, 2024 •

edited

Loading