Refactor netlist generation

[shanemmattner]

The only changes are to src/skidl/scripts/netlist_to_skidl_main.py and to src/skidl/netlist_to_skidl.py. Other files include an example kicad project, netlists from kicad and skidl, and the output folder containing generated skidl python files

This logic changes the Kicad netlist to SKiDL logic so that the SKiDL project is generated in hierarchical fashion mirroring the KiCAD schematic. Take a look at example_kicad_project_SKIDL directory for the output from example_kicad_project.net, which is the Kicad export from the project in example_kicad_project/. main.net is the SKiDL exported netlist that succesfully imports to Kicad and has the same connections as the based project.

Here is how you can try this logic:

(venv) shanemattner@Shanes-MacBook-Pro skidl % Applications/KiCad/KiCad.app/Contents/MacOS/kicad-cli sch export netlist -o example_kicad_project.net example_kicad_project/example_kicad_project.kicad_sch
(venv) shanemattner@Shanes-MacBook-Pro skidl % netlist_to_skidl -i example_kicad_project.net --output example_kicad_project_SKIDL

=== Extracting Sheet Info ===

=== Assigning Components to Sheets ===

=== Analyzing Nets ===
Wrote sheet file: example_kicad_project_SKIDL/esp32s3mini1.py
Wrote sheet file: example_kicad_project_SKIDL/_3v3_regulator.py
Wrote sheet file: example_kicad_project_SKIDL/resistor_divider1.py
Wrote sheet file: example_kicad_project_SKIDL/USB.py
(venv) shanemattner@Shanes-MacBook-Pro skidl % python3 example_kicad_project_SKIDL/main.py                  
INFO: No errors or warnings found while generating netlist.

(venv) shanemattner@Shanes-MacBook-Pro skidl % 

The core logic works as expected connection wise and could be used with the LLM logic as is. But there's still a bit of work to do to make a completely transparent Kicad > Skidl > Kicad loop. I'll try merging in this fix tomorrow to fix the component reference issue. I also need to tweak the output net names I believe.

[devbisme]

I pulled this into a new branch called netlist_to_skidl. This branch already includes the fix for issue #238. Once we have netlist_to_skidl working as we want, we can merge it into development and llm.

I moved your example into tests/examples/netlist_to_skidl. I wasn't able to convert the netlist extracted from the kicad project; there was some complaint from pyparsing about the .net file. I'm not sure why since it worked correctly for you. This is one reason I want to get rid of pyparsing: it's too hard to debug when it hits an error in the file it's parsing.

[shanemmattner]

Maybe you exported the netlist in a non-default format? I just checked out the branch you made and it seems to work well for me. Can you try generating the netlist to test with the command line tool? For me that is running:

(venv) shanemattner@Shanes-MacBook-Pro skidl % Applications/KiCad/KiCad.app/Contents/MacOS/kicad-cli sch export netlist -o example_kicad_project.net example_kicad_project/example_kicad_project.kicad_sch

I made a 2 updates to this PR and this feature should be ready now: I can succesfully export a Kicad project netlist, import it to a SKiDL based project folder, then run the SKiDL code to produce a netlist which does not change the original Kicad PCB connections. See report.txt for the import report.

[devbisme]

Can you try generating the netlist to test with the command line tool?

I tried those commands (except the KiCad project is in directory kicad_project instead of example_kicad_project) and I got the same error. I'm using KiCad 8.0.8 on Ubuntu 22.04.1. I'll have to look at it more closely and see if I can spot the error.

[shanemmattner]

Can you post the poblematic netlist file? It sounds like there's an edge case that isn't handled properly.

[devbisme]

I attached the netlist file I generated using kicad-cli. (I changed the file extension to .txt so Github would take it.)

Are we both using the same version of KiCad (8.0.8)?

A ParseException occurs in the pyparsing module that is called at the beginning. Here's a small segment of the error message so you can see why these types of errors are hard to trace:

pyparsing.exceptions.ParseException: Missing one or more required elements ({Suppress:('(') Suppress:('components') [Group:({Suppress:('(') Suppress:('comp') {{Suppress:('(') Suppress:('ref') [string enclosed in double quotes ^ string enclosed in single quotes} ^ {Suppress:([<SP><TAB><CR><LF>]...) !W:(()) Suppress:([<SP><TAB><CR><LF>]...)] Suppress:(')')} & {Suppress:('(') Suppress:('value') [string enclosed in double quotes ^ string enclosed in single quotes} ^ {Suppress:([<SP><TAB><CR><LF>]...) !W:(()) Sup

example_kicad_project.txt

[shanemmattner]

That file works fine on my machine. Here's my setup:

Kicad v8.0.6
Macbook pro M4
pyparsing Version: 3.2.1
Python 3.13.1 (venv)

Then I updated to Kicad 8.0.8 to see if that made a difference. I ran the kicad_to_netlist command as before, then parsed the resulting .net file without error:

venvshanemattner@Shanes-MacBook-Pro skidl % /Applications/KiCad/KiCad.app/Contents/MacOS/kicad-cli sch export netlist -o example_kicad_project.net example_kicad_project/example_kicad_project.kicad_sch
venvshanemattner@Shanes-MacBook-Pro skidl % netlist_to_skidl -i example_kicad_project.net --output example_kicad_project_SKIDL

=== Extracting Sheet Info ===

=== Assigning Components to Sheets ===

=== Analyzing Nets ===
Wrote sheet file: example_kicad_project_SKIDL/esp32s3mini1.py
Wrote sheet file: example_kicad_project_SKIDL/_3v3_regulator.py
Wrote sheet file: example_kicad_project_SKIDL/resistor_divider1.py
Wrote sheet file: example_kicad_project_SKIDL/USB.py

Interestingly, I don't find pyparsing anywhere in the Skidl repo. Can you point me towards where I might hunt down the error?

[devbisme]

Ah, that's embarassing: I had an old version of kinparse (my own project!) installed. Everything works now. I updated the required package versions in setup.py.

[shanemmattner]

Awesome! Let me know if there's any changes or tests you'd like implemented before finishing off this feature.

[devbisme]

I made some comments about the issue of tagging components in issue #238, but there isn't a need to address this right now since the old code has the same problem.

I probably need to add a test for the new code before merging everything into the development branch.

[devbisme]

There is already a netlist_to_skidl test in test_parse.py. I'll need to expand that to include hierarchy and a comparison that checks netlists for equivalency.

[shanemmattner]

Are you ok with making the tests? I've turned my attention towards Kicad 8 schematic generation but I'd like to finish off this feature since it could be really valuable to use along with the LLM code.

[devbisme]

I'll merge the netlist_to_skidl branch into development and then create the test later.

[devbisme]

I've merged netlist_to_skidl into development. (I'll get rid of netlist_to_skidl once you say so.)

I also merged development into llm. Which one do you want to work from for the schematic generation stuff?

[shanemmattner]

There shouldn't be much overlapping changes so merge the branches however you think best and I'll keep working off my schematic generation feature branch I have going

[shanemmattner]

Testing on llm it looks like development still needs to be merged. There's also some broken logic it seems with the llm logic so I'll try to fix that and open another PR

[devbisme]

I pushed the updated llm branch.

[shanemmattner]

Ok, good news: I made a script that can take a kicad_sch, turn it into a netlist file, then turn the netlist file into a SKIDL project, then analyze the circuit with the LLM logic:
kicad_skidl_llm.txt

Terminal output:

venvShanes-MacBook-Pro:skidl shanemattner$ python3 kicad_skidl_llm.py -s ~/Desktop/skip/electronics/PCB/control_board/control_board_0_8/control_board/control_board.kicad_sch -o control_board_v8^C
venvShanes-MacBook-Pro:skidl shanemattner$ python3 skidl_llm_test.^C
venvShanes-MacBook-Pro:skidl shanemattner$ python3 kicad_skidl_llm.py -s ~/Desktop/example_kicad_project/example_kicad_project.kicad_sch -o example_kicad_project --analyze
Found valid schematic: /Users/shanemattner/Desktop/example_kicad_project/example_kicad_project.kicad_sch
Generated netlist: example_kicad_project/example_kicad_project.net
Successfully generated netlist at: example_kicad_project/example_kicad_project.net

Generated SKiDL project files:
  USB.py
  _3v3_regulator.py
  esp32s3mini1.py
  main.py
  resistor_divider1.py
  test_examples.py
Successfully generated SKiDL project at: example_kicad_project

Running circuit analysis...

Subcircuit Docstrings:

=== Saving Query with anthropic/claude-3.5-haiku ===

Saving analysis to query.txt...
Analysis saved successfully

=== Query saved successfully ===

=== Starting Circuit Analysis with google/gemini-flash-1.5 ===

Generating analysis...

=== Analysis completed in 11.95 seconds ===

=== Starting Circuit Analysis with google/gemini-flash-1.5 ===

Generating analysis...

=== Analysis completed in 9.33 seconds ===

=== Starting Circuit Analysis with google/gemini-flash-1.5 ===

Generating analysis...

=== Analysis completed in 10.53 seconds ===

=== Starting Circuit Analysis with google/gemini-flash-1.5 ===

Generating analysis...

=== Analysis completed in 10.13 seconds ===

=== Starting Circuit Analysis with google/gemini-flash-1.5 ===

Generating analysis...

=== Analysis completed in 6.34 seconds ===

Analysis Results:

Subcircuit: top.USB0
Analysis completed in 11.95 seconds
Tokens used: 1526

Subcircuit: top._3v3_regulator0
Analysis completed in 9.33 seconds
Tokens used: 1828

Subcircuit: top.esp32s3mini10
Analysis completed in 10.53 seconds
Tokens used: 2746

Subcircuit: top.esp32s3mini10.resistor_divider10
Analysis completed in 10.13 seconds
Tokens used: 1352

Subcircuit: top.test_examples0
Analysis completed in 6.34 seconds
Tokens used: 1158

Total analysis time: 48.27 seconds
Total tokens used: 8610

Analysis results saved to: circuit_analysis.txt

circuit_analysis.txt

[shanemmattner]

Bad news: this logic doesn't work on my custom project.

• kinparse doesn't handle newer schematics netlists properly (something about textvars)
  • removing textvars results in parsing the netlist ok
• custom component fields confuse the logic. ie the C field below:

   C52 = Part('Device', 'C_Small', value='CL03A104KO3NNNC', footprint='Capacitor_SMD:C_0201_0603Metric', ref='C52', C=100nF, Sheetname='Left Leg', Sheetfile='leg.kicad_sch', ki_keywords='capacitor cap', ki_fp_filters='C_*')

Then SKIDL can't parse these components properly since they are invalid python code

• if a schematic doesn't have components (ie a power timing sheet or block diagram) then a blank function is generated in the SKIDL code and that causes compiler issues

I'm guessing there's a few other edge cases. I'll start working on these edge case solutions this week.

[devbisme]

• kinparse doesn't handle newer schematics netlists properly (something about textvars)

I can add that to the pyparsing string in kinparse if you give me an example netlist file.

• custom component fields confuse the logic. ie the C field below

Is it just a matter of adding quotes on the 100nf ? Or is it more complicated?

• if a schematic doesn't have components (ie a power timing sheet or block diagram) then a blank function is generated in the SKIDL code and that causes compiler issues

Just add a return statement in any blank function? (What does a schematic without components do? Does it just have documentation or something?)

[shanemmattner]

• Here is the netlist I was trying to parse which gave me errors
  control_board_net.txt

• adding quotes to custom fields should be easy

• adding a return statement to blank schematics should be easy

I'll take a shot at fixing the 2nd and 3rd point if you want to do the kinparse fix

[devbisme]

The new 1.2.3 version of kinparse is on PyPi and I've updated the development and llm branches of skidl to use it. Try it and make sure it behaves.

[shanemmattner]

The new 1.2.3 version of kinparse is on PyPi and I've updated the development and llm branches of skidl to use it. Try it and make sure it behaves.

Confirmed the new change is working to parse kicad netlists without error. Thank you!

I had to go back and work on the netlist_to_skidl.py code. When I tested this logic on a more complex real designs (ie proper documentation schematic files with no parts or nets, various sheets and hierarchies, etc) the logic broke. I'll put up another PR when the new logic is ready for you to review.