Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extraction: refine img src url and fix table extraction bugs #762

Merged
merged 1 commit into from
Dec 6, 2024

Conversation

unsleepy22
Copy link
Contributor

  1. refine img src url, allow for absolute and relative paths
  2. fix bugs in table extraction: 1) the last row with colspan is not well formatted in markdown/txt; 2) remove newlines in table cell markdown in some cases

Copy link

codecov bot commented Dec 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.27%. Comparing base (76200b7) to head (0be7633).
Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #762   +/-   ##
=======================================
  Coverage   99.26%   99.27%           
=======================================
  Files          21       21           
  Lines        3559     3572   +13     
=======================================
+ Hits         3533     3546   +13     
  Misses         26       26           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

trafilatura/main_extractor.py Outdated Show resolved Hide resolved
trafilatura/main_extractor.py Outdated Show resolved Hide resolved
@adbar
Copy link
Owner

adbar commented Dec 6, 2024

Great work, the tests show that you definitely improve table processing with this PR.

I just have a few comments to make the code more readable and easier to maintain.

@unsleepy22 unsleepy22 force-pushed the refine_img_src_and_table_extraction branch from 2d39b3e to 0be7633 Compare December 6, 2024 14:23
@unsleepy22
Copy link
Contributor Author

Great work, the tests show that you definitely improve table processing with this PR.

I just have a few comments to make the code more readable and easier to maintain.

All fixed, would you take a look again?

@adbar adbar merged commit 7067937 into adbar:master Dec 6, 2024
15 checks passed
@unsleepy22 unsleepy22 deleted the refine_img_src_and_table_extraction branch December 6, 2024 15:01
@unsleepy22
Copy link
Contributor Author

Great work, the tests show that you definitely improve table processing with this PR.

I just have a few comments to make the code more readable and easier to maintain.

There could be subtle bugs in table extraction as tables sometimes get really complicated, luckily I may encounter enough cases during my work (collecting LLM corpus), so hopefully bugs converge gradually and we may set up enough unit tests or real world test cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants