Issues with cards in TopLoaders and 9-section

[luzer]

hi
i am having trouble detecting cards when they are in toploaders or plastic binder sleeves?

any ideas?
CJ

[YenTheFirst]

This software is currently using a pretty naive approach to doing card detection. Basically, it

1. records a static background image
2. diffs the live video against the background, to determine what's new
3. performs edge detection on that difference
4. tries to wrap the largest possibly-skew rectangular convex hull it can around edge-esque pixels
5. if successful, consider the result a card, cut out that portion of the image, and warp it flat.
6. Once a card has been detected, it's identified by, basically, finding the Oracle image which is the most similar to it.

This is far from best-in-class detection, it was just something hacky to get the project off the ground, and with carefully arranged lighting conditions, it was good enough for the particular use case. My particular use case was indexing cards for storage in bulk boxes, so cards were removed from protection.

I've seen a few different issues related to card protectors

• Some card protectors are notably larger than the card. i.e., hard toploaders, or things like Dragon Shields. These might get detected as rectangular regions, but when trying to identify the card, there's a minor scaling and bordering effect which trips up the matching algorithm.
• I've also seen situations where the detected hull included two corners of the card protector, and two corners of the card, and the result of the 'flattening' created a trapezoidally-warped card that failed to get matched.
• Depending on lighting, shiny card protection obscure details of the card.
• If you've got multiple cards visible to the camera, like in a 9-section card sheet, the current detection algorithm just isn't equipped to handle that situation.

The best way to resolve this issue would probably be to develop a classifier that's specifically trained to detect generic MTG-card-like objects in view, so as to be able to handle a wider variety of backgrounds, and handle multiple cards at once. This would expand potential use cases from just static, controlled indexing to handling live gameplay.

In the short term, there may be some simple tweaks one can do to detect_card.py that might get it working better in your situation.

To help debug, it would be useful to get a screenshot of what's not detecting, and the included terminal output.

[luzer]

i am actually using a modified version for a pokemon card scanner i am working on -
i have problems with GX cards (anything without a yellow border)
and cards in toploaders, and cards in binder sheets

(see sample video here https://www.dropbox.com/s/azx5d44vrhjwwry/Problem%20with%20GX.mov?dl=0 )

would love any advice or help

def find_card(img, thresh_c=5, kernel_size=(3, 3), size_thresh_max=10000):
    """
    Find contours of all cards in the image
    :param img: source image
    :param thresh_c: value of the constant C for adaptive thresholding
    :param kernel_size: dimension of the kernel used for dilation and erosion
    :param size_thresh: threshold for size (in pixel) of the contour to be a candidate
    :return: list of candidate contours
    """
    # Typical pre-processing - grayscale, blurring, thresholding
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray,(1,1),1000)
    #img_thresh = cv2.adaptiveThreshold(img_gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 5, thresh_c)
    # Dilute the image, then erode them to remove minor noises
    kernel = np.ones(kernel_size, np.uint8)
    img_dilate = cv2.dilate(img_blur, kernel, iterations=3)
    img_erode = cv2.erode(img_dilate, kernel, iterations=3)

    img_filter = cv2.bilateralFilter(img_erode, 11, 17, 17)
    img_erode = cv2.Canny(img_filter, 30, 200)

    #trying to skip toploaders
    #kernel = np.ones((5, 5), np.uint8)
    #dilation = cv2.dilate(img_erode, kernel, iterations = 1)
    #cnts, hier  = cv2.findContours(dilation, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

    # Create Threshold
    #_, thresh = cv2.threshold(img_gray, 130, 255, cv2.THRESH_BINARY)


    # Find the contour
    cnts, hier = cv2.findContours(img_erode, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
    #cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:5]
    
    #cnts, hier = cv2.findContours(img_erode, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    if len(cnts) == 0:
        print('no contours')
        return []

    # The hierarchy from cv2.findContours() is similar to a tree: each node has an access to the parent, the first child
    # their previous and next node
    # Using recursive search, find the uppermost contour in the hierarchy that satisfies the condition
    # The candidate contour must be rectangle (has 4 points) and should be larger than a threshold
    cnts_rect = []
    stack = [(0, hier[0][0])]
    #print (len(stack))
    while len(stack) > 0:
        i_cnt, h = stack.pop()
        i_next, i_prev, i_child, i_parent = h
        if i_next != -1:
            stack.append((i_next, hier[0][i_next]))
        cnt = cnts[i_cnt]

        #added
        #rect = cv2.minAreaRect(cnt) # get a rectangle rotated to have minimal area
        #box = cv2.boxPoints(rect) # get the box from the rectangle
        #box = np.int0(box) # the box is now the new contour.
        #cnt = box


        size = cv2.contourArea(cnt)
        peri = cv2.arcLength(cnt, True)
        approx = cv2.approxPolyDP(cnt, 0.04 * peri, True)
        #if len(approx) :
        #    print('Size: ',size)
        #print('Length Approx: ',len(approx))

        if size >= size_thresh_max and len(approx) == 4:
            cnts_rect.append(approx)
        else:
            if i_child != -1:
                stack.append((i_child, hier[0][i_child]))
    
    #print('Length Cnts Rectange: ' ,len(cnts_rect))
    return cnts_rect

[YenTheFirst]

I skimmed through the video - I see issues with recognizing GX cards, such as at timestamp 12:45, but I don't see any toploaders nor binder sheets used in this video.

I can take a look at what the issue is with GX cards.

[luzer]

thanks so much. i will post a toploader and 9-sleeve video

[luzer]

here is a toploader and penny plastic test - https://www.dropbox.com/s/emq8f9dr7poxi86/toploader%20test%201.mov?dl=0

[YenTheFirst]

I'm currently in the process of dusting the cobwebs off this project and getting an up-to-date version published with python 3 and opencv 4. I apologize for the delays there.

There's a few things that would be helpful for making this a more actionable bug report:

• "minimally reproducing input"
  • The posted videos include UI and highlighting. This is useful for demonstrating when things are and aren't matching, but it's difficult to reproduce the exact issue without the original video.
  • The videos are relatively long, so I'm not sure which portions are meant to be a bug demonstration
  • A useful example for the bug report would be a single raw frame capture from the camera, and a description of what you expect it to match, and what it is matching
• reproducing code
  • it looks like you've forked and modified the project.
  • I can really only commit to any kind of support for the project as-is. That said, I'd be happy to give suggestions on derivative projects, and I'd be happy to consider any pull requests to this project.
  • I've tried to investigate possible issues based on your posted code, but I'd strongly suggest putting your current modifications online in their entirety.

[YenTheFirst]

An investigation of the posted input video, tested against the current project's code:

The project as-is tries to due naiive background segmentation by keeping track of a base image.
I manually selected this frame as a base, as it was the most recent stable image before the frame under investigation:

You seemed to be indicating this card as an issue, so I'm investigating this frame:

The program as-is diffs these two. Notice that the deck in frame in upper right is highlighted by this diff, and some minor camera movement is also causing the edges of the keyboard to be highlighted.

The program then does edge detection

And wraps a convex hull around long contours:

The original program is just not at all robust against any kind of noise or distraction in the frame, and is unable to detect any cards under these conditions. It was developed using a significantly more controlled environment of a plain white background, even lighting, and a single card at a time.

[YenTheFirst]

The same frame, using your modified code:

The input frame:

After your preprocessing:
Notice that the dilation/erosion starts to connect the card to the sides of the toploader.

Edge detection:
Near the blurred areas, it's not seeing a sharp edge on the card. The toploader does present a sharp edge against the background on all 4 sides.

Contour detection, all contours with random colors:
No surprises here, it's finding a connected contour all the way around the outside of the toploader. For the card itself, there's not edges present from the previous step to find a reasonable contour.

Contours selected by your logic.
color key:

• green: Contours that have both size >= size_thresh and len(approx) == 4
• blue: size >= size_thresh, approx != 4
• red: len(approx) == 4, size < size_thresh
• white: neither
  We see the toploader is detected as a candidate rectangle, but not the card inside.
  

I don't have any great suggestions for improvements to this approach off the top of my head. I'd just continue to experiment and tweak.

[luzer]

i just realized you replied! amazing

i will take a look