[Bug][GitLab] Missing the notes of Merge Request

[usharerose]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Some merge requests themselves have been collected, but their notes (comments) are missing.

It seems that only opened Merge Request would be the input of mr_notes_collector (reference), which means the following MRs' note data would not be collected

• the history MRs (closed, merged) of one GitLab project
• one MR's complete lifecycle (created, commented, and merged) fell within the gap between DevLake pipelines

What do you expect to happen

The notes have been collected

How to reproduce

Collect any GitLab project with merged or closed MRs, those MRs' related notes would be missing.

Anything else

No response

Version

v1.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

The issue you're encountering is due to the current implementation of the GetMergeRequestsIterator function, which only considers open merge requests for note collection. This is intended to speed up the process by focusing on active merge requests ^[1].

To address this and include notes from closed or merged merge requests, you would need to modify the GetMergeRequestsIterator function. Specifically, you can adjust the SQL WHERE condition to include "closed" and "merged" states in addition to "opened". Here's a snippet of how you can modify the code:

func GetMergeRequestsIterator(taskCtx plugin.SubTaskContext, apiCollector *api.StatefulApiCollector) (*api.DalCursorIterator, errors.Error) {
	db := taskCtx.GetDal()
	data := taskCtx.GetData().(*GitlabTaskData)
	clauses := []dal.Clause{
		dal.Select("gmr.gitlab_id, gmr.iid"),
		dal.From("_tool_gitlab_merge_requests gmr"),
		// collect notes and commits for opened, closed, and merged merge requests
		dal.Where(
			`gmr.project_id = ? and gmr.connection_id = ? AND state IN (?, ?, ?)`,
			data.Options.ProjectId, data.Options.ConnectionId, "opened", "closed", "merged",
		),
	}
	if apiCollector != nil {
		if apiCollector.GetSince() != nil {
			clauses = append(clauses, dal.Where("gitlab_updated_at > ?", *apiCollector.GetSince()))
		}
	}
	// construct the input iterator
	cursor, err := db.Cursor(clauses...)
	if err != nil {
		return nil, err
	}

	return api.NewDalCursorIterator(db, cursor, reflect.TypeOf(GitlabInput{}))
}

This change will allow the collection of notes from all states of merge requests, ensuring that historical and lifecycle-complete notes are included in the data collection process ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[hippojay]

This bug breaks DORA cycle time metrics. Only using open merge requests to collect merge commits and notes/comments means that it is not possible to calculate meaningful and actionable metrics.

It was introduced in this pull request #7997 and looks to be in release 1.0.2 beta 4 onwards.

Specific commit: 191f2e6

Relatively easy to reverse this change through removal of the "state =" clause, which I have briefly confirmed works in a limited test env.

[karpanin]

Yes, I have the same problem with task Collect MR Commits.

[hippojay]

I've been running with this commit reverted for a week - it has fixed all the Dora metrics, mr pickup times, and other stats.

It does run longer due to collecting more information - but is necessary for accurate data.

[klesh]

The problem should be fixed by #8256 already, I think we should release a new beta version for everybody.
@d4x1

[d4x1]

We have released a new beta version v1.0.2-beta6, just have a try. @klesh @hippojay @usharerose @karpanin