Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An error is returned when using iterator.Pager and the pageSize is not exactly the number of records that can be retrieved by the query. #383

Open
Reimei1213 opened this issue Jan 25, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@Reimei1213
Copy link

Reimei1213 commented Jan 25, 2025

What happened?

I was trying to implement pagination using iterator.Pager.
During the process, when I ran the query against bigquery-emulator with a pageSize other than the exact number of rows retrievable by the query, the following error occurred.
On the other hand, I confirmed that when the pageSize was set to exactly the number of rows retrievable by the query, the rows could be successfully retrieved.

googleapi: Error 404: job request-20250125-12077940252000-0001 is not found, notFound

What did you expect to happen?

Even if the pageSize is not set to exactly the number of rows retrievable by the query, no error occurs, and the specified number of rows can be retrieved.

How can we reproduce it (as minimally and precisely as possible)?

go version: 1.23.5
bigquery-emulator version: v0.6.6

※ The following code is intended to confirm that the expected process fails. Additionally, when the BigQuery client was switched to the real BigQuery service, it was confirmed that the expected behavior occurred.

func main() {
	const (
		projectID = "test"
		tableName = "test.dataset1.table_a"
	)

	ctx := context.Background()
	bqServer, err := server.New(server.TempStorage)
	if err != nil {
		panic(err)
	}
	if err := bqServer.SetProject(projectID); err != nil {
		panic(err)
	}

        // A file similar to testdata/data.yaml provided by bigquery-emulator.
	if err := bqServer.Load(server.YAMLSource(filepath.Join("testdata", "data.yaml"))); err != nil {
		panic(err)
	}

	testServer := bqServer.TestServer()
	defer testServer.Close()

	cli, err := bigquery.NewClient(
		ctx,
		projectID,
		option.WithEndpoint(testServer.URL),
		option.WithoutAuthentication(),
	)
	if err != nil {
		panic(err)
	}
	defer cli.Close()

	p := pager.NewPager(cli)
	if err := p.InitIterator(ctx, tableName); err != nil {
		panic(err)
	}

        // Since table_a in testdata/data.yaml contains two records, specifying pageSize as 2 results in successful execution, whereas specifying 1 or 3 causes the aforementioned error.
	pageSize := 2
	for {
		rows, err := p.GetDataPage(ctx, pageSize)
		if err != nil {
			panic(err)
		}
		for _, r := range rows {
			fmt.Println(r)
		}
		if p.GetPageToken() == "" {
			break
		}
	}
}
type Pager struct {
	cli       *bigquery.Client
	iterator  *bigquery.RowIterator
	pageToken string
}

func NewPager(cli *bigquery.Client) *Pager {
	return &Pager{
		cli: cli,
	}
}

func (p *Pager) InitIterator(ctx context.Context, tableName string) error {
	q := p.cli.Query(fmt.Sprintf("SELECT * FROM `%s` LIMIT 10", tableName))
	it, err := q.Read(ctx)
	if err != nil {
		return err
	}
	p.iterator = it
	p.pageToken = ""
	return nil
}

func (p *Pager) GetDataPage(_ context.Context, pageSize int) ([][]bigquery.Value, error) {
	var rows [][]bigquery.Value
	pager := iterator.NewPager(p.iterator, pageSize, p.pageToken)
	nextToken, err := pager.NextPage(&rows)
	if err != nil {
		return nil, err
	}
	p.pageToken = nextToken
	return rows, nil
}

func (p *Pager) GetPageToken() string {
	return p.pageToken
}

Anything else we need to know?

No response

@Reimei1213 Reimei1213 added the bug Something isn't working label Jan 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant