Remove dnetup format and add Magic Point Shop files

[alegresor]

We should remove support for the dnetup format. This requires generating matrices to have the first row be the most significant bit. This gist converts digital net files from the Magic Point Shop into the desired format by reversing the bits.

[fjhickernell]

Good idea, but let’s not use o as a variable. It can be confusing. Best, Fred Fred J. Hickernell Vice Provost for Research, research.iit.edu <https://research.iit.edu/> www.iit.edu/research/about/magazine <https://www.iit.edu/research/about/magazine> Professor, Department of Applied Mathematics Center for Interdisciplinary Scientific Computation (CISC) cos.iit.edu/cisc/ <https://cos.iit.edu/cisc/> Illinois Institute of Technology IIT Tower, Room 10F9-1, 10 W 35th St, Chicago, IL 60616 ***@***.*** ***@***.***>, +1 312 567 3470 <tel:+1%20312%20567%203470> Working remotely. Please contact me by email or phone.

…

On Dec 20, 2023, at 11:57 AM, alegresor ***@***.***> wrote: We should combine the dnet and dnetup formats by adding a Boolean parameter $o$ to dnet which specifies whether the first row is is the most significant bit, $o=1$, or the least significant bit, $o=0$. This relieves programs from having to parse the first comment line as # dnet vs # dnetup as previously specified. With this change all comments can be ignored when the file is passed to a digital net generator. You can view, comment on, or merge this pull request online at: #1 Commit Summary 9633209 <9633209> add MSB vs LSB to dnet parameters File Changes (2 files <https://github.com/QMCSoftware/LDData/pull/1/files>) M README.md <https://github.com/QMCSoftware/LDData/pull/1/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5> (14) M dnet_dnetup/mps.sobol_Cs.txt <https://github.com/QMCSoftware/LDData/pull/1/files#diff-f7ff0f065bea0ce8324966c880ca5d52f04b4fe7aff66c71c44335edecc2a2f2> (1) Patch Links: https://github.com/QMCSoftware/LDData/pull/1.patch https://github.com/QMCSoftware/LDData/pull/1.diff — Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGHS6VTXZ45P2VZ7EIM6O3YKMRGZAVCNFSM6AAAAABA5GTOW2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TCMBSGU4TOOA>. You are receiving this because you are subscribed to this thread.

[alegresor]

Good idea, but let’s not use o as a variable. It can be confusing.

Changed to $v$.

[pierrelecuyer]

Interesting point. Maybe the right question is Do we really need dnetup? When I wrote this proposal, I tried to put the minimal number of compulsory parameters or options in the files, nothing that was not essential, to make the file format as simple as possible. This is the reason why there are different file types. At first, I had only dnet, not dnetup. In my software, I use only dnet and do not support dnetup, because the generating matrices are represented in the dnet format when I generate the points, and so I want to read them in that format. It seems to me that it should be the same for most QMC software. I also never read the first line with the file type, it is always assumed. I think Dirk Nuyens asked me to add dnetup as an alternate file type because he likes to use this representation, and so I did, but without changing the dnet file format at all. The idea is to avoid changing current formats when we add new ones. I assumed that the support of dnetup would be only optional. If it was only for me, I would have only dnet.

Now if most people prefer having an additional parameter for how the columns are represented, that's fine with me. But perhaps we should try to be consistent with other file types as well. For example sobol vs soboljk. Should we put an additional parameter there as well? Or should we have a single "file type" and the format will be read as the first parameter, in all cases? I expect that not all types will be implemented in all software, and that's fine.

Let me know what you decide in the end. Regards.

-- Pierre

[pierrelecuyer]

One small comment in the proposed changes (in green): In most places, "most significant bit" should be "most significant digit", because the base b is not always 2.

[alegresor]

One small comment in the proposed changes (in green): In most places, "most significant bit" should be "most significant digit", because the base b is not always 2.

Changed in the latest commit.

[alegresor]

Interesting point. Maybe the right question is Do we really need dnetup? When I wrote this proposal, I tried to put the minimal number of compulsory parameters or options in the files, nothing that was not essential, to make the file format as simple as possible. This is the reason why there are different file types. At first, I had only dnet, not dnetup. In my software, I use only dnet and do not support dnetup, because the generating matrices are represented in the dnet format when I generate the points, and so I want to read them in that format. It seems to me that it should be the same for most QMC software. I also never read the first line with the file type, it is always assumed. I think Dirk Nuyens asked me to add dnetup as an alternate file type because he likes to use this representation, and so I did, but without changing the dnet file format at all. The idea is to avoid changing current formats when we add new ones. I assumed that the support of dnetup would be only optional. If it was only for me, I would have only dnet.

Now if most people prefer having an additional parameter for how the columns are represented, that's fine with me. But perhaps we should try to be consistent with other file types as well. For example sobol vs soboljk. Should we put an additional parameter there as well? Or should we have a single "file type" and the format will be read as the first parameter, in all cases? I expect that not all types will be implemented in all software, and that's fine.

Let me know what you decide in the end. Regards.

-- Pierre

I think only supporting dnet will help others adopt our standard and more easily write generators. I've changed the aim of this pull request to be removing support for dnetup. The new PR description includes a gist which converts generating matrices from the Magic Point Shop into the dnet format. All Magic Point Shop lattice and digital net files have been converted and added to the PR. Hopefully the Magic Point Shop and LatNet Builder will directly output the lattice and dnet formats in the future to avoid conversions.

[alegresor]

@pierrelecuyer can you give this updated PR a review?

[pierrelecuyer]

I merged the Pull request; sorry about the delay. However, it seems there are still problems with the formulas in the readme file, as I mentioned earlier by email.

Also, it seems that the readme file is essentially a copy of the document that I wrote and put here:
https://www-labs.iro.umontreal.ca/~lecuyer/myftp/papers/qmc-points-format-2023.pdf
However, I think it would be a good idea to mention it and put a link to this .pdf file.
I just made some updates in it: I removed dnetup again, added one sentence in red, and made a few very small corrections.

Unfortunately, I do not have time to check carefully if the copy on the GitHub page is exact (no errors or changes).

-- Pierre

[alegresor]

Thanks Pierre!

There is an issue for the readme markdown math not rendering #2

I created README.md as a direct copy of your .tex file which gets displayed on the front page of our repo.