Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dist: build openwrt packages #4464

Merged
merged 83 commits into from
Mar 5, 2024
Merged

Conversation

jiceatscion
Copy link
Contributor

@jiceatscion jiceatscion commented Feb 8, 2024

The method is the following:

  • Add the prebuild openwrt SDKs for the desired target platform as an http_archive dependency.
  • Slap a BUILD.bazel file on top of the SDK so bazel can make it do two things:
    • Configure a bazel toolchain around the compiler suite embedded in the SDK (musl-gcc/musl-libc).
    • Make one openwrt package for each SCION component (the pieces are provided by the SCION main workspace).
  • Declare a platform for each supported openwrt target.
  • Bind each openwrt toolchain to its matching platform.
  • Add rules to produce ipk packages for scion components (they just reference and copy the packages produced by the openwrt workspace).
  • Use multiplatform_filegroup() to transition these packages to each supported openwrt platform.
  • Add a Make target, dist-openwrt, to build the openwrt file group.

That results in the following chain of dependencies for one given component and the amd64 target:
=> Want openwrt
=> build openwrt_amd64 (among others, if any)
=> build "component.ipk for openwrt_amd64"
=> copy "component.ipk for openwrt_amd64" from @external/openwrtSDK_x86_64
=> @external/openwrtSDK_x86_64 obtains "component exec for openwrt_amd64" from @scion, then embed in component.ipk package.
=> @scion builds component with the openwrt_amd64 toolchain

Embellishments:

  • Optionally generate versioned file names for installables. That removes the need for renaming the files after the build.
  • Added coremark to our build (and as an openwrt package). The intent is to use it in benchmarking as a rough cpu power metric.
  • Executable compression when packaging for openwrt, as these devices can be storage constrained and Go executables are huge. Unfortunately there's not much I can do for the in-ram size yet.

This change is Reviewable

The method is to import openwrt_SDK as an external dependency and
slap a bazel build file on in.

The rules in that BUILD (and associated .bzl) file encapsulate a
shell script that follows the basic recipe for creating new packages
in the openwrt sdk. (https://digggy.medium.com/openwrt-quick-start-493e08ed73f
is a nice concise one).
The deviation from that recipe is that the binary to be packaged isn't
built within openwrt, it is copied from the scion build.

Building anything in a non-bazel tree while using bazel is difficult:
basel wants you to declare everything you'll touch so it can copy it to
a sandbox....no-can-do. So instead I run the build rule with "no-sandbox".

There are some other hoops to jump through due to cryptic paths shenanigans.
Tried to put comments where it mattered.
The content is rudimentary or plain wrong; placeholders to exercise
the process. Also missing: copying the router package to something
that truly has the same name; i.e. with versioning.
That config will only allow the router to stay alive if there's
two interfaces configured: 192.168.1.1 and 172.20.0.2 but that makes
my life simpler for a simple sanity check.
…ain.

That includes dynamically linked files.
This is ugly and needs cleanup but I don't want to loose it to broken laptop.
Tuck most things inside dist/openwrt.
Also did some mild cleanup in toolchain packaging.
refers to the openwrt SDK. Now it's possible to add openwrt SDKs
for other targets and build packages for them.
…mes.

For file names, development builds get a version number like:
tag-commitCntSinceTag-dirty, while clean builds get tag-commitCnrSinceTag.

For the version number inside the packages, for dirty builds it is like:
tag-commitCntSinceTag-dirty<timestamp>. The idea being that openwrt
package manager will be happy to install newly built dev packages over a
previous one (including the matching clean build) because its version
number increases. Of course, the timestamp isn't a dependency in itself so
things get rebuild only when something else changes. That's also why we do
not include it in file names.
Also add some uniformity to config files naming and
fix the makefile template to handle properly the absence
of config or init.d files.
an intermittent broken elf scion-pki too.
Added README files as non-obscure way to create subdirectires in /etc/scion
for config files that can't have a default value.

Added postinstall (to all scion packages) to create /var/lib/scion.
platform rather than command line.

While at it, added an entry in the Makefile to build the openwrt packages:
dist-openwrt

Unfortunately multiple openwrt dists (should there be another one) cannot be built at the
same time yet. The build for each target has to be serialized.
Put shared configs (samples) in a separate package.
Split DB files in per-component directory. Save them to /usr/lib upon
component stop. Restore them to /var/lib upon component start.

No more postinstall needed.
Improvements:
* Got rid of the remaining explicit x86_64 target_arch parameters; instead
  get that from selects() on the target cpu.
* Add standard headers to the toolchain (so bazel doesn't consider them
  an undeclared dependency.

Temporary workaround:
* Regardless of the above changes, platform_transition_filegroup appears
  to not forward the target_platform setting to the build of the component
  files. They get build for the default target. The work-around is to
  assert the platform explicitly when building for openwrt with
  --platforms=openwrt_amd64
In passing, had to fix the openwrt toolchain to declare ar and other
executable files correctly. Also do not link libstdc++ for no reason.

(Coremark of our openwrt-running micropc: 2865.740077)
That mistake was the reason for the failure to build ipk packages without
forcing the platform on the command line: user error calling
"bazel build //dist:openwrt" instead of "bazel build //dist:openwrt_all".
…wrt packages.

The compressed binaries are available for all targets, under the name <binary>_compressed.
Besides openwrt packages, nothing in the build depends on them, so they're not built unless
producing openwrt packages.
Copy link
Contributor Author

@jiceatscion jiceatscion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 58 files reviewed, 7 unresolved discussions (waiting on @lukedirtwalker and @matzf)


versioning.bzl line 1 at r1 (raw file):

Previously, jiceatscion wrote…

Done. I no-longer need to commit it. Dominik and Lukas are neither for or against the method. They asked however why the heck we bother naming the package files automatically from git version as we do (regardless of the method). They observed that the CI pipeline is perfectly capable of passing a version string as an argument to the build. Something that they do routinely for their own stuff. So, if we do not need to versionize packages every time we build them, but only when CI does, then we might not need any of that. Here how they do it with tar files:
pkg_tar(

name = "anapaya\_scion\_pkg\_empty",

srcs = \[

    ":empty\_pkg\_files",

\],

extension = "tar.gz",

package\_file\_name = "{pkg\_name}-{pkg\_version}.tar.gz",

package\_variables = ":pkg\_name",

tags = \["local"\],

)

pkg_name(

name = "pkg\_name",

pkg\_name = "anapaya-scion",

pkg\_version = ":version",

)

version_flag(

name = "version",

build\_setting\_default = "dev",

)

To build with specified version: "bazel build //tools/scion_pkg_builder:anapaya_scion_pkg --//tools/scion_pkg_builder:version=$${RELEASE_TAG}"

If it is not possible to do something like that, then I'm happy to report that the wrapper works like a charm but if you refuse to adopt it, I'll just go back to copying/renaming. I never cared to the level of the time we spent on this. I never expected it was going to be such a huge deal.

Working on your other remarks while you consider this question.

Following our conversation about this. Will do something along those lines.

At the request of the reviewers, no-longer auto-generate a version tag for file
names. This is to avoid the need for a bazel wrapper script or other acrobatics.
It happens that we also rarely need the versionned names; normally only when
building publishables from the CI pipeline; so an extra argument on the
command line isn't inconvenient.

We now rely on the pkg_deb support for configurable file names and have added
similar support in openwrt packaging rules. versioning.bzl, tools/bazel, and
the file copies in the Makefile have been replaced by a simple symlink of all
assets to a convenient location.

In the absence of directives, the version tag is "dev".
The version strings embeded in packages and binaries is always generated, as
before.
Copy link
Contributor Author

@jiceatscion jiceatscion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 59 files reviewed, 6 unresolved discussions (waiting on @lukedirtwalker and @matzf)


versioning.bzl line 1 at r1 (raw file):

Previously, jiceatscion wrote…

Following our conversation about this. Will do something along those lines.

done

Strip the '^v' from the git version at the source, so we can use the output
of tools/git-version or the generated git-version file interchangeably.

Build deb and openwrt packages the same way in the CI pipeline.
…ring. Like deb. So the file names are easily predictible.
Openwrt splits version from release at the last dash. So, we can't have a dash
as a separator inside the release string (for dirty builds). Use a dot instead.
Copy link
Contributor

@matzf matzf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 17 of 50 files at r1, 2 of 7 files at r3, 1 of 7 files at r4, 4 of 4 files at r5, 21 of 27 files at r6, 7 of 8 files at r7, 3 of 8 files at r8, 7 of 7 files at r9, all commit messages.
Reviewable status: all files reviewed, 14 unresolved discussions (waiting on @jiceatscion)


.buildkite/pipeline.yml line 33 at r9 (raw file):

      - tar -chaf scion-deb-arm64.tar.gz *_$(../tools/git-version)_arm64.deb
      - tar -chaf scion-deb-i386.tar.gz *_$(../tools/git-version)_i386.deb
      - tar -chaf scion-deb-armel.tar.gz *_$(../tools/git-version)_armel.deb

Nit: maybe define version as a (shell-) variable, so it's a little less verbose, e.g.
(same below for openwrt)

Suggestion:

      - version=`tools/git-version`
      - make dist-deb BFLAGS="--file_name_version=${version}"
      - cd installables;
      - tar -chaf scion-deb-amd64.tar.gz *_${version}_amd64.deb
      - tar -chaf scion-deb-arm64.tar.gz *_${version}_arm64.deb
      - tar -chaf scion-deb-i386.tar.gz *_${version}_i386.deb
      - tar -chaf scion-deb-armel.tar.gz *_${version}_armel.deb

.buildkite/pipeline.yml line 35 at r9 (raw file):

      - tar -chaf scion-deb-armel.tar.gz *_$(../tools/git-version)_armel.deb
    artifact_paths:
      - "installables/*.tar.gz"

Might have to restrict this to installables/scion-deb*.tar.gz (and analogously scion-openwrt*.tar.gz below) -- otherwise we might upload the same thing twice. Not sure if this would break something, but at least it would be wasteful.


.buildkite/pipeline.yml line 44 at r9 (raw file):

            - <a href="artifact://deb/scion-deb-arm64.tar.gz">arm64</a>
            - <a href="artifact://deb/scion-deb-i386.tar.gz">i386</a>
            - <a href="artifact://deb/scion-deb-armel.tar.gz">armel</a>

Suggestion:

            - <a href="artifact://installables/scion-deb-amd64.tar.gz">amd64</a>
            - <a href="artifact://installables/scion-deb-arm64.tar.gz">arm64</a>
            - <a href="artifact://installables/scion-deb-i386.tar.gz">i386</a>
            - <a href="artifact://installables/scion-deb-armel.tar.gz">armel</a>

.buildkite/pipeline.yml line 58 at r9 (raw file):

      - scionproto/metahook#v0.3.0:
          post-artifact: |
            cat << EOF | buildkite-agent annotate --style "info" --context "packages"

Need to pick a different --context here and in the debian step, otherwise the last messages wins.

Suggestion:

           cat << EOF | buildkite-agent annotate --style "info" --context "packages-openwrt"

.buildkite/pipeline.yml line 60 at r9 (raw file):

            cat << EOF | buildkite-agent annotate --style "info" --context "packages"
            #### Packages :openwrt:
            - <a href="artifact://deb/scion-openwrt-x86_64.tar.gz">x86_64</a>

Suggestion:

            - <a href="artifact://installables/scion-openwrt-x86_64.tar.gz">x86_64</a>

dist/BUILD.bazel line 5 at r7 (raw file):

load(":platform.bzl", "multiplatform_filegroup")
load(":platform.bzl", "DEFAULT_DEB_PLATFORMS")
load(":platform.bzl", "DEFAULT_OPENWRT_PLATFORMS")

Maybe move these definitions here, and remove the "DEFAULT_" prefix?


dist/BUILD.bazel line 153 at r7 (raw file):

scion_pkg_ipk(
    name = "gateway_ipk",
    package = "scion-gateway",

Can we rename this to scion-ip-gateway ?


dist/openwrt/musl_toolchain.bzl line 4 at r9 (raw file):

# https://github.com/bazel-contrib/musl-toolchain
# Merit is his, mistakes are ours, License is
# in LICENSE.musl_toolchain.

It's identical to the LICENSE that we already have in the repo, I think we don't need to copy the license file.
Instead, we should add the copyright / license header , something like this:

# Copyright 2023  illicitonion / Daniel Wagner-Hall 
# Modifications Copyright 2024 SCION Association
# 
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# 
#   http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Adapted from illicitonion's unreleased work: https://github.com/bazel-contrib/musl-toolchain
# Merit is his, mistakes are ours,

dist/openwrt/initds/gateway line 21 at r9 (raw file):

    fi
    # Note that this service will not start unless the local certs
    # have been generated. There are no defaults for that.

Nit: gateway does not require "local certs", but it does need more config.


dist/openwrt/initds/router line 13 at r9 (raw file):


init_conf() {
    # These defaults exist only if the testconfig package was installed.

I don't know if it's a good idea to have "production" code that must do something special to make the test setup work. Couldn't we do this setup somewhere in the openwrt_test.sh script?
Same applies to the other init scripts.

Btw, generating the "master0/1.key" is a single line of bash (base64 encode some arbitrary 16 chars). Might really be much simpler to just do it in the test script.


dist/test/openwrt_test.sh line 27 at r9 (raw file):

cleanup

if [ "$DEBUG" == 0 ]; then  # if DEBUG: keep container debian-systemd running after test

Nit:

Suggestion:

if [ "$DEBUG" == 0 ]; then  # if DEBUG: keep container openwrt-x86_64 running after test

dist/test/openwrt_test.sh line 33 at r9 (raw file):

# Note: specify absolute path to Dockerfile because docker will not follow bazel's symlinks.
# Luckily we don't need anything else in this directory.
# docker build -t debian-systemd -f $(realpath dist/test/Dockerfile) dist/test

Does not apply here.


tools/bazel-build-env line 10 at r9 (raw file):

# to generate/update a file in-time for the load phase either.
#
# Instead, prefer loading "versioning.bzl", which is generated by tools/bazel just before executing

Outdated comment?


dist/openwrt/configs/crypto/as/README line 1 at r1 (raw file):

Previously, jiceatscion wrote…

I'm glad you like the README. It was actually born of lazyness. It makes the directories exist in the eyes of bazel so they could be copied recursively when packaging instead of having to somehow describe a tree layout for the packaging makefile to create. Silly bazel. can't mirror empty dirs to the sandbox. Thinking aloud: any sort of post-install that creates default configs when needed would also be easier to construct out of a template config tree rather than hand-crafted to create every little thing. If I remove the README, I loose that.

Right. Now that this is just used for the test configs, I'm sure if this is still useful. As far as I can tell, the directories don't need to be present for running the test.

Same applies to the other dir.

Copy link
Contributor Author

@jiceatscion jiceatscion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 11 unresolved discussions (waiting on @matzf)


.buildkite/pipeline.yml line 33 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Nit: maybe define version as a (shell-) variable, so it's a little less verbose, e.g.
(same below for openwrt)

done
Wasn't sure if the commands were executed in the same shell.


.buildkite/pipeline.yml line 35 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Might have to restrict this to installables/scion-deb*.tar.gz (and analogously scion-openwrt*.tar.gz below) -- otherwise we might upload the same thing twice. Not sure if this would break something, but at least it would be wasteful.

Done
Alternatively, I was wondering if we could handle both sets in the same phase. The only distro-specific thing in the code is the icon ":debian:" or ":openwrt:" in the step label. It's probably fine to use two icons in one label. No?


.buildkite/pipeline.yml line 44 at r9 (raw file):

            - <a href="artifact://deb/scion-deb-arm64.tar.gz">arm64</a>
            - <a href="artifact://deb/scion-deb-i386.tar.gz">i386</a>
            - <a href="artifact://deb/scion-deb-armel.tar.gz">armel</a>

Done.


.buildkite/pipeline.yml line 58 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Need to pick a different --context here and in the debian step, otherwise the last messages wins.

Done.


.buildkite/pipeline.yml line 60 at r9 (raw file):

            cat << EOF | buildkite-agent annotate --style "info" --context "packages"
            #### Packages :openwrt:
            - <a href="artifact://deb/scion-openwrt-x86_64.tar.gz">x86_64</a>

Done.


dist/BUILD.bazel line 5 at r7 (raw file):

Previously, matzf (Matthias Frei) wrote…

Maybe move these definitions here, and remove the "DEFAULT_" prefix?

Done.


dist/BUILD.bazel line 153 at r7 (raw file):

Previously, matzf (Matthias Frei) wrote…

Can we rename this to scion-ip-gateway ?

Package name, done. However changing the name of the binary requires a mapping. Should I?


dist/openwrt/musl_toolchain.bzl line 4 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

It's identical to the LICENSE that we already have in the repo, I think we don't need to copy the license file.
Instead, we should add the copyright / license header , something like this:

# Copyright 2023  illicitonion / Daniel Wagner-Hall 
# Modifications Copyright 2024 SCION Association
# 
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# 
#   http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Adapted from illicitonion's unreleased work: https://github.com/bazel-contrib/musl-toolchain
# Merit is his, mistakes are ours,

Done.


dist/openwrt/initds/gateway line 21 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Nit: gateway does not require "local certs", but it does need more config.

It doesn't consume them. But the gateway won't start if the CS has no certs. At least that's what I saw. I removed the comment. It wasn't helping.


dist/openwrt/initds/router line 13 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

I don't know if it's a good idea to have "production" code that must do something special to make the test setup work. Couldn't we do this setup somewhere in the openwrt_test.sh script?
Same applies to the other init scripts.

Btw, generating the "master0/1.key" is a single line of bash (base64 encode some arbitrary 16 chars). Might really be much simpler to just do it in the test script.

The script isn't doing something special to make testing work. It's doing something special to consume packaged config files safely if they happen to exist. Removed the useless comment.

Keys: Yes, generating the keys would be simple. I thought you did not want that to happen automatically.


dist/test/openwrt_test.sh line 27 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Nit:

Done


dist/test/openwrt_test.sh line 33 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Does not apply here.

Done


tools/bazel-build-env line 10 at r9 (raw file):

Previously, matzf (Matthias Frei) wrote…

Outdated comment?

Done.


dist/openwrt/configs/crypto/as/README line 1 at r1 (raw file):

Previously, matzf (Matthias Frei) wrote…

Right. Now that this is just used for the test configs, I'm sure if this is still useful. As far as I can tell, the directories don't need to be present for running the test.

Same applies to the other dir.

You're right the test itself doesn't need them (even my version). I included them because the CS seems happy to wait for files to appear when the dirs are empty but crashes if they're missing. Kinda pointless to make the CS crash on install for lack of a pair of empty dirs. replaced this with mkdir -p in init script. Does the job just as well. So...

Copy link
Contributor Author

@jiceatscion jiceatscion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 42 of 61 files reviewed, 11 unresolved discussions (waiting on @matzf)


.buildkite/pipeline.yml line 35 at r9 (raw file):

Previously, jiceatscion wrote…

Done
Alternatively, I was wondering if we could handle both sets in the same phase. The only distro-specific thing in the code is the icon ":debian:" or ":openwrt:" in the step label. It's probably fine to use two icons in one label. No?

Answer was yes. So.. Done

The special-purpose configs contain overrides that do not attempt to preserve existing configs.
Most config files are now overrides only. The few config files that we still include along with
the binaries (daemon, gateway and dispatcher) are installed as "default", and picked-up by initd
scipts as before.
Copy link
Contributor

@matzf matzf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 14 of 19 files at r10, 6 of 6 files at r11, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @jiceatscion)


dist/BUILD.bazel line 153 at r7 (raw file):

Previously, jiceatscion wrote…

Package name, done. However changing the name of the binary requires a mapping. Should I?

Passt schon, 👎

Copy link
Contributor

@matzf matzf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @jiceatscion)

@matzf matzf changed the title Build scion components as openwrt packages. dist: build openwrt packages Mar 5, 2024
Copy link
Contributor Author

@jiceatscion jiceatscion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @jiceatscion)


dist/BUILD.bazel line 153 at r7 (raw file):

Previously, matzf (Matthias Frei) wrote…

Passt schon, 👎

Thanks!

@jiceatscion jiceatscion enabled auto-merge (squash) March 5, 2024 10:13
@jiceatscion jiceatscion merged commit a561dbe into scionproto:master Mar 5, 2024
4 checks passed
@jiceatscion jiceatscion deleted the ipk_pkg branch March 5, 2024 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants