Improved determination of what resource the job ran on

[rynge]

We should improve a Pegasus job's ability to figure out what resource it is running on. This is a two-step process:

1. The job should determine where it is running. This used to be easier based on hostnames and environments, but today where jobs can run in VMs and containers, we probably have to call an external service in those cases. For example, consider a job running in a k8s container with hostname avl43fkz4 (not enough information), it could call a new service which would return the resolved domain name of the public address the call would come from (let's say sdsc.edu).
2. The site would be stored in the Stampede database and in some cased forwarded to the Pegasus friends database.

The team discussed what the site name should be, and at least initially agreed that top level domain would be the most useful. For example, we would see entries like sdsc.edu and aws.com. There is an open question if we should have something more granular (like which resource at SDSC), and if it is even possible to determine that.

[vahi]

two avenues where this information is of interest

1. via the grafana dashboard. then it probably has to be piped via the job composite events
2. the stampede database. for a wf running on OSG then having a breakdown of what OSG sites the job ran on
3. the metrics server is not an option. since it is not aware of the jobs

For 5.1.0 we could do 1).

At the service end, the public IP is looked up against the DNS. initially the cardinality would be 1-1 lookup, and do the hostname lookup against the DNS.

the hostnames will be normalized to an institution name. so full hostnames will get mapped to site name

[vahi]

Need to make sure this reporting is turned off if user turns off metrics reporting by turning pegasus.metrics off

Also, need to update the documentation to reflect this new thing https://pegasus.isi.edu/documentation/reference-guide/funding-citing-usage-stats.html#usage-statistics-collection