i#6662 public traces: update doc (#7139)

Updates the documentation on Public Google Workload Traces.
Gives a description of the new format: the #DR_ISA_REGDEPS
synthetic ISA that instructions follow, what TRACE_MARKER_TYPE_*
are present or have been modified, what is v2p.textproto.

Issue #6662

clients/drcachesim/docs/drcachesim.dox.in

@@ -998,60 +998,124 @@ $ bin64/drrun -t drmemtrace -indir newdir -tool basic_counts
 \endcode
 
 ****************************************************************************
-\page google_workload_traces Google Workload Traces
+\page google_workload_traces Google Workload Traces (Version 2)
 
 With the rapid growth of internet services and cloud computing,
 workloads on warehouse-scale computers (WSCs) have become an important
-segment of today’s computing market. These workloads differ from
+segment of today’s computing market.  These workloads differ from
 others in their requirements of on-demand scalability, elasticity and
-availability. They have fundamentally different characteristics from
+availability.  They have fundamentally different characteristics from
 traditional benchmarks and require changes to modern computer
 architecture to achieve optimal efficiency.  Google is sharing
 instruction and memory address traces from workloads running in Google
 data centers so that computer architecture researchers can study and
 develop new architecture ideas to improve the performance and
-efficiency of this important class of workloads.
+efficiency of this important class of workloads.  To protect Google's
+intellectual property, these traces have had their original ISA replaced
+with a synthetic ISA that we call #DR_ISA_REGDEPS.  This synthetic ISA
+removes architecture specific details (e.g., the opcode of instructions),
+while still providing enough information (e.g., register dependencies,
+instruction categories) to perform meaningful analyses and simulations.
 
-\section sec_google_format Trace Format
+\section sec_google_format Public Trace Format
 
 The Google workload traces are captured using DynamoRIO's
 [drmemtrace](@ref page_drcachesim).  The traces are records of
 instruction and memory accesses as described at \ref
-sec_drcachesim_format.  We separate instruction and memory access
-records from each software thread into a separate file
-(.memtrace.gz). In addition, for each software thread, we also provide
-a branch_trace which contains execution data (taken/not taken, branch
-target) about each branch instruction (conditional, non-conditional,
-calls, etc.).  Finally, for each workload trace, we provide a thread
-statistics file (.threadstats.csv) which contains the thread ID (tid),
-instruction count, non-fetched instruction count (e.g. implicit
-instructions generated from microcode), load count, store count, and
-prefetch count.
+sec_drcachesim_format. While memory accesses are left unchanged
+compared to the original trace, instructions follow the
+#DR_ISA_REGDEPS synthetic ISA.
+
+#DR_ISA_REGDEPS has the purpose of preserving register dependencies and giving
+hints on the type of operation an instruction performs.
+
+Being a synthetic ISA, some routines that work on instructions coming from an
+actual ISA (such as #DR_ISA_AMD64) are not supported (e.g., decode_sizeof()).
+We do support decode() and decode_from_copy(): to decode an encoded #DR_ISA_REGDEPS
+instruction into an #instr_t.
+
+A #DR_ISA_REGDEPS #instr_t contains the following information:
+- Categories: composed by #dr_instr_category_t values, they indicate the type of
+  operation performed (e.g., a load, a store, a floating point math operation, a
+  branch, etc.).  Note that categories are composable, hence more than one category
+  can be set.  This information can be obtained using instr_get_category().
+- Arithmetic flags: we don't distinguish between different flags, we only report if
+  at least one arithmetic flag was read (all arithmetic flags will be set to read)
+  and/or written (all arithmetic flags will be set to written).  This information
+  can be obtained using instr_get_arith_flags().
+- Number of source and destination operands: we only consider register operands.
+  This information can be obtained using instr_num_srcs() and instr_num_dsts().
+  Memory operands can be deduced by subsequent read and write records in the trace.
+- Source operation size: is the largest source operand the original ISA instruction
+  operated on.  This information can be obtained using instr_get_operation_size().
+- List of register operand identifiers: they are contained in #opnd_t lists,
+  separated in source and destination.  Note that these #reg_id_t identifiers are
+  virtual and it should not be assumed that they belong to any DR_REG_ enum value
+  of any specific architecture. These identifiers are meant for tracking register
+  dependencies with respect to other #DR_ISA_REGDEPS instructions only.  These
+  lists can be obtained by walking the #instr_t operands with instr_get_dst() and
+  instr_get_src().
+- ISA mode: is always #DR_ISA_REGDEPS.  This information can be obtained using
+  instr_get_isa_mode().
+- Encoding bytes: an array of bytes containing the #DR_ISA_REGDEPS #instr_t
+  encoding.  Note that this information is present only for decoded instructions
+  (i.e., #instr_t generated by decode() or decode_from_copy()).  This information
+  can be obtained using instr_get_raw_bits().
+- Length: the length of the encoded instruction in bytes.  Note that this
+  information is present only for decoded instructions (i.e., #instr_t generated by
+  decode() or decode_from_copy()).  This information can be obtained using instr_length().
+  Be aware that in Google Workload Traces the instruction fetch size of a
+  #dynamorio::drmemtrace::memref_t and the instr_length() of the corresponding fetched
+  instruction do not match! To allow analyses and simulations of front-end behavior to
+  have a realistic size of fetched instructions, we kept the instruction fetch size to
+  be the same as in the original ISA instruction.
+
+Note that all routines that operate on #instr_t and #opnd_t are also supported for
+#DR_ISA_REGDEPS instructions and their operands.  However, querying information outside
+of those described above will return the zeroed value set by instr_create() or
+instr_init() when the #instr_t was created.
+
+On top of instructions and memory acceses, traces also have
+#dynamorio::drmemtrace::trace_marker_type_t markers.
+All markers of the original trace are present, except for:
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_SYSCALL_IDX
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_SYSCALL
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_SYSCALL_TRACE_START
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_SYSCALL_TRACE_END
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_SYSCALL_FAILED
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_SIGNAL_NUMBER
+Which have been removed.
+Because tracing overhead results into inflated context switches, the
+#dynamorio::drmemtrace::TRACE_MARKER_TYPE_CPU_ID values have been modified to
+"unknown CPU" to avoid confusion. We recommend users to use our scheduler
+(see \ref sec_drcachesim_sched) for a realistic schedule of a trace's threads.
+Also, we preserved the following markers:
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_FUNC_ID
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_FUNC_ARG
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_FUNC_RETVAL
+- #dynamorio::drmemtrace::TRACE_MARKER_TYPE_FUNC_RETADDR
+But only for SYS_futex functions.
+
+Finally, every trace has a v2p.textproto file associated to it, which provides a
+plausible virtual to physical mapping of the virtual addresses present in a trace
+for more realistic TLB simulations.  This is a static virtual to physical mapping
+with 2 MB pages. Users can generate different mappings (e.g., smaller page size)
+by modifying this file, or create their own mapping following the same
+v2p.textproto format.
+
+These traces are supported starting from DynamoRIO 11.3.
 
 \section sec_google_get Getting the Traces
 
 The Google Workload Traces can be downloaded from:
 
- - [Google workload trace folder](https://console.cloud.google.com/storage/browser/external-traces)
+ - [Google workload trace folder](https://console.cloud.google.com/storage/browser/external-traces-v2)
 
-Directory convention:
+Directory structure:
 - \verbatim
-  workload/trace-X/
-  \endverbatim
-  where X is sequential starting from 1
-
-Filename convention:
-- Memory trace file:
-  \verbatim
-  <uuid>.<tid>.memtrace.gz
-  \endverbatim
-- Branch trace file:
-  \verbatim
-  <uuid>.branch_trace.<tid>.csv.gz
-  \endverbatim
-- Thread statistics summary:
-  \verbatim
-  <uuid>.threadstats.csv
+  workload_name/
+    <uuid>.<tid>.memtrace.zip
+    v2p.textproto
   \endverbatim
 
 \section sec_google_help Getting Help and Reporting Bugs
@@ -1069,7 +1133,7 @@ For general questions, or if you are not sure whether the problem you
 hit is a bug in your own code or in provided code, use the
 [DynamoRIO users group mailing list/discussion forum]
 (http://groups.google.com/group/dynamorio-users) rather than
-opening an issue in the tracker. The users list will reach a wider
+opening an issue in the tracker.  The users list will reach a wider
 audience of people who might have an answer, and it will reach other
 users who may find the information beneficial.
 
@@ -1085,7 +1149,31 @@ You can contribute to the project in many ways:
 
 - Providing suggestions for improving trace formats.
 - Sharing and collaborating on architecture research.
-- Reporting issues: see \ref sec_google_help
+- Reporting issues: see \ref sec_google_help .
+
+\section sec_google_cite Cite Google Workload Traces
+
+If you would like to cite this work, you can use the following BibTeX entry:
+
+\verbatim
+@misc{ Google_Workload_Traces_Version_2,
+  title = {{Google Workload Traces Version 2}},
+  howpublished = {\url{https://console.cloud.google.com/storage/browser/external-traces-v2}},
+  note = {Accessed: yyyy-mm-dd}
+}
+\endverbatim
+
+\section sec_public_v1_deprecated Deprecated Google Workload Traces (Version 1)
+
+The previous version of Google workload traces contains a subset of the
+information of the current traces and has been deprecated.
+Please use the current version described above.
+
+The previous version can still be found at:
+
+ - [Google workload trace folder (Version 1)](https://console.cloud.google.com/storage/browser/external-traces)
+
+DynamoRIO 11.0 is the latest version that supports these traces.
 
 ****************************************************************************
 \page sec_drcachesim_config_file Configuration File