Technical Documentation: microsoft/perfview

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md)

PerfView Overview

PerfView is a free performance-analysis tool that helps isolate CPU and memory-related performance issues. It is a Windows tool, but it also has some support for analyzing data collected on Linux machines. It works for a wide variety of scenarios, but has a number of special features for investigating performance issues in code written for the .NET runtime.

If you are unfamiliar with PerfView, there are PerfView video tutorials.
Also, Vance Morrison's blog gives overview and getting
started information.

Getting PerfView

Please see the PerfView Download Page for the link and instructions for downloading the
current version of PerfView.

PerfView requires .NET Framework 4.7.2 or later, which is widely available for all supported versions of Windows.

PerfView and TraceEvent require Windows 10 or Windows Server 2016 or later when running on Windows.

There is no native ARM64 PerfView application build. PerfViewCollect requires the matching
x64 .NET runtime, whose supported Windows releases may be more restrictive.
Historical trace files and older target runtimes remain supported for analysis.

Are you here about the TraceEvent Library?

PerfView is built on a library called Microsoft.Diagnostics.Tracing.TraceEvent, that knows how to both collect and parse Event Tracing for Windows (ETW) and EventPipe (.NET Core trace) data. Thus if there is any information that PerfView collects and processes that you would like to manipulate yourself programmatically, you would probably be interested in the TraceEvent Library Documentation

Not Sure if you should use PerfView or TraceEvent?

See the scenarios document to determine which is the best choice for what you're trying to do.

Learning about PerfView

The PerfView User's Guide is part of the application itself. In addition, you can click the
Users Guide link
to see the GitHub HTML Source File rendered in your browser. You can also simply
download PerfView using the instructions above and select the Help -> User's Guide menu item.

Asking Questions / Reporting Bugs

When you have question about PerfView, your first reaction should be to search the Users Guide (Help -> User's Guide) and
see if you can find the answer already. If that does not work you can ask a question by creating a new PerfView Issue.
State your question succinctly in the title, and if necessary give details in the body of the issue, there is an issue tag
called 'question' that you should use as well that marks your issue as a question rather than some bug report.
If the question is specific to a particular trace (*.ETL.ZIP file) you can drag that file onto the issue and it will be downloaded.
This allows those watching for issues to reproduce your environment and give much more detailed and useful answers.

Note that once you have your question answered, if the issue is likely to be common, you should strongly consider updating the
documentation to include the information. The documentation is pretty much just
one file https://github.com/Microsoft/perfview/blob/main/src/PerfView/SupportFiles/UsersGuide.htm.
You will need to clone the repository and create a pull request (see OpenSourceGitWorkflow
for instructions for setting up and creating a pull request.

Reporting bugs works pretty much the same way as asking a question. It is very likely that you will want to include the *.ETL.ZIP
file needed to reproduce the problem as well as any steps and the resulting undesirable behavior.

Building PerfView Yourself

If you just want to do a performance investigation, you don't need to build PerfView yourself.
Just use the one from the PerfView Download Page.
However if you want new features or just want to contribute to PerfView to make it better
(see issues for things people want)
you can do that by following the rest of these instructions.

Tools Needed to Build PerfView

The only tool you need to build PerfView is Visual Studio 2026. The Visual Studio Community Edition
can be downloaded for free and has everything you need to fetch PerfView from GitHub, build and test it. We expect you
to download Visual Studio Community Edition if you don't already have Visual Studio 2026.

In your installation of Visual Studio, you need to ensure you have the following workloads and components installed:

.NET desktop development workload with all default components.
Desktop development with C++ workload with all default components plus the latest Windows 10 SDK.
- The Windows 10 SDK is not enabled by default in this workload, so you will need to check the box for it to be installed.
MSVC v145 - VS 2026 C++ x64/x86 build tools (Latest) component.
MSVC v145 - VS 2026 C++ x64/x86 Spectre-mitigated libs (Latest) component.
- This can be found under the 'Individual Components' tab.

A .vsconfig file is included in the root of the repository that can be used to install the necessary components. When
opening the solution in Visual Studio, it will prompt you to install any components that it thinks are missing from your
installation. Alternatively, you can import the .vsconfig in the Visual Studio Installer.
PerfView intentionally uses the latest installed MSVC v145 toolset from Visual Studio 2026.

If you get any errors compiling the ETWClrProfiler projects, it is likely because you either don't have the Windows 10 SDK
installed, or you don't have the spectre-mitigated libs installed. Please refer to the troubleshooting section for more information.

Cloning the PerfView GitHub Repository.

The first step in getting started with the PerfView source code is to clone the PerfView GitHub repository.
If you are already familiar with how GIT, GitHub, and Visual Studio 2026 GIT support works, then you can skip this section.
However, if not, the Setting up a Local GitHub repository with Visual Studio 2026 document
will lead you through the basics of doing this. All it assumes is that you have Visual Studio 2026 installed.

How to Build and Debug PerfView

PerfView is developed in Visual Studio 2026 using features through C# 7.3.

The solution file is PerfView.sln. Opening this file in Visual Studio (or double clicking on it in
the Windows Explorer) and selecting Build -> Build Solution, will build it. You can also build from the
command line using the build.cmd file at the base of the repository. The .NET SDK selected by global.json
or a compatible later patch must be installed on the machine for both command-line and Visual Studio
builds. If no compatible SDK is installed, build.cmd reports an error and provides a
winget install command using the version selected by global.json. build.cmd builds Debug by default; use
build.cmd -Configuration Release to build Release.
The build follows standard Visual Studio conventions, and the resulting PerfView.exe file ends up in
src/PerfView/bin/BuildType/net462/PerfView.exe. You need only deploy this one EXE to use it.
The solution consists of several projects, representing support DLLs and the main EXE. To run PerfView in the
debugger you need to make sure that the 'Startup Project' is set to the 'PerfView' project so that it launches
the main EXE. If the PerfView project in the Solution Explorer (on the right) is not bold, right click on the PerfView project
and select 'Set as Startup Project'. After doing this 'Start Debugging' (F5) should work.

Deploying your new version of Perfview

You will want to deploy the 'Release' rather than the 'Debug' version of PerfView. Thus, first set your build configuration
to 'Release' (Text window in the top toolbar, or right click on the .SLN file -> Configuration Manager -> Active Solution Configuration).
Next build (Build -> Build Solution (Ctrl-Shift-B)). The result will be that in the src\PerfView\bin\Release\net462 directory there will be
among other things a PerfView.exe. This one file is all you need to deploy. Simply copy it to where you wish to deploy the app.

Information for build troubleshooting.

One of the unusual things about PerfView is that it incorporates its support DLLs into the EXE itself, and these get
unpacked on first launch. This means that there are tricky dependencies in the build that are not typical. You will
see errors that certain DLLs can't be found if there were build problems earlier in the build. Typically you can fix
this simply by doing a normal (non-clean) build, since the missing file will be present from the last compilation.
If this does not fix things, see if the DLL being looked for actually exists (if it does, then rebuilding should fix it).
It can make sense to go down the projects one by one and build them individually to see which one fails 'first'.
Another unusual thing about PerfView is that it includes an extension mechanism complete with samples.
This extensions mechanism is the 'Global' project (called that because it is the Global Extension whose commands don't have an
explicit 'scope') and needs to refer to PerfView to resolve some of its references. Thus you will get many 'not found'
issues in the 'Global' project. These can be ignored until you get every other part of the build working.
One of the invariants of the repo is that if you are running Visual Studio 2026 and you simply sync and build the
PerfView.sln file, it is supposed to 'just work'. If that does not happen, and the advice above does not help, then
we need to either fix the repo or update the advice above. Thus it is reasonable to open a GitHub issue. If you
do this, the goal is to fix the problem, which means you have to put enough information into the issue to do that.
This includes exactly what you tried, and what the error messages were.
You can also build PerfView from the command line (but you still need Visual Studio 2026 installed). It is a two step process.
First you must restore all the needed nuget packages, then you do the build itself. To do this:
1. Open a developer command prompt. You can do this by hitting the windows key (by the space bar) and type
  'Developer command prompt'. You should see a entry for this that you can select (if Visual Studio 2026 is installed).
2. Change directory to the base of your PerfView source tree (where PerfView.sln lives).
3. Restore the nuget packages by typing the command 'msbuild /t:restore'
4. Build perfView by typing the command 'msbuild'
If you get an error "MSB8036: The Windows SDK version 10.0.17763.0 was not found", Or you get a 'assert.h' not found error, or
frankly any error associated with building the ETWClrProfiler dlls, you should make sure that you have the Windows 10 SDK installed. Unfortunately this library tends not to be
installed with Visual Studio anymore unless you ask for it explicitly. To fix it launch the Visual Studio Installer, modify the installation, and then look under the C++ Desktop Development and check that the Windows SDK 10.0.17763.0 option is selected. If not, select it and continue. Then try building PerfView again.
If you get an error "MSB8040: Spectre-mitigated libraries are required for this project", modify your Visual Studio
installation to ensure that you have the 'MSVC v145 - VS 2026 C++ x64/x86 Spectre-mitigated libs (Latest)' component installed.
If you get an error that the v145 platform toolset is not found, import the repository .vsconfig file in the Visual Studio Installer or install the Visual Studio 2026 C++ desktop workload.

Running Tests

PerfView has a number of *.Test projects that have automated tests. They can be run in Visual Studio by selecting the
Test -> Run -> All Tests menu item. For the most thorough results (and certainly if you intend to submit changes) you
need to run these tests with a Debug build of the product (see the text window in the top toolbar, it says 'Debug' or 'Release').
If tests fail you can right click on the failed test and select the 'Debug' context menu item to run the test under
the debugger to figure out what went wrong.

Check in testing and code coverage statistica

This repository uses Azure DevOps to automatically build and test pull requests, which allows
the community to easily view build results. The build and status reflected here is the Azure DevOps build status of the main branch.

:warning: Builds produced by Azure DevOps CI are not considered official builds of PerfView, and are not signed or otherwise
validated for safety or security in any way. This build integration is provided as a convenience for community
participants, but is not endorsed by Microsoft nor is it considered an official release channel in any way. For
information about official builds, see the PerfView Download Page page.

Contributing to PerfView

You can get a lot of value out of the source code base simply by being able to build the code yourself, debug
through it or make a local, specialized feature, but the real power of open source software happens when
you contribute back to the shared code base and thus help the community as a whole. While we encourage this it
requires significantly more effort on your part. If you are interested in stepping up, see the
PerfView Contribution Guide and PerfView Coding Standards before you start.

Code Organization

The code is broken into several main sections:

PerfView - GUI part of the application
- StackViewer - GUI code for any view with the 'stacks' suffix
- EventViewer - GUI code for the 'events' view window
- Dialogs - GUI code for a variety of small dialog boxes (although the CollectingDialog is reasonably complex)
- Memory - Contains code for memory investigations, in particular it defines 'Graph' and 'MemoryGraph' which are used
  to display node-arc graphs (e.g. GC heaps)
TraceEvent - Library that understands how to decode Event Tracing for Windows (ETW) which is used to actually
collect the data for many investigations
MainWindow - GUI code for the window that is initially launched (lets you select files or collect new data)
ETWClrProfiler* - There are two projects that build the same source either 32 or 64 bit. This is (the only) native code
project in PerfView, and implements the CLR Profiler API and emits ETW events. It is used to trace object allocation
stacks and .NET method calls.
HeapDump* There are 32 and 64 bit versions of this project. These make standalone executables that can dump the GC
heap using Microsoft.Diagnostics.Runtime APIs. This allows getting heap dumps from debugger process dumps.
Global - An example of using PerfView's extensibility mechanism
CSVReader - old code that lets PerfView read .ETL.CSV files generated by XPERF (probably will delete)
Zip - a clone of System.IO.Compression.dll so that PerfView can run on pre v4.5 runtimes (probably will delete)

Documentation CodingStandards (`documentation/CodingStandards.md`)

Coding Standard in the PerfView Codebase

If you are going to contribute to a codebase, you need to 'follow suit'
and conform to the standards that are already in place. Here is what
PerfView uses.

Indenting and other spacing conventions.

The PerfView codebase was developed using Visual Studio, and uses
indenting and spacing standards that are the default in Visual Studio.
You can use Ctrl-K Ctrl-D (reformat) to make your code conform to
this.

Layout of a Class

Items in a class should be ordered and structured to make reading the
as a public contract easy. In particular:

All private items come AFTER all public ones and are surrounded
by a '#region private' grouping. This makes Visual Studio's
outlining feature (Ctrl-M Ctrl-O) collapse things so that you
only see the public contract for the class.
Public methods should be ordered so that constructors or other
'generators' are first, then properties, then methods.
To the degree possible the most important/common methods should
come first in the class, and methods that are used together
should be near each other.
Fields should be private and placed TOGETHER, LAST in the class
(in the #region private). That makes it relatively easy for
developers to find all the state in an object (since that is what
really defines its semantics).

Naming conventions

We follow standard .NET Naming conventions (PascalCase for types,
methods, and properties; camelCase for parameters and local variables)
Private instance field names begin with a m_ (member). It is also
acceptable to use the class library convention of omitting the m (thus prefix
instance fields begin with _) If the field is static the prefix is s_.
Embedding the type in the variable (Hungarian notation) is NOT used.
Pick descriptive names. Visual Studio makes it easy to rename a
variable so fix the name if it 'morphed' as the code was written.

Minimum Commenting

PerfView is probably commented more than most code bases. We wish
to keep it that way. Here is what is expected.

If the type is public (outside the assembly) it needs a comment
and all its public members need comments.
Comments before declarations must follow the XML commenting conventions using three slashes to indicate this.
However, typically it is not necessary to explain each parameter
to a method (since you gave them really good names, right?).
You can simply use the summary tag and omit the rest (but put
the important information in the summary tag).
Field variables of a class typically DO need commenting. This is
especially true if there is some condition (invariant) that is maintained
for that variable. These are VERY valuable to the document.

When in Doubt

When in doubt, make your code look like the code around it. You can't
go too far wrong if you do that.

Documentation Downloading (`documentation/Downloading.md`)

Downloading PerfView

PerfView is a free profiling tool from Microsoft. This page tells you how to get a copy of it for yourself.
See the PerfView Overview for general information
about PerfView.

PerfView is an x64 application requiring Windows 10 or Windows Server 2016 or later
and .NET Framework 4.7.2 or later. It can collect from x86 and x64 target processes. It can also be used to capture traces on arm64 hardware.

PerfView Releases

The PerfView GitHub Releases page is now the official
way to download versions of the PerfView.
It shows the release notes and the full set of binaries for all current and past releases. If you
care about specific bug fixes or features go there.

Shortcut to Download the Latest PerfView.exe

In the common case, you only need one file, PerfView.exe, to use the tool. The most recent copy of
this file can be downloaded directly from this link.

Once you click the above link in your browser it will start downloading, the details of which vary from browser to browser.
In some cases it will prompt for more information (IE) and in others (Chrome) it may not be obvious that
you clicked on anything (look at the bottom of the pane for changes). The result, however will be a PerfView.exe on your
local machine.

Once downloaded you you can simply double click on the downloaded EXE to launch PerfView.
While Github itself and your browser do some validation, to be extra careful you can
also explicitly validate the digital signature of the downloaded file before running it.

You can also script download PerfView.exe using curl:

text

curl -L -o perfview.exe https://aka.ms/perfview/latest

Explicitly Validating PerfView's Digital Signature

Like all official Microsoft software, PerfView.exe is digitally signed
by Microsoft so you can have confidence that this software came from Microsoft
and has not been tampered with since the time it was created by Microsoft.
You can confirm this by opening the directory where you downloaded PerfView.exe and selecting
the Digital Signatures page for perfView by

Selecting PerfView -> right click -> Properties -> Digital Signatures.

You will see a sha256 signature from the 'Microsoft Corporation'.
This confirms that software is authentic. You can also do this by running the signtool command

text

signtool verify /pa /all  PerfView.exe

Additional Release Information

The aka.ms/perfview/latest shortcut allows you to quickly download
PerfView.exe which is what you need 95% of the time.
If you wish to get other files associated with the release (e.g. debugging symbols),
or wish to download older releases (not recommended), you can do
so by visiting the PerfView GitHub Releases.

Microsoft Download Site

The Microsoft Download Site for PerfView has
been retired as the official download site for PerfView. It has been replaced by
PerfView GitHub Releases page referenced above.

Documentation OpenSourceGitWorkflow (`documentation/OpenSourceGitWorkflow.md`)

Setting Up With a Fork with Visual Studio

See also Setting up a Repo in VS for important background material.
See also Simple GitHub Setup and Workflow for a simpler setup without a fork,
if you only need read-only access. Note that this information is also relevant to this (more complex)
workflow.

As mentioned, the first option is great if you only want read-only access or you have read-write access to the GitHub repo.
If you want to be able to submit pull requests, you need to create a fork (which you will have read-write access to)
and use that for your updates. To do this do the following:

Open a web browser to the GitHub project page (in our case https://github.com/Microsoft/perfview)
Make sure you have a GitHub account and that you are logged in (If you are logged into GitHub your username
in the very upper right corner of the display will be an icon for your user identity). See https://github.com
for information on getting an account. The instructions will tell you to download GIT tools. You can
do this if you like, however Visual Studio has all the GIT functionality you will need so you don't
have to have them (but it does not hurt, and they are useful for more advanced scenarios).
Click on the 'Fork' button in the upper right corner. If it asks you where to put the fork, you will wish
to put it in your account area (same as your user name). Note that if the fork already exists it simply
navigates to the existing fork, so this is also a way of finding a fork if needed.
Once that has completed (it does not take long), it will take you to a page that looks VERY similar to
the original but is in fact a clone. It should have a URL like https://github.com/vancem/perfview that has
your user name in it.
At this point you can follow the instructions above for setting up a local clone. The only difference is
that the GitHub URL will be the URL of the fork and not the main repository. You may also wish to give
it a local name that makes it clear that it is a fork (for example the local directory name for my fork is perfview-vancem).
Otherwise keeping track of which were direct clones (option 1) and which were not (option2) becomes more difficult.

Creating a 'upstream' Repository alias.

At this point you should have a local clone of your personal read-write fork of the main repository. However, this
is a FULLY INDEPENDENT clone which will NEVER SEE ANY UPDATES from the main repository unless you explicitly integrate
(merge) them. To make it easy to update, it is good to create an alias for the original main repository. Here
we show you how to do that in Visual Studio.

Click on the View -> Team Explorer menu item
Click on the top bolded header (may say 'Home' or 'Changes', ...) and right click on 'Settings'
Click on the 'Repository Setting' hyperlink
One of the sub-items in Repository setting page is 'Remotes' and under that is an 'Add' hyperlink. Click on that.
This will bring up a dialog box where you are prompted for a Name and a 'Fetch' and 'Push' location.
There is also a checkbox (which should be checked) that says the Fetch and Push are the same.
Fill in the name as 'upstream' and fill the 'Fetch' textbox with the URL OF THE MAIN REPOSITORY (in our
example this is https://github.com/Microsoft/perfview. It should automatically fill in the 'Push' dialog.
Click the 'save' button. This new name should now show up in the 'remotes' list.

What this does is name the original main repository 'upstream'. This makes it easier to refer to it in later commands
(like updating your fork with updates from the main repository). You will see that your GitHub repo already has
a short name called 'origin' (this is very standard name to use for your 'cloud' repo that your local repo works with).
'upstream' is a traditional GIT name for the repository that a fork was created from.

Pulling updates from the Main (upstream) Repository into your Fork

Sooner or later, the main repo will have updates that you will want to pull into your fork. This
section tells you how to do this.

If you go to your fork's GitHub web page (e.g. one that has your user name in it like https://github.com/vancem/perfview)
you will see that there is a line just before the description of files in the repo like

This branch is even with Microsoft:main.

Or it may say

This branch is 4 commits behind Microsoft:main.

You can see that this item is an indication of the main branch of your personal fork being 'in sync' with the main branch
of the main repo (which we have named upstream). From time to time you will want to pull in
all these changes to bring your fork up to date. We do this in three steps

Fetching all changes from the upstream repository. This step makes all the changes (commits)
in the fetched repository AVAILABLE for merging, but does not actually do any merging of branches.
Merge the changes from the upstream/main branch into the LOCAL repository's main branch.
Push the changes from the LOCAL repository's main branch into the main branch of the GitHub fork (called origin).

Step 1: Fetching All Commits for the Upstream Repository.

Click on the View -> Team Explorer menu item
Click on the bolded header (may say 'Home' or 'Changes', ...) and right click on 'Sync' option. This will bring
up the synchronization page, and one of the options will be 'Fetch'. Click on that. This brings up a dialog filled
out with 'origin' which says it will pull all changes from the 'origin' repository (which is your GitHub fork).
However we don't want that since we are only interested in the changes from the original main (which we named upstream).
You should be able to type a down arrow and 'upstream' should be one of the choices (since we added 'upstream' as a
known remote). Select 'upstream' and click the 'Fetch' button. That will fetch all of the changes from 'upstream',
and put them in the LOCAL GitHub repository. So far we have made our local repository bigger (more commits) but we have
not changed any existing branch (the transitive closure of any branch is the same as it was before).

Step 2: Merging the LOCAL repository's main branch to include changes from upstream/main

Click on the View -> Team Explorer menu item
Click on the top bolded header (may say 'Home' or 'Changes', ...) and right click on 'Branches' item which show you
all branches associated with the LOCAL repository (that is the repository clone on your local disk). There will
be at least one branch called 'main' as well as branches under 'remotes/origin' (these are branches in the
private fork on GitHub) and 'remotes/upstream' (these are branches in the original main repository).
Make sure that the 'active branch' (the one in bold) is 'main'. If it is not, double click on main to make it
the active branch. If it fails (and it could easily fail because you have modified files), you will have to
commit or undo those changes before proceeding. Ultimately you want 'main' to be the active branch.
Right Click on the 'main' branch, right click on it and select 'Merge from'. Then in the 'merge from' textbox select
upstream/main and click the 'Merge' button. This says we want to integrate (merge) all changes from the

upstream/main branch into the currently active branch (which is main).

In most cases this merge operation goes without a hitch and Visual Studio will auto-commit the merge. However if there
is a merge conflict with changes that you have made to 'main' (you really should never do that, put changes in some other
branch (see below)), then you will need to resolve them and commit the merge explicitly.

Step 3: Pushing the LOCAL repository to your GitHub Fork

Click on the View -> Team Explorer menu item
Click on the bolded header (may say 'Home' or 'Changes', ...) and right click on 'Sync' option. This will bring
up the synchronization page, and one of the options will be 'Sync'. Click on that. This will push all of your
changes to 'origin' (which is your GitHub fork) as well as pull down any changes from origin (there should be
none as you are the only one modifying this fork).

At this point you should be able to go to your GitHub web page for the fork and see a line like this

This branch is even with Microsoft:main.

Which says that you have integrated all the changes from the original main repository into your private fork.
Congratulations, you have successfully brought your private fork up to date with respect to its upstream main.

Creating a pull request to the Main Repository.

Creating a pull request starts by making the modification in the local repository in its own branch.
Thus you have to follow the instructions in Simple GitHub Setup and Workflow for creating
a new branch, committing changes to the new branch, and pushing (syncing) those changes to your personal GitHub
fork. Let's assume we did this using the branch name 'TestPullRequest' as the branch name.

One key aspect to syncing is that you must make sure that the new branch name is 'published' to the GitHub fork.
This publishing step can be done on the View -> Team Explorer -> Synchronization pane. You should see at the
bottom a hyperlink that indicates that you wish to publish the branch 'TestPullRequest' You will want to click
that hyperlink. This ensures that the name (and not just the (anonymous) commit) 'TestPullRequest' is copied into
the GitHub fork. Otherwise you will not find the new branch in the steps below.

Once the branch exists in the fork, you can go to the GitHub web page for the fork (in our example this is
https://github.com/vancem/perfview)

Near the top on the left hand side of the web page, you will find a dropdown menu which selects the 'active'
branch. It is likely to be set to the 'main' branch. Use the dropdown menu to change it to the 'TestPullRequest'
branch. Then click on the 'New Pull Request' button that is beside it.

At this point the page updates to a web page for the pull request. It prompts you to enter a description
of the pull request (but populates it with the commit message for the branch)). You will typically want to change
this to describe the pull request as a whole as well as a justification why the request should be accepted (its
advantages, etc). It will show you the delta associated with the pull request, which you can look over. When
you are happy with it, you can complete the submission of the pull request.

At that point it is out of your hands. The pull request will send mail to the maintainers of the repository
and they may comment on your change and may require changes. If you need to make changes, simply make the
changes in the branch (in our case TestPullRequest), first locally, then sync to push it to the GitHub fork.
Doing so will automatically update the pull request. Eventually the maintainers will either reject your request
(by closing it with an explanation), or accept it (in which case it will be merged into the main repository).
You can then use the 'Pulling updates' procedure above to pull changes into your GitHub fork, and from there
you can synchronize to bring it to your local repository.

Once that change is in the main repository, the branch (in our example TestPullRequest) is no longer needed.
You can go to the View -> Team Explorers -> Branches pane right click on it and select 'delete' to delete it.
The active branch must be on some other branch for the delete to succeed.

Documentation Scenarios (`documentation/Scenarios.md`)

PerfView vs. TraceEvent

Not sure which you should use? This document aims to point you in the right direction.

Start With PerfView If Your Goal Is To...

Collect an adhoc Event Tracing for Windows (ETW) trace to analyze program behavior or a performance issue.
Collect an adhoc heap snapshot to analyze a managed memory issue such as a managed memory leak.
Use the flight recorder mode to capture an ETW trace of hard to reproduce behavior.
Perform adhoc analysis of a previously collected performance trace.
Diff two performance traces to or managed memory heap snapshots to root cause a performance issue.
Use a GUI-based performance analysis tool.

Start With TraceEvent If Your Goal Is To...

Have programmatic access to trace collection and/or trace processing and analysis.
Implement a service that captures or processes traces at scale.
Build collection and/or processing capabilities into an existing application.

PerfView Limitations

PerfView is not designed to be used as a capture or processing agent in services. It is designed for use in user-interactive sessions.
PerfView is not supported on operating system SKUs such as nanoserver that do not have GUI libraries installed. See PerfViewCollect if you need to capture adhoc traces on these SKUs.

Documentation SettingUpRepoInVS (`documentation/SettingUpRepoInVS.md`)

GitHub Repository Setup with Visual Studio 2026

This section tells you how to build a project that is already hosted on GitHub,
using Visual Studio 2026. If you don't already have Visual Studio 2026, you
can get the community edition for free from
this link.
This section also goes through important routine tasks like getting the latest
changes from GitHub and submitting a pull request to the main branch.

It will show you how to do this using just Visual Studio 2026. Older versions
of Visual Studio as well as other IDEs are possible, but not covered here.
You can also use 'raw' git commands but I don't cover that here.

We use the PerfView project https://github.com/Microsoft/perfview as an example but
the vast majority of this section applies to any open source GitHub hosted project.

The step-by-step instructions will not make a whole lot of sense, however, without
a certain amount of background knowledge and a basic understanding of what GIT
and GitHub do along with some of the basic concepts associated with the GIT source code
control system. This is where we start.

Background: GitHub Repos, Commits Branches and GitHub

The set of all files versioned as a unit by the GIT source code control system is
called a repository, or repo for short. Logically, a GIT repository contains a
collection of snapshots called commits which represents a set of files at one
point in time. The data in any such snapshot is hashed (including all its meta
data like when the commit was made), and this hash is the ID for the commit.
In theory the ID is very long (64+ bytes), but in practice all
GIT commands will take an unambiguous prefix. Typically 8 Hex digits are more than
sufficient so you will see instructions talking about commit 534aab38 or commit 3abcd333.

One of the very nice properties of naming commits using a hash of the data is
that combining two repos into a single one is trivial. If we assume that the
hashing GIT does is 'perfect' (and it really is close to perfect) if two commits
have the same ID (hash) then they have to have been the same data, which means
they really are the same commit. Thus combining two repos is simply a matter of making
a union of all the commits in the two component repos.

This ability to easily combine two repositories makes GIT well suited for 'distributed'
source code control, where there is no 'main' repository that all 'clients' work
from. Instead each repository can be thought of as the 'main', and a repository
like GitHub is only special in the sense that it is a well known publishing point.

Indeed, when working with GitHub you NEVER have just one repository. At the minimum
there are two repositories you need to know about. First is the repository hosted
on GitHub, and the other is a local repository on the machine where you make your
changes. This local repository is a COMPLETE CLONE of the GitHub one (which includes
all commits and thus all history). Thus after cloning, you create many local
changes (commits) without every communicating with GitHub. Only when you want
to publish (calling pushing) or get updates (called pulling) do you need connectivity to GitHub.

Branches

A repository is a collection of MANY snapshots (commits), and most commits have a
predecessor (the snapshot before the change). Some commits have multiple natural
predecessors (as when changes from multiple sources are merged). Thus there is
the concept of 'history' of a commit where you can see the set of commits, that
over time lead to a particular place. Thus a commit can be thought of not just
as one snapshot, but in fact the complete history that is 'reachable' by following
predecessor links. GIT has a concept of a pointer to a commit (history) called
a branch. Most GIT commands have the concept of a 'current branch', and
when a repo is created, typically it creates a branch called 'main' which typically
is used to represent the 'main' or maximally shared history of code (you may
have other branches that represent releases or other specialized versions of the
code base)

While it is possible to commit your changes directly to the 'main' branch,
branches in GIT are very cheap and GIT encourages you to make a new branch (derived
from 'main') whenever independent work is being done. These independent branches
can then be merged into 'main' (or used as pull requests) independently
of each other.

Two Possible Repository Setups

This section tells you about two possible ways to use a GitHub repository, and why
you should choose one or the other.

Simple GIT Repository Setup

If one of the following two conditions hold

You have Read/Write permission to the GitHub repository of interest.
You only have read access, but you never want to 'push' changes back to the repository.

You can use a simple GIT setup with only two repositories (the one on GitHub and the local
repository on your development machine). In this setup you can pull new commits
from the GitHub repository to your local one to keep your local copy up to date.
You can even make local changes for your own personal experimentation in your local repository.
However you can only 'push' those changes back to GitHub if you have write permission to
the repo. This is fine for personal projects, but not good enough for open source projects
because there is no way for people without write access to contribute updates.

Open Source Projects and Pull Requests

In an open source project, we want the ability for ANYONE to PROPOSE a change
to the main repository, but we need an APPROVAL process so that only those with
special permissions (the maintainers) can actually update the main repository.
GitHub accomplishes this with a process known as the pull request. A pull request
is a GitHub procedure where any user can indicate that he would like a particular change
(commit) integrated to the GitHub repository. People who have write permission
to the repository can look at the change and determine if it meets
the standards of the repository and if so merge (pull) the change into it. In this way
arbitrary users can contribute fixes to a repository in a controlled way.

There is a logistic problem with pull requests on where to put the tentative (proposed)
changes. We need a commit (set of file changes) that anyone can create, but can
easily be integrated into the main GIT repository. This commit can't be created in the
main repository (because it was created by a user without write permissions to the main repository). It also
can't be in a user's local repository because that is on the user's machine and
not available to GitHub maintainers. Thus we need a repository that is in a public place
(that is on GitHub), but is writeable by a particular user. GitHub solves this problem
by making a 3rd repository which is typically called fork.

Thus the typical flow for an open source project is

Create a Fork (copy of the repo) of the open source repository (e.g. https://github.com/Microsoft/perfview)
that is writable by you. There is a button on the project's GitHub page (upper right corner called 'fork') that does
this. Every GitHub user has an 'area' assigned to their GitHub user name that can host these writable forks. There may
be other locations (shared by a group) that you may put the fork, so when clicking the 'Fork' button it may prompt
you with choices of where to put it. Typically you want to use the area associated with just your user. For example, my
GitHub user name is vancem, so the fork created when I click on the 'fork' button of https://github.com/Microsoft/perfview creates
the fork called https://github.com/vancem/perfview (notice it in the 'vancem' area of GitHub which is writable by me).
If the fork already exists, it simply takes me there (so you can find your existing forks easily).
Clone your GitHub fork locally on your machine. This works just like if you had cloned the
https://github.com/Microsoft/perfview directly, but with the significant difference that you can write
to this repository, so you can submit changes.
Prepare a change by making a new branch in your GitHub fork with the changes you want. This actually
involves several sub-steps
1. Create a new branch representing your proposed change in your local repository which is a clone of your GitHub Fork.
2. Create one or more commits in this branch that embody the change you want.
3. Push your changes to your GitHub Fork.
Once you have a branch you are happy within your own fork, there is a button on YOUR FORKs GitHub page that
allows you to submit that branch as a pull request. When you do this you write a description / rationale for The
change where you are persuading the maintainers to accept your request. There is an area for discussion, and typically
the maintainers ask for changes. You update your branch as needed, and the pull request automatically gets updated
to the latest revision of the branch. Hopefully after enough discussion and updates, the maintainers accept
the pull request and merge your branch from your GitHub fork into the main repository.
The next time you update your fork from the main repository, your main branch will have your pull request reflected in
the branch.

To Fork or not to Fork?

So there are two different ways you can set up your clone of the repository

You can simply clone the repository locally.
You can create a fork, and then clone your fork locally and work with that.

The basic answer is that (2) can do pull requests (and generally forces all updates to be pull requests),
however it is also more cumbersome (syncing to the latest code is harder and updates have the overhead of the
pull request procedure). So the simple answer is

Use (1) when you have read-write access and want a low overhead checkin process (private projects typically fall into this bucket).
Use (1) if you have read-only access and don't need to submit fixes/changes (thus you don't need pull requests)

A typical scenario is that start with the direct (unforked) option up until the point where you want to start modifying the
code base at which point you switch to the forked option.

Once you have decided on an option, see one of the following.

Documentation SimpleGitWorkflow (`documentation/SimpleGitWorkflow.md`)

Setting Up Without a Fork with Visual Studio 2026

See also Setting up a Repo in VS for important background material.
See also Open Source GitHub Setup and Workflow for the setup needed for pull requests.

Here we describe how to use Visual Studio 2026 to set a local build of the GitHub project https://github.com/Microsoft/perfview.
Because this project is open source, anyone can read it so this works even if you don't have a GitHub account, but
of course you will only have read-only access.

Open Visual Studio and select the View -> Team Explorer menu item
This brings up the Team Explorer pane. At the top of the display is a bold word (probably Home) which
indicates which Team Explorer view is being displayed. If you left click
on this bolded item you will get a menu of possible displays. Select Projects -> Manage Connections
At the bottom of this display you should see a section called 'Local GIT Repositories' and under this a
set of hyperlinks. Click on the 'Clone' hyperlink.
It will bring up a dialog box where you enter two pieces of information
a. The name of the source to clone. Enter the GitHub URL (in our example https://github.com/Microsoft/perfview)
b. The name of the local directory to place the clone (The default is probably OK, but you can decide to place it somewhere else)
Click the Clone button.

At this point Visual Studio will create a local clone of all the files in a project in the directory you specified. This represents
the final state of all files in the default branch (probably 'main'). It also clones the repository (thus the history for all time)
and places it in a hidden .git directory in the directory you specified. It is this cloned local repository where most GIT
commands you enact will operate. By default your clone ended up in %HOMEPATH%\Source\Repos\PerfView.

Once the clone is complete, you will see the new local repository added to the list under 'Local Repositories'.
You can see which one is 'active' because it is bold-faced. This is the repository that all other VS GIT commands will
operate on. Double click on the new entry (perfView in this example) to make it the active GIT repo.

This will bring you to the 'Home' display for a local GIT repository. You will see buttons for 'changes', 'branches', 'sync', etc.
At the bottom you will see a list of .SLN files. There are all of the Visual Studio solution files that were found in the clone
of the repository. This makes it convenient to open VS to do a build. In the case of PerfView, you want the PerfView.sln file.
After opening the solution file you can start building (Build -> Build Solution or Ctrl-Shift-B).

Making changes

GIT is a bit unusual in that there is no 'checkout' command. You can modify anything in the local copy of the files
at will. You can think of it as modifying a file that will cause an automatic checkout, but this works no matter how the file
was changed (thus it is more like every git command that starts by looking at files to see what the checked out set is in the repo).

Committing Trivial Changes to Master Branch

As mentioned, GIT has the concept of the 'active branch' which is the branch the 'commit' operation uses to stamp the
commit (snapshot) with the predecessor commit (snapshot). Thus the active branch is what determines the 'history' of
a commit and is very important, you want this to be accurate. After setting up a local repository, the active Branch
is almost certainly one called 'main' (but you can check by going to the View -> Team Explorer -> Branches).

Master represents the 'latest', 'default' version of the code in your repository. Thus after making updates to
particular files, simply commit them to main.

When you have a set of files you wish to commit (check in), you can either right click on the solution in the 'Solution Explorer'
and select 'Commit', or use the View -> Team Explorer menu item to get the Team explorer pane and left click on the top
bolded item and select 'Changes'. This will bring up the 'Changes' view of the Team Explorer pane that lets you commit. This
pane will show you all the files that have changed, (you can right click on each and select Compare with Unmodified to diff).
There is also a text box at the top in which you should put a message.

By convention, you should begin this comment with ONE LINE SUMMARIZES THE CHANGE AS A WHOLE. This one line is what shows up in 'short'
descriptions of the commit. After that line, you can write a longer explanation if desired.

At the top of the pane is the active branch. It is likely to say 'main', but whatever it is, this is the branch that
you are committing to.

After you have filled out the description, you can simply hit the 'Commit All' button to commit your changes.

Don't use Master Branch for Non-trivial Features.

Committing directly to the main branch is fine for small, independent edits, however it is
a bad idea for any feature that might involve multiple commits. This includes any pull
request because most pull requests are likely to have commits to respond to maintainer
feedback. (See Open Source GitHub Setup and Workflow
for more on pull requests). Instead you should give commits that are the first of a multi-part
edit their own branch.

By giving the multi-commit edit its own branch, the commits can be kept together, and this makes for significantly more
agile history (you can take or leave this feature more easily). It also makes the history significantly easier to read.

This is not hard to do, because you don't need to make the choice of whether to do the commit on
the main branch directly, or creating its own branch until you actually are trying to commit.
It is only at commit time that you need to make the choice.

To make a new branch for a feature, follow the same procedure as committing to main, but before
hitting the 'Commit All' button, click the hyperlink on the branch name at the top of the pane.

This takes you to the Branches pane, and the current branch (likely to be main) is highlighted.
You can then right click on that branch and select 'New Local Branch From' which will prompt
you for a new branch name and hit the 'Create Branch' button. This new branch is now the
active branch so you can go back to the 'Changes' pane (left click on the 'Branches' Header) and
complete the commit.

Switching Branches

Once you have more than one branch, you will want to be able to switch the active branch from
one branch to another. You do this in the 'Branches' pane (View -> Team Explorer -> Branches).
Note that whatever branch is bolded is the active branch and simply double clicking on a
branch makes it active.

When you switch branches YOU UPDATE THE LOCAL REPOSITORY WITH THE CONTENTS OF THAT BRANCH.
Thus with one double click potentially all the local files in the repo may change. If you have any uncommitted
changes, THE SWITCH IS LIKELY TO FAIL (it checks for this before doing any updates).
Typically you only switch branches if you have no modified files (in which case the switch is guaranteed to succeed).
If you hit this error (and you will), the solution is to commit your changes (if necessary to a
new temporary branch for that purpose) at which point you can switch branches.

Synchronizing the GitHub repository and the Local Repository.

After you have made edits, or when you want to pull down any GitHub changes to your local repository
you need to sync. Technically syncing is actually a combination of two operations a
push (updating the GitHub repository with what you have changed), and a pull (updating your local repository
with what is new from the GitHub repo).

One important point is that both a push and a pull operation operate on a branch (they cause the
branch to update) and both might cause conflicts (if a file has been changed in both places). Generally
speaking you should only push or pull on your local 'main' branch. Otherwise all the bad effects
of interleaving unrelated changes discussed previously come into play. If you think you need to sync
some other branch, it is more likely that you need to synchronize main, and then rebase your other
branch on top of main (TODO reference instructions for this).

To synchronize the main branch do the following

Set the active branch to 'main' with no modified files, by following the procedure 'Switching Branches'
above. You may need to commit things to a temporary local branch to get to this state, but
one way or the other you need to get there.
Go the Synchronization pane of team explorer.
Click on the 'sync' hyperlink.

If there are no conflicts doing the update, then the resulting updates are automatically committed to
the active branch.

If there are merge conflicts Visual Studio will stop and give you an opportunity to fix them.

Discarding Changes

If you decide you want to abandon changes, simply go to the View -> Team Explorer -> Changes view, select all
the files, right click and select Undo Changes.

Viewing history

From the View -> Team Explorer -> Branches view you can right click on any branch (it does not need to
be the active one) and select 'View History'.

Review

At this point we have described the critical workflows

Synchronization with what is in GitHub
Making changes
Committing (checking in)

Documentation TraceEvent TraceEventLibrary (`documentation/TraceEvent/TraceEventLibrary.md`)

The Microsoft.Diagnostics.Tracing.TraceEvent Library

The TraceEvent library is a library that allows you to both collect and process event data. It was originally designed to parse Event Tracing for Windows (ETW) events that the Windows operating system can generate. This is the library that PerfView uses to do most of its data manipulations, so if you are trying to automate the processing of some data that you can see in PerfView, there is a very good chance that you want to use the TraceEvent library to do it.

Microsoft.Diagnostics.Tracing.TraceEvent is a NuGet package available from nuget.org. This library works on both the .NET Desktop (v4.6.2 and up) as well a .NET (or .NET Core) (netstandard2.0 and up). Parts of the library work on Linux, but ETW is a Windows specific technology and thus that part (which is a lot of the package) only works on Windows.

The data model for ETW data is non-trivial, and the data is large (which means that you have to care about efficiency). There is also the question of decoding events (they are logged as binary blobs) as well as the parsing of stack information (which needs symbol files or .NET Events to decode properly). This makes the data model exposed by the TraceEvent package non-trivial and requires some explanation.

TraceEvent Programmers Guide is your guide for this. Fundamentally however, the traceEvent library simply provides a set of .NET class definitions as a NuGet package. Thus you only need to create a .NET project (it can be .NET or .NET Desktop), add a reference to the Microsoft.Diagnostics.Tracing.TraceEvent NuGet package (right click on the project -> Manage NuGet Packages) and start using the APIs. Here is a trivial example that logs all the kernel, CLR and dynamic events in an ETL (ETW Trace File).

text

using Microsoft.Diagnostics.Tracing;
using System;

class Program
{
    static void Main()
    {
        using (var source = new ETWTraceEventSource("ETWData.etl"))
        {
            // setup the callbacks
            source.Clr.All += Print;
            source.Kernel.All += Print;
            source.Dynamic.All += Print;

            // iterate over the file, calling the callbacks.
            source.Process();
        }
    }

    static void Print(TraceEvent data)
    {
        Console.WriteLine(data.ToString());
    }
}

Supported Windows Versions

On Windows, TraceEvent requires Windows 10 or Windows Server 2016 or later.
Its module initializer throws PlatformNotSupportedException on older Windows;
the CLR may wrap this in an initialization exception. This check is skipped on
Linux and macOS, preserving the library's existing cross-platform capabilities.
Older traces and target runtimes can still be analyzed on supported hosts.

TraceEvent Samples

To see more complete samples that use the APIs in more sophisticated (but not too sophisticated ways). In the Samples directory here are one or two page samples doing interesting things (collecting data, parsing from files or in real time, transforming one ETL file to another etc). Each sample is independent of the others.

The easiest way to build the latest samples is to simply clone the PerfView Repository and and build it. The samples are in the TraceEventSamples project in the PerfView solution. Simply set this project to be your Startup Project (right click on it in Solution Explorer -> Set as Startup Project) and run it (F5). By default will run all the samples.

Release Notes for the Microsoft.Diagnostics.Tracing.TraceEvent library

The Releases page for PerfView also shows the releases for the TraceEvent library. PerfView's version numbers and TraceEvent version numbers starting with version 3 are kept in lock step.

Documentation TraceEvent TraceEventProgrammersGuide (`documentation/TraceEvent/TraceEventProgrammersGuide.md`)

The TraceEvent Library Programmers Guide

Introduction: Strongly Typed (Semantic) Logging

As long as there have been programs, developers have used logging system to diagnosis functional and performance problems. Logging systems break down into two broad categories:

Weakly typed: The logging system only knows how to log strings and thus all data is converted into strings before it is logged. Printf logging is an example of this. These are the simplest logging systems and are a fine choice for ad hoc investigations by humans, however they suffer if the volume of data increases or if there is a desire to manipulate the data being logged programmatically.
Strongly typed (also known as Semantic Logging): In these logging systems each event is assigned schema which defines
1. The name of the event
2. The names and types of all the data items that are logged along with the event
3. Optional meta-data about the event (groups it belongs to, its verbosity etc.).

Because the events are strongly typed, the consumer of the logging data can assume that the events conform to the schema which makes processing the events considerably easier and less fragile. It also improves the speed and compactness of logging as well as the post processing (no need to print to strings and then parse back). The Event Tracing for Windows (ETW) system built into the Windows Operating System is an example of a strongly typed logging system (see Introduction to ETW for more)

In general strongly types logging systems make sense in the same places where strongly typed programming languages make sense, when:

When the data is likely to be consumed by other automation rather than simply viewed by humans.
When the amount of data (scale of logging) is high and therefore puts a premium on logging efficiently.
When the logging system is intended to 'last forever', and will need to be constructed in a decentralized way by many diverse people over many versions.
When the code logging the data versions at a different rate than the code processing the event data.

Server scenarios (aka the 'cloud') generally value these attributes since they are likely to want to automate the monitoring task, deal with high scale, and be developed by many programmers over time. Thus strongly typed eventing is a natural fit for the cloud.

The TraceEvent library is NuGET package that provides a set of .NET Runtime classes that make it easy to control and consume (parse) the strongly typed Event Tracing for Windows (ETW) events. It is intended to be used in conjunction with the System.Diagnostics.Tracing.EventSource class (which can generate ETW events) to form a complete, end-to-end, strongly typed logging system.

Basic Logging Architecture

There are three basic parts to and ETW based logging system as shown in the figure below

mermaid

graph LR
    provider["Event Provider (EventSource)"]
    session["Event Session (TraceEventSession)"]
    consumer["Event Consumer (TraceEventSource)"]
    file[("File")]
    provider -- "Commands" --> session
    session -- "Commands" --> provider
    session -- "Event Data" --> consumer
    provider -- "Event Data" --> file
    file --> consumer
    provider -- "Event Manifest (Schema)" --> consumer

The Event Session (represents the entity controlling the logging). The session has the ability to tell the providers of events to start and stop logging and control how verbose the logging is. It also has the ability to route the data to various places. It can indicate that the data should be directly written to a file (for maximum efficiency) or to send it to the session itself (for on the fly processing)
The Event Provider is the part of the logging system that is wired into the application to be monitored. Its job is to call a logging API when interesting things happen in the application.
The Event Consumer takes the data from the file or from the session and consumes it in some way, typically generating aggregate statistics and generating alerts.

Corresponding to each of these 'players', the TraceEvent library has a class that supports that role.

The Event Session uses the Microsoft.Diagnostics.Tracing.TraceEventSession class.
The Event Provider uses the Microsoft.Diagnostics.Tracing.EventSource class.
The Event Consumer uses the Microsoft.Diagnostics.Tracing.TraceEventSource class.

It is not uncommon for each of these 'roles' to be implemented in a separate process, but it is also common for some of the roles to be combined (most commonly the Session and the Consumer are in the same process).

Because the logging system is strongly typed, each event conforms to a particular schema and the event consumer needs to know this schema to decode the event data. The ETW system calls this schema the manifest and one representation of this is an XML file, but it also is stored in a binary form for efficient processing. One of the responsibilities of an event provider is to publish a manifest for the events that it generates so that the event consumer can find it when it needs it. As we will see there are a number of different ways of doing this. This information flow is designated in the diagram above by the red arrow.

Multi-player

An important aspect of the architecture that is not obvious in the diagram above is that each of the elements of the diagram can have multiple instances. Thus there can be:

Multiple event providers, each emitting events for the part of the system they understand, and each of which has a manifest for the events it might emit.
Multiple event sessions each of which are gathering different sets of events for different purposes. Each of these sessions has the option of logging its events to a file or to the session itself in real time.
Multiple event consumers that can process the events from a file or session. It is not recommended however to have multiple consumers feeding from the same event session in real time. Instead it is simpler and better to have multiple session each feeding a unique event consumer in the real-time case.

Asynchronous

Another fundamental property of the system that is not obvious from the diagram above is that the logging system is asynchronous. When a provider writes an event, it is a 'fire and forget' operation. The event quickly gets written to a buffer and the program continues. From this point on processing of the event is concurrent with the running of the application. This has a number of ramifications:

Logging is fast and scalable. Only the initial copy to the first buffer in the logging pipeline actually delays the program. Note that this is independent of the number of sessions or consumers. All the rest of the logging activity happens on other threads of execution and can be parallelized (thus it scales well).
Logging has minimal and predictable impact on the program. There are no 'global locks' that need to be taken by the provider when it logs events. Because of this it is much more likely that behavior of threads (races) with logging on will be essentially the same (statistically speaking) as with logging off.
There is the possibility of lost events. If the providers generate event faster than the file can store them (or the real time processor can process them), then eventually events need to be dropped. This is the price you pay for making the writing of an event be asynchronous. The system will detect that events are lost, but the event data itself is unrecoverable.

Quick Start Example

Enough theory. Let's see how this works in practice.

Component 1: the Event Provider (`EventSource`)

We start by logging some events using System.Diagnostics.Tracing.EventSource. You and paste this code into a console application project to have a fully working example (compiled against V4.5 or later of the runtime) if you wish.

csharp

using System.Diagnostics.Tracing;

[EventSource(Name = "Microsoft-Demos-MySource")]
class Logger : EventSource
{
  public void MyFirstEvent(string MyName, int MyId) { WriteEvent(1, MyName, MyId); }
  public void MySecondEvent(int MyId) { WriteEvent(2, MyId); }
  public static Logger Log = new Logger();
}

class Program
{
  static void Main(string[] args)
  {
    Logger.Log.MyFirstEvent("Hi", 1);
    Logger.Log.MySecondEvent(1);
  }
}

Above is the source code for a program that logs two events using EventSource class. To be strongly typed, we must provide a schema for each event, and we do this by defining a subclass of the EventSource class. This class will have an instance method for each event that can be logged by this provider. In the example above we define two events:

The MyFirstEvent that logs a string MyName and an integer MyId.
The MySecondEvent that logs just the integer MyId (which allows it to be correlated with the corresponding MyFirstEvent).

Notice that the method definitions provide all the information needed to generate the schema for the events. The name of the method defines the name of the event, and the argument names and types provide the names and types of each of the properties associate with the event.

In a more perfect world, humans would only author the declarations of an EventSource class since it is these declarations that specify the programmer's intent. However to make these methods actually log events, the user needs to define a 'boiler plate' body for each event that does two things:

Defines a numeric value associated with the event. This is the first parameter to the WriteEvent method call and is used to identify the event in all further processing (the event name is only used to generate the manifest). These event numbers start at 1 (0 is reserved) and by default needs to be the ordinal number of the method in the class. Thus it would be an error to reverse the order of the MyFirstEvent and MySecondEvent declarations above without also changing the first parameter to WriteEvent to match the order in the class. If this restriction bugs you, we will see how to avoid it later, but it will mean more typing on your part.
Passes along all the arguments from the method to the WriteEvent method. Because the arguments to the event method are used to generate the manifest, and the manifest is supposed to accurately describe the event, it would be an error to pass more or fewer arguments to WriteEvent. Thus the WriteEvent method is intended to be used only in this very particular way illustrated above.

The Logger class also has an attribute that defines the name for this provider to be Microsoft-Demos-MySource. If this attribute had not been provided the name of the provider would have been the name of the class without any namespace (e.g. Logger). If your provider is for more than ad-hoc logging, it is STRONGLY encouraged that you define a 'real' name for it that avoids collisions and helps your users understand what information your provider will log. We should follow the 'best practices' which the Windows Operation system group uses by making our name:

Start with the company name first (unique world-wide).
Then follow it with the name of a product or family of products.
Use further sub-groups as needed.
All separated by dashes (-).

Finally, in our example above the Logger class also defines a global static variable which creates an instance of the class that we use to log events. Having more than one instance of a particular EventSource is not recommended, since there is a cost to construct them, and having two serves no useful purpose. Thus most event sources will have a single instance, typically in a static variable that was auto-initialized as shown above.

Once we have our Logger event source defined, we simply call the event methods to log events. At this point we have an application with a fully functional ETW provider. Of course those events are off by default, so this program does not do anything yet, which brings us to step 2.

Component 2: The Event Session (`TraceEventSession`)

To turn on events we need an Event Session, which is defined by the TraceEventSession class. Typically this session will be in another process (typically some data-collection service or program but it can even be in the process logging the event). Here is code that does that. (Again you can cut and paste this into a console application which has referenced the TraceEvent Nuget Library to have a complete program)

csharp

using Microsoft.Diagnostics.Tracing.Session;

class Program
{
  static void Main()
  {
    using (var session = new TraceEventSession("MySession", "MyEventData.etl"))
    {
      session.EnableProvider("Microsoft-Demos-MySource");
      System.Threading.Thread.Sleep(10000);
    }
  }
}

In the code above, the program:

Creates a new TraceEventSession, each session is given a name that is unique ACROSS THE MACHINE. In our case we called our session MySession. Sessions CAN live beyond the lifetime of process that created them, and this name is how you refer to these sessions from other processes besides the process that created them. We also specify a file where the data is to be logged. By convention, these data files use the .ETL (Event Trace Log) suffix.
Once we have a session, we need to enable the providers. This can be done either by specifying the name of the provider or by using the unique 8 byte GUID that was assigned to the provider. Typically you will do this by name, which is what we do here, but in future example we will see cases where using the GUID is more convenient.
Waits for events to come in. In this case we simply wait 10 seconds. During this time any process that has Logger instances in it will log methods to this file. Both existing and newly created processes will log events.
Next we close the session. TraceEventSession implements IDisposable and the using clause will naturally dispose the session when the session variable goes out of scope. This causes the session to be torn down and the MyEventData.etl file will be closed.

That is it, thus running the program and then running the previous program simultaneously, will create a MyEventData.etl file with two events in it (the MyFirstEvent and MySecondEvent).

Component 3: The Event Processor (`ETWTraceEventSource`)

Now that we have a data file, we can process the data. This is what the ETWTraceEventSource class does.

csharp

using Microsoft.Diagnostics.Tracing;
using System;

class Program
{
  static void Main()
  {
    using (var source = new ETWTraceEventSource("MyEventData.etl"))
    {
      // Set up the callbacks
      source.Dynamic.All += delegate(TraceEvent data) {
        Console.WriteLine("GOT EVENT {0}", data);
      };
      source.Process(); // Invoke callbacks for events in the source
    }
  }
}

In the program above we:

Create an ETWTraceEventSource that uses the MyEventData.etl file as its data source. The ETWTraceEventSource represents the stream of events as a whole. Because this class needs to support real time processing of events it does not use the IEnumerable (pull model) for processing the events but rather provides a way of registering a callbacks.
Register callback for any events we are interested in. In this case we register a delegate that receives data (of type TraceEvent) and prints GOT EVENT and the event data for All events in the Dynamic group associated with the session. (More on this mysterious Dynamic property shortly).
Call the Process method, which causes the callbacks to actually be called. This method only returns when there are no more events (end of file), or because processing was explicitly stopped prematurely.
We dispose of the ETWTraceEventSource (the using clause does this for us when source goes out of scope). This closes the file and release all resources associated with the processing.

Running the program above will result in the following output:

text

GOT EVENT: <Event MSec="12.8213" PID="4612" TID="12372" EventName="MyFirstEvent" ProviderName="Microsoft-Demos-MySource" MyName="Hi" MyId="1"/>
GOT EVENT: <Event MSec="13.7384" PID="4612" TID="12372" EventName="MySecondEvent" ProviderName="Microsoft-Demos-MySource" MyId="1"/>

Some useful things to call out about the output:

The event is automatically stamped with a very accurate timestamp (typical resolution is 10 nsec or so) which indicates exactly when in time the event occurred.
The process and thread on which the event fired is also captured automatically.
The name of the provider as well as the event name are also captured.
All data items in the payload are also decoded. The names of the data items MyName and MyId are recorded (as well as their types) along with the specific value.

In this example we simply use the ToString method to print an XML representation of the event, however there are APIs for getting at all the data items above (including the payload values) in a convenient programmatic way. This is the real 'value add' of strongly typed logging.

At this point we have constructed an end-to-end example, creating a controller (TraceEventSession) that activated an ETW provider (our Logger implementation of EventSource) and send the data to a file which we then read with a consumer (ETWTraceEventSource) to pretty print the resulting events.

Event Parsing 'Magic' (`TraceEventParser` and derived types)

We have so far glossed over exactly how the data.ToString() call was able to determine the names of the events (e.g. MyFirstEvent) as well as the names of the arguments (e.g. MyName and MyId) of the two events that were logged. We explain this important aspect here.

The ETW architecture breaks event data needed by parsers into two categories:

Information which is known before an event is actually fired (e.g. provider names, event names, arguments, types, verbosities, ...). This schema data can be described by XML called a manifest. Each event provider conceptually has a manifest that describes the events it might log.
Data that is only known at logging time and needs to be serialized into the data stream. (e.g. the string and integer values passed to MyFirstEvent and MySecondEvent, but NOT the names MyFirstEvent and MySecondEvent (they are in the first category)).

An important architectural point is that an ETWTraceEventSource is a source of UNPARSED events. These are represented by the Microsoft.Diagnostics.Tracing.TraceEvent class. These unparsed events will only know what is known without looking at a manifest. Thus it knows things like the timestamp, process ID, provider ID (GUID) and event ID (the small integer you pass to EventSource.WriteEvent). It even has the payload blob of bytes, but it does not know how to interpret them.

This is why although you CAN subscribe to events directly from ETWTraceEventSource, (this is what the AllEvents registration point does), generally you don't want to do this because the event you get back will not be able to give you names of events or decode the payload values.

Instead you need to hook the ETWTraceEventSource up to a TraceEventParser. As its name implies, a TraceEventParser is a class that knows how to parse some set of events. For example there is a class called KernelTraceEventParser that knows how to parse most of the events that can be generated by the Windows OS kernel. When this parser is created, it is 'attached' to a particular TraceEventSource. It in turn exposes a set of callback registration points (events in the C# sense) that allow you to subscribe to particular parsed events. For example the KernelTraceEventParser has the subscription point:

csharp

public event Action<ProcessTraceData> ProcessStart;

which fires whenever a new process starts on the machine. You can subscribe to this event like so

csharp

kernelParser.ProcessStart += (ProcessTraceData data) => Console.WriteLine(data.CommandLine);

Which creates a C# delegate (the => operator) which takes ProcessTraceData, and prints out the command line that started the process. Thus a parser

Has a subscription point (a C# event) for every event it knows about (in this case we see the ProcessStart event).
For each such event, if there are payload values, then there is a specific subclass of TraceEvent, which defines properties for each data item in that event. In the example above the ProcessStart event has payload values (event arguments) like CommandLine and ParentID and you can simply use C# property syntax to get their value, and the properties have the types you would expect them to have (string for CommandLine and integer for ParentID).

Here is where the 'strong typing' of the logging becomes apparent. On the one end an EventSource logs strongly typed set of arguments (with names), and on the other end they pop out of a TraceEventParser as a class that has properties that let you fetch the values with full type fidelity.

Thus in general you get a diagram that looks like this:

mermaid

graph LR
    etw(["EtwTraceSource"])
    clr(["ClrTraceEventParser"])
    kernel(["KernelTraceEventParser"])
    registered(["RegisteredTraceEventParser"])
    dynamic(["DynamicTraceEventParser"])
    etw -- "Unparsed events" --> clr
    etw -- "Unparsed events" --> kernel
    etw -- "Unparsed events" --> registered
    etw -- "Unparsed events" --> dynamic

    gcEvent(["GC Event Callback"])
    jitEvent(["JIT Event Callback"])
    clr -- "Parsed events" --> gcEvent
    clr -- "Parsed events" --> jitEvent

    dllLoad(["DLL Load Callback"])
    threadSwitch(["Thread Switch Callback"])
    kernel -- "Parsed events" --> dllLoad
    kernel -- "Parsed events" --> threadSwitch

    iisEvent(["IIS Event Callback"])
    registered -- "Parsed events" --> iisEvent

    eventSource(["EventSource Callback"])
    dynamic -- "Parsed events" --> eventSource

Where potentially many different TraceEventParser types are 'attached' to a TraceEventSource and then in turn many callbacks are registered to the parser that knows how to decode them. The result is that you get fully parsed events in your callback code. Here is code that shows how to 'connect' the KernelTraceEventParser class to an ETWTraceEventSource and then subscribe to a the ProcessStart event and fetch out the process name and command line. Notice that in the callback delegate we specify a specific subclass of TraceEvent called ProcessTraceData which in addition to all the generic properties of an event (Name, Process, Timestamp, ...) also has properties for those specific to the ProcessStart event (e.g. ProcessName, CommandLine, ProcessID, ...).

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Parsers;
using Microsoft.Diagnostics.Tracing.Parsers.Kernel;
using System;

class Program {
  static void Main() {
    // Get a source that can return the raw events.
    using (var source = new ETWTraceEventSource("MyEventData.etl")) {
      // Connect a parser that understands kernel events
      var kernelParser = new KernelTraceEventParser(source);

      // Subscribe to a particular Kernel event
      kernelParser.ProcessStart += delegate(ProcessTraceData data) {
        Console.WriteLine("Process {0} Command Line {1}",
          data.ProcessName, data.CommandLine);
      };

      source.Process();    // call the callbacks for each event
    }
  }
}

Certain parsers (the Kernel, Clr, and Dynamic parsers) are so common that there is a shortcut property that makes it easier than the code above. In the code below we use the Kernel property on the ETWTraceEventSource object to get the kernel parser, and so the code can be simplified to the following.

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Parsers.Kernel;
using System;

class Program {
  static void Main() {
    // Get a source that can return the raw events.
    using (var source = new ETWTraceEventSource("MyEventData.etl")) {
      // Set up callback for a particular event
      source.Kernel.ProcessStart += delegate(ProcessTraceData data) {
        Console.WriteLine("Process {0} Command Line {1}",
          data.ProcessName, data.CommandLine);
      };
      source.Process();    // call the callbacks for each event
    }
  }
}

The TraceEvent library comes with a number of TraceEventParser parsers built in (in the namespace Microsoft.Diagnostics.Tracing.Parsers) including

KernelTraceEventParser - which knows about windows OS kernel events. These include DLL loading, process start, stop, CPU sampling, page faults, Disk I/O file I/O, memory, etc.
ClrTraceEventParser - which knows about Common Languages Runtime (.NET CLR) events. These include GC events, Just in Time compilation events, Exception events, ...
DynamicTraceEventParser - which knows about any event provider that have 'dynamic' manifests, in the sense that manifest information is dumped into the event stream using a standard convention. All EventSource types follow this convention, and thus all EventSources can be parsed by this parser.
RegisteredTraceEventParser - which knows about any event provider that registers itself with the operating system (using the wevtutil command)). This includes most providers that ship with the Windows operating system that are NOT the kernel provider or EventSource sources. You can see a list of such providers with the logman query providers command.
WPPTraceEventParser - which knows how to parse events written with the WPP Tracing system. Device drivers and other low-level components often use this mechanism.
JScriptTraceEventParser - which knows about the JavaScript runtime events
TPLTraceEventParser - which knows about the Task Parallel Library (another name for classes in the System.Threading.Tasks namespace).
ASPNetTraceEventParser - which knows about ASP.NET events.

As mentioned the first three providers are so common, that the ETWTraceEventSource has three shortcut properties (Kernel, Clr, and Dynamic), that allow you to access these providers in a very easy way.

We are now finally in a position to explain the mysterious piece of code in our original example that parsed EventSourceEvent:

csharp

source.Dynamic.All += delegate(TraceEvent data) { ... }

This code takes the ETWTraceEventSource source and fetches the DynamicTraceEventParser using the Dynamic property, and then asks that parser to call it back on all events THAT THAT PARSER KNOWS ABOUT. Thus All does not mean every event coming in from the source, but just all events that a DynamicTraceEventPaser can parse (which is basically any event generated from an EventSource).

Static vs. Dynamic `TraceEventParser` parsers

You will notice that in our first example where we were parsing EventSource events using the DynamicTraceEventParser the callback was declared as receiving its event data as a TraceEvent, and not some subclass (like ProcessTraceData). This might lead you to believe that the data is not parsed, and you would be half-right. Because parsers like DynamicTraceEventParser only learn about the event schema at runtime, it does not even 'make senses' to generate specific types like ProcessTraceData that have the right properties because we simply don't know these at compile time. However EventSource sources DO log this manifest information to the ETW stream itself, so the information needed to decode the ETL file is in the file at runtime. Thus the DynamicTraceEventParser CAN decode the event. In particular there are functions like TraceEvent.PayloadByName(string) which given a string name will return the payload for that property. For example here is the first example where we fetch the MyName and MyId fields from the MyFirstEvent event. At compile time we only know that the return event is a TraceEvent but the DynamicTraceEventParser is able to parse it can return type-correct values from the PayloadByName method.

csharp

using Microsoft.Diagnostics.Tracing;
using System;

class Program
{
  static void Main()
  {
    using (var source = new ETWTraceEventSource("MyEventData.etl"))
    {
      // Set up the callbacks
      source.Dynamic.AddCallbackForEvent("MyFirstEvent", delegate(TraceEvent data) {
        Console.WriteLine("GOT MyFirstEvent MyName={0} MyId={1}",
          data.PayloadByName("MyName"), data.PayloadByName("MyId"));
      });
      source.Process(); // Invoke callbacks for events in the source
    }
  }
}

Clearly this experience is a step down from what you get with a compile time solution. Certainly it is clunkier to write, and also error prone (if you misspell MyName above the compiler will not catch it like it would if we misspelled CommandLine when accessing process data). It is also MUCH less efficient to use PayloadByName than to use compile time trace parser properties. What we would have LIKED to write is something like the following

csharp

source.Dynamic.AddCallbackForEvent("MyFirstEvent", delegate(MyFirstEventTraceData data) {
  Console.WriteLine("GOT MyFirstEvent MyName={0} MyId={1}", data.MyName, data.MyId);
});

Where we have an event-specific type that with a MyName and MyId property. There are two options for doing that:

You can create a static (compile time) parser that is tailored for your EventSource and thus can return compile time types tailored for your event payloads. This is what the TraceParserGen tool is designed to do, which we will cover later.
You can take advantage of built-in DLR integration. In C#, you do this by casting to 'dynamic':
csharp
```
source.Dynamic.AddCallbackForEvent("MyFirstEvent", delegate(TraceEvent data) {
  var asDynamic = (dynamic) data;
  Console.WriteLine("GOT MyFirstEvent MyName={0} MyId={1}", asDynamic.MyName, asDynamic.MyId);
});
```
Note that unlike the TraceParserGen approach, this still has the downside of not having compile-time validation (if you misspell a property name, you won't know at compile time). Also note that the perf implications of casting to dynamic have not been measured, but it cannot be better than the PayloadByName approach (because that's what happens internally).

Other languages may have built-in DLR integration, such as PowerShell:
powershell
```
# Get the set of all MyName values
$myProc.EventsInProcess | `
     where ProviderName -eq MyProvider | `
     %{ $PSItem.MyName } | `
     Select -Unique
```

While the performance of using the DLR integration might not be as good as the TraceParserGen approach, it can be very convenient for prototyping, or just poking around in an interactive shell like PowerShell.

Lifetime constraints on `TraceEvent` objects

In the most common scenarios involving the TraceEvent library, a user program will scan over a very large stream of events in an ETL file will only care about a small fraction of the data collected. Because of this, TraceEvent is tuned to be as efficient at scanning events as possible. To achieve this TraceEvent AGGRESIVELY REUSES the System.Diagnostics.Tracining.TraceEvent objects that it passed to user code.

Consider the following simple example that scans the MyEventData.etl file for ProcessStart events from the kernel to find the first process that started during data collection.

The user registers a callback delegate with the ProcessStart events associated with the KernelTraceEventParser. This delegate will be called with a ProcessTraceEvent argument which represents the event. It is important to realize that user code cannot keep references to this object after the callback has returned. This is because the next ProcessStart event the library returns will likely overwrite this object with new data, leading to errors.

In the code above the user code wished to remember the data associated with a particular event (the first process start event), and thus has a problem. There are two options:

Create a new user-defined object and copy out the necessary data into this object at scan time. Thus no reference to the TraceEvent object survives the callback.
Use the TraceEvent.Clone() method. This method will make a copy of TraceEvent object which will not be reused by the library.

As you can see, the code above opted for the second technique. All that is required is to call the Clone() method (and probably cast it back to its most specific type), before storing it in a long-lived reference. This is a moderately expensive operation (copying 50 or so bytes of data), so if you only need a handful of fields and you have an obvious user-defined structure on which to keep them, the first option (copying what you need out of the event), is more efficient.

While cloning events a bit of a pain, by aggressively reusing event object, in the common case you can scan a file with millions of events and not have to allocate any objects in the main scanning loop. This is an important performance win.

Review of the Fundamental TraceEvent Architecture

At this point you have the fundamentals of the TraceEvent library.

TraceEventSession starts new ETW sessions and turn on ETW providers and direct the output.
ETW Providers you can turn on include:
1. The Window OS Kernel Provider, .NET and Jscript Runtime, as well as most OS components.
2. Any System.Diagnostics.Tracing.EventSource you create in your own apps.
You process the events by first hooking up a ETWTraceEventSource to the event stream (either a file or real time).
The events in the ETWTraceEventSource are unparsed. To parse them you need to hook the appropriate TraceEventParser up to your ETWTraceEventSource. These Parsers are either:
1. Static (compile time) which are easier to use and very efficient but require that you know the schema of the events you wish to use at compile time. Static parsers include most of the ones that come with the TraceEvent library as well as ones you generate from manifest files using the TraceParserGen tool (discussed later).
2. Dynamic (run time) which are parsers that can process manifest information on the fly. There are only a handful of these including DynamicTraceEventParser, RegisteredTraceEventParser, and WPPTraceEventParser. These are not as convenient to code against, and are less efficient, but are often used when the processing to be done is minimal (e.g. printing).
Once you have a parser, you subscribe to the C# events you are interested in and your callbacks will get strongly typed versions of the events that understand the payload.
Once you have hooked up your subscriptions, you call ETWTraceEventSource.Process() to start processing the ETW stream and calling the appropriate callbacks.

What You Can Do With the TraceEvent Library

To keep things concrete we 'went deep' so far, showing real code, but by necessity we restricted ourselves to the simplest possible scenario using the TraceEvent library. Now we 'go broad' and describe the breath of the library so you can determine if the library is capable of handling the scenario you have in mind.

Capabilities include:

The ability to monitor ETW events, sending them either to a file or directly to a programmatic callback in 'real time'.
The ability for those real time events to be passed to the IObservable<T> interface and thus be used by the Reactive Extensions.
The ability to turn on event providers selectively using ETW 'Keywords' and verbosity 'Levels'. You can also pass additional arguments to your provider which EventSource sources can pick up. In that way you can create very sophisticated filtering specification as well as execute simple commands (e.g. force a GC, flush the working set, and etc.).
The ability to enumerate the ETW providers on the system as well as in a particular process, and the ability to determine what ETW groups (Keywords) you can turn on.
Ability to take ETL files and merge them together into a single file.
Ability to read an ETL file or real time session and write an ETL file from it, filtering it or adding new events.
The ability to capture stack traces when events are being logged.
The ability to convert the stacks to symbolic form both for .NET, Jscript, as well as native code.
The ability to store events in a new format (ETLX) that allows the events to be accessed efficiently in a random fashion as well as to enumerate the events backwards as well as forwards, and to efficiently represent the stack information.
The ability to generate C# code that implements a strongly typed parsers for any ETW provider with a manifest (TraceParserGen).
The ability to read events written with the WPP Tracing system.
The ability to access 'Activity IDs' that allow you to track causality across asynchronous operations (if all components emits the right events).
Access Kernel events (along with stack traces), including:
- Process start/stop, Thread start/stop, DLL load and unload.
- CPU Samples every MSec (but you can control the frequency down to .125 msec).
- Every context switch (which means you know where you spend blocked item) as well as the thread that unblocked the thread.
- Page faults.
- Virtual memory allocation.
- C or C++ heap allocations.
- Disk I/O.
- File I/O (whether it hits the disk or not).
- Registry access.
- Network I/O.
- Every packet (with compete data) that comes on or off the network (network sniffer).
- Every system call.
- Sampling of processor CPU counters (instructions executed, branch mispredicts, cache misses, ...).
- Remote procedure calls.
- How the machine is configured (disk, memory, CPUs, ...).
Access CLR (.NET Runtime) events, including:
- When GCs happen.
- When allocations are made (sampling and non-sampling).
- When objects are moved during a GC.
- When methods are Just In Time (JIT) compiled.
- When exceptions are thrown and the stack at which it was thrown.
- When System.Threading.Task.Task instances are created and scheduled.
- Addition information on why a .NET assembly failed to load (to diagnose failures).
- Information to decode .NET frames in stack traces.
Access ASP.NET events which log when request come in and when various stages of the pipeline complete.
Access WCF events which log packets as the go through their pipeline.
Access JScript runtime events, including:
- Garbage collection.
- Just-in-Time (JIT) compilation of methods.
- Information to decode JScript frames in stack traces.

You can also get a reasonably good idea of what is possible by taking a look at the PerfView tool. PerfView was built on top of the TraceEvent library and all the ETW capabilities of that tool are surfaced in the TraceEvent library.

ETW Limitations

Unfortunately, there are some limitations in ETW that sometimes block it from being used in scenarios where it would otherwise be a natural fit. They are listed here for emphasis.

Provider control is machine-wide, but TraceEventProviderOptions.ProcessIDFilter and ProcessNameFilter can limit collected events to selected processes on supported Windows hosts.
Because commands are machine wide and thus give you access to all information on the system, you have to be Elevated (Admin) to turn an ETW session on or off.
By design the communication between the controllers and the providers is 'fire and forget'. Thus ETW is not intended to be a general purpose cross process communication mechanism. Don't try to use it as such.
In real time mode, events are buffered and there is at least a second or so delay (typically 3 sec) between the firing of the event and the reception by the session (to allow events to be delivered in efficient clumps of many events).
Named sessions can combine kernel and user providers. The special NT Kernel Logger session remains kernel-only; reusing its name can interfere with another tool using that session.
In general scenarios having multiple controllers (sessions) controlling the same providers is dangerous. It can be done in some cases, but there is a significant potential for interference between the sessions.
ETL files can get large. Compression is available on supported Windows hosts; historical uncompressed traces may be substantially larger.
Logging more than 10K events/sec will load the system noticeably (5%). Logging more frequently than 10K/sec should be avoided if possible. Logging 1M events/sec will completely swamp a typical machine.

Next Step: Code Samples

The best way to take the next step in learning about the TraceEvent library is to experiment with the TraceEvent code samples. This is a NuGet package that shows some simple but common scenarios. These samples are well commented with discussions on potential pitfalls and potential subtle design issues associated with the scenarios. They are worth the read. The Samples are in the TraceEvent Library Samples Package. A simple way of trying them out using Visual Studio is to:

Create a new Console program project.
Right click on the 'References' icon under the new project's XXXX.*Proj file. In solution explorer.
Select 'Managed NuGet Packages'.
Search for 'TraceEvent' in the dialog that comes up.
Select 'TraceEvent Library Samples'.

This will download the sample source code into your console application. The samples are in the TraceEventSamples directory. Take a look at them. There is a README.TXT file to get you going, which will tell you to modify your program to call:

csharp

TraceEventSamples.AllSamples.Run();

To run all the samples. They should be self-explanatory.

Diagnostic Techniques: Using PerfView

One prominent program that uses the TraceEvent library extensively is the PerfView tool. As its name suggests, this tool's primary purpose is the collection, aggregation, and display of data useful for performance investigation, however it also is a valuable tool for simply enabling and controlling ETW event providers like EventSource sources as well as displaying ETW events in a human readable fashion.

In particular you can log the data from the Microsoft-Demos-MySource EventSource we defined to a file with the PerfView command:

text

PerfView collect /OnlyProviders=*Microsoft-Demos-MySource

Please note the *. This is PerfView's way of indicating that the name should be converted to a GUID in the standard way that EventSource sources define. Once you have the data you can view it using the PerfView Events view. It is very easy to filter by process, time, event name or text in the events, as well as export the data to Excel or XML. The following PerfView videos are especially relevant.

You will find having PerfView to be very handy when debugging your own ETW processing.

Full dumps of ETW events in PerfView

Normally PerfView's event view does not show all the data in an ETW file. Things like the Provider GUID, EventID, Opcode and payload bytes are not shown because they typically are not relevant. However when you are debugging ETW processing of your own, these low level details can be critical so you would like to be able to see them. You can do this selectively in PerfView by doing the following:

Find the Event that you are interested in (typically by looking at the timestamp).
Select the time cell of that event.
Right click and select → Dump Event.

It will then open PerfView's log that will contain a XML dump of the event, including all low level information (including the raw payload bytes).

PerfView's `/keepAllEvents` feature

By default PerfView filters out some kernel and CLR events that it does not believe are of interest to you. These events happen in the beginning and end of the file (so called DC (data collection) start and stop events). Most of the time this is a good thing, but if there is a problem you are trying to debug with these events this filtering is a problem. It can be disabled by starting PerfView with the /keepAllEvents option. Also you may also have to use the File → Clear Temp Files, GUI command if PerfView had already opened that particular ETL file. Otherwise it will simply use the old (filtered) data in a temp file.

Debugging EventSource Authoring

While this document is really about TraceEvent, there is a good chance you will be authoring EventSource sources along the way. One pitfall of EventSource sources is that while they generate good diagnostic information when you make mistakes (like using the same event ID twice), these diagnostics only occur when the provider is actually enabled, and even then the exceptions that are thrown are swallowed by the runtime itself. The end effect is that the event source does not log events and it is not obvious why.

There are number of ways to diagnose error when authoring a new EventSource:

Develop the EventSource module using the EventSource NuGet package. This package as a compile time rule that will look for errors in your EventSource and warn you about them at compile time. This produces the best experience.
You should do all your development with the EventSource turned on. You can do this with the following command:
text
```
PerfView /CircularMB=10 /OnlyProviders:*Microsoft-Demos-MySource start
```
Which starts a circular logging session and leaves it on until you explicitly turn it off. Now you will get exceptions whenever your code runs, however they will still be swallowed. To fix that go to your Debug→Exceptions dialog and enable stopping on any thrown CLR exception. Any authoring mistakes will now be very obvious.
Use the ConstructionException property. By default EventSource sources NEVER throw exceptions (so that turning on logging will not induce an error). However in DEBUG code you should have logic like this in your program that explicitly looks for errors during construction
csharp
```
if (MySource.ConstructionException != null)
  throw MySource.ConstructionException;
```
This technique, along with (2) above (turning on your EventSource constantly during development) will ensure that you find errors in your EventSource promptly error.
Look at the Exceptions events in PerfView when you attempt to use your EventSource (see this blog entry for more). Basically even though the EventSource does suppress exceptions during construction, they are actually occurring (but being swallowed). PerfView can see these exceptions and display them to debug the issue.

Real Time Processing of Events

In the examples so far, we have seen how to collect events to a file, and read the events from a file. It is also possible to do this skipping the creating of the file altogether by using what are called 'Real Time' source. Here is what one looks like that watches in real time for processes to start.

csharp

using Microsoft.Diagnostics.Tracing.Parsers;
using Microsoft.Diagnostics.Tracing.Parsers.Kernel;
using Microsoft.Diagnostics.Tracing.Session;
using System;

class Program {
  static void Main() {
    // create a real time user mode session
    using (var session = new TraceEventSession("ObserveProcs")) {
      // Set up Ctrl-C to stop the session
      Console.CancelKeyPress += (object s, ConsoleCancelEventArgs args) => session.Stop();

      // Subscribe to a callback that prints the information we wish
      session.Source.Kernel.ProcessStart += delegate(ProcessTraceData data) {
        Console.WriteLine("Process {0} Command Line {1}",
          data.ProcessName, data.CommandLine);
      };

      // Turn on the process events (includes starts and stops).
      session.EnableKernelProvider(KernelTraceEventParser.Keywords.Process);

      session.Source.Process();   // Listen (forever) for events
    }
  }
}

You can see a few key differences between this code and the code that processes a file.

It combines both the collection and processing together, so you have both a TraceEventSession and a TraceEventSource.
When the TraceEventSession is created it takes only the session name, a filename parameter is not needed since there is no file generated.
We set up a handler to call Stop() on the session when Ctrl+C is pressed. This stops what would otherwise be a infinite program.
You can get the TraceEventSource you need by accessing the Source property on a TraceEventSession.
From here you subscribe to events in the same way (finding the desired parser and registering a callback).
Once you are set up, you need to turn on providers. In this case we turn on Kernel providers that generate events for processes. Note that the Kernel provider is special and you need to call a special EnableKernelProvider API. User mode ETW provider (e.g. EventSource sources) would use the EnableProvider API as before. Here we are also showing the use of 'keywords' which are bitsets that indicate which events a provider CAN log should actually be logged. In our case we ask only for the ProcessStart events and not the many other events the Kernel source can provide.
Once you are set up, you call Process() just before. For real time sessions this call will wait forever until another thread calls session.Stop() (which is what the Ctrl+C handler does).

When you run this program you will see that there is a noticeable delay of a few seconds between a process starting and the event callback being called. This is because ETW does not flush aggressively but waits a second or more before flushing. This is more efficient, but does mean a slight delay. Note that the events do have the very accurate timestamp that indicates exactly when the event occurred, so correlating events in time is still easy, there is just a delay before you get the data.

Real Time Processing with `IObservable`

The LINQ library is a set of data-base like operators for manipulating any collection of objects (more formally any IEnumerable<T>). It allows you to filter, map, select, group or sort the collection. However library does assume a 'Pull model' where the data object are passive, and the client is the master and 'pulls' the data from the collection one element at a time. The Reactive Extensions library is a .NET NuGet package that allows you do something very similar to LINQ but for a 'Push model' stream. In the push model the client registers callback functions and the send data source is the master that 'pushes' the data one object at a time to the client on its schedule.

We have been using the push model in all the TraceEvent examples so far. It is characterized by having the user code register a callback and then having thread call a Process() method to wait for incoming data. This model is forced upon you when you are doing real time monitoring, since the collection of event never ends since more events can come in at any point.

To support LINQ-like query operators for 'push style' scenarios, we need something like IEnumerable<T> but set up for a push model. That is what IObservable<T> is designed to do. The TraceEvent library exposes its streams of events as IObservable<T> objects, and this allows you to use the LINQ-like query operators that the Reactive Extensions library defines to manipulate the data. It can be quite powerful.

Here is an example of using TraceEvent's IObservable<T> support to print out a line to the console every time a process starts.

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Parsers;
using Microsoft.Diagnostics.Tracing.Parsers.Kernel;
using Microsoft.Diagnostics.Tracing.Session;
using System;

class Program {
  static void Main() {
    // create a real time user mode session
    using (var session = new TraceEventSession("ObserveProcs")) {
      // Set up Ctrl-C to stop the session
      Console.CancelKeyPress +=
        (object s, ConsoleCancelEventArgs args) => session.Stop();

      // Create a stream of process start events.
      var procStream = session.Source.Kernel.Observe<ProcessTraceData>("ProcessStart");

      // Subscribe to the stream, sending data to the console in real time.
      procStream.Subscribe(data =>
        Console.WriteLine("Process Started: Name {0} CmdLine {1}",
        data.ProcessName, data.CommandLine));

      // Turn on the process events (includes starts and stops).
      session.EnableKernelProvider(KernelTraceEventParser.Keywords.Process);

      session.Source.Process();   // Listen (forever) for events
    }
  }
}

As you can see this example is VERY similar to the previous example that used c# events to subscribe to the real time events. The only differences are:

Instead of using the C# ProcessStart event to register the callback you use the Observe method on a TraceEventParser to create an IObservable<T> for specific events.
Once you have a IObservable<T>, you call its Subscribe method to actually register the callback.

In the example above there is not a lot of value in using IObservable<T>. The value would come into play if you wish to use the Reactive Extensions LINQ-like operators to form complex transformations on the events.

Note that UNLIKE the IEnumerable (TraceEvents) interface, the IObservable<T> interfaces by default ALREADY do a Clone() of the TraceEvent object before returning it. Thus there is no need for you to call Clone() if the event came from an IObservable<T> interface.

The TraceEvent code samples NuGet package has several examples of using the IObservable<T> support.

Best Practices for Versioning ETW Events

This section is really more about EventSource (production) than TraceEvent (consumption), but is likely relevant to users of TraceEvent as well.

At some point you are going to have deployed EventSource that generate events and parsers built on TraceEvent for processing the events. Inevitably you will wish to update the events to include more data or otherwise change things. It is likely however that you do not wish to break things and you will not be able to update all event generators and consumers atomically. This is where versioning is important.

The key to making this work is compatibility, and that is not hard to achieve if a few simple rules are followed:

Events can never remove or change the meaning of existing fields. If you need to so this, you need to make new events that are logically independent from the old events.
If you wish to add new data to an existing event (the most common versioning operation), make sure you add it to the END of the existing arguments. This way the serialized format for the old fields is IDENTICAL and can be parsed by logic that has no understanding of the new data.
When you add data in this way, you should increment the version number for the event. You can do this for an EventSource by adding an attribute like [Event(ID, Version=1)]. When you don't provide a version number EventSource defaults it to 0, so you can start with Version=1 for the first update. You can have at most 255 versions (so don't version unless you need to).

If you follow these rules, TraceEvent can sort everything out for both old and new events. Old processing code will continue to see all the old properties even when the data was generated with an updated EventSource (because of rules (1) and (2)), and new code accessing old fields can use the payload length to notice that old EventSource sources have not emitted a field and thus can return a default value (e.g. 0 or an empty string). Higher level logic that uses TraceEvent can always probe the Version fields associated with an event and do special handling but often the defaults provided by TraceEvent's parsers are sufficient.

Higher level Processing: `TraceLog`

So far the only way to process data in the TraceEvent library is using the 'push' model where you subscribe to events and get a callback when they arrive (either through C# events or IObservables). There are two reasons that the push model is the 'fundamental' mechanism in TraceEvent:

The push model works for real time scenarios. The pull model (e.g. foreach) simply does not work in the real time case since real time lists have no end.
The push model naturally handles a heterogeneous list of events. Each callback gets a strongly typed data specific for that event. In a pull model (e.g. foreach), as each event is processed it must be cast to the correct type to get at its event-specific fields. This is clumsy and inefficient.

Thus the push (callback) model is encouraged, but there are definitely cases where a pull model is convenient (e.g. iterating over all events of a particular type in an ETL file) where the pull model (foreach) is not problematic and the library should support.

Perhaps more importantly, the TraceEventSource model's simplicity can also be a problem. It is not uncommon to have millions of events and the callback model really only understand one way of accessing the data - enumerating it from start to finish (well you can abort part way). For many situation you would like to organize the data in a way that is closer to how the data will be used. For example it would be useful to able to find all the process, or threads, or modules or other things without having to read every last event in the file. It would also be useful to have these things be in 'tables' of their own (that are homogeneous), and many ways of accessing them (by name, by time, etc.).

Solving these problems is what the Microsoft.Diagnostics.Tracing.TraceLog class is all about. Unlike ETWTraceEventSource (which only supports the 'push' (callback) model) and supports real time streams. TraceLog ONLY deals with files, and can provide both a push (callback) model as well as a 'pull' (foreach) model. It also contains a rich object model that allows you to:

Efficiently access a range of events in time without having to read through from the beginning (random access).
Query the set of processes, threads, and modules mentioned in the trace.
It gives every item an index (small integer with a known bound) that uniquely identifies this. This allows you to create 'side' arrays that allow you to 'hand' additional information on the item.
Processes the raw stack trace information (which is a list of physical addresses) with the module load events and JIT compile events to form symbolic stack traces.

In short, the TraceLog class preforms the next level of common semantic interpretation of the event data, as well preserving the ability to access the raw event data.

Creating a `TraceLog` (an ETLX file)

TraceLog needs to remember not only the even information but also the processed form for the process, threads, modules, and call stacks in the trace. It would have been nice to 'extend' the ETL file format but that format is private and tuned for event logging, not lookup. Thus a new format was needed, which is called ETLX. Thus the basic architecture of processing is to take a ETL file and 'convert' it to a ETLX file. The TraceLog class is then simply the programmatic interface to this ETLX data. Because users typically begin with a ETL file, the TraceLog class has a OpenOrConvert method that makes starting with an ETL file easy. Here is a simple program that prints out every event in an ETL file as XML.

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Etlx;
using System;

class Program {
  static void Main() {
    using (var traceLog = TraceLog.OpenOrConvert("MyFile.etl"))
    {
      foreach (TraceEvent data in traceLog.Events) {
        Console.WriteLine("Got Event {0}", data);
      }
    }
  }
}

The way this works is that OpenAndConvert assumes that every ETL file has an ETLX file associated with it (by changing the extension). You can give either and it will check if the ETLX file is up to date (newer than the corresponding ETL file). If it is it uses it, otherwise it generates it from the ETL file. In either case it returns the TraceLog that represents the up-to-date ETLX file data.

Once we have a TraceLog we can get at the original events using the Events property that returns an IEnumerable. Thus you can do a foreach on this and in our case, print (ToString) each event to get the XML.

But the power of the TraceLog file is that is has also computed summaries of useful information like the processes, and has integrate this all into a convenient object model. Here is code that prints the XML events but only for process of a given name and rolls it up by thread.

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Etlx;
using System;

class Program {
  static void Main() {
    using (var traceLog = TraceLog.OpenOrConvert("MyFile.etl"))
    {
      var process = traceLog.Processes.FirstProcessWithName("PerfView");
      Console.WriteLine("Process devenv Start {0:f3} end: {0:f3}",
        process.StartTimeRelativeMsec, process.EndTimeRelativeMsec);
      foreach (var thread in process.Threads)
      {
        Console.WriteLine("Thread {0}", thread.ThreadID);
        foreach (var data in thread.EventsInThread)
          Console.WriteLine("   Event {0}", data);
      }
    }
  }
}

Notice that it was trivial to find all the data associated with a particular process (a common operation) and filter to just that. Further find the thread, and look at events only on those particular threads. Notice that it could print the start and end time (information that can only be formed by combining data from two events) trivially. Also you effectively are passing over the data multiple times (for each thread) and each time only getting the filtered data. Tracelog gives you random access to the data.

One useful way to exploit the random access capability is to enumerate events backwards in time. This may seem like a strange thing to do, but it is quite useful because it is often the case that after you find a 'bad event' you need to search backwards for its cause. Doing this we the callback model is simply problematic, but with TraceLog's object model it is pretty straightforward. In the example below we have reason to believe that an exception might be caused by the JIT compilation that occurred earlier in the trace. This code could be written to find interesting cases of this bug.

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Etlx;
using Microsoft.Diagnostics.Tracing.Parsers.Clr;
using System;

class Program {
  static void Main() {
    using (var traceLog = TraceLog.OpenOrConvert("MyFile.etl")) {
      // For a particular process
      var p = traceLog.Processes.FirstProcessWithName("PerfView");

      // Find all the exceptions.
      foreach (var e in p.EventsInProcess.ByEventType<ExceptionTraceData>()) {
        Console.WriteLine("Exception {0}", e.ExceptionMessage);

        // See what method was JIT compiled just before it.
        var prevInproc = p.EventsInProcess.FilterByTime(p.StartTime, e.TimeStamp);
        foreach (var j in prevInproc.Backwards().ByEventType<MethodJittingStartedTraceData>())
        {
          Console.WriteLine("Last Jit {0}", j.MethodName);
          break;
        }
      }
    }
  }
}

In the code above we first find all the events in the PerfView process and look for exception events (all pretty straightforward so far). When we find an exception we use the FilterByTime operation to isolate just the events up to that point. We then apply (the rather magical), Backwards operator that returns a collection that is the reversed time image of those events, and filter them to just the JIT events. We find the first such event and print the information we need.

What is amazing about this is that it is efficient. All the intermediate collections are never actually formed, so what actually happens at runtime is that you start with the exception event walking backwards until you find the JIT event. Typically you find the event you are looking for very quickly (it is nearby, and you can always place a limit on how far you look back), and everything finishes quickly. It is handy.

Using Call Stacks with the TraceEvent library

Perhaps the most compelling reason to use the TraceLog class is that it support symbolic resolution of stack information associated with events. ETW has the ability to collect stack traces associated with most events, but what actually gets logged are arrays of method return addresses. In addition, some stacks are decoupled from the event they are associated with and may even be in two pieces (a kernel piece and a user mode piece). In addition, these addresses need to be resolved to the module or JIT compiled method they belong to and the physical address needs to be resolved to a symbolic name. All this complexity make it infeasible use the raw stack events 'on the fly' but the ETLX conversion does all the necessary computation to make getting these stacks relatively easy. At least currently this functionality is only available via the TraceLog class. When an ETL file is converted to ETLX the stack events are processed into an efficient and compact form and wired up to the appropriate events.

There is a complete example of using the TraceLog class to access stack traces in the TraceEvent code samples but the actual API is pretty straightforward. Here is some code that opens an ETL file finds all the exception events in the PerfView process in the trace, and prints each method name

csharp

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Etlx;
using Microsoft.Diagnostics.Tracing.Parsers.Clr;
using System;

class Program {
  static void Main() {
    using (var traceLog = TraceLog.OpenOrConvert("MyFile.etl")) {
      var p = traceLog.Processes.FirstProcessWithName("perfView");
      foreach (var ex in p.EventsInProcess.ByEventType<ExceptionTraceData>()) {
        Console.WriteLine("Exception {0}", ex.ExceptionMessage);

        var cs = ex.CallStack();
        if (cs != null) {
          Console.WriteLine("  Stack: ", cs);
          while (cs != null) {
            Console.WriteLine("    Method: {0}",
              cs.CodeAddress.FullMethodName);
            cs = cs.Caller;
          }
        }
      }
    }
  }
}

As you can see the basic logic is pretty simple. TraceEvent has a CallStack() method that returns a callstack (null if there is no stack), and each callstack has among other things a FullMethodName property and a Caller property, and you can keep calling Caller until you have enumerate the complete stack up to the top of the thread. The model is significantly richer than is outlined here (you can actually get at line number and source file information), but you get the idea, it is pretty useful.

The Ugly Details of Stack Resolution

While the user model for dealing with stack capture is pretty straightforward, unfortunately the underlying mechanism that enables it is pretty complex and fragile so it is easy for it to not work or only work partially. This is why it is a VERY good idea to start with working example (like the example in the TraceEvent code samples) and make sure that you have done all the required steps. PerfView is also useful here, because it is also an application that is doing 'all the right things' so it is good to confirm that it is working for PerfView before you start trying to debug why your code is not working properly.

Below are the steps in converting logging an event with a stack to a resolved symbolic name annotated with some things that can go wrong along the way.

When the event is logged, the ETW system tries to crawl the stack at runtime. However this can fail for various reasons:
1. On 32-bit machines the crawler assumes the compiler stores unwinding information (EBP frames) on the stack. If the compiler does not do this the stack 'breaks' (can't be unwound) and you lose any frames 'toward thread start'.
2. For 64-bit processes, the crawler needs unwind information. Historical traces collected before Windows 8 (or Windows Server 2012) may have stacks broken at JIT-compiled frames because ETW could not locate that information. This old-host limitation is not a restriction on current collection, but reading an old trace cannot recover frames that were never recorded.
If either of these happen there is nothing wrong with your code as it is an issue with the APP or with the ETW infrastructure. PerfView should have the same problem.
The raw addresses have to be associated with some entity that knows its symbolic name. At this point things work very differently for native code and code that is JIT compiled on the fly:
1. For native code, TraceEvent must find the DLL that includes the code, for this it needs information about all DLLs that where loaded in the process and what addresses they are loaded at. These are the kernel ImageLoad events.
2. For JIT compiled code it needs to know the code ranges of all JIT compiled methods. For this it needs special .NET or Jscript events specifically designed for this purpose.
If the necessary events are not present, the best that can be done is to show the address value as a hexadecimal number (which is not very helpful). Thus it is critical that these events be present. Complicating this is the fact that in many scenario of long running processes. If the process lives longer than the collection interval, then there can be image loads or JIT compilation that occurred before the trace started. We need these events as well. To get them the ETW providers involved support something called 'CAPTURE_STATE' which causes them to emit events for all past image loads or JIT compilations. The logic for capturing data must explicitly include logic for triggering this CAPTURE_STATE.
For JIT compiled code, we are mostly done, however for native code, the symbolic name has only been resolved to the DLL level. To go further you need to get the mapping from DLL address to symbolic name. This is what the debugger PDB (program database) files do. For this you need to be able to find these PDB files. There are a number of things that can go wrong:
1. You must set your _NT_SYMBOL_PATH environment variable to locations where to search for the PDBS. If you do not you will only know the module and hex address.
2. For operating system DLLs, the PDBS live on what is called a symbol server. To find these your _NT_SYMBOL_PATH must include the name for these symbol servers (the public Microsoft symbol server is SRV*https://msdl.microsoft.com/download/symbols). However to look up a DLL in the symbol server, you need a special GUID associated with the DLL, and a RAW ETL file does NOT INCLUDE this GUID!. If you try to look up the DLL's PDB on the machine where the DLL exists, TraceEvent can fetch the necessary GUID from the DLL itself, but if the ETL file was copied to another machine this will not work and the PDB cannot be fetched. Running the TraceEventSource.MergeInPlace operation rewrites the raw ETL file so that it includes the necessary DLL GUIDs and thus is a requirement if you move the data off the collection machine (and you want symbolic information for native code stacks).
3. For .NET code all the library code is precompiled (NGENed) and so is looked up using a PDB like the native case. However unlike native DLLs, the PDBs for the NGEN images are typically not saved on the Microsoft symbol server. Instead you must generate the PDBs for the NGEN images from the IL images as you need them. Again if you resolve the symbols on the machine where the collection happened, at the time you resolve the symbols TraceEvent's SymbolReader class will automatically generate the NGEN image for you and cache it, however if you move the ETL file off the machine, you need to generate the NGEN PDBs as well as merge the ETL file to get the symbolic information for the .NET code in NGEN images. This is what the SymbolReader.GenerateNGenSymbolsForModule method can help you do. TODO MORE

SymbolReader has three independent authorization checks:

SecurityCheck controls whether a PDB found in an unsafe local location, such as beside an executable or in its build directory, may be used. The callback receives the PDB path. If the property is not set or returns false, the PDB is rejected. Applications should prompt the user or allow only explicitly trusted directories.
AuthorizeDownload controls whether Source Link and HTTP source-server downloads may retrieve source code. The callback receives a DownloadAuthorizationRequest and runs immediately before the HTTP request; approval also covers redirects followed by the HTTP client. It does not run for symbol/PDB downloads, local source, embedded source, or a valid cached source file. If the property is not set, source downloads are denied by default. Applications should prompt the user or enforce an explicit host/repository allow-list.
AuthorizeSourceServerCommand controls whether a validated command derived from PDB source-server data may execute. The callback receives a SourceServerAuthorizationRequest containing the exact rebuilt command. If the property is not set or returns false, the command is not run. Applications should prompt the user before execution or use a narrowly scoped policy. Before invoking the callback, TraceEvent independently verifies that the command uses a supported executable and allowed arguments. Returning true from the callback does not bypass that validation.

using (var symbolReader = new SymbolReader(log))
{
    symbolReader.SecurityCheck = pdbPath =>
        trustedPdbDirectories.Contains(Path.GetDirectoryName(pdbPath));
    symbolReader.AuthorizeDownload = request =>
        trustedSourceHosts.Contains(request.Uri.Host);
    symbolReader.AuthorizeSourceServerCommand = request => false;
}

Each true result authorizes only the operation described by that callback invocation. Exceptions from AuthorizeDownload are logged and deny the download.

So in summary to get good stacks and have them work on any machine for any code you need to:

Collect the necessary ImageLoad and JIT compile events (including CAPTURE_STATE so you get information about events that preceded data collection start).
Merge the raw ETL files so that PDB signature information is incorporated into ETL file.
Generate the necessary NGEN PDBs and upload them along with the ETL file to the machine where symbolic resolution will happen.

Source Code Support

TODO

Building Compile-time `TraceEventParser` parsers using TraceParserGen

If you have built your own EventSource, so far the only way you have of accessing the data from your EventSource is to use the DynamicTraceEventParser class. As mentioned previously the experience coding against DynamicTraceEventParser is not great since each property of the event has to be fetched with the PayloadByName method and it is very easy to get the names wrong. What we want is a way of creating a specialized COMPILE TIME TraceEventParser that 'knows' about this ETW provider (EventSource). This is what the TraceParserGen.exe tool does. The process is really very simple:

All ETW providers, including EventSource sources, have a XML file called a 'manifest' that describes all the events and their properties.
TraceParserGen takes a XML manifest and creates a C# file that defines a TraceEventParser for the provider described in the manifest.
You can then include this C# file in your application and get a first class (and efficient) experience.

In fact most of the parsers included in the TraceEvent library were generated using the TraceParserGen tool.

The TraceParserGen tool is one of the things that is built when you build the PerfView repository.
It ends up in the src\TraceParserGen\bin\Debug\net40 directory and you can run it from there.

Getting the XML Manifest

There are two main cases for ETW files that you may want TraceEventParser parsers for:

An EventSource (which is not registered with the OS)
An ETW provider registered with the OS (something returned from logman query providers)

In both cases, the PerfView tool is an excellent way of getting the manifest you need.

Creating XML Manifests for `EventSource` Sources

Collect an ETL file that events from the EventSource you want a manifest for. For example, collect a trace for the Microsoft-Demos-MySource EventSource do
text
```
PerfView /onlyProviders=*Microsoft-Demos-MySource collect
```
which produces the file PerfviewData.etl.zip
From this data file run the following command
text
```
PerfView /noGui userCommand DumpEventSourceManifests PerfVIewData.etl.zip
```
This command reads this data file and finds the manifests that the EventSource has logged to the ETL file and extracts them into a directory called Etwmanifests. You should find a file called Microsoft-Demos-MySource.manifest.xml in that directory which is the manifest you are trying to generate.

Creating XML Manifests OS registered ETW providers

The operating system provides a wealth of ETW providers. You can get an idea using the command

text

logman query providers

which gives a list. (See the section below on discovery for more). When you have found a provider you are interested in you can create its manifest by running the command:

text

PerfView /nogui userCommand DumpRegisteredManifest PROVIDER_NAME

For example running the command

text

PerfView /nogui userCommand DumpRegisteredManifest Microsoft-Windows-Kernel-File

will generate a manifest file Microsoft-Windows-Kernel-File.manifest.xml that describes the event for the OS ETW provider Microsoft-Windows-Kernel-File.

Converting to XML Manifest to a TraceEventParser for a Provider

TODO details on getting TraceParserGen.exe

Once you have the XML manifest file, the rest is trivial simply run

text

TraceParserGen ManifestFileName

And it will generate a corresponding C# file, which you can include in your application.

TODO EXAMPLE

Event Provider Discovery

One interesting aspect of the lifecycle of an event logging session is Event Provider discovery. Simply put, if you don't already 'know' that a provider exists, how do you find out? This is what the TraceEventProviders class is about.

The first problem that hits almost immediately is the fact that the fundamental ID associated with a provider is a GUID not a name. Clearly humans would prefer a name, but how to do you get from a name to the GUID (which is what the OS APIs want). There is another problem in that each provider has 64-bit bit-vector of 'keywords' which define groups of events that you can turn on and off independently. How do we discover what our 'keyword' possibilities are for any particular provider?

Traditionally, this was solved by manifest publication. The idea is that a provider would compile its manifest to a binary form (using a tool called MC.exe), attach it to a DLL as a resource, and then run a utility called wevtutil that will publish the manifest to the operating system. Thus the OS has a list of every published event provider and this list includes its name and descriptions of all its keywords. You can use the command:

text

logman query providers

To see a list of all registered providers and once you have the name of a particular provider (say Microsoft-Windows-Kernel-Process), you can get a description of its keywords by using the command

text

logman query providers Microsoft-Windows-Kernel-Process

The functionality of these two commands is also available in the TraceEventProviders class (see GetPublishedProviders and GetProviderKeywords).

There are a couple of problems with ETW's registration scheme. The first is that it requires a step at 'install' time, which is problematic for programs that wish to keep an 'xcopy' deployment characteristic. Second, at least currently, publishing a manifest with wevtutil requires administrative permissions, which is even more problematic for many scenarios.

It is possible publish the manifest for an EventSource using the wevtutil mechanism (this is what the EventRegister tool does), however this does not solve the problem.

EventSource sources will not show up on the lists above. As mentioned previously, because EventSource sources use a standard way of generating its provider GUID from its name, you can get the GUID from the name, but you can't get the names and descriptions of the keywords for a particular EventSource (although you can turn them all on blindly which actually works pretty well).

Publication vs. Registration

Publication is all putting provide schema information (the manifest) somewhere where event consumers can find it and is not really and is a very static activity (effectively some system wide database is updated). However when an EventProvider actually starts running (e.g. an EventSource is created), the provider needs to register itself with the operating system (using its provider ID). At this time the provider also registers a callback so that if at a later time a session enables a provider this callback is invoked to update the provider.

The operating system keeps track of the providers that have been registered (even if they have not been enabled), as well as a list of all provider that have been enabled by some session (even if there are no processes that actually have register that providers. This list is useful because it is a much smaller list of 'interesting' providers (those that you could turn on right now). You can get this list with the TraceEventProviders.GetRegisteredOrEnabledProviders method.

Perhaps more useful that this is to ask for a particular process what providers are actually registered in that process. You can get this information with the command line:

text

logman query providers -pid PROCESS_ID

or programmatically with the TraceEventProviders.GetRegisteredProvidersInProcess(int) method.

Unfortunately when a provider registers with the operating system (the Win32 EventRegister API), it only registers its provider GUID, and not its name. Thus the command above and the API above cannot give the name of the provider, just its GUID. For providers registered with the OS, this is not a problem since the name can be looked up (TraceEventProviders.GetProviderName), however this will not work for EventSource since they do not register their manifest. Because the EventSource is active, it is possible to actually 'ask it' for its manifest by opening a session (it will dump its manifest as part of its activation). However this does require more work. TODO INCOMPLETE.

On the fly filtering AND Logging using `ETWReloggerTraceEventSource`

TODO

--- METRICS ---

Files Extracted: 9
Estimated Token Budget: ~36537 tokens
Canonical Reference: https://codewiki.google/github.com/microsoft/perfview

Technical Documentation: microsoft/perfview

Overview (README.md) Copy Section

PerfView Overview

Getting PerfView Copy Section

Are you here about the TraceEvent Library? Copy Section

Not Sure if you should use PerfView or TraceEvent? Copy Section

Learning about PerfView Copy Section

Asking Questions / Reporting Bugs Copy Section

Building PerfView Yourself

Tools Needed to Build PerfView Copy Section

Cloning the PerfView GitHub Repository. Copy Section

How to Build and Debug PerfView Copy Section

Deploying your new version of Perfview Copy Section

Information for build troubleshooting. Copy Section

Running Tests Copy Section

Check in testing and code coverage statistica Copy Section

Contributing to PerfView Copy Section

Code Organization Copy Section

Documentation CodingStandards (`documentation/CodingStandards.md`) Copy Section

Coding Standard in the PerfView Codebase

Indenting and other spacing conventions. Copy Chapter

Layout of a Class Copy Chapter

Naming conventions Copy Chapter

Minimum Commenting Copy Chapter

When in Doubt Copy Chapter

See Also

Documentation Downloading (`documentation/Downloading.md`) Copy Section

Downloading PerfView

PerfView Releases

Shortcut to Download the Latest PerfView.exe Copy Chapter

Explicitly Validating PerfView's Digital Signature Copy Section

Additional Release Information

Microsoft Download Site Copy Chapter

Documentation OpenSourceGitWorkflow (`documentation/OpenSourceGitWorkflow.md`) Copy Section

Setting Up *With* a Fork with Visual Studio

Creating a 'upstream' Repository alias. Copy Section

Pulling updates from the Main (upstream) Repository into your Fork Copy Chapter

Step 1: Fetching All Commits for the Upstream Repository.

Step 2: Merging the LOCAL repository's main branch to include changes from upstream/main

Step 3: Pushing the LOCAL repository to your GitHub Fork

Creating a pull request to the Main Repository. Copy Chapter

Documentation Scenarios (`documentation/Scenarios.md`) Copy Section

PerfView vs. TraceEvent

Start With PerfView If Your Goal Is To... Copy Chapter

Start With TraceEvent If Your Goal Is To... Copy Chapter

PerfView Limitations Copy Chapter

Documentation SettingUpRepoInVS (`documentation/SettingUpRepoInVS.md`) Copy Section

GitHub Repository Setup with Visual Studio 2026

Background: GitHub Repos, Commits Branches and GitHub Copy Chapter

Branches Copy Section

Two Possible Repository Setups Copy Chapter

Simple GIT Repository Setup Copy Section

Open Source Projects and Pull Requests Copy Section

To Fork or not to Fork? Copy Section

Documentation SimpleGitWorkflow (`documentation/SimpleGitWorkflow.md`) Copy Section

Setting Up *Without* a Fork with Visual Studio 2026

Making changes Copy Chapter

Committing Trivial Changes to Master Branch Copy Chapter

Don't use Master Branch for Non-trivial Features. Copy Chapter

Switching Branches Copy Chapter

Synchronizing the GitHub repository and the Local Repository. Copy Chapter

Discarding Changes Copy Chapter

Viewing history Copy Chapter

Review Copy Chapter

Documentation TraceEvent TraceEventLibrary (`documentation/TraceEvent/TraceEventLibrary.md`) Copy Section

The Microsoft.Diagnostics.Tracing.TraceEvent Library

Supported Windows Versions Copy Chapter

TraceEvent Samples Copy Chapter

Release Notes for the Microsoft.Diagnostics.Tracing.TraceEvent library Copy Chapter

Documentation TraceEvent TraceEventProgrammersGuide (`documentation/TraceEvent/TraceEventProgrammersGuide.md`) Copy Section

The TraceEvent Library Programmers Guide

Introduction: Strongly Typed (Semantic) Logging Copy Chapter

Basic Logging Architecture Copy Chapter

Multi-player Copy Section

Asynchronous Copy Section

Quick Start Example Copy Chapter

Component 1: the Event Provider (`EventSource`) Copy Section

Component 2: The Event Session (`TraceEventSession`) Copy Section

Component 3: The Event Processor (`ETWTraceEventSource`) Copy Section

Event Parsing 'Magic' (`TraceEventParser` and derived types) Copy Chapter

Overview (README.md)

Getting PerfView

Are you here about the TraceEvent Library?

Not Sure if you should use PerfView or TraceEvent?

Learning about PerfView

Asking Questions / Reporting Bugs

Tools Needed to Build PerfView

Cloning the PerfView GitHub Repository.

How to Build and Debug PerfView

Deploying your new version of Perfview

Information for build troubleshooting.

Running Tests

Check in testing and code coverage statistica

Contributing to PerfView

Code Organization

Documentation CodingStandards (`documentation/CodingStandards.md`)

Indenting and other spacing conventions.

Layout of a Class

Naming conventions

Minimum Commenting

When in Doubt

Documentation Downloading (`documentation/Downloading.md`)

Shortcut to Download the Latest PerfView.exe

Explicitly Validating PerfView's Digital Signature

Microsoft Download Site

Documentation OpenSourceGitWorkflow (`documentation/OpenSourceGitWorkflow.md`)

Setting Up With a Fork with Visual Studio

Creating a 'upstream' Repository alias.

Pulling updates from the Main (upstream) Repository into your Fork

Creating a pull request to the Main Repository.

Documentation Scenarios (`documentation/Scenarios.md`)

Start With PerfView If Your Goal Is To...

Start With TraceEvent If Your Goal Is To...

PerfView Limitations

Documentation SettingUpRepoInVS (`documentation/SettingUpRepoInVS.md`)

Background: GitHub Repos, Commits Branches and GitHub

Branches

Two Possible Repository Setups

Simple GIT Repository Setup

Open Source Projects and Pull Requests

To Fork or not to Fork?

Documentation SimpleGitWorkflow (`documentation/SimpleGitWorkflow.md`)

Setting Up Without a Fork with Visual Studio 2026

Making changes

Committing Trivial Changes to Master Branch

Don't use Master Branch for Non-trivial Features.

Switching Branches

Synchronizing the GitHub repository and the Local Repository.

Discarding Changes

Viewing history

Review

Documentation TraceEvent TraceEventLibrary (`documentation/TraceEvent/TraceEventLibrary.md`)

Supported Windows Versions

TraceEvent Samples

Release Notes for the Microsoft.Diagnostics.Tracing.TraceEvent library

Documentation TraceEvent TraceEventProgrammersGuide (`documentation/TraceEvent/TraceEventProgrammersGuide.md`)

Introduction: Strongly Typed (Semantic) Logging

Basic Logging Architecture

Multi-player

Asynchronous

Quick Start Example

Component 1: the Event Provider (`EventSource`)

Component 2: The Event Session (`TraceEventSession`)

Component 3: The Event Processor (`ETWTraceEventSource`)

Event Parsing 'Magic' (`TraceEventParser` and derived types)

Static vs. Dynamic `TraceEventParser` parsers

Lifetime constraints on `TraceEvent` objects

Review of the Fundamental TraceEvent Architecture

What You Can Do With the TraceEvent Library

ETW Limitations

Next Step: Code Samples

Diagnostic Techniques: Using PerfView

Full dumps of ETW events in PerfView

PerfView's `/keepAllEvents` feature

Debugging EventSource Authoring

Real Time Processing of Events

Real Time Processing with `IObservable`

Best Practices for Versioning ETW Events

Higher level Processing: `TraceLog`

Creating a `TraceLog` (an ETLX file)