# Technical Documentation: dotnet/roslyn > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/dotnet/roslyn) ### Overview (README.md)

Roslyn logo

The .NET Compiler Platform

Roslyn is the open-source implementation of both the C# and Visual Basic compilers with an API surface for building code analysis tools. ### C# and Visual Basic Language Feature Suggestions If you want to suggest a new feature for the C# or Visual Basic languages go here: - [dotnet/csharplang](https://github.com/dotnet/csharplang) for C# specific issues - [dotnet/vblang](https://github.com/dotnet/vblang) for VB-specific features - [dotnet/csharplang](https://github.com/dotnet/csharplang) for features that affect both languages ### Contributing All work on the C# and Visual Basic compiler happens directly on [GitHub](https://github.com/dotnet/roslyn). Both core team members and external contributors send pull requests which go through the same review process. If you are interested in fixing issues and contributing directly to the code base, a great way to get started is to ask some questions on [GitHub Discussions](https://github.com/dotnet/roslyn/discussions)! Then check out our [contributing guide](https://github.com/dotnet/roslyn/blob/main/CONTRIBUTING.md) which covers the following: - [Coding guidelines](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Contributing-Code.md) - [The development workflow, including debugging and running tests](https://github.com/dotnet/roslyn/blob/main/docs/contributing/Building%2C%20Debugging%2C%20and%20Testing%20on%20Windows.md) - [Submitting pull requests]() - Finding a bug to fix in the [IDE](https://aka.ms/roslyn-ide-bugs-help-wanted) or [Compiler](https://aka.ms/roslyn-compiler-bugs-help-wanted) - Finding a feature to implement in the [IDE](https://aka.ms/roslyn-ide-feature-help-wanted) or [Compiler](https://aka.ms/roslyn-compiler-feature-help-wanted) - Roslyn API suggestions should go through the [API review process]() ### Community The Roslyn community can be found on [GitHub Discussions](https://github.com/dotnet/roslyn/discussions), where you can ask questions, voice ideas, and share your projects. To chat with other community members, you can join the Roslyn channel on the [CSharp Community Discord](https://discord.com/invite/tGJvv88). Our [Code of Conduct](CODE-OF-CONDUCT.md) applies to all Roslyn community channels and has adopted the [.NET Foundation Code of Conduct](https://dotnetfoundation.org/code-of-conduct). ### Documentation Visit [Roslyn Architecture Overview](https://docs.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/compiler-api-model) to get started with Roslyn’s API’s. ### NuGet Feeds **The latest pre-release builds** are available from the following public NuGet feeds: - [Compiler](https://dev.azure.com/dnceng/public/_packaging?_a=feed&feed=dotnet-tools): `https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json` - [IDE Services](https://dev.azure.com/azure-public/vside/_packaging?_a=feed&feed=vssdk): `https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json` - [.NET SDK](https://dev.azure.com/dnceng/public/_packaging?_a=feed&feed=dotnet5): `https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json` [//]: # (Begin current test results) ### Continuous Integration status #### Builds |Branch|Windows Debug|Windows Release|Unix Debug| |:--:|:--:|:--:|:--:| **main**|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)| #### Desktop Unit Tests |Branch|Debug x86|Debug x64|Release x86|Release x64| |:--:|:--:|:--:|:--:|:--:| **main**|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)| #### CoreClr Unit Tests |Branch|Windows Debug|Windows Release|Linux| |:--:|:--:|:--:|:--:| **main**|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)| #### Integration Tests |Branch|Debug x86|Debug x64|Release x86|Release x64 |:--:|:--:|:--:|:--:|:--:| **main**|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=96&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=96&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=96&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=96&branchname=main&view=logs)| #### Misc Tests |Branch|Determinism|Analyzers|Build Correctness|Source build|TODO/Prototype|Spanish|MacOS| |:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| **main**|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[&configuration=Source-Build+(Managed)&label=build)](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)|[](https://dev.azure.com/dnceng-public/public/_build/latest?definitionId=95&branchname=main&view=logs)| [//]: # (End current test results) ### .NET Foundation This project is part of the [.NET Foundation](http://www.dotnetfoundation.org/projects) along with other projects like [the .NET Runtime](https://github.com/dotnet/runtime/). --- ### README (`docs/README.md`) README ====== This file is a placeholder. This directory will contain public documentation. --- ### Compilers README (`docs/compilers/README.md`) # Roslyn Compiler ## Specification The compiler specification details the supported (and semi-supported) surface area of the Roslyn VB and C# compilers. This includes 0. Command-line switches and their meaning 1. Breaking changes from previous versions of the compilers 2. Compiler behaviors that are (intentionally) contrary to the specification 3. Compiler features not described by the language specification 1. COM-specific and other Microsoft-specific features 2. "Well-known" attributes that affect compiler behavior 3. The "ruleset" file syntax and semantics 4. Features included for interoperability between C# and VB, for example 1. Named Indexers use from C# 5. Places where the compiler behavior diverges from the language specification 6. Limitations (e.g. identifier length) 7. History of language changes per version The language specification itself is not included here. ## Platforms The compiler is officially supported in the following configurations: - .NET Framework: x86, x64 and ARM64 - .NET Core: - Operating Systems: Windows, macOS, Linux - Architectures: x86, x64, and ARM64 - Mono Core: x86 and x64 The compiler takes contributions for other platforms, such as big-endian architectures, but does not officially support the compiler there. --- ### Contributing README (`docs/contributing/README.md`) Contributing === Guides for contributing to this project - Build, debugging and testing - [Unix](Building%2C%20Debugging%2C%20and%20Testing%20on%20Unix.md) - [Windows](Building%2C%20Debugging%2C%20and%20Testing%20on%20Windows.md) - [Localization in tests](Localization%20In%20Compiler%20Tests.md) - [Nullable Annotations](Nullable%20Annotations.md) - [Target Framework Strategy](target-framework-strategy.md) - [Bootstrap Builds](bootstrap-builds.md) - [API Review Process](API%20Review%20Process.md) - [Developing a Language Feature](Developing%20a%20Language%20Feature.md) - [Documentation for IDE CodeStyle analyzers](Documentation%20for%20IDE%20CodeStyle%20analyzers.md) - [IDE Design Process](ide_design_process.md) - [Powershell Guidelines](Powershell%20Guidelines.md) - [Codespaces](codespaces.md) - [Testing for Interactive readiness](Testing%20for%20Interactive%20readiness.md) - [Compiler Test Plan](Compiler%20Test%20Plan.md) - [IDE Test Plan](IDE%20Test%20Plan.md) --- ### Ide README (`docs/ide/README.md`) # Recently added productivity features and links to release notes ## TOC * [Dev16](#prod-16) * [Dev 16.9](#prod-16-9) * [Dev 16.8](#prod-16-8) * [Dev 16.7](#prod-16-7) * [Dev 16.6](#prod-16-6) * [Dev 16.5](#prod-16-5) * [Dev 16.4](#prod-16-4) * [Dev 16.3](#prod-16-3) * [Dev 16.2](#prod-16-2) * [Dev 16.1](#prod-16-1) * [Dev16.0](#prod-16-0) * [Dev15](#prod-15) * [Dev 15.9](#prod-15-9) * [Dev 15.8](#prod-15-8) * [Dev 15.7](#prod-15-7) * [Dev 15.6](#prod-15-6) * [Dev 15.5](#prod-15-5) * [Dev 15.4](#prod-15-4) ## dev16 ### 16.9 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.9)) * There is now IntelliSense completion for preprocessor symbols. * Solution Explorer now displays the new .NET 5.0 Source Generators. * Go To All won't display duplicate results across netcoreapp3.1 and netcoreapp2.0. * Quick Info now displays compiler warning IDs or numbers for suppressions. * Using directives will now automatically get added when copying and pasting types to a new file. * When pressing `;` to accept a method from a completion list, IntelliSense will now automatically insert the parentheses along with a semicolon for object creation and method calls. * Semantic colorization for C# 9.0 [records](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-9#record-types&preserve-view=true). * Refactoring that removes unnecessary discards. * Refactoring that converts a verbatim and regular string to an interpolated string preserving curly braces that were intended to go in the output. * Code fix in Visual Basic that removes the shared keyword when you convert methods that are shared to a module. * A refactoring that suggests using `new(…)` in non-contentious scenarios * A code fix that removes redundant equality expressions for both C# and Visual Basic * The .NET Code Style (IDE) analyzers can now be enforced on build * The Syntax Visualizer shows the current foreground color for enhanced colors * A new tooltip when hovering over the diagnostic ID for pragma warnings * When you type the return key from within a comment the new line is now automatically commented out * Inline parameter name hints enhancements * .NET Core Debugging with WSL 2 ### 16.8 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.8)) * Roslyn analyzers are now included in the .NET 5.0 SDK * Refactoring that introduces the new C# 9 `not` pattern matching syntax when a suppression operator is present * Inline method refactoring to help replace usages of static, instance, and extension method within a single statement body * Code fix to convert instances of `typeof` to `nameof` in C# and `GetType` to `NameOf` in Visual Basic * C# and Visual Basic support for inline parameter name hints that inserts adornments for literals, casted literals, and object instantiations prior to each argument in function calls * Refactoring that extracts members from a selected class to a new base class in both C# and Visual Basic * Code cleanup has new configuration options that can apply formatting and file header preferences set in your EditorConfig file across a single file or an entire solution * Code fix to remove the `in` keyword where the argument should not be passed by reference * Refactoring that introduces the new C#9 pattern combinators and pattern matching suggestions such as converting `==` to use `is` where applicable * Code fix to make a class abstract when you are trying to write an abstract method in a class that is not abstract * IntelliSense completion in `DateTime` and `TimeSpan` string literals automatically appear when the first quote is typed * Code fix to remove unnecessary `pragma suppressions` and unnecessary `SuppressMessageAttributes` * `Rename` and `Find All` References understands references to symbols within the target string of global `SuppressMessageAttributes` * _ByVal_ fades to say it's not necessary along with a code fix to remove the unnecessary _ByVal_ in Visual Basic * Interactive window support for multiple runtimes, such as .NET Framework and .NET Core. * Added a new _RegisterAdditionalFileAction_ API that allows analyzer authors to create an analyzer for additional files. ### 16.7 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.7)) * There is now a warning and code fix when a suppression operator is present but has no effect * A second code fix suggesting the correct negating expression is also available * You can now generate properties when generating a constructor in a type * Quick Info now displays the diagnostic ID along with a help link * There is now a quick action to add a debugger display attribute to a class * There is now a code fix for accidental assignments or comparisons to the same variable * You can now generate comparison operators for types that implement IComparable * You can now generate IEquatable operators when generating .Equals for structs * You can now create and assign properties or fields for all unused constructor parameters * There is now IntelliSense completion in DateTime and TimeSpan string literals * You can now add a parameter within the Change Signature dialog * Analyzer authors can now use CompletionProviders for IntelliSense completions when shipping their analyzers with NuGet ### 16.6 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.6)) * Add explicit cast code fix * Simplify conditional expressions refactoring * Convert regular string literals to verbatim string literals (and back) refactoring * Bulk configure the severity level of analyzer categories directly through the editor * Quick Info style support for XML comments that contain returns and value tags * Add file headers to existing files, projects, and solutions with EditorConfig * [Microsoft Fakes support for .NET Core](https://docs.microsoft.com/visualstudio/releases/2019/release-notes#microsoft-fakes-for-net-core-and-sdk-style-projects) and SDK-style projects ### 16.5 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.5)) * You can now use System.HashCode to implement the GetHashCode method when System.HashCode is available. * Refactoring to convert if statements to switch statements or switch expressions. * IntelliSense completion for unimported extension methods. You will first need to turn this option on in **Tools > Options > Text Editor > C# > Intellisense > and select Show items from unimported namespaces**. ### 16.4 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.4)) * You can now configure the severity level of a code style rule directly through the editor (through the **Ctrl+.** menu) or the error list (right-click on the error, warning, or message). This will also update EditorConfig and works for 3rd party analyzers. * Find All References now allows you to group by type and member. * Refactoring make local function static * Code fix to pass variable explicitly in local static function. * The Go To Base command can navigate up the inheritance chain. Available in the context (right-click) menu or you can type (Alt+Home). * Refactoring to add null checks for all parameters. * Methods that have no XML documentation can now automatically inherit XML documentation from the method it is overriding. Place your cursor over the undocumented method that implements a documented interface method. Quick Info will then display the XML documentation from the interface method. ### 16.3 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.3)) * Wrap chains of fluent calls with a refactoring * Introduce a local variable immediately after writing its initializer * Code Analysis page now in project properties. Right click on the project name within the solution explorer and select properties. Select Code Analysis to install analyzer packages and configure when to run code analysis. * Now, for users who turn off the completion for unimported types, it's much easier to get it back in the completion list with the new imported type filter added to the IntelliSense toggles. * There is now Quick Info style support for XML comments. Place the cursor over the method name. Quick Info will then display the supported styles from the XML comments above the code. * Rename a file when renaming an interface, enum, or class. Place the cursor in the class name and type (Ctrl + R,R) to open the Rename dialogue and check the ‘Rename file’ box. * There is now Edit and Continue support for multi-targeted projects which includes modules loaded multiple times in the same process on different domains or load contexts. In addition, developers can edit source files even when the containing project is not loaded or the application is running. ### 16.2 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.2)) * Bring back Sort Usings as a separate command from Remove Usings. Available under Edit > IntelliSense. * Convert a switch statement to a [switch expression](https://docs.microsoft.com/dotnet/csharp/whats-new/csharp-8#switch-expressions) (verify you are using C# 8 to get the switch expression feature) * Generate a parameter * [Test Explorer improvements](https://docs.microsoft.com/visualstudio/releases/2019/release-notes#test-explorer) * Significant reduction in memory consumed by the Visual Studio process and faster test discovery for solutions with large numbers of tests * Filter buttoms for passed, failed, not run tests * Additional buttons for the 'Run Failed Tests' and 'Run Previous Test Run' * Customize what columns are displayed in the Test Explorer * Specify what is displayed in each tier of the test hierarchy. The default tiers are Project, Namespace, and then Class, but additional options include Outcome or Duration groupings. * The test status window (the pane below the test list that displays the messages, output, etc.) is much more usable. Users can copy substrings of text, and the font-width is fixed for more readable output. * Playlists can be displayed in multiple tabs and are much easier to create and discard as needed. * Live Unit Testing now has its own view in the Test Explorer. It displays all tests currently included in Live Unit Testing (aka. the live test set), so testers can easily keep track of Live Unit Testing results separate from the manually run test results. * There is a target framework column that can display multi-targeted test results. ### 16.1 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.1)) * There is now experimental intellisense completion for unimported types! You now receive intellisense suggestions for types in dependencies in your project even if you have not yet added the import statement to your file. You must turn this option on in Tools > Options > Text Editor > C# > Intellisense. * Toggle Single Line Comment/Uncomment is now available through the keyboard shortcut (Ctrl+K,/). This command will add or remove a single line comment depending on whether your selection is already commented. * You can now export naming styles with the “Generate editorconfig” button located in Tools > Options > Text Editor > C# > Code Style. * You can now use a new editorconfig code style rule to require or prevent usings inside a namespace. This setting will also be exported when you use the “Generate editorconfig” button located in Tools > Options > Text Editor > C# > Code Style. * The Find All References “Kind” column now has more filter options and is aware of namespaces and types. * .NET codefixes and Refactorings * Split/Merge if * Wrap binary expressions * Unseal a class * A regex completion list can now be accessed through the intellisense menu (Ctrl + space) when inside a regex string. These completions also include an in-line description of what the suggestion does. * You can now use one-click code cleanup for projects and solutions. You can right-click on projects or the solution in the Solution Explorer and select ‘Run Code Cleanup’. * You can now use a refactoring dialog to move type to namespace or folder. Place your cursor in the class name and type (Ctrl + .) to open the quick actions and refactorings menu and select ‘Move to namespace.’ This launches a dialog where you can select the target namespace you would like to move the type to. * Toggle Block Comment/Uncomment (Ctrl+Shift+/) or through Edit > Advanced > Toggle Block Comment. * There is now a codefix for making readonly struct fields writable. Place your cursor in the struct name, type (Ctrl+.) to open the quick actions and refactorings menu, and select ‘Make readonly fields writable.’ * The codefix to add a private field from a constructor and vice versa is easier to discover and will show when any portion of the field name is selected. This refactoring now also offers all possible constructors. ### 16.0 ([release notes](https://docs.microsoft.com/visualstudio/releases/2019/release-notes-v16.0)) * .NET refactorings and codefixes: * Sync Namespace and Folder Name * Pull members up refactoring with dialog options * Wrap/indent/align lists of parameters/arguments * Convert anonymous type to tuple * Use expression/block body for lambda * Invert conditional expressions and logical operations * Convert to compound assignment * Automatically close block comment on “/” * Convert to compound assignment * Fix Implicitly-typed variables cannot be constant * Auto-fixer to replace @$" with $@" when typing interpolated verbatim string * Completion for #nullable enable|disable * Fix for unused expression values and parameters * Fix for allowing Extract Interface to remain in the same file * For cases where "await" is implied but omitted, there is now a compiler warning. * Convert a local function to a method. * Convert a tuple to a named-struct. * Convert an anonymous type to a class. * Convert an anonymous type to a tuple. * For a foreach loop to LINQ query or to LINQ method. * Generate deconstruct method * Categorize references by Read/Write in Find All References window. * Add Editorconfig when_multiline option for csharp_prefer_braces. * New classification colors are available from the .NET Compiler Platform SDK (aka Roslyn). New default colors, similar to the Visual Studio Code colors, are gradually being rolled out. You can adjust these colors in Tools > Options > Environment > Fonts and Colors or turn them off in Environment > Preview Features by unchecking the Use enhanced colors check box. We’d appreciate hearing feedback on how this change affects your workflow. * Configure [code cleanup](https://docs.microsoft.com/visualstudio/ide/code-styles-and-code-cleanup) (settings formally a part of Format Document) * [Format document global tool](https://docs.microsoft.com/visualstudio/releases/2019/release-notes#-apply-code-style-preferences) can be run from the command-line * Use [code metrics](https://docs.microsoft.com/visualstudio/code-quality/code-metrics-values) with .NET Core projects with our added compatibility. * Export editor settings to an Editorconfig file through **Tools > Options > Text Editor > C# > Code Style** with the button "Generate .editorconfig file from settings". * Use C# and Visual Basic's new Regex parser support. Regular expressions are now recognized, and language features are enabled on them. Regex strings are either recognized when a string is passed to the Regex constructor or when a string is immediately preceded with a comment containing the string language=regex. The language features included in this release are classification, brace matching, highlight references, and diagnostics. * You can now use dead code analysis for unused private members with an optional code fix to remove unused member declaration. * The Find References feature on an accessor now only returns results for that accessor. * "Using" statements can be added when code is pasted into a file. A code fix appears after pasting recognized code that prompts you to add relevant missing imports. * You can now use Find All References (Shift-F12) and CodeLens to show results from Razor (.cshtml) files in .NET Core projects. You can then navigate to the identified code in the relevant Razor files. * You will now receive a warning when running code analysis using FxCop. .NET Compiler analyzers are the recommended way to perform code analysis going forward. Read more on migrating to .NET compiler platform analyzers. ## dev15 ### dev15 Summary * Visual Studio-based code cleanup via Format Document * Refactorings: InvertIf, add parameter from method callsite, remove unnecessary parentheses, convert if-else assignment, completion for attributes and return to ternary conditional. * .editorconfig “refresh” such that changes will be automatically applied without file reload * IOperation APIs RTW * Portable PDB format to enable cross-platform debugging scenarios ### 15.9 None ### 15.8 ([release notes](https://docs.microsoft.com/visualstudio/releasenotes/vs2017-relnotes-v15.8#productivity)) * Format Document (Ctrl + K, D or Ctrl + E, D) for C# development. * .NET refactorings and codefixes: * Invert If * Add parameter from method callsite * Remove unnecessary parentheses * Use ternary conditionals in assignments and return statements * Go to Enclosing Block (Ctrl + Alt + UpArrow) * Go to Next/Previous Issue (Alt + PgUp/PgDn) skip to error, squiggle, lightbulb. * Go to Member (Ctrl + T, M) is now scoped to the file by default * Multi-caret: Insert carets with Ctrl + Alt + LeftMouseClick. * Multi-caret: Add a selection and caret at next location that matches current selection with Shift + Alt + Ins. * Access a contextual navigation menu with Alt + `. * Visual Studio Code and ReSharper (Visual Studio) keybindings ### 15.7 ([release notes](https://docs.microsoft.com/visualstudio/releasenotes/vs2017-relnotes-v15.7#dotnet_productivity)) * .NET refactorings and codefixes: * For-to-foreach, and vice versa. * Make private fields readonly. * Toggle between var and the explicit type, regardless of your code style preferences. * Go To Definition (F12) is now supported for LINQ query clauses and deconstructions. * Quick Info shows captures on lambdas and local functions, so you can see what variables are in scope. * Change Signature refactoring (Ctrl+. on signature) works on local functions. * You can edit .NET Core project files in-place, so opening containing folder, restoring tabs, and other Editor features are fully supported. IDE changes, such as adding a linked file, will be merged with unsaved changes in the editor. ### 15.6 ([release notes](https://docs.microsoft.com/visualstudio/releasenotes/vs2017-relnotes-v15.6#productivity)) * Navigate to decompiled sources (must enable in Tools > Options > Text Editor > C# > Advanced > Enable navigation to decompiled sources) * .NET EditorConfig options: dotnet_prefer_inferred_tuple_names * .NET EditorConfig options: dotnet_prefer_inferred_anonymous_type_member_names * Ctrl + D to Duplicate text * Expand Selection command that allows you to successively expand your selection to the next logical block. You can use the shortcuts Shift+Alt+= to expand and Shift+Alt+- to contract the current selection. ### 15.5 None ### 15.4 ([release notes](https://docs.microsoft.com/visualstudio/releasenotes/vs2017-relnotes-v15.4#editor)) * Editor: Control Click Go To Definition --- ### Roslyn Analyzers README (`docs/roslyn-analyzers/README.md`) # Roslyn Analyzers ## Microsoft.CodeAnalysis.Analyzers *Latest stable version:* [](https://www.nuget.org/packages/Microsoft.CodeAnalysis.Analyzers) *Latest pre-release version:* [here](https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet7/NuGet/Microsoft.CodeAnalysis.Analyzers/versions) This package contains rules for correct usage of APIs from the [Microsoft.CodeAnalysis](https://www.nuget.org/packages/Microsoft.CodeAnalysis) NuGet package, i.e. .NET Compiler Platform ("Roslyn") APIs. These are primarily aimed towards helping authors of diagnostic analyzers and code fix providers to invoke the Microsoft.CodeAnalysis APIs in a recommended manner. [More info about rules in this package](../../src/RoslynAnalyzers/Microsoft.CodeAnalysis.Analyzers/Microsoft.CodeAnalysis.Analyzers.md) ## Roslyn.Diagnostics.Analyzers *Latest stable version:* [](https://www.nuget.org/packages/Roslyn.Diagnostics.Analyzers) *Latest pre-release version:* [here](https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet7/NuGet/Roslyn.Diagnostics.Analyzers/versions) This package contains rules that are very specific to the .NET Compiler Platform ("Roslyn") project, i.e. [dotnet/roslyn](https://github.com/dotnet/roslyn) repo. This analyzer package is *not intended for general consumption* outside the Roslyn repo. [More info about rules in this package](../../src/RoslynAnalyzers/Roslyn.Diagnostics.Analyzers/Roslyn.Diagnostics.Analyzers.md) ## Microsoft.CodeAnalysis.BannedApiAnalyzers *Latest stable version:* [](https://www.nuget.org/packages/Microsoft.CodeAnalysis.BannedApiAnalyzers) *Latest pre-release version:* [here](https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet7/NuGet/Microsoft.CodeAnalysis.BannedApiAnalyzers/versions) This package contains customizable rules for identifying references to banned APIs. [More info about rules in this package](../../src/RoslynAnalyzers/Microsoft.CodeAnalysis.BannedApiAnalyzers/Microsoft.CodeAnalysis.BannedApiAnalyzers.md) For instructions on using this analyzer, see [Instructions](../../src/RoslynAnalyzers/Microsoft.CodeAnalysis.BannedApiAnalyzers/BannedApiAnalyzers.Help.md). ## Microsoft.CodeAnalysis.PublicApiAnalyzers *Latest stable version:* [](https://www.nuget.org/packages/Microsoft.CodeAnalysis.PublicApiAnalyzers) *Latest pre-release version:* [here](https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet7/NuGet/Microsoft.CodeAnalysis.PublicApiAnalyzers/versions) This package contains rules to help library authors monitoring change to their public APIs. [More info about rules in this package](../../src/RoslynAnalyzers/PublicApiAnalyzers/Microsoft.CodeAnalysis.PublicApiAnalyzers.md) For instructions on using this analyzer, see [Instructions](../../src/RoslynAnalyzers/PublicApiAnalyzers/PublicApiAnalyzers.Help.md). --- ### Wiki README (`docs/wiki/README.md`) # Getting Started * [Roslyn Overview](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Roslyn-Overview.md) * [Samples and Walkthroughs](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Samples-and-Walkthroughs.md) * [Roadmap](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Roadmap.md) * [Labels used for issues](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Labels-used-for-issues.md) * [Building, Testing, and Debugging](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Building-Testing-and-Debugging.md) * [Contributing](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Contributing-Code.md) * [FAQ](https://github.com/dotnet/roslyn/blob/main/docs/wiki/FAQ.md) * [Documentation](https://github.com/dotnet/roslyn/blob/main/docs) # Language Design * [C# Language Design repo](https://github.com/dotnet/csharplang) * [VB Language Design repo](https://github.com/dotnet/vblang) * [Language Feature Status](https://github.com/dotnet/roslyn/blob/main/docs/Language%20Feature%20Status.md) * [Language Feature Overview for C# 6 and Visual Basic 14](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Languages-features-in-C%23-6-and-VB-14.md) # Tools * [Performance considerations for large solutions](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Performance-considerations-for-large-solutions.md) * [Syntax Visualizer](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Syntax-Visualizer.md) * [Syntax Quoter](http://roslynquoter.azurewebsites.net), [Source](https://github.com/KirillOsenkov/RoslynQuoter) * [Enhanced source view](http://sourceroslyn.io), powered by Roslyn # API Changelist * [VS 2015 CTP5 Changelist](https://github.com/dotnet/roslyn/blob/main/docs/wiki/VS-2015-CTP-5-API-Changes.md) * [VS 2015 CTP6 Changelist](https://github.com/dotnet/roslyn/blob/main/docs/wiki/VS-2015-CTP-6-API-Changes.md) * [VS 2015 RC Changelist](https://github.com/dotnet/roslyn/blob/main/docs/wiki/VS-2015-RC-API-Changes.md) # Other Topics * [Project Overview](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Roslyn-Overview.md) * [Project Roadmap](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Roadmap.md) * [Labels used for Issues](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Labels-used-for-issues.md) * [Performance considerations for large solutions](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Performance-considerations-for-large-solutions.md) * [Other documentation in the repo](https://github.com/dotnet/roslyn/blob/main/docs/) ## Contributors * [Building, Testing, and Debugging](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Building-Testing-and-Debugging.md) * [Contributing Code](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Contributing-Code.md) ## Tool Authors * [Getting Started](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Getting-Started-on-Visual-Studio-2015.md) * [Samples and Walkthroughs](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Samples-and-Walkthroughs.md) * [FAQ](https://github.com/dotnet/roslyn/blob/main/docs/wiki/FAQ.md) * [API Changelist](https://github.com/dotnet/roslyn/blob/main/docs/wiki/API-Changes.md) ## Status * [Language feature status](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Languages-features-in-C%23-6-and-VB-14.md) * [Edit-and-Continue Supported Edits](https://github.com/dotnet/roslyn/blob/main/docs/wiki/EnC-Supported-Edits.md) ## .NET Core * [Runtime Code Generation](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Runtime-code-generation-using-Roslyn-compilations-in-.NET-Core-App.md) ## Interactive and Scripting * [Interactive Window Documentation](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Interactive-Window.md) * [Scripting API Samples](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Scripting-API-Samples.md) * [C# Interactive Walkthrough](https://github.com/dotnet/roslyn/blob/main/docs/wiki/C%23-Interactive-Walkthrough.md) ## Testing * [Manual Testing](https://github.com/dotnet/roslyn/blob/main/docs/wiki/Manual-Testing.md) * [How to Investigate CI Test Failures](https://github.com/dotnet/roslyn/blob/main/docs/wiki/How-To-Investigate-CI-Test-Failures.md) --- ### Wiki Roslyn Overview (`docs/wiki/Roslyn-Overview.md`) # .NET Compiler Platform ("Roslyn") Overview [Download in PDF](https://docs.microsoft.com/en-us/dotnet/opbuildpdf/csharp/roslyn-sdk/toc.pdf?branch=live) Contents * [Introduction](#introduction) * [Exposing the Compiler APIs](#exposing-the-compiler-apis) * [Compiler Pipeline Functional Areas](#compiler-pipeline-functional-areas) * [API Layers](#api-layers) * [Compiler APIs](#compiler-apis) * [Workspaces APIs](#workspaces-apis) * [Working with Syntax](#working-with-syntax) * [Syntax Trees](#syntax-trees) * [Syntax Nodes](#syntax-nodes) * [Syntax Tokens](#syntax-tokens) * [Syntax Trivia](#syntax-trivia) * [Spans](#spans) * [Kinds](#kinds) * [Errors](#errors) * [Working with Semantics](#working-with-semantics) * [Compilation](#compilation) * [Symbols](#symbols) * [Semantic Model](#semantic-model) * [Working with a Workspace](#working-with-a-workspace) * [Workspace](#workspace) * [Solutions, Projects and Documents](#solutions-projects-documents) # Summary ## Introduction Traditionally, compilers are black boxes -- source code goes in one end, magic happens in the middle, and object files or assemblies come out the other end. As compilers perform their magic, they build up deep understanding of the code they are processing, but that knowledge is unavailable to anyone but the compiler implementation wizards. The information is promptly forgotten after the translated output is produced. For decades, this world view has served us well, but it is no longer sufficient. Increasingly we rely on integrated development environment (IDE) features such as IntelliSense, refactoring, intelligent rename, “Find all references,” and “Go to definition” to increase our productivity. We rely on code analysis tools to improve our code quality and code generators to aid in application construction. As these tools get smarter, they need access to more and more of the deep code knowledge that only compilers possess. This is the core mission of Roslyn: opening up the black boxes and allowing tools and end users to share in the wealth of information compilers have about our code. Instead of being opaque source-code-in and object-code-out translators, through Roslyn, compilers become platforms—APIs that you can use for code related tasks in your tools and applications. The transition to compilers as platforms dramatically lowers the barrier to entry for creating code focused tools and applications. It creates many opportunities for innovation in areas such as meta-programming, code generation and transformation, interactive use of the C# and VB languages, and embedding of C# and VB in domain specific languages. Roslyn SDK Preview includes the latest drafts of new language object models for code generation, analysis, and refactoring. We hope to include drafts of API support for scripting and interactive use of C# and Visual Basic in a future preview. This document provides a conceptual overview of Roslyn. Further details can be found in the walkthroughs and samples included in the SDK Preview. ## Exposing the Compiler APIs ### Compiler Pipeline Functional Areas Roslyn exposes the C# and Visual Basic compiler’s code analysis to you as a consumer by providing an API layer that mirrors a traditional compiler pipeline. Each phase of this pipeline is now a separate component. First the parse phase, where source is tokenized and parsed into syntax that follows the language grammar. Second the declaration phase, where declarations from source and imported metadata are analyzed to form named symbols. Next the bind phase, where identifiers in the code are matched to symbols. Finally, the emit phase, where all the information built up by the compiler is emitted as an assembly. Corresponding to each of those phases, an object model that allows access to the information at that phase is surfaced. The parsing phase is exposed as a syntax tree, the declaration phase as a hierarchical symbol table, the binding phase as a model that exposes the result of the compiler’s semantic analysis and the emit phase as an API that produces IL byte codes. Each compiler combines these components together as a single end-to-end whole. To ensure that the public Compiler APIs are sufficient for building world-class IDE features, the language services that will be used to power the C# and VB experiences in Visual Studio vNext have been rebuilt using them. For instance, the code outlining and formatting features use the syntax trees, the Object Browser and navigation features use the symbol table, refactorings and Go to Definition use the semantic model, and Edit and Continue uses all of these, including the Emit API. These experiences may be previewed on Visual Studio 2013 through the “Roslyn” End-User Preview. This preview is required in order to build and test applications build on top of Roslyn SDK meant for integration into Visual Studio though Roslyn APIs can be used in your own applications independently of Visual Studio without requiring the End-User Preview. ### API Layers Roslyn consists of two main layers of APIs – the Compiler APIs and Workspaces APIs. #### Compiler APIs The compiler layer contains the object models that correspond with information exposed at each phase of the compiler pipeline, both syntactic and semantic. The compiler layer also contains an immutable snapshot of a single invocation of a compiler, including assembly references, compiler options, and source code files. There are two distinct APIs that represent the C# language and the Visual Basic language. These two APIs are similar in shape but tailored for high-fidelity to each individual language. This layer has no dependencies on Visual Studio components. #### Diagnostic APIs As part of their analysis the compiler may produce a set of diagnostics covering everything from syntax, semantic, and definite assignment errors to various warnings and informational diagnostics. The Compiler API layer exposes diagnostics through an extensible API allowing for user-defined analyzers to be plugged into a Compilation and user-defined diagnostics, such as those produced by tools like StyleCop or FxCop, to be produced alongside compiler-defined diagnostics. Producing diagnostics in this way has the benefit of integrating naturally with tools such as MSBuild and Visual Studio which depend on diagnostics for experiences such as halting a build based on policy, showing live squiggles in the editor and suggesting code fixes. #### Scripting APIs As a part of the compiler layer, the team created hosting/scripting APIs for executing code snippets and accumulating a runtime execution context. The REPL uses these APIs. #### Workspaces APIs The Workspaces layer contains the Workspace API, which is the starting point for doing code analysis and refactoring over entire solutions. It assists you in organizing all the information about the projects in a solution into single object model, offering you direct access to the compiler layer object models without needing to parse files, configure options or manage project to project dependencies. In addition, the Workspaces layer surfaces a set of commonly used APIs used when implementing code analysis and refactoring tools that function within a host environment like the Visual Studio IDE, such as the Find All References, Formatting, and Code Generation APIs. This layer has no dependencies on Visual Studio components. ## Working with Syntax The most fundamental data structure exposed by the Compiler APIs is the syntax tree. These trees represent the lexical and syntactic structure of source code. They serve two important purposes: 1. To allow tools - such as an IDE, add-ins, code analysis tools, and refactorings - to see and process the syntactic structure of source code in a user’s project. 2. To enable tools - such as refactorings and an IDE - to create, modify, and rearrange source code in a natural manner without having use direct text edits. By creating and manipulating trees, tools can easily create and rearrange source code. ### Syntax Trees Syntax trees are the primary structure used for compilation, code analysis, binding, refactoring, IDE features, and code generation. No part of the source code is understood without it first being identified and categorized into one of many well-known structural language elements. Syntax trees have three key attributes. The first attribute is that syntax trees hold all the source information in full fidelity. This means that the syntax tree contains every piece of information found in the source text, every grammatical construct, every lexical token, and everything else in between including whitespace, comments, and preprocessor directives. For example, each literal mentioned in the source is represented exactly as it was typed. The syntax trees also represent errors in source code when the program is incomplete or malformed, by representing skipped or missing tokens in the syntax tree. This enables the second attribute of syntax trees. A syntax tree obtained from the parser is completely round-trippable back to the text it was parsed from. From any syntax node, it is possible to get the text representation of the sub-tree rooted at that node. This means that syntax trees can be used as a way to construct and edit source text. By creating a tree you have by implication created the equivalent text, and by editing a syntax tree, making a new tree out of changes to an existing tree, you have effectively edited the text. The third attribute of syntax trees is that they are immutable and thread-safe. This means that after a tree is obtained, it is a snapshot of the current state of the code, and never changes. This allows multiple users to interact with the same syntax tree at the same time in different threads without locking or duplication. Because the trees are immutable and no modifications can be made directly to a tree, factory methods help create and modify syntax trees by creating additional snapshots of the tree. The trees are efficient in the way they reuse underlying nodes, so the new version can be rebuilt fast and with little extra memory. A syntax tree is literally a tree data structure, where non-terminal structural elements parent other elements. Each syntax tree is made up of nodes, tokens, and trivia. ### Syntax Nodes Syntax nodes are one of the primary elements of syntax trees. These nodes represent syntactic constructs such as declarations, statements, clauses, and expressions. Each category of syntax nodes is represented by a separate class derived from SyntaxNode. The set of node classes is not extensible. All syntax nodes are non-terminal nodes in the syntax tree, which means they always have other nodes and tokens as children. As a child of another node, each node has a parent node that can be accessed through the Parent property. Because nodes and trees are immutable, the parent of a node never changes. The root of the tree has a null parent. Each node has a ChildNodes method, which returns a list of child nodes in sequential order based on its position in the source text. This list does not contain tokens. Each node also has a collection of Descendant methods - such as DescendantNodes, DescendantTokens, or DescendantTrivia - that represent a list of all the nodes, tokens, or trivia that exist in the sub-tree rooted by that node. In addition, each syntax node subclass exposes all the same children through strongly typed properties. For example, a BinaryExpressionSyntax node class has three additional properties specific to binary operators: Left, OperatorToken, and Right. The type of Left and Right is ExpressionSyntax, and the type of OperatorToken is SyntaxToken. Some syntax nodes have optional children. For example, an IfStatementSyntax has an optional ElseClauseSyntax. If the child is not present, the property returns null. ### Syntax Tokens Syntax tokens are the terminals of the language grammar, representing the smallest syntactic fragments of the code. They are never parents of other nodes or tokens. Syntax tokens consist of keywords, identifiers, literals, and punctuation. For efficiency purposes, the SyntaxToken type is a CLR value type. Therefore, unlike syntax nodes, there is only one structure for all kinds of tokens with a mix of properties that have meaning depending on the kind of token that is being represented. For example, an integer literal token represents a numeric value. In addition to the raw source text the token spans, the literal token has a Value property that tells you the exact decoded integer value. This property is typed as Object because it may be one of many primitive types. The ValueText property tells you the same information as the Value property; however this property is always typed as String. An identifier in C# source text may include Unicode escape characters, yet the syntax of the escape sequence itself is not considered part of the identifier name. So although the raw text spanned by the token does include the escape sequence, the ValueText property does not. Instead, it includes the Unicode characters identified by the escape. ### Syntax Trivia Syntax trivia represent the parts of the source text that are largely insignificant for normal understanding of the code, such as whitespace, comments, and preprocessor directives. Because trivia are not part of the normal language syntax and can appear anywhere between any two tokens, they are not included in the syntax tree as a child of a node. Yet, because they are important when implementing a feature like refactoring and to maintain full fidelity with the source text, they do exist as part of the syntax tree. You can access trivia by inspecting a token’s LeadingTrivia or TrailingTrivia collections. When source text is parsed, sequences of trivia are associated with tokens. In general, a token owns any trivia after it on the same line up to the next token. Any trivia after that line is associated with the following token. The first token in the source file gets all the initial trivia, and the last sequence of trivia in the file is tacked onto the end-of-file token, which otherwise has zero width. Unlike syntax nodes and tokens, syntax trivia do not have parents. Yet, because they are part of the tree and each is associated with a single token, you may access the token it is associated with using the Token property. Like syntax tokens, trivia are value types. The single SyntaxTrivia type is used to describe all kinds of trivia. ### Spans Each node, token, or trivia knows its position within the source text and the number of characters it consists of. A text position is represented as a 32-bit integer, which is a zero-based Unicode character index. A TextSpan object is the beginning position and a count of characters, both represented as integers. If TextSpan has a zero length, it refers to a location between two characters. Each node has two TextSpan properties: Span and FullSpan. The Span property is the text span from the start of the first token in the node’s sub-tree to the end of the last token. This span does not include any leading or trailing trivia. The FullSpan property is the text span that includes the node’s normal span, plus the span of any leading or trailing trivia. For example: ``` csharp if (x > 3) { || // this is bad |throw new Exception("Not right.");| // better exception?|| } ``` The statement node inside the block has a span indicated by the single vertical bars (|). It includes the characters +throw new Exception(“Not right.”);+. The full span is indicated by the double vertical bars (||). It includes the same characters as the span and the characters associated with the leading and trailing trivia. ### Kinds Each node, token, or trivia has a RawKind property, of type System.Int32, that identifies the exact syntax element represented. This value can be cast to a language-specific enumeration; each language, C# or VB, has a single SyntaxKind enumeration that lists all the possible nodes, tokens, and trivia elements in the grammar. This conversion can be done automatically by accessing the CSharpSyntaxKind() or VisualBasicSyntaxKind() extension methods. The RawKind property allows for easy disambiguation of syntax node types that share the same node class. For tokens and trivia, this property is the only way to distinguish one type of element from another. For example, a single BinaryExpressionSyntax class has Left, OperatorToken, and Right as children. The Kind property distinguishes whether it is an AddExpression, SubtractExpression, or MultiplyExpression kind of syntax node. ### Errors Even when the source text contains syntax errors, a full syntax tree that is round-trippable to the source is exposed. When the parser encounters code that does not conform to the defined syntax of the language, it uses one of two techniques to create a syntax tree. First, if the parser expects a particular kind of token, but does not find it, it may insert a missing token into the syntax tree in the location that the token was expected. A missing token represents the actual token that was expected, but it has an empty span, and its IsMissing property returns true. Second, the parser may skip tokens until it finds one where it can continue parsing. In this case, the skipped tokens that were skipped are attached as a trivia node with the kind SkippedTokens. ## Working with Semantics Syntax trees represent the lexical and syntactic structure of source code. Although this information alone is enough to describe all the declarations and logic in the source, it is not enough information to identify what is being referenced. For example, many types, fields, methods, and local variables with the same name may be spread throughout the source. Although each of these is uniquely different, determining which one an identifier actually refers to often requires a deep understanding of the language rules. There are program elements represented in source code, and programs can also refer to previously compiled libraries, packaged in assembly files. Although no source code is available for assemblies and therefore no syntax nodes or trees, programs can still refer to elements inside them. In addition to a syntactic model of the source code, a semantic model encapsulates the language rules, giving you an easy way to make these distinctions. ### Compilation A compilation is a representation of everything needed to compile a C# or Visual Basic program, which includes all the assembly references, compiler options, and source files. Because all this information is in one place, the elements contained in the source code can be described in more detail. The compilation represents each declared type, member, or variable as a symbol. The compilation contains a variety of methods that help you find and relate the symbols that have either been declared in the source code or imported as metadata from an assembly. Similar to syntax trees, compilations are immutable. After you create a compilation, it cannot be changed by you or anyone else you might be sharing it with. However, you can create a new compilation from an existing compilation, specifying a change as you do so. For example, you might create a compilation that is the same in every way as an existing compilation, except it may include an additional source file or assembly reference. ### Symbols A symbol represents a distinct element declared by the source code or imported from an assembly as metadata. Every namespace, type, method, property, field, event, parameter, or local variable is represented by a symbol. A variety of methods and properties on the Compilation type help you find symbols. For example, you can find a symbol for a declared type by its common metadata name. You can also access the entire symbol table as a tree of symbols rooted by the global namespace. Symbols also contain additional information that the compiler determined from the source or metadata, such as other referenced symbols. Each kind of symbol is represented by a separate interface derived from ISymbol, each with its own methods and properties detailing the information the compiler has gathered. Many of these properties directly reference other symbols. For example, the ReturnType property of the IMethodSymbol class tells you the actual type symbol that the method declaration referenced. Symbols present a common representation of namespaces, types, and members, between source code and metadata. For example, a method that was declared in source code and a method that was imported from metadata are both represented by an IMethodSymbol with the same properties. Symbols are similar in concept to the CLR type system as represented by the System.Reflection API, yet they are richer in that they model more than just types. Namespaces, local variables, and labels are all symbols. In addition, symbols are a representation of language concepts, not CLR concepts. There is a lot of overlap, but there are many meaningful distinctions as well. For instance, an iterator method in C# or Visual Basic is a single symbol. However, when the iterator method is translated to CLR metadata, it is a type and multiple methods. ### Semantic Model A semantic model represents all the semantic information for a single source file. You can use it to discover the following: * The symbols referenced at a specific location in source. * The resultant type of any expression. * All diagnostics, which are errors and warnings. * How variables flow in and out of regions of source. * The answers to more speculative questions. ## Working with a Workspace The Workspaces layer is the starting point for doing code analysis and refactoring over entire solutions. Within this layer, the Workspace API assists you in organizing all the information about the projects in a solution into single object model, offering you direct access to compiler layer object models like source text, syntax trees, semantic models and compilations without needing to parse files, configure options or manage inter-project dependencies. Host environments, like an IDE, provide a workspace for you corresponding to the open solution. It is also possible to use this model outside of an IDE by simply loading a solution file. ### Workspace A workspace is an active representation of your solution as a collection of projects, each with a collection of documents. A workspace is typically tied to a host environment that is constantly changing as a user types or manipulates properties. The workspace provides access to the current model of the solution. When a change in the host environment occurs, the workspace fires corresponding events, and the CurrentSolution property is updated. For example, when the user types in a text editor corresponding to one of the source documents, the workspace uses an event to signal that the overall model of the solution has changed and which document was modified. You can then react to those changes by analyzing the new model for correctness, highlighting areas of significance, or by making a suggestion for a code change. You can also create stand-alone workspaces that are disconnected from the host environment or used in an application that has no host environment. ### Solutions, Projects, Documents Although a workspace may change every time a key is pressed, you can work with the model of the solution in isolation. A solution is an immutable model of the projects and documents. This means that the model can be shared without locking or duplication. After you obtain a solution instance from the Workspace’s CurrentSolution property, that instance will never change. However, like with syntax trees and compilations, you can modify solutions by constructing new instances based on existing solutions and specific changes. To get the workspace to reflect your changes, you must explicitly apply the changed solution back to the workspace. A project is a part of the overall immutable solution model. It represents all the source code documents, parse and compilation options, and both assembly and project-to-project references. From a project, you can access the corresponding compilation without needing to determine project dependencies or parse any source files. A document is also a part of the overall immutable solution model. A document represents a single source file from which you can access the text of the file, the syntax tree, and the semantic model. The following diagram is a representation of how the Workspace relates to the host environment, tools, and how edits are made. ## Summary Roslyn exposes a set of Compiler APIs and Workspaces APIs that provides rich information about your source code and that has full fidelity with the C# and Visual Basic languages. The transition to compilers as a platform dramatically lowers the barrier to entry for creating code focused tools and applications. It creates many opportunities for innovation in areas such as meta-programming, code generation and transformation, interactive use of the C# and VB languages, and embedding of C# and VB in domain specific languages. --- ### Compilers CSharp Readme (`docs/compilers/CSharp/readme.md`) Breaking changes (newest release on top): - [.NET 6 and Visual Studio 2022](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%206.md) - [Minor releases following .NET 5 and Visual Studio 2019 version 16.9](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20post%20DotNet%205.md) - [.NET 5 and Visual Studio 2019 version 16.9](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%205.md) - [Minor releases following Visual Studio 2019 version 16.0](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20post%20VS2019.md) - [Visual Studio 2019 version 16.0](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20VS2019.md) - [Minor releases following Visual Studio 2017 version 15.0](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20post%20VS2017.md) - [Visual Studio 2017 version 15.0](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20VS2017.md) - [Visual Studio 2015 version 14.0](https://github.com/dotnet/roslyn/blob/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20VS2015.md) --- ### Compilers Design README (`docs/compilers/Design/README.md`) Design Docs =========== This directory is a place for design documents for the Roslyn compilers. --- ### Adding Optional Parameters In Public API (`docs/Adding Optional Parameters in Public API.md`) # Adding optional parameters to public methods # Roslyn has strict requirements for maintaining source and binary API compatibility across minor version releases. Adding overloads of existing methods can be an effective way of adding to the public API without breaking backwards compatibility. However, overloads involving optional parameters can be tricky to make both source and binary compatible. The following steps should be followed when adding optional parameters to public methods. ## Steps for adding optional parameters to an existing method ## 1. Check to see if there is already a method marked with the comment ``` // BACKCOMPAT OVERLOAD -- DO NOT TOUCH ``` 2. If it does not: 1. Copy the method and make any existing optional parameters mandatory. 2. Change the body to call the original method. 3. Add the comment `// BACKCOMPAT OVERLOAD -- DO NOT TOUCH`. This method provides binary compatibility with assemblies compiled against the previous APIs. 3. Add your new parameter with a default value to the *end* of the original method. This provides source compatibility to consumers of the previous API. ## Examples ## Before: ```csharp public void O(string o1 = null, string o2 = null) { } ``` After: ```csharp public void O(string o1 = null, string o2 = null, bool o3 = false) { } // 1.0 BACKCOMPAT OVERLOAD -- DO NOT TOUCH public void O(string o1, string o2) { O(o1, o2, o3: false); } ``` ## Things to avoid ## **DO NOT add a parameter in the middle of existing optional parameters** ```csharp public void O(string o1 = null, bool o4 = false, string o2 = null, bool o3 = false, bool o5 = false) { } ``` This breaks source compat when certain arguments to a method call are skipped, like `O(null, null, o5: true);` **DO NOT add multiple overloads with optional parameters** ```csharp public void O(string o1 = null, string o2 = null, bool o3 = false) { } // 1.0 BACKCOMPAT OVERLOAD -- DO NOT TOUCH public void O(string o1 = null, string o2 = null) { O(o1, o2, o3: false); } ``` Aside from the fact that this is often ambiguous and it won’t compile, it makes verifying compatibility much harder. ## End Result ## The resulting code should 1) have only one overload with optional parameters, which 2) has the most parameters of all the overloads, and 3) all previous overloads should still exist, properly commented with their release version, and contain only mandatory parameters. ## NOTE ## After this change, if Public API Analyzer complains about it, you should copy the entry for your change from PublicAPI.Shipped.txt and then put that entry in PublicAPI.Unshipped.txt with `*REMOVED*` prefix. PublicAPI.Shipped.txt ``` txt Example.O(string o1 = null, string o2 = null) -> void ``` PublicAPI.Unshipped.txt ``` txt *REMOVED*Example.O(string o1 = null, string o2 = null) -> void ``` --- ### Breaking API Changes (`docs/Breaking API Changes.md`) API Breaking Changes ==== # Version 1.1.0 ### Removed VisualBasicCommandLineParser.ctor During a toolset update we noticed the constructor on `VisualBasicCommandLineParser` was `public`. This in turn made many of the `protected` members of `CommandLineParser` a part of the API surface as it gave external customers an inheritance path. It was never the intent for these members to be a part of the supported API surface. Creation of the parsers is meant to be done via the `Default` singleton properties. There seems to be little risk that we broke any customers here and hence we decided to remove this API. PR: https://github.com/dotnet/roslyn/pull/4169 ### Changed Simplifier methods to throw ArgumentNullExceptions Changed Simplifier.ReduceAsync, Simplifier.ExpandAsync, and Simplifier.Expand methods to throw ArgumentNullExceptions if any non-optional, nullable arguments are passed in. Previously the user would get a NullReferenceException for synchronous methods and an AggregateException containing a NullReferenceException for asynchronous methods. PR: https://github.com/dotnet/roslyn/pull/5144 # Version 1.3.0 ### Treat a method marked with both public and private flags as private The scenario is loading an assembly where some methods, fields or nested types have accessibility flags set to 7 (all three bits set), which mean public AND private. After the fix, such flags are loaded to mean private. The compat change is we’re trading a compile-time success and runtime failure (native compiler) against a compile-time error (restoring the behavior of v1.2). Details below: - The native compiler successfully compiles the method and field case (those only yield runtime error System.TypeLoadException: Invalid Field Access Flags) and reported an accessibility error on the nested type. - The 1.2 compiler generated errors: ``` error BC30390: 'C.Private Overloads Sub M()' is not accessible in this context because it is 'Private'. error BC30389: 'C.F' is not accessible in this context because it is 'Private'. error BC30389: 'C.C2' is not accessible in this context because it is 'Protected Friend'. error BC30390: 'C2.Private Overloads Sub M2()' is not accessible in this context because it is 'Private'. ``` - The 1.3 compiler crashes. - After fix, the same errors as 1.2 are generated again. PR: https://github.com/dotnet/roslyn/pull/11547 ### Don't emit bad DateTimeConstant, and load bad BadTimeConstant as default value instead The change affects compatibility in two ways: - When loading an invalid DateTimeConstant(-1), the compiler will use default(DateTime) instead, whereas the native compiler would produce code that fails to execute. - DateTimeConstant(-1) will still count when we check that you don’t specify two default values. The compiler will produce an error, instead of succeeding (and producing IL with two attributes). PR: https://github.com/dotnet/roslyn/pull/11536 # Version 4.1.0 ### Can no longer inherit from CompletionService and CompletionServiceWithProviders The constructors of Microsoft.CodeAnalysis.Completion and Microsoft.CodeAnalysis.Completion.CompletionServiceWithProviders are now internal. Roslyn does not support implementing completion for arbitrary languages. # Version 4.2.0 ### Can no longer inherit from QuickInfoService The constructors of Microsoft.CodeAnalysis.QuickInfoService are now internal. Roslyn does not support implementing completion for arbitrary languages. ### `Microsoft.CodeAnalysis.CodeStyle.NotificationOption` is now immutable All property setters now throw an exception. # Version 4.4.0 `Workspace.OnWorkspaceFailed` is no longer called when an error occurs while reading source file content from disk. The `Workspace` and `DocumentId` parameters of `TextLoader.LoadTextAndVersionAsync(Workspace, DocumentId, CancellationToken)` are deprecated. The method now receives `null` `Workspace` and `DocumentId`. # Version 4.5.0 `SymbolDisplayFormat.CSharpErrorMessageFormat` and `CSharpShortErrorMessageFormat` now include parameter names by default if used on a standalone `IParameterSymbol`. For example, parameter `p` in `void M(ref int p)` was previously formatted as `"ref int"` and now it is formatted as `"ref int p"`. # Version 4.7.0 ### `SymbolDisplayFormat` includes parameter name when invoked on `IParameterSymbol` All `SymbolDisplayFormat`s (predefined and user-created) now include parameter names by default if used on a standalone `IParameterSymbol` for consistency with predefined formats (see the breaking change for version 4.5.0 above). ### Changed `IncrementalStepRunReason` when a modified input produced a new output `IncrementalGeneratorRunStep.Outputs` previously contained `IncrementalStepRunReason.Modified` as `Reason` when the input to the step was modified in a way that produced a new output. Now the reason will be reported more accurately as `IncrementalStepRunReason.New`. # Version 4.8.0 ### Changed `Assembly.Location` behavior in non-Windows The value of `Assembly.Location` previously held the location on disk where an analyzer or source generator was loaded from. This could be either the original location or the shadow copy location. In 4.8 this will be `""` in certain cases when running on non Windows platforms. This is due the compiler server loading assemblies using `AssemblyLoadContext.LoadFromStream` instead of loading from disk. This could already happen in other load scenarios but this change moves it into mainline build scenarios. ### Deprecation warning for SyntaxNode serialization The ability to serialize/deserialize a SyntaxNode to/from a Stream has been deprecated. The code for this still exists in Roslyn, but attempting to call the APIs to perform these functions will result in 'Obsolete' warnings being reported. A future version of Roslyn will remove this functionality entirely. This functionality could only work for a host that wrote out the nodes to a stream, and later read it back in within the same process instance. It could not be used to communicate across processes, or for persisting nodes to disk to be read in at a later time by a new host sessions. This functionality originally existed for the days when Roslyn was hosted in 32bit processes with limited address space. That is no longer a mainline supported scenario. Clients can get similar functionality by persisting the text of the node, and parsing it back out when needed. PR: https://github.com/dotnet/roslyn/pull/70365 # Version 4.9.0 ### Obsoletion and removal of SyntaxNode serialization. Continuation of the deprecation that happened in 4.8.0 (see information above). In 4.9.0 this functionality is now entirely removed, and will issue both an obsoletion error, and will throw at runtime if the APIs are used. PR: https://github.com/dotnet/roslyn/pull/70277 ### Changes in `Microsoft.CodeAnalysis.Emit.EmitBaseline.CreateInitialBaseline` method A new required parameter `Compilation` has been added. Existing overloads without this parameter no longer work and throw `NotSupportedException`. ### Changes in `Microsoft.CodeAnalysis.Emit.SemanticEdit` constructors The value of `preserveLocalVariables` passed to the constructors is no longer used. # Version 5.11.0 ### Changed the `default` value of `Microsoft.CodeAnalysis.CSharp.Conversion` The `default` value of `Microsoft.CodeAnalysis.CSharp.Conversion` struct now returns `false` for all boolean properties. Previously the `Exists` and `IsExplicit` properties returned `true`: ```cs using System; using Microsoft.CodeAnalysis.CSharp; var conv = default(Conversion); Console.WriteLine(conv.Exists); // Previously was 'True', now 'False' Console.WriteLine(conv.IsExplicit); // Previously was 'True', now 'False' ``` PR: https://github.com/dotnet/roslyn/pull/84628 --- ### Language Feature Status (`docs/Language Feature Status.md`) Language Feature Status ===== This document reflects the status, and planned work in progress, for the compiler team. It is a live document and will be updated as work progresses, features are added / removed, and as work on feature progresses. This is not an exhaustive list of our features but rather the ones which have active development efforts behind them. The "Feature" column contains a link to the championed proposal or doc for the feature. The "State" column contains a link to the test plan issue for the feature. The test plan should have a "Feature - XYZ" label, which is also used on PRs and issues for the feature. To add a new feature entry (along with its test plan issue and label), use the [`new-compiler-feature` skill](../.github/skills/new-compiler-feature/). # Working Set C# Merged features should be in the bottom portion of the table (below the break), while in-progress features should be in the top portion of the table. When a feature is merged to the `main` branch, its entry should be moved to the very bottom of the table and its "State" value should be updated to record the relevant version number. | Feature | Branch | State | Developer | Reviewer | IDE Buddy | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | --------- | | [Dictionary expressions](https://github.com/dotnet/csharplang/issues/8659) | [dictionary-expressions](https://github.com/dotnet/roslyn/tree/features/dictionary-expressions) | [In progress](https://github.com/dotnet/roslyn/issues/81860) | [333fred](https://github.com/333fred) | [RikkiGibson](https://github.com/RikkiGibson), [jjonescz](https://github.com/jjonescz) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Null-conditional await](https://github.com/dotnet/csharplang/issues/8631) | [null-conditional-await](https://github.com/dotnet/roslyn/tree/features/null-conditional-await) | [In progress](https://github.com/dotnet/roslyn/issues/83237) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | TBD | TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Chained relational comparison](https://github.com/dotnet/csharplang/issues/8861) | [chained-relational-comparison](https://github.com/dotnet/roslyn/tree/features/chained-relational-comparison) | [In progress](https://github.com/dotnet/roslyn/issues/83255) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | TBD | TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Target-typed static member access](https://github.com/dotnet/csharplang/issues/9138) | [target-typed-static-member-access](https://github.com/dotnet/roslyn/tree/features/target-typed-static-member-access) | [In progress](https://github.com/dotnet/roslyn/issues/83323) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | TBD | TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Relax modifier ordering](https://github.com/dotnet/csharplang/issues/8966) | [relaxed-modifier-ordering](https://github.com/dotnet/roslyn/tree/features/relaxed-modifier-ordering) | [In progress](https://github.com/dotnet/roslyn/issues/83324) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | TBD | TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Compound assignment in initializers](https://github.com/dotnet/csharplang/issues/9896) | [compound-assignment-in-initializer](https://github.com/dotnet/roslyn/tree/features/compound-assignment-in-initializer) | [In progress](https://github.com/dotnet/roslyn/issues/83420) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | TBD | TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Extension members on typeless receivers](https://github.com/dotnet/csharplang/issues/10146) | [extension-members-on-typeless-receivers](https://github.com/dotnet/roslyn/tree/features/extension-members-on-typeless-receivers) | [In progress](https://github.com/dotnet/roslyn/issues/83428) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | TBD | TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | Runtime Async Streams | [runtime-async-streams](https://github.com/dotnet/roslyn/tree/features/runtime-async-streams) | [Test plan for "runtime-async"](https://github.com/dotnet/roslyn/issues/75960) | [jcouv](https://github.com/jcouv) | [333fred](https://github.com/333fred), [RikkiGibson](https://github.com/RikkiGibson) | | | | [Extension constants](https://github.com/dotnet/csharplang/issues/10242) | [extension-consts](https://github.com/dotnet/roslyn/tree/features/extension-consts) | [In progress](https://github.com/dotnet/roslyn/issues/84269) | [jcouv](https://github.com/jcouv) | TBD | TBD | [jcouv](https://github.com/jcouv) | | [Type Parameter Inference from Constraints](https://github.com/dotnet/csharplang/issues/9453) | [PR](https://github.com/dotnet/roslyn/pull/84655) | [In progress](https://github.com/dotnet/roslyn/issues/84868) | [agocke](https://github.com/agocke) | [333fred](https://github.com/333fred), [jjonescz](https://github.com/jjonescz) | TBD | [333fred](https://github.com/333fred) | | ------- | ------ | ----- | --------- | -------- | --------- | --------- | | [Unsafe evolution](https://github.com/dotnet/csharplang/issues/9704) | [UnsafeEvolution](https://github.com/dotnet/roslyn/tree/features/UnsafeEvolution) | [Merged as preview feature into .NET 11p2 and VS 18.6](https://github.com/dotnet/roslyn/issues/81207) | [jjonescz](https://github.com/jjonescz) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | TBD | [agocke](https://github.com/agocke) | | Runtime Async | main | [Main feature merged into main in preview](https://github.com/dotnet/roslyn/issues/75960) | [333fred](https://github.com/333fred) | [jcouv](https://github.com/jcouv), [RikkiGibson](https://github.com/RikkiGibson) | | | # Working Set VB # C# 15.0 | Feature | Branch | State | Developer | Reviewer | IDE Buddy | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | --------- | | [Collection expression arguments](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-15.0/collection-expression-arguments.md) | [collection-expression-arguments](https://github.com/dotnet/roslyn/tree/features/collection-expression-arguments) | [C# 15](https://github.com/dotnet/roslyn/issues/80613) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Unions](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-15.0/unions.md) | [Unions](https://github.com/dotnet/roslyn/tree/features/Unions) | [C# 15](https://github.com/dotnet/roslyn/issues/81074) | [AlekseyTs](https://github.com/AlekseyTs) | [RikkiGibson](https://github.com/RikkiGibson), [jjonescz](https://github.com/jjonescz) | TBD | [MadsTorgersen](https://github.com/MadsTorgersen) | | Non-virtual static interface members without DIM runtime support | main | [C# 15](https://github.com/dotnet/roslyn/pull/83097) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [RikkiGibson](https://github.com/RikkiGibson) | TBD | N/A | | [ExtendedLayoutAttribute](https://github.com/dotnet/runtime/issues/100896) | main | [Merged into 18.3](https://github.com/dotnet/roslyn/pull/78741) | [jkoritzinsky](https://github.com/jkoritzinsky) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | | | | [Closed class hierarchies](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-15.0/closed-hierarchies.md) | [closed-class](https://github.com/dotnet/roslyn/tree/features/closed-class) | [C# 15](https://github.com/dotnet/roslyn/issues/81039) | [RikkiGibson](https://github.com/RikkiGibson) | [AlekseyTs](https://github.com/AlekseyTs), [jjonescz](https://github.com/jjonescz) | TBD | [mattwar](https://github.com/mattwar) | | [Extension indexers](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-15.0/extension-indexers.md) | [extensions](https://github.com/dotnet/roslyn/tree/features/extensions) | [C# 15](https://github.com/dotnet/roslyn/issues/81505) | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv) | | [Labeled break/continue](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-15.0/labeled-break-continue.md) | [labeled-break-and-continue](https://github.com/dotnet/roslyn/tree/features/labeled-break-and-continue) | [C# 15](https://github.com/dotnet/roslyn/pull/84271) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv), [jjonescz](https://github.com/jjonescz) | [jcouv](https://github.com/jcouv) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | # C# 14.0 | Feature | Branch | State | Developer | Reviewer | IDE Buddy | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | --------- | | [`field` keyword in properties](https://github.com/dotnet/csharplang/issues/8635) | [field-keyword](https://github.com/dotnet/roslyn/tree/features/field-keyword) | [Merged into 17.12p3](https://github.com/dotnet/roslyn/issues/57012) | [Youssef1313](https://github.com/Youssef1313), [cston](https://github.com/cston) | [333fred](https://github.com/333fred), [RikkiGibson](https://github.com/RikkiGibson) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [First-class Span Types](https://github.com/dotnet/csharplang/issues/8714) | [FirstClassSpan](https://github.com/dotnet/roslyn/tree/features/FirstClassSpan) | [Merged into 17.13p1](https://github.com/dotnet/roslyn/issues/73445) | [jjonescz](https://github.com/jjonescz) | [cston](https://github.com/cston), [333fred](https://github.com/333fred) | | [333fred](https://github.com/333fred), [stephentoub](https://github.com/stephentoub) | | [Unbound generic types in `nameof`](https://github.com/dotnet/csharplang/issues/8662) | [PR](https://github.com/dotnet/roslyn/pull/75368) | [Merged into 17.13p2](https://github.com/dotnet/roslyn/pull/75368) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv), [AlekseyTs](https://github.com/AlekseyTs) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [String literals in data section as UTF8](https://github.com/dotnet/roslyn/blob/main/docs/features/string-literals-data-section.md) | [PR](https://github.com/dotnet/roslyn/pull/76036) | [Merged into 17.13p4](https://github.com/dotnet/roslyn/issues/76234) | [jjonescz](https://github.com/jjonescz) | [AlekseyTs](https://github.com/AlekseyTs), [cston](https://github.com/cston) | N/A | N/A | | [Simple lambda parameters with modifiers](https://github.com/dotnet/csharplang/issues/8637) | [PR](https://github.com/dotnet/roslyn/pull/75400) | [Merged into 17.14p1](https://github.com/dotnet/roslyn/pull/75400) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jjonescz](https://github.com/jjonescz), [cston](https://github.com/cston) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Partial Events and Constructors](https://github.com/dotnet/csharplang/issues/9058) | [PartialEventsCtors](https://github.com/dotnet/roslyn/tree/features/PartialEventsCtors) | [Merged into 17.14p3](https://github.com/dotnet/roslyn/issues/76859) | [jjonescz](https://github.com/jjonescz) | [cston](https://github.com/cston), [RikkiGibson](https://github.com/RikkiGibson) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jaredpar](https://github.com/jaredpar) | | [Extensions](https://github.com/dotnet/csharplang/issues/8697) | [extensions](https://github.com/dotnet/roslyn/tree/features/extensions) | [Merged into 17.14p3](https://github.com/dotnet/roslyn/issues/76130) | [jcouv](https://github.com/jcouv), [AlekseyTs](https://github.com/AlekseyTs) | [jjonescz](https://github.com/jjonescz), TBD | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Null-conditional assignment](https://github.com/dotnet/csharplang/issues/8677) | [null-conditional-assignment](https://github.com/dotnet/roslyn/tree/features/null-conditional-assignment) | [Merged into 17.14p3](https://github.com/dotnet/roslyn/issues/75554) | [RikkiGibson](https://github.com/RikkiGibson) | [cston](https://github.com/cston), [jjonescz](https://github.com/jjonescz) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [RikkiGibson](https://github.com/RikkiGibson) | | [Ignored directives](https://github.com/dotnet/csharplang/issues/8617) | [PR](https://github.com/dotnet/roslyn/pull/77696) | Merged into 17.14p3 | [jjonescz](https://github.com/jjonescz) | [RikkiGibson](https://github.com/RikkiGibson), [jaredpar](https://github.com/jaredpar) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jaredpar](https://github.com/jaredpar) | | [User Defined Compound Assignment Operators](https://github.com/dotnet/csharplang/issues/9101) | [UserDefinedCompoundAssignment](https://github.com/dotnet/roslyn/tree/features/UserDefinedCompoundAssignment) | [Merged to main](https://github.com/dotnet/roslyn/issues/76934) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [cston](https://github.com/cston) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [AlekseyTs](https://github.com/AlekseyTs) | | [Optional and named arguments in `Expression` trees](https://github.com/dotnet/csharplang/issues/9246) | [PR](https://github.com/dotnet/roslyn/pull/77972) | Merged to main(https://github.com/dotnet/roslyn/issues/76934) | [cston](https://github.com/cston) | [333fred](https://github.com/333fred), [AlekseyTs](https://github.com/AlekseyTs) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [cston](https://github.com/cston) | # VB in .NET 10 | Feature | Branch | State | Developer | Reviewer | IDE Buddy | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | --------- | | Recognizing 'unmanaged' constraint | main | [Merged into 17.13 P2](https://github.com/dotnet/roslyn/pull/75665) | [AlekseyTs](https://github.com/AlekseyTs) | [cston](https://github.com/cston), [333fred](https://github.com/333fred) | (no IDE impact) | [jaredpar](https://github.com/jaredpar) | | Overload Resolution Priority | main | [Merged into 17.13 P3](https://github.com/dotnet/roslyn/pull/76419) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [cston](https://github.com/cston) | (no IDE impact) | [333fred](https://github.com/333fred) | # C# 13.0 | Feature | Branch | State | Developer | Reviewer | IDE Buddy | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | --------- | | [Escape character](https://github.com/dotnet/csharplang/issues/7400) | N/A | [Merged into 17.9p1](https://github.com/dotnet/roslyn/pull/70497) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv), [RikkiGibson](https://github.com/RikkiGibson) | (no IDE impact) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Method group natural type improvements](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-13.0/method-group-natural-type-improvements.md) | main | [Merged into 17.9p2](https://github.com/dotnet/roslyn/issues/69432) | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [cston](https://github.com/cston) | (no IDE impact) | [jcouv](https://github.com/jcouv) | | [`Lock` object](https://github.com/dotnet/csharplang/issues/7104) | [LockObject](https://github.com/dotnet/roslyn/tree/features/LockObject) | [Merged into 17.10p2](https://github.com/dotnet/roslyn/issues/71888) | [jjonescz](https://github.com/jjonescz) | [cston](https://github.com/cston), [RikkiGibson](https://github.com/RikkiGibson) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) (needs IDE fixer) | [stephentoub](https://github.com/stephentoub) | | Implicit indexer access in object initializers | main | [Merged into 17.9p3](https://github.com/dotnet/roslyn/pull/70649) | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [cston](https://github.com/cston) | (no IDE impact) | | | [Params-collections](https://github.com/dotnet/csharplang/issues/7700) | main | [Merged to 17.10p3](https://github.com/dotnet/roslyn/issues/71137) | [AlekseyTs](https://github.com/AlekseyTs) | [RikkiGibson](https://github.com/RikkiGibson), [333fred](https://github.com/333fred) | [akhera99](https://github.com/akhera99) (needs IDE fixer) | [MadsTorgersen](https://github.com/MadsTorgersen), [AlekseyTs](https://github.com/AlekseyTs) | | [Ref/unsafe in iterators/async](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-13.0/ref-unsafe-in-iterators-async.md) | [RefInAsync](https://github.com/dotnet/roslyn/tree/features/RefInAsync) | [Merged into 17.11p2](https://github.com/dotnet/roslyn/issues/72662) | [jjonescz](https://github.com/jjonescz) | [AlekseyTs](https://github.com/AlekseyTs), [cston](https://github.com/cston) | (no IDE impact) | | | [Overload Resolution Priority](https://github.com/dotnet/csharplang/issues/7706) | main | [Merged to 17.12p1](https://github.com/dotnet/roslyn/issues/74131) | [333fred](https://github.com/333fred) | [jcouv](https://github.com/jcouv), [cston](https://github.com/cston) | Not yet assigned | [333fred](https://github.com/333fred) | | [Partial properties](https://github.com/dotnet/csharplang/issues/6420) | [partial-properties](https://github.com/dotnet/roslyn/tree/features/partial-properties) | [Merged into 17.11p3](https://github.com/dotnet/roslyn/issues/73090) | [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv), [333fred](https://github.com/333fred) | [Cosifne](https://github.com/Cosifne) | [333fred](https://github.com/333fred), [RikkiGibson](https://github.com/RikkiGibson) | | [Ref Struct Interfaces/`allows ref struct` constraint](https://github.com/dotnet/csharplang/issues/7608) | [RefStructInterfaces](https://github.com/dotnet/roslyn/tree/features/RefStructInterfaces) | [Merged into 17.11p2](https://github.com/dotnet/roslyn/issues/72124) | [AlekseyTs](https://github.com/AlekseyTs) | [cston](https://github.com/cston), [jjonescz](https://github.com/jjonescz) | [ToddGrun](https://github.com/ToddGrun) | [agocke](https://github.com/agocke), [jaredpar](https://github.com/jaredpar) | | [Collection expression better conversion from expression](https://github.com/dotnet/csharplang/issues/8374) | main | [Merged into 17.12p3](https://github.com/dotnet/roslyn/pull/74993) | [333fred](https://github.com/333fred) | [cston](https://github.com/cston), [AlekseyTs](https://github.com/AlekseyTs) | (no IDE impact) | [333fred](https://github.com/333fred), [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | # C# 12.0 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [ref readonly parameters](https://github.com/dotnet/csharplang/issues/6010) | [RefReadonly](https://github.com/dotnet/roslyn/tree/features/RefReadonly) | [Merged into 17.8p2](https://github.com/dotnet/roslyn/issues/68056) | [jjonescz](https://github.com/jjonescz) | [jcouv](https://github.com/jcouv), [AlekseyTs](https://github.com/AlekseyTs) | [akhera99](https://github.com/akhera99) | [jaredpar](https://github.com/jaredpar) | [Collection Expressions](https://github.com/dotnet/csharplang/issues/5354) | [CollectionLiterals](https://github.com/dotnet/roslyn/tree/features/CollectionLiterals) | [Merged into 17.7p5](https://github.com/dotnet/roslyn/issues/66418) | [cston](https://github.com/cston) | [333fred](https://github.com/333fred), [RikkiGibson](https://github.com/RikkiGibson) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Interceptors (experimental feature)](https://github.com/dotnet/csharplang/issues/7009) | [interceptors](https://github.com/dotnet/roslyn/tree/features/interceptors) | [Merged into 17.7p3](https://github.com/dotnet/roslyn/issues/67421) | [RikkiGibson](https://github.com/RikkiGibson) | [cston](https://github.com/cston), [jcouv](https://github.com/jcouv) | [akhera99](https://github.com/akhera99) | [RikkiGibson](https://github.com/RikkiGibson) | | [Inline Arrays](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-12.0/inline-arrays.md) | [InlineArrays](https://github.com/dotnet/roslyn/tree/features/InlineArrays) | [Merged into 17.7p3](https://github.com/dotnet/roslyn/issues/67826) | [AlekseyTs](https://github.com/AlekseyTs) | [cston](https://github.com/cston), [jjonescz](https://github.com/jjonescz) | [Cosifne](https://github.com/Cosifne) | | | [nameof accessing instance members](https://github.com/dotnet/csharplang/issues/4037) | main | [Merged into 17.7p1](https://github.com/dotnet/roslyn/issues/67565) | [YairHalberstadt](https://github.com/YairHalberstadt), [jjonescz](https://github.com/jjonescz) | [333fred](https://github.com/333fred), [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred) | | [Using aliases for any type](https://github.com/dotnet/csharplang/issues/4284) | main | [Merged into 17.6.P3](https://github.com/dotnet/roslyn/issues/56323) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv) [cston](https://github.com/cston) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | | [Primary Constructors](https://github.com/dotnet/csharplang/issues/2691) | [PrimaryConstructors](https://github.com/dotnet/roslyn/tree/features/PrimaryConstructors) | [Merged into 17.6.P2](https://github.com/dotnet/roslyn/issues/65697) | [AlekseyTs](https://github.com/AlekseyTs) | [cston](https://github.com/cston), [jjonescz](https://github.com/jjonescz) | [akhera99](https://github.com/akhera99) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Lambda optional parameters](https://github.com/dotnet/csharplang/issues/6051) | [lambda-default-parameters](https://github.com/dotnet/roslyn/tree/features/lambda-default-parameters) | [Merged into 17.5p2](https://github.com/dotnet/roslyn/issues/62485) | [adamperlin](https://github.com/adamperlin), [jjonescz](https://github.com/jjonescz) | [333fred](https://github.com/333fred), [cston](https://github.com/cston) | [captainsafia](https://github.com/captainsafia) | | [Experimental attribute](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-12.0/experimental-attribute.md) | main | Merged into 17.8p2 | [jcouv](https://github.com/jcouv) | [cston](https://github.com/cston), [jjonescz](https://github.com/jjonescz) | [jcouv](https://github.com/jcouv) | # C# 11.0 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [File-local types](https://github.com/dotnet/csharplang/issues/6011) | [file-types](https://github.com/dotnet/roslyn/tree/features/file-types) | [Merged into 17.4p1](https://github.com/dotnet/roslyn/issues/60819) | [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv), [cston](https://github.com/cston) | [stephentoub](https://github.com/stephentoub) | | [ref fields](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-11.0/low-level-struct-improvements.md) | [ref-fields](https://github.com/dotnet/roslyn/tree/features/ref-fields) | [Merged into 17.3p3](https://github.com/dotnet/roslyn/issues/59194) | [cston](https://github.com/cston) | [RikkiGibson](https://github.com/RikkiGibson), [AlekseyTs](https://github.com/AlekseyTs) | [jaredpar](https://github.com/jaredpar) | | [Required members](https://github.com/dotnet/csharplang/issues/3630) | main | [Merged into 17.3p2](https://github.com/dotnet/roslyn/issues/57046) | [333fred](https://github.com/333fred) | [jcouv](https://github.com/jcouv), [RikkiGibson](https://github.com/RikkiGibson) | [333fred](https://github.com/333fred) | | [DIM for Static Members](https://github.com/dotnet/csharplang/issues/4436) | [DefaultInterfaceImplementation](https://github.com/dotnet/roslyn/tree/features/DefaultInterfaceImplementation) | [Merged into 17.3p2](https://github.com/dotnet/roslyn/issues/60968) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Numeric IntPtr](https://github.com/dotnet/csharplang/issues/6065) | [numeric-intptr](https://github.com/dotnet/roslyn/tree/features/numeric-intptr) | [Merged into 17.3p2](https://github.com/dotnet/roslyn/issues/60578) | [jcouv](https://github.com/jcouv) | [cston](https://github.com/cston), [333fred](https://github.com/333fred) | [jcouv](https://github.com/jcouv) | | [Unsigned Right Shift](https://github.com/dotnet/csharplang/issues/4682) | [UnsignedRightShift](https://github.com/dotnet/roslyn/tree/features/UnsignedRightShift) | [Merged into 17.3p1](https://github.com/dotnet/roslyn/issues/60433) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs) | | [Utf8 String Literals](https://github.com/dotnet/csharplang/issues/184) | main | [Merged into 17.3p1](https://github.com/dotnet/roslyn/issues/58848) | [AlekseyTs](https://github.com/AlekseyTs) | [cston](https://github.com/cston), [RikkiGibson](https://github.com/RikkiGibson) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Pattern matching on `ReadOnlySpan`](https://github.com/dotnet/csharplang/issues/1881) | [patterns-span-char](https://github.com/dotnet/roslyn/tree/features/patterns-span-char) | [Merged into 17.3p1](https://github.com/dotnet/roslyn/issues/59191) | [YairHalberstadt ](https://github.com/YairHalberstadt) | [cston](https://github.com/cston), [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv) | | [Checked Operators](https://github.com/dotnet/csharplang/issues/4665) | [CheckedUserDefinedOperators](https://github.com/dotnet/roslyn/tree/features/CheckedUserDefinedOperators) | [Merged into 17.3p1](https://github.com/dotnet/roslyn/issues/59458) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [chsienki](https://github.com/chsienki) | [AlekseyTs](https://github.com/AlekseyTs) | | [auto-default structs](https://github.com/dotnet/csharplang/issues/5737) | main | [Merged into 17.3p1](https://github.com/dotnet/roslyn/issues/60167) | [RikkiGibson](https://github.com/RikkiGibson) | [cston](https://github.com/cston), [jcouv](https://github.com/jcouv) | | [Newlines in interpolations](https://github.com/dotnet/csharplang/issues/4935) | main | [Merged in 17.1p1](https://github.com/dotnet/roslyn/issues/57154) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv), [chsienki](https://github.com/chsienki) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [List patterns](https://github.com/dotnet/csharplang/issues/3435) | main | [Merged in 17.1p2](https://github.com/dotnet/roslyn/issues/51289) | [alrz](https://github.com/alrz) | [jcouv](https://github.com/jcouv), [333fred](https://github.com/333fred) | [333fred](https://github.com/333fred) | | [Raw string literals](https://github.com/dotnet/csharplang/issues/4304) | main | [Merged into 17.2](https://github.com/dotnet/roslyn/issues/55306) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Cache delegates for static method group](https://github.com/dotnet/roslyn/issues/5835) | main | [Merged into 17.2](https://github.com/dotnet/roslyn/pull/58288) | [pawchen](https://github.com/pawchen) | [AlekseyTs](https://github.com/AlekseyTs), [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs) | | [nameof(parameter)](https://github.com/dotnet/csharplang/issues/373) | main | [Merged into 17.3p2](https://github.com/dotnet/roslyn/issues/40524) | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv) | | [Relaxing Shift Operator](https://github.com/dotnet/csharplang/issues/4666) | main | [Merged into 17.3p2](https://github.com/dotnet/roslyn/issues/60967) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Generic attributes](https://github.com/dotnet/csharplang/issues/124) | main | [Merged into 17.0p4](https://github.com/dotnet/roslyn/issues/36285) | [AviAvni](https://github.com/AviAvni) | [RikkiGibson](https://github.com/RikkiGibson), [jcouv](https://github.com/jcouv) | [mattwar](https://github.com/mattwar) | # C# 10.0 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [Record structs](https://github.com/dotnet/csharplang/issues/4334) | main | [Merged into 16.11](https://github.com/dotnet/roslyn/issues/51199) | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv) | | [Global Using Directive](https://github.com/dotnet/csharplang/issues/3428) | main | [Merged into 16.11](https://github.com/dotnet/roslyn/issues/51307) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [cston](https://github.com/cston) | [AlekseyTs](https://github.com/AlekseyTs) | | [Improved Definite Assignment](https://github.com/dotnet/csharplang/issues/4465) | main | [Merged into 17.0](https://github.com/dotnet/roslyn/issues/51463) | [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv) | [jaredpar](https://github.com/jaredpar) | | [Constant Interpolated Strings](https://github.com/dotnet/csharplang/issues/2951) | main | [Merged into 16.9p3](https://github.com/dotnet/roslyn/pull/49676) | [kevinsun-dev](https://github.com/kevinsun-dev) | [333fred](https://github.com/333fred) | [jaredar](https://github.com/jaredpar), [agocke](https://github.com/agocke) | | [Extended property patterns](https://github.com/dotnet/csharplang/issues/4394) | main | [Merged into 17.0](https://github.com/dotnet/roslyn/issues/52468) | [alrz](https://github.com/alrz) | [jcouv](https://github.com/jcouv), [333fred](https://github.com/333fred) | [333fred](https://github.com/333fred) | | [Sealed record ToString](https://github.com/dotnet/csharplang/issues/4174) | main | [Merged](https://github.com/dotnet/roslyn/issues/52031) | [thomaslevesque](https://github.com/thomaslevesque/) | [jcouv](https://github.com/jcouv) | [333fred](https://github.com/333fred) | | [Source Generator V2 APIs](https://github.com/dotnet/roslyn/issues/51257) | main | [Merged into 17.0p2](https://github.com/dotnet/roslyn/issues/51257) | [chsienki](https://github.com/chsienki/) | [rikkigibson](https://github.com/rikkigibson), [jaredpar](https://github.com/jaredpar), [cston](https://github.com/cston) | N/A | | [Mix declarations and variables in deconstruction](https://github.com/dotnet/csharplang/issues/125) | main | [Merged into 16.10](https://github.com/dotnet/roslyn/issues/47746) | [YairHalberstadt ](https://github.com/YairHalberstadt) | [jcouv](https://github.com/jcouv) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Async method builder override](https://github.com/dotnet/csharplang/issues/1407) | main | [Merged into 17.0p2](https://github.com/dotnet/roslyn/issues/51999) | [jcouv](https://github.com/jcouv) | [cston](https://github.com/cston), [RikkiGibson](https://github.com/RikkiGibson) | [stephentoub](https://github.com/stephentoub) | | [Enhanced `#line` directive](https://github.com/dotnet/csharplang/issues/4747) | main | [Merged into 17.0p2](https://github.com/dotnet/roslyn/issues/54509) | [cston](https://github.com/cston) | [jcouv](https://github.com/jcouv), [RikkiGibson](https://github.com/RikkiGibson) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Lambda improvements](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-10.0/lambda-improvements.md) | main | [Merged into 17.0p2](https://github.com/dotnet/roslyn/issues/52192) | [cston](https://github.com/cston) | [333fred](https://github.com/333fred), [jcouv](https://github.com/jcouv) | [jaredpar](https://github.com/jaredpar) | | [Static Abstract Members In Interfaces C# 10 Preview](https://github.com/dotnet/csharplang/issues/4436) | main | [Merged into 17.0p2](https://github.com/dotnet/roslyn/issues/52221) | [AlekseyTs](https://github.com/AlekseyTs) | [333fred](https://github.com/333fred), [RikkiGibson](https://github.com/RikkiGibson) | [MadsTorgersen](https://github.com/MadsTorgersen) | | [Interpolated string improvements](https://github.com/dotnet/csharplang/issues/4487) | main | [Merged into 17.0p3](https://github.com/dotnet/roslyn/issues/51499) | [333fred](https://github.com/333fred) | [AlekseyTs](https://github.com/AlekseyTs), [chsienki](https://github.com/chsienki) | [jaredpar](https://github.com/jaredpar) | | [File-scoped namespace](https://github.com/dotnet/csharplang/issues/137) | main | [Merged into 17.0p3](https://github.com/dotnet/roslyn/issues/49000) | [RikkiGibson](https://github.com/RikkiGibson) | [jcouv](https://github.com/jcouv), [chsienki](https://github.com/chsienki) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | | [Parameterless struct constructors](https://github.com/dotnet/csharplang/issues/99) | main | [Merged into 17.0p3](https://github.com/dotnet/roslyn/issues/51698) | [cston](https://github.com/cston) | [jcouv](https://github.com/jcouv), [333fred](https://github.com/333fred) | [jcouv](https://github.com/jouv) | | [Caller expression attribute](https://github.com/dotnet/csharplang/issues/287) | main | [Merged into 17.0p3](https://github.com/dotnet/roslyn/issues/52745) | [Youssef1313](https://github.com/Youssef1313) | [333fred](https://github.com/333fred),[AlekseyTs](https://github.com/AlekseyTs) | [jcouv](https://github.com/jcouv) | # VB 17.0 | Feature | Branch | State | Developer | Reviewer | LDM Champ | |-------------------------------------------------------------------------------------------|--------|------------------------------------------------------------------|-------------------------------------------|-----------------------------------|-----------| | [Caller expression attribute](https://github.com/dotnet/csharplang/issues/287) | main | [Merged into 17.0p3](https://github.com/dotnet/roslyn/issues/52745) | [Youssef1313](https://github.com/Youssef1313) | [333fred](https://github.com/333fred),[AlekseyTs](https://github.com/AlekseyTs) | [jcouv](https://github.com/jcouv) | # VB 16.9 | Feature | Branch | State | Developer | Reviewer | LDM Champ | |-------------------------------------------------------------------------------------------|--------|------------------------------------------------------------------|-------------------------------------------|-----------------------------------|-----------| | [Enable consumption of init-only properties](https://github.com/dotnet/roslyn/pull/50414) | main | [Merged (16.9p3)](https://github.com/dotnet/roslyn/issues/50792) | [AlekseyTs](https://github.com/AlekseyTs) | [jcouv](https://github.com/jcouv) | N/A | # C# 9 | Feature | Branch | State | Developer | Reviewer | LDM Champ | |----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|------------------------------------------------------------------------------|------------------------------------------------------------------------------------|-----------------------------------------| | [Target-typed new](https://github.com/dotnet/csharplang/issues/100) | [target-typed-new](https://github.com/dotnet/roslyn/tree/features/target-typed-new) | [Merged into 16.7p1](https://github.com/dotnet/roslyn/issues/28489) | [alrz](https://github.com/alrz) | [jcouv](https://github.com/jcouv) | [jcouv](https://github.com/jcouv) | | [Skip locals init](https://github.com/dotnet/csharplang/issues/1738) | [localsinit](https://github.com/dotnet/roslyn/tree/features/localsinit) | [Merged](https://github.com/dotnet/roslyn/issues/25780) | [t-camaia](https://github.com/t-camaia), [agocke](https://github.com/agocke) | [jaredpar](https://github.com/jaredpar) | [agocke](https://github.com/agocke) | | [Lambda discard parameters](https://github.com/dotnet/csharplang/issues/111) | main | [Merged](https://github.com/dotnet/roslyn/issues/38820) | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [chsienki](https://github.com/chsienki) | [jcouv](https://github.com/jcouv) | | [Native ints](https://github.com/dotnet/csharplang/issues/435) | main | [Merged into 16.7p1](https://github.com/dotnet/roslyn/issues/38821) | [cston](https://github.com/cston) | [333fred](https://github.com/333fred), [gafter](https://github.com/gafter) | [jaredpar](https://github.com/jaredpar) | | [Attributes on local functions](https://github.com/dotnet/csharplang/issues/1888) | [local-function-attributes](https://github.com/dotnet/roslyn/tree/features/local-function-attributes) | [Merged](https://github.com/dotnet/roslyn/issues/38801) | [RikkiGibson](https://github.com/RikkiGibson) | [agocke](https://github.com/agocke) | [agocke](https://github.com/agocke) | | [Function pointers](https://github.com/dotnet/csharplang/issues/191) | main | [Merged into 16.7p3](https://github.com/dotnet/roslyn/issues/43321) | [333fred](https://github.com/333fred) | [AlekseyTs](https://github.com/AlekseyTs) | [jaredpar](https://github.com/jaredpar) | | [Pattern matching improvements](https://github.com/dotnet/csharplang/issues/2850) | main | [Merged into 16.7p1](https://github.com/dotnet/roslyn/issues/40727) | [gafter](https://github.com/gafter) | [RikkiGibson](https://github.com/RikkiGibson),[agocke](https://github.com/agocke) | [gafter](https://github.com/gafter) | | [Static lambdas](https://github.com/dotnet/csharplang/issues/275) | main | [Merged in 16.7p4](https://github.com/dotnet/roslyn/issues/39606) | [CyrusNajmabadi](https://github.com/CyrusNajmabadi) | [jcouv](https://github.com/jcouv) | [jcouv](https://github.com/jcouv) | | [Records](https://github.com/dotnet/csharplang/issues/39) | main | [Merged into 16.7p3](https://github.com/dotnet/roslyn/issues/40726) | [agocke](https://github.com/agocke) | [gafter](https://github.com/gafter), [333fred](https://github.com/333fred) | [agocke](https://github.com/agocke) | | [Target-typed conditional](https://github.com/dotnet/csharplang/issues/2460) | [target-typing](https://github.com/dotnet/roslyn/tree/features/target-typing) | [Merged into 16.7p4](https://github.com/dotnet/roslyn/issues/43186) | [gafter](https://github.com/gafter) | [agocke](https://github.com/agocke), [RikkiGibson](https://github.com/RikkiGibson) | [gafter](https://github.com/gafter) | | [Covariant](https://github.com/dotnet/csharplang/issues/49) [Returns](https://github.com/dotnet/csharplang/issues/2844) | [covariant-returns](https://github.com/dotnet/roslyn/tree/features/covariant-returns) | [Merged into 16.8p2](https://github.com/dotnet/roslyn/issues/43188) | [gafter](https://github.com/gafter) | [AlekseyTs](https://github.com/AlekseyTs), [agocke](https://github.com/agocke) | [gafter](https://github.com/gafter) | | [Extension GetEnumerator](https://github.com/dotnet/csharplang/issues/3194) | main | [Merged into 16.8p2](https://github.com/dotnet/roslyn/issues/43184) | [YairHalberstadt](https://github.com/YairHalberstadt) | [333fred](https://github.com/333fred) | [333fred](https://github.com/333fred) | | [Module initializers](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-9.0/module-initializers.md) | main | [Merged in 16.7p4](https://github.com/dotnet/roslyn/issues/40500) | [RikkiGibson](https://github.com/RikkiGibson) [jnm2](https://github.com/jnm2)| [AlekseyTs](https://github.com/AlekseyTs) | [gafter](https://github.com/gafter) | | [Extending Partial](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-9.0/extending-partial-methods.md) | main | [Merged into 16.7p2](https://github.com/dotnet/roslyn/issues/43795) | [RikkiGibson](https://github.com/RikkiGibson) | [chsienki](https://github.com/chsienki) | [jaredpar](https://github.com/jaredpar) | | [Top-level statements](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-9.0/top-level-statements.md) | [SimplePrograms](https://github.com/dotnet/roslyn/tree/features/SimplePrograms) | [Merged into 16.7p3](https://github.com/dotnet/roslyn/issues/43563) | [AlekseyTs](https://github.com/AlekseyTs) | [cston](https://github.com/cston), [RikkiGibson](https://github.com/RikkiGibson) | [MadsTorgersen](https://github.com/MadsTorgersen) | # C# 8.0 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [Default Interface Methods](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/default-interface-methods.md) | main | [Merged to dev16.1 preview2](https://github.com/dotnet/roslyn/issues/19217) | [AlekseyTs](https://github.com/AlekseyTs) | [gafter](https://github.com/gafter) | [gafter](https://github.com/gafter) | | [Nullable reference type](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/nullable-reference-types-specification.md) | main | [Merged to dev16.0 preview1](https://github.com/dotnet/roslyn/issues/22152) | [cston](https://github.com/cston), [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs), [333fred](https://github.com/333fred) | [mattwar](https://github.com/mattwar) | | [Recursive patterns](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/patterns.md) | main | [Merged to dev16.0 preview2](https://github.com/dotnet/roslyn/issues/25935) | [gafter](https://github.com/gafter) | [agocke](https://github.com/agocke), [cston](https://github.com/cston) | [gafter](https://github.com/gafter) | | [Async streams](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/async-streams.md) | main | [Merged to dev16.0 preview1](https://github.com/dotnet/roslyn/issues/24037) | [jcouv](https://github.com/jcouv) | [agocke](https://github.com/agocke) | [stephentoub](https://github.com/stephentoub) | | [Enhanced using](https://github.com/dotnet/csharplang/issues/1623) | main | [Merged to dev16.0 preview2](https://github.com/dotnet/roslyn/issues/28588) | [chsienki](https://github.com/chsienki) | [agocke](https://github.com/agocke) | [jaredpar](https://github.com/jaredpar) | | [Ranges](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/ranges.md) | main | [Merged to dev16.0 preview1](https://github.com/dotnet/roslyn/issues/23205) | [agocke](https://github.com/agocke) | [cston](https://github.com/cston) | [jaredpar](https://github.com/jaredpar) | | [Null-coalescing Assignment](https://github.com/dotnet/csharplang/issues/34) | main | [Merged to dev16.0 preview1](https://github.com/dotnet/roslyn/issues/29168) | [333fred](https://github.com/333fred) | [cston](https://github.com/cston) | [gafter](https://github.com/gafter) | [Alternative interpolated verbatim strings](https://github.com/dotnet/csharplang/issues/1630) | main | Merged to dev16.0 preview1 | [jcouv](https://github.com/jcouv) | [cston](https://github.com/cston) | [jcouv](https://github.com/jcouv) | [stackalloc in nested contexts](https://github.com/dotnet/csharplang/issues/1412) | main | Merged to dev16.2 preview1 | [gafter](https://github.com/gafter) | - | [gafter](https://github.com/gafter) | [Unmanaged generic structs](https://github.com/dotnet/csharplang/issues/1744) | main | [Merged to dev16.1 preview1](https://github.com/dotnet/roslyn/issues/31374) | [RikkiGibson](https://github.com/RikkiGibson) | - | [jaredpar](https://github.com/jaredpar) | | [Static local functions](https://github.com/dotnet/csharplang/issues/1565) | main | [Merged in dev16.0 preview2](https://github.com/dotnet/roslyn/issues/32069) | [cston](https://github.com/cston) | [jaredpar](https://github.com/jaredpar) | [jcouv](https://github.com/jcouv) | [Readonly members](https://github.com/dotnet/csharplang/issues/1710) | main | [Merged to dev16.1 preview2](https://github.com/dotnet/roslyn/issues/32911) | [RikkiGibson](https://github.com/RikkiGibson) | TBD | [jaredpar](https://github.com/jaredpar) # VB 16.0 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [Line continuation comments](https://github.com/dotnet/vblang/issues/65) | main | Merged in 16.1 (preview2) | [paul1956](https://github.com/paul1956) | [AlekseyTs](https://github.com/AlekseyTs) | [gafter](https://github.com/gafter) | | [Relax null-coalescing operator requirements](https://github.com/dotnet/vblang/issues/339) | main | Merged in 16.0 | [333fred](https://github.com/333fred) | [cston](https://github.com/cston) | [gafter](https://github.com/gafter) | # C# 7.3 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [blittable](https://github.com/dotnet/csharplang/pull/206) | None | Merged | None | - | [jaredpar](https://github.com/jaredpar) | | [Support == and != for tuples](https://github.com/dotnet/csharplang/issues/190) | main | Merged | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs) | [jcouv](https://github.com/jcouv) | | strongname | main | Merged | [tyoverby](https://github.com/tyoverby) | [agocke](https://github.com/agocke) | [jaredpar](https://github.com/jaredpar) | | [Attribute on backing field](https://github.com/dotnet/csharplang/issues/42) | features/compiler | Merged | [jcouv](https://github.com/jcouv) | [AlekseyTs](https://github.com/AlekseyTs) | [jcouv](https://github.com/jcouv) | | Ref Reassignment | main | Merged | [agocke](https://github.com/agocke) | [vsadov](https://github.com/vsadov) | [gafter](https://github.com/gafter) | | Constraints | main | Merged | [OmarTawfik](https://github.com/OmarTawfik) | [vsadov](https://github.com/vsadov) | [gafter](https://github.com/gafter) | | [Stackalloc initializers](https://github.com/dotnet/csharplang/issues/1286) | main | Merged | [alrz](https://github.com/alrz) | [vsadov](https://github.com/vsadov) | [jcouv](https://github.com/jcouv) | | [Custom fixed](https://github.com/dotnet/csharplang/issues/1314) | main | Merged | [vsadov](https://github.com/vsadov) | [jcouv](https://github.com/jcouv) | [jarepdar](https://github.com/jaredpar) | | [Indexing movable fixed buffers](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.3/indexing-movable-fixed-fields.md) | - | Merged | [vsadov](https://github.com/vsadov) | [jcouv](https://github.com/jcouv) | [jarepdar](https://github.com/jaredpar) | | [Improved overload candidates](https://github.com/dotnet/csharplang/issues/98) | features/compiler | Merged | [gafter](https://github.com/gafter) | [cston](https://github.com/cston) | [mattwar](https://github.com/mattwar) | | [Expression variables](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.3/expression-variables-in-initializers.md) | main | Merged | [AlekseyTs](https://github.com/AlekseyTs) | [gafter](https://github.com/gafter) | [AlekseyTs](https://github.com/AlekseyTs) | # C# 7.2 fixes | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [`ref` and `this` ordering in ref extension](https://github.com/dotnet/roslyn/pull/23643) | - | Merged | [alrz](https://github.com/alrz) | - | - | | [Tiebreaker for by-val and `in` overloads](https://github.com/dotnet/roslyn/pull/23122) | main | Merged | [OmarTawfik](https://github.com/OmarTawfik) | - | - | # C# 7.2 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [ref readonly](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.2/readonly-ref.md) | main | Merged | [vsadov](https://github.com/vsadov), [OmarTawfik](https://github.com/OmarTawfik) | [cston](https://github.com/cston),[gafter](https://github.com/gafter) | [jaredpar](https://github.com/jaredpar) | | [interior pointer/Span/ref struct](https://github.com/dotnet/csharplang/pull/264) | main | Merged | [vsadov](https://github.com/vsadov) | [gafter](https://github.com/gafter), [jaredpar](https://github.com/jaredpar) | [jaredpar](https://github.com/jaredpar) | | [non-trailing named arguments](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.2/non-trailing-named-arguments.md) | main | Merged | [jcouv](https://github.com/jcouv) | [gafter](https://github.com/gafter) | [jcouv](https://github.com/jcouv) | | [private protected](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.2/private-protected.md) | main | Merged | [gafter](https://github.com/gafter) | [jcouv](https://github.com/jcouv) | [gafter](https://github.com/gafter) | | [conditional ref operator](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.2/conditional-ref.md) | main | Merged | [vsadov](https://github.com/vsadov) | [cston](https://github.com/cston) | [jaredpar](https://github.com/jaredpar) | | [Digit separator after base specifier](https://github.com/dotnet/roslyn/pull/20449) | main | Merged | [alrz](https://github.com/alrz) | - | [gafter](https://github.com/gafter) | # C# 7.1 | Feature | Branch | State | Developer | Reviewer | LDM Champ | | ------- | ------ | ----- | --------- | -------- | --------- | | [Async Main](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.1/async-main.md) | main | Merged | [tyoverby](https://github.com/tyoverby) | [vsadov](https://github.com/vsadov) | [stephentoub](https://github.com/stephentoub) | | [Default Expressions](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.1/target-typed-default.md) | main | Merged | [jcouv](https://github.com/jcouv) | [cston](https://github.com/cston) | [jcouv](https://github.com/jcouv) | | [Ref Assemblies](https://github.com/dotnet/roslyn/blob/main/docs/features/refout.md) | main | Merged (IDE and project-system integrations ongoing) | [jcouv](https://github.com/jcouv) | [gafter](https://github.com/gafter) | N/A | | [Infer tuple names](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.1/infer-tuple-names.md) | main | Merged | [jcouv](https://github.com/jcouv) | [gafter](https://github.com/gafter) | [jcouv](https://github.com/jcouv) | | [Pattern-matching with generics](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.1/generics-pattern-match.md) | main | Merged | [gafter](https://github.com/gafter) | [agocke](https://github.com/agocke) | [gafter](https://github.com/gafter) | # FAQ - **Is target version a guarantee?**: No. It's explicitly not a guarantee. This is just the planned and ongoing work to the best of our knowledge at this time. - **Where are these State values defined?**: Take a look at the [Developing a Language Feature](contributing/Developing%20a%20Language%20Feature.md) document. --- ### Layering (`docs/Layering.md`) # Roslyn Architecture - Project Layering Arrows point from a project **up to** its dependencies. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### TestAccessor Pattern (`docs/TestAccessor Pattern.md`) # The `TestAccessor` Pattern Prefer testing observable behavior through existing product entry points. Do not introduce a `TestAccessor` for exhaustive unit testing or to assert a class's specific implementation. When a scenario is critical to cover and cannot be adequately tested through existing observable behavior, the `TestAccessor` pattern allows production code to expose the minimum required internal functionality without making it available to other production code. The pattern has two primary components: 1. A `TestAccessor` type, which contains the functionality available only for testing 2. A `GetTestAccessor()` method, which returns an instance of `TestAccessor` The pattern relies on enforcement of a simple rule that no production code is allowed to call a `GetTestAccessor()` method. This is enforceable either through code reviews or through an analyzer. This pattern has many advantages over alternatives: * The pattern does not require expanded accessibility (e.g. `internal` instead of `private`) for the purpose of testing * The pattern is self-documenting: all properties and methods within a `TestAccessor` type are intended only for use in test code * The pattern is consistent enough to enforce through static analysis (analyzers) * The pattern is simple enough to enforce manually (code reviews) ## The `TestAccessor` Type The `TestAccessor` type is typically defined as a nested structure. In the following example, the `SomeProductionType.TestAccessor.PrivateStateData` property allows test code to read and write the value of the private field `SomeProductionType._privateStateData` without exposing the `_privateStateData` field to other production code. ```csharp internal class SomeProductionType { private int _privateStateData; internal readonly struct TestAccessor { private readonly SomeProductionType _instance; internal TestAccessor(SomeProductionType instance) { _instance = instance; } internal ref int PrivateStateData => ref _instance._privateStateData; } } ``` ## The `GetTestAccessor()` Method The `GetTestAccessor()` method is always defined as follows: ```csharp internal TestAccessor GetTestAccessor() => new TestAccessor(this); ``` --- ### Area Owners (`docs/area-owners.md`) # Pull Requests Tagging If you need to tag folks on an issue or PR, you will generally want to tag the owners (not the lead) for [area](#areas) to which the change or issue is closest to. ## Areas | Area | Lead | Owners | |----------------------------|---------|--------------------------------| |api-Syntax |@jaredpar|@CyrusNajmabadi @333fred | |api-Symbols |@jaredpar|@AlekseyTs | |api-SemanticModel |@jaredpar|@AlekseyTs @333fred | |api-IOperation |@jaredpar|@333fred | |api-SourceGenerators |@jaredpar|@chsienki | |api-Analyzers |@jaredpar|@mavasani @333fred | |api-CodeFixes (Refactoring) |@jinuj |@mavasani @sharwell | |api-Completion |@jinuj |@genlu @CyrusNajmabadi | |api-Debugging |@jinuj |@tmat @davidwengier | |api-EditAndContinue |@jinuj |@tmat @davidwengier | |api-EmbeddedLanguages |@jinuj |@CyrusNajmabadi @joerobich | |api-Formatting |@jinuj |@sharwell @joerobich | |api-LSP |@jinuj |@dibarbet @jasonmalinowski | |api-MetadataAsSource |@jinuj |@sharwell @jasonmalinowski | |api-Navigation |@jinuj |@CyrusNajmabadi @dibarbet | |api-QuickInfo |@jinuj |@sharwell @jasonmalinowski | |api-SignatureHelp |@jinuj |@sharwell @jasonmalinowski | |api-Structure (Outlining) |@jinuj |@CyrusNajmabadi @jasonmalinowski| |api-Tagging (Classification)|@jinuj |@CyrusNajmabadi @joerobich | |api-Workspace |@jinuj |@jasonmalinowski @tmat | --- ### Roslyn Language Server Copilot Plugin (`docs/roslyn-language-server-copilot-plugin.md`) # Roslyn Language Server - Copilot Plugin The `roslyn-language-server` is a .NET tool that provides C# language intelligence (go-to-definition, find references, diagnostics, etc.) to AI coding agents via the [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/). It is distributed as a plugin through the [dotnet/skills](https://github.com/dotnet/skills) marketplace, enabling automatic installation and use in tools like GitHub Copilot CLI. ## Quick Start For this repository specifically, the `dotnet` skill from the `dotnet-agent-skills` marketplace is checked in at `.github/copilot/settings.json`, which includes the appropriate `lsp.json`. The `.vscode/settings.json` points the server at `Roslyn.slnx`. That means opening this repo in Copilot CLI is enough to have the C# LSP configured automatically. ### Install the plugin In your agent (Copilot CLI, etc.), add the marketplace and install the `dotnet` plugin: ``` /plugin marketplace add dotnet/skills /plugin install dotnet@dotnet-agent-skills ``` Restart the agent to load the plugin. Once installed, the agent will automatically have access to C# language intelligence for `.cs` files through the LSP. ### What it provides With the plugin active, the agent gains LSP-powered capabilities for C# code: - **Go to Definition** — navigate to where a symbol is defined - **Find References** — find all usages of a symbol - **Hover** — get type information and documentation - **Diagnostics** — see compiler errors and warnings - **Document Symbols** — list all symbols in a file - **And more** — any LSP feature supported by the Roslyn language server ## Prerequisites - **.NET 10 SDK** must be installed and available on `PATH`. Your project can still use an older, supported SDK. ## Automatic Project Loading The server is configured to automatically discover and load projects via either the `--autoLoadProjects [maximum]` CLI flag (defaults to a maximum of 500 projects if value omitted) or LSP `initializationOptions` in the Copilot lsp.json as defined below. ```jsonc // lsp.json { "initializationOptions": { "autoLoadProjects": 500 } } ``` For how the server chooses which solution or projects to load, and how to control it, see [Automatic project loading](../src/LanguageServer/roslyn-language-server/README.md#automatic-project-loading). ## Troubleshooting ### Verify LSP configuration is found - In Copilot CLI, running the `/lsp show` should show the server configured for C#: ``` Plugin-configured servers: • csharp: (.cs) [from dotnet] ``` ### Viewing LSP Logs Copilot CLI writes LSP server logs to the `.copilot/logs/` directory in your home folder. To inspect the language server's output: - **macOS / Linux:** `~/.copilot/logs/` - **Windows:** `%USERPROFILE%\.copilot\logs\` Look for log files related to `csharp` or `roslyn-language-server`. These contain the server's startup output, project loading progress, and any errors encountered. This is the first place to check when the language server isn't behaving as expected. ### The tool isn't being found - Ensure `dotnet` is on your `PATH`. - Check for a `nuget.config` (repo, user, or machine-level) that restricts package sources; `dnx` uses NuGet sources to resolve tools. - Verify `nuget.org` (or an other feed that mirrors `roslyn-language-server`) is enabled, for example with `dotnet nuget list source`. - Try installing the tool manually: `dotnet tool install -g roslyn-language-server --prerelease`. ### Project load issues If the LSP logs show failures loading projects - Verify that a compatible .NET SDK is installed by running `dotnet --version`. - Check that your project builds successfully with `dotnet build` before using the language server. - For large repositories with multiple solutions, configure `dotnet.defaultSolution` in `.vscode/settings.json` to specify which solution to load. ### Performance - Loading a solution with many projects may take some time. The server reports progress to the client during loading. - For large repositories, prefer loading a specific solution (via `dotnet.defaultSolution`) rather than relying on individual project discovery, which may load test projects and other projects you don't need. ## Related Links - [dotnet/skills repository](https://github.com/dotnet/skills) — Plugin marketplace for .NET agent skills ## How It Works The plugin is configured in the [dotnet/skills](https://github.com/dotnet/skills) repository via [`plugins/dotnet/lsp.json`](https://github.com/dotnet/skills/blob/main/plugins/dotnet/lsp.json): When the agent opens a workspace containing `.cs` files, it will: 1. **Install and run** the [`roslyn-language-server`](https://www.nuget.org/packages/roslyn-language-server) .NET tool on-the-fly using `dotnet dnx` (which downloads and caches the tool automatically; `--yes` skips confirmation and `--prerelease` allows prerelease versions). 2. **Communicate over stdio** (`--stdio`) using the Language Server Protocol. 3. **Automatically discover and load projects** (`--autoLoadProjects`) so that the agent immediately has full semantic understanding of the codebase. ### Command-Line Options | Option | Description | |--------|-------------| | `--stdio` | Use stdio for LSP communication (required for most agent integrations) | | `--autoLoadProjects [maximum]` | Automatically discover and load projects; optionally limit individual project discovery (currently defaults to 500) | | `--logLevel ` | Minimum log verbosity (default: `Information`) | | `--debug` | Launch the debugger on startup | --- ### Analyzers Analyzer Actions Semantics (`docs/analyzers/Analyzer Actions Semantics.md`) **Definitions:** - **Session:** For batch compilation, a single compilation. For a hosted environment, an arbitrary duration potentially encompassing many compilations. In particular, for VS, a session shares the lifetime and scope of an analyzer reference, which may correspond to a loaded project, a loaded solution, or the VS process. - **Analyzer:** An instance of a type derived from `DiagnosticAnalyzer`. - **Action:** A method instance registered via a Register\*\*\*Action method of an instance of a \*\*\*Context class, to be automatically applied as appropriate. Every action is associated with an analyzer and has a kind (which is one of compilation start, syntax tree, symbol, code block start, syntax node, code block, code block end, semantic model, compilation, compilation end). - **Operation block:** A single unit of executable code containing operation nodes, e.g. a method body, parameter default value, field initializer, etc. Operation blocks are primarily used for semantic analysis of executable code using the `IOperation` APIs. - **Code block:** A single unit of executable code containing syntax nodes, e.g. a method body, parameter default value, field initializer, etc. Code blocks are primarily used for syntax and/or semantic analysis of executable code using the language specific Syntax APIs. However, it is recommended that analyzers use Operation block and `IOperation` API for semantic analysis of executable code for improved performance and simpler, language-agnostic implementation. - **Symbol members:** List of zero or member symbols for a given `ISymbol`. For a namespace or type symbol, these are the members returned by [INamespaceOrTypeSymbol.GetMembers](http://source.roslyn.io/#Microsoft.CodeAnalysis/Symbols/INamespaceOrTypeSymbol.cs,23) API, which include nested symbols. Method, field, event and property symbols have no member symbols. **Axioms:** - Two different analyzers are assumed not to share state. No actions of a single analyzer execute concurrently, unless they explicitly configure concurrent execution in the `DiagnosticAnalyzer.Initialize` method by invoking `AnalysisContext.EnableConcurrentExecution`. Actions of different analyzers may execute concurrently. (This is likely to be refined as we develop more sophisticated concurrency strategies, but it will remain true that actions will generally not require locks to avoid races.) - The context used to register an action affects the lifetime of the registration but does not otherwise affect the semantics of applying the action. - If an analyzer has multiple applicable actions of any given kind, all of those actions are applied in an arbitrary order. **Ordering of actions:** The `DiagnosticAnalyzer.Initialize` method of an analyzer is invoked before any actions of an analyzer can be applied. The actions registered by `Initialize` methods form the initial set of actions used for each compilation. `Initialize` method may be invoked multiple times on an analyzer instance in a session, but is guaranteed to be invoked only once for any given compilation. The following rules related to action invocation apply per compilation or per any environmental change that invalidates prior analysis (e.g. changes to analyzer options or changes to non-compiled documents used by analyzers). For purposes of this specification, such environmental changes act as if they create a new compilation. There are no ordering constraints for the invocations of any actions, except those introduced by start and end actions (compilation start, compilation end, symbol start, symbol end, operation block start, operation block end, code bock start, and code block end actions). Invocations of actions of one kind may be interleaved with invocations of actions of another kind. A given host environment might appear to provide a predictable order—for example, invoking syntax tree actions before semantic model actions, or symbol actions before code block actions, but depending on such an ordering is not safe. - A compilation start action is invoked once, before any of the analyzer’s other kinds of actions are invoked. Actions registered by a compilation start action apply to that compilation only. - A syntax tree action is invoked once per source document. A syntax tree action can be expected to be invoked as early as possible after a document is parsed, but this is not guaranteed. - A symbol action is invoked once per complete semantic processing of a declaration of a type or type member, provided that symbol has a kind matching one of the kinds supplied when the action was registered. - A symbol start action is invoked exactly once for every symbol of registered `SymbolKind`, before any of the analyzer’s other kinds of actions are invoked on the symbol or the symbol's members. A symbol start action can register following actions: - Actions for syntax and operation nodes within the symbol and its members: Operation action, operation block action, operation block start action, syntax node action, code block action and code block start action. - Symbol end action for the symbol, after all other applicable actions of an analyzer have been invoked on the symbol and its members. - An operation block start action is invoked once per operation block, before any operation node actions applicable to nodes within the block. Actions registered by a operation block start action apply to that operation block only. - An operation action is invoked once per operation node, provided that the operation node has a kind matching one of the kinds supplied when the action was registered. - An operation block action is invoked once per operation block. - An operation block end action is invoked once per operation block, after invoking any operation node actions applicable to nodes within the block and any operation block actions for the block. - A code block start action is invoked once per code block, before any syntax node actions applicable to nodes within the block. Actions registered by a code block start action apply to that code block only. - A syntax node action is invoked once per syntax node, provided that the syntax node has a kind matching one of the kinds supplied when the action was registered. - A code block action is invoked once per code block. - A code block end action is invoked once per code block, after invoking any syntax node actions applicable to nodes within the block and any code block actions for the block. - A semantic model action is invoked once per source document. - A compilation action is invoked once. - A compilation end action is invoked once, after all other applicable actions of an analyzer have been invoked. Summarizing the ordering rules: - For a given compilation, compilation start actions run first and compilation end actions run last. - For a given symbol, symbol start actions run before operation/syntax/symbol actions on a symbol and its members, and these operation/syntax/symbol actions run before the symbol end actions. - For a given operation block, operation block start actions run before operation node actions, and operation node actions run before the operation block end actions. - For a given code block, code block start actions run before syntax node actions, and syntax node actions run before code block end actions. - Otherwise, there are no ordering guarantees for action invocation. **Implementation freedom:** If an implementation determines that a given action will have equivalent effects in two compilations within a session, it can omit invoking the action for the second compilation and instead retain the effects of invoking it for the first compilation. For example, in an IDE, edits to a source document will not necessarily cause actions to be invoked for unrelated symbols etc. defined in other documents. A host environment may delay invocation of actions arbitrarily and indefinitely. For example, an IDE might delay invoking some actions until a significant pause in editing activity, during which time several compilations might be initiated. Some actions might not execute at all for the interim compilations, because they will be cancelled when the next compilation begins. In such cases, it is possible for a compilation start action to run without its corresponding compilation end actions ever running for a given compilation, and likewise a code block start action can run without its corresponding code block end actions ever running. Therefore, end actions cannot be relied upon to free resources acquired by start actions. --- ### Analyzers Analyzer Samples (`docs/analyzers/Analyzer Samples.md`) **Samples Location:** Sample analyzers to demonstrate recommended implementation models for different analysis scenarios have been added to [Samples.sln](https://github.com/dotnet/roslyn-sdk/tree/main/Samples.sln). **Description:** Analyzers have been broadly categorized into the following three buckets based on the kind of analysis performed: 1. Stateless analyzers: Analyzers that report diagnostics about a specific code unit, such as a symbol, syntax node, code block, compilation, etc. Most analyzers would fall into this bucket. These analyzers register one or more actions, that: 1. Do not require maintaining state across analyzer actions and are 2. Independent of the order of execution of individual analyzer actions. 2. Stateful analyzers: Analyzers that report diagnostics about a specific code unit, but in context of an enclosing code unit such as a code block or a compilation. These represent slightly more complicated analyzers that require more powerful and wider analysis and hence need careful design to achieve efficient analyzer execution without memory leaks. These analyzers require at least one of the following kind of state manipulation for analysis: 1. Access to immutable state object(s) for the enclosing code unit, such as a compilation or the code block. For example, access to certain well known types defined in a compilation. 2. Perform analysis over the enclosing code unit, with mutable state defined and initialized in a start action for the enclosing code unit, intermediate actions that access and/or update this state, and an end action to report diagnostic on the individual code units. 3. Additional File analyzers: Analyzers that read data from non-source text files included in the project. **Contents:** Following sample analyzers, with simple unit tests, are provided: 1. [Stateless analyzers](https://github.com/dotnet/roslyn-sdk/tree/master/samples/CSharp/Analyzers/Analyzers.Implementation/StatelessAnalyzers): 1. SymbolAnalyzer: Analyzer for reporting symbol diagnostics. 2. SyntaxNodeAnalyzer: Analyzer for reporting syntax node diagnostics. 3. CodeBlockAnalyzer: Analyzer for reporting code block diagnostics. 4. CompilationAnalyzer: Analyzer for reporting compilation diagnostics. 5. SyntaxTreeAnalyzer: Analyzer for reporting syntax tree diagnostics. 6. SemanticModelAnalyzer: Analyzer for reporting syntax tree diagnostics, that require some semantic analysis. 2. [Stateful analyzers](https://github.com/dotnet/roslyn-sdk/tree/master/samples/CSharp/Analyzers/Analyzers.Implementation/StatefulAnalyzers): 1. CodeBlockStartedAnalyzer: Analyzer to demonstrate code block wide analysis. 2. CompilationStartedAnalyzer: Analyzer to demonstrate analysis within a compilation, for example analysis that depends on certain well-known symbol(s). 3. CompilationStartedAnalyzerWithCompilationWideAnalysis: Analyzer to demonstrate compilation-wide analysis. 3. [Additional File analyzers](https://github.com/dotnet/roslyn-sdk/tree/master/samples/CSharp/Analyzers/Analyzers.Implementation/AdditionalFileAnalyzers): 1. SimpleAdditionalFileAnalyzer: Demonstrates reading an additional file line-by-line and using the data in analysis. 2. XmlAdditionalFileAnalyzer: Demonstrates writing an additional file out to a `Stream` so that it can be read back as a structured document (in this case, XML). --- ### Analyzers DiagnosticSuppressorDesign (`docs/analyzers/DiagnosticSuppressorDesign.md`) # Feature Request https://github.com/dotnet/roslyn/issues/30172: _Programmatic suppression of warnings_ Provide an ability for platform/library authors to author simple, context-aware compiler extensions to programmatically suppress specific instances of reported analyzer and/or compiler diagnostics, which are always known to be false positives in the context of the platform/library. # Programmatically Suppressible Diagnostic An analyzer/compiler diagnostic would be considered a candidate for programmatic suppression if **all** of the following conditions hold: 1. _Not an error by default_: Diagnostic's [DefaultSeverity](http://source.roslyn.io/#Microsoft.CodeAnalysis/Diagnostic/Diagnostic.cs,8e27a878b4d6e40c) is *not* [DiagnosticSeverity.Error](http://source.roslyn.io/#Microsoft.CodeAnalysis/Diagnostic/DiagnosticSeverity.cs,f771032fb5a00c1c). 2. _Must be configurable_: Diagnostic is *not* tagged with [WellKnownDiagnosticTags.NotConfigurable](http://source.roslyn.io/#Microsoft.CodeAnalysis/Diagnostic/WellKnownDiagnosticTags.cs,207e57dd0b96bd4b) custom tag, indicating that its severity is configurable. 3. _No existing source suppression_: Diagnostic is *not* already suppressed in source via pragma/suppress message attribute. # Core Design 1. _For platform/library authors:_ Expose a new "DiagnosticSuppressor" public API for authoring such compiler extensions. API contract for the suppressors (detailed API with doc comments in the [last section](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#detailed-api-proposal-with-documentation-comments-from-the-draft-pr)): 1. Declaratively provide all the analyzer and/or compiler diagnostic IDs that can be suppressed by it. 2. For each such suppressible diagnostic ID, provide a unique suppression ID and justification. These are required to enable proper diagnosis and configuration of each suppression (covered in detail [later](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#example-experience-with-screenshots)). 3. Register callbacks for reported analyzer and/or compiler diagnostics with these suppressible diagnostic IDs. The callback can analyze the syntax and/or semantics of the diagnostic location and report a suppression for it. 4. DiagnosticSuppressors would *not* be able to register any other analysis callbacks, and hence cannot report any new diagnostics. 2. _For end users:_ 1. Seamless development experience when targeting such a platform/library, whereby the users do not see false positives from analyzers/compiler in their development context. 2. End users do not need to manually add and/or maintain source suppressions through pragmas/SuppressMessageAttributes for such context-specific false positives. 3. End users still have the ultimate control over suppressors (specifics covered in [later section](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#example-experience-with-screenshots)): 1. Audit: Diagnostic suppressions are logged as "Info" diagnostics on the command line, so the end users can audit each suppression in the verbose build log or msbuild binlog. Additionally, they are also logged in the [/errorlog SARIF file](https://github.com/dotnet/roslyn/blob/master/docs/compilers/Error%20Log%20Format.md) as suppressed diagnostics. 2. Configuration: Each diagnostic suppression has an associated suppression ID. This is clearly indicated in the logged "Info" diagnostics for suppressions, and the end users can disable the bucket of suppressions under any specific suppression ID with simple command line argument or ruleset entries. 3. _For analyzer driver:_ 1. Order of execution between analyzers/compiler and suppressors: For command line builds, all diagnostic suppressors will run **after** all the analyzer and compiler diagnostics have been computed. For live analysis in Visual Studio, diagnostic suppressors may be invoked with a subset of the full set of reported diagnostics, as an optimization for supporting incremental and partial analysis scenarios. 2. Order of execution between individual suppressors: Execution of diagnostic suppressors would be independent of other suppressors. Diagnostic suppressions reported by one suppressor would not affect the input set of reported diagnostics passed to any other suppressor. Each diagnostic might be programmatically suppressed by more than one suppressor, so the analyzer driver will union all the reported suppressions for each diagnostic. Command line compiler will log one suppression diagnostic per-suppression for each programmatically suppressed diagnostic. # Development Experience Example Let us consider the core example for UnityEngine covered in https://github.com/dotnet/roslyn/issues/30172: ```csharp using UnityEngine; class BulletMovement : MonoBehaviour { [SerializeField] private float speed; private void Update() { this.transform.position *= speed; } } ``` Serialization frameworks and tools embedding .NET Core or Mono often set the value of fields or call methods outside of user code. Such code is not available for analysis to the compiler and analyzers, leading to false reports such as following: 1. [CS0649](https://docs.microsoft.com/en-us/dotnet/csharp/misc/cs0649): `Field 'speed' is never assigned to, and will always have its default value 0` 2. [IDE0044 dotnet_style_readonly_field](https://docs.microsoft.com/en-us/visualstudio/ide/editorconfig-code-style-settings-reference?view=vs-2019): Make field 'speed' readonly ## New experience with screenshots: 1. _For platform/library authors:_ UnityEngine can selectively suppress instances of such diagnostics by writing a simple [SerializedFieldDiagnosticSuppressor](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-serializedfielddiagnosticsuppressor-cs) extension based on the DiagnosticSuppressor API and packaging it along with the Unity framework SDK/library. 2. _For end users:_ 1. Users do not see above false positives in the IDE live analysis or command line build. For example, see the below screenshots with and without the SerializedFieldDiagnosticSuppressor: 1. Without the suppressor: See image [here](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-without_suppressor-png) 2. With the suppressor: See image [here](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-with_suppressor-png) 2. Audit suppressions: 1. Command line build: End users will see "Info" severity suppression diagnostics in command line builds, see image [here](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-suppression_info_diagnostic-png) `info SPR0001: Diagnostic 'CS0649' was programmatically suppressed by a DiagnosticSuppressor with suppresion ID 'SPR1001' and justification 'Field symbols marked with SerializedFieldAttribute are implicitly assigned by UnityEngine at runtime and hence do not have its default value'` These diagnostics will *always* be logged in the [msbuild binlog](https://github.com/Microsoft/msbuild/blob/master/documentation/wiki/Binary-Log.md). They would not be visible in the console output for regular msbuild invocations, but increasing the msbuild verbosity to detailed or diagnostic will emit these diagnostics. 2. Visual Studio IDE: Users can see the original suppressed diagnostic in the Visual Studio Error List when they change the default filter for the "Suppression State" column in the error list to include "Suppressed" diagnostics, see image [here](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-view_suppressed_diagnostic_errorlist-png). In future, we also plan to add a new error list column, possibly named "Suppression Source", which will display the suppression ID and justification associated with each programmatic suppression. 3. Disable suppressors: End users can disable specific suppression IDs from an analyzer package using the following two mechanisms: 1. `/nowarn:<%suppression_id%>`: See image [here](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-disable_suppressor_nowarn-png) for disabling the suppression ID from the project property page, which in turn generates the nowarn command line argument. 2. Ruleset entry: See image [here](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#file-disable_suppressor_rulesetentry-png) for disabling the suppression ID using a [CodeAnalysis ruleset](https://docs.microsoft.com/en-us/visualstudio/code-quality/using-rule-sets-to-group-code-analysis-rules?view=vs-2019) # Open Questions: 1. _Should the DiagnsoticSuppressor feature be Opt-in OR on-by-default:_ This was brought up few times in the past, especially with the concern with _on-by-default_ behavior being that an end user would not see any indication in the command line that diagnostic suppressors were executed as part of their build. I think we have following options: 1. Opt-in: 1. Keep the feature behind a new, _permanent_ feature flag: This way the command line arguments will always indicate if diagnostic suppressors were involved in the build. 2. Add a new command line switch to enable suppressors 2. On-By-default: No command line argument indicating suppressors are involved in a build, but users can look at binlogs and/or verbose msbuild output logs to determine the same. The [draft PR](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#draft-pr-implementing-the-above-proposal) for this feature currently puts the feature being a feature flag. If we decide to take the _on-by-default_ route, I can revert the changes adding the feature flag. Otherwise, if we decide that we should make it opt-in with a new command line compiler switch, I would like to propose that we keep the feature flag for the initial PR, and then have a follow-up PR to add the command line switch. **Update:** We have decided to keep the feature being a temporary feature flag for initial release, which will be removed once we are confident about the feature's performance, user experience, etc. 2. _Message for the "Info" suppression diagnostic_: I have chosen the below message format (with 3 format arguments), but any suggested changes are welcome: `Diagnostic 'CS0649' was programmatically suppressed by a DiagnosticSuppressor with suppresion ID 'SPR1001' and justification 'Field symbols marked with SerializedFieldAttribute are implicitly assigned by UnityEngine at runtime and hence do not have its default value'` 3. _What should be the diagnostic ID for the "Info" suppression diagnostic?_: We have following options: 1. Use a CSxxxx/BCxxxx diagnostic ID for the suppression diagnostic so it is evident that the diagnostic is coming from the core compiler. 2. Use a distinct diagnostic prefix, such as 'SPR0001', similar to the way we report 'AD0001' diagnostics for analyzer exceptions. This provides a separate bucketing/categorization for these special suppression diagnostics. The [draft PR](https://gist.github.com/mavasani/fcac17a9581b5c54cef8a689eeec954a#draft-pr-implementing-the-above-proposal) chooses the second approach with 'SPR0001' as the diagnostic ID. **Update:** We have decided the diagnostic ID would be `SP0001`. # Draft PR implementing the above proposal: https://github.com/dotnet/roslyn/pull/36067 # Detailed API proposal with documentation comments from the Draft PR ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Analyzers FixAllProvider (`docs/analyzers/FixAllProvider.md`) # Introduction This document covers the following: - Introductory definitions - What is a FixAll occurrences code fix? - How is a FixAll occurrences code fix computed? - Adding FixAll support to your code fixer - Selecting an Equivalence key for code actions - Spectrum of FixAll providers - Built-in FixAllProvider and its limitations - Implementing a custom FixAllProvider # Definitions - **Analyzer:** An instance of a type derived from `DiagnosticAnalyzer` that reports diagnostics. - **Code fixer:** An instance of a type derived from `CodeFixProvider` that provides code fixes for compiler and/or analyzer diagnostics. - **Code refactoring:** An instance of a type derived from `CodeRefactoringProvider` that provides source code refactorings. - **Code action:** An action registered by `CodeFixProvider.RegisterCodeFixesAsync` that performs a code fix OR an action registered by `CodeRefactoringProvider.ComputeRefactoringsAsync` that performs a code refactoring. - **Equivalence Key:** A string value representing an equivalence class of all code actions registered by a code fixer or refactoring. Two code actions are treated as equivalent if they have equal `EquivalenceKey` values and were generated by the same code fixer or refactoring. - **FixAll provider:** An instance of a type derived from `FixAllProvider` that provides a FixAll occurrences code fix. A FixAll provider is associated with a corresponding code fixer by `CodeFixProvider.GetFixAllProvider` method. - **FixAll occurrences code fix:** A code action returned by `FixAllProvider.GetFixAsync`, that fixes all or multiple occurrences of diagnostics fixed by the corresponding code fixer, within a given `FixAllScope`. # What is a FixAll occurrences code fix? In layman terms, a FixAll occurrences code fix means: I have a code fix 'C', that fixes a specific instance of diagnostic 'D' in my source and I want to apply this fix to all instances of 'D' across a broader scope, such as a document or a project or the entire solution. In more technical terms: Given a particular code action registered by a code fixer to fix one or more diagnostics, a corresponding code action registered by its FixAll provider, that applies the original trigger code action across a broader scope (such as a document/project/solution) to fix multiple instances of such diagnostics. # How is a FixAll occurrences code fix computed? Following steps are used to compute a FixAll occurrences code fix: - Given a specific instance of a diagnostic, compute the set of code actions that claim to fix the diagnostic. - Select a specific code action from this set. In the Visual Studio IDE, this is done by selecting a specific code action in the light bulb menu. - The equivalence key of the selected code action represents the class of code actions that must be applied as part of a FixAll occurrences code fix. - Given this code action, get the FixAll provider corresponding to the code fixer that registered this code action. - If non-null, then request the FixAll provider for its supported FixAllScopes. - Select a specific `FixAllScope` from this set. In the Visual Studio IDE, this is done by clicking on the scope hyperlink in the preview dialog. - Given the trigger diagnostic(s), the equivalence key of the trigger code action, and the FixAll scope, invoke `FixAllProvider.GetFixAsync` to compute the FixAll occurrences code fix. # Adding FixAll support to your code fixer Follow the below steps to add FixAll support to your code fixer: - Override the `CodeFixProvider.GetFixAllProvider` method and return a non-null instance of a `FixAllProvider`. You may either use our built-in FixAllProvider or implement a custom FixAllProvider. See the following sections in this document for determining the correct approach for your fixer. - Ensure that all the code actions registered by your code fixer have a non-null equivalence key. See the following section to determine how to select an equivalence key. # Selecting an Equivalence key for code actions Each unique equivalence key for a code fixer defines a unique equivalence class of code actions. Equivalence key of the trigger code action is part of the `FixAllContext` and is used to determine the FixAll occurrences code fix. Normally, you can use the **'title'** of the code action as the equivalence key. However, there are cases where you may desire to have different values. Let us take an example to get a better understanding. Let us consider the [C# SimplifyTypeNamesCodeFixProvider](https://github.com/dotnet/roslyn/blob/main/src/Features/CSharp/Portable/SimplifyTypeNames/SimplifyTypeNamesCodeFixProvider.cs) that registers multiple code actions and also has FixAll support. This code fixer offers fixes to simplify the following expressions: - `this` expressions of the form 'this.x' to 'x'. - Qualified type names of the form 'A.B' to 'B'. - Member access expressions of the form 'A.M' to 'M'. This fixer needs the following semantics for the corresponding FixAll occurrences code fixes: - `this` expression simplification: Fix all should simplify all this expressions, regardless of the member being accessed (this.x, this.y, this.z, etc.). - Qualified type name simplification: Fix all should simplify all qualified type names 'A.B' to 'B'. However, we don't want to simplify **all** qualified type names, such as 'C.D', 'E.F', etc. as that would be too generic a fix, which is not likely intended by the user. - Member access expressions: Fix all should simplify all member access expressions 'A.M' to 'M'. It uses the below equivalence keys for its registered code actions to get the desired FixAll behavior: - `this` expression simplification: Generic resource string "Simplify this expression", which explicitly excludes the contents of the node being simplified. - Qualified type name simplification: Formatted resource string "Simplify type name A.B", which explicitly includes the contents of the node being simplified. - Member access expressions: Formatted resource string "Simplify type name A.M", which explicitly includes the contents of the node being simplified. Note that '`this` expression simplification' fix requires a different kind of an equivalence class from the other two simplifications. See method [GetCodeActionId](https://github.com/dotnet/roslyn/blob/main/src/Analyzers/Core/CodeFixes/ImplementAbstractClass/AbstractImplementAbstractClassCodeFixProvider.cs) for the actual implementation. To summarize, use the equivalence key that best suits the category of fixes to be applied as part of a FixAll operation. # Spectrum of FixAll providers When multiple fixes need to be applied to documents, there are various ways to do it: - **Sequential approach**: One way to do it is to compute diagnostics, pick one, ask a fixer to produce a code action to fix that, apply it. Now for the resulting new compilation, recompute diagnostics, pick the next one and repeat the process. This approach would be very slow but would lead to correct results (unless it doesn't converge where one fix introduces a diagnostic that was just fixed by a previous fix). We chose to not implement this approach. - **Batch fix approach**: Another way is to compute all the diagnostics, pick each diagnostic and give it to a fixer and apply it to produce a new solution. If there were 'n' diagnostics, there would be 'n' new solutions. Now just merge them all together in one go. This may produce incorrect results (when different fixes change the same region of code in different ways) but it is very fast. We have one implementation of this approach in `WellKnownFixAllProviders.BatchFixer` - **Custom approach**: Depending on the fix, there may be a custom solution to fix multiple issues. For example, consider an analyzer that simply needs to generate one file as the fix for any instance of the issue. Instead of generating the same file over and over using the previous two approaches, one could write a custom `FixAllProvider` that simply generates the file once if there were any diagnostics at all. Since there are various ways of fixing all issues, we've implemented a framework and provided the one general implementation that we think is useful in many cases. # Built-in FixAllProvider We provide a default `BatchFixAllProvider` implementation of a FixAll provider that uses the underlying code fixer to compute the FixAll occurrences code fixes. To use the batch fixer, you should return the static `WellKnownFixAllProviders.BatchFixer` instance in the `CodeFixProvider.GetFixAllProvider` override. NOTE: See the following section on **'Limitations of the BatchFixer'** to determine if the batch fixer can be used by your code fixer. Given a trigger diagnostic, a trigger code action, the underlying code fixer and the FixAll scope, the BatchFixer computes FixAll occurrences code fix with the following steps: - Compute all instances of the trigger diagnostic across the FixAll scope. - For each computed diagnostic, invoke the underlying code fixer to compute the set of code actions to fix the diagnostic. - Collect all the registered code actions that have the same equivalence key as the trigger code action. - Apply all these code actions on the original solution snapshot to compute new solution snapshots. The batch fixer only batches code action operations of type `ApplyChangesOperation` present within the individual code actions, other types of operations are ignored. - Sequentially merge the new solution snapshots into a final solution snapshot. Only non-conflicting code actions whose fix spans don't overlap the fix spans of prior merged code actions are retained. # Limitations of the BatchFixer The BatchFixer is designed for a common category of fixers where fix spans for diagnostics don't overlap with each other. For example, assume there is a diagnostic that spans a particular expression, and a fixer that fixes that expression. If all the instances of this diagnostic are guaranteed to have non-overlapping spans, then their fixes can be computed independently and this batch of fixes can be subsequently merged together. However, there are cases where the BatchFixer might not work for your fixer. Following are some such examples: - Code fixer registers code actions without an equivalence key or with a null equivalence key. - Code fixer registers non-local code actions, i.e. a code action whose fix span is completely distinct from diagnostic span. For example, a fix that adds a new declaration node. Multiple such fixes are likely to have overlapping spans and hence could be conflicting. - Diagnostics to be fixed as part of FixAll occurrences have overlapping spans. It is likely that fixes for such diagnostics will have overlapping spans too, and hence would conflict with each other. - Code fixer registers code actions with operations other than `ApplyChangesOperation`. BatchFixer ignores such operations and hence may produce unexpected results. # Implementing a custom FixAllProvider For cases where you cannot use the BatchFixer, you must implement your own `FixAllProvider`. It is recommended that you create a singleton instance of the FixAll provider, instead of creating a new instance for every `CodeFixProvider.GetFixAllProvider` invocation. Following guidelines should help in the implementation: - **GetFixAsync:** Primary method to compute the FixAll occurrences code fix for a given `FixAllContext`. You may use the set of 'GetXXXDiagnosticsAsync' methods on the `FixAllContext` to compute the diagnostics to be fixed. You must return a single code action that fixes all the diagnostics in the given FixAll scope. - **GetSupportedFixAllScopes:** Virtual method to get all the supported FixAll scopes. By default, it returns all the three supported scopes: document, project and solution scopes. Generally, you need not override this method. However, you may do so if you wish to support a subset of these scopes. - **GetSupportedFixAllDiagnosticIds:** Virtual method to get all the fixable diagnostic ids. By default, it returns the underlying code fixer's `FixableDiagnosticIds`. Generally, you need not override this method. However, you may do so if you wish to support FixAll only for a subset of these ids. See [DeclarePublicAPIFix](https://github.com/dotnet/roslyn/blob/main/src/RoslynAnalyzers/PublicApiAnalyzers/Core/CodeFixes/DeclarePublicApiFix.cs) for an example implementation of a custom FixAllProvider. --- ### Analyzers Localizing Analyzers (`docs/analyzers/Localizing Analyzers.md`) Introduction ============ This document covers the following: * Setting up your project to use localized resources * Using localized resources in Analyzers * Using localized resources in Code Fixes * Producing localized resource assemblies * Packaging resource assemblies in NuGet packages and VSIXes * Testing the localization at the command line and in VS Setting up the Project ======================= You'll need to create a .resx file to hold your localizable resources. For the purposes of this document, we'll assume you are putting them in a file named 'AnalyzerResources.resx'. Using Localized Resources in Analyzers -------------------------------------- Normally, .NET applications can simply access the resource through the type generated from the .resx file: ``` C# string message = AnalyzerResources.Message; ``` and the framework will use the culture specified in `System.Globalization.CultureInfo.CurrentUICulture` to automatically find and return the appropriately localized resource. However, the C# and VB compilers provide the `/preferreduilang` switch to specify the use of resources for a different culture from that in `CurrentUICulture`. Because of that accessing `AnalyzerResources.Message` directly within an analyzer will not return the correct resource. Instead you need to access these resources through the `LocalizableResourceString` type. For example: ``` C# LocalizableResourceString message = new LocalizableResourceString(nameof(AnalyzerResources.Message), AnalyzerResources.ResourceManager, typeof(AnalyzerResources)); ``` Using Localized Resources in Code Fixes --------------------------------------- Code Fixes, on the other hand, are not affected by the `/preferreduilang` switch. Resources in Code Fixes should be accessed in the standard way: ``` C# string codeFixMessage = AnalyzerResources.CodeFixMessage; ``` Producing Localized Resource Assemblies ======================================= To create resource assemblies, you simply need to add additional .resx files with the desired [language name](https://msdn.microsoft.com/en-us/library/windows/desktop/dd318696(v=vs.85).aspx) in the file name. For example, if you wanted to create a resource assembly with German strings, you would add AnalyzerResources.de.resx. For Japanese, you add AnalyzerResources.ja.resx, and so on. Inside these files you add items where the names match those in AnalyzerResources.resx and the values have the localized content. If AnalyzerResources.resx contains the following: Name | Value --------|------- Message | Hello! Then AnalyzerResources.de.resx should contain this: Name | Value --------|----------- Message | Guten Tag! Now when you build resource assemblies are automatically produced in language-specific sub folders: - bin\Debug\ - MyAnalyzer.dll - MyAnalyzer.pdb - de\ - MyAnalyzer.resources.dll - ja\ - MyAnalyzer.resources.dll This layout is also what the CLR expects at runtime: for a given assembly Foo.dll, the localized resources should be in language-specific folders next to the assembly. Packaging Resource Assemblies ============================= NuGet ----- NuGet packaging is straight-forward, as you simply need to duplicate the structure of the build output: ``` XML ... ``` VSIX ---- Extension projects automatically incorporate satellite assemblies into the resulting .vsix file. Testing Analyzer Localization ============================= At the command line ------------------- Once the NuGet package is installed, you can force the build to use localized resources by passing the `/preferreduilang` flag to csc.exe or vbc.exe: csc.exe /preferreduilang:de /t:library /analyzer:... Foo.cs Alternatively, you can pass it to MSBuild as a property: msbuild /p:PreferredUILang=de or set it as a property in the *proj file itself: de In VS ----- In VS analyzers use the selected UI language. This can be changed by selecting "Options..." from the "Tools" menu, and navigating to Environment -> International Settings. --- ### Analyzers Report Analyzer Format (`docs/analyzers/Report Analyzer Format.md`) Introduction ============ The C# and Visual Basic compilers support a `/reportanalyzer` switch on the command line to report additional analyzer information, such as execution time. The output contains the total wall clock time spent in executing the analyzers and the relative execution times per-analyzer. Note that the elapsed time may be less than analyzer execution time because analyzers can run concurrently. One should use this data only for comparative analysis to identify any outlier analyzer which takes signinficantly more time than other analyzers. MSBuild command ============= Use the following msbuild command line to get the analyzer performance report: ``` msbuild.exe /v:d /p:reportanalyzer=true <%project_file%> ``` Output Format ============= ``` Total analyzer execution time: XYZ seconds. NOTE: Elapsed time may be less than analyzer execution time because analyzers can run concurrently. ``` Time (s) | % | Analyzer ----------|---|---------------------- xyz1 | X | Analyzer Assembly Identity xyz2 | Y | - DiagnosticAnalyzer1 xyz3 | Z | - DiagnosticAnalyzer2 Example ============= Following is an example output from a project built with analyzers: ------------------------------------------------------------------------------------------------------- ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Analyzers Using Additional Files (`docs/analyzers/Using Additional Files.md`) Introduction ============ This document covers the following: * Uses of Additional Files * Passing Additional Files on the command line * Specifying an individual item as an `AdditionalFile` in an MSBuild project * Specifying an entire set of items as `AdditionalFile`s in an MSBuild project * Accessing and reading additional files through the `AnalyzerOptions` type * Includes sample analyzers that read additional files Uses ==== Sometimes an analyzer needs access to information that is not available through normal compiler inputs--source files, references, and options. To support these scenarios the C# and Visual Basic compilers can accept additional, non-source text files as inputs. For example, an analyzer may enforce that a set of banned terms is not used within a project, or that every source file has a certain copyright header. The terms or copyright header could be passed to the analyzer as an additional file, rather than being hard-coded in the analyzer itself. On the Command Line =================== On the command line, additional files can be passed using the `/additionalfile` option. For example: ``` csc.exe alpha.cs /additionalfile:terms.txt ``` In a Project File ================= Passing an Individual File -------------------------- To specify an individual project item as an additional file, set the item type to `AdditionalFiles`: ``` XML ``` Passing a Group of Files ------------------------ Sometimes it isn't possible to change the item type, or a whole set of items need to be passed as additional files. In this situation you can update the `AdditionalFileItemNames` property to specify which item types to include. For example, if your analyzer needs access to all .resx files in the project, you can do the following: ``` XML $(AdditionalFileItemNames);EmbeddedResource ... ``` Accessing Additional Files ========================== The set of additional files can be accessed via the `Options` property of the context object passed to a diagnostic action. For example: ```C# CompilationAnalysisContext context = ...; ImmutableArray additionFiles = context.Options.AdditionalFiles; ``` From an `AdditionalText` instance, you can access the path to the file or the contents as a `SourceText`: ``` C# AdditionText additionalFile = ...; string path = additionalFile.Path; SourceText contents = additionalFile.GetText(); ``` Samples ======= Reading a File Line-by-Line --------------------------- This sample reads a simple text file for a set of terms--one per line--that should not be used in type names. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Converting a File to a Stream ----------------------------- In cases where an additional file contains structured data (e.g., XML or JSON) the line-by-line access provided by the `SourceText` may not be desirable. One alternative is to convert a `SourceText` to a `string`, by calling `ToString()` on it. This sample demonstrates another alternative: converting a `SourceText` to a `Stream` for consumption by other libraries. The terms file is assumed to have the following format: ``` XML frob wizbang orange ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Compilers Boolean Representation (`docs/compilers/Boolean Representation.md`) Representation of Boolean Values ================================ The C# and VB compilers represent `true` (`True`) and `false` (`False`) `bool` (`Boolean`) values with the single byte values `1` and `0`, respectively, and assume that any boolean values that they are working with are restricted to being represented by these two underlying values. The ECMA 335 CLI specification permits a "true" boolean value to be represented by any nonzero value. If you use boolean values that have an underlying representation other than `0` or `1`, you can get unexpected results. This can occur in `unsafe` code in C#, or by interoperating with a language that permits other values. To avoid these unexpected results, it is the programmer's responsibility to normalize such incoming values. See also https://github.com/dotnet/roslyn/issues/24652 --- ### Compilers Co Located Core Types (`docs/compilers/Co-located core types.md`) Co-located core types ===================== ## Customer Scenario and Context The ECMA 335 spec indicates that a few types, such as `System.Object`, or `System.Int16`, can be referenced in metadata as an integer value rather than a type ref token (built-in types). Additionally, the spec indicates that certain conversions are allowed between these types and a small set of token-referenced types – we shall refer to these as special types. Finally, a few types have special runtime treatment without being referenced like the other built-in types. Scenarios 1. A compiler needs to find the type definition for a built-in type, given a reference to it. Given that the reference is not a type token, the compiler needs to know the assembly containing the def 2. A compiler encounters two definitions for a built-in type 3. A compiler is asked to type check two references to a built-in type, one of which is originating from a library compiled to an incompatible definition of core types ## Design Compilers may assume that there is exactly one assembly representing the "core assembly". This core assembly is either referenced or is currently being compiled. The core assembly must not reference any other assemblies. The core assembly contains the type definitions for the subset of built-in and special types supported on a platform. References to types having the same name as built-in types but found outside the core assembly are made via type tokens. An assembly `A` referencing only built-in types may exclude (as per ECMA spec) a reference to the core library. In the case `A` is further referenced by the compiler when compiling a second assembly `B`, the references to the built-in types in `A` are resolved by the compiler based on the core library used for compiling `B`. A platform implementation may choose to define a core assembly with only a subset of the built-in and special types. Compilation in such a scenario succeeds, as long as built-in or special types outside of the subset are not referenced, directly (by user code) or indirectly (by compiler features). This means that, while the compiler can safely assume that all built-in and special types are co-located in the same assembly, they may not all be present. If a later version of the platform introduces more built-in or special types, they must do so in the core assembly. Correspondingly, a runtime for such a platform need only support the subset in the corresponding core assembly. The list of co-located types is exactly the following: - `System.Object` - `System.Enum` - `System.MulticastDelegate` - `System.Delegate` - `System.ValueType` - `System.Void` - `System.Boolean` - `System.Char` - `System.SByte` - `System.Byte` - `System.Int16` - `System.UInt16` - `System.Int32` - `System.UInt32` - `System.Int64` - `System.UInt64` - `System.Decimal` - `System.Single` - `System.Double` - `System.String` - `System.IntPtr` - `System.UIntPtr` - `System.Array` - `System.DateTime` - `System.Collections.IEnumerable` - `System.Collections.Generic.IEnumerable<>` - `System.Collections.Generic.IList<>` - `System.Collections.Generic.ICollection<>` - `System.Collections.Generic.IEnumerator<>` - `System.Collections.IEnumerator` - `System.Nullable<>` - `System.Runtime.CompilerServices.IsVolatile` - `System.IDisposable` - `System.TypedReference` - `System.IAsyncResult` - `System.AsyncCallback` - `System.Collections.Generic.IReadOnlyList<>` - `System.Collections.Generic.IReadOnlyCollection<>` We continue to assume that references to `System.Type` are made by-enumeration value when the reference is made in a custom attribute, and as a regular reference when not. Note the departure from the ECMA spec which included `System.Type`. `System.Type` and other reflection types are intentionally not in the core library. Since `System.Type` may be referenced as a built-in type, the compiler may assume that there is at most one definition for this type available in the compilation context, and treat as an error the case where there is more than one. No definition should be accepted as long as there is no reference. Language features relying on APIs not in the list should not assume any particular assembly location for the definition of such APIs. Compilers should either use heuristics, or require explicit specification of the location of such APIs. --- ### Compilers Compiler Server (`docs/compilers/Compiler Server.md`) Compiler Server =============== As a performance optimization, the Roslyn C# and VB compilers support using a server process to execute compilations instead of executing the compilation in-process. Engineering has focused on three performance benefits from the server: 1. Re-using a single process to do compilation amortizes the cost of JITing the compiler code. This is usually the single largest performance benefit. 1. Roslyn requires assembly references to be hydrated into a rich object model to perform compilation. The compiler server holds a fixed-size cache of metadata symbols, which can help compilation times if most compilations share a sizable portion of metadata references (e.g., mscorlib). 1. On Windows process creation itself is expensive, so dispatching compilations directly from MSBuild saves creations of csc.exe. ## Usage The primary way to use the compiler server is via the `Microsoft.Build.Tasks.CodeAnalysis` MSBuild task, which ships in both the `dotnet` SDK and Visual Studio. The build task contains a build client which, rather than executing the `csc.exe` or `vbc.exe`, creates the `VBCSCompiler.exe` process and dispatches the command line arguments to the server directly. If you're not using MSBuild, there's also a flag, `/shared` which can be passed directly to `csc.exe`, which causes `csc.exe` to behave as a client and dispatches the actual compilation to the server process. In this mode `csc.exe` doesn't perform any computation itself, it just dispatches the command line to the server process (starting one if one does not already exist), and then reports the results of the compilation (including any warnings/errors produced). Importantly, the server is a performance optimization. If server compilation fails for any reason, the compiler will fall back to stand-alone compilation. For MSBuild, this means creating a standard `csc.exe` process. For `/shared` this means continuing the compilation in-process, as though `/shared` were not passed. No diagnostics are produced in this case, as there are very common environmental reasons for any given server compilation to fail. This also implies that it is always safe to kill the compiler server process, as the client will always recover gracefully. ## Architecture Roslyn supports a client-server protocol where multiple clients can dispatch to multiple servers and multiple servers can run concurrently on the machine. The client and server must run on the same machine, as the client and server only exchange a command line, not real assets. ## Troubleshooting To diagnose problems with the server compilation, you can enable an environment variable that writes a diagnostic log file: ``` (UNIX): export RoslynCommandLineLogFile=/path/to/log/file (Windows CMD, using directory path): setx RoslynCommandLineLogFile="C:\path\to\log" (Windows CMD, using file path): setx RoslynCommandLineLogFile="C:\path\to\log\file.log" ``` This file contains diagnostic logging for both the client and server, which you can distinguish using the `PID` tag for each diagnostic line. Sample output should look something like this: ``` --- PID=90433 TID=16 Ticks=1048119217: Attempt to open named pipe 'angocke.F.MW9E3dnrZddlsgI8MNFfpJyup' --- PID=90433 TID=16 Ticks=1048119217: Attempt to connect named pipe 'angocke.F.MW9E3dnrZddlsgI8MNFfpJyup' --- PID=90436 TID=1 Ticks=244465170: Keep alive timeout is: 600000 milliseconds. --- PID=90436 TID=1 Ticks=244465182: Constructing pipe 'angocke.F.MW9E3dnrZddlsgI8MNFfpJyup'. --- PID=90436 TID=1 Ticks=244465182: Successfully constructed pipe 'angocke.F.MW9E3dnrZddlsgI8MNFfpJyup'. ``` Here PID `90433` is the client, as it's attempting to open a connection (named pipe), while PID `90436` is the server, as it is constructing the named client. If it succeeds, the result looks something like: ``` --- PID=90436 TID=8 Ticks=244476082: ****C# Compilation complete. ****Return code: 0 ****Output: Microsoft (R) Visual C# Compiler version 2.9.0.63018 (e3fa7d6a) Copyright (C) Microsoft Corporation. All rights reserved. ``` Otherwise, you may be able to discover what's going wrong from other output, like exception messages. --- ### Compilers Compiler Toolset NuPkgs (`docs/compilers/Compiler Toolset NuPkgs.md`) Compiler Toolset NuPkgs === ## Summary The compiler produces the [Microsoft.Net.Compilers.Toolset NuPkg](https://www.nuget.org/packages/Microsoft.Net.Compilers.Toolset) from all of Roslyn's main branches. When this NuPkg is installed it will override the compiler that comes with MSBuild with the version from the branch it was built in. This package is meant to support the following scenarios: 1. Allows compiler team to provide rapid hot fixes to customers who hit a blocking issue. This package can be installed until the fix is available in .NET SDK or Visual Studio servicing. 1. Serves as a transport mechanism for the Roslyn binaries in the greater .NET SDK build process. 1. Allows customers to conduct experiments on various Roslyn builds, many of which aren't in an official shipping product yet. This package is **not** meant to support using newer compiler versions in an older version of MSBuild. For example using Microsoft.Net.Compilers.Toolset 3.5 (C# 8) inside MSBuild 15 is explicitly not a supported scenario. Customers who want to use the compiler as a part of their supported build infrastructure should use the [Visual Studio Build Tools SKU](https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2022]) ) or [.NET SDK](https://dotnet.microsoft.com/download/visual-studio-sdks) ## NuPkg Installation To install the NuPgk run the following: ```cmd > nuget install Microsoft.Net.Compilers.Toolset # Install C# and VB compilers ``` Daily NuGet builds of the project are also available in our [Azure DevOps feed](https://dev.azure.com/dnceng/public/_packaging?_a=feed&feed=dotnet-tools): > https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json ## Microsoft.Net.Compilers The [Microsoft.Net.Compilers](https://www.nuget.org/packages/Microsoft.Net.Compilers) NuPkg is deprecated. It is a .NET Desktop specific version of Microsoft.Net.Compilers.Toolset and will not be produced anymore after the 3.6.0 release. The Microsoft.Net.Compilers.Toolset package is a drop in replacement for it for all supported scenarios. --- ### Compilers Deterministic Inputs (`docs/compilers/Deterministic Inputs.md`) Deterministic Inputs ==================== The C# and VB compilers are fully deterministic when the `/deterministic` option is specified (this is the default in the .NET SDK). This means that the "same inputs" will cause the compilers to produce the "same outputs" byte for byte. The following are considered inputs to the compiler for the purpose of determinism: - The sequence of command-line parameters (order is important) - The precise version of the compiler used and the files included in its deployment: reference assemblies, rsp, etc ... - Current full directory path (you can reduce this to a relative path; see https://github.com/dotnet/roslyn/issues/949) - (Binary) contents of all files explicitly passed to the compiler, directly or indirectly, including - source files - referenced assemblies - referenced modules - resources - the strong name key file - `@` response files - Analyzers - Generators - Rulesets - "additional files" that may be used by analyzers and generators - The current culture if `/preferreduilang` is not specified (for the language in which diagnostics and exception messages are produced). - The current OS code page if `/codepage` is not specified and any of the input source files do not have BOM and are not UTF-8 encoded. - The existence, non-existence, and contents of files on the compiler's search paths (specified, e.g. by `/lib` or `/recurse`) - The CLR platform on which the compiler is run: - The result of `double` arithmetic performed for constant-folding may use excess precision on some platforms. - The compiler uses Unicode tables provided by the platform. - The version of the zlib library that the CLR uses to implement compression (when `/embed` or `/debug:embedded` is used). - The value of `%LIBPATH%`, as it can affect reference discovery if not fully qualified and how the runtime handles analyzer / generator dependency loading. - The full path of source files although `/pathmap` can be used to normalize this between compiles of the same code in different root directories. At the moment the compiler also depends on the time of day and random numbers for GUIDs, so it is not deterministic unless you specify `/deterministic`. ## Debugging Determinism Failures When investigating determinism issues where the same code produces different outputs, you can use several techniques to identify the cause: ### 1. Generate a Deterministic Key File Build with `/p:Features="debug-determinism"` to generate a `.key` file that documents all inputs to the compiler: ```bash dotnet build /p:Features="debug-determinism" ``` This creates an additional output file (e.g., `MyAssembly.dll.key`) in the output directory (typically in the `obj` directory for intermediate builds). The key file is a JSON document containing: - Compiler version and runtime information - All source file paths and content checksums - Referenced assemblies with their MVIDs (Module Version IDs) - Compilation options - Parse options - Emit options - Analyzer and generator information Compare the `.key` files from two supposedly identical builds to identify which inputs differ: ```bash # Unix/Linux/Mac diff build1/MyAssembly.dll.key build2/MyAssembly.dll.key # Windows (using fc) fc build1\MyAssembly.dll.key build2\MyAssembly.dll.key ``` **Note**: If the `.key` files are identical but the output assemblies differ, this indicates a compiler determinism bug. Please report such cases to the Roslyn team with a reproduction so the issue can be investigated. ### 2. Compare Metadata Using metadata-tools Install the `mdv` (MetaData Viewer) tool from the [dotnet/metadata-tools](https://github.com/dotnet/metadata-tools) repository: ```bash dotnet tool install mdv -g --prerelease --add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json ``` Generate metadata dumps for your assemblies and compare them: ```bash # Generate metadata output for each DLL mdv MyAssembly1.dll > assembly1.txt mdv MyAssembly2.dll > assembly2.txt # Compare the outputs diff assembly1.txt assembly2.txt ``` The metadata dump shows detailed information about types, methods, fields, attributes, and other metadata in the assembly. Differences in the metadata can help identify what changed between builds. ### 3. Compare Embedded Resources If your assembly contains embedded resources, verify they are identical. The recommended approach is to use ILSpy, which has an easy "Save" button to export resources from the DLL for comparison. Other tools like dnSpy or `ildasm` can also be used to extract and inspect embedded resources. Example using `ildasm` to disassemble and extract embedded `.res` files: ```bash # Disassemble and generate .res files ildasm /out=assembly1.il MyAssembly1.dll ildasm /out=assembly2.il MyAssembly2.dll # Compare the generated .res files diff assembly1.res assembly2.res ``` ### 4. Binary Diff of the DLL As a last resort, perform a hex dump comparison of the actual DLL files: ```bash # Unix/Linux/Mac - using xxd or hexdump xxd MyAssembly1.dll > assembly1.hex xxd MyAssembly2.dll > assembly2.hex diff assembly1.hex assembly2.hex # Windows - using fc with /b flag for binary comparison fc /b MyAssembly1.dll MyAssembly2.dll # Or using Format-Hex in PowerShell Format-Hex MyAssembly1.dll | Out-File assembly1.hex Format-Hex MyAssembly2.dll | Out-File assembly2.hex Compare-Object (Get-Content assembly1.hex) (Get-Content assembly2.hex) ``` A hex diff shows the exact bytes that differ, which can help identify non-deterministic data like timestamps, GUIDs, or other embedded values. ### Common Causes of Non-Determinism When debugging, look for these common issues: - **Missing `/deterministic` flag**: Ensure `/deterministic` is enabled - **Absolute paths**: Use `/pathmap` to normalize file paths - **Timestamps**: Check if PDBs or other files embed build times - **Different compiler versions**: Verify same compiler version is used - **Different reference assembly versions**: Check MVIDs (Module Version IDs) in `.key` files - **Environment variables**: Variables like `%LIBPATH%` can affect output - **Source file encoding**: Ensure consistent encoding (UTF-8 with BOM recommended) - **Generator/analyzer differences**: Verify same versions are loaded --- ### Compilers Error Log Format (`docs/compilers/Error Log Format.md`) The C# and Visual Basic compilers support a /errorlog: switch on the command line to log all diagnostics in a structured, JSON format. The log format is SARIF (Static Analysis Results Interchange Format): See https://sarifweb.azurewebsites.net/ for the format specification, JSON schema, and other related resources. ## SARIF v2 format This section provides a high level overview of the contents of the SARIF v2 error log file generated by the C# and Visual Basic compilers. A more formal and detailed schema specification for the SARIF v2 format is specified at http://json.schemastore.org/sarif-2.1.0. 1. `schema` and `version` information: The first two lines of the error log file specifies the SARIF schema and version information. For example: ```json "$schema": "http://json.schemastore.org/sarif-2.1.0", "version": "2.1.0", ``` 2. `runs` information: The core entry in the error log file is the `runs` section with a single run entry within it for the build. The run entry has the following main parts: 1. `results` section: This section contains an array of result entries, where each result corresponds to information about a reported compiler or analyzer `Diagnostic`. More details in [`Result` format for each compiler or analyzer `Diagnostic` instance](#result-format-for-each-compiler-or-analyzer-diagnostic-instance). 2. `properties` section: This section contains custom key-value properties associated with the run. Following properties are currently emitted: 1. `analyzerExecutionTime`: Execution time in seconds for execution of all analyzers, as reported with the `/reportAnalyzer` compiler switch. 3. `tools` section: This section contains information about the compiler build and version. Additionally, it contains a `rules` array, where each rule entry corresponds to metadata or `DiagnosticDescriptor` information about each reported analyzer diagnostic. More details in [`Rule` format for each analyzer supported `DiagnosticDescriptor` instance](#rule-format-for-each-analyzer-supported-diagnosticdescriptor-instance) 4. `invocations` section: This section contains information about the end user specified configurations for the rules for compiler invocation/run. We emit this section only if at least one rule had one or more severity configuration overrides for a part or whole of the compilation via options. More details in [`Rule Configuration Override` format for each rule severity override](#rule-configuration-override-format-for-each-rule-severity-override). 5. `columnKind` section: This section contains information about the unit in which the tool measures columns. C# and Visual Basic compilers uses utf16 code units. Example `runs` section, with stripped off `results`, `rules` and `ruleConfigurationOverrides` sections: ```json "runs": [ { "results": [ ], "properties": { "analyzerExecutionTime": "x.xxx" }, "tool": { "driver": { "name": "Microsoft (R) Visual C# Compiler", "version": "4.4.0-dev ()", "dottedQuadFileVersion": "42.42.42.42", "semanticVersion": "42.42.42", "language": "en-US", "rules": [ ] } }, "invocations": [ { "executionSuccessful": true, "ruleConfigurationOverrides": [ ] } ], "columnKind": "utf16CodeUnits" } ] ``` ### `Result` format for each compiler or analyzer `Diagnostic` instance The results section contains an array of result entries, where each result corresponds to information about a reported compiler or analyzer `Diagnostic`. It contains the below data: 1. `ruleId`: Rule ID or diagnostic ID for the diagnostic. 2. `ruleIndex`: Index into the `rules` section for the underlying `DiagnosticDescriptor` for the diagnostic. See [`Rule` format for each analyzer supported `DiagnosticDescriptor` instance](#rule-format-for-each-analyzer-supported-diagnosticdescriptor-instance) for more details. 3. `level`: Severity level for the diagnostic, such as `error`, `warning`, `note`, etc. 4. `message`: User facing message for the diagnostic. 5. `suppressions`: For each diagnostic instance that is suppressed with a pragma directive, SuppressMessageAttribute or via a DiagnosticSuppressor, the result entry contains `suppressions` section with the following data: 1. `kind` information: C# and Visual Basic compilers support a single `inSource` suppression kind. 2. `justification` information: Justification pertaining to the suppression. Currently, this field is populated only for `SuppressMessageAttribute` based suppression, with a non-null justification argument. 3. `suppressionType` property with one of these three values: `Pramga Directive`, `SuppressMessageAttribute`, or `DiagnosticSuppressor`. This data helps analyze the preferred in-source suppression mechanisms in a code base. 6. `locations`: One or more locations associated with the diagnostic. 7. `properties`: One or more custom key-value string pairs associated with the diagnostic. Example `result` entry: ```json { "ruleId": "CA1822", "ruleIndex": 97, "level": "warning", "message": { "text": "Member 'M2' does not access instance data and can be marked as static" }, "suppressions": [ { "kind": "inSource", "justification": "", "properties": { "suppressionType": "SuppressMessageAttribute" } } ], "locations": [ { "physicalLocation": { "artifactLocation": { "uri": "file:///C:/source/repos/ClassLibrary1/Class1.cs" }, "region": { "startLine": 13, "startColumn": 10, "endLine": 13, "endColumn": 12 } } } ], "properties": { "warningLevel": 1 } } ``` ### `Rule` format for each analyzer supported `DiagnosticDescriptor` instance The `rules` section contains a rule entry that corresponds to metadata or `DiagnosticDescriptor` information about supported descriptors for each analyzer. We report the `DiagnosticDescriptor` info for all analyzers that were provided to the compilation, regardless of whether or not they executed on the entire compilation, part of the compilation or were disabled for the entire compilation. A `rule` entry contains the below data: 1. `id`: Rule ID or Diagnostic ID associated with the descriptor. 2. `shortDescription`: User facing short description or `Title` associated with the descriptor. 3. `fullDescription`: User facing full description or `Description` associated with the descriptor. 4. `defaultConfiguration`: Default severity level for the diagnostics reported for the descriptor, such as `error`, `warning`, `note`, etc. 5. `helpUri`: Help uri for help information associated with the descriptor. 6. `properties`: One or more custom properties associated with the descriptor. It includes the following: 1. `category`: `Category` associated with the descriptor, such as, `Design`, `Performance`, `Security`, etc. 2. `isEverSuppressed` and `suppressionKinds`: If a rule had either a source suppression or was disabled for part or whole of the compilation via options, the rule metadata contains a special flag `isEverSuppressed = true` and an array `suppressionKinds` with either or both of the below suppression kinds: 1. `inSource` suppression kind for one or more reported diagnostic(s) that were suppressed through pragma directive, SuppressMessageAttribute or a DiagnosticSuppressor. 2. `external` suppression kind for diagnostic ID that is disabled either for the entire compilation (via global options such as /nowarn, ruleset, globalconfig, etc.) or for certain files or folders in the compilation (via editorconfig options). 3. `executionTimeInSeconds`: Execution time in seconds associated with the analyzer, as reported with the `/reportAnalyzer` compiler switch. Values less than `0.001` seconds are reported as `<0.001`. 4. `executionTimeInPercentage`: Execution time in percentage associated with the analyzer, as reported with the `/reportAnalyzer` compiler switch. Values less than `1` are reported as `<1`. 5. `tags`: An array of one of more `CustomTags` associated with the descriptor. Example `rule` entry: ```json { "id": "CA1001", "shortDescription": { "text": "Types that own disposable fields should be disposable" }, "fullDescription": { "text": "A class declares and implements an instance field that is a System.IDisposable type, and the class does not implement IDisposable. A class that declares an IDisposable field indirectly owns an unmanaged resource and should implement the IDisposable interface." }, "defaultConfiguration": { "level": "note" }, "helpUri": "https://docs.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1001", "properties": { "category": "Design", "isEverSuppressed": "true", "suppressionKinds": [ "external" ], "executionTimeInSeconds": "x.xxx", "executionTimeInPercentage": "xx", "tags": [ "PortedFromFxCop", "Telemetry", "EnabledRuleInAggressiveMode" ] } } ``` ### `Rule Configuration Override` format for each rule severity override The `ruleConfigurationOverrides` section contains an array of entries, with an entry for every rule severity configuration override for the whole or part of the compilation via options, i.e. editorconfig, globalconfig, command line options, ruleset, etc. Each rule configuration override entry contains the following information: 1. `descriptor`: Property containing information to map back to the relevant `descriptor` or the `rule` in the `rules` section. Currently, it contains the below child properties: 1. `id`: Rule id string for the rule or diagnostic. 2. `index`: A zero-based integral index value into the `rules` array for the rule mapping. 2. `configuration`: Configuration property for the effective overridden severity for the rule. Currently, it may include one of the below child properties: 1. `enabled`: A boolean property indicating if the rule has been explicitly disabled or enabled with the configuration override entry. 1. `level`: Property for the effective severity of the rule, such as `error`, `warning` or `note`. Note that this property is not emitted for rules disabled with configuration override, we instead emit `enabled: false` for them. Note that we can have multiple `ruleConfigurationOverride` entries for the same descriptor if the rule is configured to fire at different severities across different parts of the compilation. For example, editorconfig based severity configurations can specify different effective severity for the same rule ID for different files or folders in a project. Example `ruleConfigurationOverride` entry: ```json { "descriptor": { "id": "CA1000", "index": 0 }, "configuration": { "level": "error" } }, { "descriptor": { "id": "CA1000", "index": 0 }, "configuration": { "enabled": false } }, { "descriptor": { "id": "CA1001", "index": 1 }, "configuration": { "level": "warning" } } ``` --- ### Compilers Extension Methods In Aliased Assemblies (`docs/compilers/Extension Methods in Aliased Assemblies.md`) Extension Methods in the Global Namespace ========================================= If you create a compilation in which an imported library is assigned an alias (instead of the default "global" alias), the Roslyn compilers do not automatically see extension methods in the (unnamed top-level) global namespace in that library. You would have to import the library in your code using the alias for these methods to be treated as extension methods. Previous versions of the compiler would treat such methods as extension methods despite not being imported. See also https://github.com/dotnet/roslyn/issues/1079. --- ### Compilers IOperation Test Hook (`docs/compilers/IOperation Test Hook.md`) # IOperation Test Hook The compiler has a test hook designed to ensure that IOperation nodes and scenarios match up with information that can be retrieved from the semantic model, and to help ensure that we have code coverage that ensures IOperation will not fail or violate invariants for scenarios we don't have direct testing on. This works by running a hook whenever the test framework creates a `Compilation` object: for every syntax tree in the compilation, we enumerate every syntax node and verify that GetOperation does not crash, and returns information that matches up with the `SemanticModel` (ie, `GetTypeInfo` matches the `IOperation` type, for example). We also fetch and verify the control flow graph for every member body in the compilation and run the CFG verifier to ensure that all invariants are presevered. ## Feature Development For feature development, this means that as we shift around BoundNodes, introduce new nodes, or enable code that didn't exist before, the hook may hit failures because the IOperation machinery doesn't handle that particular node, or had logic that now triggers a `Debug.Assert` because the new code violates some pre-existing assumption of the factories. For fixing these issues, there are a couple of options: 1. Implement the IOperation changes in the same PR. This is the only acceptable way to do this in a production branch. 2. Add the new BoundKind to the catch-all switch case at the bottom of `CSharpOperationFactory/VisualBasicOperationFactory.Create`, with a prototype comment to ensure that support is implemented before the feature branch is merged back to a production branch. Sometimes, the test hits other failure spots, such as `SemanticModel` failures or mismatches. For these, the test can be marked with `ConditionalFact(typeof(NoIOperationValidation))` to skip the test in the test hook. When doing so, please mark with a prototype comment (either as a comment above or as the reason string for ConditionalFactAttribute) so that it is fixed before merging to production. ## Replicating Failures In order to replicate test failures, there are 2 options: 1. Uncomment `src/Compilers/Test/Core/Compilation/CompilationExtensions.cs:7`, which defines `ROSLYN_TEST_IOPERATION`, and run your tests. Do _not_ check this in, as it will enable the test hook for every test in every project and significantly slow down regular test runs. 2. Set the `ROSLYN_TEST_IOPERATION` environment variable and restart VS with it set. 3. Set a breakpoint at the start of `ValidateIOperations`, in `src/Compilers/Test/Core/Compilation/CompilationExtensions.cs`. When it breaks, use VS's jump to location or drag the instruction pointer past the early check and return on `EnableVerifyIOperation`, which will run the code for the current test run. In either case, after this has been done all tests will run the hook as part of running the test, and you can set breakpoints as normal. It is often helpful to run until an exception has been hit and then see what nodes were being processed, as most tests have multiple methods and understanding the specific context is useful for figuring out the earlier steps that may have gone wrong. When a test failure is isolated, please add a _dedicated_ IOperation test for this to make it easier to avoid future regressions. Preferrably, don't replicate the entire original test, just enough to hit the bug to ensure that it's protected against regressions. --- ### Compilers Interactive Imports (`docs/compilers/Interactive Imports.md`) # Imports in Scripts/Interactive ## Guiding principles Files includes by ```#load``` are each in a separate lexical scope. Declarations (including aliases) take precedence over usings (including using static) because they express intent. ## Process Steps (top to bottom) | Binding Using | Binding Code | #load ----------------------|---------------|--------------|------ Look for declarations in the current submission/file (including aliases) | Yes, excluding aliases | Yes, in submission | Yes, in file Look for declarations in preceding submissions (including aliases), in order | Yes | Yes | No Look for members in the host object (if any) | Yes | Yes | Yes Look for declarations in the global namespace | Yes | Yes | Yes Look in usings (except aliases) in the current submission/file and all preceding submissions (all at once, not looping) | No | Yes, in all submissions | Yes, in file only Look in the global imports of the current submission and all preceding submissions (all at once, not looping) | No | Yes | Yes --- ### Compilers Rule Set Format (`docs/compilers/Rule Set Format.md`) Introduction ============ This document, in conjunction with the [Rule Set Schema](..//..//src//Compilers//Core//Portable//RuleSet//RuleSetSchema.xsd), describes the structure of .ruleset files used by the C# and Visual Basic compilers to turn diagnostic analyzers on and off and to control their severity. This document only discusses the required and common parts of .ruleset files; for a full set, please see the schema file. Sample ===== The following demonstrates a small but complete example of a .ruleset file. ``` XML ``` Passing Rule Sets to the Compiler ================================= Command Line ------------ Rule set files can be passed to csc.exe and vbc.exe using the `/ruleset` switch, which takes an absolute or relative file path. For example: ``` vbc.exe /ruleset:ProjectRules.ruleset ... vbc.exe /ruleset:..\..\..\SolutionRules.ruleset ... vbc.exe /ruleset:D:\WizbangCorp\Development\CompanyRules.ruleset ... ``` MSBuild Projects ---------------- Within MSBuild project files the rule set can be specified via the `CodeAnalysisRuleSet` property. For example: ``` XML ProjectRules.ruleset ``` Note that because the rule set is specified via a *property* rather than an *item*, IDEs like Visual Studio will not show the rule set as a file in your project by default. For this reason it is common to explicitly include the file as an item as well: ``` XML ``` Elements ======== `RuleSet` --------- The `RuleSet` is the root element of the file. Attributes: * Name (**required**): A short, descriptive name for the file. * Description (**optional**): A longer description of the purpose of the rule set file. * ToolsVersion (**required**): This attribute is required for backwards compatibility with other tools that use .ruleset files. In practice it is the version of Visual Studio that produced this .ruleset file. When in doubt, simply use "12.0". Children: `Include`, `Rules` `Include` --------- Pulls in the settings from the specified rule set file. Settings in the current file override settings in the included file. Parent: `RuleSet` Attributes: * Path (**required**): An absolute or relative path to another .ruleset file. * Action (**required**): Specifies the effective action of the included rules. Must be one of the following values: * Default - The rules use the actions specified in the included file. * Error - The included rules are treated as though their action values were all "Error". * Warning - The included rules are treated as though their actions values were all "Warning". * Info - The included rules are treated as though their actions values were all "Info". * Hidden - The included rules are treated as though their actions values were all "Hidden". * None - The included rules are treated as though their actions values were all "None". Children: None. `Rules` ------- Hold settings for rules from a single diagnostic analyzer. Parent: `RuleSet` Attributes: * AnalyzerId (**required**): The name of the analyzer. In practice this is the simple name of the assembly containing the diagnostics. * RuleNamespace (**required**): This attribute is required for backwards compatibility with other tools that use .ruleset files. In practice it is generally the same as the AnalyzerId. Children: `Rule` `Rule` ------ Specifies what action to take for a given diagnostic rule. Parent: `Rules` Attributes: * Id (**required**): The ID of the diagnostic rule. * Action (**required**): One of the following values: * Error - Instances of this diagnostic are treated as compiler errors. * Warning - Instances of this diagnostic are treated as compiler warnings. * Info - Instances of this diagnostic are treated as compiler messages. * Hidden - Instances of this diagnostic are hidden from the user. * None - Turns off the diagnostic rule. Children: None. Notes: Within a `Rules` element, a `Rule` with a given `Id` may only appear once. The C# and Visual Basic compilers impose a further constraint: if multiple `Rules` elements contain a `Rule` with the same `Id` they must all specify the same `Action` value. The difference between the Hidden and None actions is subtle but important. When a diagnostic rule is set to Hidden instances of that rule are still created, but aren't displayed to the user by default. However, the host process can access these diagnostics and take some host-specific action. For example, Visual Studio uses a combination of Hidden diagnostics and a custom editor extension to "grey out" dead or unnecessary code in the editor. A diagnostic rule set to None, however, is effectively turned off. Instances of this diagnostic may not be produced at all; even if they are suppressed rather than being made available to the host. --- ### Compilers Analyzer Config (`docs/compilers/analyzer-config.md`) Compiler Support for AnalyzerConfig =================================== AnalyzerConfig is an EditorConfig-superset (https://editorconfig.org/) file format recognized by the Roslyn command line compiler. The options specified in analyzer config files are recognized by the compiler in two ways: option keys following the pattern `dotnet_diagnostic..severity = ` are parsed and interpreted by the compiler to configure the severity of compiler diagnostics. `` represents the diagnostic ID matched by the compiler, case-insensitively, to be configured. `` must be the name of a member of the [ReportDiagnostic](../../src/Compilers/Core/Portable/Diagnostic/ReportDiagnostic.cs) enum, also case-insensitive. These settings are then applied on a per-SyntaxTree to each of the files whose path matched the AnalyzerConfig name specification in the compilation. Any properties which do not have the aforementioned pattern are considered analyzer options and are placed in a PerTreeOptionsProvider on the [AnalyzerOptions type](../../src/Compilers/Core/Portable/DiagnosticAnalyzer/AnalyzerOptions.cs) for use by analyzers. AnalyzerConfig files can be passed to the command-line compiler through the `/analyzerconfig:` parameter. --- ### Compilers Terrapin (`docs/compilers/terrapin.md`) .NET and Terrapin Process === The Terrapin system is concerned with validating the provenance of NuPkgs in the .NET ecosystem. Specifically it needs to verify that DLLs in a NuPkg can be rebuilt using publicly available artifacts. This document describes the workflow that will be used by Terrapin for validating the provenance of .NET binaries, responsibilities of each team, as well as the tools the .NET team will deliver to enable the service to be successful. ## .NET Tools The .NET Team will deliver a set of tools to the Terrapin team to provide the necessary validation. The idea in providing a series of tools here vs. a single all in one tool is to help better separate out the different responsibilities involved here and letting the appropriate team handle them: 1. Providing the list of artifacts necessary for validation to succeed (owned by .NET) 1. Acquiring the artifacts necessary for validation (owned by Terrapin) 1. Performing validation and making a determination of provenance (owned by .NET) It's possible in the future we will end up providing a "all in one" experience tool but for the short term we'll be focusing on keeping these responsibilities and tools separate. ### dotnet-build-manifest-generator This is global tool which takes in a DLL / PDB file combination and will generate an artifacts manifest file which lists the set of artifacts necessary to perform a validation. This includes the version of `dotnet-build-validator` which needs to be used to perform the validation. This tool is meant to work with binaries produced by earlier versions of the .NET compiler. Every time the contents of the PDB provenance metadata change a new version of this tool will be produced that can account for the metadata. The expectation is that Terrapin will always have the latest version of this tool installed on their service. ### dotnet-build-validator A global tool which takes in the following arguments: the PE file to validate, its accompanying PDB and a directory containing the artifacts gathered from the artifacts manifest file. The tool will then return whether the provided PDB file can have it's provenance validated. The tool will have three output states: 1. Validation succeeded 1. Validation failed because the binaries do not match 1. Validation failed because a binary could not be produced This tool will be sim-shipped with the compiler: every time there is a new compiler, there will be a new version of this tool that uses those compiler binaries. This is important because the compiler does not guarantee deterministic output between versions of the compiler. Even minor version differences of the compiler can produce different IL for the same source code if the code intersects a bug fix or optimization. In order for the validator to function with high fidelity Terrapin must use the version that sim-shipped with the compiler. The expectation is that Terrapin will install this as a local tool for a given validation event. That will allow for every validation event to use a different version of `dotnet-build-validator` without having to worry about version conflicts as it would if it were installed as a global tool. ## The Terrapin Service Workflow The Terrapin service will validate a DLL by going through the following workflow for every DLL in a NuPkg file: 1. Execute `dotnet-build-manifest-generator` on the DLL + PDB combination. 1. Setup the environment for validation: 1. Transfer execution to the appropriate operating system listed in the manifest. All future actions are expected to run on this machine. 1. Install the correct .NET Runtime on the target machine (if necessary) 1. Download the artifacts listed in the artifact manifest file as well as the specified version of `dotnet-build-validator`. 1. Execute `dotnet-build-validator` providing the DLL, PDB and the directory where the artifacts were downloaded too. The Terrapin service will download the artifacts into a directory created by Terrapin. The directory will have two directories to store source files and references respectively: sources and references. The name of the artifacts in those directories will be specified in the manifest file. That will put the burden of avoiding collisions and dealing with file path fixup on the compiler team. ## Artifacts Manifest File The goal of `dotnet-build-manifest-generator` is to provide a manifest file to the Terrapin service which lists all of the artifacts necessary for a validation to be completed as well as the environment needed to run the validation. The set of artifacts that are necessary include: 1. The source files used in the compilation. 1. The reference DLLs used in the compilation. At this time the expectation is Terrapin will need to recreate much of the environment used to create the binaries in order to perform provenance validation. This includes having a matching .NET Runtime version and operating system. This is because OS dependencies, like zip compression version, can impact the deterministic output of the compiler. It's possible our experiments in Terrapin will cause us to re-evaluate how these play into determinism but for now the design is to build validation on top of our existing assumptions here. To facilitate this the artifact manifest file will include all of the necessary environment information. This will include the OS RID, the .NET Runtime version and whether a desktop or NET Core runtime was used to run the compiler. The artifacts manifest file will be a JSON file with the following format: ```json { "sources": [ { "uri": "https://github.com/dotnet/roslyn/blob/f5517311e37a3883e09dc6daa5069bbc4f44c882/src/Compilers/Shared/Csc.cs", "destFileName": "c6fea380dac7392b945fa6c63466746f830c5cc3" } ], "references": [ { "name": "System.Reflection.Metadata.dll", "mvid": "51ef3ce0-5a89-4325-b094-8a5970acc676", "destFileName": "51ef3ce0-5a89-4325-b094-8a5970acc676", } ], "environment": { "compiler-language": "csharp", "compiler-version": "3.8.0", "os": "windows", "os-version": "...", "dotnet-runtime": "net5.0.0" } } ``` ## Phases of validation There are still many open questions surrounding this effort like the impact of code pages, line endings as well as how strongly .NET runtime versions impact determinism of our outputs. The best way to answer these questions is to be pushing real customer code through the service to prove out the real impact of these problems. There are also just general process issues we need to work out between like cadence of delivery as well as just making sure all the pieces fit together. To help solve both of these we will be breaking this effort up into a series of milestones that increase in complexity. Each one moving us closer to our goals around provenance of all major NuPkgs in the ecosystem: The milestones are: - Roslyn can successfully validate dotnet/roslyn on a machine that just built dotnet/roslyn. This will give us confidence the underlying provenance checks are achievable. All of the artifacts necessary to perform validation will be located on the machine in well known locations hence that will allow us to separate out "acquisition of artifacts" - Roslyn will produce versions of dotnet-build-manifest-generator and dotnet-build-validator for Terrapin and they will validate that they can use them on projects they are hosting locally. - Rebuild of DLLs on Windows: the target of this milestone is NuPkgs produced on Windows using the .NET Framework compiler. The goal is for us to be able to rebuild the DLLs, whether validation succeeds or fails. This will help us ensure that we have the right process in place between Terrapin and .NET as well that the PDBs do truly contain enough information for us to recreate compilations. - Validation of DLLs on Windows: the target of this milestone is NuPkgs produced on Windows using the .NET Framework compiler. The goal is for us to validate the provenance of these binaries as well as find common reasons why provenance validation fails. This will let us begin the conversation with .NET customers on how to change their build to make it provenance friendly. - Validation of DLLs on Windows Part 2: the target of this milestone is NuPkgs produced on Windows using the .NET Core compiler. This will require Terrapin to take the extra environment setup step of installing the correct .NET Core runtime and using that to drive the installation process. - Validate binaries produced on Linux / Mac: the target of this milestone is NuPkgs produced on Linux / Mac. This will require Terrapin to involve multiple machines in the validation process. It will also help us understand the true impact of OS sub-versions in our ability to do provenance validation. ### FAQ ### What about binaries produced before VS 16.7 was released? The provenance validation process depends on extra metadata included in the PDB that the compiler uses to generate the artifacts manifest file as well as recreate the compilation. That extra information was first added into PDBs in the Visual Studio 16.7 release. That means any binaries produced before then will fail provenance validation. This is a known issue and the understanding is such older NuPkgs will not be supported for provenance validation. ### How do we handle source generators? The provenance validation will occur using the output of source generators rather than re-running source generators as part of the validation. This will simplify the process as well as remove the issue of determinism in source generators (customer defined tools that don't have our level of rigor) from the process. ## Open Issues ### Running the validator on .NET Framework The expectation is that for Windows we should be able to use .NET Core to run the validator for binaries that were produced using the .NET Framework compiler. The runtime differences we are worried about for Unix operating systems don't have the same problems on Windows. Hence any .NET Runtime which can execute the compiler should be able to do provenance validation. It's possible we will find differences here though in the real world. In which case we will need to have a mode for `dotnet-build-validator` where by it will shell out to a .NET Framework process to do the validation. ### Line ending issues In order for validation to succeed it is important for `dotnet-build-validator` to receive source files that have the same line endings as the one's provided to the original compilation. These factor into items like the checksum of source files stored in PDBs, the content of string literals, etc ... It's possible that the information provided by source link is not sufficient to properly recreate files with correct line endings. Consider that a `.gitattributes` file could change how line endings are used on different operating systems. This is a problem the debugger team hits today and has solved with a [simple algorithm](https://github.com/dotnet/sourcelink/pull/678/files#diff-3339ce7022ba5de3029d3f15bdf8ca585905bad4f25077ca1b1616c31d841c62R258-R290) we can replicate here. Essentially if the file has consistent LF or CRLF line endings and the SHA don't match, flip the line endings. If that matches then use that version. ### Future expansions This document is written to describe how validation will work for the C# and VB compilers but those are not the only tools in the .NET SDK which create or modify DLLs. Other tools include, but is not limited to, the F# compiler and IL trimmer. The expectation is these tools can be on boarded with minimal changes to the process outlined here. For example the artifact manifest file can be expanded to include new artifact types, new tools can be consumed by Terrapin to do the validation of various phases of the build. ### What names should we call the tools As a rule, @jaredpar is terrible at picking names. Should we pick better names for the tools? --- ### Contributing API Review Process (`docs/contributing/API Review Process.md`) # API Review Process .NET has a long-standing history of taking API usability extremely seriously. Thus, we generally review every single API that is added to the product. This page discusses how we conduct design reviews for the Roslyn components. ## Which APIs should be reviewed? The rule of thumb is that we (**dotnet/roslyn**) review every public API that is being added to this repository in the CodeAnalysis (compiler), Workspaces, Features, and EditorFeatures libraries. APIs in the LanguageServices dlls are not usually reviewed as they are not intended for general consumption; they are for use in Visual Studio, and we don't have a compat bar for these APIs. ## Steps 1. **Requester files an issue**. The issue description should contain a speclet that represents a sketch of the new APIs, including samples on how the APIs are being used. The goal isn't to get a complete API list, but a good handle on how the new APIs would roughly look like and in what scenarios they are being used. Please use [this template](https://github.com/dotnet/roslyn/issues/new?template=api-suggestion.md). The issue should have the labels `Feature Request` and `Concept-API`. [Here](https://github.com/dotnet/roslyn/issues/53410) is a good example of a strong API proposal. 2. **We assign an owner**. We'll assign a dedicated owner from our side that sponsors the issue. This is usually [the area owner](../area-owners.md#areas) for which the API proposal or design change request was filed. 3. **Discussion**. The goal of the discussion is to help the assignee to decide whether we want to pursue the proposal or not. In this phase, the goal isn't necessarily to perform an in-depth review; rather, we want to make sure that the proposal is actionable, i.e. has a concrete design, a sketch of the APIs and some code samples that show how it should be used. If changes are necessary, the owner will set the label `api-needs-work`. To make the changes, the requester should edit the top-most issue description. This allows folks joining later to understand the most recent proposal. To avoid confusion, the requester can maintain a tiny change log, like a bolded "Updates:" followed by a bullet point list of the updates that were being made. When you the feedback is addressed, the requester should notify the owner to re-review the changes. 4. **Owner makes decision**. When the owner believes enough information is available to make a decision, they will update the issue accordingly: * **Mark for review**. If the owner believes the proposal is actionable, they will label the issue with `api-ready-for-review`. * **Close as not actionable**. In case the issue didn't get enough traction to be distilled into a concrete proposal, the owner will close the issue. * **Close as won't fix as proposed**. Sometimes, the issue that is raised is a good one but the owner thinks the concrete proposal is not the right way to tackle the problem. In most cases, the owner will try to steer the discussion in a direction that results in a design that we believe is appropriate. However, for some proposals the problem is at the heart of the design which can't easily be changed without starting a new proposal. In those cases, the owner will close the issue and explain the issue the design has. * **Close as won't fix**. Similarly, if proposal is taking the product in a direction we simply don't want to go, the issue might also get closed. In that case, the problem isn't the proposed design but in the issue itself. 5. **API gets reviewed**. The group conducting the review is called *RAR*, which stands for *Roslyn API Reviewers*. In the review, we'll take notes and provide feedback. After the review, we'll publish the notes in the [API Review repository](https://github.com/dotnet/apireviews) and at the end of the relevant issue. Multiple outcomes are possible: * **Approved**. In this case the label `api-ready-for-review` is replaced with `api-approved`. * **Needs work**. If we believe the proposal isn't ready yet, we'll replace the label `api-ready-for-review` with `api-needs-work`. * **Rejected**. If we believe the proposal isn't a direction we want to pursue, we'll simply write a comment and close the issue. ## Review schedule There are three methods to get an API review (this section applies to API area owners, but is included for transparency into the process): * **Get into the backlog**. Generally speaking, filing an issue in `dotnet/roslyn` and applying the label `api-ready-for-review` on it will make your issue show up during API reviews. The downside is that we generally walk the backlog oldest-newest, so your issue might not be looked at for a while. Progress of issues can be tracked via . * **Fast track**. If you need to bypass the backlog apply both `api-ready-for-review` and `blocking`. All blocking issues are looked at before we walk the backlog. * **Dedicated review**. If an issue you are the area owner for needs an hour or longer, send an email to roslynapiowners and we book dedicated time. We also book dedicated time for language feature APIs: any APIs added as part of a new language feature need to have a dedicated review session. Rule of thumb: if the API proposal has more than a dozen APIs, the APIs have complex policy, or the APIs are part of a feature branch, then you need 60 min or more. When in doubt, send mail to roslynapiowners. The API review board currently meets every 2 weeks. ## Pull requests Pull requests against **dotnet/roslyn** that add new public API shouldn't be submitted before getting approval. Also, we don't want to get work in progress (WIP) PR's. The reason being that we want to reduce the number pending PRs so that we can focus on the work the community expects we take action on. If you want to collaborate with other people on the design, feel free to perform the work in a branch in your own fork. If you want to track your TODOs in the description of a PR, you can always submit a PR against your own fork. Also, feel free to advertise your PR by linking it from the issue you filed against **dotnet/roslyn** in the first step above. For **dotnet/roslyn** team members, PRs are allowed to be submitted and merged before a full API review session, but you should have sign-off from the area owner, and the API is expected to be covered in a formal review session before being shipped. For such issues, please work with the API area owner to make sure that the issue is marked blocking appropriately so it will be covered in short order. --- ### Contributing Building, Debugging, And Testing On Unix (`docs/contributing/Building, Debugging, and Testing on Unix.md`) # Building, Debugging and Testing on Unix This guide is meant to help developers setup an environment for debugging / contributing to Roslyn from Linux. Particularly for developers who aren't experienced with .NET Core development on Linux. ## Working with the code 1. Ensure the commands `git` and `curl` are available 1. Clone git@github.com:dotnet/roslyn.git 1. Run `./build.sh --restore` 1. Run `./build.sh --build` ## Working in Visual Studio Code 1. Install [VS Code](https://code.visualstudio.com/Download) - After you install VS Code, install the [C# extension](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) - Important tip: You can look up editor commands by name by hitting *Ctrl+Shift+P*, or by hitting *Ctrl+P* and typing a `>` character. This will help you get familiar with editor commands mentioned below. On a Mac, use *⌘* instead of *Ctrl*. 1. Install the [.NET SDK](https://dotnet.microsoft.com/en-us/download/dotnet) which matches the `sdk.version` property in [global.json](../../global.json) 3. You can build from VS Code by running the *Run Build Task* command, then selecting an appropriate task such as *build* or *build current project* (the latter builds the containing project for the current file you're viewing in the editor). 4. You can run tests from VS Code by opening a test class in the editor, then using the *Run Tests in Context* and *Debug Tests in Context* editor commands. You may want to bind these commands to keyboard shortcuts that match their Visual Studio equivalents (**Ctrl+R, T** for *Run Tests in Context* and **Ctrl+R, Ctrl+T** for *Debug Tests in Context*). 5. You can launch a new VS Code instance with the language server from your current code by running the "launch vscode with language server" task. ## Running Tests After building, the unit tests can be executed by running `./test.sh`, which runs the already-built test assemblies via `src/Tools/RunTests`. For a combined build and test run, use `./build.sh --test`, `./build.sh --testSet:compiler`, `./build.sh --testKind:ioperation`, or `./build.sh --testFramework:core` (single-hyphen spellings are also accepted). Each test option independently requests testing. These shortcuts forward the build configuration and any supplied set, kind, and framework values to RunTests; RunTests selects and validates the tests. For example, `./build.sh --configuration Release --testSet:compiler --testFramework:core --testKind:usedassemblies` builds and tests Release compiler binaries with used-assembly validation. Direct calls to `./eng/build.sh` use already-built binaries unless `--build` is also specified. Repeat `--testFramework:` to forward multiple framework selections, for example `--testFramework:core --testFramework:desktop`; do not combine them into a comma-separated value. RunTests determines which frameworks are supported on the current platform. Pass other test options to `test.sh`. For example, `./test.sh --testFramework:core --testConfiguration Release --testSet:compiler` tests the Release compiler binaries. Use `--testKind:ioperation`, `--testKind:runtimeasync --testFramework:core`, or `--testKind:usedassemblies` for additional validation, and `--env:KEY=VALUE` to set an environment variable in test processes. Runtime-async validation requires Core-only selection; the default selection includes desktop and is rejected. Run `./test.sh --help` for all options. To run all tests in a single project, it's recommended to use the `dotnet test path/to/project` command. ## GitHub The best way to clone and push is to use SSH. On Windows you typically use HTTPS and this is not directly compatible with two factor authentication (requires a PAT). The SSH setup is much simpler and GitHub has a great HOWTO for getting this setup. https://help.github.com/articles/connecting-to-github-with-ssh/ ## Debugging test failures The best way to debug is using lldb with the SOS plugin. This is the same SOS as used in WinDbg and if you're familiar with it then lldb debugging will be pretty straight forward. The [dotnet/diagnostics](https://github.com/dotnet/diagnostics) repo has more information: - [Getting LLDB](https://github.com/dotnet/diagnostics/blob/main/documentation/lldb/linux-instructions.md) - [Installing SOS](https://github.com/dotnet/diagnostics/blob/main/documentation/installing-sos-instructions.md) - [Using SOS](https://github.com/dotnet/diagnostics/blob/main/documentation/sos-debugging-extension.md) CoreCLR also has some guidelines for specific Linux debugging scenarios: - https://github.com/dotnet/coreclr/blob/main/Documentation/botr/xplat-minidump-generation.md - https://github.com/dotnet/coreclr/blob/main/Documentation/building/debugging-instructions.md#debugging-core-dumps-with-lldb. Corrections: - LLDB and createdump must be run as root - `dotnet tool install -g dotnet-symbol` must be run from `$HOME` ### Core Dumps The CoreClr does not used the standard core dumping mechanisms on Linux. Instead you must specify via environment variables that you want a core dump to be produced. The simplest setup is to do the following: ``` > export COMPlus_DbgEnableMiniDump=1 > export COMPlus_DbgMiniDumpType=4 ``` This will cause full memory dumps to be produced which can then be loaded into LLDB. A preview of [dotnet-dump](https://github.com/dotnet/diagnostics/blob/main/documentation/dotnet-dump-instructions.md) is also available for interactively creating and analyzing dumps. ### GC stress failures When you suspect there is a GC failure related to your test then you can use the following environment variables to help track it down. ``` > export COMPlus_HeapVerify=1 > export COMPlus_gcConcurrent=1 ``` The `COMPlus_HeapVerify` variable causes GC to run a verification routine on every entry and exit. Will crash with a more actionable trace for the GC team. The `COMPlus_gcConcurrent` variable removes concurrency in the GC. This helps isolate whether this is a GC failure or memory corruption outside the GC. This should be set after you use `COMPLUS_HeapVerify` to determine it is indeed crashing in the GC. Note: this variables can also be used on Windows as well. ## Ubuntu 18.04 The recommended OS for developing Roslyn is Ubuntu 18.04. This guide was written using Ubuntu 18.04 but should be applicable to most Linux environments. Ubuntu 18.04 was chosen here due to it's support for enhanced VMs in Hyper-V. This makes it easier to use from a Windows machine: full screen, copy / paste, etc ... ### Hyper-V Hyper-V has a built-in Ubuntu 18.04 image which supports enhanced mode. For more information, see a [tutorial for creating such an image](https://docs.microsoft.com/virtualization/hyper-v-on-windows/quick-start/quick-create-virtual-machine). When following this make sure to: 1. Click Installation Source and uncheck "Windows Secure Boot" 1. Complete the Ubuntu installation wizard. Full screen mode won't be available until this is done. Overall this takes about 5-10 minutes to complete. ### Source Link Many of the repositories that need to be built use source link and it crashes on Ubuntu 18.04 due to dependency changes. To disable source link add the following to the `Directory.Build.props` file in the root of the repository. ``` xml false false false ``` ### Prerequisites Make sure to install the following via `apt install` - clang - lldb - cmake - xrdp ## Troubleshooting See the [Troubleshooting section](Building,%20Debugging,%20and%20Testing%20on%20Windows.md#troubleshooting) in the Windows doc, which is not Windows-specific (use `rm -rf` in place of `Remove-Item`). --- ### Contributing Building, Debugging, And Testing On Windows (`docs/contributing/Building, Debugging, and Testing on Windows.md`) # Building, Debugging and Testing on Windows ## Working with the code Using the command line, Roslyn can be developed using the following pattern: 1. Clone https://github.com/dotnet/roslyn 1. Run Restore.cmd 1. Run Build.cmd 1. Run Test.cmd ## Recommended version of .NET Framework The minimal required version of .NET Framework is 4.7.2. ## Developing with Visual Studio 1. Use latest [Visual Studio Insiders](https://visualstudio.microsoft.com/insiders/) - Ensure Visual Studio extension development is included in the selected workloads - Ensure C# and Visual Basic, MSBuild, and .NET Core are included in the selected individual components - Restart Visual Studio 1. Install the [.NET SDK](https://dotnet.microsoft.com/en-us/download/dotnet) which matches the `sdk.version` property in [global.json](../../global.json) 1. [PowerShell 5.0 or newer](https://learn.microsoft.com/en-us/powershell/scripting/install/install-powershell-on-windows). If you are on Windows 10 or newer, you are fine; you'll only need to upgrade if you're on earlier versions of Windows. The download link is under the ["Upgrading existing Windows PowerShell"](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-windows-powershell?view=powershell-6#upgrading-existing-windows-powershell) heading. 1. Run Restore.cmd 1. Open Roslyn.slnx ## Developing with Visual Studio Code See the [Building, Debugging, and Testing on Unix](Building,%20Debugging,%20and%20Testing%20on%20Unix.md#working-in-visual-studio-code) documentation to get started developing Roslyn using Visual Studio Code. ## Running Tests There are a number of options for running the core Roslyn unit tests: ### Command Line The Test.cmd script runs unit tests on already-built test binaries via `src/Tools/RunTests`. Run `Build.cmd` separately before testing, and pass test options to `Test.cmd`. For a combined build and test run, use `Build.cmd -test`, `Build.cmd -testSet:compiler`, `Build.cmd -testKind:ioperation`, or `Build.cmd -testFramework:core`. Each test option independently requests testing. These shortcuts forward the build configuration and any supplied set, kind, and framework values to RunTests; RunTests selects and validates the tests. For example, `Build.cmd -configuration Release -testSet:compiler -testFramework:core -testKind:usedassemblies` builds and tests Release compiler binaries with used-assembly validation. Direct calls to `eng\build.ps1` use already-built binaries unless `-build` is also specified. Other test options belong on `Test.cmd`. To select multiple frameworks from PowerShell, pass an array: `.\eng\build.ps1 -build -testFramework:core,desktop`. Each array element is forwarded as a separate RunTests framework option; do not repeat the named PowerShell parameter. 1. Run the "Developer PowerShell for VS Insiders" or "Developer Command Prompt for VS Insiders" from your start menu, alternatively launch it from Visual Studio, it can be found at "Tools -> Command Line -> Developer Powershell/Developer Command Prompt". 2. Navigate to the directory of your Git clone. 3. Run `Test.cmd` in the command prompt. Pass options directly to `Test.cmd`: `--testFramework:desktop` selects .NET Framework, while `--testFramework:core` selects .NET. Repeat `--testFramework` to run both. Use `--testConfiguration Release` to test Release binaries and `--testPlatform x86` to select x86. Run `Test.cmd --help` for all options. Visual Studio integration tests have a separate entry point: run `eng\test-vsi.ps1 -configuration Debug` after building. The results of the tests can be viewed in the artifacts/TestResults directory. ### Test Explorer Tests can be run and debugged from the Test Explorer window. For best performance, we recommend the following: 1. Open **Tools → Options... → Test** 1. Check the box for **Discover tests in real time from source files** 2. Uncheck the box for **Additionally discover tests from build assemblies...** 2. Use the Search box of Test Explorer to narrow the scope of visible tests to the feature(s) you are working on 3. When you are not actively running tests, set the search query to `__NonExistent__` to hide all tests from the UI #### Testing in Linux using WSL Tests can be run and debugged under WSL. This requires a bit of setup the first time. After that, it is as easy as selecting WSL as the active environment in Test Explorer. 1. Install WSL by running `wsl --install` ([details](https://docs.microsoft.com/en-us/windows/wsl/setup/environment#get-started)) 2. In the VS installer, install ".NET Debugging with WSL" as an individual component 3. In Test Explorer, go into "Configure Remote Test Environments" (under the gear icon) and uncomment the wsl/Ubuntu environment: 4. Select that test environment from Test Explorer drop-down: 5. Run a test from Test Explorer test list. This will prompt you to install some dotnet bits into the WSL environment the first time. 6. Debug a test. This will prompt you to install some remote debugging bits into the WSL environment the first time. [More details](https://docs.microsoft.com/en-us/visualstudio/debugger/debug-dotnet-core-in-wsl-2) ### WPF Test Runner To debug through tests, you can right click the test project that contains your tests and choose **Set as Startup Project**. Then press F5. This will run the tests under the command line runner. Some members of the team have been working on a GUI runner that allows selection of individual tests, etc. Grab the source from [xunit.runner.wpf](https://github.com/pilchie/xunit.runner.wpf), build it and give it a try. ## Trying Your Changes in Visual Studio ### Deploying with command-line (recommended method) You can build and deploy with the following command: `.\Build.cmd -Restore -Configuration Release -deployExtensions -launch`. Then you can launch the `RoslynDev` hive with `devenv /rootSuffix RoslynDev`. ### Deploying with F5 The Rosyln solution is designed to support easy debugging via F5. Several of our projects produce VSIX which deploy into Visual Studio during build. The F5 operation will start a new Visual Studio instance using those VSIX which override our installed binaries. This means trying out a change to the language, IDE or debugger is as simple as hitting F5. Note that for changes to the compiler, out-of-process builds won't use the privately built version of the compiler. The startup project needs to be set to **RoslynDeployment**. This should be the default but in some cases will need to be set explicitly. To set it, right-click the RoslynDeployment project in Solution Explorer and select "Set as Startup Project". **RoslynDeployment** is a container project located in the `src/Deployment` folder that bundles and deploys all the main Roslyn extensions together. When you press F5 with RoslynDeployment set as the startup project, it will deploy all of the following extensions at once, giving you a complete debugging experience with all Roslyn components. If you're working on a specific area and want to optimize build times by deploying only the relevant extension, you can set one of the individual projects below as your startup project instead: - **Roslyn.VisualStudio.Setup**: this project can be found inside the VisualStudio\Setup folder from the Solution Explorer, and builds Roslyn.VisualStudio.Setup.vsix. It contains the core language services that provide C# and VB editing. It also contains the copy of the compiler that is used to drive IntelliSense and semantic analysis in Visual Studio. Although this is the copy of the compiler that's used to generate squiggles and other information, it's not the compiler used to actually produce your final .exe or .dll when you do a build. If you're working on fixing an IDE bug, this is the project you want to use. - **Roslyn.Compilers.Extension**: this project can be found inside the Compilers\Extension folder from the Solution Explorer, and builds Roslyn.Compilers.Extension.vsix. This deploys a copy of the command line compilers that are used to do actual builds in the IDE. It only affects builds triggered from the Visual Studio experimental instance it's installed into, so it won't affect your regular builds. Note that if you install just this, the IDE won't know about any language features included in your build. If you're regularly working on new language features, you may wish to consider building both the CompilerExtension and VisualStudioSetup projects to ensure the real build and live analysis are synchronized. - **ExpressionEvaluatorPackage**: this project can be found inside the ExpressionEvaluator\Package folder from the Solution Explorer, and builds ExpressionEvaluatorPackage.vsix. This deploys the expression evaluator and result providers, the components that are used by the debugger to parse and evaluate C# and VB expressions in the Watch window, Immediate window, and more. These components are only used when debugging. The experimental instance used by Roslyn is an entirely separate instance of Visual Studio with it's own settings and installed extensions. It's also, by default, a separate instance than the standard "Experimental Instance" used by other Visual Studio SDK projects. If you're familiar with the idea of Visual Studio hives, we deploy into the RoslynDev root suffix. ### Deploying with VSIX and Nuget package If you want to try your extension in your day-to-day use of Visual Studio, you can find the extensions you built in your Binaries folder with the .vsix extension. You can double-click the extension to install it into your main Visual Studio hive. This will replace the base installed version. Once it's installed, you'll see it marked as "Experimental" in Tools > Extensions and Updates to indicate you're running your experimental version. You can uninstall your version and go back to the originally installed version by choosing your version and clicking Uninstall. If you only install the VSIX, then the IDE will behave correctly (ie. new compiler and IDE behavior), but the Build operation or building from the command-line won't. To fix that, add a reference to the `Microsoft.Net.Compilers.Toolset` you built into your csproj. As shown below, you'll want to (1) add a nuget source pointing to your local build folder, (2) add the package reference, then (3) verify the Build Output of your project with a `#error version` included in your program. ## Trying Your Changes in Visual Studio Code See [VSCode docs](https://github.com/dotnet/vscode-csharp/blob/main/CONTRIBUTING.md#using-a-locally-developed-roslyn-server). ## Various other tips ### Referencing bootstrap compiler If you made changes to a Roslyn compiler and want to build any projects with it, you can either use the Visual Studio hive where your **CompilerExtension** is installed, or from command line, run msbuild with `/p:BootstrapBuildPath=YourBootstrapBuildPath`. `YourBootstrapBuildPath` could be any directory on your machine so long as it had csc and vbc inside it. You can check the cibuild.cmd and see how it is used. ### Troubleshooting your setup To confirm what version of the compiler is being used, include `#error version` in your program and the compiler will produce a diagnostic including its own version as well as the language version it is operating under. You can also attach a debugger to Visual Studio and check the loaded modules, looking at the folder where the various `CodeAnalysis` modules were loaded from (the `RoslynDev` should load them somewhere under `AppData`, not from `Program File`). ### Testing on the [dotnet/runtime](https://github.com/dotnet/runtime) repo 1. Make sure that you can build the `runtime` repo as baseline (run `build.cmd libs+libs.tests`, which should be sufficient to build all C# code, installing any prerequisites if prompted to, and perhaps `git clean -xdf` and `build.cmd -restore` initially - see [runtime repo documentation](https://github.com/dotnet/runtime/blob/main/docs/workflow/README.md) for specific prerequisites and build instructions) 2. `build.cmd -pack -c Release` on your `roslyn` repo - Note that `-c Debug` can also be used (along with changing `Release` to `Debug` in `RestoreAdditionalProjectSources` property value below). This will allow checking the compiler's debug assertions when building the runtime. - It's good for us to investigate scenarios where compiling the runtime libraries causes the compiler's debug assertions to fail. However, assertion failures can obscure whether or not the compiler ultimately succeeds at building the runtime and producing correct binaries. So, if the goal is to only check for "functional breaks", then using a Release mode compiler to start with can be preferable. 3. Find the compiler toolset in your NuGet package cache (its location is likely `%NUGET_PACKAGES%\microsoft.net.compilers.toolset` or `%userprofile%\.nuget\packages\microsoft.net.compilers.toolset`). 4. If there is a toolset version in the NuGet cache with the same version as the toolset you just packed in Roslyn, delete the toolset version from the cache. This allows NuGet to pick up the new packages you just created locally instead of using a stale cached version. 5. Modify your local enlistment of `runtime` similarly to [this commit](https://github.com/RikkiGibson/runtime/commit/runtime-local-roslyn-build-example) then build again - add `\artifacts\packages\Release\Shipping\` using the local path to your `roslyn` repo to `Directory.Build.props` - add `true` and `5.3.0-dev` with the package version you just packed (look in above artifacts folder) to `eng/Versions.props` ### Testing a VS insertion See internal documentation for that process [here](https://microsoft.sharepoint.com/teams/managedlanguages/_layouts/OneNote.aspx?id=%2Fteams%2Fmanagedlanguages%2Ffiles%2FTeam%20Notebook%2FRoslyn%20Team%20Notebook%20-%20New&wd=target%28Infrastructure.one%7C0E3C75FE-0827-41BF-9696-F5CD7105BC9A%2FRun%20RPS%20against%20Roslyn%20PR%7C7A74A1E4-1507-41BC-84F1-F9159C853177%2F%29). ### Testing with extra IOperation validation Run `Test.cmd --testFramework:core --testSet:compiler --testKind:ioperation`, which sets the `ROSLYN_TEST_IOPERATION` environment variable to `true` in the test processes. For running those tests in an IDE, the easiest is to find the `//#define ROSLYN_TEST_IOPERATION` directive and uncomment it. See more details in the [IOperation test hook](https://github.com/dotnet/roslyn/blob/main/docs/compilers/IOperation%20Test%20Hook.md) doc. ### Replicating Failures in the Used Assemblies leg In order to replicate test failures in that leg, there are a few options: 1. Uncomment `src/Compilers/Test/Core/Compilation/CompilationExtensions.cs:9`, which defines `ROSLYN_TEST_USEDASSEMBLIES`, and run your tests. Do _not_ check this in, as it will enable the test hook for every test in every project and significantly slow down regular test runs. 2. Set the `ROSLYN_TEST_USEDASSEMBLIES` environment variable and restart VS with it set. 3. Set a breakpoint at the start of `CSharpTestBase.VerifyUsedAssemblyReferences` in `src/Compilers/Test/Utilities/CSharp/CSharpTestBase.cs`. When it breaks, use VS's jump to location or drag the instruction pointer past the early check and return on `EnableVerifyUsedAssemblies`. When a test failure is isolated, please add a _dedicated_ test for this (ie. failing even when the Used Assemblies validation isn't enabled) to make it easier to avoid future regressions. Preferrably, don't replicate the entire original test, just enough to hit the bug to ensure that it's protected against regressions. Before pushing a relevant fix to CI, validate locally using `Test.cmd --testFramework:core --testSet:compiler --testKind:usedassemblies`. ### Running the PublicAPI fixer 1. Restore and build `Compilers.slnf`. This is necessary to ensure the source generator project is built and we can load the generator assembly when running `dotnet format`. ```ps1 C:\Source\roslyn> .\restore.cmd C:\Source\roslyn> .\build.cmd ``` 2. Invoke `dotnet format` (the one included in .NET SDK, not the global tool `dotnet-format` which is deprecated). Running only the analyzers subcommand and fixing only the "missing Public API signature" diagnostic. We must also pass the `--include-generated` flag to include source generated documents in the analysis. ```ps1 C:\Source\roslyn> cd .. C:\Source> dotnet format analyzers .\roslyn\Compilers.slnf --diagnostics=RS0016 --no-restore --include-generated -v diag ``` ## Troubleshooting If you hit an unexpected build failure, especially one that doesn't reproduce on a clean clone, a good first step is to delete the `artifacts` folder and rebuild. If that doesn't help, the bigger hammer is `git clean -xdfn` (dry run) followed by `git clean -xdf`. Note that this also deletes the locally-provisioned `.dotnet` SDK, forcing a full restore and rebuild, so prefer just deleting `artifacts` first. For example, renaming or moving your clone can leave stale cached files under `artifacts\obj\...\*.xlf\` referencing the old path, which causes `Build.cmd` to fail with errors like: ``` error MSB3103: Invalid Resx file. Could not find a part of the path '\src\RoslynAnalyzers\Text.Analyzers\Core\Dictionary.dic'. [Text.Analyzers.csproj] ``` Deleting `artifacts` and rebuilding should fix the issue. See [dotnet/arcade issue 17061](https://github.com/dotnet/arcade/issues/17061) for details. ## Contributing Please see [Contributing Code](https://github.com/dotnet/roslyn/blob/main/CONTRIBUTING.md) for details on contributing changes back to the code. --- ### Contributing Compiler Test Plan (`docs/contributing/Compiler Test Plan.md`) # Compiler Test Plan This document provides guidance for thinking about language interactions and testing compiler changes. ## General concerns - Completeness of the specification as a guide for testing (is the spec complete enough to suggest what the compiler should do in each scenario?) - *Ping* for new breaking changes and general ping for partner teams (Bill, Kathleen, Mads, IDE, Razor) - Help review external documentation - Backward and forward compatibility (interoperation with previous and future compilers, each in both directions) - Error handling/recovery (missing libraries, including missing types in mscorlib; errors in parsing, ambiguous lookup, inaccessible lookup, wrong kind of thing found, instance vs static thing found, wrong type for the context, value vs variable) - BCL (including mono) and other customer impact - Determinism - Loading from metadata (source vs. loaded from metadata) - VB/F# interop - C++/CLI interop (particularly for metadata format changes, e.g. DIMs, static abstracts in interfaces, or generic attributes) - Performance and stress testing - Can build VS - Check that `Obsolete` is honored for members used in binding/lowering - LangVersion - IL verification (file issue on `runtime` repo as needed and track [here](https://github.com/dotnet/roslyn/issues/22872)) - Does the feature use cryptographic hashes in any way? (examples: metadata names of file-local types, extension types, assembly strong naming, PDB document table, etc.) - Consider using non-cryptographic hash such as `XxHash128` instead. - If you must use a cryptographic hash in the feature implementation, then use `SourceHashAlgorithms.Default`, and not any specific hash. - A cryptographic hash must never be included in a public API name. Taking a change to the default crypto algorithm would then change public API surface, which would be enormously breaking. - **DO NOT** allow using the value of a crypto hash in a field, method or type name - **DO** allow using the value of a crypto hash in attribute or field values - Any time the compiler reads in metadata containing crypto hashes, even if it's an attribute value, ensure the crypto hash algorithm name is included in the metadata (e.g. prefixing it to the hash value), so that it can be changed over time and the compiler can continue to read both metadata using both the old and new algorithms. ## Public APIs - Public compiler APIs (including semantic model and other APIs listed below): - GetDeclaredSymbol - GetEnclosingSymbol - GetSymbolInfo - GetSpeculativeSymbolInfo - GetTypeInfo - GetSpeculativeTypeInfo - GetMethodGroup - GetConstantValue - GetAliasInfo - GetSpeculativeAliasInfo - LookupSymbols - AnalyzeStatementsControlFlow - AnalyzeStatementControlFlow - AnalyzeExpressionDataFlow - AnalyzeStatementsDataFlow - AnalyzeStatementDataFlow - ClassifyConversion - GetOperation (`IOperation`) - GetCFG (`ControlFlowGraph`), including a scenario with some nested conditional - DocumentationCommentId APIs - All newly added APIs are experimental - Tracking issue for marking APIs as non-experimental - APIs are marked with `[Experimental(RoslynExperiments.PreviewLanguageFeatureApi, UrlFormat = "link to tracking issue")]` - APIs have gone through API review ## Type and members See also [types](#types) and [members](#members) lists below. - Access modifiers (public, protected, internal, protected internal, private protected, private), static, ref - type declarations - class with or without primary constructor - record class/struct with or without positional members - struct with or without primary constructor - union - interface - type parameter - file-local types - methods - constructors (static, instance, primary) - destructors - fields (required and not) - fixed-sized buffers - properties (including get/set/init accessors, required and not) - events (including add/remove accessors) - Parameter modifiers: ref, out, in, ref readonly, params (for array, for non-array) - Attributes (including generic attributes and security attributes) - Compiler-recognized attributes should not have any effect in earlier LangVersions, except a LangVersion error should be reported when functionality depending on the attribute is used (for example, InlineArray conversion to Span). - Generics (type arguments, variance, constraints including `class`, `struct`, `new()`, `unmanaged`, `notnull`, `allows ref struct`, types and interfaces with nullability) - Default and constant values - Partial classes - Literals - Enum (implicit vs. explicit underlying type) - Expression trees - Iterators - Initializers (object, collection, dictionary) - Array (single- or multi-dimensional, jagged, initializer, fixed) - Expression-bodied methods/properties/... - Extension methods - Partial method - Named and optional parameters - String interpolation - Raw strings (including interpolation) - Properties (read-write, read-only, init-only, write-only, auto-property, expression-bodied) - Interfaces (implicit vs. explicit interface member implementation) - Delegates - Multi-declaration - NoPIA - Dynamic - Ref structs, Readonly structs - Readonly members on structs (methods, property/indexer accessors, custom event accessors) - SkipLocalsInit - Method override or explicit implementation with `where T : { class, struct, default }` - `extension` blocks (emitted with content-based names): - methods - operators (except conversion operators) - properties and indexers (including get/set/init accessors) ## Code - Operators (see Eric's list below) - Lambdas (capture of parameters or locals, target typing) - Execution order - Target typing (var, lambdas, integrals) - Conversions (boxing/unboxing) - Nullable (wrapping, unwrapping) - Overload resolution, override/hide/implement (OHI) - OverloadResolutionPriorityAttribute - Inheritance (virtual, override, abstract, new) - Anonymous types - Tuple types and literals (elements with explicit or inferred names, long tuples), tuple equality - Range literals (`1..2`) and Index operator (`^1`) - Deconstructions - Local functions - Unsafe code - LINQ - Async (task-like types) and async-iterator methods - Lvalues: the synthesized fields are mutable - Ref / out parameters - Compound operators (`+=`, `/=`, etc ..) - Assignment exprs - Ref return, ref readonly return, ref ternary, ref readonly local, ref local re-assignment, ref foreach - Ref fields - `scoped` parameters and locals - `this = e;` in `struct` .ctor - Stackalloc (including initializers) - Patterns (constant, declaration, `var`, positional, property and extended property, discard, parenthesized, type, relational, `and`/`or`/`not`, list, slice, constant `string` matching `Span`) - Switch expressions - With expressions (on record classes and on value types) - Nullability annotations (`?`, attributes) and analysis - Definite assignment analysis and auto-default struct fields - If you add a place an expression can appear in code, make sure `SpillSequenceSpiller` handles it. Test with a `switch` expression or `stackalloc` in that place. - If you add a new expression form that requires spilling, test it in the catch filter. - If you add a new expression that can be nested inside itself, test deeply nesting it. Potentially cut off direct nesting. - extension based Dispose, DisposeAsync, GetEnumerator, GetAsyncEnumerator, Deconstruct, GetAwaiter etc. - UTF8 String Literals (string literals with 'u8' or 'U8' type suffix). - Inline array element access and slicing. - Collection expressions, spread-elements (`.. expr`), and with-elements (`[with(...)]`). ## Misc - reserved keywords (sometimes contextual) - pre-processing directives - COM interop - modopt and modreq - CompilerFeatureRequiredAttribute - CompilerLoweringPreserveAttribute - ref assemblies - extern alias - UnmanagedCallersOnly - telemetry ## Testing in interaction with other components Interaction with IDE, Debugger, and EnC should be worked out with relevant teams. A few highlights: - IDE - Colorization and formatting - Typing experience and dealing with incomplete code - Intellisense (squiggles, dot completion) - "go to", Find All References, and renaming - F1 help (for new keywords) - cref comments - UpgradeProject code fixer - More: [IDE Test Plan](https://github.com/dotnet/roslyn/blob/main/docs/contributing/IDE%20Test%20Plan.md) - Debugger / EE - Stepping, setting breakpoints - Displaying Locals/Autos windows - Type and Value display - Expanding instance members - Locals/parameters/this in closure classes - Attributes on locals and expressions (e.g.: [Dynamic], [TupleElementNames]) - Compiling expressions in Immediate/Watch windows or hovering over an expression - Compiling expressions in [DebuggerDisplay("...")] - Assigning values in Locals/Autos/Watch windows - [Edit-and-continue](https://github.com/dotnet/roslyn/blob/main/docs/contributing/Testing%20for%20Interactive%20readiness.md) - Live Unit Testing (instrumentation) - Engage with VS Templates team (if applicable) ## Statements ```cs { … } ; label : … T x = whatever; // including `using` and `await` using variants M(); ++x; x++; --x; x--; a?.M(); // conditional access / null-propagating operator a?[b] = c; a?.b = c; a?[b] = c; a?.b += c; // and other compound assignment cases var x = a?.M(); // similar "value is used" versions of the above '?.' cases new C(); if (…) … else … switch(…) { … case (…) when (…): … } while(…) … do … while(…); for( … ; … ; … ) … foreach(…) … // including `await` variant fixed(…) … // (plain, or custom with `GetPinnableReference`) goto … ; throw … ; return … ; try { … } catch (…) when (…) { … } finally { … } checked { … } unchecked { … } lock(…) … // including variant on an instance of the `System.Threading.Lock` type using (…) … // including `await` variant yield return …; yield break; break; continue; break label; continue label; ``` ## Expression classifications Every expression can be classified as exactly one of these: - Value - Variable - Namespace - Type - Method group - Null literal - Default literal - Anonymous function - Property - Indexer - Event - Void-returning method call - Array initializer (\*) - __arglist (\*) (\*) Technically not an expression according to the spec. Note that only values, variables, properties, indexers and events have a type. ## Variable classifications A variable is a storage location. These are all the different ways to refer to a storage location: - Static field - Instance field - Array element - Formal param, value - Formal param, ref - Formal param, out - Local variable - Pointer dereference - __refvalue - Inline array access ## Operators ``` c# x.y f( ) a[e] x++ (including instance user defined) x-- (including instance user defined) new X() new() typeof(T) default(T) default checked(e) unchecked(e) delegate ( ) { } +x -x !x ~x ^x ++x (including instance user defined) --x (including instance user defined) (X)x x * y x / y x % y x + y x - y x << y x >> y x >>> y x < y x > y x <= y x >= y x is X x as X x == y x != y x & y x ^ y x | y x && y x || y x ?? y x ? : y : z x = y x *= y (including instance user defined) x /= y (including instance user defined) x %= y (including instance user defined) x += y (including instance user defined) x -= y (including instance user defined) x <<= y (including instance user defined) x >>= y (including instance user defined) x >>>= y (including instance user defined) x &= y (including instance user defined) x ^= y (including instance user defined) x |= y (including instance user defined) x ??= y x => { } sizeof( ) *x & x x->y e is pattern e switch { ... } await x __arglist( ) __refvalue( x, X ) __reftype( x ) __makeref( x ) ``` ## Explicit conversions - Numeric - Enum - Nullable - Reference - Unboxing - Dynamic - Type parameter - User defined - Pointer to pointer - Pointer to integral - Integral to pointer - Tuple literal - Tuple ## Implicit conversions - Identity - Numeric - Literal zero to enum (we actually do constant zero, not literal zero) - Nullable - Null literal - Reference - Boxing - Dynamic - Constant - Type parameter - User defined - Union - Anonymous function - Method group - Pointer to void pointer - Null literal to pointer - Interpolated string - Tuple literal - Tuple - Default literal - Implicit object creation (target-typed new) - Function type (in type inference comparing function types of lambdas or method groups) - UTF8 String Literal (string constant value to ```byte[]```, ```Span```, or ```ReadOnlySpan``` types) - Inline arrays (conversions to Span and ReadOnlySpan) - Collection expression conversions - Span conversions (from array to (ReadOnly)Span, or from string or (ReadOnly)Span to ReadOnlySpan) ## Types - Class - Interface - Delegate - Struct - Union - Enum - Nullable - Pointer - Type parameter ## Members - Class - Struct - Union - Interface - Enum - Delegate - Namespace - Property - Event - Constructor - Destructor - Method - Interface method - Field - User-defined indexer - User-defined operator (including checked, including instance increment/decrement and compound assignment) - User-defined conversion (including checked) ## Patterns - Discard Pattern - Var Pattern - Declaration Pattern - Constant Pattern - Constant `string` matching `Span` - Recursive Pattern - Parenthesized Pattern - `and` Pattern - `or` Pattern - `not` Pattern - Relational Pattern - Type Pattern - Union matching rules ## Metadata table numbers / token prefixes If you look at a 32 bit integer token as a hex number, the first two digits identify the “table number” and the last six digits are an offset into that table. The table numbers are: ``` 00 Module 01 TypeRef 02 TypeDef 03 FieldPtr 04 Field 05 MethodPtr 06 Method 07 ParamPtr 08 Param 09 InterfaceImpl 0A MemberRef 0B Constant 0C CustomAttr 0D FieldMarshal 0E DeclSecurity 0F ClassLayout 10 FieldLayout 11 StandAloneSig 12 EventMap 13 EventPtr 14 Event 15 PropertyMap 16 PropertyPtr 17 Property 18 MethodSemantics 19 MethodImpl 1A ModuleRef 1B TypeSpec 1C ImplMap 1D FieldRVA 1E ENCLog 1F ENCMap 20 Assembly 21 AssemblyProcessor 22 AssemblyOS 23 AssemblyRef 24 AssemblyRefProcessor 25 AssemblyRefOS 26 File 27 ExportedType 28 ManifestResource 29 NestedClass 2A GenericParam 2B MethodSpec 2C GenericConstraint ``` --- ### Contributing Developing A Language Feature (`docs/contributing/Developing a Language Feature.md`) # Developing a Language Feature Adding a new feature to C# or VB is a very serious undertaking that often takes several iterations to complete for even the (seemingly) simplest of features. This is due to both the inherent complexity of changing languages and the need to consider the effects of new features in all layers of the Roslyn codebase: IDE, debugging, scripting, etc. As such, language work occurs in a separate branch until the feature reaches a point when we are ready to merge it into the main compiler. This page discusses the process by which language feature *implementations* are considered, prototyped, and fully accepted into the language. This process is intended to be used by the compiler team and community. ## Process 1. **Feature specification filed**: The speclet should be filed as a GitHub issue and contain: * A description of the feature (including any syntax changes involved) * Discussions about impacted areas, such as overload resolution and type inference. Think through the major areas of the language specification while determine the potential impacts * Proposed changes to the API surface area. A feature speclet is different from a language design discussion. Discussions are very open-ended and often for features that simply won't work in the language. A feature speclet is a declaration of intent to do the feature as proposed. In many cases, this will come out of a language design discussion. Linking to an early prototype of the proposed feature is highly encouraged. It gives the Language Team a clear understanding of the design intent of the feature and can often serve as the initial commit in the feature branch. *Note*: The goal is not to have a 100%-complete design but instead a general understanding of how the language feature is intended to work. 1. **Discussion**: The goal of the discussion is to help clarify the design and determine if the Language Team wants to sanction the proposal. As the discussion evolves the author is encouraged to update the speclet to incorporate the changing design. This allows folks who join later to understand the most recent proposal by reading the top of the thread. 1. **Language team makes a decision**: When the Language Team believes enough information is available they will update the issue accordingly: * **Ready for prototype**: The feature is worth pursuing as a prototype for the language. Work continues in the "Prototype" phase. * **Not actionable**: The discussion didn't lead to a design that was actionable by the team. * **Declined as proposed**: The proposal is overall a good idea but the proposed implementation details are not as efficient as they could be. In most cases, the language team will attempt to steer the discussion to a cleaner design but understand that it may be simpler to close and start over with a new issue. * **Backlog**: The feature is a good idea with an acceptable design but the Language Team simply can't allocate the resources at this time for this work. * **Won't Fix**: The feature isn't taking the language in a direction the Language Team is interested in pursuing. A Language Team member will be paired with a community-driven feature when it is deemed 'Ready for prototype'. This team member will act as the primary source of contact for the community and will help drive the feature to completion. 1. **Prototype phase**: All features are initially prototyped in a separate branch. The name of the branch is `features/`. These branches are created when a feature speclet is marked as 'Ready for prototype', language team member champions the feature, and [a developer is assigned](#community-contributions) to implement the feature. The developer is responsible for keeping an up-to-date speclet of the feature at `docs/features/.md` throughout the development. When starting work on a new feature, use the `new-compiler-feature` Copilot skill (in `.github/skills/new-compiler-feature/`) to set up tracking: it creates a test plan issue, a `Feature - ` label, updates the [Language Feature Status](../Language%20Feature%20Status.md) page, and links related PRs. Run it from a PR that implements (part of) the feature. All features should be off by default until fully accepted into the language. A new feature should only be enabled when explicitly included in the `/features:` compiler option. This helps ensure the language is always in a shippable state no matter what prototypes are in the current branch. 1. **Prototype decision**: When the prototype phase ends, the Language Team will make a decision on the feature: * **Ready to integrate**: The feature implementation revealed no blocking issues, has a healthy set of test suites, and is sufficiently complete in the compiler, IDE, debugger, etc. to integrate into the main tree on a selected branch. The test suites need to focus both on the feature in isolation and when combined with other areas of the language. * **Iterate**: Experience with the prototype has produced valuable feedback suggesting modifications to the speclet. Work reverts to the Discussion state, which may lead to speclet and prototype changes (or even abandoning the feature!). This typically occurs either when the original speclet left some language details unspecified, to be resolved based on implementation experience, or when experience with the prototype suggests that some language design decisions should be reconsidered. * **Closed**: The feature hit unforeseen design issues, implementation roadblocks, etc. that caused it to be removed. 1. **Finishing**: Once a feature is integrated into the main tree, it's time to lock down and finish the implementation. This includes any remaining areas not completed in the prototype, and completing all of the corner cases and tests (lots and lots of tests!). Features which are in the main tree but not tied to a langversion switch are referred to as experimental. They can be pulled or changed at any time. 1. **Acceptance**: When the Language Team feels a feature is complete it will be accepted into the language. The corresponding feature flag will be removed and the feature will be tied to the appropriate `/langversion` option. It's important to note there are **no guarantees** about a feature being a part of the language until it has reached the 'Acceptance' state **and** shipped in an official build. Features in any other state--no matter how complete--can be pulled at the last minute if the Language Team deems it's necessary to do so. Pulling a feature after the prototype phase completes is not a decision the Language Team takes lightly; however, in the past, unforeseen circumstances have caused us to do so. The best example of this was the decision to pull parameterless struct constructors very late in the C# 6.0 timeframe. ## Community Contributions The intent of this document is to codify the process used by the Language Team to implement language features so that the community can contribute implementations. Community members can contribute to ongoing prototypes or in some cases be the primary driver of the prototype. Community members who wish to drive a particular feature can request so by commenting on an accepted feature speclet. Even when driven by a community member, a language prototype will occupy a non-trivial amount of time by the Language Team. Hence we will likely only grant this request to community members who have been actively contributing to the project in smaller areas. ## Frequently Asked Questions - **Can feature branches be rebased?**: This is left to the discretion of the developer working on the feature. Once the feature is merged into a main branch (main, feature) then rebasing is no longer allowed. - **Why was my language feature PR closed?**: Language features will not be accepted as direct PRs to a main branch. Even simple features are too involved for senior Language Team members to complete in this fashion. Features need to follow the above process in order to be incorporated into the language. - **Why are features disabled by default?**: Language features spend a considerable amount of time in the same branch as our shipping product which is often main. The main branch of Roslyn is kept in a ship ready state at all times. To facilitate this all features which haven't been fully accepted into the language must be disabled. - **What is the difference between the feature and main branches?**: The main branch represents code which will ship in updates to the current version of the compiler. The feature branches represent features that are still in progress, therefore not merged yet. - **Why weren't speclets filed for early features?**: Features such as tuples and local functions were done before this process was fully developed. Their speclets were considered internally and eventually became the `docs/feature/` file. - **Why the high bar to create a feature branch?**: Language features, even when driven by the community, require a substantial long term resource investment by the language team. The team must do regular code reviews, provide guidance, help with architectural issues, contribute, etc ... --- ### Contributing Documentation For IDE CodeStyle Analyzers (`docs/contributing/Documentation for IDE CodeStyle analyzers.md`) # Documentation for IDE CodeStyle analyzers ## Overview 1. Official documentation for all [IDE analyzer diagnostic IDs](../../src/Analyzers/Core/Analyzers/IDEDiagnosticIds.cs) is added to `dotnet/docs` repo at . 2. Each IDE diagnostic ID has a dedicated documentation page. For example: 1. Diagnostic with code style option: 2. Diagnostic without code style option: 3. Multiple diagnostic IDs sharing the same option(s): . In this case, a [redirection](https://github.com/dotnet/docs/blob/7ae7272d643aa1c6db96cad8d09d4c2332855960/.openpublishing.redirection.fundamentals.json#L11-L14) for all diagnostic IDs must be added. 3. We have tabular indices for IDE diagnostic IDs and code style options for easier reference and navigation: 1. Primary index (rule ID + code style options): 2. Secondary index (code style options only): 3. Indices for each preference/option group. For example: 1. Modifier preferences: 2. Expression-level preferences: ## Action items when adding or updating IDE analyzers Roslyn IDE team (@dotnet/roslyn-ide) is responsible for ensuring that the documentation for IDE analyzers is up-to-date. Whenever we add a new IDE analyzer, add new code style option(s) OR update semantics of existing IDE analyzers the following needs to be done: 1. **Action required in [dotnet/docs](https://github.com/dotnet/docs) repo**: 1. If you _creating a Roslyn PR for an IDE analyzer_: 1. Please follow the steps at [Contribute docs for 'IDExxxx' rules](https://docs.microsoft.com/contribute/dotnet/dotnet-contribute-code-analysis#contribute-docs-for-idexxxx-rules) to add documentation for IDE analyzers to [dotnet/docs](https://github.com/dotnet/docs) repo. 2. Ideally, the documentation PR should be created in parallel to the Roslyn PR or immediately following the approval/merge of Roslyn PR. 2. If you are _reviewing a Roslyn PR for an IDE analyzer_: 1. Please ensure that the analyzer author is made aware of the above doc contribution requirements, especially if it is a community PR. 2. If the documentation PR in is not being created in parallel to the Roslyn PR, please ensure that a [new doc issue](https://github.com/dotnet/docs/issues) is filed in `dotnet/docs` repo to track this work. 2. **No action required in Roslyn repo**: Help links have been hooked up to IDE code style analyzers in Roslyn repo in the base analyzer types, so every new IDE analyzer diagnostic ID will automatically have `https://docs.microsoft.com/dotnet/fundamentals/code-analysis/style-rules/ideXXXX` as its help link. There is no action required on the Roslyn PR to ensure this linking. If you feel any of the existing IDE rule docs have insufficient or incorrect information, please submit a PR to [dotnet/docs](https://github.com/dotnet/docs) repo to update the documentation. --- ### Contributing IDE Test Plan (`docs/contributing/IDE Test Plan.md`) This resource has moved to [../wiki/Manual-Testing.md](../wiki/Manual-Testing.md) --- ### Contributing Localization In Compiler Tests (`docs/contributing/Localization In Compiler Tests.md`) Localization in Compiler Tests === The compiler tests are structured such that they can be run on a machine using any locale. This both serves as a tool to ensure the compiler does not have any localization errors but also to ensure that developers from any locale can contribute to the project. The ability to build and execute tests should not be limited to English speaking developers. Our infrastructure runs the compiler test suite on `es-ES` machines for both .NET Core and .NET Framework. This helps us ensure that compiler tests can be authored and executed on non-English machines without issues. ## How it works To ensure that our tests run on any locale there is a general approach that is followed in the vast majority of our tests. 1. All expected baselines are expressed in invariant culture. That means that say `decimal` values are expressed using `.` separators not `,`. 2. All generated baselines, usually via `Console.Writeline` inside `CompileAndVerify`, must be generated such that they produce output in invariant culture. 3. Diagnostic tests use the `VerifyDiagnostics` helpers using `en-US` values for the messages. The implementation of `VerifyDiagnostics` will request the diagnostic message in the `en-US` culture. This helps ensure the compiler has no localization product issues by forcing a consistent culture no matter the current culture of the system. The largest source of friction that developers encounter when authoring tests is passing values to `Console.WriteLine` that have locale dependent representations. For example `double` and `decimal` values. ```csharp decimal d = 1.2; Console.WriteLine(d); ``` The above code will print `1.2` on `en-US` but `1,2` on `es-ES` machines. To fix this issue the value needs to be explicitly formatted with invariant culture. ```csharp using System.Globalization; ... decimal d = 1.2; Console.WriteLine(d.ToString(CultureInfo.InvariantCulture)); ``` This will consistently print `1.2`. ## Exceptions There are a few cases where it is very difficult to avoid locale dependent values. These include: 1. When a diagnostic includes a message from a thrown exception. 2. When a diagnostic includes a message from the underlying OS. 3. When a message is generated by a tool such as `msbuild`. In those cases the message is likely to be locale dependent. Tests that have these values should be marked is `[ConditionalFact(typeof(IsEnglishLocal))]` so they run on `en-US` machines only. ## Infrastructure ### Windows The Spanish Windows machines used in our infrastructure execute with `CurrentCulture` set to `es-ES` but `CurrentUICulture` set to `en-US`. This configuration means that roughly string formatting happens with `es-ES` but resource lookups happen with `en-US`. This setup doesn't fully test our code base so our test infrastructure will force the `CurrentUICulture` to be `CurrentCulture` when the two differ. This is done in the `TestBase` constructor, which all compiler test classes inherit from. When `CurrentUICulture` does not equal `CurrentCulture`, the constructor saves the original `CurrentUICulture` and sets `CurrentUICulture` to `CurrentCulture`. The original value is restored in `Dispose` so the change is scoped to the lifetime of each test instance. This ensures that on a Windows machine configured with `CurrentCulture = es-ES` and `CurrentUICulture = en-US`, resource string lookups also happen in `es-ES`, fully exercising the localization paths. ### Linux The Spanish Linux machines used in our infrastructure run on Ubuntu with the `LC_ALL` environment variable set to `es_ES.UTF-8`. On .NET running on Linux, `LC_ALL` determines both `CurrentCulture` and `CurrentUICulture`. This means that unlike the Windows configuration, both string formatting and resource lookups will use `es-ES`. Because both culture values are already the same, the test infrastructure does not need to force `CurrentUICulture` to `CurrentCulture` on these machines. --- ### Contributing Nullable Annotations (`docs/contributing/Nullable Annotations.md`) # Nullable Annotations This document describes how nullable annotations should be approached in the Roslyn code base. The default is to simply follow [the same guidance](https://github.com/dotnet/runtime/blob/main/docs/coding-guidelines/api-guidelines/nullability.md) as the [dotnet/runtime](https://github.com/dotnet/runtime) repository. This document serves to detail the places where the guidance differs for Roslyn and re-emphasize rules that come up frequently. ## Annotation Guidance - **DO** annotate all new code - **DO** use the null suppression operator in places where a method body has ensured a value cannot be `null` but the compiler is unable to track it. Example: ```cs struct SyntaxTrivia { void Example() { if (this.HasStructure) { Use(this.GetStructure()!); } } } ``` - **DO** use explicit validation such as `Debug.Assert` or `Contract.ThrowIfNull` to capture internal program invariants. ```cs void M1(SyntaxNode node) { Debug.Assert(node.Parent is object); ... M2(node.Parent); } ``` - **AVOID** annotating and refactoring code in the same commit. This can significantly increase the cost of reviewing. Prefer instead to separate annotations and significant refactorings into separate commits. Small changes, and modernizing style (change `default(SyntaxToken)` to `default`) are acceptable. - **DO NOT** be concerned with the validity of annotations after an object has been disposed or otherwise entered into an invalid state. For example manipulating a pooled object after it's been freed. This is a violation of the objects contract and annotations aren't meaningful here. - **DO NOT** be concerned with the validity of an enumerator's `Current` annotations when used before `MoveNext` has been called or after `MoveNext` has returned false. As with `Dispose`, such use is considered invalid, and we don't want to harm the correct consumption of `Current` by catering to incorrect consumption. - **DO** use the adjective `Require` when defining a member that exist only to provide a non-null returning version of a method that returns a nullable reference type. For example: `GetRequiredSemanticModel` model returns a non-null value from `GetSemanticModel` ## Breaking Change Guidance Roslyn is a large code base and it will take significant time for us to fully annotate the code base. This will likely take several releases for our larger libraries to complete. During this time those libraries will not consider nullable annotations to be a part of the compatibility bar. That is nullable annotations will change without an entry in the breaking change document, no compat review, etc ... Once the nullable annotations in a given library reach critical mass we will begin tracking nullable annotations as a part of our compatibility bar. The Roslyn team still reserves the right to change annotations if the language or underlying platforms we build on effectively require us to do so. However such changes will have a higher degree of scrutiny and will need approval from members of the API compat council. As specific libraries complete their nullable annotations then the annotations will be [tracked in API files](https://github.com/dotnet/roslyn-analyzers/pull/3125) and the library names will be added to this document as being complete. --- ### Contributing Powershell Guidelines (`docs/contributing/Powershell Guidelines.md`) # Powershell Guidelines Powershell is primarily used in this repo as the backbone of our infrastructure. As such Powershell scripts need to be portable, reliable and take advantage of stop on error approaches. The guidelines below are meant to push scripts to this mindset of execution. ## Coding Style 1. Opening braces always go at the end of an expression / statement. 1. Closing braces always occur on an otherwise empty line. 1. We use two space indentation. 1. Use Pascal casing for functions and where possible follow the Verb-Name convention. 1. Use Camel casing for all other identifier. 1. Use full command names instead of aliases. For example `Get-Content` vs. `gc`. Aliases can be overridden by the environment and hence are not considered portable. ## General Guidelines ### Script Template All Powershell scripts that execute in CI need to conform to the following template. ```powershell [CmdletBinding(PositionalBinding=$false)] param ( [switch]$ci = $false, [string]$configuration = "Debug") Set-StrictMode -version 2.0 $ErrorActionPreference="Stop" try { . (Join-Path $PSScriptRoot "build-utils.ps1") $prepareMachine = $ci # Body of Powershell script goes here ExitWithExitCode 0 } catch { Write-Host $_ Write-Host $_.Exception Write-Host $_.ScriptStackTrace ExitWithExitCode 1 } ``` The rationale for these parts are: - `Set-StrictMode`: forces Powershell into a stricter mode of interpretation and swaps out the "On Error Resume Next" approach for an "On Error Stop" model. Both of these help with our goals of reliability as it makes errors hard stops (unless specifically stated otherwise) - `$prepareMachine = $ci`: is necessary to ensure `ExitWithExitCode` properly exits all build processes in CI environments (conforming to arcade guidelines). - `ExitWithExitCode`: arcade standard for exiting a script that deals with process management - `PositionalBinding=$false`: alerts callers when they have incorrect parameters invoking a script ### Coding Guidelines Use `Exec-*` functions to execute programs or `dotnet` commands. This adds automatic error detection on invocation failure, incorrect parameters, etc ... ```powershell # DO NOT & msbuild /v:m /m Roslyn.slnx & dotnet build Roslyn.slnx # DO Exec-Command "msbuild" "/v:m /m Roslyn.slnx" Exec-DotNet "build Roslyn.slnx" ``` Scripts that have many executions of `dotnet` commands can store the `dotnet` command in a variable and use `Exec-Command` instead. Call `Test-LastExitCode` after invoking a powershell script to make sure failure is not ignored. ```powershell # DO NOT & eng/make-bootstrap.ps1 Write-Host "Done with Bootstrap" # DO & eng/make-bootstrap.ps1 Test-LastExitCode Write-Host "Done with Bootstrap" ``` Whenever comparing with `$null` always make sure to put `$null` on the left hand side of the operator. For non-collection types this doesn't really affect behavior. For collection types though having a collection on the left hand side changes the meaning of `-ne` and `-eq`. Instead of checking for `$null` it will instead compare collection contents. ``` powershell # DO NOT if ($e -ne $null) { ... } # DO if ($null -ne $e) { ... } ``` ### Powershell vs. pwsh The Roslyn infrastructure should use `powershell` for execution not `pwsh`. The general .NET infra still uses `Powershell` and calls into our scripts. Moving to `pwsh` in our scripts creates errors in source and unified build. Until that moves to `pwsh` our scripts need to stay on `Powershell`. `pwsh` is permitted in cases which are not part of our infrastructure. i.e., scripts that are not under the `eng/` folder, and not expected to be run by the build process. This is necessary in scenarios where the script needs to run cross-platform, such as: - VS Code helper scripts - AI skills ### Supporting .cmd file Consider adding a `.cmd` file that calls the `.ps1` file. This is to support users who are not running a powershell shell. It also helps with ensuring the script is run locally the same it is run in CI by controlling the powershell used to invoke the script. test.cmd ```cmd @echo off set PSMODULEPATH= powershell -noprofile -ExecutionPolicy Unrestricted -file "%~dp0\test.ps1" %* ``` Note: the `PSMODULEPATH` is cleared to ensure a pwsh shell [does not interfere][psmodulepath] with launching powershell. [psmodulepath]: https://github.com/PowerShell/PowerShell/discussions/24630 --- ### Contributing Testing For Interactive Readiness (`docs/contributing/Testing for Interactive readiness.md`) # Compiler debugging/EnC checklist 1. Sequence points are emitted correctly and corresponding syntax nodes are recognized as breakpoint spans by the IDE - Verify manually by launching VS with the new compiler bits and step through and place breakpoints on all syntax nodes that should allow breakpoint - Add regression compiler tests that check emitted IL with sequence points (`VerifyIL` with `sequencePoints`) - Add IDE test via `src\EditorFeatures\CSharpTest\EditAndContinue\BreakpointSpansTests.cs` (implementation `src\Features\CSharp\Portable\EditAndContinue\BreakpointSpans.cs`, which deals with mapping syntax to sequence points) 2. Sequence points in relationship with closure allocations and other hidden code (for any syntax that produces sequence points) - The debugger supports manually moving the current IP (instruction pointer) using “Set Next Statement Ctlr+Shift+F10” command. - The statement can be set to any sequence point in the current method. - Need to make sure that sequence points are emitted so that the right hidden code gets executed. - For example, the sequence point on an opening brace of a block that allocates a closure needs to precede the closure allocation instructions, so that when the IP is set to this sequence point the closure is properly allocated. ``` { // closure allocated here var x = 1; F(() => x); } ``` 3. Conditional branching has to use stloc/ldloc pattern in DEBUG build - Check the instructions and sequence points emitted for e.g. an if statement in debug build. Instead of straightforward brtrue/brfalse we allocate a temp local, store the result of the condition evaluation to that local, emit hidden sequence point, load the result from the local and then branch. This supports EnC function remapping. Only need to think about this when implementing a feature that emits conditional branches with conditions that may contain arbitrary expressions. I believe there are helpers in the lowering phase that emit conditions in general, so it should be done automatically as long as the right helpers are used. But something to be aware of. 4. Closure and lambda scopes (PDB info) - If new syntax is introduced that represents some kind of lambda (anonymous methods, local functions, LINQ queries, etc.) update helpers in `src\Compilers\CSharp\Portable\Syntax\LambdaUtilities.cs` accordingly - If a new scope is introduce that can declare variables that can be lifted into a closure - The bound node that represents the scope needs to be associated with syntax node recognized by helpers in `src\Compilers\CSharp\Portable\Syntax\LambdaUtilities.cs` (specifically `IsClosureScope`). - This requirement is enforced by an assertion in `SynthesizedClosureEnvironment` constructor. - Lambda and closure syntax offsets must be emitted to the PDB (encLambdaMap custom debug information) - The offset attribute of closure identifies the syntax node that's associated with the closure. This offset must be unique. 5. When a new symbol is introduced symbol matcher might need to be updated - Symbol matcher maps a symbol from one compilation to another. - Synthesized symbols like closures, state machines, anonymous types, lambdas, etc. also has to be mapped. - Impl: `src\Compilers\CSharp\Portable\Emitter\EditAndContinue\CSharpSymbolMatcher.cs` - Tests: `src\Compilers\CSharp\Test\Emit\Emit\EditAndContinue\SymbolMatcherTests.cs` 6. When a new syntax is introduced that may declare a user local variable or emits long lived synthesized variables (ie. state that survives between breakpoints, not just a temp within expression) - Validate that the variable slots can be mapped from new to previous compilation - This is implemented by `EncVariableSlotAllocator` using syntax offsets stored in PDB (`encLocalSlotMap` and `encLambdaMap` custom debug info). - The current mechanism might not be sufficient to support the mapping, in which case raise the issue with the IDE team to design additional PDB info to support the mapping. 7. Each new language feature should be covered by a test in Emit tests under Emit/EditAndContinue. - Some features might just need a single test others multiple tests depending on impact on EnC. - PDB tests validate these scopes. (look for `` in PDB XML). `LocalsTests.cs` EE tests also validate the scoping. 8. When a new syntax is introduced that may declare a scope for local variables the corresponding IL scopes need to be emitted correctly in the PDB - These are used by the EE to determine which variables are in scope. 9. Test debugging experience of the feature - Is useful info displayed in Watch window? - Can I evaluate expressions using this feature in Watch window? - Some features might require adding more custom PDB information to make the debug experience good (e.g. async, iterators, dynamic, etc). - Design experience improvement and custom PDB info with IDE team. --- ### Contributing Bootstrap Builds (`docs/contributing/bootstrap-builds.md`) # Bootstrap Builds The build correctness leg ensures that the latest code in dotnet/roslyn can be used to build dotnet/roslyn. Essentially our compiler can bootstrap itself. ## Why? The reason we go through so much effort is failures of this nature are incredibly hard to track down after the fact. The compiler is a complex tool and dotnet/roslyn is a large code base. It can take significant time to work out which of the ~300 commits since the last successful bootstrap is causing the current one to fail. In the past we used to run a bootstrap exercise roughly once a month. Every time there was a failure to build dotnet/roslyn with the latest compiler and it often took weeks for us to understand what change caused the failure. That resulted in a **lot** of wasted time and almost caused us to slip several releases. The build correctness jobs was introduced to solve this problem. The compiler verifies it can bootstrap on every change which means we find the failures immediately. ## Process The build correctness job works by first building the Microsoft.Net.Compilers.Toolset package. This gives us a functioning compiler with the latest changes. This build occurs using the `/define:BOOTSTRAP` which allows the compiler to make failures more actionable. This is primarily leveraged in the following ways: - Inserts a [ExitingTraceListener](https://github.com/dotnet/roslyn/blob/main/src/Compilers/Shared/ExitingTraceListener.cs) into the process trace listeners. This means any `Debug.Assert` failure will result in the compilation failing with an actionable stack trace. - Defines a [ValidateBootstrap](https://github.com/dotnet/roslyn/blob/main/src/Compilers/Core/MSBuildTask/ValidateBootstrap.cs). This lets us validate that the compiler used in the bootstrap build is actually the one we built vs. the default. This helps protect against build authoring changes which could inadvertently cause the default compiler to be used in a bootstrap build. The job then cleans out all of the artifacts from the build and starts a normal build of Roslyn.slnx but specifies `/p:BootstrapBuildPath=...`. This causes two files to be loaded: - [Bootstrap.props](https://github.com/dotnet/roslyn/blob/main/eng/targets/Bootstrap.props): loads the bootstrap compiler over the default one - [Bootstrap.targets](https://github.com/dotnet/roslyn/blob/main/eng/targets/Bootstrap.targets): verifies the bootstrap compiler was actually used This leg also ensures that the binary log and the build server log are captured in the set of published artifacts to allow for easy investigations. ## Investigating The first step for investigating a correctness build failure is downloading the log files. These are available in the published artifacts The two most interesting files are: 1. Build.Server.log: this is the text log of the compilation process. All stack traces and server error messages will be present in this file 1. Build.binlog: this is the binary log that results from building dotnet/roslyn with a bootstrap compiler. The build server log file will contain the reason why the particular request to the compiler failed. In most cases searching for one of two terms will take you straight to the failure. The first term to search for is `"Debug.Assert"` (no quotes). The most common cause of a bootstrap failure is a `Debug.Assert` call failing during compilation. This will result in an exception being added to the log with the full stack trace. For example: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` These failures are almost always due to changes being tested. Essentially the change updated compiler logic in such a way that it caused an `Assert` to fail. On occasion this will also fail because an IDE change introduces a coding pattern that sets of a latent bug in the compiler or analyzer but this is certainly the rare case. The next term to search for is `"Error "` (no quotes but keep the space). This will be added to the log every the server hits an error and needs to shut down. ```txt ID=MSBuild 60300 TID=3: Error Error: 'EndOfStreamException' 'Reached end of stream before end of read.' occurred during 'Reading response for d2c3aeac-bd8a-4251-bde0-2e11bbc57d13' Stack trace: at Microsoft.CodeAnalysis.CommandLine.BuildProtocolConstants.d__4.MoveNext() in C:\Users\jaredpar\code\wt\ros2\src\Compilers\Core\CommandLine\BuildProtocol.cs:line 641 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult() at Microsoft.CodeAnalysis.CommandLine.BuildResponse.d__5.MoveNext() in C:\Users\jaredpar\code\wt\ros2\src\Compilers\Core\CommandLine\BuildProtocol.cs:line 342 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Microsoft.CodeAnalysis.CommandLine.BuildServerConnection.<g__tryRunRequestAsync|7_1>d.MoveNext() in C:\Users\jaredpar\code\wt\ros2\src\Compilers\Shared\BuildServerConnection.cs:line 288 ``` These type of errors, when not paired with a `Debug.Assert` failure, are almost always bugs in the compiler server. Please contact the compiler team to help track down such failures. Note: when you encounter a case where the log file does not have an actionable description of why a build failed, strongly consider sending a PR that fixes this. This approach is why the log is such a valuable tool for tracking down bootstrap failures. ### Reproducing locally Locally reproducing and attaching a debugger to a bootstrap build failure is fairly simple with use of `src/Tools/Replay`, which can replay a binlog with the checked-out Roslyn compiler source. To do this, run a build locally with the `-bl` flag to get a binary log, then use replay to replay the binary log: **Windows** ```powershell .\build.ps1 -bl dotnet run --framework net10.0 --project src\Tools\Replay\ artifacts\log\Debug\Build.binlog -w ``` **Linux** ```sh ./build.sh -bl dotnet run --framework net10.0 --project src/Tools/Replay/ artifacts/log/Debug/Build.binlog -w ``` The `-w` flag will cause replay to print the process ID of the compiler and wait until you hit a key, so that you can attach a debugger to the process and set breakpoints for exceptions you are interested in. ## Debugging To debug a bootstrap build failure locally do the following. The first step is disabling the `ExitingTraceListener`. This is important for CI where the compiler needs to crash on a `Debug.Assert` failure vs. popping up a dialog that would hang CI. When debugging locally though developers want the `Debug.Assert` pops up a dialog behavior. To disable the `ExitingTraceListener` comment out the following line: https://github.com/dotnet/roslyn/blob/d73d31cbccb9aa850f3582afb464b709fef88fd7/src/Compilers/Server/VBCSCompiler/VBCSCompiler.cs#L22 Next just run the bootstrap build locally, wait for the `Debug.Assert` to trigger which pops up a dialog. From there you can attach to the VBCSCompiler process and debug through the problem ```cmd > Build.cmd -bootstrap ``` --- ### Contributing Codespaces (`docs/contributing/codespaces.md`) # Codespaces This repository provides a GitHub Codespaces configuration for full-repository Roslyn work. ## Available configuration | Configuration | Default solution | Intended usage | Minimum machine | | --- | --- | --- | --- | | `Roslyn (.NET 10)` | `Roslyn.slnx` | Full-repository work | 8 cores, 16 GB RAM, 32 GB storage | The configuration uses `.devcontainer/Dockerfile`, which starts from the .NET 10 SDK image and then installs the exact SDK pinned by `global.json`. ## Lifecycle choices The repo uses `.devcontainer/restore-workspace.sh ` in `updateContentCommand`. That placement is intentional: - `updateContentCommand` runs during Codespaces prebuild creation. - `postCreateCommand` does not run during prebuild creation. - Keeping restore in `updateContentCommand` allows GitHub Codespaces to cache the expensive first-run restore step. The configuration also installs `gh` and `pwsh` through dev container features so the container matches common Roslyn command-line workflows more closely. ## Recommended repository-level GitHub settings These settings live in the repository's **Settings > Codespaces** page rather than in the repo itself. 1. Enable a prebuild for the default branch using `.devcontainer/devcontainer.json`. 2. Start with the **On configuration change** trigger to keep storage and Actions usage under control. 3. Keep retained prebuild versions at `2` unless you have a clear need for a longer rollback window. 4. Limit prebuild regions to the regions your team actually uses. ## Validation checklist After changing Codespaces settings or the devcontainer configuration: 1. Create a fresh codespace from `.devcontainer/devcontainer.json`. 2. Confirm the expected default solution loads in VS Code. 3. Confirm `gh --version` and `pwsh --version` work in the terminal. 4. Confirm the restore step completes successfully for the selected solution. --- ### Contributing Ide Design Process (`docs/contributing/ide_design_process.md`) # IDE Team Design Process ## Discussions / Issues The goal is that contributors can create and discuss feature suggestions without needing to have everything fully figured out. Feel free to file a new [Github issue](https://github.com/dotnet/roslyn/issues/new?assignees=&labels=Area-IDE%2CFeature+Request&template=feature-suggestion.md) or start a [discussion](https://github.com/dotnet/roslyn/discussions/new). These conversations can continue at their own pace until contributors feel ready to make a more formal proposal. ## Proposals ### Adding a proposal For a feature proposal to be reviewed, a Github issue needs to be created that has the following items in its description. If these aren't there to start they eventually need to be appended as a comment to the issue before its brought up for discussion. - **Summary**: Brief summary of what this proposal is about. - **Background and Motivation**: What is the problem we are solving and in what context did you encounter it? - **Proposed Feature**: How the proposed solution solves the problem? - **Alternative Designs**: Were there other options you considered? How does this compare to features in other editors? Once you've added the details for your feature tag a member of the Roslyn-IDE team and they will add the "[Feature Request](https://github.com/dotnet/roslyn/issues?=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22Feature+Request%22)" label to your Github issue. ### Getting the issue championed by the IDE team A member of [Roslyn-IDE](https://github.com/orgs/dotnet/teams/roslyn-ide) will need to decide to "champion" that feature proposal and bring it for discussion by adding it to the "On Deck" tab in the [IDE Design Review](https://github.com/dotnet/roslyn/projects/40) project. Behind the scenes the IDE Team generally meets once a week to discuss these issues. --- ### Contributing Target Framework Strategy (`docs/contributing/target-framework-strategy.md`) # Target Framework Strategy ## Layers The roslyn repository produces components for a number of different products that push varying ship and TFM constraints on us. A summary of some of our dependencies are: - Build Tools: requires us to ship compilers on `net472` - .NET SDK: requires us to ship compilers on current servicing target framework (presently `net10.0`) - Repository Source build: requires us to ship `$(NetCurrent)` and `$(NetPrevious)` in workspaces and below (presently `net10.0` and `net9.0` respectively). This is because the output of repository source build is an input to other repository source build and those could be targeting either `$(NetCurrent)` or `$(NetPrevious)`. - Full Source build: requires us to ship `$(NetCurrent)` - Visual Studio: requires us to ship `net472` for base IDE components and `$(NetVisualStudio)` (presently `net10.0`) for private runtime components. - Visual Studio Code: expects us to ship against the same runtime as DevKit (presently `net10.0`) to avoid two runtime downloads. - MSBuildWorkspace: requires us to ship a process that must be usable on the lowest supported SDK (presently `net8.0`) - Microsoft.RoslynTools: remains compatible with the lowest runtime supported by Roslyn infrastructure automation (presently `net8.0`). It is not reasonable for us to take the union of all TFM and multi-target every single project to them. That would add several hundred compilations to any build operation which would in turn negatively impact our developer throughput. Instead we attempt to use the TFM where needed. That keeps our builds smaller but increases complexity a bit as we end up shipping a mix of TFM for binaries across our layers. ## Picking the right TargetFramework Projects in our repository should include the following values in `` based on the rules below: 1. `$(NetRoslynSourceBuild)`: code that needs to be part of source build. This property will change based on whether the code is building in a source build context or official builds. a. In official builds this will include the TFMs for `$(NetVSShared)` b. In source builds this will include `$(NetRoslyn)` 2. `$(NetVS)`: code that needs to execute on the private runtime of Visual Studio. 3. `$(NetVSCode)`: code that needs to execute in DevKit host 4. `$(NetVSShared)`: code that needs to execute in both Visual Studio and VS Code but does not need to be source built. 5. `$(NetRoslyn)`: code that needs to execute on .NET but does not have any specific product deployment requirements. For example utilities that are used by our infra, compiler unit tests, etc ... This property also controls which of the frameworks the compiler builds against are shipped in the toolset packages. This value will potentially change in source builds. 6. `$(NetRoslynAll)`: code, generally test utilities, that need to build for all .NET runtimes that we support. Note: `$(NetRoslynBuildHostNetCoreVersion)` is excluded from this set because the BuildHost is an isolated executable. 7. `$(NetRoslynWindowsTests)`: test projects that use `$(NetRoslyn)` but can only run on Windows. These tests will be automatically excluded when running on Unix/Linux because the `-windows` OS-specific TFM suffix tells the .NET SDK and the test runner to skip them on non-Windows platforms. 8. `$(NetRoslynBuildHostNetCoreVersion)`: the target used for the .NET Core BuildHost process used by MSBuildWorkspace. This may lag behind other properties as it must target the lowest supported .NET SDK (presently `net8.0` until .NET 8 EOL in November 2026). 9. `$(NetRoslynNext)`: code that needs to run on the next .NET Core version. This is used during the transition to a new .NET Core version where we need to move forward but don't want to hard code a .NET Core TFM into the build files. 10. `$(NetRoslynTools)`: the target used by the `Microsoft.RoslynTools` infrastructure tool. This may lag behind `$(NetRoslyn)` so build and insertion automation can run the tool on older supported machines. This properties `$(NetCurrent)`, `$(NetPrevious)` and `$(NetMinimum)` are not used in our project files because they change in ways that make it hard for us to maintain correct product deployments. Our product ships on VS and VS Code which are not captured by arcade `$(Net...)` macros. Further as the arcade properties change it's very easy for us to end up with duplicate entries in a `` setting. Instead our repo uses the above values and when inside source build or VMR our properties are initialized with arcade properties. **DO NOT** hard code .NET Core TFMs in project files. Instead use the properties above as that lets us centrally manage them and structure the properties to avoid duplication. It is fine to hard code other TFMs like `netstandard2.0` or `net472` as those are not expected to change. **DO NOT** use `$(NetCurrent)` or `$(NetPrevious)` in project files. These should only be used inside of `TargetFrameworks.props` to initialize the above values in certain configurations. ## Require consistent API across Target Frameworks It is important that our shipping APIs maintain consistent API surface area across target frameworks. That is true whether the API is `public` or `internal`. The reason for `public` is standard design pattern. The reason for `internal` is a combination of the following problems: - Our repository makes use of `InternalsVisibleTo` which allows other assemblies to directly reference signatures of `internal` members. - Our repository ships a mix of target frameworks. Typically workspaces and below will ship more recent TFMs than the layers above it. Compiler has to ship newer TFM for source build while IDE is constrained by Visual Studio's private runtime hence adopts newer TFM slower. - Our repository invests in polyfill APIs to make compiling against multiple TFMs in the same project a seamless experience. Taken together though this means that our `internal` surface area in many cases is effectively `public` when it comes to binary compatibility. For example a consuming project can end up with `net7.0` binaries from workspaces layer and `net6.0` binaries from IDE layer. Because there is `InternalsVisibleTo` between these binaries the `internal` API surface area is effectively `public`. This requires us to have a consistent strategy for achieving binary compatibility across TFM combinations. Consider a specific example of what goes wrong when our `internal` APIs are not consistent across TFM: - Workspaces today targets `net6.0` and `net7.0` and it contains our `EnumerableExtensions.cs` which polyfills many extensions methods on `IEnumerable`. In `net7.0` the `Order` extension method is not needed because it was put into the .NET core libraries. - Language Server Protocol targets `net6.0` and consumes the `Order` polyfill from Workspaces Let's assume for a second that we `#if` the `Order` method such that it's not present in `net7.0`. Locally this all builds because we compile the `net6.0` versions against each other so they're consistent. However if an external project which targets `net7.0` and consumes both Workspaces and Language Server Protocol then it will be in a broken state. The Protocol binary is expecting Workspaces to contain a polyfill method for `Order` but it does not since it's at `net7.0` and it was `#if` out. As a result this will fail at runtime with missing method exceptions. This problem primarily comes from our use of polyfill APIs. To avoid this we employ the following rule: > When there is a `#if` directive that matches the regex `#if !?NET.*` that declares a non-private member, there must be an `#else` that defines an equivalent binary compatible symbol This comes up in two forms: ### Pattern for types When creating a polyfill for a type use the `#if !NET...` to declare the type and in the `#else` use a `TypeForwardedTo` for the actual type. Example: ```csharp #if NET6_0_OR_GREATER using System.Runtime.CompilerServices; #pragma warning disable RS0016 // Add public types and members to the declared API (this is a supporting forwarder for an internal polyfill API) [assembly: TypeForwardedTo(typeof(IsExternalInit))] #pragma warning restore RS0016 // Add public types and members to the declared API #else using System.ComponentModel; namespace System.Runtime.CompilerServices { internal static class IsExternalInit { } } #endif ``` ### Pattern for extension methods When creating a polyfill for an extension use the `#if NET...` to declare the extension method and the `#else` to declare the same method without `this`. That will put a method with the expected signature in the binary but avoids it appearing as an extension method within that target framework. ```csharp #if NET7_0_OR_GREATER public static IOrderedEnumerable Order(IEnumerable source) where T : IComparable #else public static IOrderedEnumerable Order(this IEnumerable source) where T : IComparable #endif ``` ## Transitioning to new .NET SDK As the .NET team approaches releasing a new .NET SDK the Roslyn team will begin using preview versions of that SDK in our build. This will often lead to test failures in our CI system due to underlying behavior changes in the runtime. These failures will often not show up when running in Visual Studio due to the way the runtime for the test runner is chosen. To ensure we have a simple developer environment such project should be moved to to the `$(NetRoslynNext)` target framework. That ensures the new runtime is loaded when running tests locally. When the .NET SDK RTMs and Roslyn adopts it all occurrences of `$(NetRoslynNext)` will be moved to simply `$(NetRoslyn)`. **DO NOT** include both `$(NetRoslyn)` and `$(NetRoslynNext)` in the same project unless there is a very specific reason that both tests are adding value. The most common case is that the runtime has changed behavior and we simply need to update our baselines to match it. Adding extra TFMs for this just increases test time for very little gain. ## Checklist for updating TFMs (once a year) - Update `TargetFrameworks.props`. - Ensure we have the correct TFM for VS / VSCode (usually not the latest TFM). - Might need updating `MicrosoftNetCompilersToolsetVersion` in `eng\Versions.props` to consume latest compiler features. - Change `$(NetRoslynNext)` references to `$(NetRoslyn)` in project files. - Update TFMs in code/scripts/pipelines (search for the old `netX.0` and replace with the new `netY.0`). - Replace `#if NET7_0_OR_GREATER` corresponding to the no longer used TFM (`net7.0` in this example) with the lowest used TFM (e.g., `net8.0`: `#if NET8_0_OR_GREATER`) and similarly the negated variant `#if !NET7_0_OR_GREATER` with `#if !NET8_0_OR_GREATER`. See https://github.com/dotnet/roslyn/issues/75453#issuecomment-2405500584. - Check that the same number of tests still run in CI (they are not unintentionally filtered out by TFM). - Try an official build, a VS insertion. --- ### Features CheckedUserDefinedOperators (`docs/features/CheckedUserDefinedOperators.md`) Checked user-defined operators ===================================== C# should support defining `checked` variants of the following user-defined operators so that users can opt into or out of overflow behavior as appropriate: * The `++` and `--` unary operators (https://github.com/dotnet/csharplang/blob/main/spec/expressions.md#postfix-increment-and-decrement-operators and https://github.com/dotnet/csharplang/blob/main/spec/expressions.md#prefix-increment-and-decrement-operators). * The `-` unary operator (https://github.com/dotnet/csharplang/blob/main/spec/expressions.md#unary-minus-operator). * The `+`, `-`, `*`, and `/` binary operators (https://github.com/dotnet/csharplang/blob/main/spec/expressions.md#arithmetic-operators). * Explicit conversion operators. Proposal: - https://github.com/dotnet/csharplang/issues/4665 - https://github.com/dotnet/csharplang/blob/main/proposals/checked-user-defined-operators.md Feature branch: https://github.com/dotnet/roslyn/tree/features/CheckedUserDefinedOperators Test plan: https://github.com/dotnet/roslyn/issues/59458 --- ### Features DefaultInterfaceImplementation (`docs/features/DefaultInterfaceImplementation.md`) Default Interface Implementation ========================= The *Default Interface Implementation* feature enables a default implementation of an interface member to be provided as part of the interface declaration. Here is a link to the proposal https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/default-interface-methods.md. **What is supported:** - Supplying an implementation along with declaration of a regular interface method and recognizing that implementation as default implementation for the method when a type implements the interface. Here is an example: ``` public interface I1 { void M1() { System.Console.WriteLine("Default implementation of M1 is called!!!"); } } class Test1 : I1 { static void Main() { I1 x = new Test1(); x.M1(); } } ``` - Re-abstraction of interface implementation in derived interfaces. Types implementing the derived interface are required to supply implementation even when a base interface provides an implementation. Here is an example: ``` public interface I1 { void M1() { } int P1 { get => throw null; set => throw null; } event System.Action E1 { add => throw null; remove => throw null; } } public interface I2 : I1 { abstract void I1.M1(); abstract int I1.P1 {get;set;} abstract event System.Action I1.E1; } class Test1 : I2 // This type must implement all members of I1 { } ``` In metadata, methods representing re-abstraction have all three flags `abstract`, `virtual`, `sealed` set and do not have `newslot` flag set. - Supplying an implementation along with declaration of a property or an indexer and recognizing that implementation as default implementation for them when a type implements the interface. - Supplying an implementation along with declaration of an event and recognizing that implementation as default implementation for the event when a type implements the interface. - Using **partial**, **public**, **internal**, **private**, **protected**, **static**, **virtual**, **sealed**, **abstract**, **extern** and **async** modifiers with interface methods. - Using **public**, **internal**, **private**, **protected**, **static**, **virtual**, **sealed**, **abstract** and **extern** modifiers with interface properties. - Using **public**, **internal**, **private**, **protected**, **virtual**, **sealed**, **abstract** and **extern** modifiers with interface indexers. - Using **internal**, **private**, **protected** modifiers with interface property/indexer accessors. - Using **public**, **internal**, **private**, **protected**, **static**, **virtual**, **sealed**, **abstract** and **extern** modifiers with interface events. - Declaring types within interfaces. - Implementing interface methods in derived interfaces by using explicit implementation syntax, accessibility is **protected**, allowed modifiers: **extern** and **async**. - Implementing interface properties and indexers in derived interfaces by using explicit implementation syntax, accessibility **protected**, allowed modifiers: **extern**. - Implementing interface events in derived interfaces by using explicit implementation syntax, accessibility **protected**, no allowed modifiers. - Declaring static fields, auto-properties and field-like events. - Declaring operators ```+ - ! ~ ++ -- true false * / % & | ^ << >> > < >= <=``` in interfaces. - Base access The following forms of base-access are added (https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-11-14.md) ``` base ( ) . identifier base ( ) [ argument-list ] ``` The type-syntax can refer to one of the base classes of the containing type, or one of the interfaces implemented or inherited by the containing type. When the type-syntax refers to a class, the member lookup rules, overload resolution rules and IL emit match the rules for the 7.3 supported forms of base-access. The difference is that the specified base class is used instead of the immediate base class. The most derived implementation found must be a member of that class. When the type-syntax refers to an interface: 1. The member lookup is performed in that interface, using the regular member lookup rules within interfaces, with an exception that members of System.Object do not participate in the lookup. 2. Regular overload resolution is performed for members returned by the lookup process, virtual or abstract members are not replaced with most derived implementations at this step (unlike the case when the type-syntax refers to a class). If result of overload resolution is a virtual or abstract method, it must have an implementation within the specified interface type, an error is reported otherwise. That implementation must be accessible at the call site. If result of overload resolution is a non-virtual method, the method must be declared in the specified interface type. 3. During IL emit a **call** (non-virtual call) instruction is used to invoke methods. If result of overload resolution on the previous step is a virtual or abstract method, the implementation of the method from the specified interface is used as the target for the instruction. Given the accessibility requirements for the most specific interface implementation, accessibility of implementations provided in derived interfaces is changed to **protected**. **Open issues and work items** are tracked in https://github.com/dotnet/roslyn/issues/17952. **Parts of ECMA-335 that become obsolete/inaccurate/incomplete** >I.8.5.3.2 Accessibility of members and nested types Members (other than nested types) defined by an interface shall be public. I.8.9.4 Interface type definition Similarly, an interface type definition shall not provide implementations for any methods on the values of its type. Interfaces can have static or virtual methods, but shall not have instance methods. However, since accessibility attributes are relative to the implementing type rather than the interface itself, all members of an interface shall have public accessibility, ... I.8.11.1 Method definitions All non-static methods of an interface definition are abstract. All non-static method definitions in interface definitions shall be virtual methods. II.10.4 Method implementat ion requirements II.12 Semantics of interfaces Interfaces can have static fields and methods, but they shall not have instance fields or methods. Interfaces can define virtual methods, but only if those methods are abstract (see Partition I and §II.15.4.2.4). II.12.2 Implement ing virtual methods on interfaces If the class defines any public virtual methods whose name and signature match a virtual method on the interface, then add these to the list for that method, in type declaration order (see above). If there are any public virtual methods available on this class (directly or inherited) having the same name and signature as the interface method, and whose generic type parameters do not exactly match any methods in the existing list for that interface method for this class or any class in its inheritance chain, then add them (in type declaration order) to the list for the corresponding methods on the interface. II.15.2 Static, instance, and virtual methods It follows that instance methods shall only be defined in classes or value types, but not in interfaces or outside of a type (i.e., globally). II.22.27 MethodImpl : 0x19 The method indexed by MethodBody shall be a member of Class or some base class of Class (MethodImpls do not allow compilers to ‘hook’ arbitrary method bodies) II.22.37 TypeDef : 0x02 All of the methods owned by an Interface (Flags.Interface = 1) shall be abstract (Flags.Abstract = 1) IV.6 Implementation-specific modifications to the system libraries Interfaces and virtual methods shall not be added to an existing interface. --- ### Features ExperimentalAttribute (`docs/features/ExperimentalAttribute.md`) ExperimentalAttribute ===================== Report warnings for references to types and members marked with `Windows.Foundation.Metadata.ExperimentalAttribute`. ``` namespace Windows.Foundation.Metadata { [AttributeUsage( AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Enum | AttributeTargets.Interface | AttributeTargets.Delegate, AllowMultiple = false)] public sealed class ExperimentalAttribute : Attribute { } } ``` ## Warnings The warning message is a specific message, where `'{0}'` is the fully-qualified type or member name. ``` '{0}' is for evaluation purposes only and is subject to change or removal in future updates. ``` The warning is reported for any reference to a type or member marked with the attribute. (The `AttributeUsage` is for types only so the attribute cannot be applied to non-type members from C# or VB, but the attribute could be applied to non-type members outside of C# or VB.) ``` [Experimental] class A { internal object F; internal static object G; internal class B { } } class C { static void Main() { F(default(A)); // warning CS08305: 'A' is for evaluation purposes only ... F(typeof(A)); // warning CS08305: 'A' is for evaluation purposes only ... F(nameof(A)); // warning CS08305: 'A' is for evaluation purposes only ... var a = new A(); // warning CS08305: 'A' is for evaluation purposes only ... F(a.F); F(A.G); // warning CS08305: 'A' is for evaluation purposes only ... F(null); // warning CS08305: 'A' is for evaluation purposes only ... } static void F(T t) { } } ``` The attribute is not inherited from base types or overridden members. ``` [Experimental] class A { [Experimental] internal virtual void F() { } } class B : A // warning CS08305: 'A' is for evaluation purposes only ... { internal override void F() { base.F(); // warning CS08305: 'A.F' is for evaluation purposes only ... } } class C { static void F(A a, B b) // warning CS08305: 'A' is for evaluation purposes only ... { a.F(); // warning CS08305: 'A.F' is for evaluation purposes only ... b.F(); } } ``` There is no automatic suppression of warnings: warnings are generated for all references, even within `[Experimental]` members. Suppressing the warning requires an explicit compiler option or `#pragma`. ``` [Experimental] enum E { } [Experimental] class C { private C(E e) // warning CS08305: 'E' is for evaluation purposes only ... { } internal static C Create() // warning CS08305: 'C' is for evaluation purposes only ... { return Create(0); } #pragma warning disable 8305 internal static C Create(E e) { return new C(e); } } ``` ## ObsoleteAttribute and DeprecatedAttribute `ExperimentalAttribute` is independent of `System.ObsoleteAttribute` or `Windows.Framework.Metadata.DeprecatedAttribute`. Warnings for `[Experimental]` are reported within `[Obsolete]` or `[Deprecated]` members. Warnings and errors for `[Obsolete]` and `[Deprecated]` are reported inside `[Experimental]` members. ``` [Obsolete] class A { static object F() => new C(); // warning CS08305: 'C' is for evaluation purposes only ... } [Deprecated(null, DeprecationType.Deprecate, 0)] class B { static object F() => new C(); // warning CS08305: 'C' is for evaluation purposes only ... } [Experimental] class C { static object F() => new B(); // warning CS0612: 'B' is obsolete } ``` Warnings and errors for `[Obsolete]` and `[Deprecated]` are reported instead of `[Experimental]` if there are multiple attributes. ``` [Obsolete] [Experimental] class A { } [Experimental] [Deprecated(null, DeprecationType.Deprecate, 0)] class B { } class C { static A F() => null; // warning CS0612: 'A' is obsolete static B G() => null; // warning CS0612: 'B' is obsolete } ``` --- ### Features ExpressionVariables (`docs/features/ExpressionVariables.md`) Expression Variables ========================= The *expression variables* feature extends the features introduced in C# 7 to permit expressions containing expression variables (out variable declarations and declaration patterns) in field initializers, property initializers, ctor-initializers, and query clauses. See https://github.com/dotnet/csharplang/issues/32 and https://github.com/dotnet/csharplang/blob/main/proposals/csharp-7.3/expression-variables-in-initializers.md for more information. Current state of the feature: [X] Permit in field initializers [X] Permit in property initializers [ ] Permit in ctor-initializers [X] Permit in query clauses --- ### Features ExtendedLayoutAttribute (`docs/features/ExtendedLayoutAttribute.md`) System.Runtime.InteropServices.ExtendedLayoutAttribute Compiler Support ======================================================================= The .NET runtime team is introducing a new attribute, `System.Runtime.InteropServices.ExtendedLayoutAttribute`, which will allow the runtime team to provide additional layout options for types, primarily for interop scenarios. To provide the user experience requested by the runtime team, the C# and VB compilers will have the following support for this attribute: - If a type has the `ExtendedLayoutAttribute` applied, the compiler will emit the `TypeAttributes.ExtendedLayout` value in the type's `TypeAttributes` flags. - If a type has the `ExtendedLayoutAttribute` applied, the compiler will not allow the `StructLayoutAttribute` to be applied to the type. - (C# compiler only) If a type has the `ExtendedLayoutAttribute` applied, the compiler will not allow the `InlineArrayAttribute` to be applied to the type. - Within the Roslyn compiler, the `ITypeSymbol` for a type with the `ExtendedLayoutAttribute` applied will have the following behavior: - The `Layout` property will return a `TypeLayout` instance with the `LayoutKind` set to `Extended` (`1`), `Size` set to `0` and `Pack` set to `0`. - A type that is embedded in an assembly (using the `NoPia` technology) will have the `ExtendedLayoutAttribute` preserved on the embedded type. The Roslyn compiler will not have knowledge of the specific options available on the `ExtendedLayoutAttribute`, as these options will be defined by the runtime team and may expand over time. The compiler will not attempt to detect invalid field types for specific layout options. The runtime team may add analyzers in the future to handle these scenarios. The compiler will only recognize the presence of the attribute and its implications for the type symbol's `LayoutKind` and the emitted attribute. --- ### Features GlobalUsingDirective (`docs/features/GlobalUsingDirective.md`) Global Using Directive ========================= The *Global Using Directive* feature extends using directive syntax with an optional `global` keyword that can precede the `using` keyword. The scope of Global Using Directives spans across all compilation units in the program. Proposal: https://github.com/dotnet/csharplang/blob/master/proposals/GlobalUsingDirective.md Feature branch: https://github.com/dotnet/roslyn/tree/features/GlobalUsingDirective Test plan: https://github.com/dotnet/roslyn/issues/51307 --- ### Features NativeIntegerAttribute (`docs/features/NativeIntegerAttribute.md`) Native Integer Metadata ========= The following describes the representation of native integers in metadata. ## NativeIntegerAttribute Type references are annotated in metadata with a `NativeIntegerAttribute`. ```C# namespace System.Runtime.CompilerServices { [AttributeUsage( AttributeTargets.Class | AttributeTargets.Event | AttributeTargets.Field | AttributeTargets.GenericParameter | AttributeTargets.Parameter | AttributeTargets.Property | AttributeTargets.ReturnValue, AllowMultiple = false, Inherited = false)] public sealed class NativeIntegerAttribute : Attribute { public NativeIntegerAttribute() { TransformFlags = new[] { true }; } public NativeIntegerAttribute(bool[] flags) { TransformFlags = flags; } public readonly bool[] TransformFlags; } } ``` The `NativeIntegerAttribute` type is for compiler use only - it is not permitted in source. The type declaration is synthesized by the compiler if not already included in the compilation. Each type reference in metadata may have an associated `NativeIntegerAttribute` with a `bool[]` where each `bool` represents whether the associated type is a native integer (`nint` or `nuint` in C#) or the underlying type (`System.IntPtr` or `System.UIntPtr`). Custom modifiers associated with the type reference are ignored. A type reference may have at most one `NativeIntegerAttribute`. The attribute is not inherited. The `bool[]` is constructed as follows: - `System.IntPtr`, `System.UIntPtr`: `true` if the type represents a native integer; `false` otherwise - Generic type: the representation of the type arguments in order including containing types - Tuple: the representation of the underlying constructed type - Array: the representation of the element type - Pointer type: the representation of the pointed at type - Function pointer type: the representation of the return type followed by the parameter types in order - Other: none Given the construction above, the number of elements in the `bool[]` matches the number of references to `System.IntPtr` and `System.UIntPtr` in the type. ## Optimizations If the `bool[]` is empty, the `NativeIntegerAttribute` is omitted. If the `bool[]` contains a single `true` value, the `NativeIntegerAttribute` uses the parameter-less constructor. ## Examples ```C# System.IntPtr A; // System.IntPtr A nint B; // [NativeInteger] System.IntPtr B IEnumerable C // [NativeInteger] System.Collections.Generic.IEnumerable C (System.IntPtr, nint) D; // [NativeInteger(new[] { false, true })] System.ValueTuple D ``` --- ### Features StaticAbstractMembersInInterfaces (`docs/features/StaticAbstractMembersInInterfaces.md`) Static Abstract Members In Interfaces ===================================== An interface is allowed to specify abstract static members that implementing classes and structs are then required to provide an explicit or implicit implementation of. The members can be accessed off of type parameters that are constrained by the interface. Proposal: - https://github.com/dotnet/csharplang/issues/4436 - https://github.com/dotnet/csharplang/blob/main/proposals/static-abstracts-in-interfaces.md Feature branch: https://github.com/dotnet/roslyn/tree/features/StaticAbstractMembersInInterfaces Test plan: https://github.com/dotnet/roslyn/issues/52221 --- ### Features TopLevelStatements (`docs/features/TopLevelStatements.md`) Top-level statements ========================= Allow a sequence of *statements* to occur right before the *namespace_member_declaration*s of a *compilation_unit* (i.e. source file). The semantics are that if such a sequence of *statements* is present, the following type declaration, modulo the actual type name and the method name, would be emitted: ``` c# static class Program { static async Task Main() { // statements } } ``` Proposal: https://github.com/dotnet/csharplang/blob/main/proposals/top-level-statements.md Open issues and TODOs are tracked at https://github.com/dotnet/roslyn/issues/41704. Test plan: https://github.com/dotnet/roslyn/issues/43563. See also https://github.com/dotnet/csharplang/issues/3117. --- ### Features UsedAssemblyReferences (`docs/features/UsedAssemblyReferences.md`) Used Assembly References ========================= The *Used Assembly References* feature provides a ```GetUsedAssemblyReferences``` API on a ```Compilation``` to obtain a set of metadata assembly references that are considered to be used by the compilation. For example, if a type declared in a referenced assembly is referenced in source code within the compilation, the reference is considered to be used. Etc. See https://github.com/dotnet/roslyn/issues/37768 for more information. --- ### Features Async Main (`docs/features/async-main.md`) # Async Task Main ## [dotnet/csharplang proposal](https://github.com/dotnet/csharplang/blob/main/proposals/async-main.md) ## Technical Details * The compiler must recognize `Task` and `Task` as valid entrypoint return types in addition to `void` and `int`. * The compiler must allow `async` to be placed on a main method that returns a `Task` or a `Task` (but not void). * The compiler must generate a shim method `$EntrypointMain` that mimics the arguments of the user-defined main. * `static async Task Main(...)` -> `static void $EntrypointMain(...)` * `static async Task Main(...)` -> `static int $EntrypointMain(...)` * The parameters between the user-defined main and the generated main should match exactly. * The body of the generated main should be `return Main(args...).GetAwaiter().GetResult();` --- ### Features Async Main.Test (`docs/features/async-main.test.md`) Main areas to test: * Signature acceptance / rejection * Method body creation # Signature acceptance / rejection ## Single Mains Classes that contain only a single "main" method ### Single legal main * Past: Ok * New 7: Ok * New 7.1 Ok ### Single async (void) main * Past: ERR (Async can't be main) * New 7: ERR (Update to get this to work) * New 7.1: Ok ### Single async (Task) main * Past: ERR (No entrypoints found), WARN (has the wrong signature to be an entry point) * New 7: ERR (Update to get this to work) * New 7.1 Ok ### Single async (Task) main * Past: ERR (No entrypoints found), WARN (has the wrong signature) * New 7: ERR (Update to get this to work) * New 7.1: Ok ## Multiple Mains Classes that contain more than one main ### Multiple legal mains * Past: Err * New 7: Err * New 7.1: Err ### Single legal main, single async (void) main * Past: Err (an entrypoint cannot be marked with async) * New 7: Err (new error here?) * New 7.1: Err (new error here? "void is not an acceptable async return type") ### Single legal main, single legal Task main * Past: Ok (warning: task main has wrong signature) * New 7: Ok (new warning here) * New 7.1: Ok (new warning here?) ### Single legal main, single legal Task main * Past: Ok (warning: task main has wrong signature) * New 7: Ok (new warning here) * New 7.1: Ok (new warning here?) # Method body creation * Inspect IL for correct codegen. * Make sure that attributes are correctly applied to the synthesized mains. ## Broken methods on Task and Task * GetAwatier or GetResult are missing * GetAwaiter or GetResult don't have the required signature * Has the right shape, but types are missing ## Task or Task is missing This will be caught during entrypoint detection, should be a binding error. ## Void or int is missing If Task can be found, but void or int can't be found, then the compiler should behave gracefully. # Vlad Test Plan ## Public interface of compiler APIs -N/A? ## General functionality - `Task` and `Task` returns are allowed. Do we require `async`. - Multiple valid/invalid async candidates. - process exit code on success (void and int) and on exception - `/main:` cmd switch is still functional ## Old versions, compat - langver - when both async and regular Main avaialble. With old/new compiler versions. - async void Main. With old.new langver. With another applicable Main present. ## Type and members - Access modifiers (public, protected, internal, protected internal, private), static modifier - Parameter modifiers (ref, out, params) - STA attribute - Partial method - Named and optional parameters - `Task` - Task-Like types ## Debug - F11 works - stepping through Main --- ### Features Async Streams (`docs/features/async-streams.md`) async-streams (C# 8.0) ---------------------- Async-streams are asynchronous variants of enumerables, where getting the next element may involve an asynchronous operation. They are types that implement `IAsyncEnumerable`. ```csharp // Those interfaces will ship as part of .NET Core 3 namespace System.Collections.Generic { public interface IAsyncEnumerable { IAsyncEnumerator GetAsyncEnumerator(CancellationToken token = default); } public interface IAsyncEnumerator : System.IAsyncDisposable { System.Threading.Tasks.ValueTask MoveNextAsync(); T Current { get; } } } namespace System { public interface IAsyncDisposable { System.Threading.Tasks.ValueTask DisposeAsync(); } } ``` When you have an async-stream, you can enumerate its items using an asynchronous `foreach` statement: `await foreach (var item in asyncStream) { ... }`. An `await foreach` statement is just like a `foreach` statement, but it uses `IAsyncEnumerable` instead of `IEnumerable`, each iteration evaluates an `await MoveNextAsync()`, and the disposable of the enumerator is asynchronous. Similarly, if you have an async-disposable, you can use and dispose it with asynchronous `using` statement: `await using (var resource = asyncDisposable) { ... }` An `await using` statement is just like a `using` statement, but it uses `IAsyncDisposable` instead of `IDisposable`, and `await DisposeAsync()` instead of `Dispose()`. The user can implement those interfaces manually, or can take advantage of the compiler generating a state-machine from a user-defined method (called an "async-iterator" method). An async-iterator method is a method that: 1. is declared `async`, 2. returns an `IAsyncEnumerable` or `IAsyncEnumerator` type, 3. uses both `await` syntax (`await` expression, `await foreach` or `await using` statements) and `yield` statements (`yield return`, `yield break`). For example: ```csharp async IAsyncEnumerable GetValuesFromServer() { while (true) { IEnumerable batch = await GetNextBatch(); if (batch == null) yield break; foreach (int item in batch) { yield return item; } } } ``` Just like in iterator methods, there are several restrictions on where a yield statement can appear in async-iterator methods: - It is a compile-time error for a `yield` statement (of either form) to appear in the `finally` clause of a `try` statement. - It is a compile-time error for a `yield return` statement to appear anywhere in a `try` statement that contains any `catch` clauses. ### Detailed design for `await using` statement An asynchronous `using` is lowered just like a regular `using`, except that `Dispose()` is replaced with `await DisposeAsync()`. Note that pattern-based lookup for `DisposeAsync` binds to instance methods that can be invoked without arguments. Extension methods do not contribute. The result of `DisposeAsync` must be awaitable. ### Detailed design for `await foreach` statement An `await foreach` is lowered just like a regular `foreach`, except that: - `GetEnumerator()` is replaced with `await GetAsyncEnumerator()` - `MoveNext()` is replaced with `await MoveNextAsync()` - `Dispose()` is replaced with `await DisposeAsync()` Note that pattern-based lookup for `GetAsyncEnumerator`, `MoveNextAsync` and `DisposeAsync` binds to instance methods that can be invoked without arguments. Extension methods do not contribute. The result of `MoveNextAsync` and `DisposeAsync` must be awaitable. Disposal for `await foreach` does not include a fallback to a runtime check for an interface implementation. Asynchronous foreach loops are disallowed on collections of type dynamic, as there is no asynchronous equivalent of the non-generic `IEnumerable` interface. But wrapper types can pass non-default values (see `.WithCancellation(CancellationToken)` extension method), thereby allowing consumers of async-streams to control cancellation. A producer of async-streams can make use of the cancellation token by writing an `IAsyncEnumerator GetAsyncEnumerator(CancellationToken)` async-iterator method in a custom type. ```csharp E e = ((C)(x)).GetAsyncEnumerator(default); try { while (await e.MoveNextAsync()) { V v = (V)(T)e.Current; -OR- (D1 d1, ...) = (V)(T)e.Current; // body } } finally { await e.DisposeAsync(); } ``` ### Detailed design for async-iterator methods An async-iterator method is replaced by a kick-off method, which initializes a state machine. It does not start running the state machine (unlike kick-off methods for regular async method). The kick-off method method is marked with `AsyncIteratorStateMachineAttribute`. The state machine for an enumerable async-iterator method primarily implements `IAsyncEnumerable` and `IAsyncEnumerator`. For an enumerator async-iterator, it only implements `IAsyncEnumerator`. It is similar to a state machine produced for an async method. It contains builder and awaiter fields, used to run the state machine in the background (when an `await` is reached in the async-iterator). It also captures parameter values (if any) or `this` (if needed). But it contains additional state: - a promise of a value-or-end, - a current yielded value of type `T`, - an `int` capturing the id of the thread that created it, - a `bool` flag indicating "dispose mode", - a `CancellationTokenSource` for combining tokens (in enumerables). The central method of the state machine is `MoveNext()`. It gets run by `MoveNextAsync()`, or as a background continuation initiated from these from an `await` in the method. The promise of a value-or-end is returned from `MoveNextAsync`. It can be fulfilled with either: - `true` (when a value becomes available following background execution of the state machine), - `false` (if the end is reached), - an exception. The promise is implemented as a `ManualResetValueTaskSourceCore` (which is a re-usable and allocation-free way of producing and fulfilling `ValueTask` or `ValueTask` instances) and its surrounding interfaces on the state machine: `IValueTaskSource` and `IValueTaskSource`. See more details about those types at https://blogs.msdn.microsoft.com/dotnet/2018/11/07/understanding-the-whys-whats-and-whens-of-valuetask/ Compared to the state machine for a regular async method, the `MoveNext()` for an async-iterator method adds logic: - to support handling a `yield return` statement, which saves the current value and fulfills the promise with result `true`, - to support handling a `yield break` statement, which sets the dispose mode on and jumps to the enclosing `finally` or exit, - to dispatch execution to `finally` blocks (when disposing), - to exit the method, which disposes the `CancellationTokenSource` (if any) and fulfills the promise with result `false`, - to catch exceptions, which disposes the `CancellationTokenSource` (if any) and sets the exception into the promise. (The handling of an `await` is unchanged) This is reflected in the implementation, which extends the lowering machinery for async methods to: 1. handle `yield return` and `yield break` statements (see methods `VisitYieldReturnStatement` and `VisitYieldBreakStatement` to `AsyncIteratorMethodToStateMachineRewriter`), 2. handle `try` statements (see methods `VisitTryStatement` and `VisitExtractedFinallyBlock` in `AsyncIteratorMethodToStateMachineRewriter`) 3. produce additional state and logic for the promise itself (see `AsyncIteratorRewriter`, which produces various other members: `MoveNextAsync`, `Current`, `DisposeAsync`, and some members supporting the resettable `ValueTask` behavior, namely `GetResult`, `SetStatus`, `OnCompleted`). ```csharp ValueTask MoveNextAsync() { if (state == StateMachineStates.FinishedStateMachine) { return default(ValueTask); } valueOrEndPromise.Reset(); var inst = this; builder.Start(ref inst); var version = valueOrEndPromise.Version; if (valueOrEndPromise.GetStatus(version) == ValueTaskSourceStatus.Succeeded) { return new ValueTask(valueOrEndPromise.GetResult(version)); } return new ValueTask(this, version); // note this leverages the state machine's implementation of IValueTaskSource } ``` ```csharp T Current => current; ``` The kick-off method and the initialization of the state machine for an async-iterator method follows those for regular iterator methods. In particular, the synthesized `GetAsyncEnumerator()` method is like `GetEnumerator()` except that it sets the initial state to to StateMachineStates.NotStartedStateMachine (-1): ```csharp IAsyncEnumerator GetAsyncEnumerator(CancellationToken token) { {StateMachineType} result; if (initialThreadId == /*managedThreadId*/ && state == StateMachineStates.FinishedStateMachine) { state = InitialState; // -3 builder = AsyncIteratorMethodBuilder.Create(); disposeMode = false; result = this; } else { result = new {StateMachineType}(InitialState); } /* copy each parameter proxy, or in the case of the parameter marked with [EnumeratorCancellation] combine it with `GetAsyncEnumerator`'s `token` parameter */ } ``` For the parameter with `[EnumeratorCancellation]`, `GetAsyncEnumerator` initializes it by combining the two available tokens: ```csharp if (this.parameterProxy.Equals(default)) { result.parameter = token; } else if (token.Equals(this.parameterProxy) || token.Equals(default)) { result.parameter = this.parameterProxy; } else { result.combinedTokens = CancellationTokenSource.CreateLinkedTokenSource(this.parameterProxy, token); result.parameter = combinedTokens.Token; } ``` For a discussion of the threadID check, see https://github.com/dotnet/corefx/issues/3481 Similarly, the kick-off method is much like those of regular iterator methods: ```csharp { {StateMachineType} result = new {StateMachineType}(StateMachineStates.FinishedStateMachine); // -2 /* save parameters into parameter proxies */ return result; } ``` #### Disposal Iterator and async-iterator methods need disposal because their execution steps are controlled by the caller, which could choose to dispose the enumerator before getting all of its elements. For example, `foreach (...) { if (...) break; }`. In contrast, async methods continue running autonomously until they are done. They are never left suspended in the middle of execution from the caller's perspective, so they don't need to be disposed. In summary, disposal of an async-iterator works based on four design elements: - `yield return` (jumps to finally when resuming in dispose mode) - `yield break` (enters dispose mode and jumps to enclosing finally) - `finally` (after a `finally` we jump to the next enclosing one) - `DisposeAsync` (enters dispose mode and resumes execution) The caller of an async-iterator method should only call `DisposeAsync()` when the method completed or was suspended by a `yield return`. `DisposeAsync` sets a flag on the state machine ("dispose mode") and (if the method wasn't completed) resumes the execution from the current state. The state machine can resume execution from a given state (even those located within a `try`). When the execution is resumed in dispose mode, it jumps straight to the enclosing `finally`. `finally` blocks may involve pauses and resumes, but only for `await` expressions. As a result of the restrictions imposed on `yield return` (described above), dispose mode never runs into a `yield return`. Once a `finally` block completes, the execution in dispose mode jumps to the next enclosing `finally`, or the end of the method once we reach the top-level. Reaching a `yield break` also sets the dispose mode flag and jumps to the enclosing `finally` (or end of the method). By the time we return control to the caller (completing the promise as `false` by reaching the end of the method) all disposal was completed, and the state machine is left in finished state. So `DisposeAsync()` has no work left to do. Looking at disposal from the perspective of a given `finally` block, the code in that block can get executed: - by normal execution (ie. after the code in the `try` block), - by raising an exception inside the `try` block (which will execute the necessary `finally` blocks and terminate the method in Finished state), - by calling `DisposeAsync()` (which resumes execution in dispose mode and jumps to the enclosing finally), - following a `yield break` (which enters dispose mode and jumps to the enclosing finally), - in dispose mode, following a nested `finally`. A `yield return` is lowered as: ```csharp _current = expression; _state = ; goto ; // which does _valueOrEndPromise.SetResult(true); return; // resuming from state= will dispatch execution to this label : ; this.state = cachedState = NotStartedStateMachine; if (disposeMode) /* jump to enclosing finally or exit */ ``` A `yield break` is lowered as: ```csharp disposeMode = true; /* jump to enclosing finally or exit */ ``` ```csharp ValueTask IAsyncDisposable.DisposeAsync() { if (state >= StateMachineStates.NotStartedStateMachine /* -1 */) { throw new NotSupportedException(); } if (state == StateMachineStates.FinishedStateMachine /* -2 */) { return default; } disposeMode = true; _valueOrEndPromise.Reset(); var inst = this; _builder.Start(ref inst); return new ValueTask(this, _valueOrEndPromise.Version); // note this leverages the state machine's implementation of IValueTaskSource } ``` ##### Regular versus extracted finally When the `finally` clause contains no `await` expressions, a `try/finally` is lowered as: ```csharp try { ... finallyEntryLabel: } finally { ... } if (disposeMode) /* jump to enclosing finally or exit */ ``` When a `finally` contains `await` expressions, it is extracted before async rewriting (by AsyncExceptionHandlerRewriter). In those cases, we get: ```csharp try { ... goto finallyEntryLabel; } catch (Exception e) { ... save exception ... } finallyEntryLabel: { ... original code from finally and additional handling for exception ... } ``` In both cases, we will add a `if (disposeMode) /* jump to enclosing finally or exit */` after the block for `finally` logic. #### State values and transitions The enumerable starts with state -2. Calling GetAsyncEnumerator sets the state to -3, or returns a fresh enumerator (also with state -3). From there, MoveNext will either: - reach the end of the method (-2, we're done and disposed) - reach a `yield break` (state unchanged, dispose mode = true) - reach a `yield return` (-N, decreasing from -4) - reach an `await` (N, increasing from 0) From suspended state N or -N, MoveNext will resume execution (-1). But if the suspension was a `yield return` (-N), you could also call DisposeAsync, which resumes execution (-1) in dispose mode. When in dispose mode, MoveNext continues to suspend (N) and resume (-1) until the end of the method is reached (-2). The result of invoking `DisposeAsync` from states -1 or N is unspecified. This compiler generates `throw new NotSupportException()` for those cases. ``` DisposeAsync await +------------------------+ +------------------------> N | | | | v GetAsyncEnumerator | | resuming | -2 --------------------> -3 --------> -1 <-------------------------+ Dispose mode = false ^ | | | | done and disposed | | yield return | +-----------------------------------+ +-----------------------> -N | or exception thrown | | | | | | yield | | | break | DisposeAsync | | | +--------------------------+ | | | | | | | done and disposed v v suspension (await) +----------------------------------- -1 ------------------------> N ^ | Dispose mode = true | resuming | +--------------------------+ ``` --- ### Features Deconstruction (`docs/features/deconstruction.md`) Quickstart guide for deconstructions (C# 7.0) ---------------------------------------------- 1. Install Visual Studio 2017 2. Start a C# project 3. Add a reference to the `System.ValueTuple` package from NuGet 4. Use deconstructions: ```C# public class C { public static void Main() { int code; string message; var pair = (42, "hello"); (code, message) = pair; // deconstruct a tuple into existing variables Console.Write(message); // hello (code, message) = new Deconstructable(); // deconstruct any object with a proper Deconstruct method into existing variables Console.Write(message); // world (int code2, string message2) = pair; // deconstruct into new variables var (code3, message3) = new Deconstructable(); // deconstruct into new 'var' variables } } public class Deconstructable { public void Deconstruct(out int x, out string y) { x = 43; y = "world"; } } ``` Design ------ This design doc will cover two kinds of deconstruction: deconstruction into existing variables (*deconstruction-assignment*) and deconstruction into new variables (*deconstruction-declaration*). Here is an example of deconstruction-assignment: ```C# class C { static void Main() { long x; string y; (x, y) = new C(); System.Console.WriteLine(x + "" "" + y); } public void Deconstruct(out int a, out string b) { a = 1; b = ""hello""; } } ``` ### Deconstruction-assignment (deconstruction into existing variables): This doesn't introduce any changes to the language grammar. We have an *assignment-expression* (also simply called *assignment* in the C# grammar) where the *unary-expression* (the left-hand-side) is a *tuple-expression* containing values that can be assigned to. In short, what this does in the general case is find a `Deconstruct` method on the expression on the right-hand-side of the assignment, invoke it with the appropriate number of `out var` parameters, converts those output values (if needed) and assign them to the variables on the left-hand-side. But in the special case where the expression on the right-hand-side is a tuple (tuple expression or tuple type), then the elements of the tuple are assigned to the variables on the left-hand-side without calling Deconstruct. If the left-hand-side is nested the process will be repeated. For instance, in `(x, (y, z)) = deconstructable;`, `deconstructable` will be deconstructed into two parts and its second part will be further deconstructed. In the case where the expression on the right is a tuple expression, it is first given a type. So in `long x; string y; (x, y) = (1, null);` the literals on the right-hand-side are typed as `long` and `string` before the deconstruction even starts, which means that no conversions will be needed during the deconstruction steps. We noted already that tuples (which are syntactic sugar for the `System.ValueTuple` underlying type) don't need to invoke `Deconstruct`. The .NET framework also includes a set of `System.Tuple` types. Those are not recognized as C# tuples, and so will rely on the *Deconstruct* pattern. Those `Deconstruct` methods will be provided as extension methods for `System.Tuple` for up to 3 nestings deep (that is 21 elements). A *deconstruction-assignment* returns a tuple value (with elements using default names) which is shaped and typed like the left-hand-side and holds the (converted) parts resulting from deconstruction. #### Evaluation order The evaluation order can be summarized as: (1) all the side-effects on the left-hand-side, (2) all the `Deconstruct` invocations (if not tuple), (3) conversions (if needed), and (4) assignments. In the general case, the lowering for deconstruction-assignment would translate: `(expressionX, expressionY, expressionZ) = expressionRight` into: ``` // do LHS side-effects tempX = &evaluate expressionX tempY = &evaluate expressionY tempZ = &evaluate expressionZ // do Deconstruct evaluate right and evaluate Deconstruct in three parts (tempA, tempB and tempC) // do conversions tempConvA = convert tempA tempConvB = convert tempB tempConvC = convert tempC // do assignments tempX = tempConvA tempY = tempConvB tempZ = tempConvC ``` The evaluation order for nesting `(x, (y, z))` is: ``` // do LHS side-effects tempX = &evaluate expressionX tempY = &evaluate expressionY tempZ = &evaluate expressionZ // do Deconstruct evaluate right and evaluate Deconstruct into two parts (tempA and tempNested) evaluate Deconstruct on tempNested intwo two parts (tempB and tempC) // do conversions tempConvA = convert tempA tempConvB = convert tempB tempConvC = convert tempC // do assignments tempX = tempConvA tempY = tempConvB tempZ = tempConvC ``` The evaluation order for the simplest cases (locals, fields, array indexers, or anything returning ref) without needing conversion: ``` evaluate side-effect on the left-hand-side variables evaluate Deconstruct passing the references directly in ``` #### Resolution of the Deconstruct method The resolution is equivalent to typing `rhs.Deconstruct(out var x1, out var x2, ...);` with the appropriate number of parameters to deconstruct into. It is based on normal overload resolution. This implies that `rhs` cannot be dynamic and that none of the parameters of the `Deconstruct` method can be type arguments. A `Deconstruct(out T x1, out T x2)` method will not be found. Also, the `Deconstruct` method must be an instance method or an extension (but not a static method). It also must return `void`. ### Deconstruction-declaration (deconstruction into new variables): The *deconstruction-declaration* is also represented with an *assignment*, but the left-hand-side is either a *declaration-expression* or a tuple expression containing *declaration-expressions*. *Deconstruction-declarations* can be thought of as two steps: (1) declaring new locals, and (2) applying a *deconstruction-assignment* into those locals. The declaration of new locals by the left-hand-side of the assignment can take multiple forms. The simplest case is `(int x, string y)`, that is a tuple expression containing declaration expressions. Variants include nested declarations like `(int x, (string y, long z))` (which declares 3 locals) and implicitly typed declarations like `(var x, var y)`. The latter can also be written using the shorthand `var (x, y)`, which is a declaration expression with parenthesized designations. `var` is the only case where such shorthand is allowed (so `int (x, y)` is not legal). As in the case of *deconstruction-declarations*, tuple expressions on the right-hand-side have their type inferred from the left-hand-side. With *deconstruction-declaration*, this is also the case, except that any type is `var` in the left-hand-side, then the natural type of the element in the right-hand-side is used. For example, in `(string x, byte y, var z) = (null, 1, 2);`, `null` has type `string`, the literal `1` has type `byte` (inferred from `y`) and the literal `2` has type `int` (its natural type). In C#7.0, *deconstruction-declarations* are only allowed as top-level statements. They do not allow mixing of declaration expressions and assignable expressions (such as `(existing, var declared) = (1, 2)`). ### Grammar changes ```ANTLR expression : ... // existing | declaration_expression // new (only allowed in C#7.0 in certain contexts, such as out var, deconstruction and pattern declarations) ; declaration_expression // new : type variable_designation ; variable_designation // new : single_variable_designation | parenthesized_variable_designation | discard_designation ; single_variable_designation // new : identifier ; parenthesized_variable_designation // new : '(' variable_designation (',' variable_designation)+ ')' ; discard_designation // new : '_' ; foreach_variable_statement // new : 'foreach' '(' declaration_expression 'in' expression ')' embedded_statement ; ``` **References** [C# Design Notes for Oct 25-26, 2016](https://github.com/dotnet/csharplang/blob/main/meetings/2016/LDM-2016-10-25-26.md) [C# Design Notes for Sep 6, 2016](https://github.com/dotnet/csharplang/blob/main/meetings/2016/LDM-2016-09-06.md) [C# Design Notes for July 13, 2016](https://github.com/dotnet/csharplang/blob/main/meetings/2016/LDM-2016-07-13.md) [C# Design Notes for May 3-4, 2016](https://github.com/dotnet/csharplang/blob/main/meetings/2016/LDM-2016-05-03-04.md) [C# Design Notes for Apr 12-22, 2016](https://github.com/dotnet/csharplang/blob/main/meetings/2016/LDM-2016-04-12-22.md) [Design for declaration expressions](https://github.com/dotnet/csharplang/issues/365) The [What's new in C# 7.0](https://blogs.msdn.microsoft.com/dotnet/2016/08/24/whats-new-in-csharp-7-0) post has a section on deconstructions. **Possible future expansions** - deconstruction in [let and from clause](https://github.com/dotnet/csharplang/issues/189) - deconstruction in [lambda argument lists](https://github.com/dotnet/csharplang/issues/258) - [deconstruction patterns](https://github.com/dotnet/csharplang/issues/277) - allowing deconstructions with [mix of assignment and declaration](https://github.com/dotnet/csharplang/issues/125), thus also allowing deconstruction-declarations in expression contexts See [C# Lang](https://github.com/dotnet/csharplang) repo for more up-to-date proposals and discussions. --- ### Features Discards (`docs/features/discards.md`) Discards -------- Discards are variables which you can assign to, but cannot read from. They don't have names. Instead, they are represented by an '_' (underscore). In C#7.0, they can appear in the following contexts: - out variable declarations, such as `bool found = TryGetValue(out var _)` or `bool found = TryGetValue(out _)` - deconstruction assignments, such as `(x, _) = deconstructable;` - deconstruction declarations, such as `(var x, var _) = deconstructable;` - is patterns, such as `x is int _` - switch/case patterns, such as `case int _:` The principal representation of discards is an `_` (underscore) designation in a declaration expression. For example, `int _` in an out variable declaration or `var (_, _, x)` in a deconstruction declaration. The second representation of discards is using the expression `_` as a short-hand for `var _`, when no variable named `_` is in scope. It is allowed in out vars, deconstruction assignments and declarations, and plain assignments (`_ = IgnoredReturn();`). It is, however, not allowed in C#7.0 patterns. When a variable named `_` does exist in scope, then the expression `_` is simply a reference to that variable, as it did in earlier versions of C#. ###Grammar changes ```ANTLR declaration_expression : type variable_designation ; variable_designation : single_variable_designation | parenthesized_variable_designation | discard_designation // new ; discard_designation // new : '_' ; ``` **References** [C# Language Design Notes for Oct 25 and 26, 2016](https://github.com/dotnet/roslyn/issues/16640) [C# Design Notes for Oct 18, 2016](https://github.com/dotnet/roslyn/issues/16482) [Design proposal for discards](https://github.com/dotnet/roslyn/issues/14862) Note that those notes used to refer to discards as "wildcards". --- ### Features Dynamic Analysis.Work (`docs/features/dynamic-analysis.work.md`) Dynamic Analysis =============== This feature enables instrumentation of binaries to collect information for dynamic analysis. -------------------- TODO: - Generate code to compute MVIDs less often than once per invoked method. - Synthesize helper types to contain fields for analysis payloads so that methods of generic types can be instrumented. - Accurately identify which methods are tests. - Integrate dynamic analysis instrumentation with lowering. - Verify that instrumentation covers constructor initializers and field initializers. --- ### Features Enhanced Using (`docs/features/enhanced-using.md`) # Enhanced Using Enhanced using consists of several related features that aim to make the disposable coding pattern easier to participate in for implementers, and easier to consume for end-users. - _[Using declarations](#Using-declarations)_ aim to make consuming a disposable type simpler by allowing ```using``` to be added to a local declaration. - _[Pattern-based asynchronous disposal](#Pattern-based-asynchronous-disposal)_ allows a type to opt into asynchronous disposal without needing to implement the `IAsyncDisposable` interface. - _[Pattern-based disposal for ref structs](#Pattern-based-disposal-for-ref-structs)_ allows `ref struct`s to opt into disposal without needing to implement the `IDisposable` interface. __See__: the [corresponding proposal](https://github.com/dotnet/csharplang/blob/main/proposals/csharp-8.0/using.md) in CSharpLang. ## Using Declarations Using declarations allow a user to specify a `using` keyword as part of a local declaration statement: ```csharp using var x = ... // other statements ``` This is equivalent to declaring the local inside of a `using` statement at the same location: ```csharp using (var x = ...) { // other statements } ``` It is not valid to declare a using declaration without an initializer expression: ```csharp using IDisposable x; //error CS0210: You must provide an initializer in a fixed or using statement declaration ``` The initializer expression must result in a type that is considered to be Disposable. That is, the expression must also be valid when used directly inside a `using` statement: ```csharp using var x = using () { } // expression must also be valid in a standard using statement ``` ### Lifetime The lifetime of the local extends to the scope in which it is declared; immediately prior to the variable going out of scope, it will be disposed. ```csharp if (...) { using var x = ...; // other statements // Dispose x } ``` Using declarations in the same scope are disposed in the reverse order to which they are declared ```csharp { using var x = ...; using var y = ..., z = ...; // Dispose z // Dispose y // Dispose x } ``` As with a local declared as part of a `using` statement, a using local is `readonly` and may not be re-assigned after declaration. ```csharp using var x = ...; x = ...; // error CS1656: Cannot assign to 'x' because it is a 'using variable' ``` A using local may be used in the right hand side of an assignment or declaration. This means it is possible to capture a reference that will exist for longer than the lifetime of the using local. The reference will still be disposed when the using local is going out of scope. Interacting with the reference after disposal is undefined, but in most cases it is expected that it would result in an `ObjectDisposedException` being thrown. ```csharp IDisposable y = null; if (...) { using var x = ...; y = x; // Dispose x } y.Dispose(); // undefined. ObjectDisposedException in most cases ``` It is possible to use an existing reference as the initializer of a using local declaration. As above, the reference will be disposed when the declared local is going out of scope, but the existing reference will still be available for use via its previous declaration. ```csharp IDisposable y = ...; if (...) { using var x = y; // Dispose x, which points to the same object as y } y.Dispose(); // undefined. ObjectDisposedException in most cases ``` ### Asynchronous disposal When inside a method marked `async`, a user may optionally specify an additional `await` keyword prior to the `using` keyword, to indicate asynchronous disposal: ```csharp await using var x = ... ``` Which is equivalent to ```csharp await using (var x = ...) { } ``` The initializer expression must result in a type that is considered to be asynchronously disposable. That is, the expression must also be valid when used directly in an `await using` statement: ```csharp await using var x = await using () { } // expression must also be valid in a standard await using statement ``` Attempting to specify the `await` keyword on a using declaration in a method not marked with the `async` keyword results in a compiler error: ```csharp public void M() { await using var x = ...; //error CS4033: The 'await' operator can only be used within an async method. Consider marking this method with the 'async' modifier and changing its return type to 'Task'. } ``` ### Control flow In general, control flow is unaffected by the presence of a using declaration. However, there are some restrictions around the use of the `goto` statement when the statement and its target `label` lie either side of a using declaration. It is forbidden to jump 'forward' to a location after a using declaration, when the `goto` or the target `label` are in the same or lower scope as the declaration. ```csharp goto label1; // error CS8648: A goto cannot jump to a location after a using declaration. using var x = ...; label1: return; ``` ```csharp goto label1; // ok. using declaration is in a lower scope than goto and label { using var x = ...; } label1: return; ``` ```csharp { { goto label1; // error CS8648: A goto cannot jump to a location after a using declaration. } using var x = ...; } label1: return; ``` It is forbidden to jump 'backwards' to a location before a using declaration, when the label is in the same scope as the using declaration. ```csharp label1: using var x = ...; //error CS8649: A goto cannot jump to a location before a using declaration within the same block. goto label1; ``` Jumping 'backwards' to a label at a higher scope is specifically permitted. ```csharp label1: { using var x = ...; goto label1; // allowed. label1 is in a higher scope than the using declaration } ``` When jumping to a higher scope, any declared using variables will be disposed at the site of the `goto`. Variables not yet declared will **not** be disposed. ```csharp label1: { using var x = ...; goto label1; // dispose x using var y = ...; } ``` It is always permissible to jump to the same scope, when the `goto` and target `label` do not lie either side of a using declaration. ```csharp goto label1; // ok, not jumping over a using declaration label1: using var x = ...; label2: goto label2; // ok, not jumping over a using declaration ``` ### Other restrictions on use A using declaration may **not** appear directly inside of a case label. It may instead be used within a block inside of a case label: ```csharp switch (...) { case ...: using var x = ...; // error CS8647: A using variable cannot be used directly within a switch section (consider using braces). break; case ...: { using var y = ...; // ok // Dispose y } break; } ``` A using declaration may **not** appear directly as part of an `out` variable declaration. It is easy to emulate this behavior however, by adding a using declaration immediately after the `out` variable: ```csharp if(TryGetDisposable(out var x)) { using var y = x; // Dispose y, and thus x } ``` ## Pattern-based asynchronous disposal With pattern-based asynchronous disposal a type can be used in an `await using` statement without needing to explicitly implement `IAsyncDisposable` if it meets certain structural requirements. Specifically: "A reachable, non-generic Task-like returning instance method called DisposeAsync, that can be called with zero explicit arguments" Where reachable means legal to call from the site of the ```await using(...)``` under normal accessibility rules. ```csharp public class C { public static async Task M() { await using (AsyncDisposer a = new AsyncDisposer()) { } } } public class AsyncDisposer { public async ValueTask DisposeAsync() => Console.WriteLine("DisposeAsync"); } ``` In the situation where a type can both be implicitly converted to `IAsyncDisposable` and also fits the asynchronous disposal pattern, `IAsyncDisposable` is chosen. ### Optional arguments and params Pattern-based asynchronous disposal methods may contain optional or `params` parameters and still meet the requirements of the pattern. In general if you could write `c.DisposeAsync()` at the site of the `await using` syntax and have it be a valid call under normal language rules, then the type is considered to be asynchronously disposable. ```csharp public class AsyncDisposer { public async ValueTask DisposeAsync(int x = 0, params object[] args) => Console.WriteLine("DisposeAsync"); // valid pattern candidate } ``` ### Extension methods Extension methods may **not** be used to implement asynchronous disposal. The method must be a reachable _instance_ method in order to be considered as a valid candidate for the pattern. ### Nullable value type behavior _Note_: This is currently not working as spec'd and is tracked by [#34701](https://github.com/dotnet/roslyn/issues/34701) For nullable value types in a `using` statement, the behavior today is to call ```Dispose``` on the _underlying_ type if and only if the type has a value (i.e. ```if(t.HasValue){ t.GetValueOrDefault().Dispose() }``` ). This allows lifted types to be used as if they were the underlying type, and dispose is only called in the case they are not null. This behavior is extended to pattern-based ```DisposeAsync``` methods in `await using` statements. ```csharp public class C { public static async Task M() { StructDisposer? a = null; await using (a) { } // DisposeAsync is not invoked StructDisposer? b = new StructDisposer(); await using (b) { } // DisposeAsync is invoked } } public struct StructDisposer { public async ValueTask DisposeAsync() => Console.WriteLine("DisposeAsync"); } ``` ## Pattern-based disposal for ref structs Today `ref struct`s can not participate in the `IDisposable` pattern as they can not implement an interface. This feature allows a `ref struct` to be considered disposable if it meets certain structural requirements. Specifically: "A reachable, void returning instance method called Dispose, that can be called with zero explicit arguments" Where reachable means legal to call from the site of the `using(...)` under normal accessibility rules. ```csharp using System; public class C { public void M() { using var x = new Disposer(); } } ref struct Disposer { internal void Dispose() => Console.WriteLine("Disposed"); } ``` _Note:_ `ref struct`s can not be used in an `async` method and therefore can not participate in asynchronous disposal via `await using` ### Optional arguments and params Pattern-based disposal methods may contain optional or `params` parameters and still meet the requirements of the pattern. In general if you could write `s.Dispose()` at the site of the `using` syntax and have it be a valid call under normal language rules, then the `ref struct` is considered to be disposable. ```csharp public ref struct Disposer { public void Dispose(int x = 0, params object[] args) => Console.WriteLine("Disposed"); // valid pattern candidate } ``` ### Extension methods Extension methods may **not** be used to implement disposal. The method must be a reachable _instance_ method in order to be considered as a valid candidate for the pattern. --- ### Features Field Nullability (`docs/features/field-nullability.md`) # `field` keyword nullable analysis implementation See also [specification](https://github.com/dotnet/csharplang/blob/94205582d0f5c73e5765cb5888311c2f14890b95/proposals/field-keyword.md#nullability). ## Symbol API behavior In order to decide the NullableAnnotation of a SynthesizedBackingFieldSymbol we need to: 1. Decide if the getter associated with this field requires a null-resilience analysis. 2. If it *doesn't* require such an analysis, then the nullable annotation of the property is used. 3. If it *does*, then a binding+nullable analysis of the getter must be performed to decide the nullability of the backing field. There are some significant problems with directly exposing the nullable annotation on this field symbol. 1. Cost. It's not clear if it's OK for reading the NullableAnnotation off a field to cause us to bind and nullable analyze something. If some tooling is traversing member symbols for indexing or some such, it may be a problem if that causes unexpected method binding to occur where it didn't before. 2. Stack cycles. In the process of binding+flow analyzing, we may want to access the nullable annotation of the field. If we are *already* in the process of determining that field's nullable annotation, then we would need to "short-circuit" and return some "placeholder" value. We would need to take care to isolate the "internal" implementation from any public implementation, in order to ensure that a consistent answer is given to users of the public API. ... It's tempting to avoid exposing the inferred nullable annotation in the public symbol model. This might be problematic for automated tooling which is trying to reason about nullable initialization. For example, if a diagnostic suppressor wants to suppress `CS8618` nullable initialization warnings under certain conditions. How should it decide whether/why a property like `string Prop => field ??= GetValue();` requires initialization in constructors? It's unclear to me whether this matters. In the interests of *simplicity* (first make it correct, then make it fast), I'd like to move forward by *not* exposing the inferred annotation in the symbol model. Rather, we introduce a new internal API `SynthesizedBackingFieldSymbol.GetInferredNullableAnnotation()`, which `NullableWalker` will use to decide initial nullable state, report warnings, and so on. The implementation simply does an on-demand binding of the get accessor, then nullable analyzes and decides the nullable annotation. Care needs to be taken to avoid using this API in a re-entrant manner--so, the `NullableWalker` passes which are used to infer the nullability annotation must avoid calling it. This means that the ordinary `FieldSymbol.TypeWithAnnotations` API would not expose the inferred nullability, and neither would `IFieldSymbol.Type` or `IFieldSymbol.NullableAnnotation`. Instead the field's nullability would always match the property's. This is something I would actually like to fix, maybe before merging to main. It feels like Quick Info, etc., should expose the inferred nullability through the ordinary APIs. Once we get a handle on the behaviors of the getter null resilience, we can start talking about how to reduce cost associated with it. For example, we could try to reduce redundant binding, by forcing nullable annotations to be inferred prior to analyzing certain methods. Before processing a constructor in `MethodCompiler`, for example, maybe we would identify the getters that need to be compiled for purposes of inference, and force those to compile first. --- ### Features File Based Programs Vscode (`docs/features/file-based-programs-vscode.md`) # File-based apps VS Code support **File-based app** and **file-based program** are synonymous. See also: - [.NET SDK file-based apps](https://learn.microsoft.com/en-us/dotnet/core/sdk/file-based-apps) - [Microsoft Learn tutorial](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/tutorials/file-based-programs) - [General structure of a C# program](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/program-structure/) - [File-based app directives](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/program-structure/preprocessor-directives#file-based-app-directives) - [dotnet-run-file.md](https://github.com/dotnet/sdk/blob/main/documentation/general/dotnet-run-file.md) ## Feature overview A file-based app embeds a subset of MSBuild project capabilities into C# code, allowing single files to stand alone as ordinary projects. The following is a file-based app: ```cs Console.WriteLine("Hello World!"); ``` So is the following: ```cs #!/usr/bin/env dotnet run #:sdk Microsoft.NET.Sdk #:package Newtonsoft.Json@13.0.3 #:property LangVersion=preview using Newtonsoft.Json; Main(); void Main() { if (args is not [_, var jsonPath, ..]) { Console.Error.WriteLine("Usage: app "); return; } var json = File.ReadAllText(jsonPath); var data = JsonConvert.DeserializeObject(json); // ... } record Data(string field1, int field2); ``` The `dotnet` command line interprets the `#:` directives in source files, produces a C# project XML document in memory, and passes it off to MSBuild. The in-memory project is sometimes called a "virtual project". ## Rich miscellaneous files There is long-standing tracking to enhance the experience of working with miscellaneous files ("loose files" not associated with any project), including: - [issue 11599: lost IntelliSense when transferring a document to miscellaneous files](https://github.com/dotnet/roslyn/issues/11599) - [issue 55287: MiscellaneousFilesWorkspace design debt](https://github.com/dotnet/roslyn/issues/55287) - [issue 80743: canonical miscellaneous files project for VS Code](https://github.com/dotnet/roslyn/issues/80743). As part of the file-based app work, we can enable the following in such files without substantial issues: - Syntax diagnostics. - IntelliSense for the "default" set of references, for example those references which are included in the project created by `dotnet new console` with the current SDK. - In certain cases, we can even enable semantic diagnostics, with reasonable confidence that the resulting errors are useful to the user. These changes to misc files behavior are called "rich miscellaneous files". The implementation strategy is: the editor creates a "canonical misc files project" under the temp directory, and uses the resulting project info as a "base project" for loose files that are opened in the IDE. ## File-based app detection A C# file has multiple possible *classifications* in the editor: - **Project-Based App**. The file is part of an ordinary `.csproj` project. - **File-Based App**. The file is part of a "file-based app" project, i.e. it is either the entry point of a file-based app or it is `#:include`d by the entry point of the same. - **Miscellaneous File With Standard References and Semantic Errors**. The file is a valid entry point for a file-based app, but lacks the `#:`/`#!` directives which give us high certainty that this is the user's intent. - Tooling will light up accordingly, showing syntax errors, semantic errors, semantic info for the core library, etc. See [Rich miscellaneous files](#rich-miscellaneous-files). - These files will not be restored. - **Miscellaneous File With Standard References**. The file isn't part of any project, and heuristics indicate it's not intended to be a file-based app. The file uses the regular C# language (not the `.csx` scripting dialect). - Syntax errors and semantic info for the core library will appear in these files. - Semantic errors will not appear in these files. - **Miscellaneous File With No References**. The file isn't part of any project. It may even be a `.csx`, `.razor`, or other non-.cs type. - These files do not have any references to the core library, and do not show semantic errors. - Syntax errors, go-to-def on declarations in the same file, etc., may work. - When `enableFileBasedPrograms` is disabled, this classification is generally used instead of one of the *miscellaneous file with standard references* or *file-based app* classifications above. > [!NOTE] > This is intended to be a living document, and for the set of checks and classifications to possibly change over time depending on our needs. This is the decision tree for determining how to classify a C# file: 1. **Is the file in a currently loaded project?** - **Yes** → Classify as **Project-Based App** - **No** → Continue to next check 2. **Is `enableFileBasedPrograms` enabled?** (default: `true` in release) - **No** → Classify as **Miscellaneous File With No References** - **Yes** → Continue to next check 3. **Is the file a regular C# file? (i.e. not a `.csx` script, and not a file using a language besides C#)** - **No** → Classify as **Miscellaneous File With No References** - **Yes** → Continue to next check 4. **Does the file have an absolute path and exist on disk?** (i.e. it is not a "virtual document" created for a new, not-yet-saved file, or similar.) - **No** → Classify as **Miscellaneous File With Standard References** - **Yes** → Continue to next check 5. **Does the file have `#!` directives?** - **Yes** → Classify as **File-Based App**. Restore if needed and show semantic errors. - **No** → Continue to next check 6. **Does the file have `#:` directives?** - **No** → Go to (8) - **Yes** → Continue to next check 7. **Does the file have top-level statements?** - **Yes** → Classify as **File-Based App**. Restore if needed and show semantic errors. - **No** → Classify as **Miscellaneous File With Standard References** 8. **Is `enableFileBasedProgramsWhenAmbiguous` enabled?** (default: `false` in release, `true` in prerelease) - **No** → Classify as **Miscellaneous File With Standard References** - **Yes** → Continue to heuristic detection **Heuristic Detection (when `enableFileBasedProgramsWhenAmbiguous: true`):** 9. **Does the file have top-level statements?** - **No** → Classify as **Miscellaneous File With Standard References** - **Yes** → Continue to next check 10. **Is the file included in a `.csproj` cone?** - "Cone" means that a containing directory, at some level of nesting, has a `.csproj` file in it. - Note that this specific check is only performed at the time the file is opened. We think that the typical case is that the user will load a new project they are creating. Loading the project will cause the file to start being treated as project-based app per (1). If the user does not load the new project, then stale diagnostics may remain present until the file is closed and re-opened. - **Yes** → Classify as **Miscellaneous File With Standard References** (wait for project to load) - **No** → Classify as **Miscellaneous File With Standard References and Semantic Errors** ### Opt-out We added an opt-out flag with option name `dotnet.projects.enableFileBasedPrograms`. If issues arise with the file-based app experience, then VS Code users should set the corresponding setting `"dotnet.projects.enableFileBasedPrograms": false` to revert back to the old miscellaneous files experience. We also have a second, finer-grained opt-out flag `dotnet.projects.enableFileBasedProgramsWhenAmbiguous`. This flag is conditional on the previous flag (i.e. it is ignored when `enableFileBasedPrograms` is `false`). This is used to allow opting out only in cases where it is unclear from the single file itself, whether it should be treated as a file-based app. Presence of `#:` or `#!` directives in a `.cs` file strongly indicates that the file is a file-based app, and editor functionality will continue to light up for such files, even when `enableFileBasedProgramsWhenAmbiguous` is `false`. > [!NOTE] > The second flag is being used on a short-term basis while we work out the set of heuristics and cross-component APIs needed to efficiently and satisfactorily resolve whether a file with top-level statements but no directives is a file-based app in the context of a complex workspace. ## LSP handling of file-based apps When a C# file adds `#:` or `#!` directives, it becomes a file-based app. Conceptually, the file becomes both a C# source file and a project file in one. Conversely, when all `#:`/`#!` directives are removed, it stops being a project file, and goes back to being a C# source file only. In this scheme, we think of a file which contains `#:` as being the "entry point file" of the file-based app. We are adding support for an `#:include` directive to file-based apps, which lets users point at single files or `*` globs of C# source files, or other additional files (content, resources, etc.), which should be included in the file-based app project. This makes file-based app projects behave much more like ordinary projects in the workspace. In particular, we can have situations like the following: - `Util.cs` (an ordinary source file) - `App1.cs`, a file-based app entry point containing `#:include Util.cs` - `App2.cs`, a file-based app entry point also containing `#:include Util.cs` - `MyProject.csproj`, also containing `` Because all these projects are simply added as projects to the host workspace, it's expected that features like "active project context" and multi-targeting-aware Quick Info "just work" with all of them. One key assumption we are making is: it is not valid for a file-based app *entry point* to be a member of an ordinary project. For example, you cannot have the following: - `Util.cs` (an ordinary source file) - `App1.cs`, a file-based app entry point containing `#:include Util.cs` - `MyProject.csproj`, containing `` **<-- This part is considered malformed** An error is reported generally for presence of `#:`/`#!` directives in ordinary projects. Depending on the order that things load, such files may or may not also be detected as file-based app entry points. In this case we want the user to do one of two things to resolve the issue: 1. Delete the `#:`/`#!` directives. We will unload the file as file-based app in this case. 2. Remove the file-based app entry point as a member of any ordinary project(s). We expect the appropriate project system(s) to be able to observe either of the above changes and move the workspace into a "healthy" state once the user has corrected the error. ### `FileBasedProgramsProjectSystem` Manages projects for file-based apps and miscellaneous files. This project system effectively performs the classification process described in [File-based app detection](#file-based-app-detection) when a design-time build is performed for the project, and transitions the state of the project to match the latest classification. The project system uses the file-based app entry point file, translates it to a virtual MSBuild project, then runs a design-time build on that project. If it detects missing assets, it may also restore the virtual project. It uses file watchers to watch the project globs and redo the design-time build on relevant changes, such as changes to `#:` directives. ## Automatic discovery The Roslyn LSP will automatically discover and load file-based apps in the opened workspace folders. The user can opt out of this discovery process by setting `"dotnet.fileBasedApps.enableAutomaticDiscovery": false`. This work was tracked by [PR 82863: file-based apps automatic discovery](https://github.com/dotnet/roslyn/pull/82863). For the first release of the feature in the VS Code C# extension, the setting will be disabled-by-default in the stable release channel and enabled-by-default in the prerelease channel. Certain subfolders in a workspace are excluded from this discovery process: - Any folders which contain a `.csproj` file. - Any folders with names conventionally reserved for build artifacts, such as `artifacts`, `bin`, and `obj`. - Any folders marked "hidden" in the file system. `.git` and `.vs` typically fall into this. The first time that discovery is performed in a workspace, the LSP will read all `.cs` files in the opened workspace folders which are not excluded by the above conditions. If the file content starts with `#!`, it is marked as a file-based app and loaded. A cache file is created after each discovery pass and stored in the user temp directory. This file holds: - The time that the previous discovery pass started. - Paths of file-based apps found during the last discovery pass. - Paths of folders that were found to contain `.csproj` files during the last discovery pass. The cache data allows the following optimizations in subsequent discovery passes: - Allows not reading any C# files whose last write time is older than the cached time. - Allows reducing the number of times we list files in directories whose last write time is older than the cached time. ### `#!` requirement This design requires files to start with `#!` in order to participate in discovery. Specifically, a discoverable file must start with either the byte sequence `0x23, 0x21` (ASCII/UTF-8 `#!`), or the byte sequence `0xEF, 0xBB, 0xBF, 0x23, 0x21` (UTF-8 BOM followed by `#!`). The reason for this is: we anticipate adding support for `#:` to non-entry-point files. This means that having `#:` is not going to be enough to identify a file as definitely the entry point. Related work is tracked by: - [PR 81284: included file-based app sources](https://github.com/dotnet/roslyn/pull/81284) - [issue 82292: `#:include` scenario](https://github.com/dotnet/roslyn/issues/82292) - [PR 83185: transitive directive heuristic](https://github.com/dotnet/roslyn/pull/83185) Instead, it will be necessary to search for both `#:` and top-level statements at a minimum. This cost is acceptable for files that were explicitly opened in the editor, but is a bit steep for a broad discovery pass. For this reason, we put `#!`-at-start as a standard for entry points of file-based apps. We report a warning in files which contain both `#:include` and top-level statements, but do not have `#!` at the top. We did this in [PR 80575: File-based programs live directive diagnostics](https://github.com/dotnet/roslyn/pull/80575). ## Future considerations This section is not intended to serve as permanent documentation but as more of a roadmap for a series of changes we may make in this area in the near future. It should not be necessary to read/understand this in order to evaluate a PR currently under review. Anything that the current PR is actually implementing is covered in previous sections. ### Treating files with no directives as file-based app entry points **Miscellaneous File With Standard References and Semantic Errors** is a designation we essentially have in order to avoid restoring things we aren't 100% sure are file-based apps. This particularly includes files which have top-level statements, but no `#:`/`#!` directives. We may want to stop using this designation for files which exist on disk. Instead, we may identify files that (1) contain top-level statements, (2) are not part of an ordinary project, and (3) have no `.csproj` in-cone. We'd classify these files as file-based apps. This would improve accuracy in the editor in certain cases and make it easier to do things like avoid showing the "This is a miscellaneous file, things may be broken" popup. Related tracking includes [issue 81252: Do not restore loose files which lack file-based app directives](https://github.com/dotnet/roslyn/issues/81252) and [PR 84385: Extract file-based app entry point heuristic](https://github.com/dotnet/roslyn/pull/84385). ### Allowing non-entry-point files to contain `#:` directives We are considering adding support for non-entry-point files to contain `#:` in the future. In this case, we would need an additional bit of information to distinguish entry points from non-entry-points. Related work is tracked by: - [PR 81284: included file-based app sources](https://github.com/dotnet/roslyn/pull/81284) - [issue 82292: `#:include` scenario when entry point opened manually](https://github.com/dotnet/roslyn/issues/82292) - [PR 83185: Adjust heuristics to properly handle transitive directives](https://github.com/dotnet/roslyn/pull/83185) For *multi-file file-based apps*, users should use a `#!` at the top of the entry point file to make it easy to identify. For *single-file file-based apps*, we think that just using `#:` and top-level statements together should be enough, to identify a file that was explicitly opened in the editor as an entry point. ### Checking top-level statements presence without doing a full parse Currently there are cases where we may end up needing to do an additional parse of a file just to check if it contains top-level statements. This is generally a situation we'd like to avoid, and, would prefer to either use a pattern where the file already exists in some project and has a syntax tree we can check incrementally, or, that we devise some other solution for performing our heuristics which doesn't require a full parse. This is tracked by [issue 78878: Determine if a source file contains top-level statements without doing a full additional parse](https://github.com/dotnet/roslyn/issues/78878). --- ### Features Generators (`docs/features/generators.md`) Source Generators ================= Deprecation notice ---------- This proposal has been deprecated in favor of the newer [Source Generators](source-generators.md) proposal. --- ### Features Incremental Generators.Cookbook (`docs/features/incremental-generators.cookbook.md`) # Incremental Generators Cookbook ## Summary This document aims to be a guide to help the creation of source generators by providing a series of guidelines for common patterns. It also aims to set out what types of generators are possible under the current design, and what is expected to be explicitly out of scope in the final design of the shipping feature. **This document expands on the details in the [full design document](incremental-generators.md), please ensure you have read that first.** ## Table of contents - [Incremental Generators Cookbook](#incremental-generators-cookbook) - [Summary](#summary) - [Table of contents](#table-of-contents) - [Proposal](#proposal) - [Out of scope designs](#out-of-scope-designs) - [Language features](#language-features) - [Code rewriting](#code-rewriting) - [Conventions](#conventions) - [Pipeline model design](#pipeline-model-design) - [Use `ForAttributeWithMetadataName`](#use-forattributewithmetadataname) - [Use an indented text writer, not `SyntaxNode`s, for generation](#use-an-indented-text-writer-not-syntaxnodes-for-generation) - [Put `Microsoft.CodeAnalysis.EmbeddedAttribute` on generated marker types](#put-microsoftcodeanalysisembeddedattribute-on-generated-marker-types) - [Do not scan for types that indirectly implement interfaces, indirectly inherit from types, or are indirectly marked by an attribute from an interface or base type](#do-not-scan-for-types-that-indirectly-implement-interfaces-indirectly-inherit-from-types-or-are-indirectly-marked-by-an-attribute-from-an-interface-or-base-type) - [Designs](#designs) - [Generated class](#generated-class) - [Additional file transformation](#additional-file-transformation) - [Augment user code](#augment-user-code) - [Issue Diagnostics](#issue-diagnostics) - [INotifyPropertyChanged](#inotifypropertychanged) - [Package a generator as a NuGet package](#package-a-generator-as-a-nuget-package) - [Use functionality from NuGet packages](#use-functionality-from-nuget-packages) - [Access Analyzer Config properties](#access-analyzer-config-properties) - [Consume MSBuild properties and metadata](#consume-msbuild-properties-and-metadata) - [Unit Testing of Generators](#unit-testing-of-generators) - [Auto interface implementation](#auto-interface-implementation) - [Breaking Changes:](#breaking-changes) - [Open Issues](#open-issues) ## Proposal As a reminder, the high level design goals of source generators are: - Generators produce one or more strings that represent C# source code to be added to the compilation. - Explicitly _additive_ only. Generators can add new source code to a compilation but may **not** modify existing user code. - May access _additional files_, that is, non-C# source texts. - Run _un-ordered_, each generator will see the same input compilation, with no access to files created by other source generators. - A user specifies the generators to run via list of assemblies, much like analyzers. - Generators create a pipeline, starting from base input sources and mapping them to the output they wish to produce. The more exposed, properly equatable states exist, the earlier the compiler will be able to cut off changes and reuse the same output. ## Out of scope designs We will briefly look at the non-solvable problems as examples of the kind of problems source generators are *not* designed to solve: ### Language features Source generators are not designed to replace new language features: for instance one could imagine [records](records.md) being implemented as a source generator that converts the specified syntax to a compilable C# representation. We explicitly consider this to be an anti-pattern; the language will continue to evolve and add new features, and we don't expect source generators to be a way to enable this. Doing so would create new 'dialects' of C# that are incompatible with the compiler without generators. Further, because generators, by design, cannot interact with each other, language features implemented in this way would quickly become incompatible with other additions to the language. ### Code rewriting There are many post-processing tasks that users perform on their assemblies today, which here we define broadly as 'code rewriting'. These include, but are not limited to: - Optimization - Logging injection - IL Weaving - Call site re-writing While these techniques have many valuable use cases, they do not fit into the idea of *source generation*. They are, by definition, code altering operations which are explicitly ruled out by the source generators proposal. There are already well supported tools and techniques for achieving these kinds of operations, and the source generators proposal is not aimed at replacing them. We are exploring approaches for call site rewriting (see [interceptors.md](interceptors.md)), but those features are experimental and may change significantly or even be removed. ## Conventions ### Pipeline model design As a general guideline, source generator pipelines need to pass along models that are _value equatable_. This is critical to the incrementality of an `IIncrementalGenerator`; as soon as a pipeline step returns the same information that it returned in the previous run, the generator driver can stop running the generator and reuse the same cached data that the generator produced in the previous run. Most times a generator is triggered (particularly generators that need to look at type or method definitions, using `ForAttributeWithMetadataName`) the edit that triggered the generator will not actually have affected the things that your generator is looking at. However, because semantics can change on basically any edit, the generator driver _must_ rerun your generator again to ensure that this is the case. If your generator then produces a model with the same values as it did previously, this short-circuits the pipeline and allows us to avoid a lot of work. Here are some general guidelines around designing your models to ensure that you maintain this equality: * Use `record`s, rather than `class`es, so that value equality is generated for you. * Symbols (`ISymbol` and anything that inherits from that interface) are never equatable, and including them in your model can potentially root old compilations and force Roslyn to hold onto lots of memory that it could otherwise free. Never put these in your model types. Instead, extract the information you need from the symbols you inspect to an equatable representation: `string`s often work quite well here. * `SyntaxNode`s are also usually not equatable between runs. They're not as strongly discouraged from the initial stages of a pipeline as symbols, and an example [later down](#access-analyzer-config-properties) shows a case where you will need to include a `SyntaxNode` in model. They also don't potentially root as much memory as symbols will. However, any edit in a file will ensure that all `SyntaxNode`s from that file are no longer equatable, so they should be removed from your models as soon as possible. * The previous bullet applies to `Location`s as well. * Be careful of collection types in your models. Most built-in collection types in .NET do not do value equality by default. Arrays, `ImmutableArray` and `List`, for example, use reference equality, not value equality. We suggest that most generator authors use a wrapper type around arrays to augment them with value-based equality. ### Use `ForAttributeWithMetadataName` We highly recommend that all generator authors that need to inspect syntax do so by using a marker attribute to indicate the types or members that need to be inspected. This has multiple benefits, both for you as an author, and also for your users: * As an author, you can use `SyntaxProvider.ForAttributeWithMetadataName`. This utility method is at least 99x more efficient than `SyntaxProvider.CreateSyntaxProvider`, and in many cases even more efficient. This will help you avoid causing performance issues for your users in editors. * Your users can clearly indicate that they _intend_ to use your source generator. This intention is extremely helpful for designing a good user experience; it means that you can author Roslyn analyzers to help your users when they intended to use your generator but violated your rules in some fashion. For example, if you are generating some method body, and your generator requires that the user return a specific type, the presence of a `GenerateMe` attribute means you can write an analyzer to tell the user if their method declaration returns something that it shouldn't. ### Use an indented text writer, not `SyntaxNode`s, for generation We do not recommend generating `SyntaxNode`s when generating syntax for `AddSource`. Doing so can be complex, and it can be difficult to format it well; calling `NormalizeWhitespace` is often quite expensive, and the API is not really designed for this use-case. Additionally, to ensure immutability guarantees, `AddSource` does not accept `SyntaxNode`s. It instead requires getting the `string` representation and putting that into a `SourceText`. Instead of `SyntaxNode`, we recommend using a wrapper around `StringBuilder` that will keep track of indent level and prepend the right amount of indentation when `AppendLine` is called. See [this](https://github.com/dotnet/roslyn/issues/52914#issuecomment-1732680995) conversation on the performance of `NormalizeWhitespace` for more examples, performance measurements, and discussion on why we don't believe that `SyntaxNode`s are a good abstraction for this use case. ### Put `Microsoft.CodeAnalysis.EmbeddedAttribute` on generated marker types Users might depend on your generator in multiple projects in the same solution, and these projects will often have `InternalsVisibleTo` applied. This means that your `internal` marker attributes may be defined in multiple projects, and the compiler will warn about this. While this doesn't block compilation, it can be irritating to users. To avoid this, mark such attributes with `Microsoft.CodeAnalysis.EmbeddedAttribute`; when the compiler sees this attribute on a type from separate assembly or project, it will not include that type in lookup results. To ensure that `Microsoft.CodeAnalysis.EmbeddedAttribute` is available in the compilation, call the `AddEmbeddedAttributeDefinition` helper method in your `RegisterPostInitializationOutput` callback. Another option is to provide an assembly in your nuget package that defines your marker attributes, but this can be more difficult to author. We recommend the `EmbeddedAttribute` approach, unless you need to support versions of Roslyn lower than 4.14. ### Do not scan for types that indirectly implement interfaces, indirectly inherit from types, or are indirectly marked by an attribute from an interface or base type Using an interface/base type marker can be a very tempting and natural fit for generators. However, scanning for these types of markers is _very_ expensive, and cannot be done incrementally. Doing so can have an outsized impact on IDE and command-line performance, even for fairly small consuming users. These scenarios are: * A user implements an interface on `BaseModelType`, and then the generator looks all derived types from `BaseModelType`. Because the generator cannot know ahead of time what `BaseModelType` actually is, it means that the generator has to fetch `AllInterfaces` on every single type in the compilation so it can scan for the marker interface. This will end up occurring either on every keystroke or every file save, depending on what mode the user is running generators in; either one is disastrous for IDE performance, even when trying to optimize by scoping down the scanning to only types with a base list. * A user inherits from a generator-defined `BaseSerializerType`, and the generator looks for anything that inherits from that type, either directly or indirectly. Similar to the above scenario, the generator will need to scan all types with a base type in the entire compilation for the inherited `BaseSerializerType`, which will heavily impact IDE performance. * A generator looks among all base types/implemented interfaces for a type that is attributed with a generator's marker attribute. This is effectively either scenario 1 or 2, just with a different search criteria. * A generator leaves its marker attribute unsealed, and expects users to be able to derive their own attributes from that marker, as a source of parameter customization. This has a couple of problems: first, every attributed type needs to be checked to see if the attribute inherits from the marker attribute. While not as performance impacting as the first three scenarios, this isn't great for performance. Second, and more importantly, there is no good way to retrieve any customizations from the inherited attribute. These attributes are not instantiated by the source generator, so any parameters passed to the `base()` constructor call or values that are assigned to any properties of the base attribute are not visible to the generator. Prefer using FAWMN-driven development here, and using an analyzer to inform the user if they need to inherit from some base class for your generator to work correctly. ## Designs This section is broken down by user scenarios, with general solutions listed first, and more specific examples later on. ### Generated class **User scenario:** As a generator author I want to be able to add a type to the compilation, that can be referenced by the user's code. Common use cases include creating an attribute that will be used to drive other source generator steps. **Solution:** Have the user write the code as if the type was already present. Generate the missing type based on information available in the compilation using the `RegisterPostInitializationOutput` step. **Example:** Given the following user code: ```csharp public partial class UserClass { [GeneratedNamespace.GeneratedAttribute] public partial void UserMethod(); } ``` Create a generator that will create the missing type when run: ```csharp [Generator] public class CustomGenerator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext context) { context.RegisterPostInitializationOutput(static postInitializationContext => { postInitializationContext.AddEmbeddedAttributeDefinition(); postInitializationContext.AddSource("myGeneratedFile.cs", SourceText.From(""" using System; using Microsoft.CodeAnalysis; namespace GeneratedNamespace { [Embedded] internal sealed class GeneratedAttribute : Attribute { } } """, Encoding.UTF8)); }); } } ``` **Alternative Solution**: If you are also providing a library to your users, in addition to a source generator, simply have that library include the attribute definition. ### Additional file transformation **User scenario:** As a generator author I want to be able to transform an external non-C# file into an equivalent C# representation. **Solution:** Use the `AdditionalTextsProvider` to filter for and retrieve your files. Transform them into the code you care about, then register that code with the solution. **Example:** ```csharp [Generator] public class FileTransformGenerator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext context) { var pipeline = context.AdditionalTextsProvider .Where(static (text) => text.Path.EndsWith(".xml")) .Select(static (text, cancellationToken) => { var name = Path.GetFileName(text.Path); var code = MyXmlToCSharpCompiler.Compile(text.GetText(cancellationToken)); return (name, code); }); context.RegisterSourceOutput(pipeline, static (context, pair) => // Note: this AddSource is simplified. You will likely want to include the path in the name of the file to avoid // issues with duplicate file names in different paths in the same project. context.AddSource($"{pair.name}generated.cs", SourceText.From(pair.code, Encoding.UTF8))); } } ``` Items need to be included in your csproj files by using the `AdditionalFiles` ItemGroup: ```xml ``` ### Pre-compilation source generation > Note: this is an experimental API and requires suppressing diagnostic `RSEXPERIMENTAL007` to use. **User scenario:** As a generator author I want to produce source from non-compilation inputs (e.g. additional files, parse options) and have that source be visible to compilation-dependent code in the same generator (or in other generators), so that downstream code can call `compilation.GetTypeByMetadataName(...)` and find the types I just emitted. **Solution:** Use `RegisterPreCompilationSourceOutput`. It runs in a new phase that executes *before* the standard `RegisterSourceOutput` phase, and its emitted source is added to the compilation that the standard phase observes. Because the phase runs before the compilation is fully built, the callback can only consume non-compilation providers (additional files, parse options, analyzer config options); attempting to reach `CompilationProvider` or `SyntaxProvider` from a pre-compilation pipeline throws at runtime. **Example:** ```csharp [Generator] public class ProtoGenerator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext context) { // Pre-compilation phase: read .proto files and emit C# types. // The emitted source becomes part of the compilation that the standard // phase below sees, so compilation.GetTypeByMetadataName("MyProto.Message") // will find the generated type. var protoFiles = context.AdditionalTextsProvider .Where(static (f) => f.Path.EndsWith(".proto")); context.RegisterPreCompilationSourceOutput(protoFiles, static (ctx, file) => { var types = ParseProtoFile(file.GetText()!.ToString()); ctx.AddSource($"{Path.GetFileName(file.Path)}.g.cs", GenerateTypes(types)); }); // Standard phase: the compilation now contains the generated proto types, // so we can look them up symbolically and emit code that depends on them. context.RegisterSourceOutput(context.CompilationProvider, static (ctx, compilation) => { var messageType = compilation.GetTypeByMetadataName("MyProto.Message"); if (messageType is not null) { ctx.AddSource("Serializers.g.cs", GenerateSerializer(messageType)); } }); } } ``` **When to use this over `RegisterPostInitializationOutput`:** post-init runs even earlier, but takes no inputs at all; it is for emitting fixed source like marker attributes. If your generator needs to react to additional files, parse options, or analyzer config and have the result visible to downstream compilation-dependent code, pre-compilation is the right fit. **When to use this over `RegisterSourceOutput`:** standard `RegisterSourceOutput` always runs *after* the compilation is built and cannot contribute source back into the compilation seen by other generators. Use pre-compilation when you specifically need that cross-generator visibility, e.g. one generator emits domain types from additional files and another generator (or a later pipeline in the same generator) needs to consume those types via `CompilationProvider`. **Diagnostics:** `PreCompilationSourceProductionContext` intentionally does **not** include `ReportDiagnostic`. Pre-compilation is an early phase focused purely on producing source; diagnostic reporting should be done in a separate analyzer (see *Issue Diagnostics* below). ### Augment user code **User scenario:** As a generator author I want to be able to inspect and augment a user's code with new functionality. **Solution:** Require the user to make the class you want to augment be a `partial class`, and mark it with a unique attribute. Provide that attribute in a `RegisterPostInitializationOutput` step. Register for callbacks on that attribute with `ForAttributeWithMetadataName` to collect the information needed to generate code, and use tuples (or create an equatable model) to pass along that information. That information should be extracted from syntax and symbols; **do not put syntax or symbols into your models**. **Example:** ```csharp public partial class UserClass { [GeneratedNamespace.Generated] public partial void UserMethod(); } ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Issue Diagnostics **User Scenario:** As a generator author I want to be able to add diagnostics to the user's compilation. **Solution:** We do not recommend issuing diagnostics within generators. It is possible, but doing so without breaking incrementality is advanced topic beyond the scope of this cookbook. Instead, we suggest writing a separate analyzer for reporting diagnostics. ### INotifyPropertyChanged **User scenario:** As a generator author I want to be able to implement the `INotifyPropertyChanged` pattern automatically for a user. **Solution:** The design tenant 'Explicitly additive only' seems to be at direct odds with the ability to implement this, and appears to call for user code modification. However we can instead take advantage of explicit fields and instead of *editing* the users properties, directly provide them for listed fields. **Example:** Given a user class such as: ```csharp using AutoNotify; public partial class UserClass { [AutoNotify] private bool _boolProp; [AutoNotify(PropertyName = "Count")] private int _intProp; } ``` A generator could produce the following: ```csharp using System; using System.ComponentModel; namespace AutoNotify { [AttributeUsage(AttributeTargets.Field, Inherited = false, AllowMultiple = false)] sealed class AutoNotifyAttribute : Attribute { public AutoNotifyAttribute() { } public string PropertyName { get; set; } } } public partial class UserClass : INotifyPropertyChanged { public bool BoolProp { get => _boolProp; set { _boolProp = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs("UserBool")); } } public int Count { get => _intProp; set { _intProp = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs("Count")); } } public event PropertyChangedEventHandler PropertyChanged; } ``` ### Package a generator as a NuGet package **User scenario**: As a generator author I want to package my generator as a NuGet package for consumption. **Solution:** Generators can be packaged using the same method as an Analyzer would. Ensure the generator is placed in the `analyzers\dotnet\cs` folder of the package for it to be automatically added to the users project on install. For example, to turn your generator project into a NuGet package at build, add the following to your project file: ```xml true false ``` ### Use functionality from NuGet packages **User Scenario:** As a generator author I want to rely on functionality provided in NuGet packages inside my generator. **Solution:** It is possible to depend on NuGet packages inside of a generator, but special consideration has to be taken for distribution. Any *runtime* dependencies, that is, code that the end users program will need to rely on, can simply be added as a dependency of the generator NuGet package via the usual referencing mechanism. For example, consider a generator that creates code that relies on `Newtonsoft.Json`. The generator does not directly use the dependency, it just emits code that relies on the library being referenced in the users compilation. The author would add a reference to `Newtonsoft.Json` as a public dependency, and when the user adds the generator package it will be referenced automatically. ```xml true false ``` However, any *generation-time* dependencies, that is, used by the generator while it is is running and generating code, must be packaged directly alongside the generator assembly inside the generator NuGet package. There are no automatic facilities for this, and you will need to manually specify the dependencies to include. Consider a generator that uses `Newtonsoft.Json` to encode something to json during the generation pass, but does not emit any code the relies on it being present at runtime. The author would add a reference to `Newtonsoft.Json` but make all of its assets *private*; this ensures the consumer of the generator does not inherit a dependency on the library. The author would then have to package the `Newtonsoft.Json` library alongside the generator inside of the NuGet package. This can be achieved in the following way: set the dependency to generate a path property by adding `GeneratePathProperty="true"`. This will create a new MSBuild property of the format `PKG` where `` is the package name with `.` replaced by `_`. In our example there would be an MSBuild property called `PKGNewtonsoft_Json` with a value that points to the path on disk of the binary contents of the NuGet files. We can then use that to add the binaries to the resulting NuGet package as we do with the generator itself: ```xml true false ``` ```C# [Generator] public class JsonUsingGenerator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext context) { var pipeline = context.AdditionalTextsProvider.Select(static (text, cancellationToken) => { if (!text.Path.EndsWith("*.json")) { return default; } return (Name: Path.GetFileName(text.Path), Value: Newtonsoft.Json.JsonConvert.DeserializeObject(text.GetText(cancellationToken).ToString())); }) .Where((pair) => pair is not ((_, null) or (null, _))); context.RegisterSourceOutput(pipeline, static (context, pair) => { var sourceText = SourceText.From($$""" namespace GeneratedNamespace { internal sealed class GeneratedClass { public static const (int A, int B) SerializedContent = ({{pair.A}}, {{pair.B}}); } } """, Encoding.UTF8); context.AddSource($"{pair.Name}generated.cs", sourceText) }); } record MyObject(int A, int B); } ``` ### Access Analyzer Config properties **User Scenarios:** - As a generator author I want to access the analyzer config properties for a syntax tree or additional file. - As a generator author I want to access key-value pairs that customize the generator output. - As a user of a generator I want to be able to customize the generated code and override defaults. **Solution**: Generators can access analyzer config values via the `AnalyzerConfigOptionsProvider`. Analyzer config values can either be accessed in the context of a `SyntaxTree`, `AdditionalFile` or globally via `GlobalOptions`. Global options are 'ambient' in that they don't apply to any specific context, but will be included when requesting option within a specific context. Note that this is one of the few cases that it is necessary to put a `SyntaxNode` into a pipeline, as you need the tree in order to get the generator option. Try to get the `SyntaxNode` out of the pipeline as fast as possible to avoid making the model not correctly equatable. A generator is free to use a global option to customize its output. For example, consider a generator that can optionally emit logging. The author may choose to check the value of a global analyzer config value in order to control whether or not to emit the logging code. A user can then choose to enable the setting per project via an `.globalconfig` file: ```.globalconfig mygenerator_emit_logging = true ``` ```csharp [Generator] public class MyGenerator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext context) { var userCodePipeline = context.SyntaxProvider.ForAttributeWithMetadataName(... /* collect user code info */); var emitLoggingPipeline = context.AnalyzerConfigOptionsProvider.Select(static (options, cancellationToken) => options.GlobalOptions.TryGetValue("mygenerator_emit_logging", out var emitLoggingSwitch) ? emitLoggingSwitch.Equals("true", StringComparison.InvariantCultureIgnoreCase) : false); // Default context.RegisterSourceOutput(userCodePipeline.Combine(emitLoggingPipeline), (context, pair) => /* emit code */); } } ``` ### Consume MSBuild properties and metadata **User Scenarios:** - As a generator author I want to make decisions based on the values contained in the project file - As a user of a generator I want to be able to customize the generated code and override defaults. **Solution:** MSBuild will automatically translate specified properties and metadata into a global analyzer config that can be read by a generator. A generator author specifies the properties and metadata they want to make available by adding items to the `CompilerVisibleProperty` and `CompilerVisibleItemMetadata` item groups. These can be added via a props or targets file when packaging the generator as a NuGet package. For example, consider a generator that creates source based on additional files, and wants to allow a user to enable or disable logging via the project file. The author would specify in their props file that they want to make the specified MSBuild property visible to the compiler: ```xml ``` The value of `MyGenerator_EnableLogging` property will then be emitted to a generated analyzer config file before build, with a name of `build_property.MyGenerator_EnableLogging`. The generator is then able read this property from via the `GlobalOptions` property of the `AnalyzerConfigOptionsProvider` pipeline: ```c# context.AnalyzerConfigOptionsProvider.Select((provider, ct) => provider.GlobalOptions.TryGetValue("build_property.MyGenerator_EnableLogging", out var emitLoggingSwitch) ? emitLoggingSwitch.Equals("true", StringComparison.InvariantCultureIgnoreCase) : false); ``` A user can thus enable, or disable logging, by setting a property in their project file. Now, consider that the generator author wants to optionally allow opting in/out of logging on a per-additional file basis. The author can request that MSBuild emit the value of metadata for the specified file, by adding to the `CompilerVisibleItemMetadata` item group. The author specifies both the MSBuild itemType they want to read the metadata from, in this case `AdditionalFiles`, and the name of the metadata that they want to retrieve for them. ```xml ``` This value of `MyGenerator_EnableLogging` will be emitted to a generated analyzer config file, for each of the additional files in the compilation, with an item name of `build_metadata.AdditionalFiles.MyGenerator_EnableLogging`. The generator can read this value in the context of each additional file: ```cs context.AdditionalTextsProvider .Combine(context.AnalyzerConfigOptionsProvider) .Select((pair, ctx) => pair.Right.GetOptions(pair.Left).TryGetValue("build_metadata.AdditionalFiles.MyGenerator_EnableLogging", out var perFileLoggingSwitch) ? perFileLoggingSwitch : false); ``` In the users project file, the user can now annotate the individual additional files to say whether or not they want to enable logging: ```xml ``` Note that MSBuild properties passed to source generators via `CompilerVisibleProperty` are written into and read from an editorconfig file, [resulting in data loss for non-trivial property values](https://github.com/dotnet/roslyn/issues/51692). One possible workaround is to use a build task to apply a transport encoding preventing the data loss; as an example a semicolon separated list can be converted to a space separated list (`;` is the editorconfig comment character): ```xml <_MyInterpolatorsNamespaces>$([System.String]::Copy('$(InterpolatorsNamespaces)').Replace(';', ' ')) ``` **Full Example:** MyGenerator.props: ```xml ``` MyGenerator.csproj: ```xml true false ``` MyGenerator.cs: ```csharp [Generator] public class MyGenerator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext context) { var emitLoggingPipeline = context.AdditionalTextsProvider .Combine(context.AnalyzerConfigOptionsProvider) .Select((pair, ctx) => pair.Right.GetOptions(pair.Left).TryGetValue("build_metadata.AdditionalFiles.MyGenerator_EnableLogging", out var perFileLoggingSwitch) ? perFileLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase) : pair.Right.GlobalOptions.TryGetValue("build_property.MyGenerator_EnableLogging", out var emitLoggingSwitch) ? emitLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase) : false); var sourcePipeline = context.AdditionalTextsProvider.Select((file, ctx) => /* Gather build info */); context.RegisterSourceOutput(sourcePipeline.Combine(emitLoggingPipeline), (context, pair) => /* Add source */); } } ``` ### Unit Testing of Generators **User scenario**: As a generator author, I want to be able to unit test my generators to make development easier and ensure correctness. **Solution A**: The recommended approach is to use [Microsoft.CodeAnalysis.Testing](https://github.com/dotnet/roslyn-sdk/tree/main/src/Microsoft.CodeAnalysis.Testing#microsoftcodeanalysistesting) packages: - `Microsoft.CodeAnalysis.CSharp.SourceGenerators.Testing` - `Microsoft.CodeAnalysis.VisualBasic.SourceGenerators.Testing` TODO: https://github.com/dotnet/roslyn/issues/72149 ### Auto interface implementation **User scenario:** As a generator author I want to be able to implement the properties of interfaces passed as arguments to a decorator of a class automatically for a user **Solution:** Require the user to decorate the class with the `[AutoImplement]` Attribute and pass as arguments the types of the interfaces they want to self-implement themselves; The classes that implement the attribute have to be `partial class`. Provide that attribute in a `RegisterPostInitializationOutput` step. Register for callbacks on the classes with `ForAttributeWithMetadataName` using the fullyQualifiedMetadataName `FullyQualifiedAttributeName`, and use tuples (or create an equatable model) to pass along that information. The attribute could work for structs too, the example was kept simple on purpose for the workbook sample. **Example:** ```csharp public interface IUserInterface { int InterfaceProperty { get; set; } } public interface IUserInterface2 { float InterfacePropertyOnlyGetter { get; } } [AutoImplementProperties(typeof(IUserInterface), typeof(IUserInterface2))] public partial class UserClass { public string UserProp { get; set; } } ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Breaking Changes: * None currently ## Open Issues This section track other miscellaneous TODO items: **Framework targets**: May want to mention if we have framework requirements for the generators, e.g. they must target netstandard2.0 or similar. **Conventions**: (See TODO in [conventions](#conventions) section above). What standard conventions are we suggesting to users? **Feature detection**: Show how to create a generator that relies on specific target framework features, without depending on the TargetFramework property. --- ### Features Incremental Generators (`docs/features/incremental-generators.md`) # Incremental Generators ## Summary Incremental generators are a new API that replaces [v1 source generators](source-generators.md) to allow users to specify generation strategies that can be applied in a high performance way by the hosting layer. ### High Level Design Goals - Allow for a finer grained approach to defining a generator - Scale source generators to support 'Roslyn/CoreCLR' scale projects in Visual Studio - Exploit caching between fine grained steps to reduce duplicate work - Support generating more items than just source texts - Exist alongside `ISourceGenerator` based implementations ## Simple Example We begin by defining a simple incremental generator that extracts the contents of additional text files and makes their contents available as compile time `const`s. In the following section we'll go into more depth around the concepts shown. ```csharp [Generator] public class Generator : IIncrementalGenerator { public void Initialize(IncrementalGeneratorInitializationContext initContext) { // define the execution pipeline here via a series of transformations: // find all additional files that end with .txt IncrementalValuesProvider textFiles = initContext.AdditionalTextsProvider.Where(static file => file.Path.EndsWith(".txt")); // read their contents and save their name IncrementalValuesProvider<(string name, string content)> namesAndContents = textFiles.Select((text, cancellationToken) => (name: Path.GetFileNameWithoutExtension(text.Path), content: text.GetText(cancellationToken)!.ToString())); // generate a class that contains their values as const strings initContext.RegisterSourceOutput(namesAndContents, (spc, nameAndContent) => { spc.AddSource($"ConstStrings.{nameAndContent.name}", $@" public static partial class ConstStrings {{ public const string {nameAndContent.name} = ""{nameAndContent.content}""; }}"); }); } } ``` ## Implementation An incremental generator is an implementation of `Microsoft.CodeAnalysis.IIncrementalGenerator`. ```csharp namespace Microsoft.CodeAnalysis { public interface IIncrementalGenerator { void Initialize(IncrementalGeneratorInitializationContext initContext); } } ``` As with source generators, incremental generators are defined in external assemblies and passed to the compiler via the `-analyzer:` option. Implementations are required to be annotated with the `Microsoft.CodeAnalysis.GeneratorAttribute` with an optional parameter indicating the languages the generator supports: ```csharp [Generator(LanguageNames.CSharp)] public class MyGenerator : IIncrementalGenerator { ... } ``` An assembly can contain a mix of diagnostic analyzers, source generators and incremental generators. ### Pipeline based execution `IIncrementalGenerator` has an `Initialize` method that is called by the host[^1] exactly once, regardless of the number of further compilations that may occur. For instance a host with multiple loaded projects may share the same generator instance across multiple projects, and will only call `Initialize` a single time for the lifetime of the host. [^1]: Such as the IDE or the command-line compiler Rather than a dedicated `Execute` method, an Incremental Generator instead defines an immutable execution pipeline as part of initialization. The `Initialize` method receives an instance of `IncrementalGeneratorInitializationContext` which is used by the generator to define a set of transformations. ```csharp public void Initialize(IncrementalGeneratorInitializationContext initContext) { // define the execution pipeline here via a series of transformations: } ``` The defined transformations are not executed directly at initialization, and instead are deferred until the data they are using changes. Conceptually this is similar to LINQ, where a lambda expression might not be executed until the enumerable is actually iterated over: **IEnumerable**: ```csharp var squares = Enumerable.Range(1, 10).Select(i => i * i); // the code inside select is not executed until we iterate the collection foreach (var square in squares) { ... } ``` These transformations are used to form a directed graph of actions that can be executed on demand later, as the input data changes. **Incremental Generators**: ```csharp IncrementalValuesProvider textFiles = context.AdditionalTextsProvider.Where(static file => file.Path.EndsWith(".txt")); // the code in the Where(...) above will not be executed until the value of the additional texts actually changes ``` Between each transformation, the data produced is cached, allowing previously calculated values to be re-used where applicable. This caching reduces the computation required for subsequent compilations. See [caching](#caching) for more details. ### IncrementalValue\[s\]Provider<T> Input data is available to the pipeline in the form of opaque data sources, either an `IncrementalValueProvider` or `IncrementalValuesProvider` (note the plural _values_) where _T_ is the type of input data that is provided. An initial set of providers are created by the host, and can be accessed from the `IncrementalGeneratorInitializationContext` provided during initialization. The currently available providers are: - CompilationProvider - AdditionalTextsProvider - AnalyzerConfigOptionsProvider - MetadataReferencesProvider - ParseOptionsProvider *Note*: there is no provider for accessing syntax nodes. This is handled in a slightly different way. See [SyntaxValueProvider](#syntaxvalueprovider) for details. A value provider can be thought of as a 'box' that holds the value itself. An execution pipeline does not access the values in a value provider directly. ```ascii IValueProvider ┌─────────────┐ | | │ TSource │ | | └─────────────┘ ``` Instead, the generator supplies a set of transformations that are to be applied to the data contained within the provider, which in turn creates a new value provider. ### Select The simplest transformation is `Select`. This maps the value in one provider into a new provider by applying a transform to it. ```ascii IValueProvider IValueProvider ┌─────────────┐ ┌─────────────┐ │ │ Select │ │ │ TSource ├──────────────────────────►│ TResult │ │ │ │ │ └─────────────┘ └─────────────┘ ``` Generator transformations can be thought of as being conceptually somewhat similar to LINQ, with the value provider taking the place of `IEnumerable`. Transforms are created through a set of extension methods: ```csharp public static partial class IncrementalValueSourceExtensions { // 1 => 1 transform public static IncrementalValueProvider Select(this IncrementalValueProvider source, Func selector); public static IncrementalValuesProvider Select(this IncrementalValuesProvider source, Func selector); } ``` Note how the return type of these methods are also an instance of `IncrementalValue[s]Provider`. This allows the generator to chain multiple transformations together: ```ascii IValueProvider IValueProvider IValueProvider ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ Select │ │ Select │ │ │ TSource ├───────────────────────────►│ TResult1 │──────────────────────────►│ TResult2 │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ ``` Consider the following simple example: ```csharp // get the additional text provider IncrementalValuesProvider additionalTexts = initContext.AdditionalTextsProvider; // apply a 1-to-1 transform on each text, which represents extracting the path IncrementalValuesProvider transformed = additionalTexts.Select(static (text, _) => text.Path); // transform each extracted path into something else IncrementalValuesProvider prefixTransform = transformed.Select(static (path, _) => "prefix_" + path); ``` Note how `transformed` and `prefixTransform` are themselves an `IncrementalValuesProvider`. They represent the outcome of the transformation that will be applied, rather than the resulting data. ### Multi Valued providers An `IncrementalValueProvider` will always provide a single value, whereas an `IncrementalValuesProvider` may provide zero or more values. For example the `CompilationProvider` will always produce a single compilation instance, whereas the `AdditionalTextsProvider` will produce a variable number of values, depending on how many additional texts were passed to the compiler. Conceptually it is simple to think about the transformation of a single item from an `IncrementalValueProvider`: the single item has the selector function applied to it which produces a single value of `TResult`. For an `IncrementalValuesProvider` however, this transformation is more subtle. The selector function is applied multiple times, one to each item in the values provider. The results of each transformation are then used to create the values for the resulting values provider: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` It is this item-wise transformation that allows the caching to be particularly powerful in this model. Consider when the values inside `IncrementalValueProvider` change. Its likely that any given change will only change one item at a time rather than the whole collection (for example a user typing in an additional text only changes the given text, leaving the other additional texts unmodified). When this occurs the generator driver can compare the input items with the ones that were used previously. If they are considered to be equal then the transformations for those items can be skipped and the previously computed versions used instead. See [Comparing Items](#comparing-items) for more details. In the above diagram if `Item2` were to change we would execute the selector on the modified value producing a new value for `Result2`. As `Item1`and `Item3` are unchanged the driver is free to skip executing the selector and just use the cached values of `Result1` and `Result3` from the previous execution. ### Select Many In addition to the 1-to-1 transform shown above, there are also transformations that produce batches of data. For instance a given transformation may want to produce multiple values for each input. There are a set of `SelectMany` methods that allow a transformation of 1 to many, or many to many items: **1 to many:** ``` csharp public static partial class IncrementalValueSourceExtensions { public static IncrementalValuesProvider SelectMany(this IncrementalValueProvider source, Func> selector); } ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` **Many to many:** ``` csharp public static partial class IncrementalValueSourceExtensions { public static IncrementalValuesProvider SelectMany(this IncrementalValuesProvider source, Func> selector); } ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` For example, consider a set of additional XML files that contain multiple elements of the same type. The generator may want to treat each element as a distinct item for generation, effectively splitting a single additional file into multiple sub-items. ``` csharp // get the additional text provider IncrementalValuesProvider additionalTexts = initContext.AdditionalTextsProvider; // extract each element from each additional file IncrementalValuesProvider elements = additionalTexts.SelectMany(static (text, _) => /*transform text into an array of MyElementType*/); // now the generator can consider the union of elements in all additional texts, without needing to consider multiple files IncrementalValuesProvider transformed = elements.Select(static (element, _) => /*transform the individual element*/); ``` ### Where Where allows the author to filter the values in a value provider by a given predicate. Where is actually a specific form of select many, where each input transforms to exactly 1 or 0 outputs. However, as it is such a common operation it is provided as a primitive transformation directly. ``` csharp public static partial class IncrementalValueSourceExtensions { public static IncrementalValuesProvider Where(this IncrementalValuesProvider source, Func predicate); } ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` An obvious use case is to filter out inputs the generator knows it isn't interested in. For example, the generator will likely want to filter additional texts on file extensions: ```csharp // get the additional text provider IncrementalValuesProvider additionalTexts = initContext.AdditionalTextsProvider; // filter additional texts by extension IncrementalValuesProvider xmlFiles = additionalTexts.Where(static (text, _) => text.Path.EndsWith(".xml", StringComparison.OrdinalIgnoreCase)); ``` ### Collect When performing transformations on a value provider with multiple items, it can often be useful to view the items as a single collection rather than one item at a time. For this there is the `Collect` transformation. `Collect` transforms an `IncrementalValuesProvider` to an `IncrementalValueProvider>`. Essentially it transforms a multi-valued source into a single value source with an array of all the items. ```csharp public static partial class IncrementalValueSourceExtensions { IncrementalValueProvider> Collect(this IncrementalValuesProvider source); } ``` ```ascii IncrementalValuesProvider IncrementalValueProvider> ┌───────────┐ ┌─────────────────────────┐ │ │ Collect │ │ │ TSource ├─────────────────────────────────►│ ImmutableArray │ │ │ │ │ └───────────┘ └─────────────────────────┘ 3 Items Single Item Item1 [Item1, Item2, Item3] Item2 Item3 ``` ```csharp // get the additional text provider IncrementalValuesProvider additionalTexts = initContext.AdditionalTextsProvider; // collect the additional texts into a single item IncrementalValueProvider collected = additionalTexts.Collect(); // perform a transformation where you can access all texts at once var transform = collected.Select(static (texts, _) => /* ... */); ``` ### Multi-path pipelines The transformations described so far are all effectively single-path operations: while there may be multiple items in a given provider, each transformation operates on a single input value provider and produces a single derived output provider. While sufficient for simple operations, it is often necessary to combine the values from multiple input providers or use the results of a transformation multiple times. For this there are a set of transformations that split and combine a single path of transformations into a multi-path pipeline. ### Split It is possible to split the output of a transformation into multiple parallel inputs. Rather than having a dedicated transformation this can be achieved by simply using the same value provider as the input to multiple transforms. ```ascii IncrementalValueProvider ┌───────────┐ Select │ │ IncrementalValueProvider ┌───────────────────────►│ TResult │ ┌───────────┐ │ │ │ │ │ │ └───────────┘ │ TSource ├─────────────┤ │ │ │ └───────────┘ │ IncrementalValuesProvider │ ┌───────────┐ │ SelectMany │ │ └─────────────────────────────►│ TResult2 │ │ │ └───────────┘ ``` Those transforms can then be used as the inputs to new single path transforms, independent of one another. For example: ```csharp // get the additional text provider IncrementalValuesProvider additionalTexts = context.AdditionalTextsProvider; // apply a 1-to-1 transform on each text, extracting the path IncrementalValuesProvider transformed = additionalTexts.Select(static (text, _) => text.Path); // split the processing into two paths of derived data IncrementalValuesProvider nameTransform = transformed.Select(static (path, _) => "prefix_" + path); IncrementalValuesProvider extensionTransform = transformed.Select(static (path, _) => Path.ChangeExtension(path, ".new")); ``` `nameTransform` and `extensionTransform` produce different values for the same set of additional text inputs. For example if there was an additional file called `file.txt` then `nameTransform` would produce the string `prefix_file.txt` where `extensionTransform` would produce the string `file.new`. When the value of the additional file changes, the subsequent values produced may or may not differ. For example if the name of the additional file was changed to `file.xml` then `nameTransform` would now produce `prefix_file.xml` whereas `extensionTransform` would still produce `file.new`. Any child transform with input from `nameTransform` would be re-run with the new value, but any child of `extensionTransform` would use the previously cached version as it's input hasn't changed. ### Combine Combine is the most powerful, but also most complicated transformation. It allows a generator to take two input providers and create a single unified output provider. **Single-value to single-value**: ```csharp public static partial class IncrementalValueSourceExtensions { IncrementalValueProvider<(TLeft Left, TRight Right)> Combine(this IncrementalValueProvider provider1, IncrementalValueProvider provider2); } ``` When combining two single value providers, the resulting node is conceptually easy to understand: a new value provider that contains a `Tuple` of the two input items. ```ascii IncrementalValueProvider ┌───────────┐ │ │ │ TSource1 ├────────────────┐ │ │ │ IncrementalValueProvider<(TSource1, TSource2)> └───────────┘ │ Single Item │ ┌────────────────────────┐ │ Combine │ │ Item1 ├─────────────────────────────────────────►│ (TSource1, TSource2) │ │ │ │ IncrementalValueProvider │ └────────────────────────┘ ┌───────────┐ │ Single Item │ │ │ │ TSource2 ├────────────────┘ (Item1, Item2) │ │ └───────────┘ Single Item Item2 ``` **Multi-value to single-value:** ```csharp public static partial class IncrementalValueSourceExtensions { IncrementalValuesProvider<(TLeft Left, TRight Right)> Combine(this IncrementalValuesProvider provider1, IncrementalValueProvider provider2); } ``` When combining a multi value provider to a single value provider, however, the semantics are a little more complicated. The resulting multi-valued provider produces a series of tuples: the left hand side of each tuple is the value produced from the multi-value input, while the right hand side is always the same single value from the single value provider input. ```ascii IncrementalValuesProvider ┌───────────┐ │ │ │ TSource1 ├────────────────┐ │ │ │ └───────────┘ │ 3 Items │ IncrementalValuesProvider<(TSource1, TSource2)> │ LeftItem1 │ ┌────────────────────────┐ LeftItem2 │ Combine │ │ LeftItem3 ├─────────────────────────────────────────►│ (TSource1, TSource2) │ │ │ │ │ └────────────────────────┘ IncrementalValueProvider │ 3 Items ┌───────────┐ │ │ │ │ (LeftItem1, RightItem) │ TSource2 ├────────────────┘ (LeftItem2, RightItem) │ │ (LeftItem3, RightItem) └───────────┘ Single Item RightItem ``` **Multi-value to multi-value:** As shown by the definitions above it is not possible to combine a multi-value source to another multi-value source. The resulting cross join would potentially contain a large number of values, so the operation is not provided by default. Instead, an author can call `Collect()` on one of the input multi-value providers to produce a single-value provider that can be combined as above. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` With the above transformations the generator author can now take one or more inputs and combine them into a single source of data. For example: ```csharp // get the additional text provider IncrementalValuesProvider additionalTexts = initContext.AdditionalTextsProvider; // combine each additional text with the parse options IncrementalValuesProvider<(AdditionalText, ParseOptions)> combined = additionalTexts.Combine(initContext.ParseOptionsProvider); // perform a transform on each text, with access to the options var transformed = combined.Select(static (pair, _) => { AdditionalText text = pair.Left; ParseOptions parseOptions = pair.Right; // do the actual transform ... }); ``` If either of the inputs to a combine change, then subsequent transformation will re-run. However, the caching is considered on a pairwise basis for each output tuple. For instance, in the above example, if only additional text changes the subsequent transform will only be run for the text that changed. The other text and parse options pairs are skipped and their previously computed value are used. If the single value changes, such as the parse options in the example, then the transformation is executed for every tuple. ### SyntaxValueProvider Syntax Nodes are not available directly through a value provider. Instead, a generator author uses the special `SyntaxValueProvider` (provided via the `IncrementalGeneratorInitializationContext.SyntaxProvider`) to create a dedicated input node that instead exposes a sub-set of the syntax they are interested in. The syntax provider is specialized in this way to achieve a desired level of performance. #### CreateSyntaxProvider Currently the provider exposes a single method `CreateSyntaxProvider` that allows the author to construct an input node. ```csharp public readonly struct SyntaxValueProvider { public IncrementalValuesProvider CreateSyntaxProvider(Func predicate, Func transform); } ``` Note how this takes _two_ lambda parameters: one that examines a `SyntaxNode` in isolation, and a second one that can then use the `GeneratorSyntaxContext` to access a semantic model and transform the node for downstream usage. It is because of this split that performance can be achieved: as the driver is aware of which nodes are chosen for examination, it can safely skip the first `predicate` lambda when a syntax tree remains unchanged. The driver will still re-run the second `transform` lambda even for nodes in unchanged files, as a change in one file can impact the semantic meaning of a node in another file. Consider the following syntax trees: ```csharp // file1.cs public class Class1 { public int Method1() => 0; } // file2.cs public class Class2 { public Class1 Method2() => null; } // file3.cs public class Class3 {} ``` As an author I can make an input node that extracts the return type information ```csharp // create a syntax provider that extracts the return type kind of method symbols var returnKinds = initContext.SyntaxProvider.CreateSyntaxProvider(static (n, _) => n is MethodDeclarationSyntax, static (n, _) => ((IMethodSymbol)n.SemanticModel.GetDeclaredSymbol(n.Node)).ReturnType.Kind); ``` Initially the `predicate` will run for all syntax nodes, and select the two `MethodDeclarationSyntax` nodes `Method1()` and `Method2()`. These are then passed to the `transform` where the semantic model is used to obtain the method symbol and extract the kind of the return type for the method. `returnKinds` will contain two values, both `NamedType`. Now imagine that `file3.cs` is edited: ```csharp // file3.cs public class Class3 { public int field; } ``` The `predicate` will only be run for syntax nodes inside `file3.cs`, and will not return any as it still doesn't contain any method symbols. The `transform` however will still be run again for the two methods from `Class1` and `Class2`. To see why it was necessary to re-run the `transform` consider the following edit to `file1.cs` where we change the class's name: ```csharp // file1.cs public class Class4 { public int Method1() => 0; } ``` The `predicate` will be re-run for `file1.cs` as it has changed, and will pick out the method symbol `Method1()` again. Next, because the `transform` is re-run for _all_ the methods, the return type kind for `Method2()` is correctly changed to `ErrorType` as `Class1` no longer exists. Note that we didn't need to run the `predicate` over for nodes in `file2.cs` even though they referenced something in `file1.cs`. Because the first check is purely _syntactic_ we can be sure the results for `file2.cs` would be the same. While it may seem unfortunate that the driver must run the `transform` for all selected syntax nodes, if it did not it could end up producing incorrect data due to cross file dependencies. Because the initial syntactic check allows the driver to substantially filter the number of nodes on which the semantic checks have to be re-run, significantly improved performance characteristics are still observed when editing a syntax tree. #### ForAttributeWithMetadataName (FAWMN) One extremely common action we observe generators being written for is taking actions driven on attributes applied to specific syntax constructs. ```csharp public readonly struct SyntaxValueProvider { public IncrementalValuesProvider ForAttributeWithMetadataName( string fullyQualifiedMetadataName, Func predicate, Func transform); } ``` This area is particularly nice for optimization, as we can efficiently eliminate a significant number of syntax nodes and edits before even needing to call the provided `predicate` from the user, avoiding realizing a significant number of `SyntaxNode` instances. Roslyn can even further optimize this by tracking whether or not a given attribute could possibly be the attribute the generator cares about by maintaining a small index and comparing type names as an initial heuristic. This index is cheap to maintain and, importantly, can only have false positives, not false negatives. This allows us to eliminate 99% of syntax in a Compilation from ever needing to be checked for semantic information (to eliminate false positives from the heuristic cache) or by the user `predicate` function (saving a significant number of allocations of `SyntaxNode` instances). Given this, when at all possible, it is recommended to use attributes to drive source generators, rather than other syntax constructs. Real world testing has indicated this approach is usually 99x more efficient than `CreateSyntaxProvider`, even when the generator is otherwise not well-behaved; some pathological scenarios are even more efficient than that. Attributes are provided by the user as the fully-qualified metadata name, without the assembly name portion. For example, given the C# type `My.Namespace.MyAttribute`, the fully-qualified metadata-name would be ``My.Namespace.MyAttribute`1``. Given that attributes are usually restricted to specific constructs by an `AttributeUsage` attribute, it is common that the `predicate` a user provides will simply return `true`. For the transformation step, everything stated in the [previous section](#createsyntaxprovider) is still relevant; that step will still be rerun with every change to ensure that changed semantics are observed. ## Outputting values At some point in the pipeline the author will want to actually use the transformed data to produce an output, such as a `SourceText`. There are a set of `Register...Output` methods on the `IncrementalGeneratorInitializationContext` that allow the generator author to construct an output from a series of transformations. These output registrations are terminal, in that the they do not return a value provider and can have no further transformations applied to them. However an author is free to register multiple outputs of the same type with different input transformations. The set of output methods are - RegisterSourceOutput - RegisterImplementationSourceOutput - RegisterPostInitializationOutput **RegisterSourceOutput**: `RegisterSourceOutput` allows a generator author to produce source files and diagnostics that will be included in the users compilation. As input, it takes a `Value[s]Provider` and an `Action` that will be invoked for every value in the value provider. ``` csharp public static partial class IncrementalValueSourceExtensions { public void RegisterSourceOutput(IncrementalValueProvider source, Action action); public void RegisterSourceOutput(IncrementalValuesProvider source, Action action); } ``` The provided `SourceProductionContext` can be used to add source files and report diagnostics: ```csharp public readonly struct SourceProductionContext { public CancellationToken CancellationToken { get; } public void AddSource(string hintName, string source); public void ReportDiagnostic(Diagnostic diagnostic); } ``` For example, a generator can extract out the set of paths for the additional files and create a method that prints them out: ``` csharp // get the additional text provider IncrementalValuesProvider additionalTexts = initContext.AdditionalTextsProvider; // apply a 1-to-1 transform on each text, extracting the path IncrementalValuesProvider transformed = additionalTexts.Select(static (text, _) => text.Path); // collect the paths into a batch IncrementalValueProvider> collected = transformed.Collect(); // take the file paths from the above batch and make some user visible syntax initContext.RegisterSourceOutput(collected, static (sourceProductionContext, filePaths) => { sourceProductionContext.AddSource("additionalFiles.cs", @" namespace Generated { public class AdditionalTextList { public static void PrintTexts() { System.Console.WriteLine(""Additional Texts were: " + string.Join(", ", filePaths) + @" ""); } } }"); }); ``` **RegisterImplementationSourceOutput**: `RegisterImplementationSourceOutput` works in the same way as `RegisterSourceOutput` but declares that the source produced has no semantic impact on user code from the point of view of code analysis. This allows a host such as the IDE, to chose not to run these outputs as a performance optimization. A host that produces executable code will always run these outputs. **RegisterPostInitializationOutput**: `RegisterPostInitializationOutput` allows a generator author to provide source code immediately after initialization has run. It takes no inputs, and so cannot refer to any source code written by the user, or any other compiler inputs. Post initialization source is included in the Compilation before any other transformations are run, meaning that it will be visible as part of the rest of the regular execution pipeline, and an author may ask semantic questions about it. It is particularly useful for adding attribute definitions to the users' source code. These can then be applied by the user in their code, and the generator may find the attributed code via the semantic model. ## Handling Cancellation Incremental generators are designed to be used in interactive hosts such as an IDE. As such, it is critically important that generators respect and respond to the passed-in cancellation tokens. In general, it is likely that the amount of user computation performed per transformation is low, but often will be calling into Roslyn APIs that may have a significant performance impact. As such the author should always forward the provided cancellation token to any Roslyn APIs that accept it. For example, when retrieving the contents of an additional file, the token should be passed into `GetText(...)`: ```csharp public void Initialize(IncrementalGeneratorInitializationContext context) { var txtFiles = context.AdditionalTextsProvider.Where(static f => f.Path.EndsWith(".txt", StringComparison.OrdinalIgnoreCase)); // ensure we forward the cancellation token to GetText var fileContents = txtFiles.Select(static (file, cancellationToken) => file.GetText(cancellationToken)); } ``` This will ensure that an incremental generator correctly and quickly responds to cancellation requests and does not cause delays in the host. If the generator author is doing something expensive, such as looping over values, they should regularly check for cancellation themselves. It is recommend that the author use `CancellationToken.ThrowIfCancellationRequested()` at regular intervals, and allow the host to re-run them, rather than attempting to save partially generated results which can be extremely difficult to author correctly. ```csharp public void Initialize(IncrementalGeneratorInitializationContext context) { var txtFilesArray = context.AdditionalTextsProvider.Where(static f => f.Path.EndsWith(".txt", StringComparison.OrdinalIgnoreCase)).Collect(); var expensive = txtFilesArray.Select(static (files, cancellationToken) => { foreach (var file in files) { // check for cancellation so we don't hang the host cancellationToken.ThrowIfCancellationRequested(); // perform some expensive operation (ideally passing in the token as well) ExpensiveOperation(file, cancellationToken); } }); } ``` ## Caching While the finer grained steps allow for some coarse control of output types via the generator host, the performance benefits are only really seen when the driver can cache the outputs from one pipeline step to the next. While we have generally said that the execute method in an `ISourceGenerator` should be deterministic, incremental generators actively _require_ this property to be true. When calculating the required transformations to be applied as part of a step, the generator driver is free to look at inputs it has seen before and use previous computed and cached values of the transformation for these inputs. Consider the following transformation: ```csharp IValuesProvider transform = context.AdditionalTextsProvider .Select(static (t, _) => t.Path) .Select(static (p, _) => "prefix_" + p); ``` During the first execution of the pipeline each of the two lambdas will be executed for each additional file: AdditionalText | Select1 | Select2 ------------------------|------------|----------------- Text{ Path: "abc.txt" } | "abc.txt" | "prefix_abc.txt" Text{ Path: "def.txt" } | "def.txt" | "prefix_def.txt" Text{ Path: "ghi.txt" } | "ghi.txt" | "prefix_ghi.txt" Now consider the case where in some future iteration, the first additional file has changed and has a different path, and the second file has changed, but kept its path the same. AdditionalText | Select1 | Select2 -----------------------------|------------|----------- **Text{ Path: "diff.txt" }** | | **Text{ Path: "def.txt" }** | | Text{ Path: "ghi.txt" } | | The generator would run select1 on the first and second files, producing "diff.txt" and "def.txt" respectively. However, it would not need to re-run the select for the third file, as the input has not changed. It can just use the previously cached value. AdditionalText | Select1 | Select2 -----------------------------|----------------------|----------- **Text{ Path: "diff.txt" }** | **"diff.txt"** (new) | **Text{ Path: "def.txt" }** | **"def.txt"** (new) | Text{ Path: "ghi.txt" } | "ghi.txt" (reuse) | Next the driver would look to run Select2. It would operate on `"diff.txt"` producing `"prefix_diff.txt"`, but when it comes to `"def.txt"` it can observe that the item produced was the same as the last iteration. Even though the original input (`Text{ Path: "def.txt" }`) was changed, the result of Select1 on it was the same. Thus there is no need to re-run Select2 on `"def.txt"` as it can just use the cached value from before. Similarly the cached state of "ghi.txt" can be used. AdditionalText | Select1 | Select2 -----------------------------|----------------|---------------------- **Text{ Path: "diff.txt" }** | **"diff.txt"** | **"prefix_diff.txt"** (new) **Text{ Path: "def.txt" }** | **"def.txt"** | "prefix_def.txt" (reuse) Text{ Path: "ghi.txt" } | "ghi.txt" | "prefix_ghi.txt" (reuse) In this way, only changes that are consequential flow through the pipeline, and duplicate work is avoided. If a generator only relies on `AdditionalTexts` then the driver knows there can be no work to be done when a `SyntaxTree` changes. ### Comparing Items For a user-provided result to be comparable across iterations, there needs to be some concept of equivalence. By default the host will use `EqualityComparer` to determine equivalence. There are obviously times where this is insufficient, and there exists an extension method that allows the author to supply a comparer that should be used when comparing values for the given transformation: ```csharp public static partial class IncrementalValueProviderExtensions { public static IncrementalValueProvider WithComparer(this IncrementalValueProvider source, IEqualityComparer comparer); public static IncrementalValuesProvider WithComparer(this IncrementalValuesProvider source, IEqualityComparer comparer); } ``` Allowing the generator author to specify a given comparer. ```csharp var withComparer = context.AdditionalTextsProvider .Select(static t => t.Path) .WithComparer(myComparer); ``` Note that the comparer is on a per-transformation basis, meaning an author can specify different comparers for different parts of the pipeline. ```csharp var select = context.AdditionalTextsProvider.Select(static t => t.Path); var noCompareSelect = select.Select(...); var compareSelect = select.WithComparer(myComparer).Select(...); ``` The same select node can have no comparer when acting as input to one transformation, and still provide one when acting as input to a different transformation. The host will only invoke the given comparer when the item it is derived from has been modified. When the input value is new or being removed, or the input transformation was determined to be cached (possibly by a provided comparer) the given comparer is not considered. ### Authoring a cache friendly generator Much of the success of an incremental generator will depend on creating an optimal pipeline that is amenable to caching. This section includes some general tips and best practices to achieve that **Extract out information early**: It is best to get the information out of the inputs as early as possible in the pipeline. This ensures the host is not caching large, expensive objects such as symbols. **Use value types where possible**: Value types are more amenable to caching and usually have well defined and easy to understand comparison semantics. **Use multiple transformations**: The more transformations you break the operations into, the more opportunities there are to cache. Think of transformations as being 'check points' in the execution graph. The more check points the more chances there are to match a cached value and skip any remaining work. **Build a data model**: Rather than trying to pass each input item into a `Register...Output` method, consider building a data model to be the final item passed to the output. Use the transformations to manipulate the data model, and have well defined equality that allows you to correctly compare between revisions of the model. This also makes testing the final `Register...Outputs` significantly simpler: you can just call the method with a dummy data model and check the generated code, rather than trying to emulate the incremental transformations. **Consider the order of combines**: Ensure that you are only combining the minimal amount of information needed (this comes back to 'Extract out information early'). Consider the following (incorrect) combine where the basic inputs are combined, then used to generate some source: ```csharp public void Initialize(IncrementalGeneratorInitializationContext context) { var compilation = context.CompilationProvider; var texts = context.AdditionalTextsProvider; // Don't do this! var combined = texts.Combine(compilation); context.RegisterSourceOutput(combined, static (spc, pair) => { var assemblyName = pair.Right.AssemblyName; // produce source ... }); ``` Any time the compilation changes, which it will frequently as the user is typing in the IDE, then `RegisterSourceOutput` will get re-run. Instead, look up the compilation dependant information first, then combine _that_ with the additional files: ```csharp public void Initialize(IncrementalGeneratorInitializationContext context) { var assemblyName = context.CompilationProvider.Select(static (c, _) => c.AssemblyName); var texts = context.AdditionalTextsProvider; var combined = texts.Combine(assemblyName); context.RegisterSourceOutput(combined, (spc, pair) => { var assemblyName = pair.Right; // produce source ... }); } ``` Now, as the user types in the IDE, the `assemblyName` transform will re-run, but is very cheap and quickly returns what is likely the same value each time. That means that unless the additional texts have also changed, the host does not need to re-run the combine or re-generate any of the source. --- ### Features Interceptors (`docs/features/interceptors.md`) # Interceptors ## Summary [summary]: #summary *Interceptors* is a C# compiler feature, first shipped experimentally in .NET 8, with stable support in .NET 9.0.2xx SDK and later. An *interceptor* is a method which can declaratively substitute a call to an *interceptable* method with a call to itself at compile time. This substitution occurs by having the interceptor declare the source locations of the calls that it intercepts. This provides a limited facility to change the semantics of existing code by adding new code to a compilation (e.g. in a source generator). ```cs using System; using System.Runtime.CompilerServices; var c = new C(); c.InterceptableMethod(1); // L1: prints "interceptor 1" c.InterceptableMethod(1); // L2: prints "other interceptor 1" c.InterceptableMethod(2); // L3: prints "other interceptor 2" c.InterceptableMethod(1); // prints "interceptable 1" class C { public void InterceptableMethod(int param) { Console.WriteLine($"interceptable {param}"); } } // generated code static class D { [InterceptsLocation(version: 1, data: "...(refers to the call at L1)")] public static void InterceptorMethod(this C c, int param) { Console.WriteLine($"interceptor {param}"); } [InterceptsLocation(version: 1, data: "...(refers to the call at L2)")] [InterceptsLocation(version: 1, data: "...(refers to the call at L3)")] public static void OtherInterceptorMethod(this C c, int param) { Console.WriteLine($"other interceptor {param}"); } } ``` ## Detailed design [design]: #detailed-design ### InterceptsLocationAttribute A method indicates that it is an *interceptor* by adding one or more `[InterceptsLocation]` attributes. These attributes refer to the source locations of the calls it intercepts. ```cs namespace System.Runtime.CompilerServices { [AttributeUsage(AttributeTargets.Method, AllowMultiple = true)] public sealed class InterceptsLocationAttribute(int version, string data) : Attribute { } } ``` Any "ordinary method" (i.e. with `MethodKind.Ordinary`) can have its calls intercepted. In addition to "ordinary" forms `M()` and `receiver.M()`, a call within a conditional access, e.g. of the form `receiver?.M()` can be intercepted. A call whose receiver is a pointer member access, e.g. of the form `ptr->M()`, can also be intercepted. `[InterceptsLocation]` attributes included in source are emitted to the resulting assembly, just like other custom attributes. File-local declarations of this type (`file class InterceptsLocationAttribute`) are valid and usages are recognized by the compiler when they are within the same file and compilation. A generator which needs to declare this attribute should use a file-local declaration to ensure it doesn't conflict with other generators that need to do the same thing. #### Location encoding The arguments to `[InterceptsLocation]` are: 1. a version number. The compiler may introduce new encodings for the location in the future, with corresponding new version numbers. 2. an opaque data string. This is not intended to be human-readable. The "version 1" data encoding is a base64-encoded string consisting of the following data: - 16 byte xxHash128 content checksum of the file containing the intercepted call. - int32 in little-endian format for the position (i.e. `SyntaxNode.Position`) of the call in syntax. - utf-8 string data containing a display file name, used for error reporting. #### Position The location of the call is the location of the simple name syntax which denotes the interceptable method. For example, in `app.MapGet(...)`, the name syntax for `MapGet` would be considered the location of the call. For a static method call like `System.Console.WriteLine(...)`, the name syntax for `WriteLine` is the location of the call. If we allow intercepting calls to property accessors in the future (e.g `obj.Property`), we would also be able to use the name syntax in this way. #### Attribute creation Roslyn provides an API `GetInterceptableLocation(this SemanticModel, InvocationExpressionSyntax, CancellationToken)` for inserting `[InterceptsLocation]` into generated source code. We recommend that source generators depend on this API in order to intercept calls. See https://github.com/dotnet/roslyn/issues/72133 for further details. ### Non-invocation method usages Conversion to delegate type, address-of, etc. usages of methods cannot be intercepted. Interception can only occur for calls to ordinary member methods--not constructors, delegates, properties, local functions, operators, etc. Support for more member kinds may be added in the future. ### Arity Interceptors cannot be declared in generic types at any level of nesting. Interceptors must either be non-generic, or have arity equal to the sum of the arity of the original method's arity and containing type arities. For example: ```cs Grandparent.Parent.Original(1, false, "a"); // L1 class Grandparent { class Parent { public static void Original(T1 t1, T2 t2, T3 t3) { } } } class Interceptors { [InterceptsLocation(1, "..(refers to call at L1)")] public static void Interceptor(T1 t1, T2 t2, T3 t3) { } } ``` When an interceptor is generic, the type arguments from the original containing types and method are passed as type arguments to the interceptor, from outermost to innermost. In the above scenario, the interceptor receives `` as type arguments. If the interceptor type parameters have constraints which are violated by these type arguments, a compile-time error occurs. This substitution allows interceptors to use type parameters which aren't in scope at its declaration site. ```cs using System.Runtime.CompilerServices; class C { public static void InterceptableMethod(T1 t) => throw null!; } static class Program { public static void M(T2 t) { C.InterceptableMethod(t); // L1 } } static class D { [InterceptsLocation(1, "..(refers to call at L1)")] public static void Interceptor1(T2 t) => throw null!; } ``` ### Signature matching When a call is intercepted, the interceptor and interceptable methods must meet the signature matching requirements detailed below: - When an interceptable instance method is compared to a static interceptor method (including a classic extension method), we use the method as if it is an extension in reduced form for comparison. The first parameter of the static method is compared to the instance method `this` parameter. - The implementation currently requires the interceptor to be an extension method for this comparison to work. We plan on addressing this before releasing .NET 8. - The returns and parameters, including the `this` parameter, must have the same ref kinds and types. - A warning is reported instead of an error if a type difference is found where the types are not distinct to the runtime. For example, `object` and `dynamic`. - No warning or error is reported for a *safe* nullability difference, such as when the interceptable method accepts a `string` parameter, and the interceptor accepts a `string?` parameter. - Method names and parameter names are not required to match. - Parameter default values are not required to match. When intercepting, default values on the interceptor method are ignored. - `params` modifiers are not required to match. - `scoped` modifiers and `[UnscopedRef]` must be equivalent. - In general, attributes which normally affect the behavior of the call site, such as `[CallerLineNumber]` are ignored on the interceptor of an intercepted call. - The only exception to this is when the attribute affects "capabilities" of the method in a way that affects safety, such as with `[UnscopedRef]`. Such attributes are required to match across interceptable and interceptor methods. Arity does not need to match between intercepted and interceptor methods. In other words, it is permitted to intercept a generic method with a non-generic interceptor. ### Conflicting interceptors If more than one interceptor refers to the same location, it is a compile-time error. If an `[InterceptsLocation]` attribute is found in the compilation which does not refer to the location of an explicit method call, it is a compile-time error. ### Interceptor accessibility An interceptor must be accessible at the location where interception is occurring. An interceptor contained in a file-local type is permitted to intercept a call in another file, even though the interceptor is not normally *visible* at the call site. This allows generator authors to avoid *polluting lookup* with interceptors, helps avoid name conflicts, and prevents use of interceptors in *unintended positions* from the interceptor author's point-of-view. We may also want to consider adjusting behavior of `[EditorBrowsable]` to work in the same compilation. ### Struct receiver capture An interceptor whose `this` parameter takes a struct by-reference can generally be used to intercept a struct instance method call, assuming the methods are compatible per [Signature matching](#signature-matching). This includes situations where the receiver must be implicitly captured to temp before the invocation, even if such capture is not permitted when the interceptor is called directly. See also [12.8.9.3 Extension method invocations](https://github.com/dotnet/csharpstandard/blob/standard-v7/standard/expressions.md#12893-extension-method-invocations) in the standard. ```cs using System.Runtime.CompilerServices; struct S { public void Original() { } } static class Program { public static void Main() { new S().Original(); // L1: interception is valid, no errors. new S().Interceptor(); // error CS1510: A ref or out value must be an assignable variable } } static class D { [InterceptsLocation(1, "..(refers to call to 'Original()' at L1)")] public static void Interceptor(this ref S s) } ``` The reason we permit implicit receiver capture for the above intercepted call is: we want intercepting to be possible even when the interceptor author doesn't own the original receiver type. If we didn't do this, then intercepting `Original()` in the above example would only be possible by adding instance members to `struct S`. ### Editor experience Interceptors are treated like a post-compilation step in this design. Diagnostics are given for misuse of interceptors, but some diagnostics are only given in the command-line build and not in the IDE. There is limited traceability in the editor for which calls in a compilation are actually being intercepted. `GetInterceptorMethod(this SemanticModel, InvocationExpressionSyntax, CancellationToken)` enables analyzers to determine if a call is being intercepted, and if so, which method is intercepting the call. See https://github.com/dotnet/roslyn/issues/72093 for further details. ### User opt-in To use interceptors, the user project must specify the property ``. This is a list of namespaces which are allowed to contain interceptors. ```xml $(InterceptorsNamespaces);Microsoft.AspNetCore.Http.Generated;MyLibrary.Generated ``` It's expected that each entry in the `InterceptorsNamespaces` list roughly corresponds to one source generator. Well-behaved components are expected to not insert interceptors into namespaces they do not own. For compatibility, the property `` can be used as an alias for ``. If both properties have non-empty values, they will be concatenated together in the order `$(InterceptorsNamespaces);$(InterceptorsPreviewNamespaces)` when passed to the compiler. ### Implementation strategy During the binding phase, `InterceptsLocationAttribute` usages are decoded and the related data for each usage are collected in a `ConcurrentSet` on the compilation: - intercepted file-path and location - attribute location - attributed method symbol At this time, diagnostics are reported for the following conditions: - problems specific to the attributed interceptor method itself, for example, that it is not an ordinary method. - syntactic problems specific to the referenced location, for example, that it does not refer to an applicable simple name as defined in [Position](#position) subsection. During the lowering phase, when a given `BoundCall` is lowered: - we check if its syntax contains an applicable simple name - if so, we lookup whether it is being intercepted, based on data about `InterceptsLocationAttribute` collected during the binding phase. - if it is being intercepted, we perform an additional step after lowering of the receiver and arguments is completed: - substitute the interceptable method with the interceptor method on the `BoundCall`. - if the interceptor is a classic extension method, and the interceptable method is an instance method, we adjust the `BoundCall` to use the receiver as the first argument of the call, "pushing" the other arguments forward, similar to the way it would have bound if the original call were to an extension method in reduced form. At this time, diagnostics are reported for the following conditions: - incompatibility between the interceptor and interceptable methods, for example, in their signatures. - *duplicate* `[InterceptsLocation]`, that is, multiple interceptors which intercept the same call. - interceptor is not accessible at the call site. --- ### Features Local Functions (`docs/features/local-functions.md`) Local Functions =============== This feature is to support the definition of functions in block scope. TODO: _WRITE SPEC_ Syntax Grammar ============== This grammar is represented as a diff from the current spec grammar. ```diff declaration_statement : local_variable_declaration ';' | local_constant_declaration ';' + | local_function_declaration ; +local_function_declaration + : local_function_header local_function_body + ; +local_function_header + : local_function_modifier* return_type identifier type_parameter_list? + '(' formal_parameter_list? ')' type_parameter_constraints_clauses + ; +local_function_modifier + : 'async' + | 'unsafe' + ; +local_function_body + : block + | arrow_expression_body + ; ``` Local functions may use variables defined in the enclosing scope. The current implementation requires that every variable read inside a local function be definitely assigned, as if executing the local function at its point of definition. Also, the local function definition must have been "executed" at any use point. After experimenting with that a bit (for example, it is not possible to define two mutually recursive local functions), we've since revised how we want the definite assignment to work. The revision (not yet implemented) is that all local variables read in a local function must be definitely assigned at each invocation of the local function. That's actually more subtle than it sounds, and there is a bunch of work remaining to make it work. Once it is done you'll be able to move your local functions to the end of its enclosing block. The new definite assignment rules are incompatible with inferring the return type of a local function, so we'll likely be removing support for inferring the return type. Unless you convert a local function to a delegate, capturing is done into frames that are value types. That means you don't get any GC pressure from using local functions with capturing. --- ### Features Local Functions.Test (`docs/features/local-functions.test.md`) # Interaction with other language features: General concerns: - [ ] Error handling/recovery - [ ] Errors in parsing - [ ] Error handling for semantic errors (e.g. ambiguous lookup, inaccessible lookup, wrong kind of thing found, instance vs static thing found, wrong type for the context, value vs variable) - [ ] Public interface of compiler APIs, backcompat scenarios - [ ] Determinism - [ ] Atomicity - [ ] Edit-and-continue - [ ] Completeness of the specification as a guide for testing (e.g. is the spec complete enough to suggest what the compiler should do in each scenario?) - [ ] Other external documentation - [ ] Performance Types and members: - [ ] Access modifiers (public, protected, internal, protected internal, private), static modifier - [ ] Parameter modifiers (ref, out, params) - [ ] Attributes (including security attribute) - [ ] Generics (type arguments, constraints, variance) - [ ] default value - [ ] partial classes - [ ] literals - [ ] enum (implicit vs. explicit underlying type) - [ ] expression trees - [ ] Iterators - [ ] Initializers (object, collection, dictionary) - [ ] array (single- or multi-dimensional, jagged, initilalizer) - [ ] Expression-bodied methods/properties/... - [ ] Extension methods - [ ] Partial method - [ ] Named and optional parameters - [ ] String interpolation - [ ] Properties (read-write, read-only, write-only, auto-property, expression-bodied) - [ ] Interfaces (implicit vs. explicit interface member implementation) - [ ] delegates - [ ] Multi-declaration Code: - [ ] Operators (see Eric's list) - [ ] Operator overloading - [ ] Lvalues: the synthesized fields are mutable - [ ] Ref / out parameters - [ ] Compound operators (+=, /=, etc ..) - [ ] Assignment exprs - [ ] lambdas (capture of parameters or locals, target typing) - [ ] execution order - [ ] Target typing (var, lambdas, integrals) - [ ] Type inference - [ ] Conversions - [ ] Implicit (identity, implicit numeric, implicit enumeration, implicit nullable, null literal, implicit reference, boxing, implicit dynamic, implicit constant, user-defined implicit conversion, anonymous function, method group) - [ ] Explicit (numeric, enumeration, nullable, reference, unboxing, dynamic, user-defined) - [ ] Anonymous functions - [ ] nullable (wrapping, unwrapping) - [ ] OHI - [ ] inheritance (virtual, override, abstract, new) - [ ] overload resolution - [ ] Anonymous types - [ ] Unsafe code - [ ] LINQ - [ ] constructors, properties, indexers, events, operators, and destructors. - [X] Async - [X] Var Misc: - [ ] reserved keywords (sometimes contextual) - [ ] pre-processing directives - [ ] COM interop --- ### Features Local Functions.Work (`docs/features/local-functions.work.md`) Local Function Status ===================== This is a checklist for current and outstanding work on the local functions feature, as spec'd in [local-functions.md](./local-functions.md). -------------------- Known issues ============ Compiler: - [ ] Parser builds nodes for local functions when feature not enabled (#9940) - [ ] Compiler crash: base call to state machine method, in state machine method (#9872) - [ ] Need custom warning for unused local function (#9661) - [ ] Generate quick action does not offer to generate local (#9352) - [ ] Parser ambiguity research (#10388) - [ ] Dynamic support (#10389) - [ ] Referring to local function in expression tree (#10390) - [ ] Resolve definite assignment rules (#10391) - [ ] Remove support for `var` return type (#10392) - [ ] Update error messages (#10393) IDE: - [ ] Some selections around local functions fail extract method refactoring [ ] (#8719) - [ ] Extracting local function passed to an Action introduces compiler error [ ] (#8718) - [ ] Ctrl+. on a delegate invocation crashes VS (via Extract method) (#8717) - [ ] Inline temp introduces compiler error (#8716) - [ ] Call hierarchy search never terminates on local functions (#8654) - [ ] Nav bar doesn't support local functions (#8648) - [ ] No outlining for local functions (#8647) - [ ] Squiggles all over the place when using an unsupported modifier (#8645) - [ ] Peek definition errors out on local function (#8644) - [ ] Void keyword not recommended while declaring local function (#8616) - [ ] Change signature doesn't update the signature of the local function (#8539) Feature Completeness Progress ============================= - [x] N-level nested local functions - [x] Capture - Works alongside lambdas and behaves very similarly in fallback cases - Has zero-allocation closures (struct frames by ref) on functions never converted to a delegate and are not an iterator/async - [x] Standard parameter features - params - ref/out - named/optional - [x] Visibility - May revisit design (currently shadows, may do overloads) - [x] Generics - Nongeneric local functions in generic methods (same as lambdas). - Generic local functions in nongeneric methods. - Generic local functions in generic methods. - Arbitrary nesting of generic local functions. - Generic local functions with constraints. - [x] Inferred return type - [x] Iterators - [x] Async --- ### Features Nullable Metadata (`docs/features/nullable-metadata.md`) Nullable Metadata ========= The following describes the representation of nullable annotations in metadata. ## NullableAttribute Type references are annotated in metadata with a `NullableAttribute`. ```C# namespace System.Runtime.CompilerServices { [AttributeUsage( AttributeTargets.Class | AttributeTargets.Event | AttributeTargets.Field | AttributeTargets.GenericParameter | AttributeTargets.Parameter | AttributeTargets.Property | AttributeTargets.ReturnValue, AllowMultiple = false, Inherited = false)] public sealed class NullableAttribute : Attribute { public readonly byte[] NullableFlags; public NullableAttribute(byte flag) { NullableFlags = new byte[] { flag }; } public NullableAttribute(byte[] flags) { NullableFlags = flags; } } } ``` The `NullableAttribute` type is for compiler use only - it is not permitted in source. The type declaration is synthesized by the compiler if not already included in the compilation. Each type reference in metadata may have an associated `NullableAttribute` with a `byte[]` where each `byte` represents nullability: 0 for oblivious, 1 for not annotated, and 2 for annotated. The `byte[]` is constructed as follows: - Reference type: the nullability (0, 1, or 2), followed by the representation of the type arguments in order including containing types - Nullable value type: the representation of the type argument only - Non-generic value type: skipped - Generic value type: 0, followed by the representation of the type arguments in order including containing types - Array: the nullability (0, 1, or 2), followed by the representation of the element type - Tuple: the representation of the underlying constructed type - Type parameter reference: the nullability (0, 1, or 2, with 0 for unconstrained type parameter) Note that non-generic value types are represented by an empty `byte[]`. However, generic value types and type parameters constrained to value types have an explicit 0 in the `byte[]` for nullability. The reason generic types and type parameters are represented with an explicit `byte` is to simplify metadata import. Specifically, this avoids the need to calculate whether a type parameter is constrained to a value type when decoding nullability metadata, since the constraints may include a (valid) cyclic reference to the type parameter. ### Optimizations If the `byte[]` is empty, the `NullableAttribute` is omitted. If all values in the `byte[]` are the same, the `NullableAttribute` is constructed with that single `byte` value. (For instance, `NullableAttribute(1)` rather than `NullableAttribute(new byte[] { 1, 1 }))`.) ### Type parameters Each type parameter definition may have an associated `NullableAttribute` with a single `byte`: 1. `notnull` constraint: `NullableAttribute(1)` 2. `class` constraint in `#nullable disable` context: `NullableAttribute(0)` 3. `class` constraint in `#nullable enable` context: `NullableAttribute(1)` 4. `class?` constraint: `NullableAttribute(2)` 5. No `notnull`, `class`, `struct`, `unmanaged`, or type constraints in `#nullable disable` context: `NullableAttribute(0)` 6. No `notnull`, `class`, `struct`, `unmanaged`, or type constraints in `#nullable enable` context (equivalent to an `object?` constraint): `NullableAttribute(2)` ## NullableContextAttribute `NullableContextAttribute` can be used to indicate the nullability of type references that have no `NullableAttribute` annotations. ```C# namespace System.Runtime.CompilerServices { [System.AttributeUsage( AttributeTargets.Class | AttributeTargets.Delegate | AttributeTargets.Interface | AttributeTargets.Method | AttributeTargets.Struct, AllowMultiple = false, Inherited = false)] public sealed class NullableContextAttribute : Attribute { public readonly byte Flag; public NullableContextAttribute(byte flag) { Flag = flag; } } } ``` The `NullableContextAttribute` type is for compiler use only - it is not permitted in source. The type declaration is synthesized by the compiler if not already included in the compilation. The `NullableContextAttribute` is optional - nullable annotations can be represented in metadata with full fidelity using `NullableAttribute` only. `NullableContextAttribute` is valid in metadata on type and method declarations. The `byte` value represents the implicit `NullableAttribute` value for type references within that scope that do not have an explicit `NullableAttribute` and would not otherwise be represented by an empty `byte[]`. The nearest `NullableContextAttribute` in the metadata hierarchy applies. If there are no `NullableContextAttribute` attributes in the hierarchy, missing `NullableAttribute` attributes are treated as `NullableAttribute(0)`. The attribute is not inherited. The C#8 compiler uses the following algorithm to determine which `NullableAttribute` and `NullableContextAttribute` attributes to emit. First, `NullableAttribute` attributes are generated at each type reference and type parameter definition by: calculating the `byte[]`, skipping empty `byte[]`, and collapsing `byte[]` to single `byte` where possible. Then at each level in metadata hierarchy starting at methods: The compiler finds the most common single `byte` value across all the `NullableAttribute` attributes at that level and any `NullableContextAttribute` attributes on immediate children. If there are no single `byte` values, there are no changes. Otherwise, a `NullableContext(value)` attribute is created at that level where `value` is most common value (preferring `0` over `1` and preferring `1` over `2`), and all `NullableAttribute` and `NullableContextAttribute` attributes with that value are removed. Note that an assembly compiled with C#8 where all reference types are oblivious will have no `NullableContextAttribute` and no `NullableAttribute` attributes emitted. That is equivalent to a legacy assembly. ### Examples ```C# // C# representation of metadata [NullableContext(2)] class Program { string s; // string? [Nullable({ 2, 1, 2 }] Dictionary d; // Dictionary? [Nullable(1)] int[] a; // int[]! int[] b; // int[]? [Nullable({ 0, 2 })] object[] c; // object?[]~ } ``` ## Private members To reduce the size of metadata, the C#8 compiler can be configured to not emit attributes for members that are inaccessible outside the assembly (`private` members, and also `internal` members if the assembly does not contain `InternalsVisibleToAttribute` attributes). The compiler behavior is configured from a command-line flag. For now a feature flag is used: `-features:nullablePublicOnly`. If private member attributes are dropped, the compiler will emit a `[module: NullablePublicOnly]` attribute. The presence or absence of the `NullablePublicOnlyAttribute` can be used by tools to interpret the nullability of private members that do not have an associated `NullableAttribute` attribute. For members that do not have explicit accessibility in metadata (specifically for parameters, type parameters, events, and properties), the compiler uses the accessibility of the container to determine whether to emit nullable attributes. ```C# namespace System.Runtime.CompilerServices { [System.AttributeUsage(AttributeTargets.Module, AllowMultiple = false)] public sealed class NullablePublicOnlyAttribute : Attribute { public readonly bool IncludesInternals; public NullablePublicOnlyAttribute(bool includesInternals) { IncludesInternals = includesInternals; } } } ``` The `NullablePublicOnlyAttribute` type is for compiler use only - it is not permitted in source. The type declaration is synthesized by the compiler if not already included in the compilation. `IncludesInternal` is true if `internal` members are annotated in addition to `public` and `protected` members. ## Compatibility The nullable metadata does not include an explicit version number. Where possible, the compiler will silently ignore attribute forms that are unexpected. The metadata format described here is incompatible with the format used by earlier C#8 previews: 1. Concrete non-generic value types are no longer included in the `byte[]`, and 2. `NullableContextAttribute` attributes are used in place of explicit `NullableAttribute` attributes. Those differences mean that assemblies compiled with earlier previews may be read incorrectly by later previews, and assemblies compiled with later previews may be read incorrectly by earlier previews. --- ### Features Nullable Reference Types (`docs/features/nullable-reference-types.md`) Nullable Reference Types ========= Reference types may be nullable, non-nullable, or null-oblivious (abbreviated here as `?`, `!`, and `~`). ## Setting project level nullable context Project level nullable context can be set by using "nullable" command line switch: -nullable[+|-] Specify nullable context option enable|disable. -nullable:{enable|disable|warnings|annotations} Specify nullable context option enable|disable|warnings|annotations. Through msbuild the context could be set by supplying an argument for a "Nullable" parameter of Csc build task. Accepted values are "enable", "disable", "warnings", "annotations", or null (for the default nullable context according to the compiler). The Microsoft.CSharp.Core.targets passes value of msbuild property named "Nullable" for that parameter. Note that in previous preview releases of C# 8.0 this "Nullable" property was successively named "NullableReferenceTypes" then "NullableContextOptions". ## Annotations In source, nullable reference types are annotated with `?`. ```c# string? OptString; // may be null Dictionary? OptDictionaryOptValues; // dictionary may be null, values may be null ``` A warning is reported when annotating a reference type with `?` outside a `#nullable` context. [Metadata representation](nullable-metadata.md) ## Declaration warnings _Describe warnings reported for declarations in initial binding._ ## Flow analysis Flow analysis is used to infer the nullability of variables within executable code. The inferred nullability of a variable is independent of the variable's declared nullability. Method calls are analyzed even when they are conditionally omitted. For instance, `Debug.Assert` in release mode. ### Warnings _Describe set of warnings. Differentiate W warnings._ If the analysis determines that a null check always (or never) passes, a hidden diagnostic is produced. For example: `"string" is null`. ### Null tests A number of null checks affect the flow state when tested for: - comparisons to `null`: `x == null` and `x != null` - `is` operator: `x is null`, `x is K` (where `K` is a constant), `x is string`, `x is string s` - calls to well-known equality methods, including: - `static bool object.Equals(object, object)` - `static bool object.ReferenceEquals(object, object)` - `bool object.Equals(object)` and overrides - `bool IEquatable.Equals(T)` and implementations - `bool IEqualityComparer.Equals(T, T)` and implementations Some null checks are "pure null tests", which means that they can cause a variable whose flow state was previously not-null to update to maybe-null. Pure null tests include: - `x == null`, `x != null` *whether using a built-in or user-defined operator* - `(Type)x == null`, `(Type)x != null` - `x is null` - `object.Equals(x, null)`, `object.ReferenceEquals(x, null)` - `IEqualityComparer.Equals(x, null)` All of the above checks except for `x is null` are commutative. For example, `null == x` is also a valid pure null test. Some expressions which may not return `bool` are also considered pure null tests: - `x?.Member` will change the receiver's flow state to maybe-null unconditionally - `x ?? y` will change the LHS expression's flow state to maybe-null unconditionally Example of how a pure null test can affect flow analysis: ```cs string s = "hello"; if (s != null) { _ = s.ToString(); // ok } else { _ = s.ToString(); // warning } ``` Versus a "not pure" null test: ```cs string s = "hello"; if (s is string) { _ = s.ToString(); // ok } else { _ = s.ToString(); // ok } ``` Invocation of methods annotated with the following attributes will also affect flow analysis: - simple pre-conditions: `[AllowNull]` and `[DisallowNull]` - simple post-conditions: `[MaybeNull]` and `[NotNull]` - conditional post-conditions: `[MaybeNullWhen(bool)]` and `[NotNullWhen(bool)]` - `[DoesNotReturnIf(bool)]` (e.g. `[DoesNotReturnIf(false)]` for `Debug.Assert`) and `[DoesNotReturn]` - `[NotNullIfNotNull(string)]` - member post-conditions: `[MemberNotNull(params string[])]` and `[MemberNotNullWhen(bool, params string[])]` See https://github.com/dotnet/csharplang/blob/main/meetings/2019/LDM-2019-05-15.md The `Interlocked.CompareExchange` methods have special handling in flow analysis instead of being annotated due to the complexity of their nullability semantics. The affected overloads include: - `static object? System.Threading.Interlocked.CompareExchange(ref object? location, object? value, object? comparand)` - `static T System.Threading.Interlocked.CompareExchange(ref T location, T value, T comparand) where T : class?` When simple pre- and post-condition attributes are applied to a property, and an allowed input state is assigned to the property, the property's state is updated to be an allowed output state. For instance an `[AllowNull] string` property can be assigned `null` and still return not-null values. Enforcing nullability attributes within method bodies: - An input parameter marked with `[AllowNull]` is initialized with a maybe-null (or maybe-default) state. - An input parameter marked with `[DisallowNull]` is initialized with a not-null state. - A parameter marked with `[MaybeNull]` or `[MaybeNullWhen]` can be assigned a maybe-null value, without warning. Same for return values. Same for a nullable parameter marked with `[NotNullWhen]` (the attribute is ignored). - A parameter marked with `[NotNull]` will produce a warning when assigned a maybe-null value. Same for return values. - The state of a parameter marked with `[MaybeNullWhen]`/`[NotNullWhen]` is checked upon exiting the method instead. - A method marked with `[DoesNotReturn]` will produce a warning if it returns or exits normally. Note: we don't validate the internal consistency of auto-properties, so it is possible to misuse attributes on auto-props as on fields. For example: `[AllowNull, NotNull] public TOpen P { get; set; }`. Enforcing nullability attributes when overriding and implementing: In addition to checking the types in overrides/implementations are compatible with overridden/implemented members, we also check that the nullability attributes are compatible. - For input parameters (by-value and `in`), we check that the widest allowed value by the overridden/implemented parameter can be assigned to the overriding/implementing parameter. - For output parameters (`out` and return values), we check that the widest produced value by the overriding/implementing parameter can be assigned to the overridden/implemented parameter. - For `ref` parameters and return values, we check both. - We check that the post-condition contract `[NotNull]`/`[MaybeNull]` on input parameters is enforced by overriding/implementing members. - We check that overrides/implementations have the `[DoesNotReturn]` attribute if the overridden/implemented member has it. - We check that members used in `[MemberNotNull(...)]` or `[MemberNotNullWhen(...)]` are not null upon exit, by assuming they had maybe-null state on entry. ## `default` If `T` is a reference type, `default(T)` is `T?`. ```c# string? s = default(string); // assigns ?, no warning string t = default; // assigns ?, warning ``` If `T` is a value type, `default(T)` is `T` and any non-value-type fields in `T` are maybe null. If `T` is a value type, `new T()` is equivalent to `default(T)`. ```c# struct Pair { public T First; public U Second; } var p = default(Pair); // ok: Pair p p.Second.ToString(); // warning (object?, string) t = default; // ok t.Item2.ToString(); // warning ``` If `T` is an unconstrained type parameter, `default(T)` is a `T` that is maybe null. ```c# T t = default; // warning ``` ### Conversions Conversions can be calculated with ~ considered distinct from ? and !, or with ~ implicitly convertible to ? and !. Given `IIn` and `IOut`, with ~ distinct: - `T!` is a `T~` is a `T?` - `IIn` is a `IIn` is a `IIn` - `IOut` is a `IOut` is a `IOut` Most conversions are considered with ~ implicitly convertible to ? and !. _Describe warnings from user-defined conversions._ ### Assignment For `x = y`, the nullability of the converted type of `y` is used for `x`. A warning is reported if there is a mismatch between top-level or nested nullability comparing the inferred nullability of `x` and the declared type of `y`. The warning is a W warning when assigning `?` to `!` and the target is a local. ```c# notNull = maybeNull; // assigns ?, warning notNull = oblivious; // assigns ~, no warning oblivious = maybeNull; // assigns ?, no warning ``` ### Local declarations Nullablilty follows from assignment above. Assigning `?` to `!` is a W warning. ```c# string notNull = maybeNull; // assigns ?, warning ``` The nullability of `var` declarations is the nullable version of the inferred type. ```c# var s = maybeNull; // s is ?, no warning if (maybeNull == null) return; var t = maybeNull; // t is ? too ``` ### Suppression operator (`!`) The postfix `!` operator sets the top-level nullability to non-nullable. Conversions on a suppressed expression produce no nullability warnings. ```c# var x = optionalString!; // x is string! var y = obliviousString!; // y is string! var z = new [] { optionalString, obliviousString }!; // no change, z is string?[]! ``` A warning is reported when using the `!` operator absent a `NonNullTypes` context. Unnecessary usages of `!` do not produce any diagnostics, including `!!`. A suppressed expression `e!` can be target-typed if the operand expression `e` can be target-typed. ### Explicit cast Explicit cast to `?` changes top-level nullability. Explicit cast to `!` does not change top-level nullability and may produce W warning. ```c# var x = (string)maybeNull; // x is string?, W warning var y = (string)oblivious; // y is string~, no warning var z = (string?)notNull; // y is string?, no warning ``` A warning is reported if the type of operand including nested nullability is not explicitly convertible to the target type. ```c# sealed class MyList : IEnumerable { } var x = new MyList(); var y = (IEnumerable)x; // warning var z = (IEnumerable)x; // no warning ``` ### Method type inference We modify the spec rule for [Fixing](https://github.com/dotnet/csharplang/blob/main/spec/expressions.md#fixing "Fixing") to take account of types that may be equivalent (i.e. have an identity conversion) yet may not be identical in the set of bounds. The existing spec says (third bullet) > If among the remaining candidate types `Uj` there is a unique type `V` from which there is an implicit conversion to all the other candidate types, then `Xi` is fixed to `V`. This is not correct for C# 5 (i.e., it is a bug in the language specification), as it fails to handle bounds such as `Dictionary` and `Dictionary`, which are merged to `Dictionary`. This is [an open issue](https://github.com/ECMA-TC49-TG2/spec/issues/951) that we anticipate will be addressed in the next iteration of the ECMA specification. Handling nullability properly will require additional changes beyond those in that next iteration of the ECMA spec. When adding to the set of exact bounds of a type parameter, types that are equivalent (i.e. have an identity conversion) but not identical are merged using *invariant* rules. When adding to the set of lower bounds of a type parameter, types that are equivalent but not identical are merged using *covariant* rules. When adding to the set of upper bounds of a type parameter, types that are equivalent but not identical are merged using *contravariant* rules. In the final fixing step, types that are equivalent but not identical are merged using *covariant* rules. Merging equivalent but not identical types is done as follows: #### Invariant merging rules - Merging `dynamic` and `object` results in the type `dynamic`. - Merging tuple types that differ in element names is specified elsewhere. - Merging equivalent types that differ in nullability is performed as follows: merging the types `Tn` and `Um` (where `n` and `m` are differing nullability annotations) results in the type `Vk` where `V` is the result of merging `T` and `U` using the invariant rule, and `k` is as follows: - if either `n` or `m` are non-nullable, non-nullable. - if either `n` or `m` are nullable, nullable. - otherwise oblivious. - Merging constructed generic types is performed as follows: Merging the types `K` and `K` results in the type `K` where `Ci` is the result of merging `Ai` and `Bi` by the invariant rule. - Merging tuple types `(A1, A2, ...)` and `(B1, B2, ...)` results in the type `(C1, C2, ...)` where `Ci` is the result of merging `Ai` and `Bi` by the invariant rule. - Merging the array types `T[]` and `U[]` results in the type `V[]` where `V` is the result of merging `T` and `U` by the invariant rule. #### Covariant merging rules - Merging `dynamic` and `object` results in the type `dynamic`. - Merging tuple types that differ in element names is specified elsewhere. - Merging equivalent types that differ in nullability is performed as follows: merging the types `Tn` and `Um` (where `n` and `m` are differing nullability annotations) results in the type `Vk` where `V` is the result of merging `T` and `U` using the covariant rule, and `k` is as follows: - if either `n` or `m` are nullable, nullable. - if either `n` or `m` are oblivious, oblivious. - otherwise non-nullable. - Merging constructed generic types is performed as follows: Merging the types `K` and `K` results in the type `K` where `Ci` is the result of merging `Ai` and `Bi` by - the invariant rule if `K`'s type parameter in the `i` position is invariant. - the covariant rule if `K`'s type parameter in the `i` position is declared `out`. - the contravariant rule if the `K`'s type parameter in the `i` position is declared `in`. - Merging tuple types `(A1, A2, ...)` and `(B1, B2, ...)` results in the type `(C1, C2, ...)` where `Ci` is the result of merging `Ai` and `Bi` by the covariant rule. - Merging the array types `T[]` and `U[]` results in the type `V[]` where `V` is the result of merging `T` and `U` by the invariant rule. #### Contravariant merging rules - Merging `dynamic` and `object` results in the type `dynamic`. - Merging tuple types that differ in element names is specified elsewhere. - Merging equivalent types that differ in nullability is performed as follows: merging the types `Tn` and `Um` (where `n` and `m` are differing nullability annotations) results in the type `Vk` where `V` is the result of merging `T` and `U` using the contravariant rule, and `k` is as follows: - if either `n` or `m` are non-nullable, non-nullable. - if either `n` or `m` are oblivious, oblivious. - otherwise nullable. - Merging constructed generic types is performed as follows: Merging the types `K` and `K` results in the type `K` where `Ci` is the result of merging `Ai` and `Bi` by - the invariant rule if `K`'s type parameter in the `i` position is invariant. - the covariant rule if `K`'s type parameter in the `i` position is declared `in`. - the contravariant rule if `K`'s type parameter in the `i` position is declared `out`. - Merging tuple types `(A1, A2, ...)` and `(B1, B2, ...)` results in the type `(C1, C2, ...)` where `Ci` is the result of merging `Ai` and `Bi` by the contravariant rule. - Merging the array types `T[]` and `U[]` results in the type `V[]` where `V` is the result of merging `T` and `U` by the invariant rule. It is intended that these merging rules are associative and commutative, so that a compiler may merge a set of equivalent types pairwise in any order to compute the final result. > ***Open issue***: these rules do not describe the handling of merging a nested generic type `K.L` with `K.L`. That should be handled the same as a hypothetical type `KL` would be merged with `KL`. > ***Open issue***: these rules do not describe the handling of merging pointer types. ### Array creation The calculation of the _best type_ element nullability uses the Conversions rules above and the covariant merging rules. ```c# var w = new [] { notNull, oblivious }; // ~[]! var x = new [] { notNull, maybeNull, oblivious }; // ?[]! var y = new [] { enumerableOfNotNull, enumerableOfMaybeNull, enumerableOfOblivious }; // IEnumerable! var z = new [] { listOfNotNull, listOfMaybeNull, listOfOblivious }; // List<~>! ``` ### Anonymous types Fields of anonymous types have nullability of the arguments, inferred from the initializing expression. ```c# static void F(T x, T y) { if (x == null) return; var a = new { x, y }; // inferred as x:T, y:T (both Unannotated) a.x.ToString(); // ok (non-null tracked by the nullable flow state of the initial value for property x) a.y.ToString(); // warning } ``` ### Null-coalescing operator The top-level nullability of `x ?? y` is `!` if `x` is `!` and otherwise the top-level nullability of `y`. A warning is reported if there is a nested nullability mismatch between `x` and `y`. ### Try-finally We infer the state after a try statement in part by keeping track of which variables may have been assigned a null value in the finally block. This tracking distinguishes actual assignments from inferences: only actual assignments (but not inferences) affect the state after the try statement. See https://github.com/dotnet/roslyn/issues/34018 and https://github.com/dotnet/roslyn/pull/35276. This is not yet reflected in the language specification for nullable reference types (as we don't have a specification for how to handle try statements at this time). ## Type parameters A `class?` constraint is allowed, which, like class, requires the type argument to be a reference type, but allows it to be nullable. [Nullable strawman](https://github.com/dotnet/csharplang/issues/790) [4/25/18](https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-04-25.md) Explicit `object` (or `System.Object`) constraints of any nullability are disallowed. However, type substitution can lead to `object!` or `object~` constraints to appear among the constraint types, when their nullability is significant by comparison to other constraints. An `object!` constraint requires the type to be non-nullable (value or reference type). However, an explicit `object?` constraint is not allowed. An unconstrained (here it means - no type constraints, and no `class`, `struct`, or `unmanaged` constraints) type parameter is essentially equivalent to one constrained by `object?` when it is declared in a context where nullable annotations are enabled. If annotations are disabled, the type parameter is essentially equivalent to one constrained by `object~`. The context is determined at the identifier that declares the type parameter within a type parameter list. [4/25/18](https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-04-25.md) Note, the `object`/`System.Object` constraint is represented in metadata as any other type constraint, the type is System.Object. An explicit `notnull` constraint is allowed, which requires the type to be non-nullable (value or reference type). [5/15/19](https://github.com/dotnet/csharplang/blob/main/meetings/2019/LDM-2019-05-15.md) The rules to determine when it is a named type constraint or a special `notnull` constraint are similar to rules for `unmanaged`. Similarly, it is valid only at the first position in constraints list. A warning is reported for nullable type argument for type parameter with `class` constraint or non-nullable reference type or interface type constraint. [4/25/18](https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-04-25.md) ```c# static void F1() where T : class { } static void F2() where T : Stream { } static void F3() where T : IDisposable { } F1(); // warning F2(); // warning F3(); // warning ``` Type parameter constraints may include nullable reference type and interface types. [4/25/18](https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-04-25.md) ```c# static void F2 where T : Stream? { } static void F3() where T : IDisposable? { } F2(); // ok F3(); // ok ``` A warning is reported for inconsistent top-level nullability of constraint types. [4/25/18](https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-04-25.md) ```c# static void F4 where T : class, Stream? { } // warning static void F5 where T : Stream?, IDisposable { } // warning ``` An error is reported for duplicate constraints where constraints are compared ignoring top-level and nested nullability. ```c# class C where T : class { static void F1() where U : T, T? { } // error: duplicate constraint static void F2() where U : I, I { } // error: duplicate constraint } ``` _What are the rules for annotated (unannotated) type arguments for generic type parameters from unannotated (annotated) types and methods?_ ```c# [NotNullTypes(false)] List F1(T t) where T : class { ... } [NotNullTypes(true)] List F2(T t) where T : class { ... } [NotNullTypes(true)] List F3(T? t) where T : class { ... } var x = F1(notNullString); // List or List ? var y = F1(maybeNullString); // List or List ? var z = F2(obliviousString); // List! or List! ? var w = F3(obliviousString); // List! or List! ? ``` A warning is reported for dereferencing variables of type T, where T is an unconstrained type parameter. ```C# static void F(T t) => t.ToString(); // Warn possible null dereference ``` A warning is reported when attempting to convert a null, default, or potentially null value to an unconstrained type parameter. ```C# static T F() => default; // Warn converting default to T ``` ## Object creation An error is reported for creating an instance of a nullable reference type. ```c# new C?(); // error new List(); // ok ``` ## Generated code Older code generation strategies may not be nullable aware. Setting the project-level nullable context to "enable" could result in many warnings that a user is unable to fix. To support this scenario any syntax tree that is determined to be generated will have its nullable state implicitly set to "disable", regardless of the overall project state. A syntax tree is determined to be generated if meets one or more of the following criteria: - File name begins with: - TemporaryGeneratedFile_ - File name ends with: - .designer.cs - .generated.cs - .g.cs - .g.i.cs - Contains a top level comment that contains - `` | `true` (_needs confirmation_) | | `Nullable` | `true` | | `T? where T : class` | `true` | | `T? where T : struct` | `true` (_needs confirmation_) | | `string` | `false` | | `int` | `false` | | `T where T : class/object` | `false` | | `T where T : class?` | `false` | | `T where T : struct` | `false` | | `T` | `false` | --- ### Features Outvar (`docs/features/outvar.md`) Out Variable Declarations ========================= The *out variable declaration* feature enables a variable to be declared at the location that it is being passed as an `out` argument. ```antlr argument_value : 'out' type identifier | ... ; ``` A variable declared this way is called an *out variable*. You may use the contextual keyword `var` for the variable's type. The scope will be the same as for a *pattern-variable* introduced via pattern-matching. According to Language Specification (section 7.6.7 Element access) the argument-list of an element-access (indexing expression) does not contain ref or out arguments. However, they are permitted by the compiler for various scenarios, for example indexers declared in metadata that accept `out`. Within the scope of a local variable introduced by a local-variable-declaration, it is a compile-time error to refer to that local variable in a textual position that precedes its declaration. It is also an error to reference implicitly-typed (§8.5.1) out variable in the same argument list that immediately contains its declaration. Overload resolution is modified as follows: We add a new conversion: > There is a *conversion from expression* from an implicitly-typed out variable declaration to every type. Also > The type of an explicitly-typed out variable argument is the declared type. and > An implicitly-typed out variable argument has no type. Neither conversion from expression is better when the argument is an implicitly-typed out variable declaration. (this needs to be woven into the form of the specification) The type of an implicitly-typed out variable is the type of the corresponding parameter in the signature of the method. The new syntax node `DeclarationExpressionSyntax` is added to represent the declaration in an out var argument. #### Discussion There is a discussion thread for this feature at https://github.com/dotnet/roslyn/issues/6183. #### Open issues and TODOs: Tracked at https://github.com/dotnet/roslyn/issues/11566. --- ### Features Patterns (`docs/features/patterns.md`) > Open issues for the design and implementation of this feature can be found at [patterns.work.md](patterns.work.md). Pattern Matching for C# ======================= Pattern matching extensions for C# enable many of the benefits of algebraic data types and pattern matching from functional languages, but in a way that smoothly integrates with the feel of the underlying language. The basic features are: [record types](records.md), which are types whose semantic meaning is described by the shape of the data (treated as a separate feature); and pattern matching, which is a new form that enables extremely concise multilevel decomposition of these data types. Elements of this approach are inspired by related features in the programming languages [F#](http://www.msr-waypoint.net/pubs/79947/p29-syme.pdf "Extensible Pattern Matching Via a Lightweight Language") and [Scala](http://lampwww.epfl.ch/~emir/written/MatchingObjectsWithPatterns-TR.pdf "Matching Objects With Patterns"). ## Is Expression The `is` operator is extended to test an expression against a *pattern*. ```antlr is_pattern_expression : relational_expression 'is' pattern ; relational_expression : is_pattern_expression ; ``` This form of *relational_expression* is in addition to the existing forms in the C# specification. It is a compile-time error if the *relational_expression* to the left of the `is` token does not designate a value or does not have a type. A *constant_pattern* appearing as the right-hand-side of an *is_pattern_expression* is syntactically restricted to be a *shift_expression*, even though a *constant_pattern* appearing elsewhere can syntactically be any *expression*. For simplicity that restriction is not shown in the grammar. Every *identifier* of the pattern introduces a new local variable that is *definitely assigned* after the `is` operator is `true` (i.e. *definitely assigned when true*). > Note: There is technically an ambiguity between *type* in an `is-expression` and *constant_pattern* in an *is_pattern_expression*, either of which might be a valid parse of a qualified identifier. We try to bind it as a type for compatibility with previous versions of the language; only if that fails do we resolve it as a constant pattern. ## Patterns Patterns are used in the `is` operator and in a *switch_statement* to express the shape of data against which incoming data is to be compared. Patterns may be recursive so that parts of the data may be matched against sub-patterns. ```antlr pattern : declaration_pattern | constant_pattern | deconstruction_pattern | property_pattern | discard_pattern | var_pattern ; declaration_pattern : type simple_designation ; constant_pattern : constant_expression ; property_pattern : type? property_subpattern simple_designation? ; property_subpattern : '{' subpatterns? '}' ; subpatterns : subpattern | subpattern ',' subpatterns ; subpattern : pattern | identifier ':' pattern ; deconstruction_pattern : type? '(' subpatterns? ')' property_subpattern? simple_designation? ; simple_designation : single_variable_designation | discard_designation ; discard_pattern : discard_designation ; var_pattern : 'var' designation ; ``` It is a semantic error if any _subpattern_ of a _property_pattern_ does not contain an _identifier_ (it must be of the second form, which has an _identifier_). There is no special "null-checking pattern", as the ability to check for null falls out as a special case of a trivial property pattern. To check if the string `s` is non-null, you can write any of the following forms ``` c# if (s is object o) ... // o is of type object if (s is string x) ... // x is of type string if (s is {} x) ... // x is of type string if (s is {}) ... ``` Every pattern is bound against a hypothetical *input operand*. In the case of an *is_pattern_expression*, that is the value of the left-hand-side. In the case of a branch of a `switch` statement, it is the switch expression. In the case of a nested pattern, it is the result of extracting the relevant value from the enclosing object. ### Type Pattern The *type_pattern* both tests that the input operand holds a value of a given type and casts it to that type if the test succeeds. This introduces a local variable of the given type named by the *simple_designation*. That local variable is *definitely assigned* when the result of the pattern-matching operation is true. ```antlr type_pattern : type simple_designation ; ``` The runtime semantic of this expression is that it tests the runtime type of the input operand against the *type* in the pattern. If it is of that runtime type (or some subtype), the result of the `is operator` is `true`. If the *simple_designation* is a *single_variable_designation*, then it declares a new local variable named by the *identifier* that is assigned the value of the input operand when the result is `true`. Certain combinations of static type of the left-hand-side and the given type are considered incompatible and result in compile-time error. A value of static type `E` is said to be *pattern compatible* with the type `T` if there exists an identity conversion, an implicit reference conversion, a boxing conversion, an explicit reference conversion, or an unboxing conversion from `E` to `T`. It is a compile-time error if an expression of type `E` is not pattern compatible with the type in a type pattern that it is matched with. > Note: this is not quite correct for type parameters, which should be considered pattern compatible always. The type pattern is useful for performing run-time type tests of reference types, and replaces the idiom ```cs Type v; if (value is Type) { v = (Type)value; } ``` or ``` c# var v = expr as Type; if (v != null) { // code using v } ``` With the slightly more concise ```cs if (expr is Type v) { // code using v } ``` It is an error if *type* is a nullable value type. The type pattern can be used to test values of nullable types: a value of type `Nullable` (or a boxed `T`) matches a type pattern `T2 id` if the value is non-null and the type of `T2` is `T`, or some base type or interface of `T`. For example, in the code fragment ```cs int? x = 3; if (x is int v) { // code using v } ``` The condition of the `if` statement is `true` at runtime and the variable `v` holds the value `3` of type `int`. ### Constant Pattern A constant pattern tests the value of an expression against a constant value. The constant may be any constant expression, such as a literal, the name of a declared `const` variable, or an enumeration constant. If both *e* and *c* are of integral types, the pattern is considered matched if the result of the expression `e == c` is `true`. Otherwise the pattern is considered matching if `object.Equals(c, e)` returns `true`. In this case it is a compile-time error if the static type of *e* is not *pattern compatible* with the type of the constant. ```antlr constant_pattern : constant_expression ; ``` ### Var Pattern ``` antlr var_pattern : 'var' designation ; ``` An input operand matches the pattern `var identifier` always. In other words, a match to a *var pattern* with a *simple_designator* always succeeds. At runtime the value of the input operand is bound to a newly introduced local variable. The type of the local variable is the static type of the input operand. It is an error if the name `var` binds to a type where a *var_pattern* is used. > Note: we need to describe the semantics when the *designation* is a *tuple_designation*. ### Discard Pattern An input operand matches the discard `_` always. In other words, every expression matches the discard pattern. ### Property Pattern ``` antlr property_pattern : type? property_subpattern simple_designation? ; property_subpattern : '{' subpatterns? '}' ; subpatterns : subpattern | subpattern ',' subpatterns ; subpattern : pattern | identifier ':' pattern ; ``` A property pattern tests the input operand to see if it is an instance of a given type, and also tests some of its accessible properties or fields to see if they match given subpatterns. The type to be tested is selected as follows: - If there is a *type* part of the *property_pattern*, then that is the type to be tested. The type must not be a nullable value type. - Otherwise if the input operand is not a nullable value type, its static type is used. - Otherwise the underlying type of the input operand's type is used. The *simple_designation*, if it is a *single_variable_designation*, names a newly introduced variable of this type. Each of the *subpattern*s appearing int the *property_subpattern* specifies a property or field to be checked. The subpattern must be of the second form, with the *identifier* present. The *identifier* must name an accessible and readable instance property or field of the type to be tested. At runtime, a *subpattern* is *satisfied* if the value of that property or field, when treated as the input operand of the *subpattern*'s pattern, matches. A property pattern matches if the type test succeeds and all subpatterns are satisfied. The order in which subpatterns are matched is not specified, and a failed match may not test all subpatterns at runtime. ### Deconstruction Pattern A deconstruction pattern is similar to a *property_pattern*, but involves the deconstruction of a *tuple* or the invocation of a user-defined *Deconstruct* method. ``` antlr deconstruction_pattern : type? '(' subpatterns? ')' property_subpattern? simple_designation? ; ``` It is an error if a deconstruction pattern has a single subpattern but omits the type. The type to be tested is determined as in the *property_pattern*. That type must be a tuple type whose *cardinality* is the same as the number of *subpatterns* between parentheses, or it must be a type that contains a unique `Deconstruct` method with that number of `out` parameters, as defined for the *deconstruction assignment* feature. A subpattern between parentheses in a *deconstruction_pattern* is satisfied if the value retrieved from the input operand's deconstruction for that position, when treated as the input operand for the corresponding pattern, matches. If such a subpattern has an *identifier*, it is a compile-time error if that is not the name of the corresponding tuple element or `Deconstruct` `out` parameter. A deconstruction pattern matches if the type test succeeds and all subpatterns are satisfied. The order in which subpatterns are matched is not specified, and a failed match may not test all subpatterns at runtime. > Note: this specification does not yet treat `ITuple` for positional matching. > ## Switch Expression A *switch_expression* is added to support `switch`-like semantics for an expression context. The C# language syntax is augmented with the following syntactic productions: ``` antlr switch_expression : null_coalescing_expression switch '{' switch_expression_case_list '}' ; switch_expression_case_list : switch_expression_case | switch_expression_case_list ',' switch_expression_case ; switch_expression_case : pattern where_clause? '=>' expression ; ``` The *switch_expression* is not allowed as an *expression_statement*. The type of the *switch_expression* is the *best common type* of the expressions appearing to the right of the `=>` tokens of the *switch_expression_case*s. It is an error if the compiler can prove (using a set of techniques that has not yet been specified) that some *switch_section*'s pattern cannot affect the result because some previous pattern will always match. At runtime, the result of the *switch_expression* is the value of the *expression* of the first *switch_expression_case* for which the expression on the left-hand-side of the *switch_expression* matches the *switch_expression_case*'s pattern, and for which the expression of the *where_clause* of the *switch_expression_case*, if present, evaluates to `true`. > Note: we need to specify what happens if the set of cases is incomplete, both at compile-time and at runtime. ## Some Examples of Pattern Matching ### Arithmetic simplification Suppose we define a set of recursive types to represent expressions (per a separate proposal): ```cs abstract class Expr; class X() : Expr; class Const(double Value) : Expr; class Add(Expr Left, Expr Right) : Expr; class Mult(Expr Left, Expr Right) : Expr; class Neg(Expr Value) : Expr; ``` Now we can define a function to compute the (unreduced) derivative of an expression: ```cs Expr Deriv(Expr e) { switch (e) { case X(): return Const(1); case Const(_): return Const(0); case Add(var Left, var Right): return Add(Deriv(Left), Deriv(Right)); case Mult(var Left, var Right): return Add(Mult(Deriv(Left), Right), Mult(Left, Deriv(Right))); case Neg(var Value): return Neg(Deriv(Value)); } } ``` An expression simplifier demonstrates positional patterns: ```cs Expr Simplify(Expr e) { switch (e) { case Mult(Const(0), _): return Const(0); case Mult(_, Const(0)): return Const(0); case Mult(Const(1), var x): return Simplify(x); case Mult(var x, Const(1)): return Simplify(x); case Mult(Const(var l), Const(var r)): return Const(l*r); case Add(Const(0), var x): return Simplify(x); case Add(var x, Const(0)): return Simplify(x); case Add(Const(var l), Const(var r)): return Const(l+r); case Neg(Const(var k)): return Const(-k); default: return e; } } ``` --- ### Features Patterns.Work (`docs/features/patterns.work.md`) # Open issues for pattern-matching extensions for C# 8 ### Specification - [ ] The specification needs to be updated with the additional syntax forms being added. See https://github.com/dotnet/csharplang/issues/1054 for a summary of the proposed changes vs C# 7. ### Unimplemented parts - [ ] Give an error when a wildcard is used but something named `_` is in scope. - [ ] Implement the switch statement in the presence of recursive patterns - [x] Parsing - [x] Binding - [ ] Lowering - [ ] Code-gen - [ ] Debug info - [ ] Edit-and-continue - [ ] Implement the match expression - [ ] Parsing - [ ] Binding - [ ] Lowering - [ ] Code-gen - [ ] It is an error if the name `var` binds to a type in a var pattern. Implement and test. ### Test plan needed - [ ] We need a test plan for these additions. Here is a high-level list of some tests that are needed - [ ] Scoping mechanism works for pattern variables in recursive patterns - [ ] In scripts top-level pattern variables in recursive patterns become fields. - [ ] IDE scenarios need to be identified tested. - [ ] Fuzz generator should be extended to support recursive patterns. - [ ] Test each of the kinds of symbols that should be supported in property patterns - [ ] Test all of the ways that a name would not be valid to name a property in a property pattern - [ ] Need to design the data representation for edit-and-continue for temps in patterns - [ ] Need to precisely share temps per the edit-and-continue spec - [ ] Need bullets here or github issues for prototype comments. - [ ] Need to ensure good code quality, e.g. avoid redundant null checks preceding types tests - [ ] Need to ensure a good tradeoff between decision tree size explosion and execution of redundant tests. --- ### Features Pdb Compilation Options (`docs/features/pdb-compilation-options.md`) # Embedding Compilation options in Portable PDBs Prior to this feature the compiler did not emit all information that's necessary to reconstruct the original compilation to the output binaries. It is desirable to include such information in order to support scenarios such as validation that given binaries can be reproduced from their original sources, [post-build source analysis](https://github.com/dotnet/roslyn/issues/41395), etc. The goal of this feature is to be able to construct a compilation that is exactly the same as an initial compilation as long as it meets the conditions outlined in the assumptions below. This document is restricted to the following assumptions: 1. The full benefit is for builds with `-deterministic` and published to the symbol server. That said the compiler embeds compilation options and references to all Portable PDBs. 2. Source generator and analyzer references are not needed for this task. They may be useful, but are out of scope for this feature. 3. Any storage capacity used for PDBs and source should not impact this feature, such as compression algorithm. 4. Only Portable PDB files will be included for this spec. This feature can be expanded past these once it is implemented and proven needed elsewhere. This document will provide the expanded specification to the Portable PDB format. Any additions to that format will be ported to expand documentation provided in [dotnet-runtime](https://github.com/dotnet/runtime/blob/main/docs/design/specs/PortablePdb-Metadata.md). ## PDB Format Additions #### Compilation Metadata References custom debug information Symbol server uses a [key](https://github.com/dotnet/symstore/blob/main/docs/specs/SSQP_Key_Conventions.md#pe-timestamp-filesize) computed from the COFF header in the PE image: Timestamp: 4 byte integer Size of image: 4 byte integer Example: File name: `example.exe` COFF header Timestamp field: `0x542d5742` COFF header SizeOfImage field: `0x32000` Lookup key: `example.exe/542d574232000/example.exe` To fully support metadata references, a user will need to be able to find the exact PE image that was used in the compilation. This will be done by storing the parts that make up the symbol server key. The MVID of a reference will be stored since it's a GUID that represents the symbol. This is to future proof the information for reference lookup. At this time, only external references for the compilation will be included. Any other references may be added later in a separate blob. Metadata references are stored as binary. The binary encoding will be as follows (order matters): Name: A UTF-8 string (null terminated) Aliases: UTF-8 Comma (, ) separated list of aliases (null terminated). May be empty MetadataImageKind: byte value representing Microsoft. CodeAnalysis. MetadataImageKind EmbedInteropTypes/MetadataImageKind: byte, reads as binary values starting in order from right to left MetadataImageKind: 1 if Assembly, 0 if Module EmbedInteropTypes: 1 if true Examples: 0b11, MetadataImageKind.Assembly and EmbedInteropTypes = true 0b01, MetadataImageKind.Module and EmbedInteropTypes = true Timestamp: 4 byte integer File Size: 4 byte integer MVID: 16 byte integer (GUID) #### Retrieving Metadata References Metadata references are stored in the CustomDebugInformation in a PDB using the GUID `7E4D4708-096E-4C5C-AEDA-CB10BA6A740D` . Example how to retrieve and read this information using `System.Reflection.Metadata` : ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Compiler Options custom debug information The remaining values will be stored as key value pairs in the pdb. The storage format will be UTF8 encoded key value pairs that are null terminated. Order is not guaranteed. Any values left out can be assumed to be the default for the type. Keys may be different for Visual Basic and C#. They are serialized to reflect the command line arguments representing the same values Example: `compilerversion\01.2.3-example-sha\0sourceencoding\0utf-8\0checked\01\0unsafe\01\0langversion\0latest\0nullable\0Enable` ## List of Compiler Flags #### C# Flags That Can Be Derived From PDB or Assembly * baseaddress * checksumalgorithm * debug * deterministic * embed * filealign * highentropyva * link - Represented by a metadata reference with `EmbededInteropTypes=true` * linkresource - Will be represented in metadata reference embedded in pdb * main - Already stored in PDB as the entry point token * moduleassemblyname * modulename * nostdlib - Will be represented in metadata reference embedded in pdb * nowin32manifest * pdb * platform * publicsign * resource - Will be represented in metadata reference embedded in pdb * subsystemversion * win32icon * win32manifest * win32res #### C# Flags Not Included * bugreport * delaysign * doc * errorendlocation * errorlog * errorreport * fullpaths * incremental * keycontainer * keyfile * noconfig * nologo * nowarn * out * parallel * pathmap * preferreduilang * recurse * refout * refonly * reportanalyzer * ruleset * utf8output * version * warn * warnaserror #### Visual Basic Flags That Can Be Derived From PDB or Assembly * baseaddress * checksumalgorithm * debug * filealign * highentropyva * linkresource - Will be represented in metadata reference embedded in pdb * main * nostdlib - Will be represented in metadata reference embedded in pdb * platform * resource - Will be represented in metadata reference embedded in pdb * subsystemversion * win32icon * win32manifest * win32resource #### Visual Basic Flags Not Included * bugreport * delaysign * doc * errorreport * help * keycontainer * keyfile * libpath * moduleassemblyname * modulename * netcf * noconfig * nologo * nowarn * nowin32manifest * optioncompare * optionexplicit * optioninfer * out * parallel * preferreduilang * quiet * recurse * refonly * refout * rootnamespace * ruleset * sdkpath * utf8output * vbruntime * verbose * warnaserror #### Shared Options for C# and Visual Basic | PDB Key | Format | Default | Description | | ---------------------- | --------------------------------------- | --------- | ------------ | | language | `C#\|Visual Basic` | required | Language name. | | compiler-version | [SemVer2](https://semver.org/spec/v2.0.0.html) string | required | Full version with SHA | | runtime-version | [SemVer2](https://semver.org/spec/v2.0.0.html) string | required | [runtime version](#runtime-version) | | source-file-count | int32 | required | Count of files in the document table that are source files | | optimization | `(debug\|debug-plus\|release\|release-debug-plus)` | `'debug'` | [optimization](#optimization) | | portability-policy | `(0\|1\|2\|3)` | `0` | [portability policy](#portability-policy) | | default-encoding | string | none | [file encoding](#file-encoding) | | fallback-encoding | string | none | [file encoding](#file-encoding) | | output-kind | string | require | The value passed to `/target` | | platform | string | require | The value passed to `/platform` | #### Options For C\# See [compiler options](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/listed-alphabetically) documentation | PDB Key | Format | Default | Description | | ---------------------- | --------------------------------------- | --------- | ------------ | | language-version | `[0-9]+(\.[0-9]+)?` | required | [langversion](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/langversion-compiler-option) | | define | `,`-separated identifier list | empty | [define](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/define-compiler-option) | | checked | `(True\|False)` | `False` | [checked](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/checked-compiler-option) | | nullable | `(Disable\|Warnings\|Annotations\|Enable)` | `Disable` | [nullable](https://docs.microsoft.com/en-us/dotnet/csharp/nullable-references) | | unsafe | `(True\|False)` | `False` | [unsafe](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/unsafe-compiler-option) | #### Options For Visual Basic See [compiler options](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/compiler-options-listed-alphabetically) documentation | PDB Key | Format | Default | Description | | ---------------------- | ------------------------------------------ | -------- | ----------- | | language-version | `[0-9]+(\.[0-9]+)?` | required | [langversion](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/langversion) | | define | `,`-separated list of name `=` value pairs | empty | [define](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/define) | | checked | `(True\|False)` | `False` | Opposite of [removeintchecks](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/removeintchecks) | | option-strict | `(Off\|Custom\|On)` | required | [option strict](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/optionstrict) | | option-infer | `(True\|False)` | required | [option infer](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/optioninfer) | | option-compare-text | `(True\|False)` | required | [option compare](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/optioncompare) | | option-explicit | `(True\|False)` | required | [option explicit](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/optionexplicit) | | embed-runtime | `(True\|False)` | required | Whether or not the VB runtime was embedded into the PE | | global-namespaces | `,` -separated identifier list | empty | [imports](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/imports) | root-namespace | string | empty | [root namespace](https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/command-line-compiler/rootnamespace) #### Portability Policy Portability policy is derived from the [appconfig command option](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/appconfig-compiler-option). Since appconfig is a pointer to a file not embedded in the PDB or PE, information that will directly impact the compilation is extracted. This is stored in a flag called `portability-policy` with a numeric value from \[0-3\]. This value directly correlates to reconstructing the [AssemblyPortabilityPolicy](https://github.com/dotnet/roslyn/blob/bdb3ece74c85892709f5e42ae7d67248999ecc3b/src/Compilers/Core/Portable/Desktop/AssemblyPortabilityPolicy.cs). * 0 -> SuppressSilverlightPlatformAssembliesPortability = false, SuppressSilverlightLibraryAssembliesPortability = false * 1 -> SuppressSilverlightPlatformAssembliesPortability = true, SuppressSilverlightLibraryAssembliesPortability = false * 2 -> SuppressSilverlightPlatformAssembliesPortability = false, SuppressSilverlightLibraryAssembliesPortability = true * 3 -> SuppressSilverlightPlatformAssembliesPortability = true, SuppressSilverlightLibraryAssembliesPortability = true #### File Encoding Encoding will be stored in two keys: "default-encoding" and "fallback-encoding". If `default-encoding` is present, it represents the forced encoding used, such as by passing in "codepage" to the command line. All files without a Byte Order Mark (BOM) should be decoded using this encoding. If `fallback-encoding` is present, it means that no default encoding was specified but an encoding was detected and used. The compiler [has logic](https://github.com/dotnet/roslyn/blob/462eac607741023e5c2d518ac1045f4c6dabd501/src/Compilers/Core/Portable/EncodedStringText.cs#L32) to determine an encoding is none is specified, so the value that was computed is stored so it can be reused in future compilations. Both values are written as [WebName](https://docs.microsoft.com/en-us/dotnet/api/system.text.encoding.webname?view=netcore-3.1) values for an encoding. #### Optimization Optimization level can be specified from the command line with [optimize](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/optimize-compiler-option). It gets translated to the [OptimizationLevel](https://github.com/dotnet/roslyn/blob/e704ca635bd6de70a0250e34c4567c7a28fa9f6d/src/Compilers/Core/Portable/Compilation/OptimizationLevel.cs) that is emitted to the PDB. There are three possible values: * `debug` -> `OptimizationLevel.Debug` and DebugPlusMode is false * `debug-plus` -> `OptimizationLevel.Debug` and DebugPlusMode is true * `release` -> `OptimizationLevel.Release` and DebugPlusMode is false (ignored) #### Runtime Version The runtime version used that the compiler was running in when generating the PE. This is stored as [informational version](https://docs.microsoft.com/en-us/dotnet/api/system.reflection.assemblyinformationalversionattribute.informationalversion?view=netcore-3.1#System_Reflection_AssemblyInformationalVersionAttribute_InformationalVersion). Runtime version is stored since it can impact the unicode character interpretation and decimal arithmetics, which both play a role in how code is compiled from source. There may also be future variations where the different versions of the runtime impact compilation. ### Retriving Compiler Flags Compiler flags are stored in the CustomDebugInformation in a PDB using the GUID `B5FEEC05-8CD0-4A83-96DA-466284BB4BD8` . Example how to retrieve and read this information using `System.Reflection.Metadata` : ``` csharp var compilationOptionsGuid = new Guid("B5FEEC05-8CD0-4A83-96DA-466284BB4BD8"); var path = "path/to/pdb"; using var stream = File.OpenRead(path); using var metadataReaderProvider = MetadataReaderProvider.FromPortablePdbStream(stream); var metadataReader = metadataReaderProvider.GetMetadataReader(); foreach (var handle in metadataReader.GetCustomDebugInformation(EntityHandle.ModuleDefinition)) { var customDebugInformation = metadataReader.GetCustomDebugInformation(handle); if (metadataReader.GetGuid(customDebugInformation.Kind) == compilationOptionsGuid) { var blobReader = metadataReader.GetBlobReader(customDebugInformation.Value); // Compiler flag bytes are UTF-8 null-terminated key-value pairs var nullIndex = blobReader.IndexOf(0); while (nullIndex >= 0) { var key = blobReader.ReadUTF8(nullIndex); // Skip the null terminator blobReader.ReadByte(); nullIndex = blobReader.IndexOf(0); var value = blobReader.ReadUTF8(nullIndex); // Skip the null terminator blobReader.ReadByte(); nullIndex = blobReader.IndexOf(0); // key and value now have strings containing serialized compiler flag information Console.WriteLine($"{key} = {value}"); } } } ``` --- ### Features Pre Compilation Source Outputs (`docs/features/pre-compilation-source-outputs.md`) # Pre-Compilation Source Outputs ## Summary This document describes a new API, `RegisterPreCompilationSourceOutput`, for the incremental source generator pipeline. This API allows generators to produce source that is added to the *initial* compilation - before any compilation-dependent phases execute - while still being able to read from non-compilation input providers such as `AdditionalTextsProvider` and `ParseOptionsProvider`. This bridges the gap between `RegisterPostInitializationOutput` (which takes no inputs at all) and `RegisterSourceOutput` (which has access to the full compilation), enabling scenarios where generated source must be visible to subsequent generator phases and other generators, but the generation logic itself only depends on non-compilation inputs. ## Motivation Today, the incremental generator pipeline has a clear ordering: 1. **`RegisterPostInitializationOutput`** - Runs first, before the compilation is available. Takes no inputs. Source is added to the initial compilation. 2. **`RegisterSourceOutput` / `RegisterImplementationSourceOutput`** - Runs after the compilation (including post-init sources) is available. Can access the full compilation, syntax trees, and semantic model. This creates a gap: if a generator needs to produce source that should be visible to later phases (its own or other generators'), but the generation logic depends on non-compilation inputs like additional files or parse options, there is no appropriate API. ### Primary Motivation: Razor Performance The primary motivation for this API is a significant performance improvement for the Razor source generator. Razor has **cross-file dependencies**: a `.razor` file can reference components declared in other `.razor` files. To resolve these references, the Razor generator must compile in two phases internally: 1. **Declaration collection**: Parse all `.razor` files and collect the partial type declarations (component names, parameters, etc.). 2. **Binding and implementation**: Use those declarations to perform semantic analysis - resolving component references, validating parameter types, generating the full implementation. Today, because `RegisterSourceOutput` does not feed source back into the compilation, Razor must perform this two-phase process **entirely within its own generator**. Concretely, Razor: 1. Creates a **copy** of the compilation. 2. Adds the partial declarations into this copied compilation. 3. Performs binding against the copied compilation to discover cross-file information. 4. **Throws the copied compilation away**. 5. Produces its final output against the original compilation. This means the system creates **three compilations** during a generator run: the initial compilation (pre-SG), Razor's private intermediate compilation (for binding), and the final compilation (with SG outputs). The intermediate compilation is pure overhead - it duplicates all the work of building a compilation just so Razor can ask semantic questions about its own declarations. With `RegisterPreCompilationSourceOutput`, Razor can instead: 1. Use `RegisterPreCompilationSourceOutput` to emit partial declarations from `.razor` files. These are added to the **initial** compilation by the driver. 2. Use `RegisterSourceOutput` with the `CompilationProvider` to perform binding against the initial compilation - which now already contains the declarations. The generator produces the remaining implementation parts as standard source output. In this model, the system creates only **two compilations**: the initial compilation (pre-SG, augmented with pre-compilation sources) and the final compilation (with all SG outputs). Razor no longer needs its private intermediate compilation. Early experiments show this yields a **roughly 50% performance improvement** for Razor source generation. Critically, this improvement comes without adding any new compilation phases to the driver. The pre-compilation outputs are simply additional steps that run before the compilation is finalized - the total number of *compilation* objects created by the driver remains the same as today (the initial compilation is augmented in-place with `AddSyntaxTrees`, not rebuilt from scratch). ### Additional Scenarios **Cross-Generator Type Visibility**: As a secondary benefit, pre-compilation sources are visible to *all* generators' standard phases, not just the generator that produced them. A generator that reads `.proto` files and emits C# types via `RegisterPreCompilationSourceOutput` makes those types available to other generators that perform binding in their `RegisterSourceOutput` phase. This enables cross-generator interoperability without requiring generators to know about each other. **Configurable Post-Initialization**: Generator authors want to emit attributes or other marker types during post-initialization, but need to read `AnalyzerConfigOptions` or additional files to determine *which* sources to emit (see [#53632](https://github.com/dotnet/roslyn/issues/53632)). `RegisterPostInitializationOutput` cannot accept any inputs, forcing authors to either emit all possible sources unconditionally or split their generator into multiple packages. `RegisterPreCompilationSourceOutput` addresses this directly: the generated source is added to the initial compilation just like post-init source, but the generator can read parse options, analyzer config options, and additional files to decide what to emit. ## API ```csharp public readonly partial struct IncrementalGeneratorInitializationContext { /// /// Register a source output that will execute before the compilation is available. /// The produced source will be added to the initial compilation, making it visible /// to subsequent phases and other generators. /// /// /// The source value provider must not depend on compilation or syntax node inputs. /// Attempting to access the compilation or syntax trees during this phase will throw /// an . /// [Experimental(RoslynExperiments.PreCompilationSourceOutput)] public void RegisterPreCompilationSourceOutput( IncrementalValueProvider source, Action action); [Experimental(RoslynExperiments.PreCompilationSourceOutput)] public void RegisterPreCompilationSourceOutput( IncrementalValuesProvider source, Action action); } ``` The production context is a new type, distinct from `SourceProductionContext`: ```csharp [Experimental(RoslynExperiments.PreCompilationSourceOutput)] public readonly struct PreCompilationSourceProductionContext { public CancellationToken CancellationToken { get; } public void AddSource(string hintName, string source); public void AddSource(string hintName, SourceText sourceText); } ``` `PreCompilationSourceProductionContext` intentionally does **not** include `ReportDiagnostic`. Pre-compilation is an early phase focused purely on producing source; diagnostic reporting should be done in a separate analyzer. A new `IncrementalGeneratorOutputKind` value is also introduced: ```csharp [Flags] public enum IncrementalGeneratorOutputKind { // ... existing values ... /// /// A pre-compilation source output, registered via /// . /// Source is added to the initial compilation before compilation-dependent phases execute. /// PreCompilation = 0b10000 } ``` ### Usage Example ```csharp public void Initialize(IncrementalGeneratorInitializationContext context) { // Pre-compilation phase: read additional files and emit source // that will be visible to the compilation. var protoFiles = context.AdditionalTextsProvider .Where(f => f.Path.EndsWith(".proto")); context.RegisterPreCompilationSourceOutput(protoFiles, (ctx, file) => { var types = ParseProtoFile(file.GetText()!.ToString()); ctx.AddSource($"{file.Path}.g.cs", GenerateTypes(types)); }); // Regular phase: the compilation now contains the types generated above. context.RegisterSourceOutput(context.CompilationProvider, (ctx, compilation) => { var messageType = compilation.GetTypeByMetadataName("MyProto.MyMessage"); // messageType is non-null - it was generated in the pre-compilation phase ctx.AddSource("Serializers.g.cs", GenerateSerializer(messageType!)); }); } ``` ## Execution Model The updated pipeline execution order is: ``` 1. RegisterPostInitializationOutput +-- Source added to initial compilation (takes no inputs) 2. RegisterPreCompilationSourceOutput <- NEW +-- Reads non-compilation inputs (additional files, parse options, etc.) +-- Source added to initial compilation +-- Compilation is rebuilt with new sources 3. RegisterSourceOutput / RegisterImplementationSourceOutput +-- Reads full compilation (now includes post-init AND pre-compilation sources) +-- Source is part of final output but not fed back into compilation ``` Concretely, in the generator driver: 1. Post-initialization sources are collected as today. 2. A `DriverStateTable.Builder` is created **without** the compilation or syntax store - these are not yet available. 3. Pre-compilation output nodes are evaluated for all generators. Their sources are parsed into syntax trees. 4. The initial compilation is augmented: `compilation = compilation.AddSyntaxTrees(preCompilationTrees)`. 5. `DriverStateTable.Builder.SetCompilation` is called, which stores the compilation and creates the `SyntaxStore.Builder` internally. 6. Standard source output nodes execute against the augmented compilation. This means: - **Within a single generator**: A `RegisterSourceOutput` callback can query the semantic model and see types produced by that same generator's `RegisterPreCompilationSourceOutput`. - **Across generators**: Generator B's `RegisterSourceOutput` can see types produced by Generator A's `RegisterPreCompilationSourceOutput`. - **Incremental behavior**: Pre-compilation outputs participate in the standard incremental caching. If the inputs (e.g., additional files) haven't changed, the pre-compilation sources are cached and the compilation is not rebuilt unnecessarily. ## Phase Enforcement The `IncrementalValueProvider` / `IncrementalValuesProvider` type system does not distinguish between providers that depend on the compilation and those that do not. A generator author could write: ```csharp // This compiles fine, but will fail at runtime! context.RegisterPreCompilationSourceOutput( context.CompilationProvider, // <- compilation is not available in this phase (ctx, compilation) => { // This code never executes - an exception is thrown when evaluating the provider }); ``` To catch this, the `DriverStateTable.Builder` does **not** have the `Compilation` or `SyntaxStore` set during the pre-compilation phase. Accessing either property throws an `InvalidOperationException`. This exception is thrown from the `DriverStateTable.Builder` property getter, caught by `InputNode.UpdateStateTable` (which wraps it in a `UserFunctionException`), and handled by the pre-compilation error handler in the driver. When a pre-compilation output fails (whether from accessing compilation-dependent inputs or from any other exception), the generator is placed in **error state**: a diagnostic is reported, and the generator's standard phase is **skipped entirely**. Other generators are unaffected - their pre-compilation and standard phases continue to execute normally. ### Why Not Compile-Time Safety? A more principled approach would introduce a parallel type system (`PreCompilationValueProvider` / `PreCompilationValuesProvider`) that prevents compilation-dependent providers from being passed to `RegisterPreCompilationSourceOutput` at compile time. However, this would require duplicating every transform operation (`Select`, `Where`, `Collect`, `SelectMany`, `Combine`, `WithComparer`, `WithTrackingName`) for the pre-compilation provider types, creating a significantly larger API surface. Given that this is an experimental API, runtime enforcement with clear error messages is sufficient for the initial release. If the feature proves valuable and the runtime safety issue becomes a real pain point, a compile-time safe type system can be pursued in a future iteration. ## Incremental Step Tracking Pre-compilation outputs have a distinct step name: `WellKnownGeneratorOutputs.PreCompilationSourceOutput` (value `"PreCompilationSourceOutput"`). This is separate from `SourceOutput` and `ImplementationSourceOutput`, allowing tools to distinguish pre-compilation steps in incremental step tracking output. Per-generator `GeneratorRunStateTable.Builder` instances are shared across both the pre-compilation and standard passes, so all steps for a generator (both pre-compilation and standard) appear in a unified view. Pre-compilation sources appear in `GeneratorRunResult.GeneratedSources` alongside post-init and standard sources. ## Parse Options Reparse Like post-initialization trees, pre-compilation trees are reparsed when parse options change between driver runs. A unified `RequiresConstantTreeReparse` check handles both post-init and pre-compilation trees in a single pass. ## Implementation Architecture The output node hierarchy uses an abstract base class to share logic between source output kinds: - **`AbstractSourceOutputNode`** - Contains all shared logic: `UpdateStateTable` (caching, table building, step tracking), `AppendOutputs`, and interface boilerplate. Defines abstract members `Kind`, `StepName`, and `InvokeUserAction`. - **`SourceOutputNode`** - Sealed subclass for `Source` and `Implementation` output kinds. Creates a `SourceProductionContext` with the compilation and diagnostics. - **`PreCompilationSourceOutputNode`** - Sealed subclass for `PreCompilation` output kind. Creates a `PreCompilationSourceProductionContext` without the compilation or diagnostics. ## Comparison with Two-Phase Incremental Generators ([#81395](https://github.com/dotnet/roslyn/issues/81395)) Issue #81395 proposes a "Two-Phase Incremental Generators" model with `RegisterDeclarationOutput` and `RegisterImplementationSourceOutput`. While both proposals address the same fundamental problem - enabling cross-generator type visibility - they differ significantly in approach, scope, and trade-offs. ### Shared Goal Both proposals address the limitation that generator-produced source is not visible to subsequent compilation-dependent phases. Issue #81395 frames this primarily as a cross-generator dependency problem (Generator A's types are invisible to Generator B). This proposal frames it primarily as a performance problem (generators that need to see their own prior output must create expensive intermediate compilations as a workaround). The underlying mechanism is the same: source produced during `RegisterSourceOutput` is not fed back into the compilation. ### Key Differences | Dimension | `RegisterPreCompilationSourceOutput` | `RegisterDeclarationOutput` (#81395) | |---|---|---| | **Number of new APIs** | 1 new registration method | 2 new registration methods (declaration + implementation) | | **Compilation phases** | **No additional compilation phases** - pre-compilation outputs are added to the existing initial compilation via `AddSyntaxTrees`. The driver still creates the same number of compilations as today. | **One additional compilation phase** - a new intermediate compilation must be created from the initial compilation plus all declarations, forcing a full rebuild of the source symbol graph. | | **Input model** | Pre-compilation outputs read non-compilation inputs only (additional files, parse options). No compilation or syntax access. | Declaration outputs read the initial compilation (user code only). Can use syntax providers and semantic model. | | **What gets fed back** | Any source - full type definitions, implementations, anything the generator produces. | Intended for declarations/signatures only (though enforcement of this is an open question in #81395). | | **Generator awareness** | Generators don't need to split their logic. A single pre-compilation output can emit complete types. | Generators must explicitly split output into declarations (phase 1) and implementations (phase 2). | | **Cross-generator visibility** | Pre-compilation sources are visible to all generators' standard phases. | Declaration sources are visible to all generators' implementation phases. | | **Complexity for generator authors** | Low - add one call, ensure inputs don't depend on compilation. | Higher - must understand and correctly split declaration vs. implementation, restructure existing generators. | | **Scope of change** | Narrow: one API, one output kind, minimal driver changes. | Broad: new execution model, new APIs, new concepts, potential changes to how `RegisterSourceOutput` and `RegisterImplementationSourceOutput` interact. | ### When Each Approach is Better **`RegisterPreCompilationSourceOutput` is better when:** - The generation logic doesn't need the compilation (e.g., reading and transforming additional files, config-driven code generation). - The generator already has an internal multi-compilation workaround that this API can eliminate (e.g., Razor). - You want to emit complete types that other generators can see, not just signatures. - Performance is critical - this approach adds no additional compilation phases. - You want minimal disruption to existing generator patterns. **`RegisterDeclarationOutput` (#81395) is better when:** - The generation logic *does* need the compilation and semantic model to determine what to declare (e.g., finding types annotated with attributes, analyzing inheritance hierarchies). - You want a formal separation between "what types exist" (declarations) and "how they're implemented" (implementations), potentially enabling better incremental invalidation. - You're willing to accept the cost of an additional compilation phase for a more structured execution model. ### Complementary, Not Competing These two proposals are not mutually exclusive. They could coexist in the same pipeline: ``` 1. RegisterPostInitializationOutput -> Constant source, no inputs 2. RegisterPreCompilationSourceOutput -> Non-compilation inputs, source added to compilation 3. RegisterDeclarationOutput (future) -> Compilation inputs, declarations added to compilation 4. RegisterSourceOutput -> Full compilation, final output 5. RegisterImplementationSourceOutput -> Full compilation, implementation-only output ``` ### Performance: Compilation Phase Cost The most significant practical difference between these proposals is their cost in terms of compilation phases. As noted in the [discussion on #81395](https://github.com/dotnet/roslyn/issues/81395#issuecomment-3562700994), **each additional compilation is expensive**. Creating a new compilation forces an entirely new source symbol graph - all symbols are recreated, nothing is shared from the previous compilation, and all binding must be redone. This is the single most expensive operation in the compiler pipeline. `RegisterPreCompilationSourceOutput` **does not add any compilation phases**. The pre-compilation sources are added to the initial compilation at the same point where post-init sources are added today (via `AddSyntaxTrees`). The driver still produces exactly two compilations: the initial compilation (now augmented with pre-compilation sources) and the final compilation (with all generator outputs). The feature simply runs more *steps* within the existing phases. This makes it extremely cheap - the cost is only the generator's own logic to produce the pre-compilation sources. `RegisterDeclarationOutput` (#81395) **requires an additional compilation phase**: the initial compilation must be rebuilt with all declaration sources to produce an enriched intermediate compilation, which is then handed to the implementation phase. This means three compilations total: initial, enriched (with declarations), and final (with implementations). For generators like Razor that already create their own intermediate compilations as a workaround, this may not add net cost, but for the general case it represents a significant overhead that the pre-compilation approach avoids entirely. ## Related Issues and Documents - [#81395 - Two-Phase Incremental Generators for Cross-Generator Dependencies](https://github.com/dotnet/roslyn/issues/81395) - [#53632 - Make AnalyzerConfigOptions available during PostInitialization in SourceGenerators](https://github.com/dotnet/roslyn/issues/53632) - [#57589 - Earlier two-phase proposal discussion](https://github.com/dotnet/roslyn/issues/57589) - [Incremental Generators Design Document](incremental-generators.md) - [Source Generators Cookbook](source-generators.cookbook.md) - [Implementation PR](https://github.com/dotnet/roslyn/pull/83088) - [API Review Issue](https://github.com/dotnet/roslyn/issues/83089) --- ### Features Private Protected (`docs/features/private-protected.md`) `private protected` access modifier ================================= We propose to add a new access modifier combination `private protected` (which can appear in any order among the modifiers). This maps to the CLR notion of protectedAndInternal, and borrows the same syntax currently used in [C++/CLI](https://msdn.microsoft.com/en-us/library/ke3a209d.aspx#BKMK_Member_visibility). A member declared `private protected` can be accessed within a subclass of its container if that subclass is in the same assembly as the member. We modify the language specification as follows (additions in bold). Section numbers are not shown below as they may vary depending on which version of the specification it is integrated into. ----- > The declared accessibility of a member can be one of the following: - Public, which is selected by including a public modifier in the member declaration. The intuitive meaning of public is “access not limited”. - Protected, which is selected by including a protected modifier in the member declaration. The intuitive meaning of protected is “access limited to the containing class or types derived from the containing class”. - Internal, which is selected by including an internal modifier in the member declaration. The intuitive meaning of internal is “access limited to this assembly”. - Protected internal, which is selected by including both a protected and an internal modifier in the member declaration. The intuitive meaning of protected internal is “accessible within this assembly as well as types derived from the containing class”. - **Private protected, which is selected by including both a private and a protected modifier in the member declaration. The intuitive meaning of private protected is “accessible within this assembly by types derived from the containing class”.** ----- > Depending on the context in which a member declaration takes place, only certain types of declared accessibility are permitted. Furthermore, when a member declaration does not include any access modifiers, the context in which the declaration takes place determines the default declared accessibility. - Namespaces implicitly have public declared accessibility. No access modifiers are allowed on namespace declarations. - Types declared directly in compilation units or namespaces (as opposed to within other types) can have public or internal declared accessibility and default to internal declared accessibility. - Class members can have any of the five kinds of declared accessibility and default to private declared accessibility. [Note: A type declared as a member of a class can have any of the five kinds of declared accessibility, whereas a type declared as a member of a namespace can have only public or internal declared accessibility. end note] - Struct members can have public, internal, or private declared accessibility and default to private declared accessibility because structs are implicitly sealed. Struct members introduced in a struct (that is, not inherited by that struct) cannot have protected **,** ~~or~~ protected internal **, or private protected** declared accessibility. [Note: A type declared as a member of a struct can have public, internal, or private declared accessibility, whereas a type declared as a member of a namespace can have only public or internal declared accessibility. end note] - Interface members implicitly have public declared accessibility. No access modifiers are allowed on interface member declarations. - Enumeration members implicitly have public declared accessibility. No access modifiers are allowed on enumeration member declarations. ----- > The accessibility domain of a nested member M declared in a type T within a program P, is defined as follows (noting that M itself might possibly be a type): - If the declared accessibility of M is public, the accessibility domain of M is the accessibility domain of T. - If the declared accessibility of M is protected internal, let D be the union of the program text of P and the program text of any type derived from T, which is declared outside P. The accessibility domain of M is the intersection of the accessibility domain of T with D. - **If the declared accessibility of M is private protected, let D be the intersection of the program text of P and the program text of any type derived from T, which is declared outside P. The accessibility domain of M is the intersection of the accessibility domain of T with D.** - If the declared accessibility of M is protected, let D be the union of the program text of T and the program text of any type derived from T. The accessibility domain of M is the intersection of the accessibility domain of T with D. - If the declared accessibility of M is internal, the accessibility domain of M is the intersection of the accessibility domain of T with the program text of P. - If the declared accessibility of M is private, the accessibility domain of M is the program text of T. ----- > When a protected **or private protected** instance member is accessed outside the program text of the class in which it is declared, and when a protected internal instance member is accessed outside the program text of the program in which it is declared, the access shall take place within a class declaration that derives from the class in which it is declared. Furthermore, the access is required to take place through an instance of that derived class type or a class type constructed from it. This restriction prevents one derived class from accessing protected members of other derived classes, even when the members are inherited from the same base class. ----- > The permitted access modifiers and the default access for a type declaration depend on the context in which the declaration takes place (§9.5.2): - Types declared in compilation units or namespaces can have public or internal access. The default is internal access. - Types declared in classes can have public, protected internal, **private protected**, protected, internal, or private access. The default is private access. - Types declared in structs can have public, internal, or private access. The default is private access. ----- > A static class declaration is subject to the following restrictions: - A static class shall not include a sealed or abstract modifier. (However, since a static class cannot be instantiated or derived from, it behaves as if it was both sealed and abstract.) - A static class shall not include a class-base specification (§16.2.5) and cannot explicitly specify a base class or a list of implemented interfaces. A static class implicitly inherits from type object. - A static class shall only contain static members (§16.4.8). [Note: All constants and nested types are classified as static members. end note] - A static class shall not have members with protected **, private protected, ** or protected internal declared accessibility. > It is a compile-time error to violate any of these restrictions. ----- > A class-member-declaration can have any one of the ~~five~~ **six** possible kinds of declared accessibility (§9.5.2): public, **private protected**, protected internal, protected, internal, or private. Except for the protected internal **and private protected** combination**s**, it is a compile-time error to specify more than one access modifier. When a class-member-declaration does not include any access modifiers, private is assumed. ----- > Non-nested types can have public or internal declared accessibility and have internal declared accessibility by default. Nested types can have these forms of declared accessibility too, plus one or more additional forms of declared accessibility, depending on whether the containing type is a class or struct: - A nested type that is declared in a class can have any of ~~five~~ **six** forms of declared accessibility (public, **private protected**, protected internal, protected, internal, or private) and, like other class members, defaults to private declared accessibility. - A nested type that is declared in a struct can have any of three forms of declared accessibility (public, internal, or private) and, like other struct members, defaults to private declared accessibility. ----- > The method overridden by an override declaration is known as the overridden base method For an override method M declared in a class C, the overridden base method is determined by examining each base class type of C, starting with the direct base class type of C and continuing with each successive direct base class type, until in a given base class type at least one accessible method is located which has the same signature as M after substitution of type arguments. For the purposes of locating the overridden base method, a method is considered accessible if it is public, if it is protected, if it is protected internal, or if it is **either** internal **or private protected** and declared in the same program as C. ----- > The use of accessor-modifiers is governed by the following restrictions: - An accessor-modifier shall not be used in an interface or in an explicit interface member implementation. - For a property or indexer that has no override modifier, an accessor-modifier is permitted only if the property or indexer has both a get and set accessor, and then is permitted only on one of those accessors. - For a property or indexer that includes an override modifier, an accessor shall match the accessor-modifier, if any, of the accessor being overridden. - The accessor-modifier shall declare an accessibility that is strictly more restrictive than the declared accessibility of the property or indexer itself. To be precise: - If the property or indexer has a declared accessibility of public, the accessor-modifier may be either **private protected**, protected internal, internal, protected, or private. - If the property or indexer has a declared accessibility of protected internal, the accessor-modifier may be either **private protected**, internal, protected, or private. - If the property or indexer has a declared accessibility of internal or protected, the accessor-modifier shall be **either private protected or** private. - **If the property or indexer has a declared accessibility of private protected, the accessor-modifier shall be private.** - If the property or indexer has a declared accessibility of private, no accessor-modifier may be used. ----- > Since inheritance isn’t supported for structs, the declared accessibility of a struct member cannot be protected, **private protected**, or protected internal. ----- --- ### Features Private Protected.Work (`docs/features/private-protected.work.md`) `private protected` work list ============================= - [ ] The IDE does not offer to complete "private" after "protected" or vice versa. - [ ] This implementation requires a thorough review - [ ] The implementation requires unit tests for each specification change - [ ] A feature specification is required for the VB language as well --- ### Features Readonly Instance Members (`docs/features/readonly-instance-members.md`) # Readonly Instance Members Championed Issue: ## Summary [summary]: #summary Provide a way to specify individual instance members on a struct do not modify state, in the same way that `readonly struct` specifies no instance members modify state. It is worth noting that `readonly instance member` != `pure instance member`. A `pure` instance member guarantees no state will be modified. A `readonly` instance member only guarantees that instance state will not be modified. All instance members on a `readonly struct` are implicitly `readonly instance members`. Explicit `readonly instance members` declared on non-readonly structs would behave in the same manner. For example, they would still create hidden copies if you called an instance member (on the current instance or on a field of the instance) which was itself not-readonly. ## Design [design]: #design Allow a user to specify that an instance member is, itself, `readonly` and does not modify the state of the instance (with all the appropriate verification done by the compiler, of course). For example: ```csharp public struct Vector2 { public float x; public float y; public readonly float GetLengthReadonly() { return MathF.Sqrt(LengthSquared); } public float GetLength() { return MathF.Sqrt(LengthSquared); } public readonly float GetLengthIllegal() { var tmp = MathF.Sqrt(LengthSquared); x = tmp; // Compiler error, cannot write x y = tmp; // Compiler error, cannot write y return tmp; } public float LengthSquared { readonly get { return (x * x) + (y * y); } } } public static class MyClass { public static float ExistingBehavior(in Vector2 vector) { // This code causes a hidden copy, the compiler effectively emits: // var tmpVector = vector; // return tmpVector.GetLength(); // // This is done because the compiler doesn't know that `GetLength()` // won't mutate `vector`. return vector.GetLength(); } public static float ReadonlyBehavior(in Vector2 vector) { // This code is emitted exactly as listed. There are no hidden // copies as the `readonly` modifier indicates that the method // won't mutate `vector`. return vector.GetLengthReadonly(); } } ``` Readonly can be applied to property accessors to indicate that `this` will not be mutated in the accessor. ```csharp public int Prop { readonly get { return this._prop1; } } ``` When `readonly` is applied to the property syntax, it means that all accessors are `readonly`. ```csharp public readonly int Prop { get { return this._store["Prop2"]; } set { this._store["Prop2"] = value; } } ``` Similar to the rules for property accessibility modifiers, redundant `readonly` modifiers are not allowed on properties. ```csharp public readonly int Prop1 { readonly get => 42; } // Not allowed public int Prop2 { readonly get => this._store["Prop2"]; readonly set => this._store["Prop2"]; } // Not allowed ``` Readonly can only be applied to accessors which do not mutate the containing type. ```csharp public int Prop { readonly get { return this._prop3; } set { this._prop3 = value; } } ``` ### Auto-properties Readonly can be applied to auto-implemented properties or `get` accessors. However, the compiler will treat all auto-implemented getters as readonly regardless of whether a `readonly` modifier is present. ```csharp // Allowed public readonly int Prop1 { get; } public int Prop2 { readonly get; } public int Prop3 { readonly get; set; } // Not allowed public readonly int Prop4 { get; set; } public int Prop5 { get; readonly set; } ``` ### Events Readonly can be applied to manually-implemented events, but not field-like events. Readonly cannot be applied to individual event accessors (add/remove). ```csharp // Allowed public readonly event Action Event1 { add { } remove { } } // Not allowed public readonly event Action Event2; public event Action Event3 { readonly add { } remove { } } public static readonly event Event4 { add { } remove { } } ``` Some other syntax examples: * Expression bodied members: `public readonly float ExpressionBodiedMember => (x * x) + (y * y);` * Generic constraints: `public readonly void GenericMethod(T value) where T : struct { }` The compiler would emit the instance member, as usual, and would additionally emit a compiler recognized attribute indicating that the instance member does not modify state. This effectively causes the hidden `this` parameter to become `in T` instead of `ref T`. This would allow the user to safely call said instance method without the compiler needing to make a copy. Some more "edge" cases that are explicitly permitted: * Explicit interface implementations are allowed to be `readonly`. * Partial methods are allowed to be `readonly`. Both signatures or neither must have the `readonly` keyword. The restrictions would include: * The `readonly` modifier cannot be applied to static methods, constructors or destructors. * The `readonly` modifier cannot be applied to delegates. * The `readonly` modifier cannot be applied to members of class or interface. ## Compiler API The following public API will be added: - `bool IMethodSymbol.IsDeclaredReadOnly { get; }` indicates that the member is readonly due to having a readonly keyword, or due to being an auto-implemented instance getter on a struct. - `bool IMethodSymbol.IsEffectivelyReadOnly { get; }` indicates that the member is readonly due to IsDeclaredReadOnly, or by the containing type being readonly, except if this method is a constructor. It may be necessary to add additional public API to properties and events. This poses a challenge for properties because `ReadOnly` properties have an existing, different meaning in VB.NET and the `IMethodSymbol.IsReadOnly` API has already shipped to describe that scenario. This specific issue is tracked in https://github.com/dotnet/roslyn/issues/34213. --- ### Features Records (`docs/features/records.md`) Records for C# ============== Records are a new, simplified declaration form for C# class and struct types that combine the benefits of a number of simpler features. We describe the new features (caller-receiver parameters and *with-expression*s), give the syntax and semantics for record declarations, and then provide some examples. # caller-receiver parameters Currently a method parameter's *default-argument* must be - a *constant-expression*; or - an expression of the form `new S()` where `S` is a value type; or - an expression of the form `default(S)` where `S` is a value type We extend this to add the following - an expression of the form `this.Identifier` This new form is called a *caller-receiver default-argument*, and is allowed only if all of the following are satisfied - The method in which it appears is an instance method; and - The expression `this.Identifier` binds to an instance member of the enclosing type, which must be either a field or a property; and - The member to which it binds (and the `get` accessor, if it is a property) is at least as accessible as the method; and - The type of `this.Identifier` is implicitly convertible by an identity or nullable conversion to the type of the parameter (this is an existing constraint on *default-argument*). When an argument is omitted from an invocation of a function member for a corresponding optional parameter with a *caller-receiver default-argument*, the value of the receiver's member is implicitly passed. > **Design Notes**: the main reason for the caller-receiver parameter is to support the *with-expression*. The idea is that you can declare a method like this > ```cs > class Point > { > public readonly int X; > public readonly int Y; > public Point With(int x = this.X, int y = this.Y) => new Point(x, y); > // etc > } > ``` > and then use it like this > ```cs > Point p = new Point(3, 4); > p = p.With(x: 1); > ``` > To create a new `Point` just like an existing `Point` but with the value of `X` changed. > > It is an open question whether or not the syntactic form of the *with-expression* is worth adding once we have support for caller-receiver parameters, so it is possible we would do this *instead of* rather than *in addition to* the *with-expression*. - [ ] **Open issue**: What is the order in which a *caller-receiver default-argument* is evaluated with respect to other arguments? Should we say that it is unspecified? # with-expressions A new expression form is proposed: ```antlr primary_expression : with_expression ; with_expression : primary_expression 'with' '{' with_initializer_list '}' ; with_initializer_list : with_initializer | with_initializer ',' with_initializer_list ; with_initializer : identifier '=' expression ; ``` The token `with` is a new context-sensitive keyword. Each *identifier* on the left of a *with_initilaizer* must bind to an accessible instance field or property of the type of the *primary_expression* of the *with_expression*. There may be no duplicated name among these identifiers of a given *with_expression*. A *with_expression* of the form > *e1* `with` `{` *identifier* = *e2*, ... `}` is treated as an invocation of the form > *e1*`.With(`*identifier2*`:` *e2*, ...`)` Where, for each method named `With` that is an accessible instance member of *e1*, we select *identifier2* as the name of the first parameter in that method that has a caller-receiver parameter that is the same member as the instance field or property bound to *identifier*. If no such parameter can be identified that method is eliminated from consideration. The method to be invoked is selected from among the remaining candidates by overload resolution. > **Design Notes**: Given caller-receiver parameters, many of the benefits of the *with-expression* are available without this special syntax form. We are therefore considering whether or not it is needed. Its main benefit is allowing one to program in terms of the names of fields and properties, rather than in terms of the names of parameters. In this way we improve both readability and the quality of tooling (e.g. go-to-definition on the identifier of a *with_expression* would navigate to the property rather than to a method parameter). - [ ] **Open issue**: This description should be modified to support extension methods. - [ ] **Open issue**: Does this syntactic sugar actually pay for itself? # pattern-matching See the [Pattern Matching Specification](patterns.md) for a specification of `operator is` and its relationship to pattern-matching. > **Design Notes**: By virtue of the compiler-generated `operator is` as specified herein, and the specification for pattern-matching, a record declaration > ```cs > public class Point(int X, int Y); > ``` > will support positional pattern-matching as follows > ```cs > Point p = new Point(3, 4); > if (p is Point(3, var y)) { // if X is 3 > Console.WriteLine(y); // print Y > } > ``` # record type declarations The syntax for a `class` or `struct` declaration is extended to support value parameters; the parameters become properties of the type: ```antlr class_declaration : attributes? class_modifiers? 'partial'? 'class' identifier type_parameter_list? record_parameters? record_class_base? type_parameter_constraints_clauses? class_body ; struct_declaration : attributes? struct_modifiers? 'partial'? 'struct' identifier type_parameter_list? record_parameters? struct_interfaces? type_parameter_constraints_clauses? struct_body ; record_class_base : class_type record_base_arguments? | interface_type_list | class_type record_base_arguments? ',' interface_type_list ; record_base_arguments : '(' argument_list? ')' ; record_parameters : '(' record_parameter_list? ')' ; record_parameter_list : record_parameter | record_parameter record_parameter_list ; record_parameter : attributes? type identifier record_property_name? default_argument? ; record_property_name : ':' identifier ; class_body : '{' class_member_declarations? '}' | ';' ; struct_body : '{' struct_members_declarations? '}' | ';' ; ``` > **Design Notes**: Because record types are often useful without the need for any members explicitly declared in a class-body, we modify the syntax of the declaration to allow a body to be simply a semicolon. A class (struct) declared with the *record-parameters* is called a *record class* (*record struct*), either of which is a *record type*. - [ ] **Open issue**: We need to include *primary_constructor_body* in the grammar so that it can appear inside a record type declaration. - [ ] **Open issue**: What are the name conflict rules for the parameter names? Presumably one is not allowed to conflict with a type parameter or another *record-parameter*. - [ ] **Open issue**: We need to specify the scope of the record-parameters. Where can they be used? Presumably within instance field initializers and *primary_constructor_body* at least. - [ ] **Open issue**: Can a record type declaration be partial? If so, must the parameters be repeated on each part? ### Members of a record type In addition to the members declared in the *class-body*, a record type has the following additional members: #### Primary Constructor A record type has a `public` constructor whose signature corresponds to the value parameters of the type declaration. This is called the *primary constructor* for the type, and causes the implicitly declared *default constructor* to be suppressed. At runtime the primary constructor * initializes compiler-generated backing fields for the properties corresponding to the value parameters (if these properties are compiler-provided; [see 1.1.2](#1.1.2)); then * executes the instance field initializers appearing in the *class-body*; and then * invokes a base class constructor: * If there are arguments in the *record_base_arguments*, a base constructor selected by overload resolution with these arguments is invoked; * Otherwise a base constructor is invoked with no arguments. * executes the body of each *primary_constructor_body*, if any, in source order. - [ ] **Open issue**: We need to specify that order, particularly across compilation units for partials. - [ ] **Open Issue**: We need to specify that every explicitly declared constructor must chain to the primary constructor. - [ ] **Open issue**: Should it be allowed to change the access modifier on the primary constructor? - [ ] **Open issue**: In a record struct, it is an error for there to be no record parameters? #### Primary constructor body ```antlr primary_constructor_body : attributes? constructor_modifiers? identifier block ; ``` A *primary_constructor_body* may only be used within a record type declaration. The *identifier* of a *primary_constructor_body* shall name the record type in which it is declared. The *primary_constructor_body* does not declare a member on its own, but is a way for the programmer to provide attributes for, and specify the access of, a record type's primary constructor. It also enables the programmer to provide additional code that will be executed when an instance of the record type is constructed. - [ ] **Open issue**: We should note that a struct default constructor bypasses this. - [ ] **Open issue**: We should specify the execution order of initialization. - [ ] **Open issue**: Should we allow something like a *primary_constructor_body* (presumably without attributes and modifiers) in a non-record type declaration, and treat it like we would the code of an instance field initializer? #### Properties For each record parameter of a record type declaration there is a corresponding `public` property member whose name and type are taken from the value parameter declaration. Its name is the *identifier* of the *record_property_name*, if present, or the *identifier* of the *record_parameter* otherwise. If no concrete (i.e. non-abstract) public property with a `get` accessor and with this name and type is explicitly declared or inherited, it is produced by the compiler as follows: * For a *record struct* or a `sealed` *record class*: * A `private` `readonly` field is produced as a backing field for a `readonly` property. Its value is initialized during construction with the value of the corresponding primary constructor parameter. * The property's `get` accessor is implemented to return the value of the backing field. * Each "matching" inherited virtual property's `get` accessor is overridden. > **Design notes**: In other words, if you extend a base class or implement an interface that declares a public abstract property with the same name and type as a record parameter, that property is overridden or implemented. - [ ] **Open issue**: Should it be possible to change the access modifier on a property when it is explicitly declared? - [ ] **Open issue**: Should it be possible to substitute a field for a property? #### Object Methods For a *record struct* or a `sealed` *record class*, implementations of the methods `object.GetHashCode()` and `object.Equals(object)` are produced by the compiler unless provided by the user. - [ ] **Open issue**: We should precisely specify their implementation. - [ ] **Open issue**: We should also add the interface `IEquatable` for the record type and specify that implementations are provided. - [ ] **Open issue**: We should also specify that we implement every `IEquatable.Equals`. - [ ] **Open issue**: We should specify precisely how we solve the problem of Equals in the face of record inheritance: specifically how we generate equality methods such that they are symmetric, transitive, reflexive, etc. - [ ] **Open issue**: It has been proposed that we implement `operator ==` and `operator !=` for record types. - [ ] **Open issue**: Should we auto-generate an implementation of `object.ToString`? #### `operator is` A record type has a compiler-generated `public static void operator is` unless one with any signature is provided by the user. Its first parameter is the enclosing record type, and each subsequent parameter is an `out` parameter of the same name and type as the corresponding parameter of the record type. The compiler-provided implementation of this method shall assign each `out` parameter with the value of the corresponding property. See [the pattern-matching specification](patterns.md) for the semantics of `operator is`. #### `With` method Unless there is a user-declared member named `With` declared, a record type has a compiler-provided method named `With` whose return type is the record type itself, and containing one value parameter corresponding to each *record-parameter* in the same order that these parameters appear in the record type declaration. Each parameter shall have a *caller-receiver default-argument* of the corresponding property. In an `abstract` record class, the compiler-provided `With` method is abstract. In a record struct, or a sealed record class, the compiler-provided `With` method is `sealed`. Otherwise the compiler-provided `With` method is `virtual and its implementation shall return a new instance produced by invoking the the primary constructor with the parameters as arguments to create a new instance from the parameters, and return that new instance. - [ ] **Open issue**: We should also specify under what conditions we override or implement inherited virtual `With` methods or `With` methods from implemented interfaces. - [ ] **Open issue**: We should say what happens when we inherit a non-virtual `With` method. > **Design notes**: Because record types are by default immutable, the `With` method provides a way of creating a new instance that is the same as an existing instance but with selected properties given new values. For example, given > ```cs > public class Point(int X, int Y); > ``` > there is a compiler-provided member > ```cs > public virtual Point With(int X = this.X, int Y = this.Y) => new Point(X, Y); > ``` > Which enables an variable of the record type > ```cs > var p = new Point(3, 4); > ``` > to be replaced with an instance that has one or more properties different > ```cs > p = p.With(X: 5); > ``` > This can also be expressed using the *with_expression*: > ```cs > p = p with { X = 5 }; > ``` # 5. Examples ### record struct example This record struct ```cs public struct Pair(object First, object Second); ``` is translated to this code ```cs public struct Pair : IEquatable { public object First { get; } public object Second { get; } public Pair(object First, object Second) { this.First = First; this.Second = Second; } public bool Equals(Pair other) // for IEquatable { return Equals(First, other.First) && Equals(Second, other.Second); } public override bool Equals(object other) { return (other as Pair)?.Equals(this) == true; } public override GetHashCode() { return (First?.GetHashCode()*17 + Second?.GetHashCode()).GetValueOrDefault(); } public Pair With(object First = this.First, object Second = this.Second) => new Pair(First, Second); public static void operator is(Pair self, out object First, out object Second) { First = self.First; Second = self.Second; } } ``` - [ ] **Open issue**: should the implementation of Equals(Pair other) be a public member of Pair? - [ ] **Open issue**: This implementation of `Equals` is not symmetric in the face of inheritance. > **Design notes**: Because one record type can inherit from another, and this implementation of `Equals` would not be symmetric in that case, it is not correct. We propose to implement equality this way: > ```cs > public bool Equals(Pair other) // for IEquatable > { > return other != null && EqualityContract == other.EqualityContract && > Equals(First, other.First) && Equals(Second, other.Second); > } > protected virtual Type EqualityContract => typeof(Pair); > ``` > Derived records would `override EqualityContract`. The less attractive alternative is to restrict inheritance. ### sealed record example This sealed record class ```cs public sealed class Student(string Name, decimal Gpa); ``` is translated into this code ```cs public sealed class Student : IEquatable { public string Name { get; } public decimal Gpa { get; } public Student(string Name, decimal Gpa) { this.Name = Name; this.Gpa = Gpa; } public bool Equals(Student other) // for IEquatable { return other != null && Equals(Name, other.Name) && Equals(Gpa, other.Gpa); } public override bool Equals(object other) { return this.Equals(other as Student); } public override int GetHashCode() { return (Name?.GetHashCode()*17 + Gpa?.GetHashCode()).GetValueOrDefault(); } public Student With(string Name = this.Name, decimal Gpa = this.Gpa) => new Student(Name, Gpa); public static void operator is(Student self, out string Name, out decimal Gpa) { Name = self.Name; Gpa = self.Gpa; } } ``` ### abstract record class example This abstract record class ```cs public abstract class Person(string Name); ``` is translated into this code ```cs public abstract class Person : IEquatable { public string Name { get; } public Person(string Name) { this.Name = Name; } public bool Equals(Person other) { return other != null && Equals(Name, other.Name); } public override Equals(object other) { return Equals(other as Person); } public override int GetHashCode() { return (Name?.GetHashCode()).GetValueOrDefault(); } public abstract Person With(string Name = this.Name); public static void operator is(Person self, out string Name) { Name = self.Name; } } ``` ### combining abstract and sealed records Given the abstract record class `Person` above, this sealed record class ```cs public sealed class Student(string Name, decimal Gpa) : Person(Name); ``` is translated into this code ```cs public sealed class Student : Person, IEquatable { public override string Name { get; } public decimal Gpa { get; } public Student(string Name, decimal Gpa) : base(Name) { this.Name = Name; this.Gpa = Gpa; } public override bool Equals(Student other) // for IEquatable { return Equals(Name, other.Name) && Equals(Gpa, other.Gpa); } public bool Equals(Person other) // for IEquatable { return (other as Student)?.Equals(this) == true; } public override bool Equals(object other) { return (other as Student)?.Equals(this) == true; } public override int GetHashCode() { return (Name?.GetHashCode()*17 + Gpa.GetHashCode()).GetValueOrDefault(); } public Student With(string Name = this.Name, decimal Gpa = this.Gpa) => new Student(Name, Gpa); public override Person With(string Name = this.Name) => new Student(Name, Gpa); public static void operator is(Student self, out string Name, out decimal Gpa) { Name = self.Name; Gpa = self.Gpa; } } ``` --- ### Features Records.Work (`docs/features/records.work.md`) Records Work Items ================== This is the TODO list for the development of the records language feature for C# 7. - [ ] This checklist needs to be written - [ ] The feature needs to be implemented - [ ] The feature needs to be tested --- ### Features Ref Reassignment (`docs/features/ref-reassignment.md`) Ref Local and Parameter Reassignment ==================================== In C# 7.2, `ref` locals and parameters cannot be assigned new references, only assigned values which reassign the underlying storage to the given value. C# 7.3 introduces a new type of expression, a ref-assignment expression. The syntax is as follows ``` ref_assignment : identifier '=' 'ref' expression ``` The identifier on the left-hand side of a *ref_assignment* must be a `ref` or `ref readonly` local variable, or an `in`, `ref`, or `out` parameter. Ref assignments allow ref local variables or parameters, which represent references to storage locations, to be assigned different references. This does not include the `this` reference, even in structs. The expression on the right-hand side must have a storage location, i.e. it must be an "l-value." This l-value must be "ref-compatible" with the variable being ref-assigned, where ref-compatible means: * The type of left-hand side and the type of the right-hand side must have an identity conversion between them. * Writeable ref variables cannot be assigned read-only l-values. Readonly ref variables can be assigned either read-only or writeable l-values. The result of the ref-assignment expression is itself an "l-value" and the resulting storage location is read-only if and only if the left-hand side of the assignment is a read-only ref variable. The storage location for the right-hand side must be definitely assigned before it can be used in a ref-assignment expression. In addition, the storage location referenced by an `out` parameter must be definitely assigned before the `out` parameter is ref-reassigned to a new reference. The lifetime for a `ref` local or parameter is fixed on declaration and cannot be reassigned with ref-reassignment. The lifetime requirements for the right-hand expression are the same as for the right-hand expression in a variable assignment, identical to the lifetime for Span<T> assignments. --- ### Features Refout (`docs/features/refout.md`) # Reference assemblies Reference assemblies are metadata-only assemblies with the minimum amount of metadata to preserve the compile-time behavior of consumers (diagnostics may be affected, though). The compiler may choose to remove more metadata in later versions, if it is determined to be safe (ie. respects the principle above). ## Scenarios There are 4 scenarios: 1. The traditional one, where an assembly is emitted as the primary output (`/out` command-line parameter, or `peStream` parameter in `Compilation.Emit` APIs). 2. The IDE scenario, where the metadata-only assembly is emitted (via `Emit` API), still as the primary output. Later on (after C# 7.1), the IDE is interested to get metadata-only assemblies even when there are errors in the compilation. 3. The CoreFX scenario, where only the ref assembly is emitted, still as the primary output (`/refonly` command-line parameter) 4. The MSBuild scenario, which is the new scenario, where both a real assembly is emitted as the primary output, and a ref assembly is emitted as the secondary output (`/refout` command-line parameter, or `metadataPeStream` parameter in `Emit`). ## Definition of ref assemblies Metadata-only assembly have their method bodies replaced with a single `throw null` body, but include all members except anonymous types. The reason for using `throw null` bodies (as opposed to no bodies) is so that PEVerify could run and pass (thus validating the completeness of the metadata). Ref assemblies include an assembly-level `ReferenceAssembly` attribute. This attribute may be specified in source (then we won't need to synthesize it). Because of this attribute, runtimes will refuse to load ref assemblies for execution (but they can still be loaded in reflection-only mode). Some tools may be affected and will need to be updated (for example, `sgen.exe`). Ref assemblies further remove metadata (private members) from metadata-only assemblies: - A ref assembly only has references for what it needs in the API surface. The real assembly may have additional references related to specific implementations. For instance, the ref assembly for `class C { private void M() { dynamic d = 1; ... } }` does not reference any types required for `dynamic`. - Private function-members (methods, properties and events) are removed. If there are no `InternalsVisibleTo` attributes, do the same for internal function-members. - But all types (including private or nested types) are kept in ref assemblies. All attributes are kept (even internal ones), as well as their (internal) constructors. - All virtual methods are kept. Explicit interface implementations are kept. Explicitly-implemented properties and events are kept, as their accessors are virtual (and are therefore kept). - All fields of a struct are kept. (This is a candidate for post-C#-7.1 refinement) - Any resources included on the command-line are not emitted into ref assemblies (produced either with `/refout` or `/refonly`). (This was fixed in dev16) ## API changes ### Command-line Two mutually exclusive command-line parameters were added to `csc.exe` and `vbc.exe`: - `/refout` - `/refonly` The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below). The filename for the ref assembly should generally match that of the primary assembly. The recommended convention (used by MSBuild) is to place the ref assembly in a "ref/" sub-folder relative to the primary assembly. The `/refonly` parameter is a flag that indicates that a ref assembly should be output instead of an implementation assembly, as the primary output. The `/refonly` parameter is not allowed together with the `/refout` parameter, as it doesn't make sense to have both the primary and secondary outputs be ref assemblies. Also, the `/refonly` parameter silently disables outputting PDBs, as ref assemblies cannot be executed. The `/refonly` parameter translates to `EmitMetadataOnly` being `true`, and `IncludePrivateMembers` being `false` in the `Emit` API (see details below). Neither `/refonly` nor `/refout` are permitted with net modules (`/target:module`, `/addmodule` options). The compilation from the command-line either produces both assemblies (implementation and ref) or neither. There is no "partial success" scenario. When the compiler produces documentation, it is un-affected by either the `/refonly` or `/refout` parameters. This may change in the future. The main purpose of the `/refout` option is to speed up incremental build scenarios, so it is acceptable for the current implementation for this flag to produce a ref assembly with more metadata than `/refonly` does (for instance, anonymous types). This is a candidate for post-C#-7.1 refinement. ### CscTask/CoreCompile The `CoreCompile` target supports a new output, called `IntermediateRefAssembly`, which parallels the existing `IntermediateAssembly`. The `Csc` task supports a new output, called `OutputRefAssembly`, which parallels the existing `OutputAssembly`. Both of those basically map to the `/refout` command-line parameter. An additional task, called `CopyRefAssembly`, is provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match (by comparing their MVIDs, see details below), then the destination file is left untouched. As a side-note, `CopyRefAssembly` uses the same assembly resolution/redirection trick as `Csc` and `Vbc`, to avoid type loading problems with `System.IO.FileSystem`. ### CodeAnalysis APIs Prior to C# 7.1, it was already possible to produce metadata-only assemblies by using `EmitOptions.EmitMetadataOnly`, which is used in IDE scenarios with cross-language dependencies. With C# 7.1, the compiler now honours the `EmitOptions.IncludePrivateMembers` flag as well. When combined with `EmitMetadataOnly` or a `metadataPeStream` in `Emit`, a ref assembly is produced. Method bodies aren't compiled when using `EmitMetadataOnly`. Even the diagnostic check for emitting methods lacking a body (`void M();`) is filtered from declaration diagnostics, so such code will successfully emit with `EmitMetadataOnly`. Later on, the `EmitOptions.TolerateErrors` flag will allow emitting error types as well. `Emit` was modified to produce a new PE section called ".mvid" containing a copy of the MVID, when emitting ref assemblies. This makes it easy for `CopyRefAssembly` to extract and compare MVIDs from ref assemblies. Going back to the 4 driving scenarios: 1. For a regular compilation, `EmitMetadataOnly` is left to `false` and no `metadataPeStream` is passed into `Emit`. 2. For the IDE scenario, `EmitMetadataOnly` is set to `true`, but `IncludePrivateMembers` is left to `true`. 3. For the CoreFX scenario, ref assembly source code is used as input, `EmitMetadataOnly` is set to `true`, and `IncludePrivateMembers` is set to `false`. 4. For the MSBuild scenario, `EmitMetadataOnly` is left to `false`, a `metadataPeStream` is passed in and `IncludePrivateMembers` is set to `false`. ### Determinism We recommend that you always use determinism with reference assemblies. This minimizes the rate of change for the ref assembly, thereby maximizing the benefits they realize. That said, even if determinism isn't set, compilation of ref assemblies is [largely deterministic](http://blog.paranoidcoding.com/2016/04/05/deterministic-builds-in-roslyn.html) by default. The main exception is when using `AssemblyVersionAttribute` with a wildcard (for example, `[assembly: System.Reflection.AssemblyVersion("1.0.*")]`). In such case, the compilation is necessarily non-deterministic and therefore ref assemblies don't provide any benefits. ## MSBuild * `ProduceReferenceAssembly` (boolean) controls whether to create the item passed to the compiler task (and thus pass `/refout:`). It requires opt-in. It is recommended that `Deterministic` also be set for best result (see details above). Cannot be used in conjunction with `ProduceOnlyReferenceAssembly` * `ProduceOnlyReferenceAssembly` (boolean) controls whether to pass `/refonly` to the compiler. It requires opt-in. Cannot be used in conjunction with `ProduceReferenceAssembly` * If you encounter a problem using reference assemblies, you can set the boolean property `CompileUsingReferenceAssemblies` to `false` to avoid using ref assemblies even if the projects you reference produce them. This is unset by default and only ever checked against `false`. It is only there to provide an emergency escape hatch; a customer who hits a bug can set it to `false` and avoid the new codepaths. ## Future As mentioned above, there may be further refinements after C# 7.1: - Further reduce the metadata in ref assemblies produced by `/refout`, to match those produced by `/refonly`. - Controlling internals so they are not included despite `InternalsVisibleTo` (producing public ref assemblies) - Produce ref assemblies even when there are errors outside method bodies (emitting error types when `EmitOptions.TolerateErrors` is set) - When the compiler produces documentation, the contents produced could be filtered down to match the APIs that go into the primary output. In other words, the documentation could be filtered down when using the `/refonly` parameter. ## Open questions ## Related issues - Produce ref assemblies from command-line and msbuild (https://github.com/dotnet/roslyn/issues/2184) - Refine what is in reference assemblies and what diagnostics prevent generating one (https://github.com/dotnet/roslyn/issues/17612) - [Are private members part of the API surface?](https://blog.paranoidcoding.org/2016/02/15/are-private-members-api-surface.html) - MSBuild work items and design notes (https://github.com/Microsoft/msbuild/issues/1986) - Fast up-to-date check in project-system (https://github.com/dotnet/project-system/issues/2254) --- ### Features Sdk (`docs/features/sdk.md`) Moving dotnet/roslyn to the new SDK ============= # Overview This document tracks our progress in moving Roslyn to the new SDK and project system. The goal of this initial push is explicitly to remove the use of project.json from our code base. In doing many of our projects will move to target the new SDK, but not all. Once this change is merged we will do further cleanup to move everything to the new SDK. The projects in our repo that need to be converted break down into the following categories: - PCL: This will be converted to the new SDK and Project System - Desktop 4.6: Package consumption will change to PackageReference - Desktop 2.0: Package consumption will change to PackageReference The conversion of our projects is being almost entirely automated with the [ConvertPackageRef Tool](https://github.com/jaredpar/ConvertPackageRef). This automation is always checked in as a single commit titled "Generated code". It is frequently replaced via a rebase as the tool improves. # Branch State All deliberately disabled features are marked with the comment "DO NOT MERGE". These entries will be removed + fixed before merging ## Working - Developer work flow: Restore.cmd, Build.cmd, Test.cmd - Editing in Visual Studio ## Jenkins Legs This set of Jenkins legs is considered functional and must pass on every merge: - Unit tests debug / release on x86 / amd64 - Build correctness - Microbuild - CoreClr - Linux - Integration Tests ## Big Items These are the big items that are broken and need some work - RepoUtil: needs to understand PackageReference and can be massively simplified now - Delete the DeployCompilerGeneratorToolsRuntime project. Not needed anymore --- ### Features Skiplocalsinit (`docs/features/skiplocalsinit.md`) # Feature design for SkipLocalsInit The feature is to add a new well-known attribute to the compiler, System.Runtime.CompilerServices.SkipLocalsInit that causes method bodies nested "inside" the scope of the attribute to elide the ".locals init" CIL directive that causes the CLR to zero-init local variables and space reserved using the "localloc" instruction. "Nested inside the scope of the attribute" is a concept defined as follows: 1. When the attribute is applied to a module, all emitted methods inside that module, including generated methods, will skip local initialization. 2. When the attribute is applied to a type (including interfaces), all methods transitively inside that type, including generated methods, will skip local initialization. This includes nested types. 3. When the attribute is applied to a method, the method and all methods generated by the compiler which contain user-influenced code inside that method (e.g., local functions, lambdas, async methods) will skip local initialization. For the above, generated methods will skip local initialization only if they have user code inside them, or a significant number of compiler-generated locals. For example, the MoveNext method of an async method will skip local initialization, but the constructor of a display class probably will not, as the display class constructor likely has no locals anyway. Some notable decisions: 1. Although SkipLocalsInit does not require unsafe code blocks, it does require the `unsafe` flag to be passed to the compiler. This is to signal that in some cases code could view unassigned memory, including `stackalloc`. 2. Due to the compiler not controlling localsinit for other modules when emitting to a netmodule, SkipLocalsInit is not respected on the assembly level, even if the AttributeUsage of the SkipLocalsInitAttribute is specified that it is allowed. 3. The `Inherited` property of the SkipLocalsInitAttribute is not respected. SkipLocalsInit does not propagate through inheritance. --- ### Features Source Generators.Cookbook (`docs/features/source-generators.cookbook.md`) # Source Generators Cookbook ## Summary > **Warning**: Source generators implementing `ISourceGenerator` have been deprecated > in favor of [incremental generators](incremental-generators.md). > The incremental version of this document is [here](incremental-generators.cookbook.md). > You should implement `IIncrementalGenerator` instead of `ISourceGenerator`. This document aims to be a guide to help the creation of source generators by providing a series of guidelines for common patterns. It also aims to set out what types of generators are possible under the current design, and what is expected to be explicitly out of scope in the final design of the shipping feature. **This document expands on the details in the [full design document](source-generators.md), please ensure you have read that first.** ## Table of content - [Source Generators Cookbook](#source-generators-cookbook) - [Table of content](#table-of-content) - [Summary](#summary) - [Proposal](#proposal) - [Out of scope designs](#out-of-scope-designs) - [Language features](#language-features) - [Code rewriting](#code-rewriting) - [Conventions](#conventions) - [Designs](#designs) - [Generated class](#generated-class) - [Additional file transformation](#additional-file-transformation) - [Augment user code](#augment-user-code) - [Issue Diagnostics](#issue-diagnostics) - [INotifyPropertyChanged](#inotifypropertychanged) - [Package a generator as a NuGet package](#package-a-generator-as-a-nuget-package) - [Use functionality from NuGet packages](#use-functionality-from-nuget-packages) - [Access Analyzer Config properties](#access-analyzer-config-properties) - [Consume MSBuild properties and metadata](#consume-msbuild-properties-and-metadata) - [Unit Testing of Generators](#unit-testing-of-generators) - [Participate in the IDE experience](#participate-in-the-ide-experience) - [Serialization](#serialization) - [Auto interface implementation](#auto-interface-implementation) - [Breaking Changes:](#breaking-changes) - [Open Issues](#open-issues) ## Proposal As a reminder, the high level design goals of source generators are: - Generators produce one or more strings that represent C# source code to be added to the compilation. - Explicitly _additive_ only. Generators can add new source code to a compilation but may **not** modify existing user code. - Can produce diagnostics. When unable to generate source, the generator can inform the user of the problem. - May access _additional files_, that is, non-C# source texts. - Run _un-ordered_, each generator will see the same input compilation, with no access to files created by other source generators. - A user specifies the generators to run via list of assemblies, much like analyzers. ## Out of scope designs We will briefly look at the non-solvable problems as examples of the kind of problems source generators are *not* designed to solve: ### Language features Source generators are not designed to replace new language features: for instance one could imagine [records](records.md) being implemented as a source generator that converts the specified syntax to a compilable C# representation. We explicitly consider this to be an anti-pattern; the language will continue to evolve and add new features, and we don't expect source generators to be a way to enable this. Doing so would create new 'dialects' of C# that are incompatible with the compiler without generators. Further, because generators, by design, cannot interact with each other, language features implemented in this way would quickly become incompatible with other additions to the language. ### Code rewriting There are many post-processing tasks that users perform on their assemblies today, which here we define broadly as 'code rewriting'. These include, but are not limited to: - Optimization - Logging injection - IL Weaving - Call site re-writing While these techniques have many valuable use cases, they do not fit into the idea of *source generation*. They are, by definition, code altering operations which are explicitly ruled out by the source generators proposal. There are already well supported tools and techniques for achieving these kinds of operations, and the source generators proposal is not aimed at replacing them. ## Conventions TODO: List a set of general conventions that apply to all designs below. E.g. Re-using namespaces, generated file names etc. ## Designs This section is broken down by user scenarios, with general solutions listed first, and more specific examples later on. ### Generated class **User scenario:** As a generator author I want to be able to add a type to the compilation, that can be referenced by the user's code. **Solution:** Have the user write the code as if the type was already present. Generate the missing type based on information available in the compilation. **Example:** Given the following user code: ```csharp public partial class UserClass { public void UserMethod() { // call into a generated method GeneratedNamespace.GeneratedClass.GeneratedMethod(); } } ``` Create a generator that will create the missing type when run: ```csharp [Generator] public class CustomGenerator : ISourceGenerator { public void Initialize(GeneratorInitializationContext context) {} public void Execute(GeneratorExecutionContext context) { context.AddSource("myGeneratedFile.cs", SourceText.From(@" namespace GeneratedNamespace { public class GeneratedClass { public static void GeneratedMethod() { // generated code } } }", Encoding.UTF8)); } } ``` ### Additional file transformation **User scenario:** As a generator author I want to be able to transform an external non-C# file into an equivalent C# representation. **Solution:** Use the additional files property of the `GeneratorExecutionContext` to retrieve the contents of the file, convert it to the C# representation and return it. **Example:** ```csharp [Generator] public class FileTransformGenerator : ISourceGenerator { public void Initialize(GeneratorInitializationContext context) {} public void Execute(GeneratorExecutionContext context) { // find anything that matches our files var myFiles = context.AdditionalFiles.Where(at => at.Path.EndsWith(".xml")); foreach (var file in myFiles) { var content = file.GetText(context.CancellationToken); // do some transforms based on the file context string output = MyXmlToCSharpCompiler.Compile(content); var sourceText = SourceText.From(output, Encoding.UTF8); context.AddSource($"{file.Name}generated.cs", sourceText); } } } ``` ### Augment user code **User scenario:** As a generator author I want to be able to inspect and augment a user's code with new functionality. **Solution:** Require the user to make the class you want to augment be a `partial class`, and mark it with e.g. a unique attribute, or name. Register a `SyntaxReceiver` that looks for any classes marked for generation and records them. Retrieve the populated `SyntaxReceiver` during the generation phase and use the recorded information to generate a matching `partial class` that contains the additional functionality. **Example:** ```csharp public partial class UserClass { public void UserMethod() { // call into a generated method inside the class this.GeneratedMethod(); } } ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Issue Diagnostics **User Scenario:** As a generator author I want to be able to add diagnostics to the users compilation. **Solution:** Diagnostics can be added to the compilation via `GeneratorExecutionContext.ReportDiagnostic()`. These can be in response to the content of the users compilation: for instance if the generator is expecting a well formed `AdditionalFile` but can not parse it, the generator could emit a warning notifying the user that generation can not proceed. For code-based issues, the generator author should also consider implementing a [diagnostic analyzer](https://docs.microsoft.com/en-us/visualstudio/code-quality/roslyn-analyzers-overview?view=vs-2019) that identifies the problem, and offers a code-fix to resolve it. **Example:** ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### INotifyPropertyChanged **User scenario:** As a generator author I want to be able to implement the `INotifyPropertyChanged` pattern automatically for a user. **Solution:** The design tenant 'Explicitly additive only' seems to be at direct odds with the ability to implement this, and appears to call for user code modification. However we can instead take advantage of explicit fields and instead of *editing* the users properties, directly provide them for listed fields. **Example:** Given a user class such as: ```csharp using AutoNotify; public partial class UserClass { [AutoNotify] private bool _boolProp; [AutoNotify(PropertyName = "Count")] private int _intProp; } ``` A generator could produce the following: ```csharp using System; using System.ComponentModel; namespace AutoNotify { [AttributeUsage(AttributeTargets.Field, Inherited = false, AllowMultiple = false)] sealed class AutoNotifyAttribute : Attribute { public AutoNotifyAttribute() { } public string PropertyName { get; set; } } } public partial class UserClass : INotifyPropertyChanged { public bool BoolProp { get => _boolProp; set { _boolProp = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs("UserBool")); } } public int Count { get => _intProp; set { _intProp = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs("Count")); } } public event PropertyChangedEventHandler PropertyChanged; } ``` ### Package a generator as a NuGet package **User scenario**: As a generator author I want to package my generator as a NuGet package for consumption. **Solution:** Generators can be packaged using the same method as an Analyzer would. Ensure the generator is placed in the `analyzers\dotnet\cs` folder of the package for it to be automatically added to the users project on install. For example, to turn your generator project into a NuGet package at build, add the following to your project file: ```xml true false ``` ### Use functionality from NuGet packages **User Scenario:** As a generator author I want to rely on functionality provided in NuGet packages inside my generator. **Solution:** It is possible to depend on NuGet packages inside of a generator, but special consideration has to be taken for distribution. Any *runtime* dependencies, that is, code that the end users program will need to rely on, can simply be added as a dependency of the generator NuGet package via the usual referencing mechanism. For example, consider a generator that creates code that relies on `Newtonsoft.Json`. The generator does not directly use the dependency, it just emits code that relies on the library being referenced in the users compilation. The author would add a reference to `Newtonsoft.Json` as a public dependency, and when the user adds the generator package it will referenced automatically. The generator can check the compilation for the presence of the `Newtonsoft.Json` assembly and issue a warning or error if not present. ```xml true false ``` ```C# using System.Linq; [Generator] public class SerializingGenerator : ISourceGenerator { public void Execute(GeneratorExecutionContext context) { // check that the users compilation references the expected library if (!context.Compilation.ReferencedAssemblyNames.Any(ai => ai.Name.Equals("Newtonsoft.Json", StringComparison.OrdinalIgnoreCase))) { context.ReportDiagnostic(/*error or warning*/); } } public void Initialize(GeneratorInitializationContext context) { } } ``` However, any *generation-time* dependencies, that is, used by the generator while it is is running and generating code, must be packaged directly alongside the generator assembly inside the generator NuGet package. There are no automatic facilities for this, and you will need to manually specify the dependencies to include. Consider a generator that uses `Newtonsoft.Json` to encode something to json during the generation pass, but does not emit any code the relies on it being present at runtime. The author would add a reference to `Newtonsoft.Json` but make all of its assets *private*; this ensures the consumer of the generator does not inherit a dependency on the library. The author would then have to package the `Newtonsoft.Json` library alongside the generator inside of the NuGet package. This can be achieved in the following way: set the dependency to generate a path property by adding `GeneratePathProperty="true"`. This will create a new MSBuild property of the format `PKG` where `` is the package name with `.` replaced by `_`. In our example there would be an MSBuild property called `PKGNewtonsoft_Json` with a value that points to the path on disk of the binary contents of the NuGet files. We can then use that to add the binaries to the resulting NuGet package as we do with the generator itself: ```xml true false ``` ```C# [Generator] public class JsonUsingGenerator : ISourceGenerator { public void Execute(GeneratorExecutionContext context) { // use the newtonsoft.json library, but don't add any source code that depends on it var serializedContent = Newtonsoft.Json.JsonConvert.SerializeObject(new { a = "a", b = 4 }); context.AddSource("myGeneratedFile.cs", SourceText.From($@" namespace GeneratedNamespace {{ public class GeneratedClass {{ public static const SerializedContent = {serializedContent}; }} }}", Encoding.UTF8)); } public void Initialize(GeneratorInitializationContext context) { } } ``` ### Access Analyzer Config properties **Implementation status:** Available from VS 16.7 preview3. **User Scenarios:** - As a generator author I want to access the analyzer config properties for a syntax tree or additional file. - As a generator author I want to access key-value pairs that customize the generator output. - As a user of a generator I want to be able to customize the generated code and override defaults. **Solution**: Generators can access analyzer config values via the `AnalyzerConfigOptions` property of the `GeneratorExecutionContext`. Analyzer config values can either be accessed in the context of a `SyntaxTree`, `AdditionalFile` or globally via `GlobalOptions`. Global options are 'ambient' in that they don't apply to any specific context, but will be included when requesting option within a specific context. A generator is free to use a global option to customize its output. For example, consider a generator that can optionally emit logging. The author may choose to check the value of a global analyzer config value in order to control whether or not to emit the logging code. A user can then choose to enable the setting per project via an `.editorconfig` file: ```.editorconfig mygenerator_emit_logging = true ``` ```csharp [Generator] public class MyGenerator : ISourceGenerator { public void Execute(GeneratorExecutionContext context) { // control logging via analyzerconfig bool emitLogging = false; if (context.AnalyzerConfigOptions.GlobalOptions.TryGetValue("mygenerator_emit_logging", out var emitLoggingSwitch)) { emitLogging = emitLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase); } // add the source with or without logging... } public void Initialize(GeneratorInitializationContext context) { } } ``` ### Consume MSBuild properties and metadata **Implementation status:** Available from VS 16.7 preview3. **User Scenarios:** - As a generator author I want to make decisions based on the values contained in the project file - As a user of a generator I want to be able to customize the generated code and override defaults. **Solution:** MSBuild will automatically translate specified properties and metadata into a global analyzer config that can be read by a generator. A generator author specifies the properties and metadata they want to make available by adding items to the `CompilerVisibleProperty` and `CompilerVisibleItemMetadata` item groups. These can be added via a props or targets file when packaging the generator as a NuGet package. For example, consider a generator that creates source based on additional files, and wants to allow a user to enable or disable logging via the project file. The author would specify in their props file that they want to make the specified MSBuild property visible to the compiler: ```xml ``` The value of `MyGenerator_EnableLogging` property will then be emitted to a generated analyzer config file before build, with a name of `build_property.MyGenerator_EnableLogging`. The generator is then able read this property from via the `AnalyzerConfigOptions` property of the `GeneratorExecutionContext`: ```c# context.AnalyzerConfigOptions.GlobalOptions.TryGetValue("build_property.MyGenerator_EnableLogging", out var emitLoggingSwitch); ``` A user can thus enable, or disable logging, by setting a property in their project file. Now, consider that the generator author wants to optionally allow opting in/out of logging on a per-additional file basis. The author can request that MSBuild emit the value of metadata for the specified file, by adding to the `CompilerVisibleItemMetadata` item group. The author specifies both the MSBuild itemType they want to read the metadata from, in this case `AdditionalFiles`, and the name of the metadata that they want to retrieve for them. ```xml ``` This value of `MyGenerator_EnableLogging` will be emitted to a generated analyzer config file, for each of the additional files in the compilation, with an item name of `build_metadata.AdditionalFiles.MyGenerator_EnableLogging`. The generator can read this value in the context of each additional file: ```cs foreach (var file in context.AdditionalFiles) { context.AnalyzerConfigOptions.GetOptions(file).TryGetValue("build_metadata.AdditionalFiles.MyGenerator_EnableLogging", out var perFileLoggingSwitch); } ``` In the users project file, the user can now annotate the individual additional files to say whether or not they want to enable logging: ```xml ``` Note that MSBuild properties passed to source generators via `CompilerVisibleProperty` are written into and read from an editorconfig file, [resulting in data loss for non-trivial property values](https://github.com/dotnet/roslyn/issues/51692). One possible workaround is to use a build task to apply a transport encoding preventing the data loss; as an example a semicolon separated list can be converted to a space separated list (`;` is the editorconfig comment character): ```xml <_MyInterpolatorsNamespaces>$([System.String]::Copy('$(InterpolatorsNamespaces)').Replace(';', ' ')) ``` **Full Example:** MyGenerator.props: ```xml ``` MyGenerator.csproj: ```xml true false ``` MyGenerator.cs: ```csharp [Generator] public class MyGenerator : ISourceGenerator { public void Execute(GeneratorExecutionContext context) { // global logging from project file bool emitLoggingGlobal = false; if(context.AnalyzerConfigOptions.GlobalOptions.TryGetValue("build_property.MyGenerator_EnableLogging", out var emitLoggingSwitch)) { emitLoggingGlobal = emitLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase); } foreach (var file in context.AdditionalFiles) { // allow the user to override the global logging on a per-file basis bool emitLogging = emitLoggingGlobal; if (context.AnalyzerConfigOptions.GetOptions(file).TryGetValue("build_metadata.AdditionalFiles.MyGenerator_EnableLogging", out var perFileLoggingSwitch)) { emitLogging = perFileLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase); } // add the source with or without logging... } } public void Initialize(GeneratorInitializationContext context) { } } ``` ### Unit Testing of Generators **User scenario**: As a generator author, I want to be able to unit test my generators to make development easier and ensure correctness. **Solution A**: The recommended approach is to use [Microsoft.CodeAnalysis.Testing](https://github.com/dotnet/roslyn-sdk/tree/main/src/Microsoft.CodeAnalysis.Testing#microsoftcodeanalysistesting) packages: - `Microsoft.CodeAnalysis.CSharp.SourceGenerators.Testing` - `Microsoft.CodeAnalysis.VisualBasic.SourceGenerators.Testing` This works in the same way as analyzers and codefix testing. You add a class like the following: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Then, in your test file: ```csharp using VerifyCS = CSharpSourceGeneratorVerifier; ``` And use the following in your test method: ```csharp var code = "initial code"; var generated = "expected generated code"; await new VerifyCS.Test { TestState = { Sources = { code }, GeneratedSources = { (typeof(YourGenerator), "GeneratedFileName", SourceText.From(generated, Encoding.UTF8, SourceHashAlgorithm.Sha256)), }, }, }.RunAsync(); ``` **Solution B:** Another approach without using the testing library is that a user can host the `GeneratorDriver` directly within a unit test, making the generator portion of the code relatively simple to unit test. A user will need to provide a compilation for the generator to operate on, and can then probe either the resulting compilation, or the `GeneratorDriverRunResult` of the driver to see the individual items added by the generator. Starting with a basic generator that adds a single source file: ```csharp [Generator] public class CustomGenerator : ISourceGenerator { public void Initialize(GeneratorInitializationContext context) {} public void Execute(GeneratorExecutionContext context) { context.AddSource("myGeneratedFile.cs", SourceText.From(@" namespace GeneratedNamespace { public class GeneratedClass { public static void GeneratedMethod() { // generated code } } }", Encoding.UTF8)); } } ``` As a user, we can host it in a unit test like so: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Note: the above example uses MSTest, but the contents of the test are easily adapted to other frameworks, such as XUnit. ### Participate in the IDE experience **User scenario:** As a generator author I want to be able to interactively regenerate code as the user is editing files. **Solution:** See [incremental generators](incremental-generators.md). ### Serialization **User Scenario** Serialization is often implemented using _dynamic analysis_, i.e. serializers often use reflection to examine the runtime state of a given type and generate serialization logic. This can be expensive and brittle. If the compile-time type and the runtime-type are similar, it could be useful to move much of the cost to compile-time, instead of run-time. Source generators provide a way to do this. Since source generators can be delivered via NuGet the same way analyzers can, we anticipate this would be a use-case for a source generator library, as opposed to everyone building their own. **Solution** To start, the generator will need some way to discover which types are meant to be serializable. One indicator could be an attribute, e.g. ```C# [GeneratorSerializable] partial class MyRecord { public string Item1 { get; } public int Item2 { get; } } ``` This attribute could also be used for [Participate in the IDE experience](#participate-in-the-ide-experience), when the full scope of that feature is fully designed. In that scenario, instead of the generator finding every type marked with the given attribute, the compiler would notify the generator of every type marked with the given attribute. For now we'll assume that the types are provided to us. The first task is to decide what we want our serialization to return. Let's say we do a simple JSON serialization that produces a string like the following ```json { "Item1": "abc", "Item2": 11, } ``` For that we could add a `Serialize` method to our record type like the following: ```C# public string Serialize() { var sb = new StringBuilder(); sb.AppendLine("{"); int indent = 8; // Body addWithIndent($"\"Item1\": \"{this.Item1.ToString()}\","); addWithIndent($"\"Item2\": {this.Item2.ToString()},"); sb.AppendLine("}"); return sb.ToString(); void addWithIndent(string s) { sb.Append(' ', indent); sb.AppendLine(s); } } ``` Obviously this is heavily simplified -- this example only handles the `string` and `int` types properly, adds a trailing comma to the json output and has no error recovery, but it should serve to demonstrate the kind of code a source generator could add to a compilation. Our next task is design a generator to generate the above code, since the above code is itself customized in the `// Body` section according to the actual properties in the class. In other words, we need to generate the code which will generate the JSON format. This is a generator-generator. Let's start with a basic template. We are adding a full source generator, so we'll need to generate a class with the same name as the input class, with a public method called `Serialize`, and a filler area where we write out the properties. ```C# string template = @" using System.Text; partial class {0} {{ public string Serialize() {{ var sb = new StringBuilder(); sb.AppendLine(""{{""); int indent = 8; // Body {1} sb.AppendLine(""}}""); return sb.ToString(); void addWithIndent(string s) {{ sb.Append(' ', indent); sb.AppendLine(s); }} }} }}"; ``` Now that we know the general structure of the code, we need to examine the input type and find all the right info to fill in. This information is all available in a C# SyntaxTree in our example. Let's say we were given a `ClassDeclarationSyntax` that was confirmed to have a generation attribute attached. Then we could grab the name of the class and the name of it's properties as follows: ```C# private static string Generate(ClassDeclarationSyntax c) { var className = c.Identifier.ToString(); var propertyNames = new List(); foreach (var member in c.Members) { if (member is PropertyDeclarationSyntax p) { propertyNames.Add(p.Identifier.ToString()); } } } ``` This is really all we need. If the serialized values of the properties is their string value, the generated code just needs to call `ToString()` on them. The only remaining question is what `using`s to put at the top of the file. Since our template uses a string builder, we'll need `System.Text` for that, but all other types appear to be primitives, so that's all we'll need. Putting it all together: ```C# private static string Generate(ClassDeclarationSyntax c) { var sb = new StringBuilder(); int indent = 8; foreach (var member in c.Members) { if (member is PropertyDeclarationSyntax p) { var name = p.Identifier.ToString(); appendWithIndent($"addWithIndent($\"\\\"{name}\\\": "); if (p.Type.ToString() != "int") { sb.Append("\\\""); } sb.Append($"{{this.{name}.ToString()}}"); if (p.Type.ToString() != "int") { sb.Append("\\\""); } sb.AppendLine(",\");"); } } return $@" using System.Text; partial class {c.Identifier.ToString()} {{ public string Serialize() {{ var sb = new StringBuilder(); sb.AppendLine(""{{""); int indent = 8; // Body {sb.ToString()} sb.AppendLine(""}}""); return sb.ToString(); void addWithIndent(string s) {{ sb.Append(' ', indent); sb.AppendLine(s); }} }} }}"; void appendWithIndent(string s) { sb.Append(' ', indent); sb.Append(s); } } ``` This ties cleanly into the other serialization examples. By finding all the appropriate class declarations in the Compilation's SyntaxTrees and passing them to the above Generate method we can build new partial classes for each type that opt-ed in to generated serialization. Unlike other technologies, this serialization mechanism happens entirely at compile time and can be specialized exactly to what was written in the user class. ### Auto interface implementation TODO: ## Breaking Changes: **Implementation status:** Implemented in Visual Studio 16.8 preview3 / roslyn version 3.8.0-3.final onwards Between preview and release the following breaking changes were introduced: **Rename `SourceGeneratorContext` to `GeneratorExecutionContext`** **Rename `IntializationContext` to `GeneratorInitializationContext`** This affects user-authored generators, as it means the base interface has changed to: ```csharp public interface ISourceGenerator { void Initialize(GeneratorInitializationContext context); void Execute(GeneratorExecutionContext context); } ``` Users attempting to use a generator targeting the preview APIs against a later version of Roslyn will see an exception similar to: ```csharp CSC : warning CS8032: An instance of analyzer Generator.HelloWorldGenerator cannot be created from Generator.dll : Method 'Initialize' in type 'Generator.HelloWorldGenerator' from assembly 'Generator, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation. [Consumer.csproj] ``` The required action from the user is to rename the parameter types of the `Initialize` and `Execute` methods to match. **Rename `RunFullGeneration` to `RunGeneratorsAndUpdateCompilation`** **Add `Create()` static methods to `CSharpGeneratorDriver` and obsolete the constructor.** This affects any generator authors who have written unit tests using the `CSharpGeneratorDriver`. To create a new instance of the generator driver, the user should no longer call `new` but use one of the `CSharpGeneratorDriver.Create()` overloads. The user should no longer use the `RunFullGeneration` method, and instead call `RunGeneratorsAndUpdateCompilation` with the same arguments. ## Open Issues This section track other miscellaneous TODO items: **Framework targets**: May want to mention if we have framework requirements for the generators, e.g. they must target netstandard2.0 or similar. **Conventions**: (See TODO in [conventions](#conventions) section above). What standard conventions are we suggesting to users? **Partial methods**: Should we provide a scenario that includes partial methods? Reasons: - Control of name. The developer can control the name of the member - Generation is optional/depending on other state. Based on other information, generator might decide that the method isn't needed. **Feature detection**: Show how to create a generator that relies on specific target framework features, without depending on the TargetFramework property. --- ### Features Source Generators (`docs/features/source-generators.md`) # Source Generators ## Summary > **Warning**: Source generators implementing `ISourceGenerator` have been deprecated > in favor of [incremental generators](incremental-generators.md). Source generators aim to enable _compile time metaprogramming_, that is, code that can be created at compile time and added to the compilation. Source generators will be able to read the contents of the compilation before running, as well as access any _additional files_, enabling generators to introspect both user C# code and generator specific files. > **Note**: This proposal is separate from the [previous generator design](generators.md) ### High Level Design Goals - Generators produce one or more strings that represent C# source code to be added to the compilation. - Explicitly _additive_ only. Generators can add new source code to a compilation but may **not** modify existing user code. - Can produce diagnostics. When unable to generate source, the generator can inform the user of the problem. - May access _additional files_, that is, non-C# source texts. - Run _un-ordered_, each generator will see the same input compilation, with no access to files created by other source generators. - A user specifies the generators to run via list of assemblies, much like analyzers. ## Implementation At the simplest level source generators are an implementation of `Microsoft.CodeAnalysis.ISourceGenerator` ```csharp namespace Microsoft.CodeAnalysis { public interface ISourceGenerator { void Initialize(GeneratorInitializationContext context); void Execute(GeneratorExecutionContext context); } } ``` Generator implementations are defined in external assemblies passed to the compiler using the same `-analyzer:` option used for diagnostic analyzers. Implementations are required to be annotated with a `Microsoft.CodeAnalysis.GeneratorAttribute` attribute. An assembly can contain a mix of diagnostic analyzers and source generators. Since generators are loaded from external assemblies, a generator cannot be used to build the assembly in which it is defined. `ISourceGenerator` has an `Initialize` method that is called by the host (either the IDE or the command-line compiler) exactly once. `Initialize` passes an instance of `GeneratorInitializationContext` which can be used by the generator to register a set of callbacks that affect how future generation passes will occur. The main generation pass occurs via the `Execute` method. `Execute` passes an instance of `GeneratorExecutionContext` that provides access to the current `Compilation` and allows the generator to alter the resulting output `Compilation` by adding source and reporting diagnostics. The generator is also able to access any `AnalyzerAdditionalFiles` passed to the compiler via the `AdditionalFiles` collection, allowing for generation decisions to based on more than just the user's C# code. ```csharp namespace Microsoft.CodeAnalysis { public readonly struct GeneratorExecutionContext { public ImmutableArray AdditionalFiles { get; } public CancellationToken CancellationToken { get; } public Compilation Compilation { get; } public ISyntaxReceiver? SyntaxReceiver { get; } public void ReportDiagnostic(Diagnostic diagnostic) { throw new NotImplementedException(); } public void AddSource(string fileNameHint, SourceText sourceText) { throw new NotImplementedException(); } } } ``` It is assumed that some generators will want to generate more than one `SourceText`, for example in a 1:1 mapping for additional files. The `fileNameHint` parameter of `AddSource` is intended to address this: 1. If the generated files are emitted to disk, having some ability to put some distinguishing text might be useful. For example, if you have two `.resx` files, generating the files with simply names of `ResxGeneratedFile1.cs` and `ResxGeneratedFile2.cs` wouldn't be terribly useful -- you'd want it to be something like `ResxGeneratedFile-Strings.cs` and `ResxGeneratedFile-Icons.cs` if you had two `.resx` files named "Strings" and "Icons" respectively. 2. The IDE needs some concept of a "stable" identifier. Source generators create a couple of fun problems for the IDE: users will want to be able to set breakpoints in a generated file, for example. If a source generator outputs multiple files we need to know which is which so we can know which file the breakpoints go with. A source generator of course is allowed to stop emitting a file if its inputs change (if you delete a `.resx`, then the generated file associated with it will also go away), but this gives us some control here. This was called "hint" in that the compiler is implicitly allowed to control the filename in however it ultimately needs, and if two source generators give the same "hint" it can still distinguish them with any sort of prefix/suffix as necessary. ### IDE Integration One of the more complicated aspects of supporting generators is enabling a high-fidelity experience in Visual Studio. For the purposes of determining code correctness, it is expected that all generators will have had to be run. Obviously, it is impractical to run every generator on every keystroke, and still maintain an acceptable level of performance within the IDE. #### Progressive complexity opt-in It is expected instead that source generators would work on an 'opt-in' approach to IDE enablement. By default, a generator implementing only `ISourceGenerator` would see no IDE integration and only be correct at build time. Based on conversations with 1st party customers, there are several cases where this would be enough. However, for scenarios such as code first gRPC, and in particular Razor and Blazor, the IDE will need to be able to generate code on-the-fly as those file types are edited and reflect the changes back to other files in the IDE in near real-time. The proposal is to have a set of advanced callbacks that can be optionally implemented, that would allow the IDE to query the generator to decide what needs to be run in the case of any particular edit. For example an extension that would cause generation to run after saving a third party file might look something like: ```csharp namespace Microsoft.CodeAnalysis { public struct GeneratorInitializationContext { public void RegisterForAdditionalFileChanges(EditCallback callback){ } } } ``` This would allow the generator to register a callback during initialization that would be invoked every time an additional file changes. It is expected that there will be various levels of opt in, that can be added to a generator in order to achieve the specific level of performance required of it. What these exact APIs will look like remains an open question, and it's expected that we will need to prototype some real-world generators before knowing what their precise shape will be. ### Output files It is desirable that the generated source texts be available for inspection after generation, either as part of creating a generator or seeing what code was generated by a third party generator. By default, generated texts will be persisted to a `GeneratedFiles/{GeneratorAssemblyName}` sub-folder within `CommandLineArguments.OutputDirectory`. The `fileNameHint` from `GeneratorExecutionContext.AddSource` will be used to create a unique name, with appropriate collision renaming applied if required. For instance, on Windows a call to `AddSource("MyCode", ...);` from `MyGenerator.dll` for a C# project might be persisted as `obj/debug/GeneratedFiles/MyGenerator.dll/MyCode.cs`. File output is not required for the correct function of either command line or IDE based generation, and can be completely disabled, if required. The IDE will work on in-memory copies of the generated source texts (for 'Find all references', breakpoints etc.) and periodically flush any changes to disk. To support the use case where a user wishes to generate the source text, then commit the generated files to source control, we will allow changing the location of the generated files via an appropriate command line switch, and matching MSBuild property (naming still to be determined). In these cases it will be up to the user if they wish to generate over the files again in the future (in which case they would still be generated, but output to a source controlled location), or remove the generators and perform the action as a one time step. It is currently an open question how for example, the action of setting a breakpoint in a disk-based generated file will function. TK: how do we save PDBs/Source link etc? ### Editing experiences for third party languages One of the interesting scenarios that source generators will enable is essentially the 'embedding' of C# within other languages (and vice versa). This is how Razor works today, and the Razor team maintains a significant language service investment in Visual Studio to enable it. A possible goal of this project would be to find a generic way to represent this: that would allow the Razor team to reduce their tooling investment, while allowing third parties the opportunity to enable the same sort of experiences (including 'Go to definition', 'Find all references' etc.) relatively cheaply. The current thinking is to have some form of 'side-channel' available to the generator. As the generator emits source text, it would indicate where in the original document this was generated from. This would allow the compiler API to track e.g. a generated `Symbol` as having an `OriginalDefinition` that represents a span of third party source text (such as a Razor tag in a `.cshtml` file). We discussed embedding this directly in the source text via `#pragma` but this would require language changes and limit the feature to a specific version of C#. Other considerations could be specially formed comments or `#if FALSE --` blocks. In general a 'side-channel' approach seems preferable to specially crafted grammar in the generated text. This is not necessarily a goal required for the success of Source Generators; Razor’s language service can be updated to work with source generators if it proves to be infeasible, but it certainly something we want to consider as part of the work. ### MSBuild Integration It is expected that generators will need some form of configuration system, and we intend to allow certain properties to flow through from MSBuild to facilitate this. > **Note**: This is still under design and open to change. ### Performance targets Ultimately, the performance of the feature is going to be somewhat dependent on the performance of the generators written by customers. Progressive opt-in, and build-time only by-default will allow the IDE to mitigate many of the potential performance problems posed by third party generators. However, there is still a risk that third-party generators will cause unacceptable performance problems for the IDE, and the design of the feature will need to keep this in mind. For 1st party generators, especially Razor and Blazor, we aim at a minimum to match the existing performance seen by users today. It is expected that even naïve generator-based implementations will perform significantly faster than the existing tooling, due to less communication overhead and duplicated work, but improving the speed of these experiences is not a primary goal of this project. ### Language Changes This design does not currently propose altering the language, it is purely a compiler feature. The previous design for source generators introduced the `replace` and `original` keywords. This proposal removes these, as the source generated is purely additional and so there is no need for them. We expect that most scenarios are possible with the existing use of `partial` definitions; as a V1 we expect to ship in this state. If concrete scenarios are later shown that can’t be achieved with the V1 approach we would consider allowing modification as a V2. ## Use cases We've identified several first and third party candidates that would benefit from source generators: - ASP.Net: Improve startup time - Blazor and Razor: Massively reduce tooling burden - Azure Functions: regex compilation during startup - Azure SDK - [gRPC](https://docs.microsoft.com/en-us/aspnet/core/grpc/?view=aspnetcore-3.1) - Resx file generation - [System.CommandLine](https://github.com/dotnet/command-line-api) - Serializers - [SWIG](http://www.swig.org/) ## Discussion / Open Issues / TODOs: **Interface vs Class for ISourceGenerator**: We discussed about this being an interface or class. Analyzers chose to have a abstract base class, but we weren't sure what we'd end up a need since ultimately we only had one method on this. Keeping it an interface also was more natural since we have other interfaces that implement this interface as well for optional light-up. **IDependsOnCompilationGenerator**: We did discuss if there should be an IDependsOnCompilationGenerator to formally state that you actually use a compilation. After all, if you don't use the compilation then we know your performance in the IDE is greatly simplified. However every scenario we've had for reading additional files has also needed the compilation, so we simply weren't sure what that was going to bring. **Breakpoints in generated files**: Do we map this back to the in-memory file? **Should generators be push or pull**: Source generators are pull-based, analyzers are push-based (registration based). Should we use a push-based model for generators as well? - If we go down the push-based model, walking the tree should make sure to continue to produce events for as many nodes as possible, even with errors, as generators will often work in the presence - The events that we use today for analyzers may require may more work to produce, since we expect analyzers to run during full compilation, while generators may not want to even construct the symbol table - The progressive-performance-opt-in model may work better in a push-based model, since you would only register for the things you care about **Should we share more with the analyzer type hierarchy?**: We would still need to differentiate analyzers from generators, since they would be generated at different times (generator diagnostics only on the first compilation, analyzer diagnostics only on the second compilation) **Can we predict how often some of our sample customers (Razor?) will have to run the generators?**: They can't predict that right now, and the incorporation of timers into their current generation makes it very difficult to predict the consequences of only event-based generation **Do we have a priority list of the most important customers?**: No, we should work out priority in order to prioritize features. **Security Review**: Do generators create any new security risks not already posed via analyzers and nuget? --- ### Features String Literals Data Section (`docs/features/string-literals-data-section.md`) # String literals in data section This opt-in **experimental** Roslyn feature allows changing how string literals in C# programs are emitted into PE files (`.dll`/`.exe`). By default, string literals are emitted to the UserString heap which is limited to [2^24 bytes](https://github.com/dotnet/roslyn/issues/9852). When the limit is reached, the following compiler error is reported by Roslyn: ``` error CS8103: Combined length of user strings used by the program exceeds allowed limit. Try to decrease use of string literals. ``` By turning on [the feature flag](#configuration), string literals ([where possible](#eligible-string-literals)) are instead emitted as UTF-8 data into a different section of the PE file which does not have the same limit. The emit format is similar to [explicit u8 string literals][u8-literals]. The feature is currently implemented only for C#, not VB. > [!WARNING] > This feature is currently experimental and can be changed or removed at any time. ## Configuration The feature flag can take a non-negative integer threshold. Only string literals whose length (number of characters, not bytes) is greater than the threshold are emitted using the utf8 encoding strategy. If the flag is set, but no value or an unrecognized value is specified, the threshold defaults to 100. Specifying 0 means all non-empty string literals are considered for the feature. The feature is turned off if - the feature flag is not specified, or - the string `off` is provided as the feature flag value, or - `null` is provided as the feature flag value (usually only possible using Roslyn APIs like `ParseOptions.WithFeatures`). The feature flag can be specified on the command line like `/features:experimental-data-section-string-literals` or `/features:experimental-data-section-string-literals=20`, or in a project file in a `` like `$(Features);experimental-data-section-string-literals` or `$(Features);experimental-data-section-string-literals=20`. > [!NOTE] > This configuration is useful for experimenting (to see the impact of the threshold on runtime performance and IL size) > and also allowing users to avoid a potential overhead of many type definitions generated if their program has many short string literals > which do not need this new emit strategy because they can fit into the size-limited UserString heap. ## Eligible string literals A string literal is emitted with the utf8 encoding if and only if all of these are true: - it can be encoded as UTF-8, i.e., the following call does not throw: ```cs new System.Text.Encoding.UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true).GetBytes(literal) ``` - its `Length` is greater than the threshold specified by [the feature flag](#configuration). ## Emit The standard string literal emit strategy emits `ldstr "string"`. That puts the string literal on the UserString heap of the PE file. The utf8 string literal encoding emit strategy emits `ldsfld` of a field in a generated class instead. For every unique string literal, a unique internal static class is generated which: - has name composed of `` followed by a hex-encoded XXH128 hash of the string (collisions [should not happen][xxh128] with XXH128, but if there are string literals which would result in the same XXH128 hash, a compile-time error is reported), - is nested in the `` type to avoid polluting the global namespace and to avoid having to enforce name uniqueness across modules, - has one internal static readonly `string` field which is initialized in a static constructor of the class, - is marked `beforefieldinit` so the static constructor can be called eagerly if deemed better by the runtime for some reason. There is also an internal static readonly `.data` field generated into `` containing the actual bytes, similar to [u8 string literals][u8-literals] and [constant array initializers][constant-array-init]. This field uses hex-encoded SHA-256 hash for its name and collisions are currently not reported by the compiler. These other scenarios might also reuse the data field, e.g., the following statements could all reuse the same data field: ```cs ReadOnlySpan a = new byte[6] { 72, 101, 108, 108, 111, 46 }; ReadOnlySpan b = stackalloc byte[6] { 72, 101, 108, 108, 111, 46 }; ReadOnlySpan c = "Hello."u8; string d = "Hello."; // assuming this string literal is eligible for the `ldsfld` emit strategy ``` The initialization calls `.BytesToString` helper which in turn calls `Encoding.UTF8.GetString`. This is an optimization so each of the generated static constructors is slightly smaller in IL size. These size savings can add up since one class is generated per one eligible string literal. A compile-time error is reported if the `Encoding.UTF8.GetString` API is not available and needs to be used (the user can then either ensure the API is available or turn off the feature flag). The following example demonstrates the code generated for string literal `"Hello."`. ```cs using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using System.Text; [CompilerGenerated] internal static class { internal static readonly __StaticArrayInitTypeSize=6 2BBCE396D68A9BBD2517F6B7064666C772694CD92010887BA379E10E7CDDA960 = /* IL: data(48 65 6C 6C 6F 2E) */; [StructLayout(LayoutKind.Explicit, Pack = 1, Size = 6)] internal struct __StaticArrayInitTypeSize=6 { } internal static class F6D59F18520336E6E57244B792249482 { internal static readonly string s; unsafe static F6D59F18520336E6E57244B792249482() { s = .BytesToString((byte*)Unsafe.AsPointer(ref .2BBCE396D68A9BBD2517F6B7064666C772694CD92010887BA379E10E7CDDA960), 6); } } private unsafe static string BytesToString(byte* bytes, int length) { return Encoding.UTF8.GetString(bytes, length); } } ``` ### ILVerify and PEVerify The actual generated code does not call `(byte*)Unsafe.AsPointer`. It simply emits `ldsflda` of the data field. The same approach is taken by other scenarios like [u8 string literals][u8-literals] and [constant array initializers][constant-array-init]. That works but does not pass IL/PE verification. ### Ref assemblies The synthesized types are not part of ref assemblies. That makes them smaller and incremental compilation is faster because it does not need to recompile dependent projects when only string literal contents change. This is automatically implemented, because during metadata-only compilation, method bodies are not emitted, and the synthesized types used by this feature are only emitted lazily as part of generating method body instructions. ## Diagnostics The error CS8103 (Combined length of user strings used by the program exceeds allowed limit) is updated to suggest the feature flag, albeit with a disclaimer during the experimental phase of the feature. ## Runtime Throughput of `ldstr` vs `ldsfld` is very similar (both result in one or two move instructions). In the `ldsfld` emit strategy, the `string` instances won't ever be collected by the GC once the generated class is initialized. `ldstr` has similar behavior (GC does not collect the string literals either until the assembly is unloaded), but there are some optimizations in the runtime around `ldstr`, e.g., they are loaded into a different frozen heap so machine codegen can be more efficient (no need to worry about pointer moves). Generating new types by the compiler means more type loads and hence runtime impact, e.g., startup performance and the overhead of keeping track of these types. On the other hand, the PE size might be smaller due to UTF-8 vs UTF-16 encoding, which can result in memory savings since the binary is also loaded to memory by the runtime. See [below](#runtime-overhead-benchmark) for a more detailed analysis. The generated types are returned from reflection like `Assembly.GetTypes()` which might impact the performance of Dependency Injection and similar systems. ### Runtime overhead benchmark | [cost per string literal](https://github.com/jkotas/stringliteralperf) | feature on | feature off | | --- | --- | --- | | bytes | 1037 | 550 | | microseconds | 20.3 | 3.1 | The benchmark results above [show](https://github.com/dotnet/roslyn/pull/76139#discussion_r1944144978) that the runtime overhead of this feature per 100 char string literal is ~500 bytes of working set memory (~2x of regular string literal) and ~17 microseconds of startup time (~7x of regular string literal). The startup time overhead does depend on the length of the string literal. It is cost of the type loads and JITing the static constructor. The working set has two components: private working set (r/w pages) and non-private working set (r/o pages backed by the binary). The private working set overhead (~600 bytes) does not depend on the length of the string literal. Again, it is the cost of the type loads and the static constructor code. Non-private working set is reduced by this feature since the binary is smaller. Once the string literal is about 600 characters, the private working set overhead and non-private working set improvement will break even. For string literals longer than 600 characters, this feature is total working set improvement.
Why 600 bytes? When the feature is off, ~550 bytes cost of 100 char string literal is composed from: - The string in the binary (~200 bytes). - The string allocated on the GC heap (~200 bytes). - Fixed overheads: metadata encoding, runtime hashtable of all allocated string literals, code that referenced the string in the benchmark (~150 bytes). When the feature is on, ~1050 bytes cost of 100 char string literal is composed from: - The string in the binary (~100 bytes). - The string allocated on the GC heap (~200 bytes). - Fixed overheads: metadata encoding, the extra types, code that referenced the string in the benchmark (~750 bytes). 750 - 150 = 600. Vast majority of it are the extra types. A bit of the extra fixed overheads with the feature on is probably in the non-private working set. It is difficult to measure it since there is no managed API to get private vs. non-private working set. It does not impact the estimate of the break-even point for the total working set.
## Implementation `CodeGenerator` obtains [configuration of the feature flag](#configuration) from `Compilation` passed to its constructor. `CodeGenerator.EmitConstantExpression` switches to the utf8 encoding strategy for [eligible string literals](#eligible-string-literals). That uses the following API responsible for synthesizing the `` class and returning reference to its `string` field: ```cs IFieldReference ITokenDeferral.GetFieldForDataString(ImmutableArray data, SyntaxNode syntaxNode, DiagnosticBag diagnostics) ``` Similar to `ITokenDeferral.GetFieldForData`, this calls into `PrivateImplementationDetails` which is responsible for synthesizing the classes and helpers in a thread-safe and deterministic manner. There are implementations of various `Cci` interfaces which represent the metadata, rooted in a `DataStringHolder` which is the `INestedTypeDefinition` for the `` class. These are implemented in `Microsoft.CodeAnalysis`. The alternative (used by inline arrays, for example) would be to implement the `Symbol`s in the language-specific projects (and then use the standard `Cci` adapters), but that seems to require similar amount of implemented abstract properties/methods of the `Symbol` as the implementations of `Cci` interfaces require. But implementing `Cci` directly allows us to reuse the same implementation for VB if needed in the future. ## Future work and alternatives ### Edit and Continue Hot reload currently does not support `.data` field replacement. That can be implemented in the future. ### AOT Ahead-of-time compilation tools would need to be updated to recognize this new pattern of string literal use in place of `ldstr`. ### Automatic threshold The threshold could be determined automatically with some objective, for example, use the utf8 encoding emit strategy for the lowest number of string literals necessary to avoid overflowing the UserString heap. The set of string literals is not known up front in the compiler, it is discovered lazily (and in parallel) by the emit layer. However, we could continue emitting `ldstr` instructions and fix them up in a separate phase after we have seen all the string literals (and hence can determine the automatic threshold). The `ldstr` and `ldsfld` instructions have the same size, so fixup would be a straightforward replace. This fixup phase already exists in the compiler in `MetadataWriter.WriteInstructions` (it's also how the string literals are emitted into the UserString heap). It is called from `SerializeMethodBodies` which precedes `PopulateSystemTables` call, hence synthesizing the utf8 string classes in the former should be possible and they would be emitted in the latter. Alternatively, we could collect string literals during binding, then before emit sort them by length and content (for determinism) to find the ones that are over the threshold and should be emitted with this new strategy. ### Statistics The compiler could emit an info diagnostic with useful statistics for customers to determine what threshold to set. For example, a histogram of string literal lengths encountered while emitting the program. Tools like ILSpy can already be used to obtain some related data, e.g., count, length, and contents of string literals in the UserString heap. ### Single blob We could merge all UTF-8 byte sequences for affected strings into a single blob - a virtual User-String heap. The sequences can be deduped as we dedupe entries in User-String heap. We can even attempt to do some vary basic compression, for example, in the name heap that we generate (contains type and member names, etc.), names with shared content can overlap. We would generate a single `__StaticArrayInitTypeSize=*` structure for the entire virtual heap blob and add a single `.data` field to `` that points to the blob. At runtime, we would do an offset to where the required data reside in the blob and decode the required length from UTF-8 to UTF-16. However, this would be unfriendly to IL trimming. ### Configuration/emit granularity Instead of one global feature flag, the emit strategy could be controlled using compiler-recognized attributes (applicable to assemblies or classes). Furthermore, we could emit more than one string per one class. That could be configurable as well. One interesting strategy would be grouping strings which occur in one user-defined type into one compiler-generated `` type. The idea is that strings from one class are likely used "together" so there is no performance impact from initializing them all at once and we save on metadata. ### GC To avoid rooting the `string` references forever, we could turn the fields into `WeakReference`s (note that this would be quite expensive for both direct overhead and indirectly for the GC due to longer GC pause times). Or we could avoid the caching altogether (each eligible `ldstr` would be replaced with a direct call to `Encoding.UTF8.GetString`). This could be configurable as well. ### Runtime support We could emit the strings to data fields similarly but we would not synthesize the `string` fields and static constructors. Instead, at the place of `ldstr`, we would directly call a runtime-provided helper, passing a pointer to the data field. That runtime helper would be an intrinsic recognized by the JIT (with a fallback calling `Encoding.UTF8.GetString`). The runtime would then be responsible for loading and caching the strings, perhaps similarly to how `ldstr` works (using a frozen GC heap). ### Custom lazy initialization All the data and string fields could be part of one class, e.g., ``, and the compiler could emit custom lazy initialization instead of relying on static constructors, something like: ```cs static class { private static string _str1; internal static string Str1 => _str1 ??= Load(str1DataSection); } ``` However, that would likely result in worse machine code due to more branches and function calls. ### String interning The compiler should report a diagnostic when the feature is enabled together with `[assembly: System.Runtime.CompilerServices.CompilationRelaxations(0)]`, i.e., string interning enabled, because that is incompatible with the feature. ### Avoiding hash collisions Instead of XXH128 for the type names and SHA-256 for the data field names, we could use index-based names. - The compiler could assign names lazily based on metadata tokens which are deterministic. If building on the current approach, that might require some refactoring, because internal data structures in the compiler might not be ready for lazy names like that. But it would be easier if combined with the first strategy suggested for [automatic threshold](#automatic-threshold) above, where we would not synthesize the types until very late in the emit phase (during fixup of the metadata tokens). - We could build on the second strategy suggested for [automatic threshold](#automatic-threshold) where we would collect string literals during binding (and perhaps also constant arrays and u8 strings if we want to extend this support to them as well), then before emit we would sort them by length and content and assign indices to them to be then used for the synthesized names. [u8-literals]: https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-11.0/utf8-string-literals [constant-array-init]: https://github.com/dotnet/roslyn/pull/24621 [xxh128]: https://github.com/Cyan4973/xxHash/blob/86f6400a2a14ea7123ada691c89faf1d2a6a126f/tests/collisions/README.md --- ### Features Task Types (`docs/features/task-types.md`) Async Task Types in C# ====================== Extend `async` to support _task types_ that match a specific pattern, in addition to the well known types `System.Threading.Tasks.Task` and `System.Threading.Tasks.Task`. ## Task Type A _task type_ is a `class` or `struct` with an associated _builder type_ identified with `System.Runtime.CompilerServices.AsyncMethodBuilderAttribute`. The _task type_ may be non-generic, for async methods that do not return a value, or generic, for methods that return a value. To support `await`, the _task type_ must have a corresponding, accessible `GetAwaiter()` method that returns an instance of an _awaiter type_ (see _C# 7.7.7.1 Awaitable expressions_). ```cs [AsyncMethodBuilder(typeof(MyTaskMethodBuilder<>))] class MyTask { public Awaiter GetAwaiter(); } class Awaiter : INotifyCompletion { public bool IsCompleted { get; } public T GetResult(); public void OnCompleted(Action completion); } ``` ## Builder Type The _builder type_ is a `class` or `struct` that corresponds to the specific _task type_. The _builder type_ can have at most 1 type parameter and must not be nested in a generic type. The _builder type_ has the following `public` methods. For non-generic _builder types_, `SetResult()` has no parameters. ```cs class MyTaskMethodBuilder { public static MyTaskMethodBuilder Create(); public void Start(ref TStateMachine stateMachine) where TStateMachine : IAsyncStateMachine; public void SetStateMachine(IAsyncStateMachine stateMachine); public void SetException(Exception exception); public void SetResult(T result); public void AwaitOnCompleted( ref TAwaiter awaiter, ref TStateMachine stateMachine) where TAwaiter : INotifyCompletion where TStateMachine : IAsyncStateMachine; public void AwaitUnsafeOnCompleted( ref TAwaiter awaiter, ref TStateMachine stateMachine) where TAwaiter : ICriticalNotifyCompletion where TStateMachine : IAsyncStateMachine; public MyTask Task { get; } } ``` ## Execution The types above are used by the compiler to generate the code for the state machine of an `async` method. (The generated code is equivalent to the code generated for async methods that return `Task`, `Task`, or `void`. The difference is, for those well known types, the _builder types_ are also known to the compiler.) `Builder.Create()` is invoked to create an instance of the _builder type_. If the state machine is implemented as a `struct`, then `builder.SetStateMachine(stateMachine)` is called with a boxed instance of the state machine that the builder can cache if necessary. `builder.Start(ref stateMachine)` is invoked to associate the builder with compiler-generated state machine instance. The builder must call `stateMachine.MoveNext()` either in `Start()` or after `Start()` has returned to advance the state machine. After `Start()` returns, the `async` method calls `builder.Task` for the task to return from the async method. Each call to `stateMachine.MoveNext()` will advance the state machine. If the state machine completes successfully, `builder.SetResult()` is called, with the method return value if any. If an exception is thrown in the state machine, `builder.SetException(exception)` is called. If the state machine reaches an `await expr` expression, `expr.GetAwaiter()` is invoked. If the awaiter implements `ICriticalNotifyCompletion` and `IsCompleted` is false, the state machine invokes `builder.AwaitUnsafeOnCompleted(ref awaiter, ref stateMachine)`. `AwaitUnsafeOnCompleted()` should call `awaiter.OnCompleted(action)` with an action that calls `stateMachine.MoveNext()` when the awaiter completes. Similarly for `INotifyCompletion` and `builder.AwaitOnCompleted()`. ## Overload Resolution Overload resolution is extended to recognize _task types_ in addition to `Task` and `Task`. An `async` lambda with no return value is an exact match for an overload candidate parameter of non-generic _task type_, and an `async` lambda with return type `T` is an exact match for an overload candidate parameter of generic _task type_. Otherwise if an `async` lambda is not an exact match for either of two candidate parameters of _task types_, or an exact match for both, and there is an implicit conversion from one candidate type to the other, the from candidate wins. Otherwise recursively evaluate the types `A` and `B` within `Task1` and `Task2` for better match. Otherwise if an `async` lambda is not an exact match for either of two candidate parameters of _task types_, but one candidate is a more specialized type than the other, the more specialized candidate wins. --- ### Features Throwexpr (`docs/features/throwexpr.md`) ## Throw expression We extend the set of expression forms to include ```antlr throw_expression : 'throw' null_coalescing_expression ; null_coalescing_expression : throw_expression ; ``` The type rules are as follows: - A *throw_expression* has no type. - A *throw_expression* is convertible to every type by an implicit conversion. The flow-analysis rules are as follows: - For every variable *v*, *v* is definitely assigned before the *null_coalescing_expression* of a *throw_expression* iff it is definitely assigned before the *throw_expression*. - For every variable *v*, *v* is definitely assigned after *throw_expression*. A *throw expression* is permitted in only the following syntactic contexts: - As the second or third operand of a ternary conditional operator `?:` - As the second operand of a null coalescing operator `??` - As the body of an expression-bodied lambda or method. > Note: the rest of the semantics of the throw expression are identical to the semantics of the *throw_statement* in the current language specification. --- --- METRICS --- - Files Extracted: 98 - Estimated Token Budget: ~187237 tokens - Canonical Reference: https://codewiki.google/github.com/dotnet/roslyn