If you want to set a custom code analysis rule set on an ASP.NET Core project, there is currently no UI tooling in Visual Studio 2017 to support this scenario. However, you can add it manually to the csproj file.

To set a custom code analysis rule set on an ASP.NET Core project, right click on the project in solution explorer, select Edit <YourProjectName>.csproj, and then inside <PropertyGroup> add a new tag called CodeAnalysisRuleSet containing a path to the custom ruleset. The path can be a full path or a path relative to the project file. Here is an example of how to do that:

<PropertyGroup>
  ...
  <CodeAnalysisRuleSet>..\custom.ruleset</CodeAnalysisRuleSet>
</PropertyGroup>

For other project types, you can set the custom code analysis rule set through the Code Analysis Settings tab in the solution properties window or you can set it directly on the project through the Code Analysis tab in the project properties window.