excessive memory usage in IntelliSense with MSVC non-conforming preprocessor (OpenUSD/Pixar)

[jkriegshauser]

Environment

• OS and Version: Windows 11 10.0.26100 x64
• VS Code Version: 1.96.4 commit cd4ee3b1c348a
• C/C++ Extension Version: 1.22.11 (ms-vscode.cpptools); 1.3.0 (ms-vscode.cpptools-extension-pack)
• If using SSH remote, specify OS of remote machine:

Bug Summary and Steps to Reproduce

Bug Summary:
On a very large C++ codebase, cpptools-srv.exe will consume memory, starving out the rest of the system:

I have C_Cpp: Max Memory set, but it does not appear to be used:

Steps to reproduce:

1. Open a very large C++ directory, potentially multiple in multiple instances of VSCode
2. Wait
3. Observe cpptools-srv.exe processes consuming significant amounts of memory

Expected behavior:

1. Max memory limit is enforced

Configuration and Logs

{
    "configurations": [
        {
            "name": "Win32",
            "intelliSenseMode": "msvc-x64",
            "cStandard": "c11",
            "cppStandard": "c++17",
            "compileCommands": "${workspaceFolder}/_build/windows-x86_64/release/compile_commands-full.json",
            "includePath": [
                "${default}",
                "${workspaceFolder}/include"
            ]
        },
    ],
    "version": 4
}



-------- Diagnostics - 1/27/2025, 5:55:42 PM
Version: 1.22.11
Current Configuration:
{
    "name": "Win32",
    "includePath": [
        "c:/src/<redacted>/**"
    ],
    "defines": [
        "_DEBUG",
        "UNICODE",
        "_UNICODE"
    ],
    "windowsSdkVersion": "10.0.22621.0",
    "compilerPath": "cl.exe",
    "cStandard": "c17",
    "cppStandard": "c++17",
    "intelliSenseMode": "windows-msvc-x64",
    "compilerPathInCppPropertiesJson": "cl.exe",
    "intelliSenseModeIsExplicit": false,
    "cStandardIsExplicit": false,
    "cppStandardIsExplicit": false,
    "mergeConfigurations": false,
    "compilerPathIsExplicit": false,
    "browse": {
        "path": [
            "c:/src/kit/**",
            "${workspaceFolder}"
        ],
        "limitSymbolsToIncludedHeaders": true
    }
}
Modified Settings:
{
    "C_Cpp.maxMemory": 4000
}
Additional Tracked Settings:
{
    "editorTabSize": 4,
    "editorInsertSpaces": true,
    "editorAutoClosingBrackets": "languageDefined",
    "filesEncoding": "utf8",
    "filesAssociations": {},
    "filesExclude": {
        "**/.git": true,
        "**/.svn": true,
        "**/.hg": true,
        "**/CVS": true,
        "**/.DS_Store": true,
        "**/Thumbs.db": true
    },
    "filesAutoSaveAfterDelay": false,
    "editorInlayHintsEnabled": true,
    "editorParameterHintsEnabled": true,
    "searchExclude": {
        "**/node_modules": true,
        "**/bower_components": true,
        "**/*.code-search": true
    },
    "workbenchSettingsEditor": "ui"
}
cpptools version (native): 1.22.11.0
Current database path: C:\USERS\REDACTED\APPDATA\LOCAL\MICROSOFT\VSCODE-CPPTOOLS\D04989D4087A51DA9667FC71DEFAB4CC\.BROWSE.VC.DB
No active translation units.

------- Workspace parsing diagnostics -------
Number of files discovered (not excluded): 155248
Number of files parsed: 30183

Other Extensions

No response

Additional context

No response

[sean-mcmanus]

The settings are behaving as expected -- they don't limit the memory usage of a single cpptools-srv process.

When you run the C/C++: Log Diagnostics command after opening a source file -- what memory usage is reported per TU? Over 15 GB? If so, that seems like it might be a bug. Your logging indicates no active files have been started.

The cpptools-srv process only has memory for the current TU (source file and headers), so having a "large code base" should not make a difference.

[sean-mcmanus]

@jkriegshauser Also, your repro steps don't indicate that you have opened any files? No cpptools-srv process should launch if you don't open any files.

[jkriegshauser]

Oh, I didn't realize it was per TU. I opened one VSCode instance with two source windows open. I have several source files open in the left source window and only one in the right source window. Active, the left source file is a .cpp file with 10,978 lines and in the right, a .cpp file with 1,028 lines.

I have 3 cpptools-srv.exe processes running each consuming nearly 15 GB of RAM each:

Though every time I do C/C++: Log Diagnostics it says No active translation units. I even bound it to a key and execute the key with the larger source file active and get the same message:

-------- Diagnostics - 1/28/2025, 8:41:03 AM
Version: 1.22.11
Current Configuration:
{
    "name": "Win32",
    "includePath": [
        "c:/src/kit/**"
    ],
    "defines": [
        "_DEBUG",
        "UNICODE",
        "_UNICODE"
    ],
    "windowsSdkVersion": "10.0.22621.0",
    "compilerPath": "cl.exe",
    "cStandard": "c17",
    "cppStandard": "c++17",
    "intelliSenseMode": "windows-msvc-x64",
    "compilerPathInCppPropertiesJson": "cl.exe",
    "intelliSenseModeIsExplicit": false,
    "cStandardIsExplicit": false,
    "cppStandardIsExplicit": false,
    "mergeConfigurations": false,
    "compilerPathIsExplicit": false,
    "browse": {
        "path": [
            "c:/src/redacted/**",
            "${workspaceFolder}"
        ],
        "limitSymbolsToIncludedHeaders": true
    }
}
Modified Settings:
{
    "C_Cpp.maxMemory": 4000
}
Additional Tracked Settings:
{
    "editorTabSize": 4,
    "editorInsertSpaces": true,
    "editorAutoClosingBrackets": "languageDefined",
    "filesEncoding": "utf8",
    "filesAssociations": {},
    "filesExclude": {
        "**/.git": true,
        "**/.svn": true,
        "**/.hg": true,
        "**/CVS": true,
        "**/.DS_Store": true,
        "**/Thumbs.db": true
    },
    "filesAutoSaveAfterDelay": false,
    "editorInlayHintsEnabled": true,
    "editorParameterHintsEnabled": true,
    "searchExclude": {
        "**/node_modules": true,
        "**/bower_components": true,
        "**/*.code-search": true
    },
    "workbenchSettingsEditor": "ui"
}
cpptools version (native): 1.22.11.0
Current database path: C:\USERS\REDACTED\APPDATA\LOCAL\MICROSOFT\VSCODE-CPPTOOLS\D04989D4087A51DA9667FC71DEFAB4CC\.BROWSE.VC.DB
No active translation units.

------- Workspace parsing diagnostics -------
Number of files discovered (not excluded): 157125
Number of files parsed: 2

[sean-mcmanus]

@jkriegshauser Those might be dangling processes. Can you kill them? Does your C/C++ logging with C_Cpp.loggingLevel set to "Debug" indicate the IntelliSense process is shutting down or crashing?

[jkriegshauser]

I had 13 cpptools-srv processes running, the most using ~26 GB RAM. I killed them all with taskkill and 5 restarted. I see some debug logging happening now after opening it. When I killed the processes I saw this appear in the logs:

IntelliSense process crash detected: handle_initialize
IntelliSense process crash detected: handle_initialize
IntelliSense process crash detected: handle_initialize
loggingLevel has changed to: Debug
IntelliSense process crash detected: handle_update_intellisense
IntelliSense process crash detected: handle_update_intellisense
IntelliSense process crash detected: handle_update_intellisense
IntelliSense process crash detected: handle_initialize
Update IntelliSense time (sec): 3776.863
IntelliSense process crash detected: handle_contains
IntelliSense process crash detected: handle_contains
IntelliSense process crash detected: handle_contains
IntelliSense process crash detected: handle_contains
IntelliSense process crash detected: handle_contains

[sean-mcmanus]

@jkriegshauser Are the crash logging from your killing the processes or from the processes actually crashing? If it's an actual crash you could potentially get a crash call stack via attaching a debugger before it crashes (https://github.com/microsoft/vscode-cpptools/wiki/Attaching-debugger-to-cpptools-or-cpptools%E2%80%90srv). It sounds like our IntelliSense parser may be hitting some bug while processing your TU. If it's using CPU while memory is increasing, attaching a debugger and hitting a breakpoint might show what processing is causing the issue if it's stuck in some loop that is allocating memory.

[jkriegshauser]

I think those were from me killing the process. I have some huge memory processes:

I used procdump to get mini-dumps of them and the biggest ones have threads doing this:

	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!expand_top_level_pcc_macro()	Unknown
 	cpptools-srv.exe!macro_invocation(struct a_symbol *,int *)	Unknown
 	cpptools-srv.exe!get_token(void)	Unknown
 	cpptools-srv.exe!scan_nonmember_declaration(struct a_decl_parse_state *,struct a_source_range *)	Unknown
 	cpptools-srv.exe!declaration(int,int,int,int,struct a_param_id *,struct a_source_range *,struct a_decl_parse_state *)	Unknown
 	cpptools-srv.exe!process_translation_unit(char const *,int,struct an_exported_template_file *)	Unknown
 	cpptools-srv.exe!cfe_main(int,char * * const)	Unknown
 	cpptools-srv.exe!cfe_main_exception_handler(int,char * * const)	Unknown
 	cpptools-srv.exe!invoke_edge_compiler()	Unknown
 	cpptools-srv.exe!edge_compiler_main(int,char const * * const)	Unknown
 	cpptools-srv.exe!preparse(int,char const * *,class a_scout_store *,class edge::translation_unit *)	Unknown
 	cpptools-srv.exe!a_compiler_thread::compiler_thread_routine(class a_compiler_thread *)	Unknown
 	cpptools-srv.exe!msvc::thread_helper_t::thread_entry(void *)	Unknown
 	cpptools-srv.exe!thread_start<unsigned int (__cdecl*)(void *),1>()	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Unfortunately the symbols are minimal so I don't see any local variables or anything. But it definitely looks like there's some recursion happening. Would be nice to know what macro is kicking it off

[sean-mcmanus]

@jkriegshauser I've sent an email to the team involved. Are you able to reduce the TU to locate the macro that is triggering the excessive/infinite recursion?

[jkriegshauser]

@sean-mcmanus not easily. I'm not even sure which file it's parsing. I looked at strings in the minidump that I gathered and didn't see any .obj or .cpp files, but several headers. Having PDB files with local variables might help. This tool isn't open-source I take it?