Monitoring certain system calls done by a process in Windows

There are several options on Windows.

Windows Performance Toolkit can be used to enable tracing of various system events, including file I/O, and includes tools for processing and viewing these events. You can use xperf to begin trace variously classes of events and save to an ETL file that you can then process or view using the same tools later.

Process Monitor from Sysinternals is another, very easy to use, option, and enables you to quickly see all file and registry accesses any process on the system is doing. You can also run Process Monitor in an automated fashion.

If you'd like to do this completely programmatically, you can use the ETW functions (StartTrace, EnableTrace, etc.) to snap file I/O events and save to an ETL file. Sample code here.

API Monitor by Rohitab Batra is very good for system calls.

On Windows, you can use Process Monitor to monitor process activity (I/O and registry). I guess this fits your need if you don't really want to know the system calls.

And you can use winapioverride32 to monitor API calls.