How to use Function keys as standard key in one app only?

Try combining FunctionFlip with BetterTouchTool:

  • Set the keys you want "flipped" with FunctionFlip
  • Set up application specific keys with BetterTouchTool

Use Karabiner

This application can provide the exact functionality the OP asked for. user495470's answer about KeyRemap4MacBook was helpful in pointing me in the right direction. However, it's outdated and no longer works as written because KeyRemap4MacBook has been updated. It is now called Karabiner. It looks to be well-maintained and is free and open source software (not the case with BetterTouchTool mentioned in another answer which requires payment after 45 days).

Setting it up

First download and install the Karabiner application. Once installed, in order to remap the function keys as standard function keys, you will need to add a "Complex-Modification" in preferences. There are several of these complex modifications prebuilt and downloadable from the Karabiner website, but you can also write your own in json.

Use the following code and place it in a file (named something like "functionkeys.json") in the ~/.config/karabiner/assets/complex_modifications folder. Once it is there, go to Karabiner preferences and click Complex-Modifications, then Add rule. You should see the text Function Keys in Terminal Apps and there click Enable All.

How to change which foreground application causes the effect

This code specifically provides function key standard functionality while using any of several terminal apps (and nothing when using any other app). This code also allows media key usage in terminal apps by using fn + function key.

While the OP asked about a generic specific application, VirtualBox was mentioned in particular. The below code could be changed to create the function key functionality in whatever application(s) was required by replacing the contents of each instance of "Bundle Identifiers: [ ... ]" with any application(s). (VirtualBox may be "^org\\.virtualbox\\.VirtualBoxVM" although I haven't tried it.) Use the included Karabiner EventViewer to find the application identifier for any application desired, and then plug in the values.

{
  "title": "Function Keys in Terminal Apps",
  "rules": [
    {
      "description": "Use regular function keys in terminal",
      "manipulators": [
        {
          "type": "basic",
          "from": {
            "key_code": "f1"
          },
          "to": [
            {
              "key_code": "f1",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f2"
          },
          "to": [
            {
              "key_code": "f2",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f3"
          },
          "to": [
            {
              "key_code": "f3",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f4"
          },
          "to": [
            {
              "key_code": "f4",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f5"
          },
          "to": [
            {
              "key_code": "f5",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f6"
          },
          "to": [
            {
              "key_code": "f6",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f7"
          },
          "to": [
            {
              "key_code": "f7",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f8"
          },
          "to": [
            {
              "key_code": "f8",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f9"
          },
          "to": [
            {
              "key_code": "f9",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f10"
          },
          "to": [
            {
              "key_code": "f10",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f11"
          },
          "to": [
            {
              "key_code": "f11",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f12"
          },
          "to": [
            {
              "key_code": "f12",
              "modifiers": [
                "fn"
              ]
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f1",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f1"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f2",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f2"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f3",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f3"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f4",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f4"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f5",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f5"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f6",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f6"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f7",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f7"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f8",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f8"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f9",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f9"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f10",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f10"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f11",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f11"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "f12",
            "modifiers": {
              "mandatory": [
                "fn"
              ]
            }
          },
          "to": [
            {
              "key_code": "f12"
            }
          ],
          "conditions": [
            {
              "type": "frontmost_application_if",
              "bundle_identifiers": [
                "^com\\.apple\\.Terminal$",
                "^com\\.googlecode\\.iterm2$",
                "^co\\.zeit\\.hyperterm$",
                "^co\\.zeit\\.hyper$",
                "^io\\.alacritty$",
                "^net\\.kovidgoyal\\.kitty$"
              ]
            }
          ]
        }
      ]
    }
  ]
}