Trying to detect if any checked values contains a specific string

There were multiple errors in this code. I also changed a bit to make it simpler.

  1. You have to use .indexOf() not .indexOF(). Small typos like those can throw your whole code off.
  2. For document.querySelectorAll, you only need input:checked. #question5 is completely unecessary (since none of the values have an id of question5).
  3. .querySelectorAll returns a NodeList which is more like an object than an array. Therefore, .indexOf() does not work on it. You must use a for loop or a .forEach() like I did in the below snippet.
  4. I set the default value of answer to "no", and if it finds photo or name, it will set the value to "yes" (which is easier better than having an if else statement for this).

The snippet below works for you.

function admissible() {
  var b = 4;
  var answer = 'no';
  var elementsList = document.querySelectorAll("input:checked");

  //get the selected checkbox values for the missing elements
  if (elementsList.length >= 0) { //Code works only if some checkbox is checked

    elementsList.forEach(e => {
      if (e.value == "photo" || e.value == "name") {
        answer = "yes";
      }
    });

  } else {
    //display error: you must select a value
  }
  console.log(answer);
}
<div class="questionholder" id="question5">
  <div>
    <h5>Select all elements</h5>
    <input class="input5" type="checkbox" id="elementName" name="element" value="name"><label for="elementName"><p class="radioChoice">Name</p></label>
    <input class="input5" type="checkbox" id="elementPhoto" name="element" value="photo"><label for="elementPhoto"><p class="radioChoice">Photo</p></label>
    <input class="input5" type="checkbox" id="elementSignature" name="element" value="signature"><label for="elementSignature"><p class="radioChoice">Signature</p></label>
  </div>
  <div class="holdButtons">
    <a class="text2button" onclick="admissible()">Next</a>
  </div>
</div>


There are a lot of errors in your posted code. The code in the example below fixes them. The comments explain how it is solved.

Small summary:

Query selector

"#question5" + " input:checked"

The queryselector above looks wrong. It will work, but you can combine it into one string.

"#question5 > div:first-child > input:checked"

This query selector will make a safer selection. It selects through document.querySelectorAll all elements of type input that are checked within the first div child element of the element with id question5.

Iterating over a node list

Since document.querySelectorAll returns a nodelist (array like) which is not actually an array. In your code you used length check to <= 0. This will always execute the if. Drop the =. Now the if only fires when the nodelist has at least one entry.

Then we cast the nodeList to an array using Array.from and loop over it with Array.prototype.some. Which iterates over every element and if one condition matches it will return true.

We use String.prototype.search instead of String.prototype.indexOf since we can use a regexp to look for the value name|photo in one go. The | (pipe) tells the code to either look for name or photo. If the result is not -1 we found the value.

Working example

function admissible(method) {
  var b = 4;
  var answer = '';
  var elementsList = document.querySelectorAll("#question5 > div:first-child > input:checked"); //fixed the query selector

  //get the selected checkbox values for the missing elements
  if (elementsList.length > 0) { //Code works only if some checkbox is checked
    //cast the node list to array and use array.some to iterate over entries. If one value is found answerFilter will be true.  

    if (method == "some") {
      var answerFilter = Array.from(elementsList).some(function(element) {
        return element.value.search(/name|photo/i) != -1; //check with regexp if value contains photo or name
      });
    }
    else
    {
    
      //map returns an array and lets us return a value we want.
      var answerFilter = Array.from(elementsList).map(function(element) {
        return element.value;
      });
      //now use indexOf like in the original code
      if (answerFilter.indexOf("name") > -1 || answerFilter.indexOf("photo") > -1)
      {
        answerFilter = true;
      }
      else
      {
        answerFilter = false;
      }
    }
    answerFilter ? answer = "yes" : answer = "no"; //use shorthand if/else

  } else {
    //display error: you must select a value
  }
  console.log(answer);
}
* {
  font-family: tahoma;
  font-size: 10pt;
}

div.questionholder>div:first-child {
  display: grid;
  grid-template-columns: 20px auto;
  width: 100%;
}

h5 {
  grid-area: 1 2;
  white-space: nowrap;
}

label {
  grid-column: 2;
}

label>p {
  display: inline;
}

input {
  grid-column: 1;
}

div.holdButtons {
  margin: 10px 5px;
}

div.holdButtons>a {
  padding: 5px;
  background-color: #f4f4f4;
  cursor: pointer;
  border-radius: 4px;
}

div.holdButtons>a:hover {
  background-color: #e1e1e1;
}
<div class="questionholder" id="question5">
  <div>
    <h5>Select all elements</h5>
    <input class="input5" type="checkbox" id="elementName" name="element" value="name"><label for="elementName"><p class="radioChoice">Name</p></label>
    <input class="input5" type="checkbox" id="elementPhoto" name="element" value="photo"><label for="elementPhoto"><p class="radioChoice">Photo</p></label>
    <input class="input5" type="checkbox" id="elementSignature" name="element" value="signature"><label for="elementSignature"><p class="radioChoice">Signature</p></label>
  </div>
  <div class="holdButtons">
    <a class="text2button" onclick="admissible('some')">Next <i>(Array.some)</i></a>
    <a class="text2button" onclick="admissible('map')">Next <i>(Array.map)</i></a>
  </div>
</div>