How to include unfruitful attempts to use software that were not explored much?

I would have to disagree with the previous answers. I would consider documenting what did not work. I am not saying that I would definitely do it, but I am saying that ruling out the idea, out of hand, is not good advice. I have 27 years of experience in software development and project management, and I would very seriously think about at least a short section where I would lay out what I tried. Sounds like there are some serious holes in your knowledge of those libraries and your attempts to use them, which might make it difficult to document what you did, but I encourage you to give it a shot and run it by your advisor.

In the end, you could take the section out if it ends up detracting from the finished thesis. But I find that kind of honest appraisal of failed attempts to be often more informative than the success stories. I find it to be counterproductive to the community and the researcher when academia is reluctant or completely unable to admit when something did not work or the researcher had a knowledge gap. I find the disarming honesty of "I tried and it did not work" elevates my opinion of a researcher. But there is so little of it, that I find myself going to non-academic sources much more often than not, when I want to find answers that are real and practical.

Be careful in how you write it, and how you make the final decision to include it or not. If you can say, I tried this library. This is what it gave me. This is why it was not suitable. I tried to find relevant documentation to resolve the issues, but no such documentation could be found. And in the end, the ability of this project to integrate a proper library is demonstrated in the use of X library, which worked. If you can frame it like that, informative, honest, yet demonstrating your competence by highlighting, once again, that you did successfully integrate a library that worked, you could add some real value to your thesis.

The main reason for not including such an admission, in my opinion, would only be if your ability to integrate the third library was more luck than real skill, if it did not demonstrate a concerted effort aided by much better features and documentation. If that is the case, then forget what I have said and go with the other answers' advice.

Report what did work. Forget the rest. Anyone looking to reproduce or expand on your results needs to know the software you did use.

Edit: I note with some chagrin that I answered other than what OP asked, namely how to include unfruitful attempts to use software X and Y.

At the point where you describe your use of package Z, include a footnote that you first attempted to use X and Y, but received implausible results. One sentence is enough. That's enough to warn anyone who cares that you had difficulty, but does not belabor problems that you cannot explain. It might even prompt someone to ask you for more information some day.