slow query performance issue with partition and max

I think you need a NONCLUSTERED INDEX for your query that you describes above.

If you don't have any idea about INDEX, I mean you can not identify a witch field of your table NONCLUSTERED INDEXneed then simply, you just create an execution plan from SQL Server 2008 Management Studio and SQL Server intelligence gives you missing index details and shows a green color text that is details of the missing index.

you can move your mouse pointer on missing Index text and SQL Server 2008 Management Studio intelligence will show the T-SQL code that is required to create the missing index or you can press your mouse to right-click on missing index text then select the missing index details option from the list to see the details of the missing index.

For more information, you can visit this article Create Missing Index From the Actual Execution Plan

I hope this solution helps you.

Filtering by calculated fields used to affect performance negatively. You can do your other filters first, and as a last step do the calculated filter, to have less rows to match. Maybe it will fill TEMPDB because it will store the intermediate recordset there, but in this case you either increase the size of it, or use another method.
Here is your second query written like this (maybe you need to adjust it, I just wrote it in Notepad++:

SELECT MAX(id) AS id
FROM (
    SELECT F.id, F.sequence_invoice % @number_service_installed as [idServizio]
    FROM jfel_tagxml_invoicedigi F
         INNER JOIN jfel_invoice_state S ON F.id = S.idinvoice
    WHERE S.idstato = @idstato
        AND S.id = F.idstatocorrente
        -- AND F.sequence_invoice % @number_service_installed = @idServizio
)
WHERE idServizio = @idServizio
;

Instead of the subquery, you can try a temp table or CTE as well, maybe one is the clear winner above the others, worth a try for all if you want maximum performance.

The data calculation is Non-Sargable, you could try using a variable with OPTION RECOMPILE:

DECLARE @d Date
SET @d = dateadd(DAY,-5,getdate())

SELECT top 1 F.id as id 
FROM jfel_tagxml_invoicedigi AS F 
INNER JOIN jfel_invoice_state AS S 
ON F.idstatocorrente = S.id
WHERE S.idstato= 1 AND S.id = F.idstatocorrente 
and S.datastato > @d
AND F.progressivo_fattura % 1 = 0
ORDER BY S.datastato
OPTION (RECOMPILE)