Answer: 20

Small Hint:

Let $S_k=ax^k+by^k$ and derive a recurrence using $x+y$ and $xy$

Big Hint:

Use $S_3$ and $S_4$ to solve for the two recurrence coefficients before computing $S_5$

Solution:

Let $S_k=ax^k+by^k,$ $p=x+y,$ and $q=xy.$ Since $x$ and $y$ each satisfy $t^2=pt-q,$ $$S_{k+2}=pS_{k+1}-qS_k.$$ Using $S_1=3,$ $S_2=7,$ $S_3=16,$ and $S_4=42$ gives $$\begin{aligned}7p-3q&=16,\\16p-7q&=42.\end{aligned}$$ Solving yields $p=-14$ and $q=-38.$ Therefore $$\begin{aligned}S_5&=pS_4-qS_3\\&=-14(42)+38(16)\\&=20.\end{aligned}$$