Since the remainder on dividing f(x) by x2 - 2 is bx we know that
f(x) = x5 - 3x4 - 6x3 + 3ax2 - 24 = (x2 - 2) g(x) + bx
for some polynomial g(x).
Since (x2 - 2) is zero when x = √2 and when x = -√2, substituting x = √2 and x = -√2 in the equation above give two equations in the unknowns a and b. Solve for a and b.