Theorem: Two of a, b, c are multiples of 3.

Proof: Modulo 3, if a=0, a²=0; otherwise a²=1.

If all of a, b, c are multiples of 3, so are abc, a², b² and c², so a²+b²+c²+2 is not, so {a, b, c} is not a solution.

If one of a, b, c is a multiple of 3, so is abc. a², b² and c² are 0, 1 and 1, so a²+b²+c²+2=1, so {a, b, c} is not a solution.

If none of a, b, c are multiples of 3, a², b² and c² are 1, 1 and 1, so a²+b²+c²+2=2. Thus abc=2, so a, b and c are 1, 1 and 2, or 2, 2 and 2. If all possible cases are checked, working modulo 9, it will be found that (a²+b²+c²+2, abc) are (2, 5), (5, or (8, 2), so {a, b, c} is not a solution.

Thus two of a, b, c are multiples of 3 as required.

The following theorem can be applied to the more general problem of finding triples {a, b, c} where

a²+b²+c² + f = abc

where f is given. Setting f to 2 gives the present problem.

Theorem: If a, b, c_1 are positive integers and {a, b, c_1} is a solution, so also is {a, b, ab-c_1}, and these are the only solutions {a, b, c}.

Proof: Given a, b:

a²+b²+c²+f – abc = 0
c² – (ab)c + (a²+b²+f) = 0
c = { ab +- sqrt[ a²b² - 4(a²+b²+f)] } / 2
= (ab +- sqrt(D))/2, D = a²b² – 4(a²+b²+f)

If sqrt(D) is an integer of the same parity as ab, then this provides solutions

c_1 = (ab – sqrt(D))/2
c_2 = (ab + sqrt(D))/2

thus

c_2 = ab – c_1

as required. Otherwise, it provides no solutions.

This may be used to find arbitrarily many solutions involving a number a, given a starting solution {a, b, c}. For example:

{3, 3, 4}, {3, 4, 9}, {3, 9, 23}, {3, 23, 60}, {3, 60, 157}, {3, 157, 411}, {3, 411, 1076},…

Repetitively setting the new a and b to the old b and c, and taking the larger solution as the new c yields a series of solutions including some with a, b, c all greater than N given any arbitrary N:
{3, 3, 4}, {3, 4, 9}, {4, 9, 33}, {9, 33, 293}, {33, 293, 9660}, {233, 9660, 2830347}, {9660, 2830347, 27341151727}, …

Some solutions where a, b, c are all less than 10:
{a, b, c} = {3, 3, 4}, {3, 3, 5}, {3, 4, 9}
A solution where a, b, c are all greater than 10:
{12, 31, 369}, {23, 60, 1377}, {31, 81, 2508}, {60, 157, 9417}, {81, 212, 17169}; {33, 123, 4055}; {12, 57, 679}, {57, 273, 15556}
Some solutions which are a mixture of less than 10 and greater than 10:
{a, b, c} = {3, 5, 12}, {3, 9, 23}, {4, 9, 33}

Let f(a, b, c) = LHS – RHS = a²+b²+c²+2 – abc.

For given a and b, f is a cubic in c. If neither a nor b is small, then a few numerical tests gave me the impression that the only c that is positive and not small for which f is zero is between ab-3 and ab-2:

If c=ab-2,
f = a²+b²+(ab-2)²+2-ab(ab-2)
= a²+b²+a²b²-4ab+4+2-a²b²+2ab
= a²+b²-2ab+6
= (a-b)²+6
which is positive.

If c=ab-3,
f = a²+b²+(ab-3)²+2-ab(ab-3)
= a²+b²+a²b²-6ab+9+2-a²b²+3ab
= a²+b²-3ab+11
(negative?)

In fact this is not so; the root (value of c for which f=0) might be less than ab-3. Nonetheless, I decided to set c=ab-3 and try various values in turn for a. Hence the triples I found above. Trying c=ab-4 and ab-5 found a few more, too. There are probably some more solutions out there.