The fractional part vanishes when the argument is an integer; in this case, for
[tex]\left\{\dfrac1{1-xy}\right\} = 0 \iff \dfrac1{1-xy} = n \iff xy = \dfrac{n-1}n[/tex]
which are hyperbolas in the [tex](x,y)[/tex]-plane.
Observe that between neighboring hyperbolas, we have
[tex]\dfrac{n-1}n < xy < \dfrac n{n+1} \\\\ ~~~~ \implies \dfrac1{n+1} < 1-xy < \dfrac1n \\\\ ~~~~ \implies n < \dfrac1{1-xy} < n+1 \\\\ ~~~~ \implies \left\{\dfrac1{1-xy}\right\} = \dfrac1{1-xy} - \left\lfloor\dfrac1{1-xy}\right\rfloor = \dfrac1{1-xy} - n[/tex]
Split up the integral over [tex][0,1)^2[/tex] along the curves [tex]xy=\frac{n-1}n[/tex]. The subregions somewhat resemble the layers or scales of an onion (see attached plot with the first 5 "scales").
Let [tex]S_n[/tex] denote the [tex]n[/tex]-th ([tex]n\in\Bbb N[/tex]) "scale", starting from the blue region closest to the origin and counting diagonally upward in the direction of (1, 1).
In Cartesian coordinates, the integral over [tex]n[/tex]-th "scale" is
[tex]\displaystyle \iint_{S_n} x \left(\frac1{1-xy} - n\right) \, dy \, dx \\\\\\ ~~~~~~~~= \int_{(n-1)/n}^{n/(n+1)} \int_{(n-1)/(nx)}^1 x \left(\frac1{1-xy} - n\right) \, dy dx \\\\\\ ~~~~~~~~~~~~~ + \int_{n/(n+1)}^1 \int_{(n-1)/(nx)}^{n/((n+1)x)} x \left(\frac1{1-xy} - n\right) \, dx[/tex]
(see attached plot of the 2nd "scale" for reference)
The integral is trivial, so I'll leave it to you to confirm that it drastically reduces to
[tex]\displaystyle \iint_{S_n} x \left(\frac1{1-xy} - n\right) \, dy \, dx = \frac1{2n (n+1)^2} = \frac12 \left(\frac1n - \frac1{n+1} - \frac1{(n+1)^2}\right)[/tex]
Now we recover the original integral by summing over [tex]\Bbb N[/tex].
[tex]\displaystyle \int_0^1 \int_0^1 x \left\{\frac1{1-xy}\right\} \, dy \, dx = \frac12 \sum_{n=1}^\infty \left(\frac1n - \frac1{n+1} - \frac1{(n+1)^2}\right) \\\\ ~~~~~~~~ = \frac12 \left(\left(1-\frac12\right)+\left(\frac12-\frac13\right)+\left(\frac13-\frac14\right)+\cdots\right) - \frac12 \sum_{n=2}^\infty \frac1{n^2} \\\\ ~~~~~~~~ = \frac12 - \frac12 \left(\sum_{n=1}^\infty \frac1{n^2} - 1\right) \\\\ ~~~~~~~~ = \frac12 - \frac12 \left(\frac{\pi^2}6 - 1\right) = \boxed{1 - \frac{\pi^2}{12}}[/tex]
