Let D be a finite set of integers. The distance graph G(D) has the set of integers as vertices and two vertices at distance d ∈D are adjacent in G(D). A conjecture of Xuding Zhu states that if the chromatic number of G (D) achieves its maximum value |D|+1 then the graph has a clique of order |D|. We prove that the chromatic number of a distance graph with D={ a,b,c,d} is five if and only if either D={ 1,2,3,4k} or D={ a,b,a+b,a+2b} with a ≡0 mod 2 and b ≡1 mod 2. This confirms Zhu's conjecture for |D|=4.