Hi, I have the chance to work on Mediterranean species and I own many olives.
Generally olives are strong trees but they are highly susceptible to three factors: level of drainage of the soil mix, air temperature and light exposure. They grow slowly in clay soils and soils with high water-retention capacity: olives’ roots love air and dry soils (best mix in my opinion: 30% pumice - 30% lava - 30% compost). Consider that many olives thrive in dry sunny lands (i.e. Southern Italy, Greece). The soil mix you are using, in my opinion, looks a bit compact and sandy; moreover the depth of the pot could (my guess) enhance water retention in the root area. If this is the case, leaves turn yellow and fall. I’d change the watering schedule and place it in full sun until it recovers and then repot it in a different soil mix.