User behaviour tracking using etags

Introduction 

Tracking in e-commerce apps is essential to increment ROI in ads campaigns and also to have information to increment sales numbers. At Ensolvers, for one of our customers we had to implement a very complex tracking system using a plethora of methods like cookies, pixels, fingerprinting and ETags. 

These approaches vary slightly between each other, some of the most common are

• cookies being created when visiting a certain page and then used to get that information and know where the user had been
• transparent pixels, that, when rendered, send all the information available at that point and allow the platform to univocally identify a person
• fingerprinting, using a browser resources and libraries to get a simple “fingerprint” (supposed to be unique to each browser/PC), generating a hash from it, and then use it to identify that browser on subsequent requests. 
• ETags, (the ones in which we are focusing on here) where a couple of headers are used to save a hash on the communication between the client and the backend, allowing the backend to know if the requests are coming from the same client.

‍

What are ETags and why they are used?

ETag (aka entity tag) is an HTTP response header used as an identifier for a specific version of a resource; i.e it tells the browser if a resource was changed, and if that’s the case, ignore the cache and get the new content. This allows to save bandwidth, avoiding to send the full response and making caching more efficient.

ETags mechanism is implemented as a header exchange - depicted in the picture below. It starts with the client asking for a resource (1), then the server responding with the resource and adding an ETag header (hash-like, depending on implementation) representing the version of the resource provided (2). All the following calls from the client will include that generated header so the server can compare and match the value to see if the resource is the same or was changed (3-5). If the resource is still the same, the server will reply (4) with a 304 NOT MODIFIED status and no further information included; otherwise it will send (6) the new resource data with a 200 OK status and the new ETag value, which will be updated on the client side.

‍

As you can see, except for the hash value, no extra information from the client is involved, making the use of ETags not only an excellent tracking option, but a modern solution to comply with privacy policies.

The first question the reader may ask is: if we already have cookies for this, why are ETags required? One of the most important reasons, in particular in the e-commerce area, is that the use of cookies is being slowly restricted starting with the recent GDPR regulations in the EU making some of our tracking methods non-compliant with this document. 

As the problem started to rise on the tracking end, we decided to combine the classical sales tracking methods with ETags as a back, to individually attribute a session to each potential buyer (usually called "lead" in e-commerce jargon).

Implementation + code

Our custom solution is not as straightforward as it sounds on the previous section. We had made some tweaks to adapt the ETag approach to our old version and combine them to get even better results. Some of these mechanisms include:

• Cookies: classic cookies usage when allowed
• ETags: replacing the cookies with the ETag exchange using the ETag hash as a session
• Cookies-ETag hybrid: both mechanisms used, not only storing cookies on the client side, but also using the ETags header.
• Fingerprint: implemented to use as explained above; didn’t work as expected due to new privacy policies

The next code snippet shows, in a Java-like pseudocode, how a response is built when the ETag header is present. 

‍

‍

As you can see, we don’t modify the ETag hash ever; that’s because once we create an ETag for a specific browser, we don’t want to change it so we keep tracking the next requests as the same “source”. Where we do create and include the new sessionId and the header is when the request doesn’t have the “If-None-Match” header, meaning that the “communication” is new and no tracking was started. Also, we can see that, besides the ETag, we are sharing no personal or private information from the user, since we are only interested in tracking the results of the campaign to improve it in the future.

Caveats and Results

Adding this mechanism and combining it with some others that we described above allowed us to improve tracking results considerably for our customer. However, with ETags there are two cases considered as new communication that imposes some limitations on this technique. The first one arises when a specific browser performs the very first communication with the server - so, as with cookies, we have no way of associate this request with previous ones. The other one is strictly related to the functionality to which ETags were meant to: caching; i.e. when the client clears the browser cache or request an explicit resource reload, flushing away every resource stored for the site and requesting them again, as a fresh communication.

Conclusion

According to our experience, the ETags + cookies approach:

• Is definitely a good choice if you want to track clients in an efficient and simple way. 
• Allows a server to recognize the same client in each subsequent request without using cookies or any other non-compliant privacy method. 
• No personal data extracted from the client making the communication anonymous. 
• It is easy to implement and add to every existing app without the trouble of adding new libraries or frameworks, just a couple of headers.

In general, if you want a simple tracking method to know where different requests are coming from and uniquely identify them, this is one approach that you should consider.