A Brief History Of The User-Agent

In the beginning there was NCSA Mosaic…and the user agent string already existed.

NCSA Mosaic (Mosaic for short) was released in 1993 and is credited with triggering the initial explosion in web popularity.

In 1990, Tim Berners-Lee developed the first web browser, WorldWideWeb (later renamed Nexus). This browser was quickly followed up by Line Mode Browser in 1991 and subsequently by other pioneering browsers such as MidasWWW, ViolaWWW, Erwise and Cello. By the time Mosaic was released, HTTP request headers already facilitated the optional User-Agent field.

Mosaic’s initial user agent string was something similar to “NCSA_Mosaic/1.0” – a simpler time. The User-Agent field was straightforward, it comprised the product name followed by an optional slash and then the version.

The User-Agent field was introduced for analytical purposes and for identifying issues. Additionally, it was explicitly recommended that it “should be included”. As detailed in the W3C HTTP archive (1992), referring to the User-Agent field:

"This line if present gives the software program used by the original client. This is for statistical purposes and the tracing of protocol violations. It should be included. The first white space delimited word must be the software product name, with an optional slash and version designator. Other products which form part of the user agent may be put as separate words.

        <field>   =   User-Agent: <product>+

        <product> =   <word> [/<version>]

        <version> =   <word>

Example:

               User-Agent:  LII-Cello/1.0  libwww/2.5"

Different tokens could, and began to, be added to the user agent string, new browsers with different capabilities began to enter the market, and people began using the user agent string to identify and declare compatibility. The time of the simple, straightforward user agent string had ended.

In late 1993, the definition of the User-Agent HTTP header field was updated to combat this, going so far as to detail that “a user agent SHOULD NOT generate a User-Agent field containing needlessly fine-grained detail…overly long and detailed User-Agent field values increase request latency and the risk of a user being identified against their wishes (‘fingerprinting’).”

Fast forward to today and user agents look more like:

Mozilla/5.0 (Windows Phone 10.0; Android 6.0.1; Microsoft; RM-1152) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Mobile Safari/537.36 Edge/15.15254

This is a far cry from “NCSA_Mosaic/1.0”, however, it’s more reflective of the modern era, with our increasingly fragmented device landscape and capability options, in addition to our drive to continually enhance end-user experiences.

Related article: User Agent Parsing: How It Works And How It Can Be Used

Another Plan To Deprecate And Freeze The User-Agent

The User-Agent field is accessible in the HTTP header, which means that user agent strings can be parsed. Some people have a bad impression of user agent parsing due to, among other reasons, limitations around accuracy, performance, maintenance and identifying device capabilities. Valuable and actionable device insights can only be unravelled accurately by sustained and dedicated attention to mapping and interpreting entropy (DeviceAtlas does this, which enables you to focus more on delivering value to your end users).

Building and maintaining in-house device detection solutions that leverage user agent parsing can be very difficult and off-putting for many. However, user agent analysis is common practice with some 82% of the Alexa 100 using Adaptive Web Design in their websites.

There has been increasing discussions around accuracy and privacy recently in relation to user agents particularly due to Google’s intent to deprecate and freeze the user agent (read the Chrome feature description, here) gaining greater public awareness.

Apple decided to freeze the user agent string in late 2017, but they reversed their decision once they were educated by the industry as to the impacts – this Twitter thread gives an insight into industry feedback received. Google’s decision to freeze the user agent is also being challenged by industry, however, they are continuing with their progress (feedback can be logged, here).

It appears that trillion-dollar companies are willing to force through changes to web standards that will materially impact smaller players, even if these changes limit the capabilities of those smaller players. Apple listened to the industry and reversed their decision, but it is yet to be seen if Google will follow suit.

It is proposed now, however, that “Client Hints” replace the user agent.

- Replacing The User Agent

A key goal of freezing the user agent and introducing Client Hints is to increase privacy by reducing fingerprinting. However, it is not yet clear that this will yield a net improvement on privacy – read the reported issue “Client-Hints exposes fingerprint values to additional parties and logging sensitive locations” for more details.

The freezing of the user agent and requiring the implementation of Client Hints is a significant upheaval of the device detection landscape. An upheaval that does not appear to have had sufficient consideration.

For example, as of writing, there will be a negative impact on performance as Client Hints are only available to the server on the second request – the ability to serve optimised content on the first request will be compromised. See the reported issue “Performance impact of moving User-Agent header to Client Hint headers” for more details. This will have a massive impact on the online advertising industry.

It’s not clear that there has been sufficient consultation with industry to ensure that the purported benefits justify the change. Additionally, it’s not clear that the proposed changes have been assessed from a competition perspective and there’s a lack of clarity around considerations to enable businesses to adapt effectively within the proposed timeframe.

What is clear, however, is that this is an evolving initiative that can impact countless companies. As we mentioned to our customers, support for Client Hints in DeviceAtlas is planned, with availability targeting Google’s announced dates.

Conclusion

Companies have been leveraging the user agent, since its inception in the early 90’s, to enhance their end user’s experience. User agent strings have been growing in complexity in line with the increasingly complex device landscape. With the reported aim of increasing accuracy and privacy, which is being debated, the user agent could get the cold shoulder from Google’s proposed freezing.

It is not clear that the planned replacement, Clients Hints, has been given sufficient external consultation to ensure that the purported benefits justify the change. What is clear, however, is that the upcoming changes, if not reversed (like Apple’s attempt), will have a substantial impact on the web and on the bottom line of countless businesses. However, DeviceAtlas customers can rest assured that they’ll continue to receive the most accurate device detection and intelligence solution on the market – DeviceAtlas is indexed on a range of keys: user agent string, make/model, TAC, and it is planned to add Client Hints as a further index.